Behavior recognition model training method, behavior recognition method, and related device

Abstract

The application provides a behavior recognition model training method, detection data of V human body key points of a single human body instance in a single frame sample image and an adjacency relationship between the V human body key points are acquired; a spatial graph convolution model is constructed, the spatial graph convolution model comprises a backbone network and a classification head, the backbone network comprises a plurality of spatial graph convolution units, the spatial graph convolution unit comprises a feature dimension increasing convolution layer, a spatial graph convolution layer and a feature fusion convolution layer connected in series, and the spatial graph convolution layer is determined according to the adjacency relationship between the V human body key points; the spatial graph convolution model is trained through a training data set, and a behavior recognition model is obtained after training. Only single frame human body key points are used as an input mode, time features do not need to be extracted, compared with a video input mode, the deployment cost is lower, processes such as visual tracking and id matching are not needed, the behavior recognition process is more efficient, the demand for computing resources is reduced, and the method is suitable for deployment on a low-end embedded chip.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for training a behavior recognition model, a behavior recognition method, and related equipment. Background Technology

[0002] Human behavior recognition is widely used in visual surveillance, autonomous driving, video retrieval, and human-computer interaction. Early on, human behavior recognition primarily used RGB / grayscale video as input. In recent years, the rise of deep learning has significantly improved the accuracy of human behavior recognition. However, using video as input requires a high-performance environment, making it difficult to deploy on low- to mid-range embedded chips. Human keypoints, on the other hand, are simple, information-rich, and robust to background, making them an important component of behavior recognition input modalities. Graph convolution is a deep learning model focused on human keypoints. In behavior recognition, since behaviors often contain temporal information, human keypoints are multi-frame inputs. Graph convolution learns the temporal and spatial features of human keypoints across multiple frames, combining temporal feature extraction, spatial feature extraction, visual tracking, and ID matching to determine the behavior category. However, in some static actions, such as sitting, standing, and lying down, temporal information is not needed. Using traditional graph convolution would extract redundant temporal features across multiple frames and perform visual tracking and ID matching, requiring more computational resources and reducing the efficiency of static behavior recognition. Summary of the Invention

[0003] This invention provides a method for training a behavior recognition model, aiming to solve the problem that in existing static behavior recognition processes, graph convolution extracts redundant temporal features across multiple frames, requiring more computational resources and reducing the efficiency of static behavior recognition. This method extracts spatial features of key points in a single frame using a spatial graph convolution model, and then classifies these key points using a classification head to achieve recognition. A spatial graph convolution layer is constructed based on the adjacency relationships between human key points. This layer extracts spatial features between human key points that conform to static behavior, and classifies behavior categories using these spatial features. It uses only single-frame human key points as the input modality, eliminating the need for a temporal convolution layer to extract temporal features, resulting in lower deployment costs compared to video input modalities. Furthermore, it is geared towards static behavior, requiring only single-frame key points as input, eliminating processes such as visual tracking and ID matching. The spatial graph convolution model's behavior recognition process is more efficient, reducing computational resource requirements and making it suitable for deployment on low- to mid-range embedded chips.

[0004] In a first aspect, embodiments of the present invention provide a method for training a behavior recognition model, wherein the behavior recognition model is used for behavior recognition of a single frame image, and the method includes:

[0005] Obtain the human keypoint detection results of at least one human instance in a single frame sample image. The human keypoint detection results of each human instance include the detection data of V human keypoints, where V is greater than 1, and each human instance corresponds to a behavior label.

[0006] A spatial graph convolutional model is constructed, which includes a backbone network and a classification head. The backbone network includes multiple spatial graph convolutional units. Each spatial graph convolutional unit includes a cascaded feature upscaling convolutional layer, a spatial graph convolutional layer, and a feature fusion convolutional layer. The spatial graph convolutional layer is determined based on the adjacency relationship between the V human body key points. The classification head includes a fully connected layer related to the number of behavior categories.

[0007] The training dataset is determined based on the human keypoint detection results of at least one human instance in the single-frame sample image and the behavior labels corresponding to each human instance. The spatial graph convolution model is trained using the training dataset, and the behavior recognition model is obtained after training is completed.

[0008] Optionally, the construction of the spatial graph convolution model includes:

[0009] Based on the adjacency relationships between the V human body key points, construct the adjacency matrix corresponding to the V human body key points;

[0010] Using the adjacency matrix as the spatial graph convolution kernel, a spatial graph convolutional layer is obtained, wherein the spatial graph convolutional layer includes the spatial graph convolution kernel;

[0011] The feature up-dimensionality convolutional layer, the spatial graph convolutional layer, and the feature fusion convolutional layer are sequentially connected in series to obtain a spatial graph convolutional unit.

[0012] Multiple spatial graph convolutional units are sequentially connected in series to obtain a backbone network;

[0013] Based on the number of behavior categories, a corresponding fully connected layer is constructed, and a classification layer is constructed based on the fully connected layer. The classification layer is then connected to the backbone network to obtain a spatial graph convolutional model.

[0014] Optionally, constructing the adjacency matrix corresponding to the V human body key points based on the adjacency relationships between the V human body key points includes:

[0015] The first adjacency matrix is ​​constructed based on the adjacency relationship between key points of the same human body.

[0016] The central key point is determined from the V human body key points, and a second adjacency matrix is ​​constructed based on the adjacency relationship between two different human body key points pointing to the central key point.

[0017] A third adjacency matrix is ​​constructed based on the adjacency relationship between two different human body key points facing away from the central key point.

[0018] Based on the first adjacency matrix, the second adjacency matrix, and the third adjacency matrix, the adjacency matrices corresponding to the key points of the V individuals are determined.

[0019] Optionally, the detection data includes multiple data components corresponding to key points on the human body.

[0020] The step of determining the training dataset based on the human keypoint detection results of at least one human instance in the single-frame sample image and the behavior labels corresponding to each human instance includes:

[0021] For each human body instance in the single frame sample image, tensor reconstruction processing is performed on the detection data of the V human body key points based on multiple data components to obtain sample tensor data corresponding to each human body instance. The sample tensor data includes multiple channels, and each data component corresponds to one channel.

[0022] The training dataset is obtained based on the sample tensor data corresponding to the human body instance and the behavior label corresponding to the human body instance.

[0023] Optionally, training the spatial graph convolutional model using the training dataset to obtain the behavior recognition model after training includes:

[0024] The sample tensor data is fed into the spatial convolution model, and the sample tensor data is subjected to feature-incrementing convolution processing through the feature-incrementing convolution layer to obtain the dimension-incrementing features of the sample tensor data. The dimension of the dimension-incrementing features is greater than the dimension of the sample tensor data.

[0025] The spatial graph convolutional layer is used to perform spatial convolution processing on the dimensionality-upgraded features to obtain multiple sub-spatial features of the sample tensor data.

[0026] The feature fusion convolutional layer performs fusion convolution processing on multiple subspace features to obtain the spatial fusion features of the sample tensor data;

[0027] The spatial fusion features are classified using the classification layer to obtain the classification result of the sample tensor data;

[0028] Calculate the loss value between the classification result of the sample tensor data and the corresponding behavior label of the sample tensor data. Based on the loss value, update and iterate the model parameters of the spatial convolution model. When the loss value reaches the preset loss value or the number of iterations reaches the preset number of iterations, stop training and obtain the trained spatial graph convolution model as the behavior recognition model.

[0029] Secondly, embodiments of the present invention provide a behavior recognition method, the behavior recognition method comprising the following steps:

[0030] Acquire an image to be identified, wherein the image to be identified includes at least one human body to be identified;

[0031] Human key point detection processing is performed on the image to be identified to obtain the human key point detection results of at least one human body to be identified in the image to be identified. The human key point detection results of each human body to be identified include the detection data of V human key points, where V is greater than 1.

[0032] The human key point detection results of at least one human body in the image to be identified are input into the behavior recognition model for behavior recognition processing to obtain the behavior recognition result of the image to be identified. The behavior recognition model is trained by any of the behavior recognition model training methods in the embodiments of the present invention.

[0033] Thirdly, embodiments of the present invention provide a behavior recognition model training device, wherein the behavior recognition model is used for behavior recognition of a single frame image, and the behavior recognition model training device includes:

[0034] The first acquisition module is used to acquire the human key point detection results of at least one human instance in a single frame sample image. The human key point detection results of each human instance include the detection data of V human key points, where V is greater than 1, and each human instance corresponds to a behavior label.

[0035] The model construction module is used to construct a spatial graph convolutional model, which includes a backbone network and a classification head. The backbone network includes multiple spatial graph convolutional units, and each spatial graph convolutional unit includes a cascaded feature upscaling convolutional layer, a spatial graph convolutional layer, and a feature fusion convolutional layer. The spatial graph convolutional layer is determined based on the adjacency relationship between the V human body key points. The classification head includes a fully connected layer related to the number of behavior categories.

[0036] The model training module is used to determine the training dataset based on the human keypoint detection results of at least one human instance in the single-frame sample image and the behavior labels corresponding to each human instance, and to train the spatial graph convolutional model using the training dataset, thereby obtaining the behavior recognition model.

[0037] Fourthly, embodiments of the present invention provide a behavior recognition device, the behavior recognition device comprising:

[0038] The first acquisition module is used to acquire an image to be identified, wherein the image to be identified includes at least one human body to be identified.

[0039] The detection module is used to perform human key point detection processing on the image to be identified, and obtain the human key point detection results of at least one human body to be identified in the image to be identified. The human key point detection results of each human body to be identified include the detection data of V human key points, where V is greater than 1.

[0040] The recognition module is used to input the human key point detection results of at least one human body in the image to be recognized into the behavior recognition model for behavior recognition processing, so as to obtain the behavior recognition result of the image to be recognized. The behavior recognition model is trained by any of the behavior recognition model training methods in the embodiments of the present invention.

[0041] Fifthly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the behavior recognition model training method provided in embodiments of the present invention, or when the processor executes the computer program, it implements the steps in the behavior recognition method provided in embodiments of the present invention.

[0042] Sixthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps in the behavior recognition model training method provided in embodiments of the present invention, or when the computer program is executed by a processor, it implements the steps in the behavior recognition method provided in embodiments of the present invention.

[0043] In this embodiment of the invention, the detection results of human keypoints of at least one human instance in a single frame sample image are obtained. The detection results of human keypoints of each human instance include detection data of V human keypoints, where V is greater than 1, and each human instance corresponds to a behavior label. A spatial graph convolutional model is constructed, which includes a backbone network and a classification head. The backbone network includes multiple spatial graph convolutional units, and each spatial graph convolutional unit includes a cascaded feature upscaling convolutional layer, a spatial graph convolutional layer, and a feature fusion convolutional layer. The spatial graph convolutional layer is determined based on the adjacency relationship between the V human keypoints. The classification head includes a fully connected layer related to the number of behavior categories. A training dataset is determined based on the detection results of human keypoints of at least one human instance in a single frame sample image and the behavior labels corresponding to each human instance. The spatial graph convolutional model is trained using the training dataset, and the behavior recognition model is obtained after training. This method extracts spatial features of key points in a single frame using a spatial graph convolutional model, and then classifies these key points using a classification head to achieve recognition. A spatial graph convolutional layer is constructed based on the adjacency relationships between human key points. This layer extracts spatial features that conform to static behaviors between human key points, and then classifies behaviors using these spatial features. It uses only single-frame human key points as input modalities, eliminating the need for temporal convolutional layers to extract temporal features, resulting in lower deployment costs compared to video input modalities. Furthermore, it is geared towards static behaviors, requiring only single-frame key points as input, eliminating processes such as visual tracking and ID matching, making the behavior recognition process more efficient and reducing computational resource requirements. This makes it suitable for deployment on low- to mid-range embedded chips. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart of a behavior recognition model training method provided in an embodiment of the present invention;

[0046] Figure 2 This is a flowchart of a behavior recognition method provided in an embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram of the structure of a behavior recognition model training device provided in an embodiment of the present invention;

[0048] Figure 4 This is a schematic diagram of the structure of a behavior recognition device provided in an embodiment of the present invention;

[0049] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] like Figure 1 As shown, Figure 1 This is a flowchart illustrating a method for training a behavior recognition model according to an embodiment of the present invention. The behavior recognition model is used for behavior recognition in a single frame image, and the training method for this behavior recognition model includes the following steps:

[0052] 101. Obtain the human keypoint detection results of at least one human instance in a single frame sample image.

[0053] In this embodiment of the invention, the above-mentioned behavior recognition model training method can be applied to a server. After training, the behavior recognition model can be applied to the behavior recognition task of a single frame image. The behavior recognition task of a single frame image can be understood as a static behavior recognition task. The static behavior can include behaviors such as sitting, standing, lying down, leaning, and squatting. The static behavior refers to the action behavior that can be determined without time.

[0054] A certain number of single-frame sample images can be collected as a sample dataset. The sample dataset includes multiple single-frame sample images and corresponding behavior labels for the single-frame sample images. Each single-frame sample image includes at least one human instance, and each human instance corresponds to a behavior label. The behavior label is used to represent the behavior category corresponding to the single-frame sample image. The behavior category can be a static behavior category, which can include behaviors such as sitting, standing, lying down, leaning, and squatting.

[0055] For each single-frame sample image, human keypoint detection can be performed using a human keypoint detection model to obtain the human keypoint detection results for at least one human instance in the single-frame sample image. For a single human instance, V human keypoint detection data can be selected from the human keypoint detection results according to the order or category of human keypoints, where V is greater than 1. The human keypoint detection model can be MoveNet, BlazePose, PoseNet, or other human keypoint detection models. Furthermore, the aforementioned V human keypoints can be the V human keypoints with the highest correlation obtained from correlation analysis of static behavior, or the V human keypoints obtained by dimensionality reduction after principal component analysis of static behavior, or the V human keypoints obtained based on empirical rules, or the V human keypoints obtained based on the annotation rules of various datasets. The aforementioned human keypoint detection results include the detection data of human keypoints, which can be the location, confidence score, and other data of each human keypoint.

[0056] For example, the aforementioned key points of the human body can include: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle, totaling 17 key points. These 17 key points can be represented sequentially using numbers 0-16. For instance, the 17 key points can be represented in the following order: 0 nose, 1 left eye, 2 right eye, 3 left ear, 4 right ear, 5 left shoulder, 6 right shoulder, 7 left elbow, 8 right elbow, 9 left wrist, 10 right wrist, 11 left hip, 12 right hip, 13 left knee, 14 right knee, 15 left ankle, and 16 right ankle.

[0057] 102. Construct a spatial graph convolution model.

[0058] In this embodiment of the invention, the spatial graph convolutional model may include a backbone network and a classification head. The backbone network may include multiple spatial graph convolutional units. The spatial graph convolutional units may include a series of feature upscaling convolutional layers, spatial graph convolutional layers, and feature fusion convolutional layers. The spatial graph convolutional layers are determined based on the adjacency relationships between the V human body key points. The classification head includes a fully connected layer related to the number of behavior categories.

[0059] It should be noted that graph convolutional models are primarily used to process node graphs, which include nodes and the connections between them. Using a node graph as input to a graph convolutional model, in this embodiment of the invention, human keypoints can be treated as nodes, and the adjacency relationships between these keypoints can be used as the connections between nodes. Thus, the human keypoints of a single frame image are treated as a node graph and input into the graph convolutional model. The aforementioned spatial graph convolutional model can be understood as a graph convolutional model that only extracts spatial features between human keypoints for behavior classification, without involving the extraction of temporal features. Specifically, the aforementioned spatial graph convolutional model includes a backbone network and a classification head. The backbone network is used to extract spatial features between human keypoints, and the classification head is used to classify the extracted spatial features to obtain the corresponding behavior category, which is then used as the behavior recognition result.

[0060] The aforementioned backbone network includes multiple spatial convolutional units connected in series. These spatial convolutional units include feature upscaling convolutional layers, spatial graph convolutional layers, and feature fusion convolutional layers.

[0061] The aforementioned feature-enhancing convolutional layer is used to enhance the dimensionality of the input human key point detection data to obtain higher-dimensional features. The aforementioned feature-enhancing convolutional layer can be composed of K×K convolutional kernels, where K is greater than or equal to 1. The convolutional kernel of the aforementioned feature-enhancing convolutional layer can be a 1×1 convolutional kernel.

[0062] The aforementioned spatial graph convolutional layer is used to extract the spatial relationships between key points on the human body. Specifically, the spatial graph convolutional layer extracts spatial features from the increased-dimensional features to obtain the spatial features between key points on the human body. This spatial graph convolutional layer can be constructed using V×V convolutional kernels, which are determined by the adjacency relationships between V key points on the human body. It should be noted that these V key points on the human body have adjacency relationships, which indicate the proximity between each key point. For example, the left elbow is adjacent to the left wrist and left shoulder, respectively. These adjacent key points are directly connected on the human body. Specifically, a V×V matrix can be constructed, where a matrix unit v of the V×V matrix... i,j This represents the adjacency value between the i-th and j-th human keypoints. If an adjacency exists, the value is 1; otherwise, it is 0. By utilizing the adjacency relationships between human keypoints to construct a spatial graph convolutional layer, we can focus more on the spatial features between human keypoints under different types of actions, thereby extracting more accurate spatial features and improving the accuracy of behavior recognition in single-frame images.

[0063] The aforementioned feature fusion convolutional layer is used to perform feature fusion operations on different spatial features to obtain fused features. The aforementioned feature fusion convolutional layer can be composed of K×K convolutional kernels, where K is greater than or equal to 1, and the convolutional kernel of the aforementioned feature fusion convolutional layer can be a 1×1 convolutional kernel.

[0064] The aforementioned classification head is used to classify the fused features to obtain the behavior category corresponding to a single frame sample image. This classification head may include a fully connected layer, which comprises M output neurons, each corresponding to a behavior category. The fully connected layer maps the fused features to obtain classification scores for each category, and the behavior category with the highest classification score is determined as the behavior category corresponding to the single frame sample image. In some possible embodiments, an adaptive mean pooling layer may be included before the fully connected layer. This adaptive mean pooling layer can sample the feature dimension of the integrated features to 1×1, facilitating mapping by the fully connected layer.

[0065] 103. The training dataset is determined based on the detection results of human key points of at least one human instance in a single frame sample image and the behavior labels corresponding to each human instance. The spatial graph convolution model is trained using the training dataset, and the behavior recognition model is obtained after training is completed.

[0066] In this embodiment of the invention, a certain number of single-frame sample images can be collected as a sample dataset. The sample dataset includes multiple single-frame sample images, each of which includes at least one human instance. Each human instance corresponds to a behavior label, and each single-frame sample image corresponds to at least one behavior label. The behavior label is used to represent the behavior category corresponding to the human instance. The behavior category can be a static behavior category, which may include behaviors such as sitting, standing, lying down, leaning, and squatting. Human keypoint detection is performed on each single-frame sample image in the sample dataset to obtain the human keypoint detection results for at least one human instance in the single-frame sample image. For a single human instance, the detection data of V human keypoints of each human instance in each single-frame sample image can be associated with the behavior label corresponding to each human instance in the human keypoint detection results to obtain a training dataset.

[0067] The spatial graph convolutional model is trained in a supervised manner using the training dataset. After training, the trained spatial graph convolutional model is used as the behavior recognition model. The V human key points corresponding to each human instance in a single frame of the image to be recognized are sequentially input into the behavior recognition model. The behavior recognition model outputs the behavior category and category confidence of each human instance in a single frame of the image to be recognized.

[0068] Specifically, the detection data of V human keypoints for a single human instance can be represented by [C,1,V], where C represents the detection data of human keypoints, 1 represents a single frame sample image, and V represents V human keypoints. The detection data of human keypoints [C,1,V] is input into a spatial graph convolutional model. The feature dimension-upgrading convolutional layer changes the feature dimension of the input human keypoints [C,1,V] to obtain the dimension-upgraded features [T,C]. out ,1,V],C out The feature dimension of [T, V, V] is greater than the feature dimension of C. The spatial graph convolutional layer uses the adjacency matrix [T, V, V] as the convolution kernel, where T represents the number of channels in the adjacency matrix, and extracts spatial features through matrix calculation. Specifically, the increased-dimensional features [T, C]... out Matrix operations are performed between the convolution kernels [1,V] and [T,V,V], and these matrix operations occur between the keypoint feature subspace [1,V] and the adjacent matrix subspace [V,V], resulting in T [C] kernels. out The subspace features of [1,V] are expressed as [T,C] out [1,V]. Feature fusion convolutional layer: Convolutional layer fuses T spatial features to obtain fused features [C]. out ,1,V]. Fusion features [C out [1,V] is mapped from the fully connected layer in the classification head to [class_num], where class_num is the number of behavior categories. After performing softmax on the class_num dimension, a one-hot vector is obtained, and its maximum value is the classification score. The classification category indexed by the maximum value is the classification category of a single human instance. The classification category of a single human instance is determined as the recognition result of a single human instance. The recognition result of a single human instance and the behavior label of a single human instance are used to calculate the loss value between the recognition result and the behavior label of a single human instance. The loss value between the recognition result and the behavior label of a single human instance is determined as the loss value of a single human instance. The loss value of a single frame image is obtained by taking the loss values ​​of all human instances in a single frame sample image. With the goal of minimizing the loss value of a single frame image, the model parameters of the spatial graph convolutional model are updated through the backpropagation algorithm. The above parameter update process is iterated until the loss value of a single frame image converges at the minimum or the number of iterations reaches the preset number. Training stops, and the trained spatial graph convolutional model is obtained. The trained spatial graph convolutional model is determined as the behavior recognition model. In this case, the kernel parameters in the spatial graph convolutional layer do not participate in the updating of model parameters, that is, the adjacency relation values ​​in the adjacency matrix [T,V,V] are not updated.

[0069] In this embodiment of the invention, the detection results of human keypoints of at least one human instance in a single frame sample image are obtained. The detection results of human keypoints of each human instance include detection data of V human keypoints, where V is greater than 1, and each human instance corresponds to a behavior label. A spatial graph convolutional model is constructed, which includes a backbone network and a classification head. The backbone network includes multiple spatial graph convolutional units, and each spatial graph convolutional unit includes a cascaded feature upscaling convolutional layer, a spatial graph convolutional layer, and a feature fusion convolutional layer. The spatial graph convolutional layer is determined based on the adjacency relationship between the V human keypoints. The classification head includes a fully connected layer related to the number of behavior categories. A training dataset is determined based on the detection results of human keypoints of at least one human instance in a single frame sample image and the behavior labels corresponding to each human instance. The spatial graph convolutional model is trained using the training dataset, and the behavior recognition model is obtained after training. This method extracts spatial features of key points in a single frame using a spatial graph convolutional model, and then classifies these key points using a classification head to achieve recognition. A spatial graph convolutional layer is constructed based on the adjacency relationships between human key points. This layer extracts spatial features that conform to static behaviors between human key points, and then classifies behaviors using these spatial features. It uses only single-frame human key points as input modalities, eliminating the need for temporal convolutional layers to extract temporal features, resulting in lower deployment costs compared to video input modalities. Furthermore, it is geared towards static behaviors, requiring only single-frame key points as input, eliminating processes such as visual tracking and ID matching, making the behavior recognition process more efficient and reducing computational resource requirements. This makes it suitable for deployment on low- to mid-range embedded chips.

[0070] Optionally, in the step of constructing the spatial graph convolutional model, an adjacency matrix corresponding to V human keypoints can be constructed based on the adjacency relationship between V human keypoints; the adjacency matrix can be used as the spatial graph convolution kernel to obtain a spatial graph convolutional layer, which includes the spatial graph convolution kernel; the feature upscaling convolutional layer, the spatial graph convolutional layer, and the feature fusion convolutional layer can be sequentially concatenated to obtain a spatial graph convolutional unit; multiple spatial graph convolutional units can be sequentially concatenated to obtain a backbone network; a corresponding fully connected layer can be constructed based on the number of behavior categories, and a classification layer can be constructed based on the fully connected layer. The classification layer can then be connected to the backbone network to obtain the spatial graph convolutional model.

[0071] In this embodiment of the invention, the aforementioned V key human body points are adjacent to each other. This adjacency relationship indicates the proximity between each key human body point. For example, the left elbow is adjacent to both the left wrist and left shoulder. These adjacent key human body points are directly connected on the body. There is no third key human body point and no non-body gap between any two adjacent key human body points. Similarly, there is no third key human body point and no non-body gap between the left elbow and left wrist. If the left hip and left wrist were directly connected, there would be a non-body gap, therefore there is no adjacency relationship between the left hip and left wrist. Specifically, a V×V matrix can be constructed, where a matrix element v of the V×V matrix... i,j This represents the adjacency value between the i-th human keypoint and the j-th human keypoint. If an adjacency exists, the adjacency value is 1; otherwise, the adjacency value is 0.

[0072] Taking V individuals as an example, with 17 key points (0 for nose, 1 for left eye, 2 for right eye, 3 for left ear, 4 for right ear, 5 for left shoulder, 6 for right shoulder, 7 for left elbow, 8 for right elbow, 9 for left wrist, 10 for right wrist, 11 for left hip, 12 for right hip, 13 for left knee, 14 for right knee, 15 for left ankle, and 16 for right ankle), a 17×17 adjacency matrix can be constructed.

[0073] After obtaining the adjacency matrix, it is used as the convolution kernel of the spatial graph convolutional layer. After obtaining the spatial graph convolutional layer, the feature upscaling convolutional layer, the spatial graph convolutional layer, and the feature fusion convolutional layer are sequentially concatenated to obtain a spatial graph convolutional unit. Multiple spatial graph convolutional units are concatenated to obtain the backbone network. After the backbone network, a classification head is concatenated to obtain the spatial graph convolutional model.

[0074] In this embodiment of the invention, a spatial graph convolutional layer is constructed by utilizing the adjacency relationship between key points of the human body. This allows for greater focus on the spatial features between key points of the human body under different types of actions, thereby extracting more accurate spatial features and improving the accuracy of behavior recognition in a single frame image.

[0075] Optionally, in the step of constructing the adjacency matrix corresponding to V human keypoints based on the adjacency relationship between V human keypoints, a first adjacency matrix can be constructed based on the adjacency relationship between keypoints of the same human body; the central keypoint is determined from the V human keypoints, and a second adjacency matrix is ​​constructed based on the adjacency relationship between two different human keypoints pointing to the central keypoint; a third adjacency matrix is ​​constructed based on the adjacency relationship between two different human keypoints facing away from the central keypoint; and the adjacency matrix corresponding to V human keypoints is determined based on the first adjacency matrix, the second adjacency matrix, and the third adjacency matrix.

[0076] In this embodiment of the invention, the adjacency relationship between the same human body key points refers to the adjacency relationship between a human body key point and itself, and the first adjacency matrix is ​​an identity matrix. The first adjacency matrix can be illustrated in Table 1, as shown in Table 1:

[0077] Table 1

[0078] 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 4 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 5 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 8 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 9 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 10 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 11 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 12 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 13 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 14 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 15 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1

[0079] In Table 1, 0-16 represent 17 corresponding human body key points, and matrix unit v i,j This represents the adjacency value between the i-th and j-th human keypoints. When i ≠ j, there is no adjacency relationship, so the adjacency value is 0; when i = j, there is an adjacency relationship, so the adjacency value is 1. The corresponding first adjacency matrix can be determined according to Table 1 above.

[0080] The adjacency relationship between two different human body key points pointing towards the central key point refers to two adjacent human body key points in centripetal motion. That is, the direction from the first human body key point to the second human body key point is the direction pointing towards the central key point. For example, taking the nose as the central key point, the two different human body key points pointing towards the central key point can include: (left ankle, left knee), (left knee, left hip), (right ankle, right knee), (right knee, right hip), (left hip, left shoulder), (right hip, right shoulder), (left wrist, left elbow), (left elbow, left shoulder), (right wrist, right elbow), (right elbow, right shoulder), (left shoulder, nose), (right shoulder, nose), (left eye, nose), (left ear, left eye), (right eye, nose), (right ear, right eye). This can be understood as the adjacency relationship of human body key points from the limbs towards the nose. Based on the above adjacency relationship, a second adjacency matrix is ​​obtained. The second adjacency matrix is ​​an upper triangular matrix. The above second adjacency matrix can be illustrated by Table 2, as shown in Table 2:

[0081] Table 2

[0082]

[0083]

[0084] In Table 2, 0-16 represent 17 corresponding human body key points, namely: 0 nose, 1 left eye, 2 right eye, 3 left ear, 4 right ear, 5 left shoulder, 6 right shoulder, 7 left elbow, 8 right elbow, 9 left wrist, 10 right wrist, 11 left hip, 12 right hip, 13 left knee, 14 right knee, 15 left ankle, 16 right ankle, etc. The corresponding adjacency relationships are: (15 left ankle, 13 left knee), (13 left knee, 11 left hip), (1... 6 Right ankle, 14 Right knee), (14 Right knee, 12 Right hip), (11 Left hip, 5 Left shoulder), (12 Right hip, 6 Right shoulder), (9 Left wrist, 7 Left elbow), (7 Left elbow, 5 Left shoulder), (10 Right wrist, 8 Right elbow), (8 Right elbow, 6 Right shoulder), (5 Left shoulder, 0 Nose), (6 Right shoulder, 0 Nose), (1 Left eye, 0 Nose), (3 Left ear, 1 Left eye), (2 Right eye, 0 Nose), (4 Right ear, 2 Right eye). The corresponding second adjacency matrix can be determined based on Table 2 above.

[0085] The aforementioned adjacency relationship between two different human body key points facing away from the central key point refers to two adjacent human body key points in centrifugal motion. That is, the direction from the first human body key point to the second human body key point is the direction away from the central key point. For example, taking the nose as the central key point, two different human body key points facing away from the central key point could include: (left knee, left ankle), (left hip, left knee), (right knee, right ankle), (right hip, right knee), (left shoulder, left hip), (right shoulder, right hip), (left elbow, left wrist), (left shoulder) The following relationships can be understood as the adjacency relationships of key points on the human body from the limbs towards the nose: (left elbow), (right elbow, right wrist), (right shoulder, right elbow), (nose, left shoulder), (nose, right shoulder), (nose, left eye), (left eye, left ear), (nose, right eye), (right eye, right ear). Based on these relationships, a third adjacency matrix is ​​obtained. This third adjacency matrix is ​​a lower triangular matrix, and it is the inversion of the second adjacency matrix. Specifically, the third adjacency matrix can be illustrated in Table 3.

[0086] Table 3

[0087]

[0088]

[0089] In Table 2, 0-16 represent 17 corresponding human body key points, namely: 0 nose, 1 left eye, 2 right eye, 3 left ear, 4 right ear, 5 left shoulder, 6 right shoulder, 7 left elbow, 8 right elbow, 9 left wrist, 10 right wrist, 11 left hip, 12 right hip, 13 left knee, 14 right knee, 15 left ankle, 16 right ankle, etc. The corresponding adjacency relationships are: (13 left knee, 15 left ankle), (11 left hip, 13 left knee), (1... 4 Right knee, 16 Right ankle), (12 Right hip, 14 Right knee), (5 Left shoulder, 11 Left hip), (6 Right shoulder, 12 Right hip), (7 Left elbow, 9 Left wrist), (5 Left shoulder, 7 Left elbow), (8 Right elbow, 10 Right wrist), (6 Right shoulder, 8 Right elbow), (0 Nose, 5 Left shoulder), (0 Nose, 6 Right shoulder), (0 Nose, 1 Left eye), (1 Left eye, 3 Left ear), (0 Nose, 2 Right eye), (2 Right eye, 4 Right ear). The corresponding third adjacency matrix can be determined based on Table 3 above.

[0090] The first, second, and third adjacency matrices are taken as three channels, each of which is a V×V matrix. The three channels are combined to obtain the adjacency matrix [3,V,V] corresponding to the human body key points, where 3 is the number of channels and [V,V] is the tensor corresponding to the channel.

[0091] Optionally, in the step of determining the training dataset based on the detection results of human keypoints of at least one human instance in a single-frame sample image and the behavior labels corresponding to each human instance, for each human instance in the single-frame sample image, tensor reconstruction processing can be performed on the detection data of V human keypoints based on multiple data components to obtain sample tensor data corresponding to the single-frame sample image. The sample tensor data includes multiple channels, with each data component corresponding to one channel, and each single-frame sample image corresponding to one sample tensor data. The training dataset is obtained based on the sample tensor data corresponding to the human instance and the behavior labels corresponding to the human instance.

[0092] In this embodiment of the invention, the data components may include the position components and confidence components of human key points. The position components may be the horizontal and vertical coordinates of the human key points, and the confidence components are the confidence scores of the human key points. The confidence scores are output by the human key point detection model and are used to express the degree of confidence that the point is a human key point.

[0093] For a single human instance in a single frame sample image, each data component of the human keypoint can be treated as a channel. After tensor reconstruction of the detection data of each human keypoint, the sample tensor data of a single human instance can be obtained [C]. in [,1,V], where Cin This includes channels corresponding to multiple data components. In the sample tensor data, 1 represents a single frame sample image, and V represents V human body keypoints. For example, if the data components of a human body keypoint are the x-coordinate, y-coordinate, and confidence level (conf), then the x-coordinate (x) is used as one channel, the y-coordinate (y) as another channel, and the confidence level (conf) as yet another channel. Combining these three channels yields C. in If each sample image contains at least one human instance, then each sample image can correspond to at least one sample tensor data.

[0094] Sample tensor data of all human instances [C in [1,V] and the corresponding human instance's behavior label Y t By performing correlation, the training dataset is obtained. In the training dataset, the sample tensor data [C]... in The data is directly input into the training spatial graph convolutional model. It should be noted that the process of converting human keypoint detection data into sample tensor data can be integrated into the input port of the spatial graph convolutional model, allowing direct input of human keypoint detection data. Furthermore, in the training dataset, the sample tensor data corresponding to human instances and the corresponding behavior labels can be organized on a single-frame sample image basis, or directly on a sample tensor basis.

[0095] Optionally, in the step of training the spatial graph convolutional model using the training dataset to obtain the behavior recognition model, the sample tensor data can be fed into the spatial convolutional model. The sample tensor data is then subjected to feature-enhancing convolutional processing through a feature-enhancing convolutional layer to obtain enhanced features with a dimension greater than that of the sample tensor data. These enhanced features are then subjected to spatial convolution processing through a spatial graph convolutional layer to obtain multiple sub-space features of the sample tensor data. These multiple sub-space features are then fused through a feature fusion convolutional layer to obtain spatial fusion features of the sample tensor data. Finally, the spatial fusion features are classified through a classification layer to obtain the classification result of the sample tensor data. The loss value between the classification result of the sample tensor data and the corresponding behavior label is calculated. Based on the loss value, the model parameters of the spatial convolutional model are updated iteratively. Training stops when the loss value reaches a preset value or the number of iterations reaches a preset number of iterations, resulting in the trained spatial graph convolutional model as the behavior recognition model.

[0096] In this embodiment of the invention, the sample tensor data [C] can be... inThe input, 1,V]] is fed into the spatial graph convolution model, and the input sample tensor data is changed through the feature upscaling convolution layer in the spatial graph convolution unit. [C] in The feature dimensions of [1,V] are used to analyze the sample tensor data [C]. in Upsizing [1,V] yields the upsizing features [T,C] out ,1,V],C out The feature dimension is greater than C in The spatial graph convolutional layer uses the adjacency matrix [T,V,V] as the convolution kernel, where T represents the number of channels in the adjacency matrix, to extract spatial features through matrix calculations. Specifically, the upgraded features [T,C]... out Matrix operations are performed between the convolution kernels [1,V] and [T,V,V], and these operations occur between the keypoint feature subspace [1,V] and the adjacent matrix subspace [V,V], resulting in T [C] kernels. out The subspace features of [1,V] are expressed as [T,C] out [1,V]. Feature fusion convolutional layer: Convolutional layer fuses T spatial features to obtain fused features [C]. out ,1,V].

[0097] The aforementioned backbone network uses multiple spatial graph convolutional units to sequentially perform the spatial graph convolution operation, resulting in the final fused feature [C]. out ,1,V]. Final fusion features [C out The classification head [1, V] is mapped to [class_num] by the fully connected layer, where class_num is the number of behavior categories. A softmax operation on the class_num dimension yields a one-hot vector, the maximum value of which is the classification score. The classification category indexed by the maximum value is the classification category of a single human instance. The classification category of a single human instance is determined as the recognition result Y of that single human instance. The recognition result Y of a single human instance is then compared with the behavior label Y of that single human instance. t Loss calculation is performed to obtain the loss value loss(Y,Y) between the recognition result of a single human instance and the behavior label of a single human instance. t To minimize the loss value loss(Y,Y) t With the objective of , the model parameters of the spatial graph convolutional model are updated using the backpropagation algorithm. This parameter update process is iterated until the loss value loss(Y,Y) is reached. t Training stops when the model converges at the minimum or the preset number of iterations is reached, resulting in a trained spatial graph convolutional model. This trained spatial graph convolutional model is then designated as the behavior recognition model. Note that the kernel parameters in the spatial graph convolutional layers do not participate in updating the model parameters; that is, the adjacency relationship values ​​in the adjacency matrix [T,V,V] are not updated.

[0098] In the forward inference phase, for example, the backbone network may include 9 layers of spatial graph convolutional units, and the original sample tensor data C in The channel dimension is 3, and the first to fourth layers are C. out The channel dimension is 64, and the 5th to 7th layers are C. out The channel dimension is 128, with 8 to 9 layers of C. out The channel dimension is 256, resulting in a feature set of [256, 1, 17]. It should be noted that this configuration is only an example; in practice, the spatial graph convolutional layer and C can be adjusted to be larger or deeper, or smaller or shallower, depending on different needs. out The classification head consists of an adaptive mean pooling layer and a fully connected layer. The adaptive mean pooling layer samples the feature dimension to 1×1, that is, [256,1,17] is sampled to

[256] . Then, it is mapped to [class_num] by the fully connected layer, where class_num is the number of classes. After performing softmax on the class_num dimension, a one-hot vector is obtained. The maximum value is the classification score. The larger the value, the more reliable the classification result. The index of the maximum value is the classification class.

[0099] Taking batch training as an example, in the forward inference phase, N sample tensor data [N, C] are collected from the training dataset. in The spatial graph convolutional model is trained batch by batch, with [1, V] as a single batch. Specifically, the input sample tensor data [N, C] is modified by the feature-enhancing convolutional layer in the spatial graph convolutional unit. in The feature dimensions of [1, V] are used to analyze the sample tensor data [N, C]. in Upsizing is performed on [1,V] to obtain the upsizing features [N,T,C] out ,1,V],C out The feature dimension is greater than C in The spatial graph convolutional layer uses the adjacency matrix [T,V,V] as the convolution kernel, where T represents the number of channels in the adjacency matrix, to extract spatial features through matrix calculations. Specifically, the upgraded features [N,T,C]... out Matrix operations are performed between the convolution kernels [1,V] and [T,V,V], and these operations occur between the keypoint feature subspace [1,V] and the adjacent matrix subspace [V,V], resulting in T [C] kernels. out The subspace features of [1,V] are expressed as [T,C] out [1,V]. Feature fusion convolutional layer: Convolutional layer fuses T spatial features to obtain fused features [C]. out The aforementioned backbone network consists of multiple spatial graph convolutional units, which sequentially perform the spatial graph convolution operation to obtain the final fused features [N,C].out [,1,V]. Final fusion features [N,C] out [1,V] is mapped to [N,class_num] by the fully connected layer in the classification head, where class_num is the number of categories of behavior. After performing softmax on the class_num dimension, a one-hot vector is obtained, and its maximum value is the classification score. The classification category indexed by the maximum value is the classification category of the sample tensor data.

[0100] like Figure 2 As shown, Figure 2 This is a flowchart of a behavior recognition method provided in an embodiment of the present invention. The behavior recognition method includes the following steps:

[0101] 201. Obtain the image to be recognized.

[0102] In this embodiment of the invention, the behavior recognition method described above can be applied to servers or terminal devices that require behavior recognition. The image to be recognized can be an image uploaded by a user or an image captured by an image acquisition device. The behavior recognition method can be applied to behavior recognition tasks for single-frame images. These single-frame image behavior recognition tasks can be understood as static behavior recognition tasks. Static behaviors can include sitting, standing, lying down, leaning, squatting, etc., and refer to actions that can be determined without time constraints. The image to be recognized includes at least one human body to be recognized.

[0103] 202. Perform human key point detection processing on the image to be identified to obtain the human key point detection results of at least one human body in the image to be identified.

[0104] In this embodiment of the invention, human keypoint detection can be performed using a human keypoint detection model to obtain the detection results of at least one human body in the image to be identified. From the human keypoint detection results, V human keypoint detection data are selected according to the order or category of human keypoints, where V is greater than 1. The human keypoint detection model can be MoveNet, BlazePose, PoseNet, or other human keypoint detection models. Furthermore, the aforementioned V human keypoints can be the V human keypoints with the highest correlation obtained from correlation analysis of static behavior, or the V human keypoints obtained by dimensionality reduction after principal component analysis of static behavior, or the V human keypoints obtained based on empirical rules, or the V human keypoints obtained based on the annotation rules of various datasets. The aforementioned human keypoint detection results include human keypoint detection data, which can be data such as the position and confidence level of each human keypoint.

[0105] 203. Input the human key point detection results of at least one human body in the image to be identified into the behavior recognition model for behavior recognition processing to obtain the behavior recognition result of the image to be identified.

[0106] In this embodiment of the invention, the behavior recognition model is trained using any behavior recognition model training method according to this embodiment. During training, the behavior recognition model extracts spatial features of key points in a single frame using a spatial graph convolutional model, and classifies these key points using a classification head to achieve recognition. A spatial graph convolutional layer is constructed based on the adjacency relationship between human key points. This layer extracts spatial features that conform to static behavior between human key points, and classifies behavior categories using these spatial features. It only requires key points of at least one human body in a single frame of the image to be recognized as the input modality, eliminating the need for a temporal convolutional layer to extract temporal features, resulting in lower deployment costs compared to video input modalities. Furthermore, it is geared towards static behavior, requiring only key points of the human body in a single frame of the image to be recognized, eliminating processes such as visual tracking and ID matching, making the behavior recognition process more efficient and improving its effectiveness.

[0107] like Figure 3 As shown, an embodiment of the present invention provides a behavior recognition model training device, which includes:

[0108] The first acquisition module 301 is used to acquire the human key point detection results of at least one human instance in a single frame sample image. The human key point detection results of each human instance include the detection data of V human key points, where V is greater than 1, and each human instance corresponds to a behavior label.

[0109] The model construction module 302 is used to construct a spatial graph convolutional model, which includes a backbone network and a classification head. The backbone network includes multiple spatial graph convolutional units, and each spatial graph convolutional unit includes a cascaded feature upscaling convolutional layer, a spatial graph convolutional layer, and a feature fusion convolutional layer. The spatial graph convolutional layer is determined based on the adjacency relationship between the V human body key points. The classification head includes a fully connected layer related to the number of behavior categories.

[0110] The model training module 303 is used to determine the training dataset based on the human key point detection results of at least one human instance in the single frame sample image and the behavior labels corresponding to each human instance, and to train the spatial graph convolution model through the training dataset to obtain the behavior recognition model after training is completed.

[0111] Optionally, the model construction module 302 is further configured to construct an adjacency matrix corresponding to the V human keypoints based on the adjacency relationships between the V human keypoints; use the adjacency matrix as a spatial graph convolution kernel to obtain a spatial graph convolution layer, the spatial graph convolution layer including the spatial graph convolution kernel; sequentially concatenate the feature upscaling convolution layer, the spatial graph convolution layer, and the feature fusion convolution layer to obtain a spatial graph convolution unit; sequentially concatenate multiple spatial graph convolution units to obtain a backbone network; construct a corresponding fully connected layer based on the number of behavior categories, and construct a classification layer based on the fully connected layer, and connect the classification layer to the backbone network to obtain a spatial graph convolution model.

[0112] Optionally, the model construction module 302 is further configured to construct a first adjacency matrix based on the adjacency relationship between the same human body key points; determine the center key point among the V human body key points; construct a second adjacency matrix based on the adjacency relationship between two different human body key points pointing to the center key point; construct a third adjacency matrix based on the adjacency relationship between two different human body key points facing away from the center key point; and determine the adjacency matrix corresponding to the V human body key points based on the first adjacency matrix, the second adjacency matrix, and the third adjacency matrix.

[0113] Optionally, the detection data includes multiple data components corresponding to human key points. The model training module 303 is further configured to perform tensor reconstruction processing on the detection data of the V human key points for each human instance in the single frame sample image, based on the multiple data components, to obtain sample tensor data corresponding to each human instance. The sample tensor data includes multiple channels, and each data component corresponds to one channel. Based on the sample tensor data corresponding to the human instance and the behavior label corresponding to the human instance, a training dataset is obtained.

[0114] Optionally, the model training module 303 is further configured to input the sample tensor data into the spatial convolutional model, perform feature-enhanced convolution processing on the sample tensor data through the feature-enhanced convolutional layer to obtain enhanced features of the sample tensor data, wherein the dimension of the enhanced features is greater than the dimension of the sample tensor data; perform spatial convolution processing on the enhanced features through the spatial graph convolutional layer to obtain multiple sub-space features of the sample tensor data; perform fusion convolution processing on the multiple sub-space features through the feature fusion convolutional layer to obtain spatial fusion features of the sample tensor data; perform classification processing on the spatial fusion features through the classification layer to obtain the classification result of the sample tensor data; calculate the loss value between the classification result of the sample tensor data and the corresponding behavior label of the sample tensor data; update and iterate the model parameters of the spatial convolutional model based on the loss value; stop training when the loss value reaches a preset loss value or the number of iterations reaches a preset number of iterations, and obtain the trained spatial graph convolutional model as a behavior recognition model.

[0115] The behavior recognition model training device provided in this embodiment of the invention can implement all the processes of the behavior recognition model training method in the above-described method embodiments, and can achieve the same beneficial effects. To avoid repetition, it will not be described again here.

[0116] like Figure 4 As shown, an embodiment of the present invention provides a behavior recognition device, the behavior recognition model training device comprising:

[0117] The second acquisition module 401 is used to acquire an image to be identified, wherein the image to be identified includes at least one human body to be identified.

[0118] The detection module 402 is used to perform human key point detection processing on the image to be identified, and obtain the human key point detection result of at least one human body to be identified in the image to be identified. The human key point detection result of each human body to be identified includes the detection data of V human key points, where V is greater than 1.

[0119] The recognition module 403 is used to input the human key point detection results of at least one human body to be recognized into the behavior recognition model for behavior recognition processing, and to obtain the behavior recognition result of the image to be recognized. The behavior recognition model is trained by any of the behavior recognition model training methods in the embodiments of the present invention.

[0120] The behavior recognition device provided in this embodiment of the invention can implement all the processes implemented by the behavior recognition method in the above-described method embodiments, and can achieve the same beneficial effects. To avoid repetition, it will not be described again here.

[0121] See Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 5 As shown, it includes: a memory 502, a processor 501, and a computer program for training a behavior recognition model or a behavior recognition method stored in the memory 502 and capable of running on the processor 501, wherein:

[0122] The processor 501 calls the computer program stored in the memory 502 for the behavior recognition model training method and executes the following steps:

[0123] Obtain the human keypoint detection results of at least one human instance in a single frame sample image. The human keypoint detection results of each human instance include the detection data of V human keypoints, where V is greater than 1, and each human instance corresponds to a behavior label.

[0124] A spatial graph convolutional model is constructed, which includes a backbone network and a classification head. The backbone network includes multiple spatial graph convolutional units. Each spatial graph convolutional unit includes a cascaded feature upscaling convolutional layer, a spatial graph convolutional layer, and a feature fusion convolutional layer. The spatial graph convolutional layer is determined based on the adjacency relationship between the V human body key points. The classification head includes a fully connected layer related to the number of behavior categories.

[0125] The training dataset is determined based on the human keypoint detection results of at least one human instance in the single-frame sample image and the behavior labels corresponding to each human instance. The spatial graph convolution model is trained using the training dataset, and the behavior recognition model is obtained after training is completed.

[0126] Optionally, the constructed spatial graph convolutional model executed by processor 501 includes:

[0127] Based on the adjacency relationships between the V human body key points, construct the adjacency matrix corresponding to the V human body key points;

[0128] Using the adjacency matrix as the spatial graph convolution kernel, a spatial graph convolutional layer is obtained, wherein the spatial graph convolutional layer includes the spatial graph convolution kernel;

[0129] The feature up-dimensionality convolutional layer, the spatial graph convolutional layer, and the feature fusion convolutional layer are sequentially connected in series to obtain a spatial graph convolutional unit.

[0130] Multiple spatial graph convolutional units are sequentially connected in series to obtain a backbone network;

[0131] Based on the number of behavior categories, a corresponding fully connected layer is constructed, and a classification layer is constructed based on the fully connected layer. The classification layer is then connected to the backbone network to obtain a spatial graph convolutional model.

[0132] Optionally, the step of processor 501 constructing an adjacency matrix corresponding to the V human body key points based on the adjacency relationships between the V human body key points includes:

[0133] The first adjacency matrix is ​​constructed based on the adjacency relationship between key points of the same human body.

[0134] The central key point is determined from the V human body key points, and a second adjacency matrix is ​​constructed based on the adjacency relationship between two different human body key points pointing to the central key point.

[0135] A third adjacency matrix is ​​constructed based on the adjacency relationship between two different human body key points facing away from the central key point.

[0136] Based on the first adjacency matrix, the second adjacency matrix, and the third adjacency matrix, the adjacency matrices corresponding to the key points of the V individuals are determined.

[0137] Optionally, the detection data includes multiple data components corresponding to human key points. The processor 501 executes the human key point detection results based on at least one human instance in the single-frame sample image and the behavior labels corresponding to each human instance to determine the training dataset, including:

[0138] For each human body instance in the single frame sample image, tensor reconstruction processing is performed on the detection data of the V human body key points based on multiple data components to obtain sample tensor data corresponding to each human body instance. The sample tensor data includes multiple channels, and each data component corresponds to one channel.

[0139] The training dataset is obtained based on the sample tensor data corresponding to the human body instance and the behavior label corresponding to the human body instance.

[0140] Optionally, the processor 501 performs the step of training the spatial graph convolutional model using the training dataset to obtain the behavior recognition model, including:

[0141] The sample tensor data is fed into the spatial convolution model, and the sample tensor data is subjected to feature-incrementing convolution processing through the feature-incrementing convolution layer to obtain the dimension-incrementing features of the sample tensor data. The dimension of the dimension-incrementing features is greater than the dimension of the sample tensor data.

[0142] The spatial graph convolutional layer is used to perform spatial convolution processing on the dimensionality-upgraded features to obtain multiple sub-spatial features of the sample tensor data.

[0143] The feature fusion convolutional layer performs fusion convolution processing on multiple subspace features to obtain the spatial fusion features of the sample tensor data;

[0144] The spatial fusion features are classified using the classification layer to obtain the classification result of the sample tensor data;

[0145] Calculate the loss value between the classification result of the sample tensor data and the corresponding behavior label of the sample tensor data. Based on the loss value, update and iterate the model parameters of the spatial convolution model. When the loss value reaches the preset loss value or the number of iterations reaches the preset number of iterations, stop training and obtain the trained spatial graph convolution model as the behavior recognition model.

[0146] The electronic device provided in this embodiment of the invention can implement all the processes of the behavior recognition model training method in the above-described method embodiments, and can achieve the same beneficial effects. To avoid repetition, it will not be described again here.

[0147] Alternatively, processor 701 may invoke the computer program for the behavior recognition method stored in memory 702 to perform the following steps:

[0148] Acquire an image to be identified, wherein the image to be identified includes at least one human body to be identified;

[0149] Human key point detection processing is performed on the image to be identified to obtain the human key point detection results of at least one human body to be identified in the image to be identified. The human key point detection results of each human body to be identified include the detection data of V human key points, where V is greater than 1.

[0150] The human key point detection results of at least one human body in the image to be identified are input into the behavior recognition model for behavior recognition processing to obtain the behavior recognition result of the image to be identified. The behavior recognition model is trained by any of the behavior recognition model training methods in the embodiments of the present invention.

[0151] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the behavior recognition model training method or behavior recognition method provided in this invention and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0152] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0153] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for training a behavior recognition model, characterized in that, The behavior recognition model is used for behavior recognition in a single frame image, and the method includes the following steps: Obtain the human keypoint detection results of at least one human instance in a single frame sample image. The human keypoint detection results of each human instance include the detection data of V human keypoints, where V is greater than 1, and each human instance corresponds to a behavior label. A spatial graph convolutional model is constructed, which includes a backbone network and a classification head. The backbone network includes multiple spatial graph convolutional units. Each spatial graph convolutional unit includes a cascaded feature upscaling convolutional layer, a spatial graph convolutional layer, and a feature fusion convolutional layer. The spatial graph convolutional layer is determined based on the adjacency relationship between the V human body key points. The classification head includes a fully connected layer related to the number of behavior categories. The training dataset is determined based on the human keypoint detection results of at least one human instance in the single frame sample image and the behavior labels corresponding to each human instance. The spatial graph convolution model is trained using the training dataset, and the behavior recognition model is obtained after training is completed. The construction of the spatial graph convolution model includes: Based on the adjacency relationships between the V human body key points, construct the adjacency matrix corresponding to the V human body key points; Using the adjacency matrix as the spatial graph convolution kernel, a spatial graph convolutional layer is obtained, wherein the spatial graph convolutional layer includes the spatial graph convolution kernel; The feature up-dimensionality convolutional layer, the spatial graph convolutional layer, and the feature fusion convolutional layer are sequentially connected in series to obtain a spatial graph convolutional unit. Multiple spatial graph convolutional units are sequentially connected in series to obtain a backbone network; Based on the number of behavior categories, a corresponding fully connected layer is constructed, and a classification layer is constructed based on the fully connected layer. The classification layer is then connected to the backbone network to obtain a spatial graph convolutional model.

2. The behavior recognition model training method as described in claim 1, characterized in that, The step of constructing the adjacency matrix corresponding to the V human body key points based on the adjacency relationships between the V human body key points includes: The first adjacency matrix is ​​constructed based on the adjacency relationship between key points of the same human body. The central key point is determined from the V human body key points, and a second adjacency matrix is ​​constructed based on the adjacency relationship between two different human body key points pointing to the central key point. A third adjacency matrix is ​​constructed based on the adjacency relationship between two different human body key points facing away from the central key point. Based on the first adjacency matrix, the second adjacency matrix, and the third adjacency matrix, the adjacency matrices corresponding to the key points of the V individuals are determined.

3. The behavior recognition model training method as described in claim 1 or 2, characterized in that, The detection data includes multiple data components corresponding to human key points. The step of determining the training dataset based on the human key point detection results of at least one human instance in the single-frame sample image and the behavior labels corresponding to each human instance includes: For each human body instance in the single frame sample image, tensor reconstruction processing is performed on the detection data of the V human body key points based on multiple data components to obtain sample tensor data corresponding to each human body instance. The sample tensor data includes multiple channels, and each data component corresponds to one channel. The training dataset is obtained based on the sample tensor data corresponding to the human body instance and the behavior label corresponding to the human body instance.

4. The behavior recognition model training method as described in claim 3, characterized in that, The step of training the spatial graph convolutional model using the training dataset to obtain the behavior recognition model includes: The sample tensor data is input into the spatial graph convolution model, and the sample tensor data is subjected to feature dimensionality-upgrading convolution processing through the feature dimensionality-upgrading convolution layer to obtain the dimensionality-upgrading features of the sample tensor data. The dimension of the dimensionality-upgrading features is greater than the dimension of the sample tensor data. The spatial graph convolutional layer is used to perform spatial convolution processing on the dimensionality-upgraded features to obtain multiple sub-spatial features of the sample tensor data. The feature fusion convolutional layer performs fusion convolution processing on multiple subspace features to obtain the spatial fusion features of the sample tensor data; The spatial fusion features are classified using the classification layer to obtain the classification result of the sample tensor data; Calculate the loss value between the classification result of the sample tensor data and the corresponding behavior label of the sample tensor data. Based on the loss value, update and iterate the model parameters of the spatial graph convolution model. When the loss value reaches the preset loss value or the number of iterations reaches the preset number of iterations, stop training and obtain the trained spatial graph convolution model as the behavior recognition model.

5. A behavior recognition method, characterized in that, The behavior recognition method includes the following steps: Acquire an image to be identified, wherein the image to be identified includes at least one human body to be identified; Human key point detection processing is performed on the image to be identified to obtain the human key point detection results of at least one human body to be identified in the image to be identified. The human key point detection results of each human body to be identified include the detection data of V human key points, where V is greater than 1. The human key point detection results of at least one human body in the image to be identified are input into the behavior recognition model for behavior recognition processing to obtain the behavior recognition result of the image to be identified. The behavior recognition model is trained according to any one of the behavior recognition model training methods in claims 1 to 4.

6. A behavior recognition model training device, characterized in that, The behavior recognition model is used for behavior recognition in a single frame image, and the behavior recognition model training device includes: The first acquisition module is used to acquire the human key point detection results of at least one human instance in a single frame sample image. The human key point detection results of each human instance include the detection data of V human key points, where V is greater than 1, and each human instance corresponds to a behavior label. The model construction module is used to construct a spatial graph convolutional model, which includes a backbone network and a classification head. The backbone network includes multiple spatial graph convolutional units, and each spatial graph convolutional unit includes a cascaded feature upscaling convolutional layer, a spatial graph convolutional layer, and a feature fusion convolutional layer. The spatial graph convolutional layer is determined based on the adjacency relationship between the V human body key points. The classification head includes a fully connected layer related to the number of behavior categories. The model training module is used to determine the training dataset based on the human key point detection results of at least one human instance in the single frame sample image and the behavior labels corresponding to each human instance, and to train the spatial graph convolution model using the training dataset, thereby obtaining the behavior recognition model after training is completed. The model construction module is further configured to construct an adjacency matrix corresponding to the V human keypoints based on the adjacency relationships between the V human keypoints; use the adjacency matrix as a spatial graph convolution kernel to obtain a spatial graph convolution layer, the spatial graph convolution layer including the spatial graph convolution kernel; sequentially concatenate the feature upscaling convolution layer, the spatial graph convolution layer, and the feature fusion convolution layer to obtain a spatial graph convolution unit; sequentially concatenate multiple spatial graph convolution units to obtain a backbone network; construct a corresponding fully connected layer based on the number of behavior categories, and construct a classification layer based on the fully connected layer, and connect the classification layer to the backbone network to obtain a spatial graph convolution model.

7. A behavior recognition device, characterized in that, The behavior recognition device includes: The first acquisition module is used to acquire an image to be identified, wherein the image to be identified includes at least one human body to be identified. The detection module is used to perform human key point detection processing on the image to be identified, and obtain the human key point detection results of at least one human body to be identified in the image to be identified. The human key point detection results of each human body to be identified include the detection data of V human key points, where V is greater than 1. The recognition module is used to input the human key point detection results of at least one human body in the image to be recognized into the behavior recognition model for behavior recognition processing, so as to obtain the behavior recognition result of the image to be recognized. The behavior recognition model is trained according to the behavior recognition model training method according to any one of claims 1 to 4.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the behavior recognition model training method as described in any one of claims 1 to 4, or the processor executes the computer program to implement the steps in the behavior recognition method as described in claim 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps in the behavior recognition model training method as described in any one of claims 1 to 4, or, when executed by a processor, implements the steps in the behavior recognition method as described in claim 5.

Images