Universal action recognition method and device based on semantic feature and structural feature matching, computer equipment and readable storage medium

Abstract

The invention discloses a universal action recognition method and device based on semantic feature and structural feature matching, computer equipment and a readable storage medium, and the method comprises the steps: firstly obtaining a to-be-detected picture, and obtaining a human body position region through a human body recognition model; and respectively extracting semantic and structural features of a to-be-detected action through an action self-encoding model and a human body posture estimation model, acquiring semantic and structural features of a reference action from the reference feature library, performing feature matching based on the semantic and structural features to determine a target reference picture, and taking a corresponding action category as a recognition result of the to-be-detected picture. And semantic and structural features are integrated, so that the accuracy and universality of action recognition are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a general action recognition method, device, computer equipment and readable storage medium based on matching of semantic features and structural features. Background Art

[0002] In today's digital age, motion recognition is of great significance in many fields, such as security monitoring, intelligent human-computer interaction, sports analysis, etc. Traditional motion recognition methods often have limitations, such as relying on only a single feature for recognition, resulting in insufficient accuracy. Summary of the invention

[0003] The object of the present invention is to provide a general action recognition method, device, computer equipment and readable storage medium based on matching of semantic features and structural features.

[0004] In a first aspect, an embodiment of the present invention provides a general action recognition method based on matching of semantic features and structural features, comprising:

[0005] Acquire a picture to be tested, and input the picture to be tested into a pre-trained human body recognition model to obtain a human body position area;

[0006] Calling a pre-trained action autoencoder model to extract features of the human body position area to obtain semantic features of the action to be tested;

[0007] Calling a pre-trained human posture estimation model to extract features of the human body position area to obtain structural features of the action to be measured;

[0008] Acquire benchmark action semantic features and benchmark action structural features corresponding to a plurality of benchmark images from a preset benchmark feature library, each of the benchmark images being marked with a corresponding benchmark action category;

[0009] According to the multiple benchmark action semantic features and the multiple benchmark action structural features, feature matching is performed on the semantic features of the action to be tested and the structural features of the action to be tested, a target benchmark picture with the highest matching degree is determined, and the target benchmark action category corresponding to the target benchmark picture is used as the action recognition result of the picture to be tested.

[0010] In a possible implementation, the action autoencoder model is obtained in the following manner, including:

[0011] Constructing an initial action autoencoder model, the initial action autoencoder model includes five cascaded downsampling components and five upsampling components, the downsampling component is a Conv-BN-PRelu-Conv-BN-PRelu structure, the upsampling component is a DeConv-BN-PRelu-DeConv-BN-PRelu structure, each of the downsampling components and each of the upsampling components is configured with a channel attention component, and the channel attention component includes a global pooling layer, a fully connected layer, and a normalization layer;

[0012] The initial action autoencoder model is trained based on an unsupervised mechanism until a preset mean square error loss function training termination condition is reached to obtain the action autoencoder model.

[0013] In a possible implementation, calling a pre-trained human posture estimation model to extract features of the human body position area to obtain structural features of the action to be measured includes:

[0014] Calling a pre-trained human posture estimation model to perform key point detection on the human body position area to obtain multiple human body key point position information;

[0015] Determine the corresponding adjacency matrix and angle matrix according to the position information of the plurality of human body key points;

[0016] The structural features of the action to be measured are calculated according to the preset key point weights, the adjacency matrix and the angle matrix.

[0017] In a possible implementation manner, the step of calculating the structural features of the action to be measured according to the preset key point weights, the adjacency matrix, and the angle matrix includes:

[0018] According to the formula: S2 = A n×n D n×n W n×n Calculate and obtain the structural characteristics of the action to be tested;

[0019] Among them, S2 is the structural feature of the action to be tested, A n×n is an adjacency matrix, which is used to characterize the relationship between n key points of the human body. n×n is an angle matrix used to represent the horizontal angles of the lines connecting n key points of the human body. n×n The preset key weights.

[0020] In a possible implementation, the method of performing feature matching on the semantic features of the action to be tested and the structural features of the action to be tested according to the plurality of the benchmark action semantic features and the plurality of the benchmark action structural features, determining a target benchmark picture with the highest matching degree, and taking the target benchmark action category corresponding to the target benchmark picture as the action recognition result of the picture to be tested, includes:

[0021] Calculating the semantic cosine similarity between the semantic feature of the action to be tested and the semantic feature of each benchmark action;

[0022] Calculating the structural cosine similarity between the semantic feature of the action to be tested and each structural feature of the benchmark action;

[0023] Determine whether there is a pending reference picture whose semantic cosine similarity and structural cosine similarity are both greater than a preset cosine similarity threshold;

[0024] If it exists, taking the undetermined reference picture as the target reference picture, and taking the target reference action category corresponding to the target reference picture as the action recognition result of the picture to be tested;

[0025] If not, the action recognition result of the image to be tested is determined as an unknown action category.

[0026] In a possible implementation, the method further includes:

[0027] In the case where there are multiple pending reference pictures, calculating the Euclidean distance between the pending action semantic feature and the pending action structural feature corresponding to each pending reference picture and the semantic feature and the structural feature of the action to be tested;

[0028] According to the calculation result of the Euclidean distance, the undetermined reference picture with the highest matching degree is determined as the target reference picture.

[0029] In a possible implementation manner, obtaining the image to be tested includes:

[0030] Obtain the image to be tested and the business scenario corresponding to the image to be tested;

[0031] The obtaining of the benchmark action semantic features and the benchmark action structural features corresponding to the plurality of benchmark images from the preset benchmark feature library includes:

[0032] According to the business scenario, benchmark action semantic features and benchmark action structural features corresponding to a plurality of benchmark images are obtained from a preset benchmark feature library.

[0033] In a second aspect, an embodiment of the present invention provides a general action recognition device based on matching of semantic features and structural features, comprising:

[0034] An acquisition module is used to acquire a picture to be tested, and input the picture to be tested into a pre-trained human body recognition model to obtain a human body position area; call a pre-trained action autoencoder model to extract features from the human body position area to obtain semantic features of the action to be tested; call a pre-trained human posture estimation model to extract features from the human body position area to obtain structural features of the action to be tested; obtain benchmark action semantic features and benchmark action structural features corresponding to multiple benchmark pictures from a pre-set benchmark feature library, each of the benchmark pictures is marked with a corresponding benchmark action category;

[0035] The recognition module is used to perform feature matching on the semantic features of the action to be tested and the structural features of the action to be tested according to multiple semantic features of the benchmark actions and multiple structural features of the benchmark actions, determine the target benchmark image with the highest matching degree, and use the target benchmark action category corresponding to the target benchmark image as the action recognition result of the image to be tested.

[0036] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor and a non-volatile memory storing computer instructions, wherein when the computer instructions are executed by the processor, the computer device executes the method described in the first aspect.

[0037] In a fourth aspect, an embodiment of the present invention provides a readable storage medium, wherein the readable storage medium includes a computer program, and when the computer program is executed, the computer device where the readable storage medium is located is controlled to execute the method described in the first aspect.

[0038] Compared with the prior art, the beneficial effects provided by the present invention include: adopting a general action recognition method, device, computer equipment and readable storage medium based on semantic feature and structural feature matching disclosed by the present invention, obtaining a human body position area by acquiring a picture to be tested through a human body recognition model, and then extracting the semantic and structural features of the action to be tested respectively through an action autoencoder model and a human body posture estimation model, obtaining the semantic and structural features of the benchmark action from a benchmark feature library, determining the target benchmark picture based on feature matching of the two, and using the corresponding action category as the recognition result of the picture to be tested, and improving the accuracy and versatility of action recognition by combining semantic and structural features. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can also be obtained based on these drawings without creative work.

[0040] Figure 1 A schematic diagram of the steps of a general action recognition method based on matching of semantic features and structural features provided by an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of a framework of a general action recognition method based on matching of semantic features and structural features provided by an embodiment of the present invention;

[0042] Figure 3 A schematic diagram of the overall structure of the autoencoder model provided by an embodiment of the present invention;

[0043] Figure 4 A schematic diagram of the component structure of the autoencoder model provided in an embodiment of the present invention;

[0044] Figure 5 A schematic diagram of key points of a human body provided by an embodiment of the present invention;

[0045] Figure 6 A schematic block diagram of the structure of a general action recognition device based on semantic feature and structural feature matching provided by an embodiment of the present invention;

[0046] Figure 7 A schematic block diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0048] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.

[0049] In order to solve the technical problems in the aforementioned background technology, Figure 1 A flowchart of a general action recognition method based on matching of semantic features and structural features provided in an embodiment of the present disclosure is provided. The general action recognition method based on matching of semantic features and structural features is introduced in detail below.

[0050] Step S201, obtaining a picture to be tested, and inputting the picture to be tested into a pre-trained human body recognition model to obtain a human body position area;

[0051] Step S202, calling a pre-trained action autoencoder model to extract features of the human body position area to obtain semantic features of the action to be tested;

[0052] Step S203, calling a pre-trained human posture estimation model to extract features of the human body position area to obtain structural features of the action to be measured;

[0053] Step S204, obtaining benchmark action semantic features and benchmark action structural features corresponding to a plurality of benchmark images from a preset benchmark feature library, wherein each of the benchmark images is marked with a corresponding benchmark action category;

[0054] Step S205, based on the multiple benchmark action semantic features and the multiple benchmark action structural features, feature matching is performed on the semantic features of the action to be tested and the structural features of the action to be tested, the target benchmark picture with the highest matching degree is determined, and the target benchmark action category corresponding to the target benchmark picture is used as the action recognition result of the picture to be tested.

[0055] In the embodiment of the present invention, for example, it is assumed that the server is processing a surveillance video analysis task of a large shopping mall to identify various actions of customers in the mall. The server obtains pictures to be tested from the mall's monitoring system. These pictures to be tested include scenes of various areas of the mall, such as pictures of different locations such as the mall entrance, corridors, and store interiors.

[0056] The server has already trained a human body recognition model based on a large amount of image data containing human bodies. When the server inputs the acquired test image into the human body recognition model, the model starts to recognize the human body.

[0057] For example, for a test image of a shopping mall entrance, the model will detect the location area of ​​all people in the image. If the image shows a family of three walking into a shopping mall, the model will accurately mark the body contour range of each of the three people, that is, the human body location area. This area can be represented by a rectangular box or an irregular polygon, which accurately delineates the position of the human body in the image, providing a basis for subsequent action analysis.

[0058] The training process of the human body recognition model involves deep learning algorithms. It learns various characteristics of the human body through technologies such as convolutional neural networks (CNN). During the training process, a large number of labeled human images are input into the network, and the network continuously adjusts its internal weight parameters to improve the accuracy of human body recognition. When faced with new test images, the model can accurately locate the location area of ​​the human body based on the learned feature patterns.

[0059] In the shopping mall monitoring scenario, after obtaining the body position area of ​​the family of three in front, the server will call the pre-trained motion autoencoder model to extract the semantic features of the motion.

[0060] This action autoencoder model is built and trained according to a specific structure. For each of the three people in the family, the model will focus on the image information in the area where their body is located.

[0061] For example, for the body position area of ​​the father, if he is carrying a shopping bag and walking with his body slightly leaning forward, the action autoencoder model will deeply analyze the image pixels, textures and other information in this area. It gradually extracts the semantic features related to this action through its own downsampling component and upsampling component structure. The Conv-BN-PRelu-Conv-BN-PRelu structure of the downsampling component will perform convolution operations and normalization on the image to reduce the dimension of the image while retaining key information. The global pooling layer in the channel attention component will comprehensively count the features of the entire area, and the fully connected layer and normalization layer will further process these statistical information, so that the model can pay attention to the importance differences between different channels.

[0062] Through such a series of complex operations, the action autoencoder model extracts a unique semantic feature of the action to be tested for the action of the father walking with the shopping bag. This feature can be a high-dimensional vector that contains various information related to the semantics of the action, such as the overall rhythm of the action, the fluency of the posture, and other semantic-level representations.

[0063] The initial action autoencoder model constructed by the action autoencoder model contains five cascaded downsampling components and five upsampling components. This structure helps to gradually extract and restore image features. The structure of the downsampling component can effectively reduce the resolution of the image, while extracting key features through operations such as convolution and activation functions. The channel attention component is designed to enable the model to better focus on the important information of different channels in the image, thereby improving the accuracy of feature extraction.

[0064] When training this action autoencoder model, the training is based on an unsupervised mechanism. Unsupervised training means that there is no need to manually label the data with information such as action categories. The model continuously adjusts its own parameters by minimizing the preset mean square error loss function. For example, if the training data set contains a large number of human images with different actions, the model will try to reconstruct the features of the input image so that the mean square error between the reconstructed features and the original features is minimized. When this error reaches the preset training termination condition, the model training is considered complete, so that the action semantic features of the new human position area can be accurately extracted.

[0065] Let's continue with the example of a family of three in a shopping mall. The server calls the pre-trained human pose estimation model to extract features of their human position area.

[0066] For the mother's body position area, if she is reaching for a product on the shelf, the human posture estimation model will first perform key point detection, which can identify the position information of multiple key parts of the mother's body, such as the head, shoulders, arm joints, waist, knees, etc. The position information of these key parts is like the coordinate points of the human posture.

[0067] Suppose the model detects that the mother's right arm is straight and her left hand is placed on the side of the body. Based on the position information of these multiple key points of the human body, the model will determine the corresponding adjacency matrix and angle matrix. The adjacency matrix can represent the connection relationship between each key point, for example, the shoulder and arm joint are connected, and the angle matrix represents the horizontal angle of the line between the key points, such as the horizontal angle of the line from the shoulder to the arm joint when the arm is straight.

[0068] Then, based on the preset key point weights (this weight can be determined based on experience or during the model training process, for example, the weight of the arm joints may be relatively high because the arm movement is more important in determining the overall movement structure), the adjacency matrix, and the angle matrix, the action structural feature to be tested of the mother reaching out to take the product is calculated. This structural feature can be a numerical vector that reflects the structural relationship between the various parts of the mother's body in this action.

[0069] The human posture estimation model uses deep learning algorithms, especially variants of convolutional neural networks (CNNs), to detect key points. During the training process, the model learns key point patterns in a large amount of human posture data. By performing convolution operations and pooling operations on the input human position area image, the positions of each key point are gradually located.

[0070] The determination of the adjacency matrix and angle matrix is ​​based on the physiological structure and kinematics of the human body. There are certain connection relationships between the key points of the human body, and these relationships will show different geometric structures under different actions. By calculating the connection relationship (adjacency matrix) and angle relationship (angle matrix) between these key points, combined with the preset key point weights, a vector that can fully reflect the structural characteristics of the action can be obtained. This calculation method can accurately capture the structural changes of the human body under different actions.

[0071] In the scenario of shopping mall monitoring, the benchmark feature library of the server stores a large number of benchmark images with pre-labeled action categories. These benchmark images cover various typical action scenes that may appear in the shopping mall.

[0072] For example, a benchmark image is labeled with the benchmark action category "picking up products". This image shows a customer standing in front of a shelf, looking at the products with his eyes and picking up the products with his hands. The server will obtain the benchmark action semantic features and benchmark action structural features corresponding to this image from the benchmark feature library.

[0073] As for semantic features, they contain information related to the semantics of the action of picking up products, which can be extracted through the previous similar action autoencoder model. As for structural features, they are obtained by analyzing the human posture in the picture through the human posture estimation model, such as the customer's body slightly bent to approach the shelf, and the hands moving within a certain range, etc., which are related to the action structure.

[0074] Similarly, there are benchmark images marked as different benchmark action categories such as "payment" and "consulting the clerk". The server will obtain their corresponding benchmark action semantic features and benchmark action structural features from the benchmark feature library.

[0075] The benchmark feature library is established during the system construction and pre-training phase. When building this library, a large number of pictures covering various action scenes are first collected, and these pictures are accurately labeled with action categories. Then, the same action autoencoder model and human posture estimation model as the test pictures are used to extract features from these benchmark pictures, and benchmark action semantic features and benchmark action structural features are obtained respectively, and these features are stored in the benchmark feature library together with the corresponding benchmark action categories. In this way, when performing action recognition on the test pictures, the necessary reference features can be obtained from this library for comparative analysis.

[0076] Taking the example of a family of three in a shopping mall, suppose that the server has extracted the semantic features and structural features of the action to be tested for the action of the father walking with a shopping bag, and obtained the benchmark action semantic features and benchmark action structural features of many benchmark images from the benchmark feature library.

[0077] First, the server calculates the semantic cosine similarity between the semantic features of the action to be tested and the semantic features of each benchmark action. For example, for the benchmark action semantic features corresponding to the benchmark action category of "normal walking", the semantic cosine similarity between them is calculated. At the same time, the structural cosine similarity between the semantic features of the action to be tested and the structural features of each benchmark action is calculated.

[0078] Assuming that the semantic cosine similarity and structural cosine similarity corresponding to the benchmark action category of "normal walking" are both relatively high and are both greater than the preset cosine similarity threshold, then this benchmark image will be temporarily determined as a pending benchmark image.

[0079] If there is only this one pending benchmark picture, it will be determined as the target benchmark picture, and its corresponding benchmark action category of "normal walking" will be used as the action recognition result of the father in this test picture.

[0080] However, if there are multiple pending benchmark images, for example, there is also a benchmark image corresponding to the benchmark action category of "walking quickly to the destination" that also meets the condition that the semantic and structural cosine similarities are greater than the threshold. In this case, the server calculates the Euclidean distance between the pending action semantic features and pending action structural features corresponding to each pending benchmark image and the semantic features and structural features of the action to be tested.

[0081] Assume that after calculation, the Euclidean distance corresponding to the pending benchmark image "normal walking" is smaller, which means that it has a higher overall match with the action of the father walking with the shopping bag. Then the pending benchmark image "normal walking" will be determined as the target benchmark image, and its corresponding "normal walking" benchmark action category is the final action recognition result.

[0082] If no benchmark image satisfies the condition that both semantic cosine similarity and structural cosine similarity are greater than the preset cosine similarity threshold, the server will determine the action recognition result of the image to be tested as an unknown action category. For example, if the father makes a very strange action and there is no benchmark action with a high matching degree in the benchmark feature library, it will be marked as an unknown action category.

[0083] Semantic cosine similarity is an indicator that measures the similarity between two vectors in the semantic space. By calculating the cosine value of the angle between the semantic feature vector of the action to be tested and the semantic feature vector of the benchmark action, the semantic similarity between them can be obtained. Structural cosine similarity is similar and is used to measure the similarity between action structural features.

[0084] The preset cosine similarity threshold is determined based on experience or during the model training and verification phase, and is used to screen out possible matching reference images. The Euclidean distance calculates the straight-line distance between two vectors in space. When there are multiple reference images to be determined, the smaller the Euclidean distance, the closer the two feature vectors are overall, so that the target reference image with the highest degree of matching can be determined more accurately. This multi-stage, multi-index feature matching method can improve the accuracy and reliability of action recognition.

[0085] In the embodiment of the present invention, the action self-encoding model is obtained in the following manner and can be implemented through the following examples.

[0086] Constructing an initial action autoencoder model, the initial action autoencoder model includes five cascaded downsampling components and five upsampling components, the downsampling component is a Conv-BN-PRelu-Conv-BN-PRelu structure, the upsampling component is a DeConv-BN-PRelu-DeConv-BN-PRelu structure, each of the downsampling components and each of the upsampling components is configured with a channel attention component, and the channel attention component includes a global pooling layer, a fully connected layer, and a normalization layer;

[0087] The initial action autoencoder model is trained based on an unsupervised mechanism until a preset mean square error loss function training termination condition is reached to obtain the action autoencoder model.

[0088] In the embodiment of the present invention, illustratively, the server starts to construct an initial motion autoencoding model, which is specially designed to effectively extract semantic features of human motion.

[0089] It contains five cascaded downsampling components and five upsampling components. The design of this structure is based on the needs of image feature extraction and restoration. The downsampling component is used to gradually reduce the resolution of the input image (here is the human body position area image) and extract the key features in the image; the upsampling component is used to gradually restore the downsampled features to a feature representation close to the original image, so that a feature vector containing rich semantic information can be obtained in the end.

[0090] Downsampling component-Conv-BN-PRelu-Conv-BN-PRelu structure:

[0091] Convolutional layer (Conv):

[0092] For the first convolution layer in the downsampling component, the server sets appropriate parameters such as convolution kernel size and step size when building the model. For example, the convolution kernel size may be set to 3×3 with a step size of 1. When a human body position area image is input, this convolution layer performs convolution operations on each pixel in the image and the pixels around it. Taking the shopping mall monitoring scene as an example, if the input is a human body position area image of a customer bending over to pick up a product, the convolution layer will identify local features in the image, such as the texture of the customer's clothes, the local shape of the body contour, etc. It slides the convolution kernel on the image, calculates the convolution result of each position, and preliminarily extracts some basic features related to the action.

[0093] The second convolutional layer further performs convolution operations on the output of the first convolutional layer. For example, it can further refine the feature extraction, and may perform a deeper analysis of the features such as body contours and clothing textures extracted by the first convolutional layer, such as identifying the relationship between wrinkles on clothing and body bending movements.

[0094] Batch Normalization Layer (BN):

[0095] After each convolutional layer is a batch normalization layer. When processing image data of a large number of customers in a shopping mall, the distribution of the input data may be different due to factors such as lighting and shooting angles of different images. The batch normalization layer normalizes the features output by the convolutional layer. For example, for a group of images of customers under different lighting conditions, the batch normalization layer normalizes the feature values ​​of each batch of images after convolution so that their mean and variance are within a certain range, thereby improving the training efficiency and stability of the model.

[0096] For continuous action images of the same customer, due to possible slight changes in lighting or slight jitters of the camera, the batch normalization layer can also ensure that the model has consistent feature processing for these images.

[0097] PRelu activation function:

[0098] After each convolutional layer and batch normalization layer is the PRelu activation function. The PRelu activation function introduces nonlinear factors to the features. In the shopping mall scene, when the model processes images of customers in different postures, such as the customer's movement from standing to squatting, the shape and posture of the body change nonlinearly. The PRelu activation function can better capture this nonlinear relationship, allowing the model to better learn the feature representation under different motion states. Without the activation function, the model can only process linear relationships and cannot accurately represent complex human motion features.

[0099] Upsampling component-DeConv-BN-PRelu-DeConv-BN-PRelu structure:

[0100] Deconvolution layer (DeConv):

[0101] The deconvolution layer in the upsampling component is the inverse operation of the convolution layer in the downsampling component. In the shopping mall monitoring scenario, assuming that the customer action features processed by the downsampling component are low-resolution representations, the deconvolution layer will gradually increase the resolution of the feature map. For example, for a customer bending action feature that has been downsampled to a smaller size, the deconvolution layer will expand this low-resolution feature based on the pattern it has learned, trying to restore it to a representation close to the original image feature.

[0102] The second deconvolution layer further refines the restoration process, gradually restoring more information in the original image, such as the overall outline of the customer's body and details of the posture.

[0103] Batch Normalization layer (BN) and PRelu activation function

[0104] Similar to the downsampling component, the batch normalization layer and PRelu activation function in the upsampling component also play an important role. The batch normalization layer ensures the stability of features during the deconvolution process, avoiding unstable model training due to changes in data distribution. The PRelu activation function continues to introduce nonlinear factors to the deconvolution features, allowing the model to better reconstruct the complex action semantic features in the original image.

[0105] Channel attention components - global pooling layer, fully connected layer and normalization layer:

[0106] Global pooling layer:

[0107] A channel attention component is configured in each downsampling component and upsampling component, and the global pooling layer is the first operation of the channel attention component. Taking the customer image in the mall monitoring as an example, when a customer's body position area image is input and partially processed by the downsampling component, the global pooling layer will perform a pooling operation on the entire feature map in the spatial dimension. It will compress the entire feature map into a vector of channel dimension, which contains the comprehensive information of the entire feature map in each channel. For example, for a feature map containing multiple channels (which may represent different feature dimensions, such as features of different parts of the body, different directional features of actions, etc.), the global pooling layer will count all pixel values ​​on each channel (such as averaging) to obtain a value that reflects the overall features of the entire feature map on the channel, and finally form a vector of channel dimension.

[0108] Fully connected layer:

[0109] After receiving the channel dimension vector output by the global pooling layer, the fully connected layer will perform a linear transformation on this vector. In the shopping mall scenario, the fully connected layer will readjust the importance of different channels according to the patterns that the model has learned. For example, if a channel is very important for representing the action of a customer reaching out to grab a product (such as a channel representing the direction of the arm movement), the fully connected layer will increase the weight of this channel in the entire feature representation; while for some less important channels (such as channels that may be related to background noise), their weights will be reduced.

[0110] Normalization layer:

[0111] The normalization layer normalizes the output of the fully connected layer. When processing a large number of image data of different customers and different actions in a shopping mall, the normalization layer can ensure that the features of different images after passing through the channel attention component remain consistent in the numerical range. In this way, in subsequent model operations, the combination and processing of features can be more stable, improving the model's ability to accurately extract semantic features of different actions.

[0112] The server collects a large amount of unlabeled image data to train the initial action autoencoder model. These image data come from a wide range of sources, including frames of human body movements captured from various public image datasets and online videos. In the case of shopping mall monitoring, it can also include some shopping mall internal monitoring video images with unlabeled action categories.

[0113] For example, these images contain various actions of people in different environments, such as walking in the park, walking on the street, doing various daily activities indoors, etc. These image data do not need to be pre-labeled with action categories because an unsupervised training mechanism is adopted.

[0114] During the training process, the goal of the initial action autoencoder model is to gradually compress the features of the input image through the downsampling component, and then reconstruct the feature representation of the input image as accurately as possible through the upsampling component. For each input unlabeled image (such as an image of a person bending over to pick up something in a room), the model first compresses the image features through the downsampling component. During the downsampling process, the high-resolution features of the image are gradually converted into low-resolution feature representations through the collaborative work of convolutional layers, batch normalization layers, and activation functions, while extracting key action features.

[0115] Then, the upsampling component tries to reconstruct the features of the original image based on the features obtained by downsampling. In this process, the deconvolution layer, batch normalization layer, and activation function work together to gradually restore the low-resolution features to a feature representation close to the original image. During the reconstruction process, the channel attention component continuously adjusts the weights of each channel to better focus on the channels that have an important influence on the action semantic features.

[0116] The server calculates the preset mean square error loss function. The mean square error loss function is used to measure the difference between the features reconstructed by the model and the features of the original input image. For example, for an input image containing a customer reaching for a product, the features reconstructed by the model and the features of the original image may differ in each pixel or feature dimension. The mean square error loss function calculates the average of the sum of the squares of these differences.

[0117] During the training process, the server will continuously input new unlabeled images into the model, calculate the value of the mean square error loss function, and adjust the model parameters based on this value. For example, if the mean square error between the reconstructed features and the original features of a certain image is large, it means that the model is not good at extracting and reconstructing features for this image. The model will adjust the parameters of the convolutional layers, batch normalization layers, activation functions in the downsampling and upsampling components, and the fully connected layers in the channel attention component through the back propagation algorithm to reduce the mean square error.

[0118] As unlabeled image data is continuously input for training, the value of the mean square error loss function will gradually decrease. When the value of this mean square error loss function reaches the preset training termination condition, for example, when the mean square error is reduced to a very small value (such as 0.001), it indicates that the model has been able to extract and reconstruct features of the input image well. At this time, the model training is considered to be completed and the action autoencoder model is obtained. This action autoencoder model can be used to extract action semantic features from images of customer body position areas in shopping mall monitoring.

[0119] In the embodiment of the present invention, the calling of a pre-trained human posture estimation model to perform feature extraction on the human body position area to obtain the structural features of the action to be measured can be implemented through the following examples.

[0120] Calling a pre-trained human posture estimation model to perform key point detection on the human body position area to obtain multiple human body key point position information;

[0121] Determine the corresponding adjacency matrix and angle matrix according to the position information of the plurality of key points of the human body;

[0122] The structural features of the action to be measured are calculated according to the preset key point weights, the adjacency matrix and the angle matrix.

[0123] In the embodiment of the present invention, for example, it is assumed that the server is processing a video surveillance screen of a sports event to identify the action posture of the athlete. Each frame of the image obtained from the video is used as the image to be tested. After obtaining the human body position area (for example, the human body contour area of ​​the athlete on the track in a track and field competition scene), the server calls the pre-trained human body posture estimation model to perform key point detection.

[0124] This human pose estimation model is trained based on a large amount of image data containing human pose annotations, which mark the accurate positions of multiple key points such as head, neck, shoulders, elbows, wrists, chest, waist, knees, ankles, etc.

[0125] For the human body position area of ​​track and field athletes, the model first identifies the location of the head, which can be determined by searching for specific patterns in the image, such as the outline of the hair, facial features, etc. The model then accurately locates the location of the shoulders, which may be determined by comparing them with surrounding body parts such as the neck and arms. Similarly, for the elbows and wrists, the model detects their locations based on the shape of the arms and the characteristics of the joints. The location of the waist may be determined by the curve of the body and the connection relationship with the upper and lower body parts, and the knees and ankles are located based on the bend and shape characteristics of the legs.

[0126] At the moment a high jumper takes off, the model can accurately detect the position of each key point of his body. The head is in an upward posture, the shoulders are open and raised, the elbows are bent, the wrists are in a specific position as the arms swing, the waist is bent to prepare for the jump, the knees are bent to store power, and the ankles provide support and elasticity for the jump. These key point position information is expressed in the form of coordinates, for example, in the image coordinate system, each key point has a corresponding x and y coordinate value, which accurately describes the position of each part of the athlete's body at the moment of take-off.

[0127] The human posture estimation model is usually built based on the convolutional neural network (CNN). During the training process, a large number of human posture images with annotations are input, and the model gradually learns the characteristic patterns of each key point in different human postures through operations such as convolutional layers and pooling layers.

[0128] For example, in the convolution layer, convolution kernels of different sizes scan the image and extract local features, which are then downsampled by the pooling layer to reduce the amount of data while retaining key information. As the network goes deeper, the model can learn more abstract features, thereby accurately identifying the key point locations under different human postures.

[0129] During the detection process, the model inputs the human body position area image into the network. After a series of convolution and pooling operations, the final fully connected layer or convolution layer outputs the position information of each key point. This information is expressed in the form of coordinate values ​​in the image.

[0130] Continuing with the example of the moment when a high jumper takes off, the server determines the adjacency matrix and the angle matrix based on the position information of multiple key points of the human body that have been detected.

[0131] For the adjacency matrix, the physiological structure of the human body determines that there is a natural connection relationship between each key point. For example, the head is connected to the neck, the neck is connected to the shoulder, the shoulder is connected to the elbow, the elbow is connected to the wrist, the shoulder is also connected to the chest, the chest is connected to the waist, the waist is connected to the knee, and the knee is connected to the ankle. According to this connection relationship, the elements in the adjacency matrix can be expressed as 0 or 1. If two key points are connected, the corresponding element is 1, otherwise it is 0. This matrix clearly shows the connection relationship between each key point.

[0132] For the angle matrix, the server calculates the horizontal angle of the line between the key points. For example, for the two key points of shoulder (S) and elbow (E), a coordinate system is established with the shoulder as the origin, and the angle between the line from the shoulder to the elbow and the horizontal direction (x-axis) is calculated. If the athlete's arm is bent upward at the moment of take-off, this angle can be an acute angle, assuming 30 degrees. Similarly, calculate the angles between other key points, such as the angle between the neck and the shoulder, the elbow and the wrist, etc.

[0133] In a dance performance, when dancers make complex movements, the server will also determine the adjacency matrix and angle matrix of their key points. For different dance movements, although the adjacency matrix remains unchanged based on the basic structure of the human body, the angle matrix will change greatly. For example, in a rotation, the angles of various parts of the body will continue to change as the rotation progresses, and these angle changes are accurately recorded through the angle matrix.

[0134] The adjacency matrix is ​​determined based on the knowledge of the human body's anatomical structure, which is pre-coded into the algorithm. Through simple logical judgment, the adjacency matrix is ​​constructed according to the connection relationship of each key point.

[0135] The calculation of the angle matrix is ​​based on geometric principles. In a two-dimensional image coordinate system, for the coordinates of any two key points, the angle between their connecting line and the horizontal direction (x-axis) can be calculated by trigonometric functions. By performing such calculations for each pair of connected key points, an angle matrix can be constructed.

[0136] Still taking the moment when a high jumper takes off as an example, the server calculates the structural features of the action to be tested based on the preset key point weights, adjacency matrix and angle matrix.

[0137] The preset key point weights are determined according to the importance of different key points in describing the action structure. For example, in a high jump action, the weights of the ankle, knee, and waist may be relatively high, because these parts play a key role in the power transmission and body posture adjustment during the take-off process; while the weights of the wrist and elbow may be relatively low.

[0138] By calculating all the connected key points and adding up the results, we can get the action structure feature of the high jumper at the moment of take-off. This feature is a value that combines the weight, connection relationship and angle information of each key point, which can fully describe the action structure of the athlete at the moment of take-off.

[0139] In a yoga practice scenario, different yoga postures have different action structure characteristics. For example, in the tree pose, when standing on one leg, the key point weights of the ankle, knee and waist of the standing leg are very important in maintaining body balance. By calculating the sum of the product of the adjacency matrix, angle matrix and preset weights of these parts and other related parts, the action structure characteristics of the yoga posture to be tested are obtained, so that different yoga postures can be accurately distinguished.

[0140] The principle of this calculation process is to quantify the structural information of human posture. The adjacency matrix represents the connection relationship between key points and determines which parts of the angle information are relevant; the angle matrix provides the angle information of each connected part and reflects the relative direction of each part of the body; the preset key point weights are weighted according to the importance of different key points in the action.

[0141] By calculating these three factors according to the formula, the geometric and physiological relationship of the human body movement structure is converted into a quantifiable feature vector. This feature vector can mathematically represent the structural characteristics of the movement and can be used for subsequent comparison and matching with the benchmark movement structure characteristics, thereby realizing movement recognition.

[0142] In the embodiment of the present invention, the calculation of the structural features of the action to be measured based on the preset key point weights, the adjacency matrix and the angle matrix can be implemented through the following examples.

[0143] According to the formula: S2 = A n×n D n×n W n×n Calculate and obtain the structural characteristics of the action to be tested;

[0144] Among them, S2 is the structural feature of the action to be tested, A n×n is an adjacency matrix, which is used to characterize the relationship between n key points of the human body. n×n is an angle matrix used to represent the horizontal angles of the lines connecting n key points of the human body. n×n The preset key weights.

[0145] In the embodiment of the present invention, for example, it is assumed that the server is processing video surveillance data of a dance performance, with the purpose of identifying different dance moves of the dancers.

[0146] In a dance performance, dancers make various complex movements and postures. After processing the dancer's body position area through the human posture estimation model, the server obtains the position information of multiple human key points, and then determines the adjacency matrix, angle matrix and preset key point weights.

[0147] For example, for a specific dance move, the key points of the human body include the head, neck, shoulders, elbows, wrists, waist, knees, and ankles. The adjacency matrix determines the connection relationship between these key points based on the physiological structure of the human body. For example, the head is connected to the neck, the neck is connected to the shoulders, etc. The corresponding matrix elements are 1, and the unconnected elements are 0.

[0148] The angle matrix accurately reflects the horizontal angles of the lines connecting each key point. In dance movements, these angles change a lot at different times. For example, when a dancer stretches his arms, the horizontal angles of the lines connecting the shoulder and elbow, and the lines connecting the elbow and wrist will change significantly; when a dancer bends his knees, the horizontal angles of the lines connecting the waist and knees, and the lines connecting the knees and ankles will also change.

[0149] The preset key point weights are set according to the characteristics of dance movements. In dance, the movements of the waist, shoulders and knees often have a greater impact on the entire dance posture, so the key point weights of these parts are relatively high.

[0150] According to the above formula, for all connected key point combinations, their contributions to the action structure features are calculated based on their relationships in the adjacency matrix, the angles in the angle matrix, and the preset key point weights, and these contributions are accumulated. For example, after calculating all key point combinations, the final accumulated result is the action structure feature to be tested for this hand-raising rotation action. This feature is a numerical value that integrates the weights, connection relationships, and angle information of each key point in the dance action, which can accurately describe the structural characteristics of this specific dance action.

[0151] In this way, the server can calculate the corresponding action structure features to be tested for each action posture in the dance performance, thereby providing a basis for subsequent comparison with the benchmark action structure features and action recognition. In different dance styles, such as classical dance, modern dance or folk dance, due to the differences in posture, rhythm and style of the action, the calculated action structure features will be significantly different, which helps to accurately identify and distinguish different types of dance movements.

[0152] In an embodiment of the present invention, feature matching is performed on the semantic features of the action to be tested and the structural features of the action to be tested based on a plurality of the benchmark action semantic features and a plurality of the benchmark action structural features, a target benchmark picture with the highest matching degree is determined, and the target benchmark action category corresponding to the target benchmark picture is used as the action recognition result of the picture to be tested, which can be implemented through the following examples.

[0153] Calculating the semantic cosine similarity between the semantic feature of the action to be tested and the semantic feature of each benchmark action;

[0154] Calculating the structural cosine similarity between the semantic feature of the action to be tested and each structural feature of the benchmark action;

[0155] Determine whether there is a pending reference picture whose semantic cosine similarity and structural cosine similarity are both greater than a preset cosine similarity threshold;

[0156] If it exists, taking the undetermined reference picture as the target reference picture, and taking the target reference action category corresponding to the target reference picture as the action recognition result of the picture to be tested;

[0157] If not, the action recognition result of the image to be tested is determined as an unknown action category.

[0158] In the embodiment of the present invention, for example, it is assumed that the server is processing image data in a shopping mall monitoring system to identify various actions of customers. The server has obtained the semantic features of the action to be tested through the previous steps, and obtained multiple benchmark action semantic features from the benchmark feature library.

[0159] For example, the action to be tested is the action of a customer reaching out to grab a product in front of a shelf in a shopping mall. The corresponding semantic feature of the action to be tested is a high-dimensional vector, which contains various information related to the semantics of the action, such as the smoothness of the action, the intention of extending the arm, etc.

[0160] The benchmark action semantic features in the benchmark feature library cover the corresponding semantic features of various common shopping mall actions, such as "walking", "picking goods", "queueing to pay", etc. For the semantic feature of the benchmark action of "picking goods", it is also a high-dimensional vector, which can contain semantic information such as the customer's concentration when selecting goods and the frequency of hand movements.

[0161] The server calculates the semantic cosine similarity between the semantic feature of the action to be tested and the semantic feature of the benchmark action of "selecting products".

[0162] During the calculation process, the server multiplies the corresponding elements of the two vectors and accumulates them to obtain the vector dot product, then calculates the modulus of the two vectors respectively, and finally calculates the semantic cosine similarity through the formula. For example, if the calculated semantic cosine similarity is 0.8, it means that the action to be tested has a high semantic similarity with the benchmark action of "selecting products".

[0163] The server will perform such calculations for each benchmark action semantic feature in the benchmark feature library, such as calculating the semantic cosine similarity with action semantic features such as "walking" and "queueing to pay".

[0164] Taking sports event monitoring as an example, suppose the server is analyzing the actions of athletes. The action to be tested is a basketball player shooting. The obtained structural features of the action to be tested include the structural relationship of various parts of the athlete's body at the moment of shooting, such as the extension angle of the arm, the inclination of the body, and other information.

[0165] The benchmark feature library contains benchmark action structural features of various sports actions, such as the structural features of "dribbling", "passing", "defending", etc. For the structural feature of the benchmark action "shooting", it is also a vector, which contains the structural relationship of various parts of the body in the standard shooting posture.

[0166] The server calculates the structural cosine similarity between the structural features of the action to be tested and the structural features of the benchmark action "shooting". Similarly, according to the cosine similarity formula, the similarity between the two vectors in the structural feature space is calculated. Assuming that the calculated structural cosine similarity is 0.75, this indicates that in terms of structure, the action to be tested has a certain similarity with the benchmark action "shooting".

[0167] Just like calculating semantic cosine similarity, the server performs such calculation for each benchmark action structure feature.

[0168] In the airport security scenario, the server has calculated all semantic cosine similarities and structural cosine similarities. Assume that the preset cosine similarity threshold is 0.7.

[0169] For the action of a passenger undergoing security check, the server compares it with various security check-related actions in the benchmark feature library (such as "raise hand for inspection" and "turn around", etc.). If there is a benchmark action, such as "raise hand for inspection", whose semantic cosine similarity is 0.8 and structural cosine similarity is 0.72, and both similarities are greater than the preset cosine similarity threshold of 0.7, then the benchmark image containing the action of "raise hand for inspection" is determined as the pending benchmark image.

[0170] If for all benchmark actions, there is no such situation, that is, no semantic cosine similarity and structural cosine similarity of a benchmark action are greater than the preset threshold at the same time, then it means that no action with a sufficiently high matching degree with the action to be tested is found in the existing benchmark feature library.

[0171] On the factory production line, the server determines whether there is a pending reference picture through the previous steps. If there is, for example, when identifying the action of a worker operating a certain device, it is found that the reference picture corresponding to the reference action of "operating the device button" is a pending reference picture, then this pending reference picture is determined as the target reference picture, and the reference action category of "operating the device button" is used as the recognition result of the worker's action in the image to be tested.

[0172] If there is no pending benchmark image, for example, when a new special action is detected that is not covered in the benchmark feature library (it may be a temporary unique operation of a worker), the server will determine the action recognition result of the image to be tested as an unknown action category, which will help the subsequent analysis of new actions and the update of the benchmark feature library.

[0173] In the embodiments of the present invention, the following implementation modes are also provided.

[0174] In the case where there are multiple pending reference pictures, calculating the Euclidean distance between the pending action semantic feature and the pending action structural feature corresponding to each pending reference picture and the semantic feature and the structural feature of the action to be tested;

[0175] According to the calculation result of the Euclidean distance, the undetermined reference picture with the highest matching degree is determined as the target reference picture.

[0176] In the embodiment of the present invention, for example, it is assumed that the server is responsible for identifying the actions of patients and medical staff in a hospital ward to ensure the safety and normal nursing process in the ward.

[0177] When analyzing the actions in the ward, the server has completed the previous steps, namely, calculating the semantic cosine similarity between the semantic features of the action to be tested and the semantic features of each benchmark action, and the structural cosine similarity between the structural features of the action to be tested and the structural features of each benchmark action, and determined multiple benchmark images to be determined based on a preset cosine similarity threshold.

[0178] For example, the action to be tested is the action of a nurse at the bedside, which may include elements such as leaning forward, bending the arms, and some hand manipulation. After the previous similarity calculation, it was found that the benchmark images corresponding to three benchmark actions became the benchmark images to be determined, namely "measuring the patient's temperature", "adjusting the infusion speed", and "helping the patient turn over". These three actions have a high similarity with the action to be tested in terms of semantics and structure, so the benchmark images corresponding to them are determined as the benchmark images to be determined.

[0179] For the pending benchmark image of "measuring the patient's temperature", the server obtains the corresponding pending action semantic features and pending action structural features. The pending action semantic features can include semantic information such as the nurse's concentration when measuring the temperature and the gentleness of the action; the pending action structural features include the positional relationship of the nurse's body parts in this action, such as the relative position of the arm and the patient's body, the bending angle of the body, etc.

[0180] Repeat the above process for the undetermined benchmark image of "adjusting the infusion speed". Its undetermined action semantic features include semantic information such as the nurse checking the infusion equipment and operating the adjustment device; the undetermined action structural features include structural information such as the relative position of the nurse's hand and the infusion equipment, and body posture.

[0181] The Euclidean distance of semantic features and structural features are calculated separately according to the Euclidean distance formula, and then the total Euclidean distance is obtained by adding them together.

[0182] For the pending benchmark image of "helping the patient turn over", the pending action semantic features include semantic information such as the collaboration information between the nurse and the patient and the strength of the action; the pending action structural features include the positional relationship of various parts of the nurse's body during the process of assisting in turning over.

[0183] Similarly, the Euclidean distances of semantic features and structural features are calculated and added together to obtain the total Euclidean distance.

[0184] After comparison and calculation, the server determines that the overall difference between the pending action corresponding to the pending reference image of "adjusting the infusion speed" and the action to be tested is the smallest based on the corresponding Euclidean distance, that is, the degree of match is the highest.

[0185] Therefore, the server determines the pending benchmark image of "adjusting the infusion speed" as the target benchmark image, and uses the benchmark action category of "adjusting the infusion speed" as the final recognition result of the nurse action in the test image. In this way, through the calculation of the Euclidean distance, when there are multiple pending benchmark images, the benchmark action category that best matches the test action can be more accurately determined, thereby improving the accuracy of action recognition.

[0186] Euclidean distance is a method to measure the distance between two points (here, feature vectors) in a multidimensional space. In the context of action recognition, both semantic feature vectors and structural feature vectors represent the feature representation of actions in different dimensions. By calculating the Euclidean distance, we can comprehensively consider the differences in semantics and structure.

[0187] When there are multiple undetermined reference images, the Euclidean distance between the undetermined action features corresponding to each undetermined reference image and the action features to be tested reflects the overall difference between them. The smaller the distance, the closer it is to the action to be tested in terms of semantics and structure. By comparing these Euclidean distances and selecting the undetermined reference image corresponding to the minimum distance as the target reference image, the most matching action category can be selected from multiple similar candidate actions, thereby improving the accuracy and reliability of the entire action recognition system.

[0188] In the embodiment of the present invention, the obtaining of the image to be tested may be implemented through the following example.

[0189] Obtain the image to be tested and the business scenario corresponding to the image to be tested;

[0190] The obtaining of the benchmark action semantic features and the benchmark action structural features corresponding to the plurality of benchmark images from the preset benchmark feature library includes:

[0191] According to the business scenario, benchmark action semantic features and benchmark action structural features corresponding to a plurality of benchmark images are obtained from a preset benchmark feature library.

[0192] In the embodiment of the present invention, illustratively, in an intelligent transportation system, the server is responsible for identifying the actions of pedestrians, vehicle drivers, etc. on the road.

[0193] The server obtains the images to be tested from the traffic monitoring camera network. These cameras are distributed at various intersections, main roads, and parking lots in the city. At the same time, the server obtains the business scenario information corresponding to these images to be tested.

[0194] For example, the image to be tested is taken by a surveillance camera at an intersection, and its corresponding business scenario is "intersection traffic supervision". The images to be tested in this scenario include pedestrians crossing the road, vehicles turning, and traffic police directing traffic.

[0195] For example, the image to be tested taken by the camera at the entrance of a parking lot corresponds to the business scenario of "parking lot vehicle and personnel entry and exit management". The image may include actions such as the driver parking to get a card and pedestrians guiding vehicles.

[0196] The server obtains these test images and the corresponding business scenario information in order to perform action recognition more accurately in the future. This is because the types of actions and their characteristics vary greatly in different business scenarios. In the intersection scenario, the focus is on actions related to compliance with traffic rules, such as whether pedestrians cross the road according to traffic lights, whether vehicles turn correctly, etc. In the parking lot scenario, more attention is paid to actions related to parking operations and personnel guidance.

[0197] The server receives the images to be tested from the bank's internal surveillance cameras. The corresponding business scenario is "bank business hall security and service monitoring."

[0198] The benchmark feature library stores a large number of benchmark images in different business scenarios and their corresponding benchmark action semantic features and benchmark action structural features. For the business scenario of the bank business hall, the server obtains the features of the benchmark images related to it from the benchmark feature library.

[0199] For example, the benchmark images may include images of tellers handling business for customers (such as counting money, stamping, etc.), customers operating self-service terminals (such as withdrawing money, making inquiries, etc.), security guards patrolling (such as walking, observing the surrounding environment, etc.), etc.

[0200] For the benchmark image of a teller handling business for a customer, the benchmark action semantic features include semantic information related to the business operation process. For example, the semantic features of the money counting action can include the speed and concentration of the money counting, and the semantic features of the stamping action can include the accuracy of the stamping position and the standardization of the action.

[0201] Its basic action structure characteristics include the positional relationship of various parts of the teller's body during the operation, such as the movement trajectory of the arm when counting money, the relative position of the hand, the counting machine and the coins, the swing amplitude of the arm when stamping, and the forward leaning degree of the body.

[0202] For the benchmark images of customers operating at the self-service terminal, the benchmark action semantic features contain semantic information such as the customer's operation intention and operation proficiency. For example, the degree of caution in entering the password when withdrawing money, the decisiveness in choosing the withdrawal amount, etc.

[0203] The baseline motion structure features include the relative position of the customer's body and the self-service terminal, the hand movements on the operating screen and keyboard, etc.

[0204] For the benchmark images of security patrols, the benchmark action semantic features contain semantic information such as the security guard’s alertness and the regularity of the patrol route.

[0205] The structural features of the baseline action include the size of the security guard's walking steps, the angle at which the body turns to observe the surrounding environment, etc.

[0206] By obtaining relevant benchmark action semantic features and benchmark action structural features from the benchmark feature library according to the business scenario, the server can more specifically identify the actions in the test image. For example, in the bank lobby scenario, when the server identifies the teller's actions in the test image, the benchmark action features related to the teller's business can provide an accurate reference to improve the accuracy and efficiency of action recognition. Similarly, the action recognition of customers and security guards can also be more effectively carried out within this specific business scenario framework.

[0207] In order to more clearly describe the solution provided by the embodiment of the present invention, a relatively complete implementation method is provided below.

[0208] The present invention proposes a general action recognition method based on semantic feature and structural feature matching, which implements a general feature extraction method, namely, first using the YOLOv5 model to detect the human body, locate the human body area and crop it, and then using the action autoencoder model AAEM to extract the semantic feature S1 of the human behavior action; then using the AlphaPose model to detect the key points of the human body, which is used to calculate the structural feature S2 of the key points of the human body; based on this feature extraction method, the present invention needs to extract key frames from the action video as reference pictures, and extract the semantic features and structural features of the reference pictures, and save them in a feature library. In practical applications, it is only necessary to use the same method to extract the semantic features and structural features of the picture to be tested, and then perform feature matching in the feature library of the reference picture to obtain the matching results. The implementation method technology of the present invention relates to the fields of deep learning, image processing, and computer vision technology, and can be applied to video surveillance, human-computer interaction, video behavior analysis and other fields.

[0209] To achieve the above purpose, the present invention adopts the following technical solution, which mainly includes 4 stages. Figure 2 .

[0210] The first stage is human body detection, which uses the pre-trained YOLOv5 model to detect and locate the area of ​​the human body in the image;

[0211] The second stage is to build and train an action autoencoder model AAEM to extract the semantic features S1 of human action.

[0212] The third stage is human key point detection, which uses the pre-trained AlphaPose model to detect human key points and calculate the structural features S2 of human key points;

[0213] The fourth stage is to construct a benchmark image dataset of action key frames; the benchmark images serve as reference images and templates for benchmark action recognition, and the semantic features B S1 and structural feature BS2 , and saved in the system's feature library.

[0214] The fifth stage is to achieve feature matching and extract the semantic features T of the test image. S1 and structural features T S2 ,Then feature matching is performed in the feature library of the benchmark image, and the most similar benchmark image is selected as the matching result.

[0215] The first stage is human detection: As an important application in the field of computer vision, human detection aims to accurately identify and locate the human body in an image or video. This technology is not only crucial for security monitoring, intelligent security, human-computer interaction and other fields, but also the basis for modern intelligent systems to achieve efficient action recognition and behavior analysis. To achieve this goal, this solution directly uses the pre-trained model YOLOv5, which has been trained on the COCO (Common Objects in Context) dataset for target detection tasks, covering various complex scenes and multiple object categories, including human bodies. By using the YOLOv5 model, we can quickly and accurately identify human bodies in images and accurately locate their location areas. This provides a solid foundation for subsequent action recognition, behavior analysis and other work.

[0216] The second stage is to build and train an action autoencoder model AAEM to extract the semantic features S1 of human action. The action autoencoder model consists of two parts: the encoder and the decoder. The encoder is used to map the input action picture data to a low-dimensional semantic feature space, and the decoder is used to reconstruct the representation of the low-dimensional semantic feature space back to the original action picture data. Please refer to Figure 3 and Figure 4 .

[0217] An action autoencoder model (AAEM) is constructed, in which the encoder consists of 5 downsampling modules (Block), which maps the input 224×224 RGB image to a low-dimensional semantic feature space and encodes it into a 512 feature vector; the decoder consists of 5 upsampling modules (upBlock), which reconstructs the 512 feature vector back to the original image; the Block module adopts the Conv (convolution)-BN-PRelu-Conv-BN-PRelu structure, and the upBlock module adopts the DeConv (deconvolution)-BN-PRelu-DeConv-BN-PRelu structure. These two structures can not only effectively extract the key features in the image and realize the functions of upsampling and downsampling, but also introduce the nonlinear activation function PRelu; the activation function has a learnable parameter w, which can adaptively learn the parameters of the corrected linear unit, improve the fitting ability of the model, and reduce the risk of overfitting, so that the model can learn more complex mapping relationships. In addition, both the Block and upBlock modules add a channel attention module, which assigns different importance weights to different channels, so that the model can focus on features that are more critical to the current task, thereby improving the overall feature representation ability of the model. The channel attention module is mainly composed of the Global Pooling-FC-sigmoid structure, and is specifically implemented as follows:

[0218] Global Pooling is responsible for performing a global pooling operation on the feature map of each channel to obtain a one-dimensional vector. This vector contains the feature information of all positions in the channel and is crucial for the subsequent learning process.

[0219] The fully connected layer (FC) performs a fully connected operation on the vector output by the Global Pooling layer and learns the weight of each channel through nonlinear transformation. These weights reflect the importance of different channels to the current task and are the core of the attention mechanism.

[0220] Normalization layer: It normalizes the weights output by the FC layer (such as the sigmoid function) so that the sum of the weights of all channels is 1. In this way, the weight of each channel becomes a probability value between 0 and 1, that is, the attention coefficient of each channel is obtained, which is convenient for subsequent network layers to perform weighted operations.

[0221] Attention is applied, and the obtained channel attention coefficient vector is multiplied by the original feature map by channel element-wise (element-wise multiplication), so as to weight the features of each channel. The weighted feature map retains the original spatial structure, but the influence of each channel has been adaptively adjusted.

[0222] Training the AAEM model: The AAEM model is an autoencoder model, which is a data dimension compression and feature representation method based on unsupervised learning. It does not require the image category to be labeled, so the data labeling cost of the AAEM model is almost zero, and the model can be directly trained using images of existing open source data. This solution directly uses the open source dataset ImageNet, which contains more than 14 million images and covers more than 20,000 categories. This solution uses ImageNet's ILSVRC2012_img_train as the training set and ILSVRC2012_img_val as the validation set; uses the training set to train and optimize the AAEM model; and uses the test set to test the AAEM model and evaluate the performance of the model indicators. During model training, the input data is uniformly scaled to 224×224, the batch size batch_size=256, the optimization algorithm uses the Adam optimizer, the initial learning rate is set to lr=0.001, the mean square error loss function (MSE Loss) is used, and the entire data set is iterated epoch=200 times; during the training process, the program calculates the MSE of the model in the test set once after completing a complete data iteration, and saves the model weight with the lowest mean square error MSE; in subsequent actual use, only the model weight file and the corresponding inference program code need to be retained.

[0223] Deploy the AAEM model: After training, the action autoencoder model is obtained. In actual use, this solution only needs the encoder part to extract image features, and the decoder does not participate in the inference calculation;

[0224] The third stage is to detect key points of the human body using the pre-trained AlphaPose model to calculate the structural features S2 of the key points of the human body. The specific implementation steps are as follows:

[0225] For key point detection of human body, this solution directly uses the pre-trained model AlphaPose, which has been fully trained on the key point detection task of the COCO (Common Objects inContext) dataset and supports the detection of 17 key points of the whole body. By using the AlphaPose model, we can predict the 17 key points of the human body, including the position information of key points such as the head, hands, feet, and body. Please refer to Figure 5 .

[0226] Calculate the key point structure feature S2 of the human body:

[0227] S2=A n×n D n×n W n×n

[0228] For the AlphaPose model trained on the COCO dataset, n=17, which means that there are 17 human key points in its output;

[0229] A is an adjacency matrix, which is used to represent the point-to-point connection relationship between human key points. 0 means no connection and 1 means connection. According to the human key point diagram of the COCO dataset, when (i, j) belongs to the set {(1, 2), (0, 1), (0, 2), (1, 3), (2, 4), (3, 5), (4, 6), (15, 13), (13, 11), (16, 14), (14, 12), (11, 12), (5, 11), (6, 12), (5, 6), (5, 7), (6, 8), (7, 9), (8, 10)}, A (i,j) =1, when (i,j) does not belong to the set, then A (i,j) =0

[0230] D is the angle matrix, which is used to measure the horizontal angle between any two key points;

[0231]

[0232] Where I represents the pixel coordinates (x, y) in the image.

[0233] W weight matrix, used to configure the weight between any two key points, the default value is 1.

[0234] The fourth stage is to construct a benchmark image dataset of action key frames; the benchmark images serve as reference images and templates for benchmark action recognition, and the semantic features B S1 and structural feature B S2 , and saved in the system's feature library.

[0235] The fifth stage is to achieve feature matching and extract the semantic features T of the test image. S1 and structural features T S2 , then perform feature matching in the feature library of the benchmark image and select the most similar benchmark image as the matching result. The specific method is as follows:

[0236] The test image is first subjected to YOLOv5 human body detection and the human body area is cropped. Then, the AAEM encoder is used to extract the semantic features of the human body area. S1, and then use the AlphaPose model to obtain 17 key points of the human body and calculate the structural features T of the key points of the human body S2 .

[0237] Calculate the semantic feature cosine similarity SIMI1 and structural feature cosine similarity SIMI2 between the test image and the benchmark image in the feature library.

[0238]

[0239] When the semantic feature cosine similarity SIMI1 and the structural feature cosine similarity SIMI2 are greater than the confidence thresholds T1 and T2 respectively, the test image matching result is the action category corresponding to the reference image; otherwise, it belongs to an unknown category.

[0240]

[0241] Please refer to Figure 6 , Figure 6 A general action recognition device 110 based on semantic feature and structural feature matching provided in an embodiment of the present invention includes:

[0242] The acquisition module 1101 is used to acquire a picture to be tested, and input the picture to be tested into a pre-trained human body recognition model to obtain a human body position area; call a pre-trained action autoencoder model to extract features from the human body position area to obtain semantic features of the action to be tested; call a pre-trained human posture estimation model to extract features from the human body position area to obtain structural features of the action to be tested; obtain benchmark action semantic features and benchmark action structural features corresponding to multiple benchmark pictures from a pre-set benchmark feature library, each of the benchmark pictures is marked with a corresponding benchmark action category;

[0243] The recognition module 1102 is used to perform feature matching on the semantic features of the action to be tested and the structural features of the action to be tested according to multiple semantic features of the benchmark actions and multiple structural features of the benchmark actions, determine the target benchmark image with the highest matching degree, and use the target benchmark action category corresponding to the target benchmark image as the action recognition result of the image to be tested.

[0244] It should be noted that the implementation principle of the aforementioned general action recognition device 110 based on semantic feature and structural feature matching can refer to the implementation principle of the aforementioned general action recognition method based on semantic feature and structural feature matching, which will not be repeated here. It should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into a physical entity, or physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be fully implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the general action recognition device 110 based on semantic feature and structural feature matching can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and a processing element of the above device can call and execute the functions of the general action recognition device 110 based on semantic feature and structural feature matching. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in a processor element or an instruction in software form.

[0245] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (digital signal processors, DSP), or one or more field programmable gate arrays (FPGA), etc. For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0246] The embodiment of the present invention provides a computer device 100, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned general action recognition device 110 based on matching of semantic features and structural features. Figure 7 As shown, Figure 7The computer device 100 provided in the embodiment of the present invention is a structural block diagram. The computer device 100 includes a general action recognition device 110 based on semantic feature and structural feature matching, a memory 111, a processor 112 and a communication unit 113.

[0247] To achieve data transmission or interaction, the memory 111, the processor 112 and the communication unit 113 are electrically connected to each other directly or indirectly. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The universal action recognition device 110 based on semantic feature and structural feature matching includes at least one software function module that can be stored in the memory 111 in the form of software or firmware or solidified in the operating system (OS) of the computer device 100. The processor 112 is used to execute the universal action recognition device 110 based on semantic feature and structural feature matching stored in the memory 111, such as the software function modules and computer programs included in the universal action recognition device 110 based on semantic feature and structural feature matching.

[0248] An embodiment of the present invention provides a readable storage medium, which includes a computer program. When the computer program is executed, the computer device where the readable storage medium is located is controlled to execute the aforementioned general action recognition device 110 based on matching of semantic features and structural features.

[0249] For illustrative purposes, the foregoing description is made with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Numerous modifications and variations are possible in accordance with the above teachings. These embodiments are selected and described in order to best illustrate the principles of the present disclosure and its practical application, so that those skilled in the art can best utilize the present disclosure and utilize various embodiments with different modifications to suit the intended specific application.

Claims

1. A general action recognition method based on matching semantic features and structural features, characterized in that: include: Acquire a picture to be tested, and input the picture to be tested into a pre-trained human body recognition model to obtain a human body position area; Calling a pre-trained action autoencoder model to extract features from the human body position area to obtain semantic features of the action to be tested; Calling a pre-trained human posture estimation model to extract features of the human body position area to obtain structural features of the action to be measured; Acquire benchmark action semantic features and benchmark action structural features corresponding to a plurality of benchmark images from a preset benchmark feature library, each of the benchmark images being marked with a corresponding benchmark action category; According to the multiple benchmark action semantic features and the multiple benchmark action structural features, feature matching is performed on the semantic features of the action to be tested and the structural features of the action to be tested, a target benchmark picture with the highest matching degree is determined, and the target benchmark action category corresponding to the target benchmark picture is used as the action recognition result of the picture to be tested.

2. The method according to claim 1, characterized in that The action autoencoder model is obtained by the following method, including: Constructing an initial action autoencoder model, the initial action autoencoder model includes five cascaded downsampling components and five upsampling components, the downsampling component is a Conv-BN-PRelu-Conv-BN-PRelu structure, the upsampling component is a DeConv-BN-PRelu-DeConv-BN-PRelu structure, each of the downsampling components and each of the upsampling components is configured with a channel attention component, and the channel attention component includes a global pooling layer, a fully connected layer, and a normalization layer; The initial action autoencoder model is trained based on an unsupervised mechanism until a preset mean square error loss function training termination condition is reached to obtain the action autoencoder model.

3. The method according to claim 1, characterized in that The calling of a pre-trained human posture estimation model to extract features of the human body position area to obtain structural features of the action to be measured includes: Calling a pre-trained human posture estimation model to perform key point detection on the human body position area to obtain multiple human body key point position information; Determine the corresponding adjacency matrix and angle matrix according to the position information of the plurality of human body key points; The structural features of the action to be measured are calculated according to the preset key point weights, the adjacency matrix and the angle matrix.

4. The method according to claim 3, characterized in that The step of calculating the structural features of the action to be measured according to the preset key point weights, the adjacency matrix and the angle matrix includes: According to the formula: S2 = A n×n D n×n W n×n Calculate and obtain the structural characteristics of the action to be tested; Among them, S2 is the structural feature of the action to be tested, A n×n is an adjacency matrix, which is used to characterize the relationship between n key points of the human body. n×n is an angle matrix used to represent the horizontal angles of the lines connecting n key points of the human body. n×n The preset key weights.

5. The method according to claim 1, characterized in that The method of performing feature matching on the semantic features of the action to be tested and the structural features of the action to be tested according to the plurality of the semantic features of the benchmark action and the plurality of the structural features of the benchmark action, determining a target benchmark picture with the highest matching degree, and taking the target benchmark action category corresponding to the target benchmark picture as the action recognition result of the picture to be tested, comprises: Calculating the semantic cosine similarity between the semantic feature of the action to be tested and the semantic feature of each benchmark action; Calculating the structural cosine similarity between the semantic feature of the action to be tested and each structural feature of the benchmark action; Determine whether there is a pending reference picture whose semantic cosine similarity and structural cosine similarity are both greater than a preset cosine similarity threshold; If it exists, taking the undetermined reference picture as the target reference picture, and taking the target reference action category corresponding to the target reference picture as the action recognition result of the picture to be tested; If not, the action recognition result of the image to be tested is determined as an unknown action category.

6. The method according to claim 5, characterized in that The method further comprises: In the case where there are multiple pending reference pictures, calculating the Euclidean distance between the pending action semantic feature and the pending action structural feature corresponding to each pending reference picture and the semantic feature and the structural feature of the action to be tested; According to the calculation result of the Euclidean distance, the undetermined reference picture with the highest matching degree is determined as the target reference picture.

7. The method according to claim 1, characterized in that The step of obtaining the image to be tested includes: Obtain the image to be tested and the business scenario corresponding to the image to be tested; The obtaining of the benchmark action semantic features and the benchmark action structural features corresponding to the plurality of benchmark images from the preset benchmark feature library includes: According to the business scenario, benchmark action semantic features and benchmark action structural features corresponding to a plurality of benchmark images are obtained from a preset benchmark feature library.

8. A general action recognition device based on matching of semantic features and structural features, characterized in that: include: The acquisition module is used to acquire the image to be tested, and input the image to be tested into a pre-trained human body recognition model to obtain a human body position area; call the pre-trained action autoencoder model to extract features of the human body position area to obtain semantic features of the action to be tested; Calling a pre-trained human posture estimation model to extract features of the human body position area to obtain structural features of the action to be measured; Acquire benchmark action semantic features and benchmark action structural features corresponding to a plurality of benchmark images from a preset benchmark feature library, each of the benchmark images being marked with a corresponding benchmark action category; The recognition module is used to perform feature matching on the semantic features of the action to be tested and the structural features of the action to be tested according to multiple semantic features of the benchmark actions and multiple structural features of the benchmark actions, determine the target benchmark image with the highest matching degree, and use the target benchmark action category corresponding to the target benchmark image as the action recognition result of the image to be tested.

9. A computer device, characterized in that: The computer device comprises a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device executes the method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium includes a computer program, and when the computer program is executed, the computer device where the readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

Images