A method and apparatus for recognizing an action of a target object
Patent Information
- Application Number
- CN202110653416.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-06-11
- Publication Date
- 2026-10-20
- Estimated Expiration
- 2041-06-11
AI Technical Summary
[0003]在现有技术中，在利用卷积神经网络对目标对象的动作进行识别时，会通过对每一时刻全体的关键点的坐标进行卷积以识别出目标对象的动作类型，而该方法会忽略关键点在空间上的局部信息，从而导致识别出的目标对象的动作类型准确性较低
[0023]本实施例通过获取目标对象的多个关键点在预设图像序列中各自对应的坐标点，得到多个坐标点，然后对多个坐标点进行一维卷积获取目标对象的多个第一局部特征，即获取到目标对象的多个关键点对应的空间局部运动特征，因不同的第一局部特征对动作识别的重要程度是不同的，因此为了对第一局部特征的重要程度进行区分，确定出多个第一局部特征各自对应的第一权重系数，并根据多个第一局部特征及多个第一权重系数，获取到目标对象的多个第一加权局部特征，从而使得在进行动作识别的过程中将更多的资源分配到重要的第一局部特征上，则根据多个第一加权局部特征获取的目标对象对应的全局特征可以更能准确的体现出目标对象的全局运动信息，有效地统筹了部分关键点动作特征与全局动作特征的相互关系，提高识别出的目标对象的第一动作类型的准确性。
Smart Images
Figure CN113435272B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of convolutional neural network technology, and more specifically, to a method and apparatus for action recognition of a target object. Background Technology
[0002] Convolutional Neural Networks (CNNs) are a popular area in machine learning, and action recognition is one of their important applications.
[0003] In existing technologies, when using convolutional neural networks to identify the actions of a target object, the coordinates of all key points at each moment are convolved to identify the action type of the target object. However, this method ignores the local spatial information of the key points, resulting in low accuracy in identifying the action type of the target object. Summary of the Invention
[0004] To address the aforementioned technical problems, this disclosure is proposed. Embodiments of this disclosure provide a method, apparatus, computer-readable storage medium, and electronic device for recognizing the action of a target object. It utilizes one-dimensional convolution to obtain first local features of the target object and uses first weight coefficients corresponding to the first local features to distinguish the importance of multiple first local features, thereby better highlighting the contribution of certain key points to the overall action recognition, resulting in higher accuracy in the ultimately recognized action type of the target object.
[0005] According to a first aspect of this disclosure, a method for action recognition of a target object is provided, comprising:
[0006] Obtain the coordinates of multiple key points of the target object in a preset image sequence to obtain multiple coordinate points;
[0007] One-dimensional convolution is performed on the multiple coordinate points to obtain multiple first local features of the target object;
[0008] Determine the first weight coefficient corresponding to each of the plurality of first local features to obtain the plurality of first weighted local features of the target object;
[0009] Based on the multiple first weighted local features, the global features corresponding to the target object are obtained;
[0010] Based on the global features, the first action type corresponding to the target object is obtained.
[0011] According to a second aspect of this disclosure, a target-corresponding action recognition device is provided, comprising:
[0012] The coordinate point acquisition module is used to acquire the coordinate points corresponding to multiple key points of the target object in a preset image sequence, thereby obtaining multiple coordinate points.
[0013] The local feature acquisition module is used to perform one-dimensional convolution on the multiple coordinate points to obtain multiple first local features of the target object;
[0014] The weighted feature acquisition module is used to determine the first weight coefficient corresponding to each of the plurality of first local features, so as to obtain the plurality of first weighted local features of the target object;
[0015] The global feature acquisition module is used to acquire the global features corresponding to the target object based on the plurality of first weighted local features;
[0016] The action type acquisition module is used to acquire the first action type corresponding to the target object based on the global features.
[0017] According to a third aspect of this disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the above-described target object action recognition method.
[0018] According to a fourth aspect of this disclosure, an electronic device is provided, the electronic device comprising:
[0019] processor;
[0020] Memory used to store the processor's executable instructions;
[0021] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the above-described target object action recognition method.
[0022] The target object action recognition method, apparatus, computer-readable storage medium, and electronic device disclosed herein have at least the following beneficial effects:
[0023] This embodiment obtains the coordinates of multiple key points of the target object in a preset image sequence, and then performs one-dimensional convolution on these coordinates to obtain multiple first local features of the target object, i.e., obtains the spatial local motion features corresponding to multiple key points of the target object. Since different first local features have different degrees of importance for action recognition, in order to distinguish the importance of the first local features, first weight coefficients corresponding to each of the multiple first local features are determined. Based on the multiple first local features and the multiple first weight coefficients, multiple first weighted local features of the target object are obtained. This allows more resources to be allocated to important first local features during the action recognition process. The global features of the target object obtained based on the multiple first weighted local features can more accurately reflect the global motion information of the target object, effectively coordinating the relationship between some key point action features and global action features, and improving the accuracy of the first action type of the target object being identified. Attached Figure Description
[0024] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.
[0025] Figure 1 This is a flowchart illustrating an exemplary embodiment of the action recognition method for a target object provided in this disclosure. Figure 1 ;
[0026] Figure 2 This is a flowchart illustrating step 101 of the action recognition method for a target object provided in an exemplary embodiment of this disclosure;
[0027] Figure 3 This is a flowchart illustrating step 102 of the action recognition method for a target object provided in an exemplary embodiment of this disclosure;
[0028] Figure 4 This is a schematic diagram of the skeleton points in the action recognition method for a target object provided by an exemplary embodiment of the present disclosure;
[0029] Figure 5 This is a flowchart illustrating step 104 of the action recognition method for a target object provided in an exemplary embodiment of this disclosure;
[0030] Figure 6 This is a schematic diagram of a neural network structure in an action recognition method for a target object provided by an exemplary embodiment of the present disclosure;
[0031] Figure 7 This is a flowchart illustrating step 103 of the action recognition method for a target object provided in an exemplary embodiment of this disclosure;
[0032] Figure 8 This is a schematic diagram illustrating the acquisition of the first weight coefficient in an action recognition method for a target object provided by an exemplary embodiment of this disclosure;
[0033] Figure 9 This is a flowchart illustrating an exemplary embodiment of the action recognition method for a target object provided in this disclosure. Figure 2 ;
[0034] Figure 10 This is a schematic diagram of another neural network structure in the action recognition method for a target object provided by an exemplary embodiment of the present disclosure;
[0035] Figure 11 This is a flowchart illustrating step 106 of the action recognition method for a target object provided in an exemplary embodiment of this disclosure;
[0036] Figure 12 This is a schematic diagram illustrating the acquisition of the fourth weight coefficient in an action recognition method for a target object provided by an exemplary embodiment of this disclosure;
[0037] Figure 13 This is a schematic diagram of the structure of a target object action recognition device provided in an exemplary embodiment of the present disclosure;
[0038] Figure 14 This is a schematic diagram of the structure of another target object action recognition device provided in an exemplary embodiment of the present disclosure;
[0039] Figure 15 This is a schematic diagram of the structure of the second recognition processing module in a target object action recognition device provided in an exemplary embodiment of the present disclosure;
[0040] Figure 16 This is a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation
[0041] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.
[0042] Public Overview
[0043] When using convolutional neural networks to identify the actions of a target object, it is crucial to effectively coordinate the relationship between the motion features of some key points and the global motion information. However, in existing technologies, when convolving the coordinates of all key points at each moment to identify the action type of the target object, the local information of the key points in space is ignored, resulting in low accuracy in identifying the action type of the target object.
[0044] This embodiment obtains the coordinates of multiple key points of the target object in a preset image sequence, and then performs one-dimensional convolution on these coordinates to obtain multiple first local features of the target object, i.e., obtains the spatial local motion features corresponding to multiple key points of the target object. Since different first local features have different degrees of importance for action recognition, in order to distinguish the importance of the first local features, first weight coefficients corresponding to each of the multiple first local features are determined. Based on the multiple first local features and the multiple first weight coefficients, multiple first weighted local features of the target object are obtained. This allows more resources to be allocated to important first local features during the action recognition process. The global features of the target object obtained based on the multiple first weighted local features can more accurately reflect the global motion information of the target object, effectively coordinating the relationship between some key point action features and global action features, and improving the accuracy of the first action type of the target object being identified.
[0045] After introducing the basic concept of this application, various non-limiting embodiments of the technical solution provided in this application will be described in detail below with reference to the accompanying drawings.
[0046] Exemplary methods
[0047] Figure 1 This is a flowchart illustrating an exemplary embodiment of the action recognition method for a target object provided in this disclosure.
[0048] This embodiment can be applied to electronic devices, specifically servers or general computers. For example... Figure 1 As shown, an exemplary embodiment of this disclosure provides a method for recognizing the action of a target object, which includes at least the following steps:
[0049] Step 101: Obtain the coordinates of multiple key points of the target object in the preset image sequence to obtain multiple coordinate points.
[0050] In one embodiment, the preset image sequence is a series of frame images carrying a target object that have been acquired in advance. The target object can be a person, and the multiple key points of the target object can be the skeleton points of the human body.
[0051] In one embodiment, the coordinate points are the locations of key points of the target object in the image, which can indicate the spatial information of the target object.
[0052] Step 102: Perform one-dimensional convolution on multiple coordinate points to obtain multiple first local features of the target object.
[0053] In one embodiment, one-dimensional convolution is to use a 1*N convolution kernel to perform convolution calculation on the input object, i.e., multiple coordinate points.
[0054] In one embodiment, the first local feature is a feature extracted by convolution calculation of local coordinate points or local regions of an image.
[0055] Step 103: Determine the first weight coefficients corresponding to each of the multiple first local features to obtain multiple first weighted local features of the target object.
[0056] In one embodiment, a first weight coefficient represents the importance of a first local feature. For example, the larger the value of the first weight coefficient, the more important the corresponding first local feature is and the greater its contribution to the overall action recognition.
[0057] In one embodiment, the first weighted local feature is the result of multiplying the first local feature by the first weight coefficient.
[0058] Step 104: Based on multiple first weighted local features, obtain the global features corresponding to the target object.
[0059] In one embodiment, global features refer to features about the global region extracted by convolutional calculation of multiple first weighted local features.
[0060] Step 105: Based on global features, obtain the first action type corresponding to the target object.
[0061] In one embodiment, after obtaining global features, the global motion information of the target object can be understood, and the action type of the target object can be identified based on the global features to obtain the first action type corresponding to the target object.
[0062] In the above embodiments, multiple coordinate points are obtained by acquiring the coordinate points corresponding to multiple key points of the target object in a preset image sequence. Then, one-dimensional convolution is performed on the multiple coordinate points to obtain multiple first local features of the target object, that is, the spatial local motion features corresponding to multiple key points of the target object are obtained. Since different first local features have different degrees of importance to action recognition, the more important the first local feature, the greater its contribution to action recognition. Therefore, in order to distinguish the importance of the first local features, the first weight coefficients corresponding to each of the multiple first local features are determined. Based on the multiple first local features and the multiple first weight coefficients, multiple first weighted local features of the target object are obtained. This allows more resources to be allocated to important first local features during the action recognition process. The global features of the target object obtained based on the multiple first weighted local features can more accurately reflect the global motion information of the target object, effectively coordinating the relationship between some key point action features and global action features, and improving the accuracy of the first action type of the target object.
[0063] Figure 2 As shown Figure 1 The illustrated embodiment is a flowchart illustrating the steps of obtaining the coordinates of multiple key points of the target object in a preset image sequence to obtain multiple coordinate points.
[0064] like Figure 2 As shown above, in the above Figure 1 Based on the illustrated embodiment, in this exemplary embodiment, step 101, which involves obtaining multiple coordinate points, includes the following steps:
[0065] Step 1011: Determine the coordinates of each of the multiple key points of the target object in the current frame image to obtain the first number of first coordinate points.
[0066] In one embodiment, the current frame image refers to the latest unprocessed frame image acquired at the current moment, and the first coordinate point is the coordinate point corresponding to the key point in the current frame image, indicating the position of the key point in the current frame image. Specifically, image recognition is performed on the current frame image to extract the coordinate points corresponding to multiple key points of the target object in the current frame image, thereby obtaining a first number of first coordinate points.
[0067] Step 1012: Determine the coordinates of each of the multiple key points of the target object in the historical frame images of a preset number of frames prior to the current frame image, and obtain the second number of second coordinate points.
[0068] In one embodiment, a historical frame image refers to a preset number of frame images acquired before the current frame image. The second coordinate point is the coordinate point corresponding to the target object in the historical frame image, indicating the position of the key point in the historical frame image. Each historical frame image will have multiple second coordinate points, so a second number of second coordinate points will be obtained in the preset number of historical frame images.
[0069] Step 1013: Determine the first number of first coordinate points and the second number of second coordinate points as the coordinate points corresponding to multiple key points of the target object in the preset image sequence, thereby obtaining multiple coordinate points. The preset image sequence consists of the current frame image and the historical frame images of the preset number of frames.
[0070] In one embodiment, the preset image sequence consists of the current frame image and a preset number of historical frame images. The coordinates of key points of the target object in different frame images of the preset image sequence can reflect the motion information of the target object. Therefore, a first number of first coordinate points and a second number of second coordinate points are determined as the coordinate points corresponding to multiple key points of the target object in the preset image sequence, resulting in multiple coordinate points as input objects. Specifically, an image acquisition device can be directly connected to obtain frame images at different times, thereby enabling real-time recognition of the target object's actions.
[0071] In the above embodiments, the coordinate points corresponding to the key points of the target object in the current frame image and the historical frame image are determined respectively, thereby obtaining a first number of first coordinate points and a second number of second coordinate points. In this way, the coordinate points corresponding to the key points of the target object in the preset image sequence composed of the current frame image and the historical frame image can be determined. Based on the determined coordinate points, the change of the position information of the key points of the target object in space can be obtained, which is beneficial to accurately identify the action of the target object.
[0072] Figure 3 As shown Figure 1 The illustrated embodiment is a flowchart of the steps to obtain multiple first local features of the target object by performing one-dimensional convolution on multiple coordinate points.
[0073] like Figure 3 As shown above, in the above Figure 1 Based on the illustrated embodiment, in an exemplary embodiment of this application, step 102, which involves obtaining multiple first weighted local features of the target object, includes the following steps:
[0074] Step 1021: Perform one-dimensional convolution on multiple coordinate points to obtain point-level features of the target object.
[0075] In one embodiment, point-level features are features extracted after performing a one-dimensional convolution calculation on each coordinate point, thus enabling the extraction of motion information of each keypoint within a preset image sequence. Specifically, a 1*1 convolution kernel is used to perform convolution calculations on multiple input coordinate points.
[0076] Step 1022: Perform channel concatenation and one-dimensional convolution on the point-level features corresponding to adjacent key points to obtain multiple first local features of the target object.
[0077] In one embodiment, adjacent key points are identified among the key points of the target object. The point-level features corresponding to each adjacent key point are then concatenated by channel splicing, that is, the point-level features corresponding to each adjacent key point are fused together. A one-dimensional convolution is then performed on the concatenated point-level features to extract the first local features corresponding to the adjacent key points. The extracted first local features can display the implicit information between adjacent key points. Thus, the first local features carry semantic information about the mutual movement between two adjacent key points. In this way, more information can be incorporated into the action recognition of the target object to ensure the accuracy of the finally recognized action of the target object.
[0078] In one possible scenario, if the target object is a person, then the key points are the person's skeletal structure (e.g., ...). Figure 4 As shown in the diagram, when designing the neural network structure, the coordinates of the skeleton points in the frame image are used as input information. The first convolutional layer performs convolution calculations on each coordinate point to obtain the point-level features of the target object. Then, the... Figure 4 The point-level features corresponding to adjacent skeleton points are concatenated by channels to obtain the first local features corresponding to adjacent key points. For example, the point-level features corresponding to skeleton points 5 and 6 are concatenated by channels.
[0079] In the above embodiments, point-level features are extracted from multiple coordinate points to obtain motion information of each key point in a preset image sequence. The point-level features corresponding to adjacent key points are fused and one-dimensionally convolved to obtain implicit information between adjacent key points. This allows more information to be incorporated into the action recognition of the target object, thereby ensuring the accuracy of the identified action type of the target object.
[0080] Figure 5 As shown Figure 3 The illustrated embodiment is a flowchart of the steps for obtaining the global features corresponding to the target object.
[0081] like Figure 5 As shown above, in the above Figure 3 Based on the illustrated embodiment, in an exemplary embodiment of this application, step 104, which involves obtaining multiple first weighted local features of the target object, includes the following steps:
[0082] Step 1041: Perform one-dimensional convolution on multiple first weighted local features to obtain multiple second local features of the target object.
[0083] In one embodiment, the second local feature is a feature extracted by convolution calculation of a local region among multiple first weighted local features. The semantic information carried by the second local feature is higher than that carried by the first local feature. It is obtained by logical reasoning from the semantic information carried by the first local feature. For example, the semantic information carried by the second local feature is hand, foot, spine, etc.
[0084] Specifically, design Figure 6 The neural network shown uses layer 1 (convolutional layer) to extract point-level features, layer 2 to extract first local features, and layer 3 to extract second local features. It should be noted that the input to each layer is a concatenation of the features output from each coordinate point of the previous layer along the channel dimension. To emphasize the importance of different coordinate points in the input, each layer's input includes features such as... Figure 6 The channel correlation block shown is used to learn the weight coefficients corresponding to different local features.
[0085] Step 1042: Determine the second weight coefficients corresponding to each of the multiple second local features to obtain multiple second weighted local features of the target object.
[0086] In one embodiment, a second weighting coefficient represents the importance of a second local feature.
[0087] In one embodiment, the second weighted local feature is the result of multiplying the second local feature by the second weight coefficient.
[0088] Step 1043: Group the multiple second-weighted local features according to their respective semantic information, and perform channel concatenation and one-dimensional convolution to obtain at least two third local features of the target object.
[0089] In one embodiment, the second weighted local features carry different semantic information. They are first grouped according to semantic information, with those carrying the same or similar semantic information grouped together, and channels within each group are concatenated. Then, a one-dimensional convolution is performed on the concatenated second weighted local features to extract at least two third local features. The third local features carry more advanced semantic information than the second local features, derived through logical reasoning from the second local features, such as upper and lower halves, left and right halves, or upper left and lower left halves, etc. Specific applications... Figure 6 The third local feature is extracted from layer 4.
[0090] Step 1044: Determine the third weight coefficients corresponding to at least two third local features to obtain at least two third weighted local features of the target object.
[0091] In one embodiment, a third weighting coefficient represents the importance of a third local feature.
[0092] In one embodiment, the third weighted local feature is the result of multiplying the third local feature by the third weight coefficient.
[0093] Step 1045: Obtain the global features of the target object based on at least two third-weighted local features.
[0094] In one embodiment, the global feature is the feature extracted by convolving at least two third-weighted local features over a whole region, and the semantic information carried by the global feature is overall information. Specifically, it utilizes... Figure 6 Global features are extracted from layer 5.
[0095] In the above embodiments, feature extraction of key points of the target object from point level features, first local features, first weighted local features, second local features, second weighted local features, third local features, third weighted local features to global features is realized, which makes the constructed neural network feature hierarchy strong, and by introducing an attention mechanism, the relationship between the motion features of some key points of the target object and the global motion information is effectively coordinated, thereby ensuring the accuracy of the action type of the target object finally identified.
[0096] Figure 7 As shown Figure 1 The illustrated embodiment is a flowchart of determining the first weight coefficients corresponding to multiple first local features to obtain multiple first weighted local features of the target object.
[0097] like Figure 7 As shown above, in the above Figure 1 Based on the illustrated embodiment, in an exemplary embodiment of this application, step 103, which involves obtaining multiple first weighted local features of the target object, includes the following steps:
[0098] Step 1031: After channel pooling, the multiple first local features are introduced into the activation function through a fully connected layer to determine the first weight coefficients corresponding to each of the multiple first local features.
[0099] In one embodiment, channel pooling is performed on the first local features, and then the multiple channel-pooled first local features are input into a fully connected layer. An activation function is then introduced to determine the first weight coefficient corresponding to each first local feature. Specifically, as shown... Figure 8As shown, the activation function can be the sigmoid function, which determines the first weight coefficient with a value range of 0 to 1. Since the extracted local features exist in the form of feature images, therefore... Figure 8 The Feature Map in this embodiment is the first local feature. Different first local features are obtained after convolving multiple coordinate points with different convolution kernels. Each local feature represents a channel. Therefore, channel pooling of the first local feature can obtain the channel that contributes more to action recognition. That is, the first local feature corresponding to the channel is more important. Then, the feature is mapped to a size through a fully connected layer, and the first weight coefficient in the range of 0 to 1 is output using the sigmoid activation function.
[0100] Step 1032: Based on multiple first local features and their respective first weight coefficients, obtain multiple first weighted local features of the target object.
[0101] In one embodiment, the first weight coefficient corresponding to each of the first local features is multiplied with the first local features to obtain multiple first weighted local features of the target object.
[0102] In the above embodiments, channel pooling is used to obtain the first local features that contribute significantly to action recognition, and a first weight coefficient of numerical size is output through a fully connected layer and an activation function. This allows more resources to be concentrated on the important first local features after obtaining the first weighted local features based on the first local features and the first weight coefficient, thereby ensuring the accuracy of the action type of the target object finally identified.
[0103] Figure 9 As shown Figure 5 The illustrated embodiments also include flowcharts.
[0104] like Figure 9 As shown above, in the above Figure 5 Based on the illustrated embodiments, an exemplary embodiment of this application may further include the following steps:
[0105] Step 106: Based on multiple first local features, multiple second local features, and global features, obtain the second action type corresponding to the target object.
[0106] In one embodiment, the design is as follows Figure 10 The dual-branch neural network shown uses one branch to extract features from multiple coordinate points step by step, namely, to obtain multiple first local features, multiple second local features, and global features. The other branch uses the semantic information carried by the global features as a guide to more accurately identify the local features in the shallow layer of the network, so as to obtain a more accurate second action type corresponding to the target object.
[0107] In the above embodiments, a dual-branch neural network is used to simultaneously identify multiple coordinate points, providing two classification results: a first action type and a second action type, to further ensure the accuracy of the identified action type. Furthermore, by introducing another branch neural network, the network avoids getting stuck in an infinite loop during gradient backpropagation. The two branches can be trained simultaneously, mutually reinforcing each other and ultimately achieving good recognition results.
[0108] Figure 11 As shown Figure 9 The illustrated embodiment is a flowchart illustrating the steps for obtaining the second action type corresponding to the target object based on multiple first local features, multiple second local features, and global features.
[0109] like Figure 11 As shown above, in the above Figure 9 Based on the illustrated embodiment, in an exemplary embodiment of this application, step 106, which involves obtaining the second action type corresponding to the target object, includes the following steps:
[0110] Step 1061: Fuse multiple first local features with global features, and introduce an activation function to determine the fourth weight coefficients corresponding to different positions of the multiple first local features.
[0111] In one embodiment, because global features carry higher-level semantic information, they can be used to guide feature extraction in the first local features to obtain the importance of different positions in the first local features for overall action recognition, i.e., the fourth weighting coefficient. Specifically, as shown... Figure 12 As shown, the embedded global feature is transformed in dimension by a fully connected layer, then fused with the first local feature map, and then the sigmoid activation function is introduced to obtain the fourth weight coefficient in the range of 0 to 1.
[0112] Step 1062: Based on multiple first local features and the fourth weight coefficients corresponding to different positions of the multiple first local features, obtain multiple fourth weighted local features.
[0113] In one embodiment, the fourth weighted local feature is obtained by multiplying the first local feature and the fourth weight coefficient.
[0114] Step 1063: Fuse multiple second local features with global features, and introduce an activation function to determine the fifth weight coefficients corresponding to different positions of the multiple second local features.
[0115] In one embodiment, the semantic information carried by the global features is used to guide the feature extraction in the second local features, thereby obtaining the importance of different positions in the second local features, i.e., the fifth weight coefficient.
[0116] Step 1064: Based on multiple second local features and the fifth weight coefficients corresponding to different positions of the multiple second local features, obtain multiple fifth weighted local features.
[0117] In one embodiment, the fifth weighted local feature is obtained by multiplying the second local feature and the fifth weight coefficient.
[0118] Step 1065: Based on multiple fourth-weighted local features and multiple fifth-weighted local features, obtain the second action type corresponding to the target object.
[0119] In one embodiment, the fourth weighted local features and the fifth weighted local features are aggregated and passed through a fully connected layer to obtain the second action type corresponding to the target object.
[0120] In the above embodiments, the semantic information carried by the global features is used to guide the feature extraction of the first local features and the second local features, so as to obtain the importance of different positions in the first local features and the second local features, thereby allocating more resources to important local features to facilitate the acquisition of a more accurate second action type.
[0121] Exemplary device
[0122] Based on the same concept as the method embodiments of this application, this application also provides a target-corresponding action recognition device.
[0123] Figure 13 This is a schematic diagram of the structure of a target-corresponding action recognition device provided for an exemplary embodiment of this application.
[0124] The coordinate point acquisition module 131 is used to acquire the coordinate points corresponding to each of the multiple key points of the target object in a preset image sequence, thereby obtaining multiple coordinate points;
[0125] The local feature acquisition module 132 is used to perform one-dimensional convolution on the multiple coordinate points to obtain multiple first local features of the target object;
[0126] The weighted feature acquisition module 133 is used to determine the first weight coefficient corresponding to each of the plurality of first local features, so as to obtain the plurality of first weighted local features of the target object;
[0127] The global feature acquisition module 134 is used to acquire the global features corresponding to the target object based on the plurality of first weighted local features;
[0128] The first recognition processing module 135 is used to obtain the first action type corresponding to the target object based on the global features.
[0129] like Figure 14 As shown, in an exemplary embodiment of this application, the coordinate point acquisition module 131 includes:
[0130] The first coordinate acquisition unit 1311 is used to determine the coordinate points corresponding to multiple key points of the target object in the current frame image, and obtain a first number of first coordinate points;
[0131] The second coordinate acquisition unit 1312 is used to determine the coordinate points corresponding to multiple key points of the target object in historical frame images of a preset number of frames before the current frame image, and obtain a second number of second coordinate points.
[0132] The third coordinate acquisition unit 1313 determines the first number of first coordinate points and the second number of second coordinate points as the coordinate points corresponding to multiple key points of the target object in the preset image sequence, thereby obtaining multiple coordinate points. The preset image sequence consists of the current frame image and the historical frame images of the preset number of frames.
[0133] like Figure 14 As shown, in an exemplary embodiment of this application, the local feature acquisition module 132 includes:
[0134] The point-level feature acquisition unit 1321 is used to perform one-dimensional convolution on multiple coordinate points to obtain the point-level features of the target object;
[0135] The first feature acquisition unit 1322 is used to perform channel splicing and one-dimensional convolution on the point-level features corresponding to adjacent key points to obtain multiple first local features of the target object.
[0136] like Figure 14 As shown, in an exemplary embodiment of this application, the weighted feature acquisition module 133 includes:
[0137] The first coefficient acquisition unit 1331 is used to introduce multiple first local features after channel pooling into an activation function after passing through a fully connected layer, so as to determine the first weight coefficients corresponding to each of the multiple first local features.
[0138] The second feature acquisition unit 1332 acquires multiple first weighted local features of the target object based on multiple first local features and the first weight coefficients corresponding to each of the multiple first local features.
[0139] like Figure 14 As shown, in an exemplary embodiment of this application, the global feature acquisition module 134 includes:
[0140] The third feature acquisition unit 1341 is used to perform one-dimensional convolution on multiple first weighted local features to obtain multiple second local features of the target object;
[0141] The fourth feature acquisition unit 1342 is used to determine the second weight coefficients corresponding to each of the multiple second local features, so as to obtain multiple second weighted local features of the target object;
[0142] The fifth feature acquisition unit 1343 is used to group multiple second weighted local features according to their respective semantic information and perform channel concatenation and one-dimensional convolution to obtain at least two third local features of the target object.
[0143] The sixth feature acquisition unit 1344 is used to determine the third weight coefficients corresponding to at least two third local features, so as to obtain at least two third weighted local features of the target object;
[0144] The seventh feature acquisition unit 1345 is used to acquire the global features of the target object based on at least two third weighted local features.
[0145] like Figure 14 As shown in an exemplary embodiment of this application, the action recognition device corresponding to the target further includes:
[0146] The second recognition processing module 136 is used to obtain the second action type corresponding to the target object based on multiple first local features, multiple second local features and global features.
[0147] like Figure 15 As shown, in one exemplary embodiment of this application, the second identification processing module 136 includes:
[0148] The second coefficient acquisition unit 1361 is used to fuse multiple first local features with global features and introduce an activation function to determine the fourth weight coefficients corresponding to different positions of multiple first local features.
[0149] The eighth feature acquisition unit 1362 is used to acquire multiple fourth weighted local features based on multiple first local features and the fourth weight coefficients corresponding to different positions of the multiple first local features;
[0150] The third coefficient acquisition unit 1363 is used to fuse multiple second local features with global features and introduce an activation function to determine the fifth weight coefficients corresponding to different positions of multiple second local features.
[0151] The ninth feature acquisition unit 1364 acquires multiple fifth weighted local features based on multiple second local features and the fifth weight coefficients corresponding to different positions of the multiple second local features;
[0152] The tenth feature acquisition unit 1365 acquires the second action type corresponding to the target object based on multiple fourth weighted local features and multiple fifth weighted local features.
[0153] Exemplary electronic devices
[0154] Figure 16 A block diagram of an electronic device according to an embodiment of this application is illustrated.
[0155] like Figure 16 As shown, the electronic device 100 includes one or more processors 101 and memory 102.
[0156] The processor 101 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 100 to perform desired functions.
[0157] The memory 102 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 101 may execute the program instructions to implement the action recognition methods corresponding to the objectives of the various embodiments of this application described above, and / or other desired functions.
[0158] In one example, the electronic device 100 may also include an input device 103 and an output device 104, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).
[0159] Of course, for the sake of simplicity, Figure 16 Only some of the components of the electronic device 100 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 100 may include any other suitable components depending on the specific application.
[0160] Exemplary computer program products and computer-readable storage media
[0161] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the action recognition methods corresponding to the targets according to various embodiments of this application as described in the "Exemplary Methods" section of this specification.
[0162] The computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of this application. The programming languages include object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0163] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the action recognition method corresponding to the target according to various embodiments of this application as described in the "Exemplary Methods" section above.
[0164] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0165] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.
[0166] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.
[0167] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.
[0168] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.
[0169] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.
Claims
1. A method for recognizing the action of a target object, comprising: Obtain the coordinates of multiple key points of the target object in a preset image sequence to obtain multiple coordinate points; One-dimensional convolution is performed on the multiple coordinate points to obtain multiple first local features of the target object; Determine the first weight coefficient corresponding to each of the plurality of first local features to obtain the plurality of first weighted local features of the target object; Based on the multiple first weighted local features, the global features corresponding to the target object are obtained; Based on the global features, the first action type corresponding to the target object is obtained; The step of obtaining the global features corresponding to the target object based on the plurality of first weighted local features includes: One-dimensional convolution is performed on the plurality of first weighted local features to obtain a plurality of second local features of the target object; Determine the second weight coefficient corresponding to each of the plurality of second local features to obtain the plurality of second weighted local features of the target object; Based on the semantic information corresponding to each of the multiple second weighted local features, group them and perform channel concatenation and one-dimensional convolution to obtain at least two third local features of the target object; Determine the third weight coefficients corresponding to each of the at least two third local features to obtain at least two third weighted local features of the target object; Based on the at least two third-weighted local features, the global features corresponding to the target object are obtained.
2. The method according to claim 1, wherein, The step of determining the first weight coefficient corresponding to each of the plurality of first local features to obtain the plurality of first weighted local features of the target object includes: The channel pooled first local features are passed through a fully connected layer and then introduced into an activation function to determine the first weight coefficient corresponding to each of the first local features. Based on the plurality of first local features and the first weight coefficients corresponding to each of the plurality of first local features, a plurality of first weighted local features of the target object are obtained.
3. The method according to claim 1, wherein, The step of performing one-dimensional convolution on the plurality of coordinate points to obtain multiple first local features of the target object includes: Perform one-dimensional convolution on the multiple coordinate points to obtain the point-level features of the target object; Channel concatenation and one-dimensional convolution are performed on the point-level features corresponding to adjacent key points to obtain multiple first local features of the target object.
4. The method according to claim 3, wherein, The method further includes: Based on the plurality of first local features, the plurality of second local features, and the global features, the second action type corresponding to the target object is obtained.
5. The method according to claim 4, wherein, The step of obtaining the second action type corresponding to the target object includes: The plurality of first local features are fused with the global features, and an activation function is introduced to determine the fourth weight coefficients corresponding to different positions of the plurality of first local features. Based on the plurality of first local features and the fourth weight coefficients corresponding to different positions of the plurality of first local features, a plurality of fourth weighted local features are obtained; The multiple second local features are fused with the global features, and an activation function is introduced to determine the fifth weight coefficients corresponding to different positions of the multiple second local features. Based on the plurality of second local features and the fifth weight coefficients corresponding to different positions of the plurality of second local features, a plurality of fifth weighted local features are obtained; Based on the plurality of fourth-weighted local features and the plurality of fifth-weighted local features, the second action type corresponding to the target object is obtained.
6. The method according to any one of claims 1-5, wherein, The step of obtaining the coordinates of multiple key points of the target object in a preset image sequence, resulting in multiple coordinate points, includes: Determine the coordinates of multiple key points of the target object in the current frame image to obtain the first number of first coordinate points; In the historical frame images of a preset number of frames prior to the current frame image, the coordinate points corresponding to each of the multiple key points of the target object are determined to obtain a second number of second coordinate points. The first number of first coordinate points and the second number of second coordinate points are determined as the coordinate points corresponding to the multiple key points of the target object in the preset image sequence, thereby obtaining multiple coordinate points. The preset image sequence consists of the current frame image and the preset number of historical frame images.
7. A target object action recognition device, comprising: The coordinate point acquisition module is used to acquire the coordinate points corresponding to multiple key points of the target object in a preset image sequence, thereby obtaining multiple coordinate points. The local feature acquisition module is used to perform one-dimensional convolution on the multiple coordinate points to obtain multiple first local features of the target object; The weighted feature acquisition module is used to determine the first weight coefficient corresponding to each of the plurality of first local features, so as to obtain the plurality of first weighted local features of the target object; The global feature acquisition module is used to acquire the global features corresponding to the target object based on the plurality of first weighted local features; An action type acquisition module is used to acquire the first action type corresponding to the target object based on the global features; The global feature acquisition module is specifically used for: performing one-dimensional convolution on the plurality of first weighted local features to obtain a plurality of second local features of the target object; determining the second weight coefficients corresponding to each of the plurality of second local features to obtain a plurality of second weighted local features of the target object; grouping according to the semantic information corresponding to each of the plurality of second weighted local features and performing channel concatenation and one-dimensional convolution to obtain at least two third local features of the target object; Determine the third weight coefficients corresponding to each of the at least two third local features to obtain at least two third weighted local features of the target object; based on the at least two third weighted local features, obtain the global features corresponding to the target object.
8. A computer-readable storage medium storing a computer program for performing the action recognition method for a target object according to any one of claims 1-6.
9. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the target object action recognition method according to any one of claims 1-6.