Object action recognition method, object action recognition apparatus, and electronic device

Abstract

A method, apparatus and electronic device for recognizing an action of an object are disclosed. The method comprises: obtaining a series of coordinates of each key point in a plurality of key points of an object over a period of continuous time; performing one-dimensional convolution on the series of coordinates of each key point to obtain a point-level feature of the object; performing grouping and one-dimensional convolution based on the point-level feature to obtain a local feature; using a global network based on the point-level feature to obtain a global feature; and recognizing an action of the object based on the local feature and the global feature. In this way, the accuracy of action recognition is improved.

Description

Technical Field

[0001] This application relates to the field of machine vision, and more specifically, to an object action recognition method, an object action recognition device, and an electronic device. Background Technology

[0002] In the field of machine vision, action recognition and behavior understanding based on machine vision can significantly impact all aspects of people's lives because vision is the most important information in interpersonal interactions, helping people quickly obtain key information. For example, a person's gestures and other actions can reflect their attitude, underlying intentions, and emotions.

[0003] Human motion recognition can be performed based on human skeleton point information. This requires capturing the correlation information between spatial skeleton points in the sequence data itself, as well as the motion information of each skeleton point in the temporal dimension.

[0004] In addition, relevant information can also be obtained by recognizing the actions of objects other than people. For example, by recognizing specific actions of running machinery, it can be determined whether the machine is operating normally.

[0005] Therefore, it is desirable to provide an improved object action recognition scheme. Summary of the Invention

[0006] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide an object action recognition method, an object action recognition device, and an electronic device, which obtain point-level features of an object based on a series of coordinates of key points over a continuous time period, and obtain local and global features based on the point-level features to recognize the object's actions, thereby improving the accuracy of action recognition.

[0007] According to one aspect of this application, an object action recognition method is provided, comprising: obtaining a series of coordinates of each key point over a continuous time period among a plurality of key points of an object; performing a one-dimensional convolution on the series of coordinates of each key point to obtain point-level features of the object; performing grouping and one-dimensional convolution based on the point-level features to obtain local features; using a global network based on the point-level features to obtain global features; and recognizing the action of the object based on the local features and the global features.

[0008] According to another aspect of this application, an object action recognition device is provided, comprising: a coordinate acquisition unit for acquiring a series of coordinates of each key point over a continuous time period among a plurality of key points of an object; a point-level feature acquisition unit for performing a one-dimensional convolution on the series of coordinates of each key point acquired by the coordinate acquisition unit to obtain point-level features of the object; a local feature acquisition unit for grouping and performing a one-dimensional convolution on the point-level features obtained by the point-level feature acquisition unit to obtain local features; a global feature acquisition unit for using a global network to obtain global features based on the point-level features obtained by the point-level feature acquisition unit; and an object action recognition unit for recognizing the action of the object based on the local features obtained by the local feature acquisition unit and the global features obtained by the global feature acquisition unit.

[0009] According to another aspect of this application, an electronic device is provided, comprising: a processor; and a memory storing computer program instructions that, when executed by the processor, cause the processor to perform the content-recommended method as described above.

[0010] According to another aspect of this application, a computer-readable medium is provided that stores computer program instructions thereon, which, when executed by a processor, cause the processor to perform the content recommendation method as described above.

[0011] The object action recognition method, object action recognition device, and electronic device provided in this application can obtain the point-level features of an object based on a series of coordinates of key points of the object over a continuous period of time, and obtain local and global features based on the point-level features to recognize the object's actions, thereby improving the accuracy of action recognition. Attached Figure Description

[0012] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain the application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0013] Figure 1 The illustration shows a flowchart of an object action recognition method according to an embodiment of this application.

[0014] Figure 2 The illustration shows a schematic diagram of a human skeleton according to an embodiment of this application.

[0015] Figure 3The diagram illustrates a network architecture of an object action recognition method according to an embodiment of this application.

[0016] Figure 4 The illustration shows a flowchart of a first example of obtaining local and global features at each level in the object action recognition method according to an embodiment of this application.

[0017] Figure 5 The illustration shows a flowchart of a second example of obtaining local and global features at each level in the object action recognition method according to an embodiment of this application.

[0018] Figure 6 The illustration shows a schematic architecture diagram of the global network in the object action recognition method according to an embodiment of this application.

[0019] Figure 7 The figure shows a block diagram of an object action recognition device according to an embodiment of the present application.

[0020] Figure 8 The diagram illustrates a block diagram of a first example of a local feature acquisition unit and a global feature acquisition unit in an object action recognition device according to an embodiment of this application.

[0021] Figure 9 The diagram illustrates a second example of a local feature acquisition unit and a global feature acquisition unit in an object action recognition device according to an embodiment of this application.

[0022] Figure 10 A block diagram of an electronic device according to an embodiment of this application is illustrated. Detailed Implementation

[0023] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0024] Application Overview

[0025] As mentioned above, current object action recognition schemes, such as human action recognition based on human skeleton information, include schemes that integrate information from various modalities such as depth and RGB images for classification, schemes that extract information from time series using recurrent neural networks, and schemes that use a dual-branch mode (dual-branch dual-stream) to extract spatiotemporal features on feature maps using local and non-local convolutional networks. In addition, there are schemes that use one-dimensional convolution to predict and classify actions.

[0026] Currently, although recurrent neural networks can effectively process sequence information, the models are not flexible enough for input skeleton information of different time lengths, and their ability to learn the spatial relationships between skeleton points is insufficient.

[0027] In addition, current methods that extract spatiotemporal information in the temporal and spatial dimensions by designing local and non-local network structures can learn some spatiotemporal information, but when performing convolutions in different dimensions on the same feature map, the learned information is relatively redundant, and the extraction of spatiotemporal features is not deep enough.

[0028] While current global networks can effectively extract global information about human skeleton points, they lack sufficient control over local details.

[0029] Current one-dimensional convolution schemes identify and predict by convolving all joints at each time step. While this can extract structural information of skeleton points at different time steps, it ignores spatial local information, and the model is somewhat dependent on the sequence length. Another one-dimensional convolution scheme performs one-dimensional convolution based on semantic structural features, but it fails to capture well the contributions of channels, i.e., different joints, to the features implicit in the action.

[0030] Therefore, the key to object action recognition is how to effectively coordinate the relationship between the motion features of some key points of an object and the global motion information.

[0031] To address the aforementioned technical problems, the basic concept of this application is to obtain point-level features for each point of an object, obtain local and global features based on the point-level features, and fuse the local and global features to identify the object's actions.

[0032] Specifically, the object action recognition method, object action recognition device, and electronic device provided in this application first obtain a series of coordinates of each key point over a continuous time period among multiple key points of an object, then perform one-dimensional convolution on the series of coordinates of each key point to obtain point-level features of the object, then group and perform one-dimensional convolution based on the point-level features to obtain local features, and then use a global network based on the point-level features to obtain global features, and finally recognize the action of the object based on the local features and the global features.

[0033] The object action recognition method, object action recognition device, and electronic device provided in this application can effectively extract features for each key point by performing one-dimensional convolution on a series of coordinates of each key point over a continuous time period among multiple key points of an object, thereby effectively utilizing the features of each key point to recognize the object.

[0034] Furthermore, the object action recognition method, object action recognition device, and electronic device provided in this application can effectively extract local semantic information in the object's action process by grouping point-level features and performing one-dimensional convolution, thereby improving the accuracy of object action recognition.

[0035] Furthermore, the object action recognition method, object action recognition device, and electronic device provided in this application can further improve the accuracy of object action recognition by combining local and global features to recognize actions.

[0036] In the object action recognition method, object action recognition device and electronic device provided in this application, the object can be a person or other moving objects used to recognize their actions, such as robots or engineering machinery.

[0037] After introducing the basic principles of this application, various non-limiting embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0038] Exemplary methods

[0039] Figure 1 The illustration shows a flowchart of an object action recognition method according to an embodiment of this application.

[0040] like Figure 1 As shown, the object action recognition method according to an embodiment of this application includes the following steps.

[0041] Step S110: Obtain a series of coordinates over a continuous time period for each of the multiple key points of the object. Here, the key points of the object refer to points related to the object's actions. Taking a person as an example, the key points of a person are points on the person's skeleton diagram, such as... Figure 2 As shown. Figure 2 The illustration shows a schematic diagram of a human skeleton according to an embodiment of this application.

[0042] like Figure 2 As shown, the human skeleton diagram includes multiple key points, and each key point represents a specific part of the human body, for example... Figure 2 The joints 9, 10, 11, 12, 25, and 24 in the diagram represent the human hand, while... Figure 2 The joints 5, 6, 7, 23, and 22 in the diagram represent the person's other hand.

[0043] Furthermore, taking a human as an example, a human action necessarily occurs over a continuous period of time. Therefore, a human action can be expressed through a series of coordinates of key points over a continuous period of time. Moreover, these coordinates can be two-dimensional coordinates, such as (x, y) coordinates, or three-dimensional coordinates, such as (x, y, z) coordinates.

[0044] Figure 3 The diagram illustrates a network architecture for an object action recognition method according to an embodiment of this application. Figure 3 As shown, P1, P2, P3, P4, ..., Pn-2, Pn-1, Pn represent the sequence coordinates of key points, where P1 is represented, for example, as (x1, y1)*T, and T represents multiple moments in continuous time.

[0045] Step S120: Perform a one-dimensional convolution on the series of coordinates of each key point to obtain the point-level features of the object. For example, perform a one-dimensional convolution on the series of coordinates of each key point of the human body. This allows the extraction of the point-level features of the object. The point-level features represent the motion information of each key point in a continuous time series. In other words, the behavior information of the object in the overall space is composed of the combination of the local motion feature information of each key point.

[0046] like Figure 3 As shown, through one-dimensional convolution, point-level features F1, F2, F3, F4, ..., Pn-2, Pn-1, Pn can be obtained from the series of coordinates P1, P2, P3, P4, ..., Pn-2, Pn-1, Pn of each keypoint. Each point-level feature is represented in the form of C*T, where C represents the number of channels of the point-level feature, and T is the same as above, representing multiple different time points represented by the point-level feature.

[0047] Step S130: Grouping and performing one-dimensional convolution based on the point-level features to obtain local features. The point-level features can reflect the structural information of key points at different times and, to a certain extent, the local spatial information of the key points. Further grouping and performing one-dimensional convolution on the point-level features can further extract local semantic information during the object's action process.

[0048] The following text will further explain how to obtain semantic information at each level by grouping the point-level features and performing one-dimensional convolution. For example, as... Figure 3 As shown, local features F are obtained based on the point-level features F1, F2, F3, F4, ..., Fn-2, Fn-1, Fn. L .

[0049] Step S140: Obtain global features using a global network based on the point-level features. By using a global network, the correlation between features of different key points under a behavior can be captured effectively. Taking the human body as an example, for a human behavior, the motion information of key points in different parts of the human body is different. Through a global network, the mapping relationship between a human behavior and key points in different parts of the human body can be analyzed more comprehensively.

[0050] The following text will further explain how to obtain global features using a global network based on the aforementioned point-level features. For example, as... Figure 3 As shown, the global feature F is obtained based on the point-level features F1, F2, F3, F4, ..., Fn-2, Fn-1, Fn. G .

[0051] Step S150: Identify the object's action based on the local features and the global features. That is, based on the local features and the global features obtained as described above, local semantic information and global features can be fully combined to achieve accurate action recognition.

[0052] For example, such as Figure 3 As shown, by combining local features F L and global features F G For example, local features F L After dimensionality transformation, it is compared with the global feature F G Cascading can yield a feature map F used to represent the actions of an object. R Then, based on the feature map F R Identify the actions of the object.

[0053] Therefore, the object action recognition method according to the embodiments of this application can effectively extract the features of each key point by performing one-dimensional convolution on a series of coordinates of each key point over a continuous time period among multiple key points of the object, thereby effectively utilizing the features of each key point to recognize the object.

[0054] Furthermore, the object action recognition method according to the embodiments of this application can effectively extract local semantic information in the object's action process by grouping point-level features and performing one-dimensional convolution, thereby improving the accuracy of object action recognition.

[0055] Furthermore, the object action recognition method according to the embodiments of this application can further improve the accuracy of object action recognition by combining local features and global features to recognize actions.

[0056] Figure 4 The illustration shows a flowchart of a first example of obtaining local and global features at each level in the object action recognition method according to an embodiment of this application.

[0057] like Figure 4 As shown, in Figure 1 Based on the illustrated embodiment, step S130 includes the following steps.

[0058] Step S131 involves grouping the point-level features according to their semantic connectivity and then performing channel-wise concatenation and one-dimensional convolution to obtain multiple first sub-local features. For example, as described above... Figure 2 Taking the human skeleton diagram shown as an example, the connection relationship between each joint point represents its semantic connection relationship. For example, the connection between joint points 9 and 10 represents the connection relationship of the upper arm of a person.

[0059] Based on semantic connectivity, the point-level features can be grouped. For example, the point-level features of joints 9 and 10 can be grouped together, then concatenated by channel into a feature vector, and further one-dimensional convolution can be performed to obtain the first sub-local features representing the semantic information of the mutual motion between joints 9 and 10.

[0060] For example, as described above, the point-level feature can be represented by C*T. Thus, by concatenating by channel, a first point-level feature, such as C1*T, and a second point-level feature, such as C2*T, can be concatenated into a first sub-local feature of (C1+C2)*T.

[0061] like Figure 3 As shown, for point-level features F1, F2, F3, F4, ..., Fn-2, Fn-1, Fn, they are grouped and then concatenated by channel and one-dimensional convolution is performed to obtain multiple first sub-local features A1, A2, A3, ..., Am-1, Am.

[0062] Step S132: Obtain the local feature based on the plurality of first sub-local features. Through step S131, each first sub-local feature representing the semantic information of the mutual movement between adjacent keypoints can be obtained. Then, the local feature can be obtained based on the plurality of first sub-local features, for example, by concatenating the plurality of first sub-local features by channel to obtain the local feature. That is, the local feature has the same dimension as the first sub-local features, for example, it is still represented as C*T.

[0063] like Figure 3 As shown, local feature F is obtained based on the plurality of first sub-local features A1, A2, A3, ..., Am-1, Am. L .

[0064] In this way, by grouping point-level features according to semantic connections and performing channel-wise concatenation and one-dimensional convolution to obtain multiple first sub-local features, the obtained local features can reflect the semantic information of the interconnection between key points and the motion information of their relative positions, thereby making action recognition based on local features more accurate.

[0065] Further as Figure 4 As shown, step S140 includes the following steps.

[0066] Step S141 involves dimensionally concatenating the plurality of first sub-local features and using the global network to obtain the first sub-global feature from the concatenated first sub-local features. That is, dimensionally concatenating the plurality of first sub-local features allows the global feature to be obtained using a two-dimensional convolutional global network.

[0067] For example, as mentioned above, the first sub-local feature is still represented by C*T. The multiple first sub-local features are concatenated by dimensions to obtain a global feature of C*T*N, where N represents the number of dimensions. Therefore, for the global feature, the first sub-global feature can be obtained, for example, by using a two-dimensional convolutional neural network as the global network.

[0068] For example, such as Figure 3 As shown, the multiple first sub-local features A1, A2, A3, ..., Am-1, Am are passed through a global network G1 to obtain the global feature F. G1 .

[0069] Step S142: Obtain the global feature based on the first sub-global feature. For example, the multiple first sub-global features are concatenated by channel to obtain the global feature. That is, the global feature has the same dimension as the first sub-global feature, for example, it is still represented by C*T*N.

[0070] For example, such as Figure 3 As shown, based on global feature F G1 Obtain global features F G .

[0071] In this way, by obtaining global features through the first sub-local features that can reflect the semantic relationship between the connections between each key point, the global features can contain semantic information that reflects the relative relationship between the features of each key point, thereby improving the accuracy of action recognition.

[0072] Figure 5 The illustration shows a flowchart of a second example of obtaining local and global features at each level in the object action recognition method according to an embodiment of this application.

[0073] like Figure 5 As shown, in Figure 4 Based on the illustrated embodiment, step S132 includes the following steps.

[0074] Step S133: The plurality of first sub-local features are grouped according to the adjacency relationship of the object's local structure, and then concatenated by channel and performed one-dimensional convolution to obtain second sub-local features. For example, as described above... Figure 2Taking the human skeleton diagram shown as an example, grouping the joints according to their parts can represent the adjacent relationships of their local structures. For example, joints 9, 10, 11, 12, 24 and 25 represent the adjacent relationships of the local structures of a human arm.

[0075] Based on the adjacency relationship in the local structure, the first sub-local features can be grouped. For example, the first sub-local features of joints 9 and 10, joints 11 and 12, and joints 24 and 25 can be grouped together, then concatenated by channel into a feature vector, and further one-dimensional convolution can be performed to obtain the second sub-local features representing the semantic information of the hand part.

[0076] For example, as described above, the first sub-local feature can be represented by C*T. Thus, by concatenating channels, a first sub-local feature of, for example, C1*T and a second sub-local feature of, for example, C2*T can be concatenated into a second sub-local feature of (C1+C2)*T.

[0077] like Figure 3 As shown, for multiple first sub-local features A1, A2, A3, ..., Am-1, Am, they are grouped and then concatenated by channel and subjected to one-dimensional convolution to obtain multiple second sub-local features B1, B2, B3, B4, and B5. For example, as shown in... Figure 2 Taking the human skeleton diagram shown as an example, the second sub-local features B1, B2, B3, B4 and B5 represent the partial semantic features of the left arm, right arm, left leg, right leg and torso of the human body, respectively.

[0078] Step S134: Obtain the local feature based on the second sub-local feature. Through step S133, second sub-local features representing partial semantic information of a certain part of an object can be obtained. Then, the local feature can be obtained based on the multiple second sub-local features, for example, by concatenating the multiple second sub-local features by channel to obtain the local feature. That is, the local feature has the same dimension as the second sub-local feature, for example, it is still represented as C*T.

[0079] like Figure 3 As shown, local feature F is obtained based on the multiple first sub-local features B1, B2, B3, B4, and B5. L .

[0080] In this way, by grouping the first sub-local features according to the adjacency relationship of the object's local structure and performing channel-wise concatenation and one-dimensional convolution to obtain multiple second sub-local features, the obtained local features can reflect the semantic information of the relationship between the various key features of the object according to the local structure, thereby making action recognition based on local features more accurate.

[0081] Furthermore, such as Figure 5 As shown, in Figure 4 Based on the illustrated embodiment, step S142 includes the following steps.

[0082] S143, the second sub-local features are dimensionally concatenated, and the second sub-global features are obtained from the concatenated second sub-local features using the global network. That is, by dimensionally concatenating the multiple second sub-local features, global features can be obtained by using a two-dimensional convolutional global network.

[0083] For example, as mentioned above, the second sub-local feature is still represented by C*T. The multiple second sub-local features are concatenated by dimensions to obtain a global feature of C*T*N, where N represents the number of dimensions. Therefore, for the global feature, the second sub-global feature can be obtained, for example, by using a two-dimensional convolutional neural network as the global network.

[0084] For example, such as Figure 3 As shown, the multiple second sub-local features B1, B2, B3, B4, and B5 are passed through the global network G2 to obtain the global feature F. G2 .

[0085] S144, combine the first sub-global feature and the second sub-global feature to obtain the global feature. That is, the global feature has the same dimension as the first sub-global feature and the second sub-global feature, for example, it is still represented by C*T*N.

[0086] For example, such as Figure 3 As shown, the global feature F G1 and global features F G2 Cascade to obtain global feature F G .

[0087] In this way, by obtaining global features through second sub-local features that can reflect the adjacency relationship between each key point according to the local structure, the global features can contain semantic information reflecting the relationship between the features of each key point according to the local structure, thereby improving the accuracy of action recognition.

[0088] Further reference Figure 3 Based on the second sub-local features B1, B2, B3, B4 and B5, the objects can be further grouped according to their respective halves, and then concatenated by channel and one-dimensional convolution can be performed to obtain the third sub-local features that can contain the semantic information of the objects according to their respective halves.

[0089] like Figure 3As shown, the second sub-local features B1, B2, B3, B4, and B5 are grouped according to the left, right, upper, and lower halves of the object, and then concatenated by channel and performed one-dimensional convolution to obtain four third sub-local features C1, C2, C3, and C4. Then, the local feature F is obtained based on these four third sub-local features C1, C2, C3, and C4. L .

[0090] In this way, by grouping the second sub-local features according to the semantic relationship between each half of the object and performing channel-wise concatenation and one-dimensional convolution to obtain multiple third sub-local features, the obtained local features can reflect the semantic information of the relationship between the key features of the object according to the relationship between each half of the object, thereby making action recognition based on local features more accurate.

[0091] It is worth noting that, in the embodiments of this application, the division of the object into its halves is not limited to the division into left half, right half, upper half and lower half. It can also be the division into upper left half, upper right half, lower left half and lower right half, or other division methods, as long as the whole is divided into a specified number of parts according to the structural semantics of the object.

[0092] Further reference Figure 3 Based on the third sub-local features C1, C2, C3 and C4, each pair of features can be further grouped and concatenated by channel and one-dimensional convolution to obtain two fourth sub-local features that can contain the overall semantic information of the object.

[0093] like Figure 3 As shown, the third sub-local features C1, C2, C3, and C4 are grouped according to the upper and lower halves and left and right halves of the object, and then concatenated by channel and one-dimensional convolution are performed to obtain two fourth sub-local features D1 and D2. Then, the local feature F is obtained based on the two fourth sub-local features D1 and D2. L .

[0094] In this way, by grouping the third sub-local features according to the semantic relationships of the overall structure of the object and performing channel-wise concatenation and one-dimensional convolution to obtain two fourth sub-local features, the obtained local features can reflect the semantic information of the relationship between the various key features of the object according to the overall structure of the object, thereby making action recognition based on local features more accurate.

[0095] It is worth noting that, in the embodiments of this application, the grouping of the upper and lower halves and the left and right halves of the object is only an example. Grouping can also be done according to the upper left and lower right halves of the object, or other grouping methods. Furthermore, when using a division method such as upper left, upper right, lower left, and lower right halves, grouping can also be done according to the upper left and upper right halves of the object, and the lower left and lower right halves of the object.

[0096] Further reference Figure 3 Based on two fourth sub-local features D1 and D2, they can first be concatenated by channel to obtain concatenated local features E, and then one-dimensional convolution can be performed on the concatenated local features E to obtain the local features F. L .

[0097] In this way, by concatenating two fourth sub-local features, the obtained local features can include as much of the semantic information of the object as possible, and by extracting the structural information of all local features through one-dimensional convolution, thus making action recognition based on local features more accurate.

[0098] Below, we will refer to Figure 6 To illustrate the global network in the object action recognition method according to embodiments of this application.

[0099] Figure 6 The illustration shows a schematic architecture diagram of the global network in the object action recognition method according to an embodiment of this application.

[0100] like Figure 6 As shown, for example, for such Figure 3 The multiple first sub-local features A1, A2, A3, ..., Am-1, Am shown are concatenated dimensionally to obtain the first concatenated sub-local feature H1. Assuming that each first sub-local feature is C*T, then the first concatenated sub-local feature H1 is C*T*m.

[0101] Then, as Figure 6As shown, the first spliced ​​sub-local feature H1 is subjected to a first two-dimensional convolution R1 to obtain a first two-dimensional convolution feature H2. Next, the first two-dimensional convolution feature H2 is subjected to a dimension transformation DM1 to obtain a first dimension-transformed feature H3. Then, the first dimension-transformed feature H3 is subjected to a second two-dimensional convolution R2 to obtain a second two-dimensional convolution feature H4. Then, the second two-dimensional convolution feature H4 is subjected to pooling PO1 to obtain a first pooling feature H5. Then, the first pooling feature H5 is subjected to a third two-dimensional convolution R3 to obtain a third two-dimensional convolution feature H6. Then, the third two-dimensional convolution feature H6 is subjected to pooling PO2 to obtain a second pooling feature H7. Then, the second pooling feature H7 is subjected to a fourth two-dimensional convolution R4 to obtain a fourth two-dimensional convolution feature H8. Finally, the fourth two-dimensional convolution feature H8 is input into a fully connected layer FC1 to obtain the first sub-global feature F. G1 .

[0102] Similarly, for such Figure 3 The multiple second sub-local features B1, B2, B3, B4 and B5 shown are concatenated dimensionally to obtain the second concatenated sub-local feature K1. Assuming that each second sub-local feature is C*T, then the second concatenated sub-local feature K1 is C*T*5.

[0103] Then, as Figure 6 As shown, the fifth two-dimensional convolution R5 is performed on the second concatenated sub-local feature K1 to obtain the fifth two-dimensional convolution feature K2. Then, the fifth two-dimensional convolution feature K2 is transformed by dimension DM2 to obtain the second dimension transformed feature K3. Then, the second dimension transformed feature K3 is transformed by a sixth two-dimensional convolution R6 to obtain the sixth two-dimensional convolution feature K4. Then, the sixth two-dimensional convolution feature K4 is pooled by PO3 to obtain the third pooled feature K5. Then, the third pooled feature K5 is transformed by a seventh two-dimensional convolution R7 to obtain the seventh two-dimensional convolution feature K6. Then, the seventh two-dimensional convolution feature K6 is pooled by PO4 to obtain the fourth pooled feature K7. Then, the fourth pooled feature K7 is transformed by an eighth two-dimensional convolution R8 to obtain the eighth two-dimensional convolution feature K8. Finally, the eighth two-dimensional convolution feature K8 is input into the fully connected layer FC2 to obtain the second sub-global feature F. G2 .

[0104] Thus, through such Figure 6 The global network shown can effectively learn the global features of object actions, thereby improving the accuracy of action recognition.

[0105] Exemplary device

[0106] Figure 7 The illustration shows a block diagram of an object action recognition device according to an embodiment of this application.

[0107] like Figure 7 As shown, the object action recognition device 200 according to an embodiment of this application includes: a coordinate acquisition unit 210, used to acquire a series of coordinates of each key point in a continuous time period among a plurality of key points of an object; a point-level feature acquisition unit 220, used to perform a one-dimensional convolution on the series of coordinates of each key point acquired by the coordinate acquisition unit 210 to obtain point-level features of the object; a local feature acquisition unit 230, used to group and perform one-dimensional convolution on the point-level features obtained by the point-level feature acquisition unit 220 to obtain local features; a global feature acquisition unit 240, used to use a global network to obtain global features based on the point-level features obtained by the point-level feature acquisition unit 220; and an object action recognition unit 250, used to recognize the action of the object based on the local features obtained by the local feature acquisition unit 230 and the global features obtained by the global feature acquisition unit 240.

[0108] Figure 8 The diagram illustrates a block diagram of a first example of a local feature acquisition unit and a global feature acquisition unit in an object action recognition device according to an embodiment of this application.

[0109] like Figure 8 As shown, in Figure 7 Based on the embodiment shown, the local feature acquisition unit 230 includes: a grouped convolution subunit 231, used to group the point-level features obtained by the point-level feature acquisition unit 220 according to semantic connection relationships and perform channel-wise concatenation and one-dimensional convolution to obtain multiple first sub-local features; and a local feature acquisition subunit 232, used to obtain the local features based on the multiple first sub-local features obtained by the grouped convolution subunit 231.

[0110] The global feature acquisition unit 240 includes: a global network subunit 241, used to concatenate the multiple first sub-local features obtained by the grouped convolution subunit 231 and use the global network to obtain the first sub-global feature from the concatenated first sub-local features; and a global feature acquisition subunit 242, used to obtain the global feature based on the first sub-global feature obtained by the global network subunit 241.

[0111] Figure 9 The diagram illustrates a second example of a local feature acquisition unit and a global feature acquisition unit in an object action recognition device according to an embodiment of this application.

[0112] like Figure 9 As shown, in Figure 8Based on the embodiment shown, the local feature acquisition subunit 232 includes: a grouped convolution module 233, used to group the plurality of first sub-local features obtained by the grouped convolution subunit 231 according to the adjacency relationship on the local structure of the object and perform channel-wise concatenation and one-dimensional convolution to obtain second sub-local features; and a local feature acquisition module 234, used to obtain the local features based on the second sub-local features obtained by the grouped convolution module 233.

[0113] The global feature acquisition subunit 242 includes: a global network module 243, used to concatenate the second sub-local features obtained by the grouped convolution module 233 and use the global network to obtain a second sub-global feature from the concatenated second sub-local features; and a global feature acquisition module 244, used to combine the first sub-global feature obtained by the global network subunit 241 and the second sub-global feature obtained by the global network module 243 to obtain the global feature.

[0114] In one example, in the object action recognition device 200 described above, the local feature acquisition module 234 is used to: group the second sub-local features according to the left half, right half, upper half and lower half of the object and perform channel-wise concatenation and one-dimensional convolution to obtain four third sub-local features; and obtain the local features based on the four third sub-local features.

[0115] In one example, in the object action recognition device 200 described above, the local feature acquisition module 234 is used to: group the four third sub-local features according to the upper and lower halves and left and right halves of the object and perform channel-wise concatenation and one-dimensional convolution to obtain two fourth sub-local features; and obtain the local features based on the two fourth sub-local features.

[0116] In one example, in the object action recognition device 200 described above, the local feature acquisition module 234 is used to: concatenate the two fourth sub-local features by channel to obtain concatenated local features; and to perform a one-dimensional convolution on the concatenated local features to obtain the local features.

[0117] In one example, in the object action recognition device 200 described above, the global network subunit 241 is configured to: concatenate the plurality of first sub-local features in terms of dimensions to obtain first concatenated sub-local features; perform a first two-dimensional convolution on the first concatenated sub-local features to obtain first two-dimensional convolution features; perform a dimensional transformation on the first two-dimensional convolution features to obtain first dimensional transformation features; perform a second two-dimensional convolution on the first dimensional transformation features to obtain second two-dimensional convolution features; perform pooling on the second two-dimensional convolution features to obtain first pooling features; perform a third two-dimensional convolution on the first pooling features to obtain third two-dimensional convolution features; perform pooling on the third two-dimensional convolution features to obtain second pooling features; perform a fourth two-dimensional convolution on the second pooling features to obtain fourth two-dimensional convolution features; and input the fourth two-dimensional convolution features into a fully connected layer to obtain the first sub-global features.

[0118] In one example, in the object action recognition device 200 described above, the global network module 243 is configured to: concatenate the plurality of second sub-local features in terms of dimensions to obtain second concatenated sub-local features; perform a fifth two-dimensional convolution on the second concatenated sub-local features to obtain a fifth two-dimensional convolution feature; perform a dimensional transformation on the fifth two-dimensional convolution feature to obtain a second dimensional transformation feature; perform a sixth two-dimensional convolution on the second dimensional transformation feature to obtain a sixth two-dimensional convolution feature; perform pooling on the sixth two-dimensional convolution feature to obtain a third pooling feature; perform a seventh two-dimensional convolution on the third pooling feature to obtain a seventh two-dimensional convolution feature; perform pooling on the seventh two-dimensional convolution feature to obtain a fourth pooling feature; perform an eighth two-dimensional convolution on the fourth pooling feature to obtain an eighth two-dimensional convolution feature; and input the eighth two-dimensional convolution feature into a fully connected layer to obtain the second sub-global feature.

[0119] Here, those skilled in the art will understand that the specific functions and operations of each unit and module in the object action recognition device 200 described above have been referenced above. Figures 1 to 6 The object action recognition method has been described in detail in the description of the method, and therefore, its repeated description will be omitted.

[0120] As described above, the object motion recognition device 200 according to the embodiments of this application can be implemented in various terminal devices, such as cameras for recognizing human movements. In one example, the object motion recognition device 200 according to the embodiments of this application can be integrated into the terminal device as a software module and / or a hardware module. For example, the object motion recognition device 200 can be a software module in the operating system of the terminal device, or it can be an application developed for the terminal device; of course, the object motion recognition device 200 can also be one of many hardware modules of the terminal device.

[0121] Alternatively, in another example, the object motion recognition device 200 and the terminal device can also be separate devices, and the object motion recognition device 200 can be connected to the terminal device via wired and / or wireless networks, and transmit interactive information in accordance with an agreed data format.

[0122] Exemplary electronic devices

[0123] Below, for reference Figure 10 This describes an electronic device according to embodiments of the present application.

[0124] Figure 10 A block diagram of an electronic device according to an embodiment of this application is illustrated.

[0125] like Figure 10 As shown, the electronic device 10 includes one or more processors 11 and memory 12.

[0126] The processor 13 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0127] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the object action recognition methods of the various embodiments of this application described above and / or other desired functions. Various contents such as point-level features, local features, and global features may also be stored in the computer-readable storage medium.

[0128] In one example, the electronic device 10 may also include an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0129] The input device 13 may include, for example, a keyboard, a mouse, etc.

[0130] The output device 14 can output various information to the outside, including the results of object action recognition. The output device 14 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0131] Of course, for the sake of simplicity, Figure 10 Only some of the components of the electronic device 10 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0132] Exemplary computer program products and computer-readable storage media

[0133] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the object action recognition methods according to various embodiments of this application as described in the "Exemplary Methods" section of this specification.

[0134] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0135] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the object action recognition method according to various embodiments of this application described in the "Exemplary Methods" section of this specification.

[0136] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0137] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0138] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0139] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0140] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0141] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. An object action recognition method, comprising: Obtain a series of coordinates over a continuous time period for each of the multiple key points of an object; One-dimensional convolution is performed on the series of coordinates of each key point to obtain the point-level features of the object; each point-level feature is represented in the form of C*T, where C represents the number of channels of the point-level feature and T represents the multiple different time points represented by the point-level feature; Based on the point-level features, multi-level grouping and one-dimensional convolution are performed to obtain local features; Based on the point-level features, a global network is used to obtain global features; and The actions of the object are identified based on the local features and the global features; The global network is a two-dimensional convolutional neural network; Among them, obtaining local features by performing multi-level grouping and one-dimensional convolution based on the point-level features includes: The point-level features are grouped according to semantic connectivity and then concatenated by channel and subjected to one-dimensional convolution to obtain multiple first sub-local features; and Obtaining the local feature based on the plurality of first sub-local features includes: grouping the plurality of first sub-local features according to the adjacency relationship in the local structure of the object and performing channel-wise concatenation and one-dimensional convolution to obtain second sub-local features; and obtaining the local feature based on the second sub-local features. The use of a global network based on the point-level features to obtain global features includes: The plurality of first sub-local features are concatenated dimensionally, and the first sub-global feature is obtained from the concatenated first sub-local features using the global network; and Obtaining the global feature based on the first sub-global feature includes: concatenating the dimensions of the second sub-local feature and using the global network to obtain the second sub-global feature from the concatenated second sub-local feature; and combining the first sub-global feature and the second sub-global feature to obtain the global feature. The actions for identifying the object based on the local features and the global features include: After dimensional transformation of the local features, they are concatenated with the global features to obtain a feature map representing the object's actions, and the object's actions are identified based on the feature map.

2. The object action recognition method as described in claim 1, wherein, Obtaining the local features based on the second sub-local features includes: The second sub-local features are grouped according to the left, right, upper, and lower halves of the object, and then concatenated by channel and subjected to one-dimensional convolution to obtain four third sub-local features; and The local features are obtained based on the four third sub-local features.

3. The object action recognition method as described in claim 2, wherein, The local features obtained based on the four third sub-local features include: The four third sub-local features are grouped according to the upper and lower halves and left and right halves of the object, and then concatenated by channel and one-dimensional convolution is performed to obtain two fourth sub-local features; and The local features are obtained based on the two fourth sub-local features.

4. The object action recognition method as described in claim 3, wherein, The local features obtained based on the two fourth sub-local features include: The two fourth sub-local features are concatenated by channel to obtain concatenated local features; and The spliced ​​local features are obtained by performing a one-dimensional convolution.

5. The object action recognition method as described in claim 1, wherein, The process of concatenating the multiple first sub-local features dimensionally and obtaining the first sub-global feature from the concatenated first sub-local features using the global network includes: The multiple first sub-local features are concatenated dimensionally to obtain the first concatenated sub-local features; Perform a first two-dimensional convolution on the first spliced ​​sub-local features to obtain the first two-dimensional convolutional features; The first two-dimensional convolutional features are subjected to dimensionality transformation to obtain the first dimension-transformed features; The first-dimensional transformed features are subjected to a second two-dimensional convolution to obtain the second two-dimensional convolutional features; Pooling is performed on the second two-dimensional convolutional features to obtain the first pooled features; The first pooling feature is subjected to a third two-dimensional convolution to obtain a third two-dimensional convolution feature; The third two-dimensional convolutional features are pooled to obtain the second pooled features; The second pooling feature is subjected to a fourth two-dimensional convolution to obtain a fourth two-dimensional convolution feature; and The fourth two-dimensional convolutional feature is input into the fully connected layer to obtain the first sub-global feature.

6. The object action recognition method as described in claim 1, wherein, The process of concatenating the second sub-local features dimensionally and obtaining the second sub-global features from the concatenated second sub-local features using the global network includes: The multiple second sub-local features are concatenated dimensionally to obtain the second concatenated sub-local features; The second spliced ​​sub-local features are subjected to a fifth two-dimensional convolution to obtain the fifth two-dimensional convolution features; The fifth two-dimensional convolutional feature is subjected to dimensionality transformation to obtain the second dimensional transformation feature; The second-dimensional transformed features are subjected to a sixth two-dimensional convolution to obtain the sixth two-dimensional convolutional features; The sixth two-dimensional convolutional feature is pooled to obtain the third pooled feature; The third pooling feature is subjected to a seventh two-dimensional convolution to obtain a seventh two-dimensional convolution feature; The seventh two-dimensional convolutional feature is pooled to obtain the fourth pooled feature; The fourth pooling feature is subjected to an eighth two-dimensional convolution to obtain an eighth two-dimensional convolution feature; and The eighth two-dimensional convolutional feature is input into the fully connected layer to obtain the second sub-global feature.

7. An object action recognition device, comprising: The coordinate acquisition unit is used to acquire a series of coordinates of each key point over a continuous time period among multiple key points of an object; A point-level feature acquisition unit is used to perform a one-dimensional convolution on the series of coordinates of each key point acquired by the coordinate acquisition unit to obtain the point-level features of the object; each point-level feature is represented in the form of C*T, where C represents the number of channels of the point-level feature and T represents multiple different time points represented by the point-level feature. The local feature acquisition unit is used to perform multi-level grouping and one-dimensional convolution based on the point-level features obtained by the point-level feature acquisition unit to obtain local features; A global feature acquisition unit is configured to obtain global features using a global network based on the point-level features obtained by the point-level feature acquisition unit, wherein the global network is a two-dimensional convolutional neural network; and An object action recognition unit is used to recognize the action of the object based on the local features obtained by the local feature acquisition unit and the global features obtained by the global feature acquisition unit; Among them, grouping and one-dimensional convolution based on the point-level features to obtain local features includes: The point-level features are grouped according to semantic connectivity and then concatenated by channel and subjected to one-dimensional convolution to obtain multiple first sub-local features; and Obtaining the local feature based on the plurality of first sub-local features includes: grouping the plurality of first sub-local features according to the adjacency relationship in the local structure of the object and performing channel-wise concatenation and one-dimensional convolution to obtain second sub-local features; and obtaining the local feature based on the second sub-local features. The use of a global network based on the point-level features to obtain global features includes: The plurality of first sub-local features are concatenated dimensionally, and the first sub-global feature is obtained from the concatenated first sub-local features using the global network; and Obtaining the global feature based on the first sub-global feature includes: concatenating the dimensions of the second sub-local feature and using the global network to obtain the second sub-global feature from the concatenated second sub-local feature; and combining the first sub-global feature and the second sub-global feature to obtain the global feature. The actions for identifying the object based on the local features and the global features include: After dimensional transformation of the local features, they are concatenated with the global features to obtain a feature map representing the object's actions, and the object's actions are identified based on the feature map.

8. An electronic device, comprising: processor; as well as A memory storing computer program instructions that, when executed by the processor, cause the processor to perform the object action recognition method as described in any one of claims 1-6.

Images