Method, device and equipment for identifying motion of moving objects, and non-volatile storage medium

Abstract

This application discloses a method, apparatus, and device for identifying the motion of a moving object, as well as a non-volatile storage medium. The method comprises: acquiring a motion image of the moving object; analyzing the motion image to obtain a sequence of joint points of the moving object; extracting spatiotemporal dynamic features of the motion of the moving object from the sequence of joint points; and identifying the motion of the moving object based on the spatiotemporal dynamic features. This application addresses the technical problem in related art of ignoring the dependencies between interspaced, non-directly connected joint points when identifying pedestrian motion, resulting in inaccurate recognition results.

Description

Technical Field

[0001] The present application relates to the field of behavior recognition technology, and in particular to a method, apparatus and device for identifying the actions of moving objects, and a non-volatile storage medium. Background Art

[0002] Nowadays, deep learning technology is widely used in various fields. In the field of computer vision, action recognition is the first step in the development from static vision to dynamic vision. The biggest feature of pedestrian skeleton data is that it is a non-Euclidean space graph data, with bones only linking specific joints.

[0003] Because pedestrian skeleton data is highly robust to complex scenarios such as viewpoint, occlusion, and lighting, joint-based pedestrian action recognition has long been considered a promising research topic. Existing methods primarily use graph convolutional neural networks (GCNs) for action recognition. These methods focus on learning the topological structure of graphs, specifically the relationships between directly connected nodes in the graph data (such as wrists and fingers), while ignoring the dependencies between intervening, indirect nodes (such as hands and feet). However, in human actions, hands and feet are often highly correlated, and this information is crucial for pedestrian action recognition. Furthermore, GCN-based methods require numerous model parameters and heavy computation, making them unsuitable for practical deployment.

[0004] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0005] The embodiments of the present application provide a method, apparatus, and device, as well as a non-volatile storage medium for identifying the actions of moving objects, to at least solve the technical problem in the related art of ignoring the dependencies between interspaced and non-directly connected joints when identifying pedestrian actions, resulting in inaccurate recognition results.

[0006] According to one aspect of an embodiment of the present application, a method for identifying a moving object action is provided, comprising: acquiring a moving object action image; analyzing the moving object action image to obtain a moving object joint point sequence; extracting spatiotemporal dynamic features of the moving object action from the moving object joint point sequence; and identifying the moving object action based on the spatiotemporal dynamic features.

[0007] According to another aspect of an embodiment of the present application, a method for identifying the action of a moving object is also provided, including: obtaining an action image of a moving object; analyzing the above-mentioned action image of the moving object to obtain a sequence of joint points of the moving object; setting the above-mentioned joint point sequence of the moving object as an input parameter of a spatiotemporal graph convolutional network model, and outputting the recognition result of the action of the moving object, wherein the above-mentioned spatiotemporal graph convolutional network model includes: a multi-layer spatiotemporal graph convolution calculation unit, and the above-mentioned multi-layer spatiotemporal graph convolution calculation unit is used to extract the spatiotemporal dynamic features of the above-mentioned action of the moving object from the above-mentioned joint point sequence of the moving object, and the above-mentioned spatiotemporal dynamic features are used to predict the above-mentioned recognition result.

[0008] According to another aspect of an embodiment of the present application, a method for identifying the action of a moving object is also provided, including: responding to a control operation received by a client, obtaining an action recognition request, wherein the information carried in the above-mentioned action recognition request includes: an action image of the moving object to be identified; based on the above-mentioned action recognition request, calling a software service of a server on the above-mentioned client to analyze the above-mentioned action image of the moving object to obtain a sequence of joint points of the moving object, and performing action recognition on the above-mentioned joint point sequence of the moving object to obtain an identification result of the action of the moving object, wherein the above-mentioned software service is used to provide a spatiotemporal graph convolutional network model, and the above-mentioned spatiotemporal graph convolutional network model includes: a multi-layer spatiotemporal graph convolution calculation unit, and the above-mentioned multi-layer spatiotemporal graph convolution calculation unit is used to extract the spatiotemporal dynamic features of the above-mentioned moving object action from the above-mentioned motion object joint point sequence, and the above-mentioned spatiotemporal dynamic features are used to predict the above-mentioned recognition result.

[0009] According to another aspect of an embodiment of the present application, a method for identifying the action of a moving object is also provided, including: receiving an action recognition request from a client, wherein the information carried in the above-mentioned action recognition request includes: an action image of the moving object to be identified; based on the above-mentioned action recognition request, calling a software service on the server to analyze the above-mentioned action image of the moving object to obtain a sequence of joint points of the moving object, and performing action recognition on the above-mentioned joint point sequence of the moving object to obtain a recognition result of the action of the moving object, wherein the above-mentioned software service is used to provide a spatiotemporal graph convolutional network model, and the above-mentioned spatiotemporal graph convolutional network model includes: a multi-layer spatiotemporal graph convolution calculation unit, and the above-mentioned multi-layer spatiotemporal graph convolution calculation unit is used to extract the spatiotemporal dynamic features of the above-mentioned moving object action from the above-mentioned joint point sequence of the moving object, and the above-mentioned spatiotemporal dynamic features are used to predict the above-mentioned recognition result; and feeding back the above-mentioned recognition result to the above-mentioned client.

[0010] According to another aspect of an embodiment of the present application, a device for identifying the action of a moving object is also provided, including: an acquisition module for acquiring an action image of a moving object; an analysis module for analyzing the above-mentioned action image of the moving object to obtain a sequence of joint points of the moving object; an extraction module for extracting the spatiotemporal dynamic features of the action of the moving object from the above-mentioned sequence of joint points of the moving object; and an identification module for identifying the above-mentioned action of the moving object based on the above-mentioned spatiotemporal dynamic features.

[0011] According to another aspect of an embodiment of the present application, a device for identifying the action of a moving object is also provided, including: an acquisition unit for acquiring an action image of a moving object; an analysis unit for analyzing the above-mentioned action image of the moving object to obtain a sequence of joint points of the moving object; an identification unit for setting the above-mentioned joint point sequence of the moving object as an input parameter of a spatiotemporal graph convolutional network model, and outputting the identification result of the action of the moving object, wherein the above-mentioned spatiotemporal graph convolutional network model includes: a multi-layer spatiotemporal graph convolution calculation unit, and the above-mentioned multi-layer spatiotemporal graph convolution calculation unit is used to extract the spatiotemporal dynamic features of the above-mentioned action of the moving object from the above-mentioned joint point sequence of the moving object, and the above-mentioned spatiotemporal dynamic features are used to predict the above-mentioned recognition result.

[0012] According to another aspect of an embodiment of the present application, a non-volatile storage medium is further provided, wherein the non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute any one of the above-mentioned motion object action recognition methods.

[0013] According to another aspect of an embodiment of the present application, a motion object action recognition device is also provided, including: a processor; and a memory, connected to the above-mentioned processor, for providing the above-mentioned processor with instructions for processing the following processing steps: obtaining a motion object action image; analyzing the above-mentioned motion object action image to obtain a motion object joint point sequence; extracting the spatiotemporal dynamic features of the motion object action from the above-mentioned motion object joint point sequence; and recognizing the above-mentioned motion object action based on the above-mentioned spatiotemporal dynamic features.

[0014] In an embodiment of the present application, a motion image of a moving object is obtained; the motion image of the moving object is analyzed to obtain a sequence of joint points of the moving object; the spatiotemporal dynamic features of the motion of the moving object are extracted from the sequence of joint points of the moving object; and the motion of the moving object is identified based on the spatiotemporal dynamic features.

[0015] It is easy to notice that the embodiment of the present application determines the joint point sequence of the moving object by analyzing the acquired motion image of the moving object, and simultaneously extracts two features, namely, time and space features, between the joint points from the joint point sequence of the moving object, namely, the spatiotemporal dynamic features. Since the spatiotemporal dynamic features fuse static and dynamic information, the above-mentioned moving object movements are identified based on the spatiotemporal dynamic features, thereby achieving the purpose of paying attention to the relationship between directly connected joint points and the dependency relationship between spaced and non-directly connected joint points at the same time, thereby achieving the technical effect of improving the accuracy of identifying the movement of the moving object, and further solving the technical problem in the related technology of ignoring the dependency relationship between spaced and non-directly connected joint points when identifying pedestrian movements, resulting in inaccurate recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 This is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for identifying motion objects according to an embodiment of the present application;

[0018] Figure 2 is a flow chart of a method for identifying a moving object action according to an embodiment of the present application;

[0019] Figure 3 is a schematic structural diagram of an optional spatiotemporal graph convolutional network model according to an embodiment of the present application;

[0020] Figure 4 is a schematic structural diagram of an optional spatiotemporal graph convolution calculation unit according to an embodiment of the present application;

[0021] Figure 5 is a flow chart of another method for identifying motion of a moving object according to an embodiment of the present application;

[0022] Figure 6 is a flow chart of another method for identifying motion of a moving object according to an embodiment of the present application;

[0023] Figure 7 is a flowchart of another method for identifying motion of a moving object according to an embodiment of the present application;

[0024] Figure 8 1 is a schematic structural diagram of a moving object action recognition device according to an embodiment of the present application;

[0025] Figure 91 is a schematic structural diagram of a moving object action recognition device according to an embodiment of the present application;

[0026] Figure 10 is a structural diagram of a moving object action recognition device according to an embodiment of the present application;

[0027] Figure 11 This is a structural block diagram of a computer terminal according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following interpretations:

[0031] Deep Learning: Deep learning refers to a collection of machine learning algorithms applied to multi-layer neural networks to solve various problems involving images, text, and other data. Deep learning can generally be categorized as a neural network, though its specific implementation varies greatly. The core of deep learning is feature learning, which aims to obtain hierarchical feature information through layered networks, thereby solving important problems that previously required manual feature design.

[0032] Action Recognition: Action recognition studies the movements of targets in a sequence, such as determining whether a person is walking, jumping, or waving. It has important applications in video surveillance, video recommendation, and human-computer interaction.

[0033] Skeleton: A skeleton is typically represented by interconnected joints and bones, used to characterize the positions and connections between the key parts of a moving object. The joints corresponding to interconnected parts of a moving object are also connected, while the joints corresponding to non-adjacent parts are also unconnected. This is a typical example of graph data in a non-Euclidean space.

[0034] Graph Convolutional Networks: Traditional convolutional neural networks can only process Euclidean spatial data, while graph convolutional neural networks are used to process non-Euclidean spatial data, such as traffic networks and social networks. In these data, the local structure of each node is different and no longer meets the translation invariance requirement.

[0035] Example 1

[0036] According to an embodiment of the present application, an embodiment of a method for identifying the action of a moving object is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0037] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for identifying motion objects is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (illustrated as 102a, 102b, ..., 102n) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0038] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0039] The memory 104 can be used for storing software programs and modules of application software, such as the program instructions / data storage device corresponding to the method for motion recognition of moving objects in the embodiment of the present application. The processor 102 performs various functional applications and data processing by running the software programs and modules stored in the memory 104, i.e., realizes the above-mentioned method for motion recognition of moving objects. The memory 104 can include a high-speed random access memory and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memory or other non-volatile solid-state memories. In some instances, the memory 104 can further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. The example of the above-mentioned network includes but is not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0040] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0041] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0042] Under the above operating environment, this application provides Figure 2 An embodiment of a method for identifying a moving object action is shown. Figure 2 is a flow chart of a method for identifying a moving object action according to an embodiment of the present application. Figure 2 As shown, the above-mentioned moving object action recognition method includes:

[0043] Step S202, obtaining a motion image of a moving object;

[0044] Step S204, analyzing the motion image of the moving object to obtain a joint point sequence of the moving object;

[0045] Step S206, extracting the spatiotemporal dynamic features of the motion of the motion object from the joint point sequence of the motion object;

[0046] Step S208: identifying the motion of the moving object based on the spatiotemporal dynamic features.

[0047] In an embodiment of the present application, a motion image of a moving object is obtained; the motion image of the moving object is analyzed to obtain a sequence of joint points of the moving object; the spatiotemporal dynamic features of the motion of the moving object are extracted from the sequence of joint points of the moving object; and the motion of the moving object is identified based on the spatiotemporal dynamic features.

[0048] It is easy to notice that the embodiment of the present application determines the joint point sequence of the moving object by analyzing the acquired motion image of the moving object, and simultaneously extracts two features, namely, time and space features, between the joint points from the joint point sequence of the moving object, namely, the spatiotemporal dynamic features. Since the spatiotemporal dynamic features fuse static and dynamic information, the above-mentioned moving object movements are identified based on the spatiotemporal dynamic features, thereby achieving the purpose of paying attention to the relationship between directly connected joint points and the dependency relationship between spaced and non-directly connected joint points at the same time, thereby achieving the technical effect of improving the accuracy of identifying the movement of the moving object, and further solving the technical problem in the related technology of ignoring the dependency relationship between spaced and non-directly connected joint points when identifying pedestrian movements, resulting in inaccurate recognition results.

[0049] Optionally, the above-mentioned motion object includes one of the following: a human motion object, a human-like motion object, an animal motion object, and a machine-simulated motion object.

[0050] In an optional embodiment, the above-mentioned motion image of the moving object may be a motion image of the moving object walking, jumping, running, waving, swinging arms, etc., and by analyzing the joint points of the moving object in the above-mentioned motion image of the moving object, a motion object joint point sequence is obtained. The embodiment of the present application adopts a new type of graph convolutional neural network model, for example, a spatiotemporal graph convolutional network model, to extract the spatiotemporal dynamic features of the motion of the moving object from the above-mentioned motion object joint point sequence; and then the above-mentioned motion of the moving object is identified based on the above-mentioned spatiotemporal dynamic features to obtain the action category.

[0051] The graph convolutional neural network model proposed in the embodiments of this application is a basic neural network model for joint-based action recognition tasks. The network structure can be arbitrarily edited according to business needs, increasing or decreasing the network depth and width. Under the conditions of the same network width and depth, the embodiments of this application can use 1 / 5 of the network parameters and 1 / 10 of the computational effort to surpass the best existing neural network action recognition algorithms.

[0052] In an optional embodiment, extracting the spatiotemporal dynamic features from the joint point sequence of the moving object includes:

[0053] Step S302: inputting the joint point sequence of the moving object into a spatiotemporal graph convolutional network model, wherein the spatiotemporal graph convolutional network model includes: a multi-layer spatiotemporal graph convolution calculation unit;

[0054] Step S304: Utilize the multi-layer spatiotemporal graph convolution calculation unit to extract the spatiotemporal dynamic features.

[0055] As an optional embodiment, the structural diagram of the above-mentioned spatio-temporal inception graph convolutional network model (STIGCN) is as follows: Figure 3 As shown in the figure, the input Previous layer of the spatiotemporal graph convolutional network model is a sequence of moving object joints (for example, a sequence of human joints) composed of moving object joints, and the output recognition result is the predicted action category.

[0056] like Figure 3 As shown in the figure, the above-mentioned spatio-temporal graph convolution network model is composed of multiple layers of spatio-temporal graph convolution calculation units (Spatio-Temporal Inception Block) connected in front and back, namely, spatio-temporal graph convolution calculation unit 1, spatio-temporal graph convolution calculation unit 2, ... spatio-temporal graph convolution calculation unit N. The function of the spatio-temporal graph convolution calculation unit is to extract the spatio-temporal dynamic features of pedestrian movements. Each spatio-temporal graph convolution calculation unit includes SI (spatial graph convolution unit) and TI (temporal graph convolution unit). The spatio-temporal dynamic features extracted by the multiple layers of spatio-temporal graph convolution calculation units are then subjected to mean pooling processing by the mean pooling layer Avg Pooling of the spatio-temporal graph convolution network model, and neuron random dropout processing is performed by the neuron random dropout layer Dropout, and finally the above-mentioned spatio-temporal dynamic features are output through the fully connected layer FC.

[0057] In an optional embodiment, Figure 4 is a schematic structural diagram of an optional spatiotemporal graph convolution calculation unit according to an embodiment of the present application, such as Figure 4As shown, each layer of the multi-layer spatiotemporal graph convolution calculation unit includes: a spatial path, a temporal path, and a residual path.

[0058] As an optional embodiment, extracting the spatiotemporal dynamic features using the multi-layer spatiotemporal graph convolution calculation unit includes:

[0059] Step S402: extracting spatial features between multiple joints of the motion of the motion object from the joint sequence of the motion object on the spatial branch.

[0060] Step S404: extracting time features between the plurality of joint points from the joint point sequence of the moving object on the time branch;

[0061] Step S406: transmitting the original input information of each layer of spatiotemporal graph convolution calculation unit on the residual branch;

[0062] Step S408 , obtaining the spatiotemporal dynamic features by adding the spatial features, the temporal features and the original input information.

[0063] Optionally, the above-mentioned spatial branch includes: multiple sampling units, multiple convolution units and a fusion unit.

[0064] In an optional embodiment, on the spatial branch, extracting the spatial features from the joint point sequence of the moving object includes:

[0065] Step S502: extracting a plurality of scale connection features corresponding to the plurality of joint points using the plurality of sampling units, wherein each of the plurality of sampling units samples a different scale connection feature;

[0066] Step S504: performing feature processing on the multiple scale connection features using the convolution units corresponding to each sampling unit in the multiple convolution units to obtain multiple processing results;

[0067] Step S506: Use the fusion unit to fuse the multiple processing results to obtain the spatial features.

[0068] Still Figure 4 As shown, the number of sampling units (Adjacency Sampling) on ​​the above control branch can be 4, and the number of convolution units (Convolution) can also be 4.

[0069] In an optional embodiment, the multiplicity of the plurality of scale connection features is determined by the number of the plurality of joint points connected in sequence, and the number of the plurality of sampling units is the same as the multiplicity of the plurality of scale connection features.

[0070] In an embodiment of the present application, the four sampling units in the above-mentioned spatial branch are respectively used to extract multiple scale connection features corresponding to the above-mentioned multiple joint points. For example, each sampling unit is respectively used to extract the relationship features between 1 to 4-fold connections between the joint points. Assume that there are 5 joint points connected in sequence, abcde, where the connection between a and b is called a 1-fold connection, the connection between a and c is called a 2-fold connection, the connection between a and d is called a 3-fold connection, and the connection between a and e is called a 4-fold connection.

[0071] Through the above-mentioned extraction processing of the sampling unit, the spatial branch is able to extract multiple scale connection features, and the convolution units corresponding to each sampling unit in the multiple convolution units in the spatial branch are used to perform feature processing on the above-mentioned multiple scale connection features to obtain multiple processing results; then the above-mentioned fusion unit (Fusion) is used to fuse the above-mentioned multiple processing results (feature combinations) to obtain the above-mentioned spatial features.

[0072] In an optional embodiment, Figure 4 As shown, the above-mentioned time branch includes: at least one sampling unit (Motion Sampling), multiple convolution units (Convolution) and a fusion unit (Fusion).

[0073] As an optional embodiment, in the time branch, extracting the time feature from the joint point sequence of the moving object includes:

[0074] Step S602: using the at least one sampling unit to extract second-order temporal features corresponding to the plurality of joint points;

[0075] Step S604: using at least one of the plurality of convolution units to perform feature processing on the second-order time series features to obtain a plurality of processing results;

[0076] Step S606: Use the fusion unit to fuse the multiple processing results to obtain the time feature.

[0077] In the above optional embodiment, the at least one sampling unit in the above time branch is used to extract the second-order time series features corresponding to the above-mentioned multiple joint points, that is, the motion features of the joint points between adjacent frames, and the second-order time series features are processed by at least one of the above-mentioned multiple convolution units through the convolution unit to obtain multiple processing results, and then the above-mentioned fusion unit is used to fuse the above-mentioned multiple processing results to obtain the above-mentioned time features.

[0078] In an embodiment of the present application, the original input information of each layer of the spatiotemporal graph convolution calculation unit is transmitted on the above-mentioned residual branch; the above-mentioned spatiotemporal dynamic features are obtained by adding the above-mentioned spatial features, the above-mentioned temporal features and the above-mentioned original input information.

[0079] The graph convolutional neural network model provided in the embodiment of the present application has a simple network structure. The network structure can be flexibly edited according to different applications, the depth and width can be increased or decreased, and end-to-end training can be performed. It can use 1 / 5 of the network parameters and 1 / 10 of the computational complexity, which exceeds the best existing neural network action recognition algorithm and is easier to use in practice.

[0080] The embodiment of the present application proposes a new basic network model that can be used for skeletal motion recognition of moving objects, namely a spatiotemporal graph convolutional network model, which can simultaneously extract the dependency relationships between joints of multiple scales, that is, multiple scale connection features corresponding to multiple joints. Moreover, the spatiotemporal graph convolutional network model can simultaneously extract both time and space features between joints, integrating static and dynamic information, thereby achieving the purpose of paying attention to the relationship between directly connected joints and the dependency relationships between spaced and non-directly connected joints at the same time, thereby achieving the technical effect of improving the accuracy of identifying the motion of moving objects.

[0081] As an optional embodiment, the present application solution can, but is not limited to, use the open source deep learning algorithm framework PyTorch for algorithm development, and all newly developed codes are Python codes.

[0082] Under the above operating environment, this application provides Figure 5 Another motion object action recognition method shown in FIG. Figure 5 is a flow chart of another method for identifying motion of a moving object according to an embodiment of the present application. Figure 5 As shown, the above-mentioned moving object action recognition method includes:

[0083] Step S702, obtaining a motion image of a moving object;

[0084] Step S704, analyzing the motion image of the moving object to obtain a joint point sequence of the moving object;

[0085] Step S706: Set the above-mentioned motion object joint point sequence as the input parameter of the spatiotemporal graph convolutional network model, and output the recognition result of the motion of the motion object, wherein the above-mentioned spatiotemporal graph convolutional network model includes: a multi-layer spatiotemporal graph convolution calculation unit, and the above-mentioned multi-layer spatiotemporal graph convolution calculation unit is used to extract the spatiotemporal dynamic features of the above-mentioned motion of the motion object from the above-mentioned motion object joint point sequence, and the above-mentioned spatiotemporal dynamic features are used to predict the above-mentioned recognition result.

[0086] In an embodiment of the present application, a motion image of a moving object is obtained; the motion image of the moving object is analyzed to obtain a sequence of joint points of the moving object; the sequence of joint points of the moving object is set as an input parameter of a spatiotemporal graph convolutional network model, and the recognition result of the motion of the moving object is output, wherein the spatiotemporal graph convolutional network model includes: a multi-layer spatiotemporal graph convolution calculation unit, the multi-layer spatiotemporal graph convolution calculation unit is used to extract the spatiotemporal dynamic features of the motion of the moving object from the spatiotemporal joint sequence of the moving object, and the spatiotemporal dynamic features are used to predict the recognition result.

[0087] It is easy to notice that the embodiment of the present application determines the joint point sequence of the moving object by analyzing the acquired motion image of the moving object, and uses the spatiotemporal graph convolutional network model to simultaneously extract the time and space features between the joint points from the joint point sequence of the moving object, namely the spatiotemporal dynamic features. Since the spatiotemporal dynamic features fuse static and dynamic information, and the above-mentioned moving object movements are identified based on the spatiotemporal dynamic features, the purpose of paying attention to the relationship between directly connected joint points and the dependency relationship between spaced and non-directly connected joint points is achieved, thereby achieving the technical effect of improving the accuracy of identifying the movement of the moving object, and further solving the technical problem in the related technology of ignoring the dependency relationship between spaced and non-directly connected joint points when identifying pedestrian movements, resulting in inaccurate recognition results.

[0088] Optionally, the above-mentioned motion object includes one of the following: a human motion object, a human-like motion object, an animal motion object, or a machine-simulated motion object.

[0089] In an optional embodiment, the above-mentioned motion image of the moving object may be a motion image of the moving object walking, jumping, running, waving, swinging arms, etc., and by analyzing the joint points of the moving object in the above-mentioned motion image of the moving object, a motion object joint point sequence is obtained. The embodiment of the present application adopts a new type of graph convolutional neural network model, for example, a spatiotemporal graph convolutional network model, to extract the spatiotemporal dynamic features of the motion of the moving object from the above-mentioned motion object joint point sequence; and then the above-mentioned motion of the moving object is identified based on the above-mentioned spatiotemporal dynamic features to obtain the action category.

[0090] As an optional embodiment, the input of the spatio-temporal graph convolutional network model is a sequence of moving object joints composed of moving object joints, and the output is the predicted action category. The above-mentioned spatio-temporal graph convolutional network model is composed of multiple layers of spatio-temporal graph convolution calculation units (Spatio-Temporal Inception Block) connected front to back. The function of the spatio-temporal graph convolution calculation unit is to extract the spatio-temporal dynamic characteristics of pedestrian movements.

[0091] The graph convolutional neural network model proposed in the embodiments of this application is a basic neural network model for joint-based action recognition tasks. The network structure can be arbitrarily edited according to business needs, increasing or decreasing the network depth and width. Under the conditions of the same network width and depth, the embodiments of this application can use 1 / 5 of the network parameters and 1 / 10 of the computational effort to surpass the best existing neural network action recognition algorithms.

[0092] Under the above operating environment, this application provides Figure 6 Another motion object action recognition method shown in FIG. Figure 6 is a flow chart of another method for identifying motion of a moving object according to an embodiment of the present application. Figure 6 As shown, the above-mentioned moving object action recognition method includes:

[0093] Step S802: Responding to the control operation received by the client, obtaining a motion recognition request, wherein the information carried in the motion recognition request includes: a motion image of the moving object to be recognized;

[0094] Step S804: Based on the action recognition request, the software service of the server is called on the client to analyze the action image of the moving object to obtain a sequence of joint points of the moving object, and the action recognition is performed on the joint point sequence of the moving object to obtain a recognition result of the action of the moving object. The software service is used to provide a spatiotemporal graph convolutional network model, and the spatiotemporal graph convolutional network model includes: a multi-layer spatiotemporal graph convolution calculation unit, and the multi-layer spatiotemporal graph convolution calculation unit is used to extract the spatiotemporal dynamic features of the motion of the moving object from the joint point sequence of the moving object, and the spatiotemporal dynamic features are used to predict the recognition result.

[0095] In an embodiment of the present application, an action recognition request is obtained by the client responding to the received control operation, wherein the information carried in the above-mentioned action recognition request includes: an action image of the moving object to be identified; based on the above-mentioned action recognition request, the software service of the server is called on the above-mentioned client to analyze the above-mentioned action image of the moving object to obtain a sequence of joint points of the moving object, and action recognition is performed on the above-mentioned joint point sequence of the moving object to obtain an identification result of the action of the moving object, wherein the above-mentioned software service is used to provide a spatiotemporal graph convolutional network model, and the above-mentioned spatiotemporal graph convolutional network model includes: a multi-layer spatiotemporal graph convolution calculation unit, and the above-mentioned multi-layer spatiotemporal graph convolution calculation unit is used to extract the spatiotemporal dynamic features of the above-mentioned moving object action from the above-mentioned motion object joint point sequence, and the above-mentioned spatiotemporal dynamic features are used to predict the above-mentioned recognition result.

[0096] It is easy to notice that the embodiment of the present application obtains an action recognition request by responding to the received control operation through the client. Based on the above action recognition request, the server-side software service is called on the above client to analyze the obtained motion image of the moving object, determine the motion object joint sequence, and perform action recognition on the above motion object joint sequence to obtain the recognition result of the motion of the moving object, that is, the multi-layer spatiotemporal graph convolution calculation unit in the spatiotemporal graph convolution network model provided by the software service is used to extract the spatiotemporal dynamic features of the above motion object action from the above motion object joint sequence, and predict the above recognition result. Since the spatiotemporal dynamic features fuse static and dynamic information, and the motion of the moving object is recognized based on the spatiotemporal dynamic features, the purpose of paying attention to the relationship between directly connected joints and the dependency between spaced and non-directly connected joints at the same time is achieved, thereby achieving the technical effect of improving the accuracy of identifying the motion of the moving object, and further solving the technical problem in the related technology that the dependency between spaced and non-directly connected joints is ignored when identifying pedestrian actions, resulting in inaccurate recognition results.

[0097] Optionally, the above-mentioned motion object includes one of the following: a human motion object, a human-like motion object, an animal motion object, and a machine-simulated motion object.

[0098] In an optional embodiment, the motion image of the moving object may be a motion image of the moving object walking, jumping, running, waving, swinging arms, and the like.

[0099] It should be noted that the executing entity of the embodiment of the present application is the client, which responds to the received control operation, obtains an action recognition request, and carries the motion image of the motion object to be recognized in the above action recognition request, and sends the action recognition request to the server to request the server to recognize the motion image of the motion object to be recognized. Since the software service of the server provides a new type of graph convolutional neural network model, for example, a spatiotemporal graph convolutional network model, by analyzing the motion object joints in the above motion object motion image, a motion object joint sequence is obtained. The multi-layer spatiotemporal graph convolution calculation unit included in the above spatiotemporal graph convolutional network model extracts the spatiotemporal dynamic features of the motion of the motion object from the above motion object joint sequence; then the motion of the motion object is recognized based on the above spatiotemporal dynamic features, and the action category of the motion of the motion object is predicted.

[0100] As an optional embodiment, the input of the spatio-temporal graph convolutional network model is a sequence of moving object joints composed of moving object joints, and the output is the predicted action category. The above-mentioned spatio-temporal graph convolutional network model is composed of multiple layers of spatio-temporal graph convolution calculation units (Spatio-Temporal Inception Block) connected front to back. The function of the spatio-temporal graph convolution calculation unit is to extract the spatio-temporal dynamic characteristics of pedestrian movements.

[0101] It should be noted that the graph convolutional neural network model proposed in the embodiments of this application is a basic neural network model for joint-based action recognition tasks. The network structure can be arbitrarily edited according to business needs, and the network depth and width can be increased or decreased. Under the conditions of the same network width and depth, the embodiments of this application can use 1 / 5 of the network parameters and 1 / 10 of the computational effort to surpass the best existing neural network action recognition algorithm.

[0102] In the above operating environment, this application also provides Figure 7 Another embodiment of a method for identifying a moving object action is shown in FIG. Figure 7 is a flow chart of another method for identifying motion objects according to an embodiment of the present application. Figure 7 As shown, the above-mentioned moving object action recognition method includes:

[0103] Step S902: receiving a motion recognition request from a client, wherein the information carried in the motion recognition request includes: a motion image of a moving object to be recognized;

[0104] Step S904: Based on the action recognition request, a software service on the server is called to analyze the motion image of the moving object to obtain a sequence of joint points of the moving object, and perform action recognition on the sequence of joint points of the moving object to obtain a recognition result of the motion of the moving object, wherein the software service is used to provide a spatiotemporal graph convolutional network model, and the spatiotemporal graph convolutional network model includes: a multi-layer spatiotemporal graph convolution calculation unit, and the multi-layer spatiotemporal graph convolution calculation unit is used to extract the spatiotemporal dynamic features of the motion of the moving object from the sequence of joint points of the moving object, and the spatiotemporal dynamic features are used to predict the recognition result;

[0105] Step S906: Feedback the recognition result to the client.

[0106] In an embodiment of the present application, a motion recognition request is received from a client through a server, wherein the information carried in the above motion recognition request includes: a motion image of the moving object to be recognized; based on the above motion recognition request, a software service on the server is called to analyze the above motion image to obtain a motion object joint sequence, and motion recognition is performed on the above motion object joint sequence to obtain a recognition result of the motion of the moving object, wherein the above software service is used to provide a spatiotemporal graph convolutional network model, and the above spatiotemporal graph convolutional network model includes: a multi-layer spatiotemporal graph convolution calculation unit, and the above-mentioned multi-layer spatiotemporal graph convolution calculation unit is used to extract the spatiotemporal dynamic features of the above motion object action from the above motion object joint sequence, and the above spatiotemporal dynamic features are used to predict the above recognition result; the above recognition result is fed back to the above client.

[0107] It is easy to notice that the embodiment of the present application obtains and sends an action recognition request to the server by responding to the received control operation through the client, calling the software service of the server to analyze the obtained motion image of the moving object, determine the motion object joint sequence, and perform action recognition on the above motion object joint sequence to obtain the recognition result of the motion of the moving object, that is, the multi-layer spatiotemporal graph convolution calculation unit in the spatiotemporal graph convolution network model provided by the software service is used to extract the spatiotemporal dynamic features of the above motion object action from the above motion object joint sequence, and predict the above recognition result. Since the spatiotemporal dynamic features fuse static and dynamic information, and the motion of the moving object is recognized based on the spatiotemporal dynamic features, the purpose of paying attention to the relationship between directly connected joints and the dependency relationship between spaced and non-directly connected joints at the same time is achieved, thereby achieving the technical effect of improving the accuracy of identifying the motion of the moving object, and then solving the technical problem in the related technology that the dependency relationship between spaced and non-directly connected joints is ignored when identifying pedestrian actions, resulting in inaccurate recognition results.

[0108] Optionally, the above-mentioned motion object includes one of the following: a human motion object, a human-like motion object, an animal motion object, and a machine-simulated motion object.

[0109] In an optional embodiment, the motion image of the moving object may be a motion image of the moving object walking, jumping, running, waving, swinging arms, and the like.

[0110] It should be noted that the executing entity of the embodiment of the present application is the server. After the client responds to the received control operation and obtains the action recognition request, it sends the action recognition request to the server to request the server to identify the motion image of the motion object to be identified. Since the software service of the server provides a new type of graph convolutional neural network model, for example, a spatiotemporal graph convolutional network model, by analyzing the motion object joints in the above-mentioned motion object action image, a motion object joint sequence is obtained. The above-mentioned spatiotemporal graph convolutional network model includes a multi-layer spatiotemporal graph convolution calculation unit that extracts the spatiotemporal dynamic features of the motion of the motion object from the above-mentioned motion object joint sequence; then the above-mentioned motion object action is identified based on the above-mentioned spatiotemporal dynamic features, and the action category of the motion object action is predicted.

[0111] As an optional embodiment, the input of the spatio-temporal graph convolutional network model is a sequence of moving object joints composed of moving object joints, and the output is the predicted action category. The above-mentioned spatio-temporal graph convolutional network model is composed of multiple layers of spatio-temporal graph convolution calculation units (Spatio-Temporal Inception Block) connected front to back. The function of the spatio-temporal graph convolution calculation unit is to extract the spatio-temporal dynamic characteristics of pedestrian movements.

[0112] It should be noted that the graph convolutional neural network model proposed in the embodiments of this application is a basic neural network model for joint-based action recognition tasks. The network structure can be arbitrarily edited according to business needs, and the network depth and width can be increased or decreased. Under the conditions of the same network width and depth, the embodiments of this application can use 1 / 5 of the network parameters and 1 / 10 of the computational effort to surpass the best existing neural network action recognition algorithm.

[0113] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0114] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a non-volatile storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the above-mentioned methods of each embodiment of the present application.

[0115] Example 2

[0116] According to an embodiment of the present application, there is also provided an embodiment of a device for implementing the above-mentioned method for identifying the motion of a moving object. Figure 8 FIG. 1 is a schematic structural diagram of a moving object action recognition device according to an embodiment of the present application. Figure 8 As shown, the device includes: an acquisition module 60, an analysis module 62, an extraction module 64 and an identification module 66, wherein:

[0117] The acquisition module 60 is used to acquire the motion image of the moving object; the analysis module 62 is used to analyze the above motion image of the moving object to obtain a sequence of joint points of the moving object; the extraction module 64 is used to extract the spatiotemporal dynamic features of the motion of the moving object from the above sequence of joint points of the moving object; the recognition module 66 is used to recognize the above motion of the moving object based on the above spatiotemporal dynamic features.

[0118] It should be noted that the acquisition module 60, analysis module 62, extraction module 64, and identification module 66 described above correspond to steps S202 to S208 in Example 1. The examples and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0119] According to an embodiment of the present application, there is also provided an embodiment of a device for implementing the above-mentioned method for identifying the motion of a moving object. Figure 9 FIG. 1 is a schematic structural diagram of a moving object action recognition device according to an embodiment of the present application. Figure 9 As shown, the device includes: an acquisition unit 70, an analysis unit 72 and an identification unit 74, wherein:

[0120] An acquisition unit 70 is used to acquire a motion image of a moving object; an analysis unit 72 is used to analyze the motion image of the moving object to obtain a sequence of joint points of the moving object; an identification unit 74 is used to set the sequence of joint points of the moving object as an input parameter of a spatiotemporal graph convolutional network model, and output an identification result of the motion of the moving object, wherein the spatiotemporal graph convolutional network model includes: a multi-layer spatiotemporal graph convolution calculation unit, and the multi-layer spatiotemporal graph convolution calculation unit is used to extract the spatiotemporal dynamic features of the motion of the moving object from the sequence of joint points of the moving object, and the spatiotemporal dynamic features are used to predict the identification result.

[0121] It should be noted that the acquisition unit 70, analysis unit 72, and identification unit 74 described above correspond to steps S702 to S706 in Example 1. The examples and application scenarios implemented by the three units and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in the embodiment.

[0122] It should also be noted that the preferred implementation of this embodiment can be found in the relevant description in Example 1 and will not be repeated here.

[0123] Example 3

[0124] According to an embodiment of the present application, an embodiment of a moving object action recognition device is also provided. The moving object action recognition device can be any computing device in a computing device group. Figure 10 is a structural diagram of a moving object action recognition device according to an embodiment of the present application, such as Figure 10 As shown, the moving object action recognition device includes: a processor 800 and a memory 802, wherein:

[0125] Processor 800; and memory 802, connected to the above-mentioned processor 800, used to provide the above-mentioned processor with instructions for processing the following processing steps: obtaining a motion image of a moving object; analyzing the above-mentioned motion image of the moving object to obtain a sequence of joint points of the moving object; extracting the spatiotemporal dynamic features of the motion of the moving object from the above-mentioned sequence of joint points of the moving object; and identifying the motion of the moving object based on the above-mentioned spatiotemporal dynamic features.

[0126] In an embodiment of the present application, a motion image of a moving object is obtained; the motion image of the moving object is analyzed to obtain a sequence of joint points of the moving object; the spatiotemporal dynamic features of the motion of the moving object are extracted from the sequence of joint points of the moving object; and the motion of the moving object is identified based on the spatiotemporal dynamic features.

[0127] It is easy to notice that the embodiment of the present application determines the joint point sequence of the moving object by analyzing the acquired motion image of the moving object, and simultaneously extracts two features, namely, time and space features, between the joint points from the joint point sequence of the moving object, namely, the spatiotemporal dynamic features. Since the spatiotemporal dynamic features fuse static and dynamic information, the above-mentioned moving object movements are identified based on the spatiotemporal dynamic features, thereby achieving the purpose of paying attention to the relationship between directly connected joint points and the dependency relationship between spaced and non-directly connected joint points at the same time, thereby achieving the technical effect of improving the accuracy of identifying the movement of the moving object, and further solving the technical problem in the related technology of ignoring the dependency relationship between spaced and non-directly connected joint points when identifying pedestrian movements, resulting in inaccurate recognition results.

[0128] It should also be noted that the preferred implementation of this embodiment can be found in the relevant description in Example 1 and will not be repeated here.

[0129] Example 4

[0130] According to an embodiment of the present application, a computer terminal embodiment is also provided, which can be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal can also be replaced by a terminal device such as a mobile terminal.

[0131] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of a computer network.

[0132] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the motion object action recognition method: obtaining a motion object action image; analyzing the above-mentioned motion object action image to obtain a motion object joint point sequence; extracting the spatiotemporal dynamic characteristics of the motion object action from the above-mentioned motion object joint point sequence; and recognizing the above-mentioned motion object action based on the above-mentioned spatiotemporal dynamic characteristics.

[0133] Optionally, Figure 11 is a structural block diagram of a computer terminal according to an embodiment of the present application, such as Figure 11 As shown, the computer terminal may include: one or more (only one is shown in the figure) processors 902 , a memory 904 , and a peripheral interface 906 .

[0134] Wherein, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the motion object motion recognition method and device in the embodiment of the present application, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned motion object motion recognition method. The memory may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0135] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain the motion image of the moving object; analyze the above-mentioned motion image of the moving object to obtain a sequence of joint points of the moving object; extract the spatiotemporal dynamic characteristics of the motion of the moving object from the above-mentioned sequence of joint points of the moving object; and identify the above-mentioned motion of the moving object based on the above-mentioned spatiotemporal dynamic characteristics.

[0136] Optionally, the processor may also execute the program code of the following steps: inputting the joint point sequence of the moving object into the spatiotemporal graph convolutional network model, wherein the spatiotemporal graph convolutional network model includes: a multi-layer spatiotemporal graph convolution calculation unit; and extracting the spatiotemporal dynamic features using the multi-layer spatiotemporal graph convolution calculation unit.

[0137] Optionally, the processor may also execute the program code of the following steps: on the spatial branch, extracting the spatial features between the multiple joint points of the motion of the motion object from the joint point sequence of the motion object; on the temporal branch, extracting the temporal features between the multiple joint points from the joint point sequence of the motion object; on the residual branch, transmitting the original input information of each layer of the spatiotemporal graph convolution calculation unit; and obtaining the spatiotemporal dynamic features by adding the spatial features, the temporal features and the original input information.

[0138] Optionally, the processor may also execute the program code of the following steps: using the multiple sampling units to extract multiple scale connection features corresponding to the multiple joint points, wherein each of the multiple sampling units samples different scale connection features; using the convolution units corresponding to each sampling unit in the multiple convolution units to perform feature processing on the multiple scale connection features to obtain multiple processing results; and using the fusion unit to fuse the multiple processing results to obtain the spatial features.

[0139] Optionally, the processor may also execute the program code of the following steps: using the at least one sampling unit to extract the second-order time series features corresponding to the multiple joint points; using at least one of the multiple convolution units to perform feature processing on the second-order time series features to obtain multiple processing results; and using the fusion unit to perform fusion processing on the multiple processing results to obtain the time features.

[0140] Optionally, the processor may also execute the program code of the following steps: obtaining a motion image of a moving object; analyzing the motion image of the moving object to obtain a sequence of joint points of the moving object; setting the sequence of joint points of the moving object as an input parameter of a spatiotemporal graph convolutional network model, and outputting the recognition result of the motion of the moving object, wherein the spatiotemporal graph convolutional network model includes: a multi-layer spatiotemporal graph convolution calculation unit, the multi-layer spatiotemporal graph convolution calculation unit is used to extract the spatiotemporal dynamic features of the motion of the moving object from the sequence of joint points of the moving object, and the spatiotemporal dynamic features are used to predict the recognition result.

[0141] An embodiment of the present application provides a solution for motion object action recognition, which comprises obtaining a motion object action image; analyzing the motion object action image to obtain a motion object joint point sequence; extracting the spatiotemporal dynamic features of the motion object action from the motion object joint point sequence; and recognizing the motion object action based on the spatiotemporal dynamic features.

[0142] It is easy to notice that the embodiment of the present application determines the joint point sequence of the moving object by analyzing the acquired motion image of the moving object, and simultaneously extracts two features, namely, time and space features, between the joint points from the joint point sequence of the moving object, namely, the spatiotemporal dynamic features. Since the spatiotemporal dynamic features fuse static and dynamic information, the above-mentioned moving object movements are identified based on the spatiotemporal dynamic features, thereby achieving the purpose of paying attention to the relationship between directly connected joint points and the dependency relationship between spaced and non-directly connected joint points at the same time, thereby achieving the technical effect of improving the accuracy of identifying the movement of the moving object, and further solving the technical problem in the related technology of ignoring the dependency relationship between spaced and non-directly connected joint points when identifying pedestrian movements, resulting in inaccurate recognition results.

[0143] It can be understood by those skilled in the art that Figure 11 The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (MID), a PAD, or other terminal devices. Figure 11 It does not limit the structure of the above electronic device. For example, the computer terminal may also include Figure 11More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 11 Different configurations shown.

[0144] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable non-volatile storage medium. The non-volatile storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0145] Example 5

[0146] According to an embodiment of the present application, an embodiment of a non-volatile storage medium is also provided. Optionally, in this embodiment, the non-volatile storage medium can be used to store the program code executed by the moving object action recognition method provided in the above embodiment 1.

[0147] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0148] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: obtaining a motion image of a moving object; analyzing the above-mentioned motion image of the moving object to obtain a sequence of joint points of the moving object; extracting the spatiotemporal dynamic features of the motion of the moving object from the above-mentioned sequence of joint points of the moving object; and identifying the above-mentioned motion of the moving object based on the above-mentioned spatiotemporal dynamic features.

[0149] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: inputting the above-mentioned motion object joint point sequence into the spatiotemporal graph convolutional network model, wherein the above-mentioned spatiotemporal graph convolutional network model includes: a multi-layer spatiotemporal graph convolution calculation unit; and extracting the above-mentioned spatiotemporal dynamic features using the above-mentioned multi-layer spatiotemporal graph convolution calculation unit.

[0150] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: on the above-mentioned spatial branch, extracting spatial features between multiple joint points of the above-mentioned moving object action from the above-mentioned moving object joint point sequence; on the above-mentioned time branch, extracting time features between the above-mentioned multiple joint points from the above-mentioned moving object joint point sequence; on the above-mentioned residual branch, transmitting the original input information of each layer of the spatiotemporal graph convolution calculation unit; and obtaining the above-mentioned spatiotemporal dynamic features by adding the above-mentioned spatial features, the above-mentioned time features and the above-mentioned original input information.

[0151] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: using the above-mentioned multiple sampling units to extract multiple scale connection features corresponding to the above-mentioned multiple joint points, wherein each of the above-mentioned multiple sampling units samples different scale connection features respectively; using the convolution units corresponding to each sampling unit in the above-mentioned multiple convolution units to perform feature processing on the above-mentioned multiple scale connection features to obtain multiple processing results; using the above-mentioned fusion unit to fuse the above-mentioned multiple processing results to obtain the above-mentioned spatial features.

[0152] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: using the above-mentioned at least one sampling unit to extract second-order time series features corresponding to the above-mentioned multiple joint points; using at least one convolution unit among the above-mentioned multiple convolution units to perform feature processing on the above-mentioned second-order time series features to obtain multiple processing results; using the above-mentioned fusion unit to fuse the above-mentioned multiple processing results to obtain the above-mentioned time features.

[0153] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: obtaining a motion image of a moving object; analyzing the above-mentioned motion image of the moving object to obtain a motion object joint point sequence; setting the above-mentioned motion object joint point sequence as an input parameter of a spatiotemporal graph convolutional network model, and outputting the recognition result of the motion of the moving object, wherein the above-mentioned spatiotemporal graph convolutional network model includes: a multi-layer spatiotemporal graph convolution calculation unit, and the above-mentioned multi-layer spatiotemporal graph convolution calculation unit is used to extract the spatiotemporal dynamic features of the above-mentioned motion of the moving object from the above-mentioned motion object joint point sequence, and the above-mentioned spatiotemporal dynamic features are used to predict the above-mentioned recognition results.

[0154] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0155] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0156] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0157] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0159] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable non-volatile storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a non-volatile storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the above-mentioned methods of each embodiment of the present application. The aforementioned non-volatile storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0160] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for motion recognition of a moving object, characterized in that: include: Acquiring a motion image of a moving object, wherein the moving object includes one of the following: a human moving object, a human-like moving object, an animal moving object, or a machine-simulated moving object; Analyzing the motion image of the moving object to obtain a joint point sequence of the moving object; Utilizing a spatiotemporal graph convolutional network model, extracting spatiotemporal dynamic features of a motion of a motion object from a sequence of joint points of the motion object, wherein the spatiotemporal dynamic features include spatial features, the spatial features are determined based on a plurality of scale connection features corresponding to a plurality of joint points in the sequence of joint points of the motion object, the plurality of scale connection features being used to describe connection features of direct and indirect connections between the plurality of joint points, the plurality of scale connection features being obtained by sampling a plurality of sampling units, each of the plurality of sampling units respectively sampling a relational feature corresponding to multiple connections between the plurality of joint points as the plurality of scale connection features, and the plurality of sampling units being included in a spatial branch of the spatiotemporal graph convolutional network model; The motion of the moving object is identified based on the spatiotemporal dynamic features.

2. The method according to claim 1, characterized in that Extracting the spatiotemporal dynamic features from the joint point sequence of the moving object using the spatiotemporal graph convolutional network model includes: Inputting the joint point sequence of the moving object into the spatiotemporal graph convolutional network model, wherein the spatiotemporal graph convolutional network model comprises: a multi-layer spatiotemporal graph convolution calculation unit; The multi-layer spatiotemporal graph convolution calculation unit is used to extract the spatiotemporal dynamic features.

3. The method according to claim 2, characterized in that Each layer of the multi-layer spatiotemporal graph convolution calculation unit includes: a spatial branch, a temporal branch, and a residual branch. Extracting the spatiotemporal dynamic features using the multi-layer spatiotemporal graph convolution calculation unit includes: On the spatial branch, extracting spatial features between multiple joint points of the motion of the motion object from the joint point sequence of the motion object; On the time branch, extracting time features between the plurality of joint points from the joint point sequence of the moving object; On the residual branch, the original input information of each layer of spatiotemporal graph convolution calculation unit is transmitted; The spatiotemporal dynamic feature is obtained by adding the spatial feature, the temporal feature and the original input information.

4. The method according to claim 3, characterized in that The spatial branch includes: a plurality of sampling units, a plurality of convolution units and a fusion unit. In the spatial branch, extracting the spatial features from the joint point sequence of the moving object includes: Extracting a plurality of scale connection features corresponding to the plurality of joint points using the plurality of sampling units, wherein each of the plurality of sampling units samples a different scale connection feature; Using the convolution units corresponding to each sampling unit in the plurality of convolution units to perform feature processing on the plurality of scale connection features to obtain a plurality of processing results; The fusion unit is used to perform fusion processing on the multiple processing results to obtain the spatial feature.

5. The method according to claim 4, characterized in that The multiplicities of the plurality of scale connection features are determined by the number of the plurality of joint points connected in sequence, and the number of the plurality of sampling units is the same as the multiplicities of the plurality of scale connection features.

6. The method according to claim 3, characterized in that The time branch includes: at least one sampling unit, multiple convolution units and a fusion unit. In the time branch, extracting the time feature from the joint point sequence of the moving object includes: extracting second-order time series features corresponding to the plurality of joint points using the at least one sampling unit; Using at least one convolution unit among the multiple convolution units to perform feature processing on the second-order time series features to obtain multiple processing results; The fusion unit is used to perform fusion processing on the multiple processing results to obtain the time feature.

7. A method for identifying the motion of a moving object, characterized in that: include: Acquiring a motion image of a moving object, wherein the moving object includes one of the following: a human moving object, a human-like moving object, an animal moving object, or a machine-simulated moving object; Analyzing the motion image of the moving object to obtain a joint point sequence of the moving object; The motion object joint point sequence is set as the input parameter of the spatiotemporal graph convolutional network model, and the recognition result of the motion of the motion object is output, wherein the spatiotemporal graph convolutional network model includes: a multi-layer spatiotemporal graph convolution calculation unit, the multi-layer spatiotemporal graph convolution calculation unit is used to extract the spatiotemporal dynamic features of the motion of the motion object from the motion object joint point sequence, the spatiotemporal dynamic features are used to predict the recognition result, the spatiotemporal dynamic features include spatial features, the spatial features are determined based on multiple scale connection features corresponding to multiple joint points in the motion object joint point sequence, the multiple scale connection features are used to describe the connection features of direct connections and indirect connections between the multiple joint points, the multiple scale connection features are obtained by sampling by multiple sampling units, each of the multiple sampling units respectively samples the relationship features corresponding to the multiple connections between the multiple joint points as the multiple scale connection features, and the multiple sampling units are included in the spatial branch of the spatiotemporal graph convolutional network model.

8. A method for identifying the motion of a moving object, characterized in that: include: In response to the control operation received by the client, obtaining a motion recognition request, wherein the information carried in the motion recognition request includes: a motion image of a motion object to be recognized, wherein the motion object includes one of the following: a human motion object, a human-like motion object, an animal motion object, and a machine-simulated motion object; Based on the action recognition request, a software service of a server is called on the client to analyze the motion image of the moving object to obtain a sequence of joint points of the moving object, and action recognition is performed on the sequence of joint points of the moving object to obtain a recognition result of the motion of the moving object. The software service is used to provide a spatiotemporal graph convolutional network model, and the spatiotemporal graph convolutional network model includes: a multi-layer spatiotemporal graph convolution calculation unit, and the multi-layer spatiotemporal graph convolution calculation unit is used to extract the spatiotemporal dynamic features of the motion of the moving object from the sequence of joint points of the moving object. The spatiotemporal dynamic features are used to predict the recognition result. The spatiotemporal dynamic features include spatial features, which are determined based on multiple scale connection features corresponding to multiple joint points in the sequence of joint points of the moving object. The multiple scale connection features are used to describe the connection features of direct connections and indirect connections between the multiple joint points. The multiple scale connection features are obtained by sampling by multiple sampling units, and each of the multiple sampling units respectively samples the relationship features corresponding to the multiple connections between the multiple joint points as the multiple scale connection features. The multiple sampling units are included in the spatial branches of the spatiotemporal graph convolutional network model.

9. A method for identifying the motion of a moving object, characterized in that: include: Receiving a motion recognition request from a client, wherein the information carried in the motion recognition request includes: a motion image of a motion object to be recognized, wherein the motion object includes one of the following: a human motion object, a human-like motion object, an animal motion object, and a machine-simulated motion object; Based on the action recognition request, a software service on the server is called to analyze the motion image of the moving object to obtain a sequence of joint points of the moving object, and action recognition is performed on the sequence of joint points of the moving object to obtain a recognition result of the motion of the moving object, wherein the software service is used to provide a spatiotemporal graph convolutional network model, and the spatiotemporal graph convolutional network model includes: a multi-layer spatiotemporal graph convolution calculation unit, the multi-layer spatiotemporal graph convolution calculation unit is used to extract the spatiotemporal dynamic features of the motion of the moving object from the sequence of joint points of the moving object, the spatiotemporal dynamic features are used to predict the recognition result, the spatiotemporal dynamic features include spatial features, the spatial features are determined based on multiple scale connection features corresponding to multiple joint points in the sequence of joint points of the moving object, the multiple scale connection features are used to describe the connection features of direct connections and indirect connections between the multiple joint points, the multiple scale connection features are obtained by sampling by multiple sampling units, each of the multiple sampling units respectively samples the relationship features corresponding to the multiple connections between the multiple joint points as the multiple scale connection features, and the multiple sampling units are included in the spatial branches of the spatiotemporal graph convolutional network model; The recognition result is fed back to the client.

10. A moving object action recognition device, characterized in that: include: An acquisition module is used to acquire a motion image of a moving object, wherein the moving object includes one of the following: a human moving object, a human-like moving object, an animal moving object, and a machine-simulated moving object; An analysis module, configured to analyze the motion image of the moving object to obtain a joint point sequence of the moving object; an extraction module, configured to extract spatiotemporal dynamic features of a motion of a moving object from a sequence of joint points of the moving object using a spatiotemporal graph convolutional network model, wherein the spatiotemporal dynamic features include spatial features, the spatial features being determined based on a plurality of scale connection features corresponding to a plurality of joint points in the sequence of joint points of the moving object, the plurality of scale connection features being used to describe connection features of direct and indirect connections between the plurality of joint points, the plurality of scale connection features being obtained by sampling a plurality of sampling units, each of the plurality of sampling units respectively sampling a relational feature corresponding to multiple connections between the plurality of joint points as the plurality of scale connection features, and the plurality of sampling units being included in a spatial branch of the spatiotemporal graph convolutional network model; The recognition module is used to recognize the motion of the moving object based on the spatiotemporal dynamic features.

11. A moving object action recognition device, characterized in that: include: An acquisition unit, configured to acquire a motion image of a moving object, wherein the moving object comprises one of the following: a human moving object, a human-like moving object, an animal moving object, or a machine-simulated moving object; An analysis unit, configured to analyze the motion image of the moving object to obtain a joint point sequence of the moving object; An identification unit is used to set the motion object joint point sequence as an input parameter of a spatiotemporal graph convolutional network model and output an identification result of the motion of the motion object, wherein the spatiotemporal graph convolutional network model includes: a multi-layer spatiotemporal graph convolution calculation unit, the multi-layer spatiotemporal graph convolution calculation unit is used to extract the spatiotemporal dynamic features of the motion of the motion object from the motion object joint point sequence, the spatiotemporal dynamic features are used to predict the identification result, the spatiotemporal dynamic features include spatial features, the spatial features are determined based on multiple scale connection features corresponding to multiple joint points in the motion object joint point sequence, the multiple scale connection features are used to describe the connection features of direct and indirect connections between the multiple joint points, the multiple scale connection features are obtained by sampling by multiple sampling units, each of the multiple sampling units respectively samples the relationship features corresponding to the multiple connections between the multiple joint points as the multiple scale connection features, and the multiple sampling units are included in the spatial branch of the spatiotemporal graph convolutional network model.

12. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the moving object action recognition method according to any one of claims 1 to 9.

13. A moving object action recognition device, characterized in that: include: processor; as well as A memory, connected to the processor, configured to provide the processor with instructions for processing the following processing steps: Acquiring a motion image of a moving object, wherein the moving object includes one of the following: a human moving object, a human-like moving object, an animal moving object, or a machine-simulated moving object; Analyzing the motion image of the moving object to obtain a joint point sequence of the moving object; Utilizing a spatiotemporal graph convolutional network model, extracting spatiotemporal dynamic features of a motion of a motion object from a sequence of joint points of the motion object, wherein the spatiotemporal dynamic features include spatial features, the spatial features are determined based on a plurality of scale connection features corresponding to a plurality of joint points in the sequence of joint points of the motion object, the plurality of scale connection features being used to describe connection features of direct and indirect connections between the plurality of joint points, the plurality of scale connection features being obtained by sampling a plurality of sampling units, each of the plurality of sampling units respectively sampling a relational feature corresponding to multiple connections between the plurality of joint points as the plurality of scale connection features, and the plurality of sampling units being included in a spatial branch of the spatiotemporal graph convolutional network model; The motion of the moving object is identified based on the spatiotemporal dynamic features.

Images