Space-time action recognition method based on artificial intelligence and related device

Abstract

The application provides a spatiotemporal action recognition method and device based on artificial intelligence, electronic equipment and storage medium. The spatiotemporal action recognition method based on artificial intelligence comprises the following steps: a first recognition network is built, the first recognition network comprises a target detection module, an interaction feature extraction module and an action recognition module; the first recognition network is trained to obtain a second recognition network, the input of the second recognition network is a video segment, and the output is position information and an action category of each recognition object in a to-be-recognized video frame of the video segment; under different playing speeds, video frames in a preset time period before and after the to-be-recognized video frame are selected to form a plurality of video segments; the plurality of video segments are sequentially input into the second recognition network, and initial recognition results of the video segments are output; and all the initial recognition results are fused to obtain a spatiotemporal action recognition result of the to-be-recognized video frame. The application can improve the accuracy of spatiotemporal action recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a spatio-temporal action recognition method based on artificial intelligence and related equipment. BACKGROUND

[0002] In the field of finance, banking or medical rehabilitation, it is necessary to perform spatio-temporal action recognition on each person in the video to determine whether there is abnormal behavior, so as to ensure the safety of bank customers or patients. In the spatio-temporal action recognition task, not only the interval at which the action appears in the video and the corresponding category need to be recognized, but also the spatial position of the person needs to be marked with a bounding box in the spatial range.

[0003] At present, it is often necessary to detect the person in the video frame, and then directly obtain the action recognition result by taking the video frames in a fixed time interval as the context information of the person. However, this method ignores the phenomenon that the time intervals of different actions are very different, and does not consider the interaction information between the person and the environment, other people and objects, resulting in a large error in the spatio-temporal action recognition result. SUMMARY

[0004] In view of the above, it is necessary to propose a spatio-temporal action recognition method based on artificial intelligence and related equipment to solve the technical problem of how to improve the accuracy of the spatio-temporal action recognition result, wherein the related equipment includes a spatio-temporal action recognition device based on artificial intelligence, an electronic device and a storage medium.

[0005] The present application provides a spatio-temporal action recognition method based on artificial intelligence, which comprises:

[0006] A first recognition network is built, which comprises a target detection module, an interaction feature extraction module and an action recognition module. The target detection module is used to locate the position information of all recognition objects in the video frame to be recognized. The interaction feature extraction module is used to extract the interaction information between each recognition object and the environment and other recognition objects in the video frame to be recognized. The action recognition module is used to recognize the action category of all recognition objects in the video frame to be recognized.

[0007] The first recognition network is trained to obtain a second recognition network. The input of the second recognition network is a video segment, and the output is the position information and the action category of each recognition object in the video frame to be recognized corresponding to the video segment. The video frame to be recognized is the video frame at the middle position of the video segment.

[0008] Video frames in a preset time period before and after any video frame to be recognized in a video to be recognized are selected to form a video segment, and the playback speed of the video to be recognized is adjusted to obtain a plurality of video segments corresponding to the video frame to be recognized.

[0009] inputting the plurality of video clips into the second recognition network in sequence, and outputting an initial recognition result of each video clip, the initial recognition result comprising initial position information and an initial action category of each recognition object in the to-be-recognized video frame;

[0010] fusing the initial recognition results of different video clips to obtain a spatio-temporal action recognition result of the to-be-recognized video frame, the spatio-temporal action recognition result comprising optimized position information and an optimized action category of each recognition object in the to-be-recognized video frame.

[0011] In some embodiments, the first recognition network comprises:

[0012] The target detection module takes the to-be-recognized video frame as input and outputs position information of all recognition objects in the to-be-recognized video frame, the position information comprising a center point coordinate and a width-height size of a rectangular frame surrounding the recognition object;

[0013] The to-be-recognized video frame is cropped based on the rectangular frame of the recognition object to obtain a cropped image of each recognition object;

[0014] The interaction feature extraction module takes the video clip and the cropped image of each recognition object as input, and comprises an environment feature extraction layer and an interaction feature extraction layer;

[0015] The environment feature extraction layer is configured to extract static features and dynamic features of the video clip and concatenate the static features and the dynamic features as environment features;

[0016] In the interaction feature extraction layer, a convolutional neural network is used to perform multiple convolution operations on the cropped image of each recognition object to obtain a pose feature of each recognition object, and the pose feature and the environment features are concatenated to obtain a first interaction feature of each recognition object, the first interaction feature being used to represent interaction information between the recognition object and the environment;

[0017] The first interaction features of different recognition objects are processed based on a self-attention mechanism to construct a second interaction feature of each recognition object, the second interaction feature being used to represent interaction information between the recognition object and other recognition objects;

[0018] The first interaction feature and the second interaction feature of the same recognition object are added to obtain a fused interaction feature of each recognition object, and the fused interaction feature is input into the action recognition module to obtain an action category of all recognition objects in the to-be-recognized video frame;

[0019] The position information and the action category of all recognition objects in the to-be-recognized video frame are taken as output results of the first recognition network.

[0020] In some embodiments, before the splicing the pose features and the environment features to obtain the first interaction feature of each identified object, the method further comprises:

[0021] performing an ROI Align operation on the pose features of each identified object, the ROI Align operation being configured to process pose features of different sizes into a preset size.

[0022] In some embodiments, the processing the first interaction features of different identified objects based on the self-attention mechanism to construct the second interaction feature of each identified object comprises:

[0023] arranging the first interaction features of all identified objects in a preset order along a column direction to obtain a first interaction matrix, each row in the first interaction matrix corresponding to the first interaction feature of an identified object;

[0024] multiplying the first interaction matrix with a parameterized matrix to obtain a Q matrix, a K matrix and a V matrix, the Q matrix, the K matrix and the V matrix respectively comprising Q vectors, K vectors and V vectors of all identified objects;

[0025] processing the Q matrix, the K matrix and the V matrix based on the self-attention mechanism to construct a second interaction matrix, each row in the second interaction matrix corresponding to the second interaction feature of an identified object, the second interaction matrix satisfying the relationship:

[0026]

[0027] wherein Q, K and V represent the Q matrix, the K matrix and the V matrix respectively, K T represents the transpose of the K matrix, d is the number of all identified objects, and F2 represents the second interaction matrix.

[0028] extracting the second interaction feature of each identified object from the second interaction matrix.

[0029] In some embodiments, the training the first identification network to obtain a second identification network comprises:

[0030] collecting a plurality of video clips with annotation information to construct a training set, the annotation information comprising real position information and real action categories of all identified objects in a to-be-detected video frame corresponding to the video clip;

[0031] selecting a video clip from the training set and inputting the first identification network to obtain an identification result, the identification result comprising predicted position information and predicted action categories of all identified objects in a to-be-detected video frame corresponding to the video clip;

[0032] calculate a cost function based on the recognition result and the annotation information, the cost function satisfying a relationship formula:

[0033]

[0034] wherein B is a number of recognized objects in a video frame to be detected, Z i and Y i are predicted position information and a predicted action category of a recognized object i in the video frame to be detected, and are real position information and a real action category of the recognized object i in the video frame to be detected, denotes a Euclidean distance between Z i and , and denotes a cross-entropy loss between Y i and , and Loss is a value of the cost function;

[0035] updating the trainable parameters of the first recognition network based on a back propagation algorithm;

[0036] continuously selecting video clips from the training set to update the trainable parameters of the first recognition network until the value of the cost function is less than a preset value, and obtaining a second recognition network.

[0037] In some embodiments, the selecting video frames within a preset time period before and after the arbitrary video frame to be recognized from the video to be recognized to form a video clip comprises:

[0038] selecting video frames within a preset time period before the video frame to be recognized as a first video frame sequence;

[0039] selecting video frames within a preset time period after the video frame to be recognized as a second video frame sequence;

[0040] comparing the number of video frames in the first video frame sequence and the second video frame sequence;

[0041] if the number of video frames in the first video frame sequence is greater than the second video frame sequence, supplementing blank images at the end of the second video frame sequence to make the number of video frames in the first video frame sequence equal to the second video frame sequence;

[0042] if the number of video frames in the first video frame sequence is less than the second video frame sequence, supplementing blank images at the beginning of the first video frame sequence to make the number of video frames in the first video frame sequence equal to the second video frame sequence;

[0043] when the number of video frames in the first video frame sequence is equal to the second video frame sequence, sequentially arranging the first video frame sequence, the to-be-identified video frame and the second video frame sequence together to form a video clip.

[0044] In some embodiments, the fusing the initial recognition results of different video clips to obtain the spatio-temporal action recognition result of the to-be-identified video frame comprises:

[0045] extracting initial action categories of each recognition object in the to-be-identified video frame from all the initial recognition results;

[0046] from all the initial action categories of the same recognition object, selecting an initial action category with the highest occurrence frequency as an optimized action category of the recognition object;

[0047] extracting initial position information of each recognition object in the to-be-identified video frame from all the initial recognition results, and calculating the mean value of all the initial position information of the same recognition object to obtain optimized position information of each recognition object;

[0048] taking the optimized position information and the optimized action category of each recognition object in the to-be-identified video frame as the spatio-temporal action recognition result.

[0049] Embodiments of the present application also provide a spatio-temporal action recognition device based on artificial intelligence, which comprises:

[0050] a building unit configured to build a first recognition network, wherein the first recognition network comprises a target detection module, an interactive feature extraction module and an action recognition module, the target detection module is configured to locate position information of all recognition objects in a to-be-identified video frame, the interactive feature extraction module is configured to extract interactive information between each recognition object and the environment and other recognition objects in the to-be-identified video frame, and the action recognition module is configured to recognize action categories of all recognition objects in the to-be-identified video frame;

[0051] a training unit configured to train the first recognition network to obtain a second recognition network, wherein the input of the second recognition network is a video clip, and the output is position information and action categories of each recognition object in a to-be-identified video frame corresponding to the video clip, and the to-be-identified video frame is a video frame at a middle position of the video clip;

[0052] a selection unit configured to select video frames within a preset time period before and after any to-be-identified video frame in a to-be-identified video to form a video clip, and adjust the playing speed of the to-be-identified video to obtain a plurality of video clips corresponding to the to-be-identified video frame;

[0053] an output unit, configured to input the plurality of video clips into the second recognition network in sequence, and output an initial recognition result of each video clip, the initial recognition result comprising initial position information and an initial action category of each recognition object in the to-be-recognized video frame;

[0054] a fusion unit, configured to fuse the initial recognition results of different video clips to obtain a spatio-temporal action recognition result of the to-be-recognized video frame, the spatio-temporal action recognition result comprising optimized position information and an optimized action category of each recognition object in the to-be-recognized video frame.

[0055] The embodiment of the present application also provides an electronic device, which comprises:

[0056] a memory, configured to store at least one instruction;

[0057] a processor, configured to execute the instruction stored in the memory to implement the spatio-temporal action recognition method based on artificial intelligence.

[0058] The embodiment of the present application also provides a computer readable storage medium, which stores at least one instruction, and the at least one instruction is executed by a processor in an electronic device to implement the spatio-temporal action recognition method based on artificial intelligence.

[0059] To sum up, the present application adjusts the playing speed of a to-be-recognized video to collect video clips comprising different context information, and one to-be-recognized video frame corresponds to a plurality of video clips; each video clip is input into a second recognition network to extract interaction information between each recognition object in the to-be-recognized video frame and the environment and other recognition objects, and an initial recognition result is obtained based on the interaction information; finally, the initial recognition results of different video clips are fused to obtain a spatio-temporal action recognition result, thereby improving the accuracy of the spatio-temporal action recognition result. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 is a flowchart of a preferred embodiment of the spatio-temporal action recognition method based on artificial intelligence involved in the present application.

[0061] Figure 2 is a structural schematic diagram of the first recognition network involved in the present application.

[0062] Figure 3 is a function module diagram of a preferred embodiment of the spatio-temporal action recognition apparatus involved in the present application.

[0063] Figure 4 is a structural schematic diagram of an electronic device of a preferred embodiment of the spatio-temporal action recognition method based on artificial intelligence involved in the present application. DETAILED DESCRIPTION

[0064] In order to more clearly understand the purpose, features and advantages of the present application, the present application is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict. In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application, and the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments.

[0065] In addition, the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly and specifically limited.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing the specific embodiments of the present application, and are not intended to limit the present application. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0067] The embodiments of the present application provide a spatio-temporal action recognition method based on artificial intelligence, which can be applied to one or more electronic devices. The electronic device is a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions. The hardware thereof includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0068] The electronic device can be any electronic product that can interact with the customer, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive Internet Protocol Television (IPTV), a smart wearable device, etc.

[0069] The electronic device can also include a network device and / or a client device. The network device includes, but is not limited to, a single network server, a server group composed of multiple network servers, or a cloud composed of a large number of hosts or network servers based on cloud computing.

[0070] The network in which the electronic device is located includes, but is not limited to, the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), etc.

[0071] As shown in Figure 1 , it is a flow chart of a preferred embodiment of the spatio-temporal action recognition method based on artificial intelligence according to the present application. The order of the steps in the flow chart can be changed according to different needs, and some steps can be omitted. The spatio-temporal action recognition method based on artificial intelligence provided by the embodiments of the present application can be applied to any scene that needs spatio-temporal action recognition, and the method can be applied to products in these scenes, such as abnormal behavior recognition in the fields of finance, banking, or medical rehabilitation, etc.

[0072] S10, a first recognition network is built, the first recognition network includes a target detection module, an interaction feature extraction module, and an action recognition module, the target detection module is used to locate the position information of all recognition objects in a to-be-recognized video frame, the interaction feature extraction module is used to extract the interaction information between each recognition object and the environment and other recognition objects in the to-be-recognized video frame, and the action recognition module is used to recognize the action category of all recognition objects in the to-be-recognized video frame.

[0073] In an optional embodiment, the input of the first recognition network is any video segment, and the expected output is the position information and the action category of each recognition object in the to-be-recognized video frame corresponding to the video segment. The to-be-recognized video frame is a video frame at the middle position of the video segment, the recognition object is the subject of the action in the video segment, which can be a person, a robot, an animal, etc., and the action category and the recognition object are related to a specific application scenario. For example, in the field of medical rehabilitation, the recognition object is a patient, and the action category is different rehabilitation actions.

[0074] Please refer to Figure 2 , Figure 2 is a structural schematic diagram of the first recognition network involved in the present application. In an optional embodiment, the first recognition network includes:

[0075] The input of the target detection module 10 is the to-be-recognized video frame, and the output is the position information of all recognition objects in the to-be-recognized video frame, the position information including the center point coordinates and the width and height dimensions of the rectangular frame surrounding the recognition object.

[0076] cropping the to-be-identified video frame based on the rectangular box of the identified object to obtain a cropped image of each identified object;

[0077] The input of the interaction feature extraction module 20 is the video segment and the cropped image of each identified object, and the interaction feature extraction module 20 comprises an environment feature extraction layer 201 and an interaction feature extraction layer 202.

[0078] The environment feature extraction layer 201 is configured to extract static features and dynamic features of the video segment, and splice the static features and the dynamic features together as environment features.

[0079] In the interaction feature extraction layer 202, the cropped image of each identified object is subjected to multiple convolution operations by a convolutional neural network to obtain a posture feature of each identified object, and the posture feature and the environment feature are spliced to obtain a first interaction feature of each identified object, which is used to represent the interaction information between the identified object and the environment.

[0080] The first interaction features of different identified objects are processed based on a self-attention mechanism to construct a second interaction feature of each identified object, which is used to represent the interaction information between the identified object and other identified objects.

[0081] The first interaction feature and the second interaction feature of the same identified object are added to obtain a fusion interaction feature of each identified object, and the fusion interaction feature is input into the action recognition module 30 to obtain the action category of all identified objects in the to-be-identified video frame.

[0082] The position information and the action category of all identified objects in the to-be-identified video frame are taken as the output result of the first identification network.

[0083] The target detection module 10 is any one of existing target detection networks such as YOLO, CenterNet, Faster RCNN, etc.

[0084] The environment feature extraction layer 201 is a fast-slow dual-channel network SlowFast. SlowFast can apply two parallel convolutional neural networks to process the same video segment, one is a Slow branch for extracting static features in the video segment, and the other is a Fast branch for extracting dynamic features in the video, and the feature vectors output by the two branches are spliced together as the environment features of the video segment.

[0085] The interaction feature extraction layer comprises a convolutional neural network and a self-attention layer, the convolutional neural network is a ResNet, a Densenet or other existing neural network, and the self-attention layer is an attention mechanism in a Transformer.

[0086] The action recognition module 30 is a fully connected network comprising a classifier, and the classifier is a softmax function.

[0087] In an optional embodiment, before the splicing of the posture feature and the environment feature to obtain the first interaction feature of each identified object, the method further comprises:

[0088] An ROI Align operation is performed on the posture feature of each identified object, and the ROI Align operation is used to process posture features of different sizes into a preset size.

[0089] It should be noted that in the video frame to be detected, due to different heights and distances of different identified objects from the video acquisition device, the sizes of the cropped images of different identified objects are different, and thus the sizes of the posture features corresponding to different identified objects are different. The ROI Align operation can process posture features of different sizes into the same preset size. In this way, the sizes of the first interaction features of each identified object can be the same, and thus errors can be avoided in the process of constructing the second interaction feature.

[0090] In an optional embodiment, the processing of the first interaction features of different identified objects based on the self-attention mechanism to construct the second interaction feature of each identified object comprises:

[0091] Arranging the first interaction features of all identified objects in a preset order along the column direction to obtain a first interaction matrix, each row in the first interaction matrix corresponding to the first interaction feature of an identified object;

[0092] Multiplying the first interaction matrix by a parameterized matrix to obtain a Q matrix, a K matrix and a V matrix, the Q matrix, the K matrix and the V matrix comprising Q vectors, K vectors and V vectors of all identified objects, respectively;

[0093] Processing the Q matrix, the K matrix and the V matrix based on the self-attention mechanism to construct a second interaction matrix, each row in the second interaction matrix corresponding to the second interaction feature of an identified object, and the second interaction matrix satisfying the relationship:

[0094]

[0095] wherein Q, K and V represent the Q matrix, the K matrix and the V matrix, respectively, and K Tdenotes the transpose of the K matrix, d is the number of all identified objects, F2 is the second interaction matrix;

[0096] The second interaction feature of each identified object is extracted from the second interaction matrix.

[0097] In the optional embodiment, the size of the first interaction feature is 1 row and N columns, wherein the value of N is related to the network structure of the first identification network, and the number of identified objects in the video frame to be identified is denoted as M, and the size of the first interaction matrix is M rows and N columns.

[0098] In the optional embodiment, the Q matrix, the K matrix and the V matrix satisfy the relationship:

[0099]

[0100]

[0101]

[0102] wherein, are the parameterized matrices corresponding to the Q matrix, the K matrix and the V matrix respectively, and the size of the parameterized matrix is N rows and P columns, P is pre-set, and the values in the parameterized matrix are trainable parameters, and the specific values are determined by the training process of the first identification network; the size of the Q matrix, the K matrix and the V matrix is M rows and P columns, that is, the size of the Q vector, the K vector and the V vector of the identified object is 1 row and P columns.

[0103] It should be noted that in the self-attention mechanism, the Q vector, the K vector and the V vector can be regarded as split vectors of the first interaction feature, and each first interaction feature is split into a Q vector, a K vector and a V vector.

[0104] The second interaction matrix is calculated based on the Q matrix, the K matrix and the V matrix, so the size information of the second interaction matrix is M rows and P columns, and each row of the second interaction matrix corresponds to the second interaction feature of an identified object, that is, the size of the second interaction feature of the identified object is 1 row and P columns.

[0105] In this way, the first identification network is built, the first identification network can extract the interaction information between each identified object and the environment and other identified objects in the video frame to be identified, and obtain the action category and position information of each identified object in the video frame to be identified based on the interaction information, thereby improving the accuracy of action recognition.

[0106] S11, training the first recognition network to obtain a second recognition network, an input of the second recognition network being a video clip and an output of the second recognition network being position information and an action category of each recognition object in a to-be-recognized video frame corresponding to the video clip, the to-be-recognized video frame being a video frame at a middle position of the video clip.

[0107] In an optional embodiment, in order to constrain the first recognition network to accurately output the position information and the action category of each recognition object in the to-be-recognized video frame corresponding to the video clip, the first recognition network needs to be trained to update trainable parameters of the first recognition network. The trainable parameters of the first recognition network include weights in convolution layers and full connection layers in the target detection module, the action recognition module and the environment feature extraction layer, and a parameterized matrix in the interaction feature extraction layer.

[0108] In an optional embodiment, the training of the first recognition network to obtain the second recognition network includes:

[0109] collecting a plurality of video clips with annotation information to construct a training set, the annotation information including real position information and real action categories of all recognition objects in a to-be-detected video frame corresponding to the video clip;

[0110] selecting a video clip from the training set and inputting the first recognition network to obtain a recognition result, the recognition result including predicted position information and predicted action categories of all recognition objects in a to-be-detected video frame corresponding to the video clip;

[0111] calculating a cost function based on the recognition result and the annotation information, the cost function satisfying a relationship formula:

[0112]

[0113] wherein B is a number of recognition objects in the to-be-detected video frame, Z i and Y i are respectively predicted position information and predicted action categories of a recognition object i in the to-be-detected video frame, and are respectively real position information and real action categories of the recognition object i in the to-be-detected video frame, represents a Euclidean distance between Z i and , and represents a cross-entropy loss between Y i and , and Loss is a value of the cost function.

[0114] updating the trainable parameters of the first recognition network based on a back propagation algorithm;

[0115] continuously picking video clips from the training set to update the trainable parameters of the first recognition network until the value of the cost function is less than a preset value, obtaining a second recognition network.

[0116] The preset value is 0.001.

[0117] In this way, the second recognition network is obtained, and the second recognition network can accurately output the position information and the action category of each recognition object in the to-be-recognized video frame corresponding to the video clip.

[0118] S12, selecting video frames within a preset time period before and after any one to-be-recognized video frame in a to-be-recognized video to form a video clip, and adjusting the playing speed of the to-be-recognized video to obtain a plurality of video clips corresponding to the to-be-recognized video frame.

[0119] In an optional embodiment, the to-be-recognized video is any video that needs to be recognized in time and space, and all video frames in the to-be-recognized video are the to-be-recognized video frames.

[0120] In an optional embodiment, the selecting video frames within a preset time period before and after any one to-be-recognized video frame in a to-be-recognized video to form a video clip comprises:

[0121] selecting video frames within a preset time period before the to-be-recognized video frame as a first video frame sequence;

[0122] selecting video frames within a preset time period after the to-be-recognized video frame as a second video frame sequence;

[0123] comparing the number of video frames in the first video frame sequence and the second video frame sequence;

[0124] if the number of video frames in the first video frame sequence is greater than the second video frame sequence, supplementing blank images at the end of the second video frame sequence to make the number of video frames in the first video frame sequence equal to the second video frame sequence;

[0125] if the number of video frames in the first video frame sequence is less than the second video frame sequence, supplementing blank images at the beginning of the first video frame sequence to make the number of video frames in the first video frame sequence equal to the second video frame sequence;

[0126] when the number of video frames in the first video frame sequence is equal to the second video frame sequence, sequentially arranging the first video frame sequence, the to-be-recognized video frame and the second video frame sequence together to form a video clip.

[0127] The preset time period is 5 seconds; the blank image is an image equal in size to the video frame and having pixel values all of 0; the first video frame sequence reflects information before the to-be-identified video frame, and the second video frame sequence reflects information after the to-be-identified video frame, so that the video clip provides context information for the to-be-identified video frame, for assisting in spatio-temporal behavior identification of the to-be-identified video frame.

[0128] In an actual scene, because durations of different actions differ greatly, for example, a dunking action can last for 10 seconds, and a pole vaulting action can take a minute or even more, in order to ensure that the video clip can provide sufficient context information and be applicable to spatio-temporal action detection of different actions, the application adjusts the playing speed of the to-be-identified video multiple times to obtain video clips containing different context information.

[0129] In an optional embodiment, the playing speed includes 0.5 times speed, normal speed, 2 times speed, 3 times speed, and 4 times speed, and the number of times of adjustment of the playing speed and the specific playing speed are preset; one video clip is collected each time the playing speed is adjusted, and one to-be-identified video frame can correspond to multiple video clips. The faster the playing speed, the more the number of video frames contained in a unit of time, the more the number of video frames in the video clip corresponding to the playing speed, and the more the context information provided by the video clip.

[0130] In this way, for the to-be-identified video frame in the to-be-identified video, multiple video clips are collected, different video clips provide different lengths of context information for the to-be-identified video frame, richer environmental features can be obtained, context information loss caused by great differences in durations of different actions is avoided, and the accuracy of subsequent spatio-temporal behavior identification is improved.

[0131] S13, sequentially inputting the multiple video clips into the second identification network to output initial identification results of each video clip, the initial identification results including initial position information and initial action categories of each identification object in the to-be-identified video frame.

[0132] In an optional embodiment, the multiple video clips corresponding to the to-be-identified video frame are respectively inputted into the second identification network to obtain initial identification results of each video clip, that is, one to-be-identified video frame corresponds to multiple initial identification results, and different initial identification results correspond to different context information.

[0133] In this way, the video clips containing different context information are inputted into the second identification network to obtain multiple initial identification results corresponding to the to-be-identified video frame.

[0134] S14, fusing the initial recognition results of different video clips to obtain the spatio-temporal action recognition result of the to-be-recognized video frame, wherein the spatio-temporal action recognition result comprises optimized position information and an optimized action category of each recognition object in the to-be-recognized video frame.

[0135] In an optional embodiment, the step of fusing the initial recognition results of different video clips to obtain the spatio-temporal action recognition result of the to-be-recognized video frame comprises:

[0136] extracting initial action categories of each recognition object in the to-be-recognized video frame from all the initial recognition results;

[0137] selecting, from all the initial action categories of the same recognition object, an initial action category with the highest occurrence frequency as the optimized action category of the recognition object;

[0138] extracting initial position information of each recognition object in the to-be-recognized video frame from all the initial recognition results, and calculating the mean value of all the initial position information of the same recognition object to obtain the optimized position information of each recognition object;

[0139] taking the optimized position information and the optimized action category of each recognition object in the to-be-recognized video frame as the spatio-temporal action recognition result.

[0140] In this way, the multiple initial recognition results corresponding to the to-be-recognized video frame are fused to obtain the final spatio-temporal action recognition result, thereby improving the accuracy of spatio-temporal behavior detection.

[0141] As can be seen from the above technical solutions, the application adjusts the playing speed of the to-be-recognized video to collect video clips comprising different context information, and one to-be-recognized video frame corresponds to multiple video clips; each video clip is input into a second recognition network to extract interaction information between each recognition object in the to-be-recognized video frame and the environment and other recognition objects, and initial recognition results are obtained based on the interaction information; finally, the initial recognition results of different video clips are fused to obtain the spatio-temporal action recognition result, thereby improving the accuracy of the spatio-temporal action recognition result.

[0142] Please refer to Figure 3 , Figure 3 is a functional module diagram of a preferred embodiment of the spatio-temporal action recognition device based on artificial intelligence according to the application. The spatio-temporal action recognition device based on artificial intelligence 11 comprises a building unit 110, a training unit 111, a selecting unit 112, an output unit 113, and a fusing unit 114. The module / unit referred to in the application refers to a series of computer readable instruction segments that can be executed by the processor 13 and can complete a fixed function, which are stored in the memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0143] In an optional embodiment, the building unit 110 is configured to build a first recognition network, the first recognition network comprising a target detection module, an interaction feature extraction module, and an action recognition module, the target detection module being configured to locate position information of all recognition objects in a to-be-recognized video frame, the interaction feature extraction module being configured to extract interaction information between each recognition object and the environment and other recognition objects in the to-be-recognized video frame, and the action recognition module being configured to recognize action categories of all recognition objects in the to-be-recognized video frame.

[0144] In an optional embodiment, the building the first recognition network comprises:

[0145] The target detection module receives the to-be-recognized video frame as input and outputs position information of all recognition objects in the to-be-recognized video frame, the position information comprising center point coordinates and width and height dimensions of a rectangular frame surrounding each recognition object;

[0146] The to-be-recognized video frame is cropped based on the rectangular frame of each recognition object to obtain a cropped image of each recognition object;

[0147] The interaction feature extraction module receives the video segment and the cropped image of each recognition object as input, and comprises an environment feature extraction layer and an interaction feature extraction layer;

[0148] The environment feature extraction layer is configured to extract static features and dynamic features of the video segment and concatenate the static features and the dynamic features as environment features;

[0149] In the interaction feature extraction layer, a convolutional neural network is used to perform multiple convolution operations on the cropped image of each recognition object to obtain pose features of each recognition object, and the pose features and the environment features are concatenated to obtain first interaction features of each recognition object, the first interaction features being used to represent interaction information between the recognition object and the environment;

[0150] The first interaction features of different recognition objects are processed based on a self-attention mechanism to construct second interaction features of each recognition object, the second interaction features being used to represent interaction information between the recognition object and other recognition objects;

[0151] The first interaction features and the second interaction features of the same recognition object are added to obtain fusion interaction features of each recognition object, and the fusion interaction features are input into the action recognition module to obtain action categories of all recognition objects in the to-be-recognized video frame;

[0152] The position information and the action categories of all recognition objects in the to-be-recognized video frame are taken as output results of the first recognition network.

[0153] In an optional embodiment, before the splicing the pose feature and the environment feature to obtain the first interaction feature of each identified object, the method further comprises:

[0154] performing an ROI Align operation on the pose feature of each identified object, the ROI Align operation being configured to process the pose feature of different sizes into a preset size.

[0155] In an optional embodiment, the processing of the first interaction features of different identified objects based on the self-attention mechanism to construct the second interaction feature of each identified object comprises:

[0156] arranging the first interaction features of all identified objects in a preset order along a column direction to obtain a first interaction matrix, each row in the first interaction matrix corresponding to the first interaction feature of an identified object;

[0157] multiplying the first interaction matrix by a parameterized matrix to obtain a Q matrix, a K matrix and a V matrix, the Q matrix, the K matrix and the V matrix respectively comprising Q vectors, K vectors and V vectors of all identified objects;

[0158] processing the Q matrix, the K matrix and the V matrix based on the self-attention mechanism to construct a second interaction matrix, each row in the second interaction matrix corresponding to the second interaction feature of an identified object, the second interaction matrix satisfying the relationship:

[0159]

[0160] wherein Q, K and V represent the Q matrix, the K matrix and the V matrix respectively, K T represents the transpose of the K matrix, d is the number of all identified objects, and F2 represents the second interaction matrix.

[0161] extracting the second interaction feature of each identified object from the second interaction matrix.

[0162] In an optional embodiment, the training unit 111 is configured to train the first identification network to obtain a second identification network, an input of the second identification network being a video clip and an output of the second identification network being the position information and the action category of each identified object in a to-be-identified video frame corresponding to the video clip, the to-be-identified video frame being a video frame at a middle position of the video clip.

[0163] In an optional embodiment, the training of the first identification network to obtain the second identification network comprises:

[0164] collecting a plurality of video clips with labeled information to construct a training set, the labeled information including real position information and real action category of all identified objects in a video frame corresponding to the video clips to be detected;

[0165] selecting a video clip from the training set and inputting the first identification network to obtain an identification result, the identification result including predicted position information and predicted action category of all identified objects in a video frame corresponding to the video clip to be detected;

[0166] calculating a cost function based on the identification result and the labeled information, the cost function satisfying a relationship:

[0167]

[0168] wherein B is the number of identified objects in the video frame to be detected, Z i and Y i are the predicted position information and the predicted action category of identified object i in the video frame to be detected, respectively, and are the real position information and the real action category of identified object i in the video frame to be detected, respectively, denotes the Euclidean distance between Z i and , and denotes the cross-entropy loss between Y i and , and Loss is the value of the cost function;

[0169] updating the trainable parameters of the first identification network based on a back propagation algorithm;

[0170] constantly selecting a video clip from the training set to update the trainable parameters of the first identification network until the value of the cost function is less than a preset value, and obtaining a second identification network.

[0171] In an optional embodiment, the selecting unit 112 is configured to select video frames within a preset time period before and after any one of the video frames to be identified to form a video clip, and adjust the playing speed of the video to be identified to obtain a plurality of video clips corresponding to the video frames to be identified.

[0172] In an optional embodiment, the selecting video frames within a preset time period before and after any one of the video frames to be identified to form a video clip includes:

[0173] selecting video frames within a preset time period before the video frame to be identified from the video to be identified as a first video frame sequence;

[0174] select video frames in a preset time period after the to-be-identified video frame from the to-be-identified video as a second video frame sequence;

[0175] compare the number of video frames in the first video frame sequence and the second video frame sequence;

[0176] if the number of video frames in the first video frame sequence is greater than the second video frame sequence, supplement blank images at the end of the second video frame sequence to make the number of video frames in the first video frame sequence equal to the second video frame sequence;

[0177] if the number of video frames in the first video frame sequence is less than the second video frame sequence, supplement blank images at the beginning of the first video frame sequence to make the number of video frames in the first video frame sequence equal to the second video frame sequence;

[0178] when the number of video frames in the first video frame sequence is equal to the second video frame sequence, sequentially arrange the first video frame sequence, the to-be-identified video frame and the second video frame sequence together to form a video clip.

[0179] In an optional embodiment, the output unit 113 is configured to input the plurality of video clips into the second identification network in sequence, and output an initial identification result of each video clip, wherein the initial identification result comprises initial position information and an initial action category of each identified object in the to-be-identified video frame.

[0180] In an optional embodiment, the fusion unit 114 is configured to fuse the initial identification results of different video clips to obtain a spatio-temporal action identification result of the to-be-identified video frame, wherein the spatio-temporal action identification result comprises optimized position information and an optimized action category of each identified object in the to-be-identified video frame.

[0181] In an optional embodiment, the fusion of the initial identification results of different video clips to obtain the spatio-temporal action identification result of the to-be-identified video frame comprises:

[0182] extracting the initial action category of each identified object in the to-be-identified video frame from all the initial identification results;

[0183] from all the initial action categories of the same identified object, selecting the initial action category with the highest occurrence frequency as the optimized action category of the identified object;

[0184] extracting the initial position information of each identified object in the to-be-identified video frame from all the initial identification results, and calculating the mean value of all the initial position information of the same identified object to obtain the optimized position information of each identified object;

[0185] The optimized position information and the optimized action category of each recognition object in the to-be-identified video frame are taken as the spatio-temporal action recognition result.

[0186] From the above technical solutions, it can be seen that the application adjusts the playing speed of the to-be-identified video to collect video clips including different context information, and one to-be-identified video frame corresponds to multiple video clips; each video clip is input into the second recognition network to extract the interaction information between each recognition object and the environment and other recognition objects in the to-be-identified video frame, and an initial recognition result is obtained based on the interaction information; finally, the initial recognition results of different video clips are fused to obtain a spatio-temporal action recognition result, thereby improving the accuracy of the spatio-temporal action recognition result.

[0187] Please refer to Figure 4 is a structural schematic diagram of an electronic device provided by an embodiment of the application. The electronic device 1 includes a memory 12 and a processor 13. The memory 12 is configured to store computer readable instructions, and the processor 13 is configured to execute the computer readable instructions stored in the memory to implement the spatio-temporal action recognition method based on artificial intelligence according to any of the above embodiments.

[0188] In an optional embodiment, the electronic device 1 further includes a bus, a computer program stored in the memory 12 and executable on the processor 13, such as a spatio-temporal action recognition program based on artificial intelligence.

[0189] Figure 4 Only the electronic device 1 with the memory 12 and the processor 13 is shown, and those skilled in the art can understand that, Figure 4 The structure shown does not constitute a limitation on the electronic device 1, and can include fewer or more components than shown, or combine certain components, or different component arrangements.

[0190] In combination with Figure 1 The memory 12 in the electronic device 1 stores a plurality of computer readable instructions to implement a spatio-temporal action recognition method based on artificial intelligence, and the processor 13 can execute the plurality of instructions to implement:

[0191] A first recognition network is built, the first recognition network includes a target detection module, an interaction feature extraction module and an action recognition module, the target detection module is configured to locate the position information of all recognition objects in a to-be-identified video frame, the interaction feature extraction module is configured to extract the interaction information between each recognition object and the environment and other recognition objects in the to-be-identified video frame, and the action recognition module is configured to identify the action category of all recognition objects in the to-be-identified video frame;

[0192] training the first recognition network to obtain a second recognition network, an input of the second recognition network being a video clip and an output of the second recognition network being position information and an action category of each recognition object in a to-be-recognized video frame corresponding to the video clip, the to-be-recognized video frame being a video frame at a middle position of the video clip;

[0193] selecting video frames in a preset time period before and after any to-be-recognized video frame in a to-be-recognized video to form a video clip, and adjusting a playing speed of the to-be-recognized video to obtain a plurality of video clips corresponding to the to-be-recognized video frame;

[0194] inputting the plurality of video clips into the second recognition network in sequence to output an initial recognition result of each video clip, the initial recognition result including initial position information and an initial action category of each recognition object in the to-be-recognized video frame;

[0195] fusing the initial recognition results of different video clips to obtain a spatio-temporal action recognition result of the to-be-recognized video frame, the spatio-temporal action recognition result including optimized position information and an optimized action category of each recognition object in the to-be-recognized video frame.

[0196] Specifically, the specific implementation method of the processor 13 to the above instructions can refer to Figure 1 The descriptions of related steps in the corresponding embodiments are not repeated here.

[0197] Those skilled in the art can understand that the schematic diagram is only an example of the electronic device 1 and does not constitute a limitation on the electronic device 1. The electronic device 1 can be in a bus type structure or a star type structure. The electronic device 1 can further include more or less other hardware or software or different component arrangements, for example, the electronic device 1 can further include an input / output device, a network access device, and the like.

[0198] It should be noted that the electronic device 1 is only an example. Other existing or future electronic products can also be applicable to the present application and should be included in the protection scope of the present application and included by reference.

[0199] The memory 12 includes at least one type of readable storage medium, which can be non-volatile or volatile. The readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. The memory 12 can be an internal storage unit of the electronic device 1 in some embodiments, such as a mobile hard disk of the electronic device 1. The memory 12 can also be an external storage device of the electronic device 1 in other embodiments, such as a plug-in mobile hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on the electronic device 1. The memory 12 can be used to store application software and various data installed on the electronic device 1, such as the code of the spatiotemporal action recognition program based on artificial intelligence, and can also be used to temporarily store data that has been output or will be output.

[0200] The processor 13 can be composed of an integrated circuit in some embodiments, such as a single packaged integrated circuit or a plurality of packaged integrated circuits with the same or different functions, including one or more combinations of central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the electronic device 1, which connects all components of the electronic device 1 through various interfaces and lines, executes programs or modules stored in the memory 12 (such as the spatiotemporal action recognition program based on artificial intelligence), and calls data stored in the memory 12 to perform various functions of the electronic device 1 and process data.

[0201] The processor 13 executes the operating system and various application programs installed on the electronic device 1. The processor 13 executes the application programs to implement the steps in each of the above-mentioned spatiotemporal action recognition methods based on artificial intelligence, such as Figure 1 the steps shown.

[0202] For example, the computer program can be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules / units can be a series of computer-readable instruction segments that can complete a specific function, which are used to describe the execution process of the computer program in the electronic device 1. For example, the computer program can be divided into a building unit 110, a training unit 111, a selection unit 112, an output unit 113, and a fusion unit 114.

[0203] The integrated units in the form of software function modules can be stored in a computer readable storage medium. The software function modules are stored in a storage medium and include a plurality of instructions for causing a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to execute part of the space-time action recognition method based on artificial intelligence described in the embodiments of the present application.

[0204] The modules / units integrated in the electronic device 1, if realized in the form of software function modules and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiments can also be implemented by a computer program to instruct related hardware devices to complete, and the computer program can be stored in a computer readable storage medium. When the processor executes the computer program, the steps of each method embodiment can be implemented.

[0205] The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, and other memories, etc.

[0206] Further, the computer readable storage medium can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function, etc.; and the data storage area can store data created according to the use of the blockchain node, etc.

[0207] The blockchain referred to in the present application is a new application mode of distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, and other computer technologies. Blockchain, in essence, is a decentralized database, which is a series of data blocks associated using cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity (anti-fake) of the information and generate the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer, etc.

[0208] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus or the like. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one arrow is shown in Figure 4

[0209] The embodiment of the present application further provides a computer readable storage medium (not shown in the figure), which stores computer readable instructions. The computer readable instructions are executed by a processor in an electronic device to implement the artificial intelligence based space-time action recognition method in any of the above embodiments.

[0210] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the above described device embodiment is merely illustrative, and for example, the division of the modules is merely a logical function division, and there can be another division manner in actual implementation.

[0211] The modules described as separated components can or can not be physically separated, and the components displayed as modules can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. According to actual needs, some or all of the modules can be selected to achieve the purpose of the embodiment of the present application.

[0212] In addition, each functional module in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of hardware plus software functional modules.

[0213] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The plurality of units or devices stated in the specification can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.

[0214] ​Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. An artificial intelligence-based spatio-temporal action recognition method, characterized by, The method comprises: a first recognition network is built, the first recognition network comprises a target detection module, an interaction feature extraction module and an action recognition module, the target detection module is used for positioning the position information of all recognition objects in a to-be-recognized video frame, the interaction feature extraction module is used for extracting the interaction information between each recognition object and the environment and other recognition objects in the to-be-recognized video frame, and the action recognition module is used for recognizing the action category of all recognition objects in the to-be-recognized video frame; the first recognition network is trained to obtain a second recognition network, the input of the second recognition network is a video clip, and the output is the position information and the action category of each recognition object in a to-be-recognized video frame corresponding to the video clip, the to-be-recognized video frame is a video frame at the middle position of the video clip; video frames in a preset time period before and after any to-be-recognized video frame in a to-be-recognized video are selected to form a video clip, and the playing speed of the to-be-recognized video is adjusted to obtain a plurality of video clips corresponding to the to-be-recognized video frame; the plurality of video clips are sequentially input into the second recognition network, and the initial recognition result of each video clip is output, the initial recognition result comprises initial position information and initial action category of each recognition object in the to-be-recognized video frame; the initial recognition results of different video clips are fused to obtain the spatio-temporal action recognition result of the to-be-recognized video frame, and the spatio-temporal action recognition result comprises optimized position information and optimized action category of each recognition object in the to-be-recognized video frame. 2.The spatio-temporal action recognition method based on artificial intelligence of claim 1, wherein, The first recognition network is built, comprising: the input of the target detection module is the to-be-recognized video frame, and the output is the position information of all recognition objects in the to-be-recognized video frame, the position information comprises the center point coordinates and the width and height dimensions of the rectangular frame surrounding the recognition object; the to-be-recognized video frame is cropped based on the rectangular frame of the recognition object to obtain a cropped picture of each recognition object; the input of the interaction feature extraction module is the video clip and the cropped picture of each recognition object, and the interaction feature extraction module comprises an environment feature extraction layer and an interaction feature extraction layer; the environment feature extraction layer is used for extracting static features and dynamic features of the video clip, and splicing the static features and the dynamic features together as environment features; in the interaction feature extraction layer, the cropped picture of each recognition object is subjected to multiple convolution operations by using a convolutional neural network to obtain the posture features of each recognition object, the posture features and the environment features are spliced to obtain the first interaction features of each recognition object, and the first interaction features are used for representing the interaction information between the recognition object and the environment; the first interaction features of different recognition objects are processed based on a self-attention mechanism to construct the second interaction features of each recognition object, and the second interaction features are used for representing the interaction information between the recognition object and other recognition objects; the first interaction features and the second interaction features of the same recognition object are added to obtain the fusion interaction features of each recognition object, and the fusion interaction features are input into the action recognition module to obtain the action category of all recognition objects in the to-be-recognized video frame; The position information and the action category of all the identified objects in the to-be-identified video frame are taken as output results of the first identification network. 3.The spatio-temporal action recognition method based on artificial intelligence of claim 2, wherein, Before the splicing of the posture feature and the environment feature to obtain the first interaction feature of each identified object, the method further comprises: performing an ROI Align operation on the posture feature of each identified object, the ROI Align operation being used to process posture features of different sizes into a preset size. 4.The spatio-temporal action recognition method based on artificial intelligence of claim 2, wherein, The processing of the first interaction features of different identified objects based on the self-attention mechanism to construct the second interaction feature of each identified object comprises: arranging the first interaction features of all the identified objects in a preset order along a column direction to obtain a first interaction matrix, each row in the first interaction matrix corresponding to the first interaction feature of an identified object; multiplying the first interaction matrix by a parameterized matrix to obtain a Q matrix, a K matrix and a V matrix, the Q matrix, the K matrix and the V matrix respectively comprising Q vectors, K vectors and V vectors of all the identified objects; processing the Q matrix, the K matrix and the V matrix based on the self-attention mechanism to construct a second interaction matrix, each row in the second interaction matrix corresponding to the second interaction feature of an identified object, the second interaction matrix satisfying a relationship: wherein Q, K, V represent a Q matrix, a K matrix, and a V matrix, respectively, K T denotes a transpose of the K matrix, d is a number of all recognition objects, F2 denotes a second interaction matrix; extracting the second interaction feature of each identified object from the second interaction matrix. 5.The spatio-temporal action recognition method based on artificial intelligence of claim 1, wherein, The training of the first identification network to obtain a second identification network comprises: collecting a plurality of video clips with labeled information to construct a training set, the labeled information including real position information and real action categories of all the identified objects in the to-be-detected video frames corresponding to the video clips; selecting a video clip from the training set and inputting the first identification network to obtain an identification result, the identification result including predicted position information and predicted action categories of all the identified objects in the to-be-detected video frames corresponding to the video clip; calculating a cost function based on the identification result and the labeled information, the cost function satisfying a relationship: wherein B is the number of recognized objects in the video frame to be detected, Z i and Y i are respectively the predicted position information and the predicted action category of the recognized object i in the video frame to be detected, and are respectively the real position information and the real action category of the recognized object i in the video frame to be detected, denotes the Euclidean distance between Z i and , and denotes the cross-entropy loss between Y i and , and Loss is the value of the cost function. updating trainable parameters of the first identification network based on a back propagation algorithm; continuously selecting video clips from the training set to update the trainable parameters of the first identification network until the value of the cost function is less than a preset value, thereby obtaining a second identification network. 6.The spatio-temporal action recognition method based on artificial intelligence of claim 1, wherein, The selection of video frames within a preset time period before and after the to-be-identified video frame from the to-be-identified video to form a video clip comprises: selecting video frames within a preset time period before the to-be-identified video frame as a first video frame sequence from the to-be-identified video; selecting video frames within a preset time period after the to-be-identified video frame as a second video frame sequence from the to-be-identified video; comparing the number of video frames in the first video frame sequence and the second video frame sequence; if the number of video frames in the first video frame sequence is greater than that in the second video frame sequence, supplementing blank images at the end of the second video frame sequence to make the number of video frames in the first video frame sequence equal to that in the second video frame sequence; if the number of video frames in the first video frame sequence is less than the second video frame sequence, supplementing blank images at the front of the first video frame sequence to make the number of video frames in the first video frame sequence equal to the second video frame sequence; when the number of video frames in the first video frame sequence is equal to the second video frame sequence, arranging the first video frame sequence, the video frame to be recognized and the second video frame sequence in sequence to form a video clip. 7.The spatio-temporal action recognition method based on artificial intelligence of claim 1, wherein, the spatio-temporal action recognition result of the video frame to be recognized is obtained by fusing initial recognition results of different video clips, and the spatio-temporal action recognition result includes: extracting initial action categories of each recognition object in the video frame to be recognized from all initial recognition results; from all initial action categories of the same recognition object, selecting an initial action category with the highest occurrence frequency as an optimized action category of the recognition object; extracting initial position information of each recognition object in the video frame to be recognized from all initial recognition results, and calculating the mean value of all initial position information of the same recognition object to obtain optimized position information of each recognition object; taking the optimized position information and the optimized action category of each recognition object in the video frame to be recognized as the spatio-temporal action recognition result.

8. An artificial intelligence-based spatio-temporal action recognition apparatus, characterized by, The device includes: a building unit configured to build a first recognition network, the first recognition network including a target detection module, an interactive feature extraction module and an action recognition module, the target detection module being configured to locate position information of all recognition objects in a video frame to be recognized, the interactive feature extraction module being configured to extract interactive information between each recognition object in the video frame to be recognized and the environment and other recognition objects, and the action recognition module being configured to recognize action categories of all recognition objects in the video frame to be recognized; a training unit configured to train the first recognition network to obtain a second recognition network, the second recognition network having a video clip as an input and having position information and action categories of each recognition object in a video frame to be recognized corresponding to the video clip as an output, the video frame to be recognized being a video frame at a middle position of the video clip; a selection unit configured to select video frames within a preset time period before and after any one video frame to be recognized in a video to be recognized to form a video clip, and adjust a playing speed of the video to be recognized to obtain a plurality of video clips corresponding to the video frame to be recognized; an output unit configured to input the plurality of video clips into the second recognition network in sequence, and output initial recognition results of each video clip, the initial recognition results including initial position information and initial action categories of each recognition object in the video frame to be recognized; a fusion unit configured to fuse initial recognition results of different video clips to obtain a spatio-temporal action recognition result of the video frame to be recognized, the spatio-temporal action recognition result including optimized position information and optimized action categories of each recognition object in the video frame to be recognized.

9. An electronic device, comprising: The electronic device includes: a memory storing computer readable instructions; and a processor executing the computer readable instructions stored in the memory to implement the spatio-temporal action recognition method based on artificial intelligence according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by the processor to implement the artificial intelligence-based spatio-temporal action recognition method in any one of claims 1 to 7.

Images