Motion recognition method, device, electronic device and storage medium

Abstract

The present disclosure provides an action recognition method, device, electronic device, storage medium and program product, which relate to the field of artificial intelligence technology, especially to the field of visual computer technology, the field of deep learning technology and the field of three-dimensional reconstruction technology. The specific implementation scheme is: based on the image sequence, determine the skeleton key point group sequence of the target object; based on the skeleton key point group sequence, determine the skeleton angle sequence of the target object; and based on the image feature sequence, the skeleton key point feature sequence and the skeleton angle feature sequence, perform action recognition on the target object to determine the action recognition result of the target object, the image feature sequence is generated based on the image sequence, the skeleton key point feature sequence is generated based on the skeleton key point group sequence, and the skeleton angle feature sequence is generated based on the skeleton angle sequence.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of visual computer technology, the field of deep learning technology and the field of three-dimensional reconstruction technology, and specifically to a method, device, electronic device, storage medium and program product for motion recognition. Background Art

[0002] Computer vision technology is a science that studies how to use computers to "see". Computer vision technology can be applied to scenarios such as image recognition, image semantic understanding, image retrieval, 3D object reconstruction, virtual reality, and action recognition. For each scenario, it is worth exploring how to use computer vision technology to generate reasonable and accurate results. Summary of the invention

[0003] The present disclosure provides a motion recognition method, device, electronic device, storage medium and program product.

[0004] According to one aspect of the present disclosure, a method for action recognition is provided, comprising: determining a skeleton key point group sequence of a target object based on an image sequence; determining a skeleton angle sequence of the target object based on the skeleton key point group sequence; and performing action recognition on the target object based on an image feature sequence, a skeleton key point feature sequence and a skeleton angle feature sequence to determine an action recognition result of the target object, wherein the image feature sequence is generated based on the image sequence, the skeleton key point feature sequence is generated based on the skeleton key point group sequence, and the skeleton angle feature sequence is generated based on the skeleton angle sequence.

[0005] According to another aspect of the present disclosure, there is provided a motion recognition device, comprising: a key point determination module, for determining a skeleton key point group sequence of a target object based on an image sequence; an angle determination module, for determining a skeleton angle sequence of the target object based on the skeleton key point group sequence; and a motion recognition module, for performing motion recognition on the target object based on an image feature sequence, a skeleton key point feature sequence and a skeleton angle feature sequence, and determining a motion recognition result of the target object, wherein the image feature sequence is generated based on the image sequence, the skeleton key point feature sequence is generated based on the skeleton key point group sequence, and the skeleton angle feature sequence is generated based on the skeleton angle sequence.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method as disclosed in the present disclosure.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method of the present disclosure.

[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method of the present disclosure is implemented.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1 An exemplary system architecture to which the method and device for motion recognition according to an embodiment of the present disclosure can be applied is schematically shown;

[0012] Figure 2 The flowchart of the action recognition method according to the embodiment of the present disclosure is schematically shown;

[0013] Figure 3 A network structure diagram of an action recognition model according to an embodiment of the present disclosure is schematically shown;

[0014] Figure 4 The flowchart of determining the skeleton key point group sequence according to the embodiment of the present disclosure is schematically shown;

[0015] Figure 5 A block diagram schematically shows a motion recognition device according to an embodiment of the present disclosure; and

[0016] Figure 6 A block diagram of an electronic device suitable for implementing a motion recognition method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0017] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0018] The present disclosure provides a motion recognition method, device, electronic device, storage medium and program product.

[0019] According to an embodiment of the present disclosure, the action recognition method may include: determining a skeleton key point group sequence of a target object based on an image sequence. Determining a skeleton angle sequence of a target object based on a skeleton key point group sequence. Performing action recognition on the target object based on an image feature sequence, a skeleton key point feature sequence, and a skeleton angle feature sequence to determine an action recognition result of the target object. The image feature sequence is generated based on an image sequence, the skeleton key point feature sequence is generated based on a skeleton key point group sequence, and the skeleton angle feature sequence is generated based on a skeleton angle sequence.

[0020] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0021] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0022] Figure 1 An exemplary system architecture to which the method and apparatus for motion recognition according to an embodiment of the present disclosure can be applied is schematically shown.

[0023] It should be noted that Figure 1 The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, the exemplary system architecture to which the motion recognition method and apparatus can be applied may include a terminal device, but the terminal device may implement the motion recognition method and apparatus provided by the embodiments of the present disclosure without interacting with the server.

[0024] like Figure 1 As shown, the system architecture 100 according to this embodiment may include an information collection device 101, a network 102 and a server 103. The network 102 is used to provide a medium for a communication link between the information collection device 101 and the server 103. The network 102 may include various connection types, such as wired and / or wireless communication links, etc.

[0025] The user can use the information collection device 101 to interact with the server 103 through the network 102 to receive or send videos, etc. Various communication client applications can be installed on the information collection device 101, such as instant messaging tools, email clients and / or social platform software, etc. (only as examples).

[0026] The information collection device 101 may be any electronic device having a display screen and supporting web browsing, including but not limited to one or more of an event camera, a depth camera, a binocular camera, and a monocular camera.

[0027] The server 103 may be a server that provides various services, such as a background management server (only an example) that provides support for the video sent by the user using the information collection device 101. The background management server may analyze and process the received video.

[0028] The motion recognition method provided in the embodiment of the present disclosure can generally be performed by the server 103. Accordingly, the motion recognition device provided in the embodiment of the present disclosure can generally be set in the server 103. The motion recognition method provided in the embodiment of the present disclosure can also be performed by a server or server cluster that is different from the server 103 and can communicate with the information collection device 101 and / or the server 103. Accordingly, the motion recognition device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 103 and can communicate with the information collection device 101 and / or the server 103.

[0029] For example, in a sports stadium, an event camera can be hoisted on the top of the sports stadium, and a binocular camera and a depth camera can be installed on the side walls of the stadium and around the competition venue. The server can receive videos from the event camera, the binocular camera, and the depth camera. An image sequence is obtained from the video to determine the skeleton key point group sequence of the target object. Based on the skeleton key point group sequence, the skeleton angle sequence of the target object is determined. Based on the image feature sequence, the skeleton key point feature sequence, and the skeleton angle feature sequence, the target object is subjected to action recognition to determine the action recognition result of the target object. Alternatively, a server or server cluster that can communicate with the information acquisition device 101 and / or the server 103 analyzes the image sequence and finally determines the action recognition result of the target object, such as an athlete.

[0030] It should be understood that Figure 1 The number of information collection devices, networks and servers in the embodiment is only for illustration. Any number of information collection devices, networks and servers may be provided according to the implementation requirements.

[0031] It should be noted that the sequence numbers of the operations in the following method are only used as representations of the operations for the purpose of description, and should not be regarded as representing the execution order of the operations. Unless explicitly stated, the method does not need to be executed completely in the order shown.

[0032] Figure 2 The flowchart of the action recognition method according to the embodiment of the present disclosure is schematically shown.

[0033] like Figure 2As shown, the method includes operations S210 to S230.

[0034] In operation S210 , a skeleton key point group sequence of a target object is determined based on an image sequence.

[0035] In operation S220, a skeleton angle sequence of the target object is determined based on the skeleton key point group sequence.

[0036] In operation S230, action recognition is performed on the target object based on the image feature sequence, the skeleton key point feature sequence, and the skeleton angle feature sequence to determine an action recognition result of the target object.

[0037] According to an embodiment of the present disclosure, an image feature sequence is generated based on an image sequence, a skeleton key point feature sequence is generated based on a skeleton key point group sequence, and a skeleton angle feature sequence is generated based on a skeleton angle sequence.

[0038] According to an embodiment of the present disclosure, an image sequence may include multiple frames of images with time sequence. For example, an image sequence includes an image captured at time t1, an image captured at time t2, and an image captured at time t3. Time t1, time t2, and time t3 are three consecutive moments.

[0039] According to an embodiment of the present disclosure, based on an image sequence, a skeleton key point group sequence of a target object can be determined. The skeleton key point group sequence can include multiple skeleton key point groups with temporal correspondence to the image sequence. Each skeleton key point group includes multiple skeleton key points such as skeleton joints, and each skeleton key point can include three-dimensional key point coordinates.

[0040] For example, based on an image sequence including an image collected at t1, an image collected at t2, and an image collected at t3, a skeleton key point group of a target object at t1, a skeleton key point group at t2, and a skeleton key point group at t3 can be determined to form a skeleton key point group sequence. Each skeleton key point group can include arm key points, shoulder key points, wrist key points, leg key points, ankle key points, and crotch key points of a target object such as a human body.

[0041] According to an embodiment of the present disclosure, based on a skeleton key point group sequence, a skeleton angle sequence of a target object can be determined. For example, for each skeleton key point group, based on the three-dimensional key point coordinates of each skeleton key point, a coordinate vector of each segment of skeleton is obtained. Based on the coordinate vector of each segment of skeleton, a skeleton angle is determined. For example, a skeleton key point group includes three-dimensional key point coordinates of a right shoulder key point, a right elbow key point and a right hand key point, respectively. Based on the three-dimensional key point coordinates of the right shoulder key point and the right elbow key point respectively, a coordinate vector of a skeleton between a right shoulder and a right elbow is determined. Based on the three-dimensional key point coordinates of the right elbow key point and the right hand key point respectively, a coordinate vector of a skeleton between a right elbow and a right hand is determined. Based on the coordinate vector of a skeleton between a right shoulder and a right elbow and a coordinate vector of a skeleton between a right elbow and a right hand, a skeleton angle of a right shoulder, a right elbow and a right hand is determined. By analogy, based on a skeleton key point group sequence, a skeleton angle sequence corresponding one to one to a skeleton key point group sequence can be determined. The bone angle sequence may include multiple bone angles with time series correlation, but is not limited to this. The bone angle sequence may also include multiple bone angle groups with time series correlation, and each bone angle group includes multiple bone angles.

[0042] According to an embodiment of the present disclosure, based on an image feature sequence, a skeleton key point feature sequence and a skeleton angle feature sequence, a target object is subjected to action recognition, and the action recognition result of the target object is determined, which may include: inputting the image feature sequence, the skeleton key point feature sequence and the skeleton angle feature sequence into an action recognition model to obtain the action recognition result. However, it is not limited to this. It may also include: matching the image feature sequence, the skeleton key point feature sequence and the skeleton angle feature sequence with the template feature sequence to determine the matching result. When determining that the matching result is used to characterize the match between the two, the action pattern matched with the template feature sequence is used as the action recognition result.

[0043] According to the relevant examples, the action recognition result of the target object can be determined based on one or two of the image feature sequence, the skeleton key point feature sequence and the skeleton angle feature sequence. Compared with the above method, the action recognition result of the target object is determined based on the image feature sequence, the skeleton key point feature sequence and the skeleton angle feature sequence, which can combine various forms of data, making the data rich in variety and large in information, thereby improving the accuracy of the action recognition result.

[0044] According to the embodiments of the present disclosure, Figure 2The operation S230 shown, based on the image feature sequence, the skeleton key point feature sequence and the skeleton angle feature sequence, performs action recognition on the target object and determines the action recognition result of the target object, which may include: for each image feature in the image feature sequence, according to the time series relationship, determining the target skeleton key point feature that matches the image feature from the skeleton key point feature sequence, and determining the target skeleton angle feature that matches the image feature from the skeleton angle feature sequence. Based on the image feature, the target skeleton key point feature and the target skeleton angle feature, generate a target comprehensive feature to obtain a target comprehensive feature sequence. Based on the target comprehensive feature sequence, perform action recognition on the target object and determine the action recognition result of the target object.

[0045] According to an embodiment of the present disclosure, a target skeleton key point feature that matches the image feature is determined from the skeleton key point feature sequence according to a time sequence relationship. For example, for the image feature at time t1, the skeleton key point feature at time t1 can be determined from the skeleton key point feature sequence as the target skeleton key point feature.

[0046] According to an embodiment of the present disclosure, a target bone angle feature that matches the image feature is determined from the bone angle feature sequence according to a time sequence relationship. For example, for the image feature at time t1, the bone angle feature at time t1 can be determined from the bone angle feature sequence as the target bone angle feature.

[0047] According to an embodiment of the present disclosure, generating a target comprehensive feature based on image features, target skeleton key point features and target skeleton angle features may include: splicing the image features, target skeleton key point features and target skeleton angle features to generate the target comprehensive feature. However, it is not limited to this. It may also include: splicing the image features, target skeleton key point features and target skeleton angle features to generate an initial comprehensive feature. Feature extraction is performed on the initial comprehensive feature to obtain the target comprehensive feature.

[0048] According to an embodiment of the present disclosure, based on the target comprehensive feature sequence, the target object is subjected to action recognition, and the action recognition result of the target object is determined, which may include: inputting the target comprehensive feature sequence into the action recognition module to obtain the action recognition result. The action recognition module may include a codec (Transformer) and a classifier, but is not limited to this, and may also include an LSTM (Long Short-Term Memory) and a classifier. The classifier may include a fully connected layer and an activation function. The network structure of the action recognition module is not limited, as long as it is a module that can obtain the action recognition result based on the target comprehensive feature sequence.

[0049] According to the embodiments of the present disclosure, based on the target comprehensive feature sequence, the target object is subjected to action recognition, and multiple features of different types such as image features, target skeleton key point features and target skeleton angle features can be combined, and the associated time series features in the target comprehensive feature sequence can also be used. This can make the action recognition result accurate and effective, and the processing efficiency is high.

[0050] According to an embodiment of the present disclosure, when executing Figure 2 In operation S210 shown, before determining the skeleton key point group sequence of the target object based on the image sequence, the action recognition method may further include the following operations.

[0051] For example, feature extraction is performed on an image sequence and an event data sequence with the same time sequence as the image sequence to obtain an image feature sequence. Feature extraction is performed on a bone angle sequence to obtain a bone angle feature sequence. Feature extraction is performed on a bone key point group sequence to obtain a bone key point feature sequence.

[0052] According to an embodiment of the present disclosure, images in an image sequence may include color images such as RGB images. The event data sequence may correspond to the image sequence one by one. Each event data may be acquired by an event camera. Based on the event mechanism, for each captured pixel position, when the pixel value changes by more than a predetermined pixel threshold, the event data of the pixel is generated. The pixel value may include the grayscale value, brightness value or light intensity of the pixel. The event data may include: the pixel coordinates of the event, the timestamp of the event occurrence and the polarity of the event occurrence. The polarity of the event occurrence can be used to characterize the change of the pixel value, for example, to characterize the polarity of the pixel value change from low to high or to characterize the polarity of the pixel value change from high to low.

[0053] According to an embodiment of the present disclosure, extracting features from an image sequence and an event data sequence to obtain an image feature sequence may include: inputting the image sequence and the event data sequence into an image feature extraction module to obtain an image feature sequence. The image feature extraction module may include RestNet (residual network), but is not limited thereto. As long as it is a network structure that can be used to extract image features.

[0054] According to an optional embodiment of the present disclosure, the operation of "extracting features from the image sequence and the event data sequence to obtain an image feature sequence" may be replaced by the operation of "extracting features from the image sequence to obtain an image feature sequence".

[0055] Compared with the image feature sequence obtained by only extracting features from the image sequence, the image feature sequence obtained using the feature extraction method provided by the embodiment of the present disclosure can use the event data sequence to denoise the image sequence, highlight the features of the dynamic target objects in the image sequence, and filter out the features of the static objects in the image sequence, so that the features in the image feature sequence are effective and accurate.

[0056] According to an embodiment of the present disclosure, feature extraction is performed on a bone angle sequence to obtain a bone angle feature sequence, which may include: inputting the bone angle sequence into a bone angle feature extraction module to obtain a bone angle feature sequence. The bone angle feature extraction module may include a multilayer perceptron (MLP), but is not limited thereto, as long as it is a network structure or algorithm that can convert bone angle data into a feature vector.

[0057] According to an embodiment of the present disclosure, feature extraction is performed on a skeleton key point group sequence to obtain a skeleton key point feature sequence, which may include: inputting the skeleton key point group sequence into a key point feature extraction module to obtain a skeleton key point feature sequence. The key point feature extraction module may include Point Net (point cloud segmentation network), but is not limited thereto, as long as it is a network structure or algorithm that can extract features from key point data.

[0058] According to the embodiments of the present disclosure, different feature extraction methods are selected for different data, which is highly targeted and has high processing efficiency.

[0059] Figure 3 The network structure diagram of the action recognition model according to the embodiment of the present disclosure is schematically shown.

[0060] like Figure 3 As shown, the action recognition method can be performed using an action recognition model. The action recognition model can include an image feature extraction module M310, a bone angle feature extraction module M320, a key point feature extraction module M330, and an action recognition module M340.

[0061] The image sequence 310 and the event data sequence 320 can be input into the image feature extraction module M310 to obtain the image feature sequence 330. The bone angle sequence 340 is input into the bone angle feature extraction module M320 to obtain the bone angle feature sequence 350. The bone key point group sequence 360 ​​is input into the key point feature extraction module M330 to obtain the bone key point feature sequence 370. The image features, the target bone key point features and the target bone angle features are spliced ​​to generate the target comprehensive features. Thus, the target comprehensive feature sequence 380 is obtained. The target comprehensive feature sequence 380 is input into the action recognition module M340 to obtain the action recognition result 390.

[0062] According to the embodiments of the present disclosure, Figure 2 The operation S210 shown, obtaining a skeleton key point group sequence of the target object based on the image sequence, may include: obtaining a skeleton key point group sequence of the target object based on the image sequence and an event data sequence with the same time sequence as the image sequence.

[0063] According to an embodiment of the present disclosure, obtaining a skeleton key point group sequence of a target object based on an image sequence and an event data sequence with the same time sequence as the image sequence may include: inputting the image sequence and the event data sequence into a second image feature extraction module to obtain a second image feature sequence. Inputting the second image feature sequence into a key point recognition module to obtain a skeleton key point group sequence.

[0064] According to the embodiment of the present disclosure, the second image feature extraction module may be the same as or different from the image feature extraction module provided in the embodiment of the present disclosure, as long as it can extract features from the image and reduce noise on the image sequence using the event data sequence.

[0065] According to an embodiment of the present disclosure, the key point recognition module may include a visual codec, but is not limited thereto, and may also be a lightweight visual codec or a Mobile Pose (a lightweight pose estimation) model.

[0066] According to an optional embodiment of the present disclosure, inputting the second image feature sequence into a key point recognition module to obtain a skeleton key point group sequence may include: inputting the second image feature sequence into a posture simulation model to obtain a model parameter sequence. Inputting the model parameter sequence into the posture model to obtain an updated posture simulation model sequence. Based on the updated posture simulation model sequence, a skeleton key point group sequence is obtained. However, it is not limited to this. The second image feature sequence can also be directly input into the key point recognition module to obtain a skeleton key point group sequence.

[0067] According to an embodiment of the present disclosure, model parameters may include posture parameters and shape parameters. The posture simulation model may include a pre-set three-dimensional model, a three-dimensional model whose morphological changes are controlled by a fixed number of parameters. For example, a model parameter sequence may be input into the posture simulation model to obtain an updated posture simulation model sequence. The updated posture simulation model sequence presents a morphology that matches the angles of each bone of the target object in the image sequence. In the case where the target object is a human object, the posture simulation model may include an SMPL (Skinned Multi-Person Linear Model) model. Based on the updated posture simulation model sequence, a skeleton key point group sequence is obtained.

[0068] According to an embodiment of the present disclosure, the skeleton key point group sequence of the target object is determined based on an image sequence and an event data sequence with the same time sequence as the image sequence. The event data sequence can be used to perform denoising on the image sequence to remove noise in the image sequence, thereby making the obtained skeleton key point group sequence highly accurate.

[0069] According to the embodiments of the present disclosure, Figure 2 The operation S210 shown is to determine a skeleton key point group sequence of the target object based on an image sequence and an event data sequence having the same time sequence as the image sequence, and may also include a correction operation.

[0070] For example, based on the image sequence and the event data sequence with the same timing as the image sequence, the initial skeleton key point group sequence of the target object is obtained. Based on the image sequence and the depth data sequence, the auxiliary skeleton key point group sequence of the target object is determined. Based on the initial skeleton key point group sequence and the auxiliary skeleton key point group sequence, the skeleton key point group sequence is determined.

[0071] According to an embodiment of the present disclosure, a skeleton key point group sequence obtained based on an image sequence and an event data sequence having the same timing as the image sequence can be used as an initial skeleton key point group sequence. Based on the image sequence and the depth data sequence, an auxiliary skeleton key point group sequence of the target object is determined. The initial skeleton key point group sequence is corrected using the auxiliary skeleton key point group sequence, so that the accuracy of the skeleton key point group sequence is high.

[0072] According to an embodiment of the present disclosure, each image in the image sequence may include an image captured by a binocular camera. Each depth data in the depth data sequence may include data captured by a depth camera. The depth data may include the distance between each pixel of the target object and the depth camera in the world coordinate system.

[0073] According to an embodiment of the present disclosure, a 3D somatosensory camera may be used to simultaneously capture a color image and a depth image to obtain an image sequence and a depth data sequence.

[0074] According to an embodiment of the present disclosure, determining the auxiliary skeleton key point group sequence of the target object based on the image sequence and the depth data sequence may include: determining the target depth data matching the image from the depth data sequence. Based on the image and the target depth data, determining the three-dimensional key point coordinate group of the target object in the real world to obtain the auxiliary skeleton key point group. Then determining the auxiliary skeleton key point group sequence.

[0075] According to an embodiment of the present disclosure, determining a skeleton key point group sequence based on an initial skeleton key point group sequence and an auxiliary skeleton key point group sequence may include: determining the skeleton key point group sequence based on the initial skeleton key point group sequence when it is determined that the initial skeleton key point group sequence matches the auxiliary skeleton key point group sequence. When it is determined that the initial skeleton key point group sequence does not match the auxiliary skeleton key point group sequence, updating the initial skeleton key point group sequence based on the auxiliary skeleton key point group sequence to obtain the skeleton key point group sequence.

[0076] Figure 4 The flowchart of determining the skeleton key point group sequence according to an embodiment of the present disclosure is schematically shown.

[0077] like Figure 4 As shown, determining the skeleton key point group sequence may include operations S410-S450.

[0078] In operation S410, an initial skeleton key point group sequence of the target object is obtained based on an image sequence and an event data sequence having the same time sequence as the image sequence.

[0079] In operation S420, a sequence of auxiliary skeleton key point groups of the target object is determined based on the image sequence and the depth data sequence.

[0080] In operation S430, it is determined whether the initial skeleton key point group sequence and the auxiliary skeleton key point group sequence match each other. If it is determined that the initial skeleton key point group sequence and the auxiliary skeleton key point group sequence match each other, operation S440 is performed. Otherwise, operation S450 is performed.

[0081] In operation S440, a skeleton key point group sequence is determined based on the initial skeleton key point group sequence.

[0082] In operation S450, the initial skeleton key point group sequence is updated based on the auxiliary skeleton key point group sequence to obtain a skeleton key point group sequence.

[0083] According to the embodiments of the present disclosure, Figure 4The operation S430 shown in the figure determines whether the initial skeleton key point group sequence and the auxiliary skeleton key point group sequence match, and may include: for the initial skeleton key point group in the initial skeleton key point group sequence, determining the target auxiliary skeleton key point group that matches the initial skeleton key point group timing. Compare each initial skeleton key point in the initial skeleton key point group with the target auxiliary skeleton key point in the target auxiliary skeleton key point group that matches the initial skeleton key point pixel position, determine the key point coordinate difference between the initial skeleton key point and the target auxiliary skeleton key point, and when the key point coordinate difference is greater than the key point coordinate threshold, it is explained that the initial skeleton key point does not match the target auxiliary skeleton key point. When the key point coordinate difference is less than or equal to the key point coordinate threshold, it is explained that the initial skeleton key point matches the target auxiliary skeleton key point. When the number of unmatched initial skeleton key points in the initial skeleton key point group reaches a predetermined number threshold, it is explained that the initial skeleton key point group sequence and the auxiliary skeleton key point group sequence do not match. When the number of unmatched initial skeleton key point groups reaches a predetermined number threshold, it is explained that the initial skeleton key point group sequence and the auxiliary skeleton key point group sequence do not match.

[0084] According to an embodiment of the present disclosure, when determining the match between the initial skeleton key point group sequence and the auxiliary skeleton key point group sequence, it is indicated that the accuracy of the initial skeleton key point group sequence meets a predetermined condition, and the initial skeleton key point group sequence can be used as the skeleton key point sequence. However, it is not limited to this. It is also possible to add and sum the initial skeleton key point group sequence and the auxiliary skeleton key point group sequence according to the temporal relationship to obtain the skeleton key point sequence.

[0085] According to an embodiment of the present disclosure, when it is determined that there is a mismatch between the initial skeleton key point group sequence and the auxiliary skeleton key point group sequence, the initial skeleton key point group sequence can be updated based on the image sequence and the depth data sequence to obtain the skeleton key point group sequence.

[0086] According to an embodiment of the present disclosure, based on an image sequence and a depth data sequence, an initial skeleton key point group sequence is updated to obtain a skeleton key point group sequence, which may include: determining a target initial skeleton key point that does not match an auxiliary skeleton key point group sequence from the initial skeleton key point group sequence, and based on the image sequence and the depth data sequence, determining the texture node data of each of the multiple texture nodes associated with the target initial skeleton key point in the real world, such as the three-dimensional coordinates of the texture node in the world coordinate system. Based on the multiple texture node data, a fitting skeleton key point is determined. The target initial skeleton key point is replaced with the fitting skeleton key point to complete the update.

[0087] For example, the target initial skeleton key point is, for example, an initial elbow key point, and the three-dimensional key point coordinates between the initial elbow key point and the auxiliary elbow key point do not match. Texture node data, such as texture node three-dimensional coordinates, of texture nodes around the elbow joint point associated with the initial elbow key point can be determined based on the image sequence and the depth data sequence. The fitting skeleton key point can be determined based on multiple texture node data. For example, based on multiple texture node data around the initial elbow key point, the three-dimensional key point coordinates of the fitting elbow key point are determined by weighted summation.

[0088] According to an embodiment of the present disclosure, an auxiliary skeleton key point group sequence is used to determine the initial skeleton key point group sequence, and data obtained by different processing methods can be used to corroborate each other to improve the accuracy of the skeleton key point group sequence. In the case where the initial skeleton key point group sequence is determined to have a large error with the actual result, the image sequence and the depth data sequence can be used to update the initial skeleton key point group sequence, thereby improving the effectiveness of the skeleton key point group sequence. In addition, it is also possible to use a variety of data of different types to improve data utilization. The action recognition method provided by the embodiment of the present disclosure is applied to the training or competition scenes of athletes, which will enhance the value of the action recognition results and the skeleton key point group sequence as a reference result.

[0089] According to an embodiment of the present disclosure, when executing Figure 2 The operation S230 shown is to perform action recognition on the target object based on the image feature sequence, the skeleton key point feature sequence and the skeleton angle feature sequence. After the action recognition result of the target object is determined, the action recognition result may also include the following operations.

[0090] For example, based on the event data sequence and the image sequence, the motion trajectory of the associated object is determined. The motion made by the associated object is generated based on the motion of the target object. Based on the motion trajectory, the action mode of the associated object is determined. When it is determined that the action mode matches the action recognition result, the action recognition result is used as the target action recognition result.

[0091] According to an embodiment of the present disclosure, the associated object may be an object associated with the target object. In a ball sports scene, the target object may be a player, and the associated object may be a ball, such as a badminton, a football, or a basketball.

[0092] According to an embodiment of the present disclosure, determining the motion trajectory of the associated object based on the event data sequence and the image sequence may include: using the event data sequence to filter out background information in the image sequence to obtain a target image sequence containing the associated object and the target object. Object detection may be performed on the target image sequence to obtain a detection frame result sequence of the associated object. Based on the detection frame result sequence, the motion trajectory of the associated object is determined.

[0093] According to an embodiment of the present disclosure, the motion mode of the associated object can be determined based on the motion trajectory. However, it is not limited to this. The moving speed can also be determined based on the motion trajectory and the motion duration. The motion mode of the associated object is determined based on the motion trajectory and the moving speed. The motion made by the associated object is generated based on the motion of the target object. There is an influence of force between the two. The accuracy of the motion recognition result can be assisted in determining based on the motion mode. When it is determined that the motion mode matches the motion recognition result, the motion recognition result is used as the target motion recognition result. When it is determined that the motion mode does not match the motion recognition result, the motion recognition method can be re-executed to improve the accuracy of the motion recognition result.

[0094] According to the embodiments of the present disclosure, the motion recognition method provided by the embodiments of the present disclosure can also determine the motion trajectory, action pattern and other results of the associated object, and combine them with the target action recognition results of the target object to improve the richness of the recognition results.

[0095] According to an embodiment of the present disclosure, when executing Figure 2 In operation S210 shown, before determining the skeleton key point group sequence of the target object based on the image sequence, the action recognition method may further include: acquiring the image sequence.

[0096] For example, the video is subjected to frame extraction processing to obtain continuous video frames. A plurality of target video frames are determined from the continuous video frames according to a predetermined frame extraction method. An image sequence is determined based on the plurality of target video frames.

[0097] According to an embodiment of the present disclosure, the predetermined frame extraction method may refer to extracting a plurality of target video frames from continuous video frames at a certain time interval.

[0098] According to the embodiments of the present disclosure, by determining the image sequence in the above manner, it is possible to improve processing efficiency while ensuring processing accuracy and avoiding missing key actions.

[0099] According to an embodiment of the present disclosure, the image sequence may include multiple image sequences, and the multiple image sequences are temporally related sequences to each other.

[0100] For example, a predetermined frame extraction method is used to determine the target video frame at time t1, the target video frame at time t2, and the target video frame at time t3 from the continuous video frames as image sequence 1. The target video frame at time t3, the target video frame at time t4, and the target video frame at time t5 are determined from the continuous video frames as image sequence 2. Multiple image sequences are obtained. There are video frames with overlapping time periods between the multiple image sequences.

[0101] According to an embodiment of the present disclosure, in Figure 2After operation S230 shown, the action recognition method provided by the embodiment of the present disclosure may further include the following operations.

[0102] For example, the action recognition results of the plurality of image sequences are determined to obtain a plurality of action recognition results. When it is determined that the plurality of action recognition results are all used to represent the same action, a target video segment is determined from the video based on the plurality of image sequences.

[0103] According to an embodiment of the present disclosure, when it is determined that multiple action recognition results are used to represent the same action, a target video segment is determined from the video based on multiple image sequences. When it is determined that multiple action recognition results are used to represent different actions, a target image sequence can be determined from multiple image sequences. Based on the target image sequence, a target video segment is determined from the video frame. In this way, key video segments can be archived for subsequent viewing. This reduces resource consumption while avoiding data loss.

[0104] According to the embodiments of the present disclosure, the action recognition method provided by the embodiments of the present disclosure is applied to the badminton sports scene, and multiple cameras set with multiple perspectives, such as event cameras, depth cameras, binocular cameras, etc., can be used to obtain image sequences, depth data sequences, and event data sequences. The action recognition method is operated and processed on the image sequences, depth data sequences, and event data sequences to realize the dynamic tracking of athletes in the badminton court, the recognition of actions such as steps, swings, and smashes, as well as the determination of results such as badminton pattern recognition, badminton landing points, and ball speed detection. Quantitative analysis data is provided to realize multi-dimensional analysis.

[0105] By utilizing the action recognition method provided in the embodiment of the present disclosure, invalid actions can be automatically eliminated while saving manpower, and key actions can be classified and archived for easy review.

[0106] Figure 5 The block diagram of the motion recognition device according to the embodiment of the present disclosure is schematically shown.

[0107] like Figure 5 As shown, the action recognition device 500 includes: a key point determination module 510 , an angle determination module 520 and an action recognition module 530 .

[0108] The key point determination module 510 is used to determine a skeleton key point group sequence of the target object based on the image sequence.

[0109] The angle determination module 520 is used to determine the bone angle sequence of the target object based on the bone key point group sequence.

[0110] The action recognition module 530 is used to perform action recognition on the target object based on the image feature sequence, the skeleton key point feature sequence and the skeleton angle feature sequence, and determine the action recognition result of the target object. The image feature sequence is generated based on the image sequence, the skeleton key point feature sequence is generated based on the skeleton key point group sequence, and the skeleton angle feature sequence is generated based on the skeleton angle sequence.

[0111] According to an embodiment of the present disclosure, the action recognition module includes: a matching submodule, a splicing submodule and a recognition submodule.

[0112] The matching submodule is used to match each image feature in the image feature sequence.

[0113] According to the time sequence relationship, the target bone key point features that match the image features are determined from the bone key point feature sequence, and the target bone angle features that match the image features are determined from the bone angle feature sequence.

[0114] The splicing submodule is used to generate target comprehensive features based on image features, target skeleton key point features and target skeleton angle features to obtain a target comprehensive feature sequence.

[0115] The recognition submodule is used to perform action recognition on the target object based on the target comprehensive feature sequence and determine the action recognition result of the target object.

[0116] According to an embodiment of the present disclosure, the key point determination module includes: a first key point determination submodule.

[0117] The first key point determination submodule is used to determine a skeleton key point group sequence of the target object based on an image sequence and an event data sequence with the same time sequence as the image sequence.

[0118] According to an embodiment of the present disclosure, the key point determination module includes: a second key point determination submodule, a third key point determination submodule and a fourth key point determination submodule.

[0119] The second key point determination submodule is used to obtain an initial skeleton key point group sequence of the target object based on the image sequence and the event data sequence with the same time sequence as the image sequence.

[0120] The third key point determination submodule is used to determine the auxiliary skeleton key point group sequence of the target object based on the image sequence and the depth data sequence.

[0121] The fourth key point determination submodule is used to determine the skeleton key point group sequence based on the initial skeleton key point group sequence and the auxiliary skeleton key point group sequence.

[0122] According to an embodiment of the present disclosure, the fourth key point determination submodule includes: a first key point determination unit and a second key point determination unit.

[0123] The first key point determination unit is used to determine the skeleton key point group sequence based on the initial skeleton key point group sequence when it is determined that the initial skeleton key point group sequence matches the auxiliary skeleton key point group sequence.

[0124] The second key point determination unit is used to update the initial skeleton key point group sequence based on the image sequence and the depth data sequence to obtain the skeleton key point group sequence when it is determined that there is a mismatch between the initial skeleton key point group sequence and the auxiliary skeleton key point group sequence.

[0125] According to an embodiment of the present disclosure, the action recognition device further includes: an image feature extraction module, an angle feature extraction module and a key point feature extraction module.

[0126] The image feature extraction module is used to extract features from an image sequence and an event data sequence with the same time sequence as the image sequence to obtain an image feature sequence.

[0127] The angle feature extraction module is used to extract features from the bone angle sequence to obtain the bone angle feature sequence.

[0128] The key point feature extraction module is used to extract features from the skeleton key point group sequence to obtain the skeleton key point feature sequence.

[0129] According to an embodiment of the present disclosure, the action recognition device further includes: a trajectory determination module, a pattern recognition module and a first result determination module.

[0130] The trajectory determination module is used to determine the motion trajectory of the associated object based on the event data sequence and the image sequence. The motion of the associated object is generated based on the motion of the target object.

[0131] The pattern recognition module is used to determine the action pattern of the associated object based on the motion trajectory.

[0132] The first result determination module is used to use the action recognition result as a target action recognition result when it is determined that the action pattern matches the action recognition result.

[0133] According to an embodiment of the present disclosure, the action recognition device further includes: a frame splitting module, a frame extraction module and an image determination module.

[0134] The frame decomposition module is used to decomposition the video to obtain continuous video frames.

[0135] The frame extraction module is used to determine a plurality of target video frames from continuous video frames according to a predetermined frame extraction method.

[0136] The image determination module is used to determine an image sequence based on multiple target video frames.

[0137] According to an embodiment of the present disclosure, the image sequence includes a plurality of.

[0138] According to an embodiment of the present disclosure, the action recognition device further includes: a second result determination module and a segment determination module.

[0139] The second result determination module is used to determine the action recognition results of each of the multiple image sequences to obtain multiple action recognition results.

[0140] The segment determination module is used to determine a target video segment from a video based on multiple image sequences when it is determined that multiple action recognition results are all used to represent the same action.

[0141] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0142] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method as in the embodiment of the present disclosure.

[0143] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute a method as described in an embodiment of the present disclosure.

[0144] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the method according to the embodiment of the present disclosure is implemented.

[0145] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0146] like Figure 6As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0147] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0148] The computing unit 601 can be various general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the action recognition method. For example, in some embodiments, the action recognition method can be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the action recognition method described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the action recognition method by any other appropriate means (e.g., by means of firmware).

[0149] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0150] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0151] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0152] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0153] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0154] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0155] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0156] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for motion recognition, comprising: Based on the image sequence, determining a skeleton key point group sequence of the target object; Based on the skeleton key point group sequence, determining the skeleton angle sequence of the target object; Based on the image feature sequence, the skeleton key point feature sequence and the skeleton angle feature sequence, the target object is subjected to action recognition to determine the action recognition result of the target object, wherein the image feature sequence is generated based on the image sequence, the skeleton key point feature sequence is generated based on the skeleton key point group sequence, and the skeleton angle feature sequence is generated based on the skeleton angle sequence; Determining a motion trajectory of an associated object based on the event data sequence and the image sequence, wherein the associated object is associated with and different from the target object, and the motion made by the associated object is generated based on the motion of the target object; Based on the motion trajectory, determining the motion mode of the associated object; and In the case where it is determined that the action pattern matches the action recognition result, taking the action recognition result as a target action recognition result, Wherein, determining the motion trajectory of the associated object based on the event data sequence and the image sequence includes: Using the event data sequence, background information in the image sequence is filtered out to obtain a target image sequence including associated objects and target objects; Performing target detection on the target image sequence to obtain a detection frame result sequence of associated objects; and Based on the detection box result sequence, the motion trajectory of the associated object is determined.

2. The method according to claim 1, wherein: The step of performing action recognition on the target object based on the image feature sequence, the skeleton key point feature sequence and the skeleton angle feature sequence to determine the action recognition result of the target object includes: For each image feature in the image feature sequence, According to the time sequence relationship, determine the target bone key point feature that matches the image feature from the bone key point feature sequence, and determine the target bone angle feature that matches the image feature from the bone angle feature sequence; Generate a target comprehensive feature based on the image feature, the target bone key point feature and the target bone angle feature to obtain a target comprehensive feature sequence; and Based on the target comprehensive feature sequence, action recognition is performed on the target object to determine the action recognition result of the target object.

3. The method according to claim 1 or 2, wherein: The method of determining a skeleton key point group sequence of the target object based on the image sequence includes: Based on the image sequence and an event data sequence having the same time sequence as the image sequence, a skeleton key point group sequence of the target object is determined.

4. The method according to claim 1 or 2, wherein: The method of determining a skeleton key point group sequence of the target object based on the image sequence includes: Based on the image sequence and an event data sequence having the same time sequence as the image sequence, obtaining an initial skeleton key point group sequence of the target object; Determining an auxiliary skeleton key point group sequence of the target object based on the image sequence and the depth data sequence; and The skeleton key point group sequence is determined based on the initial skeleton key point group sequence and the auxiliary skeleton key point group sequence.

5. The method according to claim 4, wherein: The step of determining the skeleton key point group sequence based on the initial skeleton key point group sequence and the auxiliary skeleton key point group sequence comprises: In the case where it is determined that the initial skeleton key point group sequence matches the auxiliary skeleton key point group sequence, determining the skeleton key point group sequence based on the initial skeleton key point group sequence; and When it is determined that the initial skeleton key point group sequence and the auxiliary skeleton key point group sequence do not match each other, the initial skeleton key point group sequence is updated based on the image sequence and the depth data sequence to obtain the skeleton key point group sequence.

6. The method according to claim 1 or 2, further comprising: Extracting features from the image sequence and an event data sequence having the same time sequence as the image sequence to obtain the image feature sequence; Performing feature extraction on the bone angle sequence to obtain the bone angle feature sequence; as well as Feature extraction is performed on the skeleton key point group sequence to obtain the skeleton key point feature sequence.

7. The method according to claim 1 or 2, further comprising: Perform frame splitting processing on the video to obtain continuous video frames; Determining a plurality of target video frames from the continuous video frames according to a predetermined frame extraction method; as well as The image sequence is determined based on the plurality of target video frames.

8. The method according to claim 7, wherein: The image sequence includes a plurality of; The method further comprises: Determining the action recognition results of each of the plurality of image sequences to obtain a plurality of action recognition results; and When it is determined that the multiple action recognition results are all used to represent the same action, a target video segment is determined from the video based on the multiple image sequences.

9. A motion recognition device, comprising: A key point determination module, used for determining a skeleton key point group sequence of a target object based on an image sequence; An angle determination module, used for determining a skeleton angle sequence of the target object based on the skeleton key point group sequence; An action recognition module is used to perform action recognition on the target object based on an image feature sequence, a skeleton key point feature sequence and a skeleton angle feature sequence, and determine an action recognition result of the target object, wherein the image feature sequence is generated based on the image sequence, the skeleton key point feature sequence is generated based on the skeleton key point group sequence, and the skeleton angle feature sequence is generated based on the skeleton angle sequence; a trajectory determination module, configured to determine a motion trajectory of an associated object based on the event data sequence and the image sequence, wherein the associated object is associated with and different from the target object, and the motion made by the associated object is generated based on the motion of the target object; a pattern recognition module, configured to determine a motion pattern of the associated object based on the motion trajectory; and A first result determination module is used to use the action recognition result as a target action recognition result when it is determined that the action pattern matches the action recognition result. Wherein, the trajectory determination module is configured as follows: Using the event data sequence, background information in the image sequence is filtered out to obtain a target image sequence including associated objects and target objects; Performing target detection on the target image sequence to obtain a detection frame result sequence of associated objects; and Based on the detection box result sequence, the motion trajectory of the associated object is determined.

10. The device according to claim 9, wherein: The action recognition module comprises: The matching submodule is used to match each image feature in the image feature sequence. According to the time sequence relationship, determine the target bone key point feature that matches the image feature from the bone key point feature sequence, and determine the target bone angle feature that matches the image feature from the bone angle feature sequence; A splicing submodule, used to generate a target comprehensive feature based on the image feature, the target skeleton key point feature and the target skeleton angle feature to obtain a target comprehensive feature sequence; and The recognition submodule is used to perform action recognition on the target object based on the target comprehensive feature sequence and determine the action recognition result of the target object.

11. The device according to claim 9 or 10, wherein: The key point determination module includes: The first key point determination submodule is used to determine the skeleton key point group sequence of the target object based on the image sequence and the event data sequence with the same time sequence as the image sequence.

12. The device according to claim 9 or 10, wherein: The key point determination module includes: A second key point determination submodule is used to obtain an initial skeleton key point group sequence of the target object based on the image sequence and an event data sequence with the same time sequence as the image sequence; A third key point determination submodule is used to determine an auxiliary skeleton key point group sequence of the target object based on the image sequence and the depth data sequence; and The fourth key point determination submodule is used to determine the skeleton key point group sequence based on the initial skeleton key point group sequence and the auxiliary skeleton key point group sequence.

13. The device according to claim 12, wherein: The fourth key point determination submodule includes: A first key point determination unit is used to determine the skeleton key point group sequence based on the initial skeleton key point group sequence when it is determined that the initial skeleton key point group sequence matches the auxiliary skeleton key point group sequence; and The second key point determination unit is used to update the initial skeleton key point group sequence based on the image sequence and the depth data sequence to obtain the skeleton key point group sequence when it is determined that there is a mismatch between the initial skeleton key point group sequence and the auxiliary skeleton key point group sequence.

14. The device according to claim 9 or 10, further comprising: An image feature extraction module, used for extracting features from the image sequence and an event data sequence having the same time sequence as the image sequence, to obtain the image feature sequence; An angle feature extraction module, used for extracting features from the bone angle sequence to obtain the bone angle feature sequence; as well as The key point feature extraction module is used to extract features from the skeleton key point group sequence to obtain the skeleton key point feature sequence.

15. The device according to claim 9 or 10, further comprising: A frame splitting module is used to split the video into frames to obtain continuous video frames; A frame extraction module, used for determining a plurality of target video frames from the continuous video frames according to a predetermined frame extraction method; as well as The image determination module is used to determine the image sequence based on the multiple target video frames.

16. The device according to claim 15, wherein: The image sequence includes a plurality of; The device also includes: A second result determination module is used to determine the action recognition results of each of the plurality of image sequences to obtain a plurality of action recognition results; and The segment determination module is used to determine a target video segment from the video based on the multiple image sequences when it is determined that the multiple action recognition results are all used to represent the same action.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 8.

19. A computer program product, comprising a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 8.

Images