Training method, device, equipment, storage medium and product of action recognition model

Abstract

The present application provides a training method, device, equipment, storage medium and product for an action recognition model, which belongs to the field of artificial intelligence technology and can be applied to the processing scenario of multimedia resources. The method includes: obtaining multiple sample videos; segmenting the target object in the sample video to obtain multiple action parts corresponding to the sample video, and the multiple action parts are the action parts of the target object; determining the relative position vectors between the multiple action parts, and obtaining the relative position vector corresponding to the sample video, and the relative position vector is used to represent the positional relationship between the multiple action parts; clustering the multiple sample videos based on the relative position vectors corresponding to the multiple sample videos, and obtaining multiple video clusters, the video cluster includes at least one sample video, and the actions of the sample videos in the same video cluster are the same; based on the multiple video clusters, training the action recognition model. This method improves the training efficiency of the action recognition model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method, device, equipment, storage medium and product for a motion recognition model. Background Art

[0002] With the development of artificial intelligence technology, the application of action recognition models is becoming more and more extensive; for example, action recognition models can be applied in intelligent education scenarios; through action recognition models, the actions in the teacher's teaching video can be identified, and then the teacher can be evaluated by artificial intelligence (AI) based on the identified action categories.

[0003] In the related art, supervised deep learning algorithms are used to train action recognition models, that is, the action recognition models are trained through labeled sample videos. The action labels in the sample videos are annotated by humans. Since the amount of sample video data is large, it takes a lot of time to manually annotate the action labels of the sample videos, thereby reducing the training efficiency of the action recognition model. Summary of the invention

[0004] The embodiments of the present application provide a method, apparatus, device, storage medium and product for training an action recognition model, which can improve the training efficiency of the action recognition model. The technical solution is as follows:

[0005] On the one hand, a method for training an action recognition model is provided, the method comprising:

[0006] Get multiple sample videos;

[0007] Segmenting the target object in the sample video to obtain a plurality of action parts corresponding to the sample video, wherein the plurality of action parts are action parts of the target object, and the target object is an object performing an action;

[0008] Determine the relative position vectors between the multiple action parts to obtain the relative position vector corresponding to the sample video, wherein the relative position vector is used to represent the positional relationship between the multiple action parts;

[0009] Clustering the plurality of sample videos based on the relative position vectors respectively corresponding to the plurality of sample videos to obtain a plurality of first video clusters, wherein the first video cluster includes at least one sample video, and the actions of the sample videos in the same first video cluster are the same;

[0010] Based on the multiple first video clusters, an action recognition model is trained.

[0011] On the other hand, a training device for an action recognition model is provided, the device comprising:

[0012] A first acquisition module, used to acquire multiple sample videos;

[0013] a segmentation module, configured to segment the target object in the sample video to obtain a plurality of action parts corresponding to the sample video, wherein the plurality of action parts are action parts of the target object, and the target object is an object performing an action;

[0014] A determination module, used to determine the relative position vectors between the multiple action parts, and obtain the relative position vector corresponding to the sample video, wherein the relative position vector is used to represent the positional relationship between the multiple action parts;

[0015] A clustering module, configured to cluster the plurality of sample videos based on relative position vectors respectively corresponding to the plurality of sample videos to obtain a plurality of first video clusters, wherein the first video cluster includes at least one sample video, and the actions of the sample videos in the same first video cluster are the same;

[0016] A training module is used to train an action recognition model based on the multiple first video clusters.

[0017] In some embodiments, the determination module is used to determine a first action part and multiple second action parts among the multiple action parts based on the part information of the multiple action parts corresponding to the sample video, wherein the first action part is the benchmark action part of the target object; and determine the relative position vectors of the multiple second action parts relative to the first action part to obtain the relative position vector corresponding to the sample video.

[0018] In some embodiments, the determination module is used to respectively determine a first center position and a second center position, wherein the first center position is the center position of the first action part, and the second center position is the center position of the second action part; the vector between the first center position and the second center position is determined as the relative position vector of the second action part relative to the first action part; the relative position vectors of the multiple second action parts relative to the first action part form a relative position vector corresponding to the sample video.

[0019] In some embodiments, the determination module is used to respectively determine a first center position and multiple boundary positions, the first center position being the center position of the first action part, and the multiple boundary positions being the boundary positions of the second action part; determine the vectors between the first center position and the multiple boundary positions respectively; concatenate the vectors between the first center position and the multiple boundary positions respectively to obtain a relative position vector of the second action part relative to the first action part; and combine the relative position vectors of the multiple second action parts relative to the first action part to form a relative position vector corresponding to the sample video.

[0020] In some embodiments, the sample video includes multiple video frames, and the relative position vectors corresponding to the sample video include relative position vectors corresponding to the multiple video frames respectively; the clustering module is used to splice the multiple relative position vectors corresponding to the same video frame to obtain a first relative position vector corresponding to the same video frame; splice the first relative position vectors corresponding to multiple video frames of the same sample video to obtain a second relative position vector corresponding to the sample video; based on the second relative position vectors corresponding to the multiple sample videos respectively, cluster the multiple sample videos to obtain the multiple first video clusters.

[0021] In some embodiments, the clustering module is used to determine the distance between any two second relative position vectors based on the second relative position vectors respectively corresponding to the multiple sample videos; when the distance is not greater than a preset distance, the two sample videos corresponding to the two second relative position vectors are aggregated into the same first video cluster.

[0022] In some embodiments, the apparatus further comprises:

[0023] A second acquisition module is used to acquire a target video, where the target video is a video of a to-be-identified action category;

[0024] The input-output module is used to input the target video into the action recognition model, output the action category of the target video and the relative position vectors of multiple action parts of the target object in the target video, and the relative position vectors are used to explain why the target video is identified as the action category.

[0025] In some embodiments, the training module is used to determine a first action part and multiple second action parts among the multiple action parts based on the part information of the multiple action parts, wherein the first action part is a baseline action part of the target object; determine a target action part from the multiple second action parts, wherein the target action part contains the least pixels; cluster the multiple sample videos based on the first action parts and the target action parts to obtain multiple second video clusters, wherein the second video cluster includes at least one sample video, and the actions of the sample videos in the same second video cluster are the same; and train the action recognition model based on the multiple first video clusters and the multiple second video clusters.

[0026] In some embodiments, the training module is used to determine a first similarity and a second similarity between any two sample videos, where the first similarity is the similarity between first action parts of the two sample videos, and the second similarity is the similarity between target action parts of the two sample videos; based on the first similarity and the second similarity, a third similarity between the two sample videos is determined; when the third similarity is greater than a preset similarity, the two sample videos are aggregated into the same second video cluster.

[0027] In some embodiments, the segmentation module is used to extract multiple video frames from the sample video; segment the target object included in the video frame to obtain multiple action parts corresponding to the video frame; and combine the action parts corresponding to the multiple video frames into multiple action parts corresponding to the sample video.

[0028] On the other hand, a computer device is provided, which includes a processor and a memory, wherein the memory is used to store at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the training method of the action recognition model in the embodiment of the present application.

[0029] On the other hand, a computer-readable storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement a training method for an action recognition model as in an embodiment of the present application.

[0030] On the other hand, a computer program product or a computer program is provided, which includes a computer program code, which is stored in a computer-readable storage medium, and a processor of a computer device reads the computer program code from the computer-readable storage medium, and the processor executes the computer program code, so that the computer device executes the training method of the action recognition model provided in various optional implementations of the above-mentioned aspects.

[0031] An embodiment of the present application provides a method for training an action recognition model, which segments an object in a sample video and obtains multiple action parts of the object. Since the relative position vectors between the multiple action parts can effectively represent the action performed by the object, the multiple sample videos are clustered based on the relative position information corresponding to the multiple sample videos, and multiple video clusters with accurate classification can be obtained. In this way, the action recognition model is trained based on the sample videos in the multiple video clusters obtained by clustering, which avoids the process of labeling action labels for multiple sample videos in turn, thereby improving the training efficiency of the action recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 It is a schematic diagram of an implementation environment of a method for training an action recognition model provided in an embodiment of the present application;

[0034] Figure 2 is a flow chart of a method for training an action recognition model provided in an embodiment of the present application;

[0035] Figure 3 is a flow chart of a method for training an action recognition model provided in an embodiment of the present application;

[0036] Figure 4 is an image schematic diagram of a video frame provided in an embodiment of the present application;

[0037] Figure 5 is an image schematic diagram of another video frame provided in an embodiment of the present application;

[0038] Figure 6 is a schematic diagram of a relative position vector provided in an embodiment of the present application;

[0039] Figure 7 is a flow chart of a method for training an action recognition model provided in an embodiment of the present application;

[0040] Figure 8 is a flow chart of a method for training an action recognition model provided in an embodiment of the present application;

[0041] Fig. 9 This is a schematic diagram of an application of an action recognition model provided by an embodiment of the present application in an intelligent education scenario;

[0042] Fig.10 It is a block diagram of a training device for an action recognition model provided in an embodiment of the present application;

[0043] Fig.11 is a block diagram of a terminal provided in an embodiment of the present application;

[0044] Fig.12 This is a block diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0046] In this application, the terms "first", "second", etc. are used to distinguish identical or similar items with substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there any limitation on quantity and execution order.

[0047] In the present application, the term "at least one" means one or more, and the term "plurality" means two or more.

[0048] It should be noted that in the implementation methods of this application, data related to sample videos, part information, etc. are involved. When the above embodiments of this application are applied to products or technologies, relevant licenses or consents need to be obtained, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0049] The following explains the terms used in this application.

[0050] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0051] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0052] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0053] The following is an introduction to the implementation environment involved in this application:

[0054] The training method of the action recognition model provided in the embodiment of the present application can be executed by a computer device. In some embodiments, the computer device is a terminal 101 or a server 102. The following first takes the computer device as a server 102 as an example to introduce the implementation environment diagram of the training method of the action recognition model provided in the embodiment of the present application. Figure 1 The implementation environment includes a terminal 101 and a server 102. The terminal 101 and the server 102 can be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0055] In some embodiments, the terminal 101 is a smart phone, a tablet computer, a laptop computer, a desktop computer, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, etc., but is not limited thereto. In some embodiments, the server 102 is an independent server and can also be a server cluster or a distributed system of multiple physical server projects. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server 102 is used to provide background services for applications installed on the terminal 101. In some embodiments, the server 102 is mainly responsible for computing work, and the terminal 101 is responsible for secondary computing work; or, the server 102 is responsible for secondary computing services, and the terminal 101 is responsible for the main computing work; or, the server 102 and the terminal 101 use a distributed computing architecture for collaborative computing.

[0056] In some embodiments, a computer device is used to obtain a sample video, perform model training based on the sample video, and obtain a motion recognition model. Optionally, the motion recognition model is widely used in scenarios such as smart education and social entertainment. For example, the motion recognition model is applied to a smart education scenario to identify the actions in a teacher's lecture video, so that the teacher can be evaluated by AI based on the identified action category. For example, the motion recognition model is applied to a social entertainment scenario to identify the actions in a social entertainment video, so that the video can be determined to include restrictive content based on the identified action category. In some embodiments, the motion recognition model is also used in business scenarios such as video understanding, wonderful video highlights, video subtitle generation, and motion recognition, as an underlying guarantee for video analysis tasks, to improve the efficiency of video analysis processing.

[0057] Figure 2 is a flowchart of a method for training an action recognition model according to an embodiment of the present application, see Figure 2 In the present application, the training method of the action recognition model includes the following steps:

[0058] 201. Obtain multiple sample videos.

[0059] 202. Segment the target object in the sample video to obtain multiple action parts corresponding to the sample video, where the multiple action parts are action parts of the target object, and the target object is an object performing an action.

[0060] 203. Determine a relative position vector between the multiple action parts, and obtain a relative position vector corresponding to the sample video, where the relative position vector is used to represent the positional relationship between the multiple action parts.

[0061] 204. Cluster the multiple sample videos based on relative position vectors of action parts corresponding to the multiple sample videos to obtain multiple first video clusters, wherein the first video cluster includes at least one sample video, and the actions of the sample videos in the same first video cluster are the same.

[0062] 205. Train an action recognition model based on the multiple first video clusters.

[0063] In some embodiments, determining the relative position vectors between the plurality of action parts to obtain the relative position vectors corresponding to the sample video includes:

[0064] Based on the part information of the multiple action parts corresponding to the sample video, determine a first action part and multiple second action parts among the multiple action parts, the first action part being a reference action part of the target object;

[0065] Determine the relative position vectors of the plurality of second motion parts relative to the first motion part, and obtain the relative position vector corresponding to the sample video.

[0066] In some embodiments, determining the relative position vectors of the plurality of second motion parts relative to the first motion part to obtain the relative position vector corresponding to the sample video includes:

[0067] Determine a first center position and a second center position respectively, the first center position is the center position of the first action part, and the second center position is the center position of the second action part;

[0068] Determine the vector between the first center position and the second center position as the relative position vector of the second action part relative to the first action part;

[0069] The relative position vectors of the plurality of second motion parts relative to the first motion part are combined into a relative position vector corresponding to the sample video.

[0070] In some embodiments, determining the relative position vectors of the plurality of second motion parts relative to the first motion part to obtain the relative position vector corresponding to the sample video includes:

[0071] Determine a first center position and a plurality of boundary positions respectively, the first center position being the center position of the first action part, and the plurality of boundary positions being the boundary positions of the second action part respectively;

[0072] Determine vectors between the first center position and the plurality of boundary positions respectively;

[0073] splicing the vectors between the first center position and the plurality of boundary positions respectively to obtain a relative position vector of the second action part relative to the first action part;

[0074] The relative position vectors of the plurality of second motion parts relative to the first motion part are combined into a relative position vector corresponding to the sample video.

[0075] In some embodiments, the sample video includes a plurality of video frames, and the relative position vector corresponding to the sample video includes relative position vectors corresponding to the plurality of video frames respectively;

[0076] Based on the relative position vectors respectively corresponding to the multiple sample videos, the multiple sample videos are clustered to obtain multiple first video clusters, including:

[0077] splicing multiple relative position vectors corresponding to the same video frame to obtain a first relative position vector corresponding to the same video frame;

[0078] splicing first relative position vectors corresponding to multiple video frames of the same sample video to obtain a second relative position vector corresponding to the sample video;

[0079] Based on the second relative position vectors respectively corresponding to the multiple sample videos, the multiple sample videos are clustered to obtain multiple first video clusters.

[0080] In some embodiments, clustering the plurality of sample videos based on the second relative position vectors respectively corresponding to the plurality of sample videos to obtain a plurality of first video clusters includes:

[0081] Determine the distance between any two second relative position vectors based on the second relative position vectors respectively corresponding to the plurality of sample videos;

[0082] When the distance is not greater than the preset distance, the two sample videos corresponding to the two second relative position vectors are aggregated into the same first video cluster.

[0083] In some embodiments, training an action recognition model based on a plurality of first video clusters includes:

[0084] Based on the part information of the plurality of action parts, determining a first action part and a plurality of second action parts among the plurality of action parts, the first action part being a reference action part of the target object;

[0085] Determine a target action part from the plurality of second action parts, wherein the target action part contains the least number of pixels;

[0086] Based on the first action parts and the target action parts of the multiple sample videos, the multiple sample videos are clustered to obtain multiple second video clusters, wherein the second video cluster includes at least one sample video, and the actions of the sample videos in the same second video cluster are the same;

[0087] Based on the plurality of first video clusters and the plurality of second video clusters, an action recognition model is trained.

[0088] In some embodiments, based on the first action parts and the target action parts of the multiple sample videos, the multiple sample videos are clustered to obtain multiple second video clusters, including:

[0089] Determine a first similarity and a second similarity between any two sample videos, where the first similarity is the similarity between first action parts of the two sample videos, and the second similarity is the similarity between target action parts of the two sample videos;

[0090] Based on the first similarity and the second similarity, determining a third similarity between the two sample videos;

[0091] When the third similarity is greater than the preset similarity, the two sample videos are aggregated into the same second video cluster.

[0092] In some embodiments, the target object in the sample video is segmented to obtain multiple action parts corresponding to the sample video, including:

[0093] Extracting a plurality of video frames from a sample video;

[0094] Segmenting the target object included in the video frame to obtain multiple action parts corresponding to the video frame;

[0095] The action parts corresponding to the multiple video frames are respectively combined into multiple action parts corresponding to the sample video.

[0096] In some embodiments, the method further comprises:

[0097] Obtain a target video, which is a video of the action category to be identified;

[0098] The target video is input into the action recognition model, and the action category of the target video and the relative position vectors of multiple action parts of the target object in the target video are output. The relative position vector is used to explain why the target video is identified as an action category.

[0099] An embodiment of the present application provides a method for training an action recognition model, which segments an object in a sample video to obtain multiple action parts of the object. Since the relative position vectors between the multiple action parts can effectively represent the action performed by the object, the multiple sample videos are clustered based on the relative position vectors corresponding to the multiple sample videos, and multiple video clusters with accurate classification can be obtained. In this way, the action recognition model is trained based on the sample videos in the multiple video clusters obtained by clustering, which avoids the process of labeling action labels for multiple sample videos in turn, thereby improving the training efficiency of the action recognition model.

[0100] Figure 3 1 is a flow chart of a method for training an action recognition model provided in an embodiment of the present application. The method for training an action recognition model comprises the following steps:

[0101] 301. A computer device obtains multiple sample videos.

[0102] The sample video includes multiple video frames, and the video duration of the sample video can be set and changed as needed; optionally, the video duration of the sample video is 1 second.

[0103] In one implementation, a computer device directly obtains a sample video. In another implementation, a computer device obtains at least one initial sample video, and the video duration of the initial sample video is greater than the video duration of the sample video; the computer device splits the initial sample video to obtain multiple sample videos. Optionally, the splitting time interval of the initial sample video can be set and changed as needed; for example, the computer device determines the splitting time interval according to the video type of the initial sample video, splits the initial sample video into m segments of sample videos, and optionally obtains multiple sample videos with a video duration of 1 second. In this implementation, since the initial sample video has a longer duration and contains more actions, the initial sample video is split into multiple sample videos with shorter durations, so that the action changes in the sample video are smaller, and it is convenient to process the sample video to improve the accuracy of action recognition in the sample video.

[0104] In some embodiments, the sample video includes videos in at least one of the smart education scene and the social entertainment scene. Optionally, if the sample video is a video in the smart education scene, the sample video includes positive sample videos in which the teacher in the video makes actions such as writing on the blackboard and explaining, and negative sample videos in which the teacher does not make any actions. Optionally, if the sample video is a video in the social entertainment scene, the sample video includes positive sample videos in which the object in the video makes restrictive actions, and negative sample videos in which the object does not make any restrictive actions.

[0105] 302. The computer device segments the target object in the sample video to obtain multiple action parts corresponding to the sample video.

[0106] The multiple action parts are the action parts of the target object, and the target object is the object that performs the action. The multiple action parts include at least one of the four limb parts and the trunk part of the target object; optionally, the multiple action parts include the trunk part, the left upper limb part, the left lower limb part, the right upper limb part and the right lower limb part.

[0107] In some embodiments, the computer device segments the target object included in any video frame of the multiple video frames included in the sample video to obtain multiple action parts corresponding to the arbitrary video frame. The computer device groups the action parts corresponding to the multiple video frames into multiple action parts corresponding to the sample video.

[0108] In some embodiments, the computer device extracts multiple video frames from the sample video. The computer device segments the target object included in the video frame to obtain multiple action parts corresponding to the video frame. The computer device combines the action parts corresponding to the multiple video frames into multiple action parts corresponding to the sample video.

[0109] The action information of the target object in the multiple video frames is used to represent the action information of the target object in the sample video. The number of multiple video frames can be set and changed as needed; in one implementation, the computer device determines the number of multiple video frames to be extracted based on at least one of the video type, action recognition requirements and resource configuration of the sample video. The video type is used to indicate the type of the sample video, and the video categories include educational videos, food videos, social entertainment videos, etc. The number of video frames extracted from sample videos of different video types is different. The action recognition requirements include the accuracy requirements for action recognition in the sample video, the number requirements for the number of action categories, etc. For example, for educational videos, the accuracy requirements for corresponding action recognition are higher, and the number of corresponding extracted video frames is larger. The higher the resource configuration of the computer device, the more video frames are extracted, and the lower the resource configuration, the fewer video frames are extracted. In another implementation, the computer device sets the number of video frames to be extracted by default, such as the computer device sets the number of video frames n extracted from a sample video of 1 second by default to 4.

[0110] The computer device may randomly extract video frames or extract video frames at equal time intervals. Randomly extracting video frames means randomly extracting n video frames from the sample video. Equal time interval extraction of video frames means that the time intervals between the extracted multiple video frames are the same. For example, if the total duration of the sample video is t and the number of extracted video frames is n, then the times at which the multiple video frames are extracted are 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41,

[0111] Among them, multiple video frames correspond to one image respectively. The computer device uses a self-supervised common part segmentation algorithm (SCOPS) to segment the target object in the image corresponding to the multiple video frames, and obtain multiple action parts of the target object, which facilitates the subsequent determination of the relative position vectors of the multiple action parts.

[0112] It should be noted that, since the SCOPS algorithm takes into account several characteristics of good image segmentation and encodes the prior knowledge of these characteristics into the loss function, it can achieve accurate segmentation of the target object. These characteristics include geometric concentration, robustness to changes, semantic consistency and foreground and background differentiation. Among them, geometric concentration refers to the geometric concentration of blocks in multiple parts and their close positions. Robustness to changes means that the block segmentation is robust to the deformation of the target object due to changes in posture and changes in the camera and viewpoint. Semantic consistency means that the blocks should have semantic consistency between different object instances, and different object instances refer to the target objects included in different video frames. Foreground and background differentiation refers to the segmented action part appearing on the foreground rather than the background. In this application, the foreground refers to the target object, and the union of multiple action parts constitutes the foreground, so that the segmentation of the image is more focused on the foreground in the partition diagram and ignores the background, which meets the segmentation requirements of the target object in the image in the embodiment of the present application.

[0113] For example, Figure 4 It is an image schematic diagram of a video frame provided according to an embodiment of the present application, including a target object performing a turning and jumping action, which is an image before segmentation. Figure 5 is another video frame image provided by an embodiment of the present application, which is Figure 4 The image shown is a schematic diagram of the effect of segmentation processing by the SCOPS algorithm, where the target object in the foreground is segmented into five action parts. Optionally, the five action parts are represented by five different color blocks of red, yellow, blue, green and gray, respectively, to represent different action parts.

[0114] 303. The computer device determines a first action part and multiple second action parts among the multiple action parts based on the part information of the multiple action parts corresponding to the sample video.

[0115] Wherein, the first action part is the reference action part of the target object. In some embodiments, the part information includes contour information of the action part. The computer device determines the minimum circumscribed rectangles of the multiple action parts respectively based on the contour information corresponding to the multiple action parts; then the computer device determines the first action part based on the minimum circumscribed rectangles corresponding to the multiple action parts, and the other parts other than the first action part are the second action parts. Wherein, the minimum circumscribed rectangle refers to a rectangle that contains the action part and is tangent to the contour of the action part, and is unique; the aspect ratio of the minimum circumscribed rectangle of the first action part is closest to the preset value 1, that is, the first action part is the trunk part, and the multiple second action parts are limbs.

[0116] 304. The computer device determines relative position vectors of multiple second motion parts relative to the first motion part, and obtains a relative position vector corresponding to the sample video.

[0117] In some embodiments, the step of the computer device determining the relative position vectors of the plurality of second motion parts relative to the first motion part, and obtaining the relative position vector corresponding to the sample video includes: the computer device determines the first center position and the second center position respectively, the first center position is the center position of the first motion part, and the second center position is the center position of the second motion part. The computer device determines the vector between the first center position and the second center position as the relative position vector of the second motion part relative to the first motion part. The computer device combines the relative position vectors of the plurality of second motion parts relative to the first motion part into a relative position vector corresponding to the sample video.

[0118] For example, Figure 6 It is a schematic diagram of a relative position vector provided in an embodiment of the present application, wherein the circle is the center point of the action part, and the arrow is the relative position vector of the second action part relative to the first action part, thereby abstracting the positional relationship between the first action part and multiple second action parts into a relative position vector, so as to facilitate the subsequent clustering of sample videos based on the relative position vectors between the first action part and multiple second action parts.

[0119] In this embodiment, since the center position of the action part can effectively represent the position information of the action part, the vectors between the center positions of the multiple second action parts and the center position of the first action part are determined to serve as the relative position vectors of the multiple second action parts relative to the first action part, respectively. This can improve the accuracy and representativeness of the relative position vectors of the multiple action parts obtained.

[0120] In some embodiments, the computer device determines the relative position vectors of multiple second action parts relative to the first action part, and the step of obtaining the relative position vector corresponding to the sample video includes: the computer device determines the first center position and multiple boundary positions respectively, the first center position is the center position of the first action part, and the multiple boundary positions are the boundary positions of the second action part respectively. The computer device determines the vectors between the first center position and the multiple boundary positions respectively. The computer device splices the vectors between the first center position and the multiple boundary positions respectively to obtain the relative position vector of the second action part relative to the first action part. The computer device combines the relative position vectors of the multiple second action parts relative to the first action part into a relative position vector corresponding to the sample video. Optionally, the multiple boundary positions include at least one of the topmost, bottommost, leftmost and rightmost boundary positions of the action part.

[0121] In this embodiment, since the multiple boundary positions of the second action part can effectively represent the position information of the second action part, the position vector between the multiple boundary positions of the second action parts and the center position of the first action part is further determined as the relative position vector of the second action part relative to the first action part. This can improve the accuracy and representativeness of the relative position vectors of the multiple action parts obtained.

[0122] 305. The computer device clusters the multiple sample videos based on the relative position vectors respectively corresponding to the multiple sample videos to obtain multiple first video clusters.

[0123] The first video cluster includes at least one sample video, and the sample videos in the same first video cluster have the same actions. The sample video includes multiple video frames, and the relative position vector corresponding to the sample video includes relative position vectors corresponding to multiple video frames.

[0124] In some embodiments, the computer device splices multiple relative position vectors corresponding to the same video frame to obtain a first relative position vector corresponding to the same video frame. The computer device splices the first relative position vectors corresponding to multiple same video frames of the same sample video to obtain a second relative position vector corresponding to the sample video. The computer device clusters the multiple sample videos based on the second relative position vectors corresponding to the multiple sample videos to obtain multiple first video clusters.

[0125] In one implementation, a computer device determines the pixel points corresponding to multiple second action parts in the same video frame, and based on the pixel points respectively contained in the multiple second action parts, sequentially splices the multiple relative position vectors of the same video frame to obtain the first relative position vector corresponding to the same video frame. Optionally, the arrangement order of the multiple relative position vectors in the first relative position vector is positively correlated with the pixel points contained in the second action part corresponding to it, that is, the computer device sorts the relative position vector of the second action part containing the most pixel points in the front, and the relative position vector of the second action part containing the least pixel points in the back. Alternatively, the arrangement order of the multiple relative position vectors in the first relative position vector is negatively correlated with the pixel points contained in the second action part corresponding to it; that is, the computer device sorts the relative position vector of the second action part containing the least pixel points in the front, and sorts the relative position vector of the second action part containing the most pixel points in the back; in the embodiment of the present application, this is not specifically limited.

[0126] Optionally, if the computer device determines the first position relative vector based on the first center position of the first action part and the boundary positions of multiple second action parts, and the multiple boundary positions include the topmost, bottommost, leftmost and rightmost boundary positions of the second action part, then for the same second action part, the computer device sorts the multiple relative position vectors of the same second action part in the order of topmost, bottommost, leftmost and rightmost to obtain the third relative position vector of the same second action part, and then based on the pixel points respectively contained in the multiple second action parts in the same video frame, the multiple third relative position vectors of the same video frame are spliced ​​in turn to obtain the first relative position vector corresponding to the same video frame.

[0127] Optionally, the relative position vector is a 2-dimensional position vector, in which the elements respectively represent the horizontal coordinate and the vertical coordinate of the vector relative to the origin of the first center position. If the multiple second action parts are limbs, and the computer device determines the first relative position vector based on the first center position of the first action part and the second center positions of the multiple second action parts, then the dimension of the first relative position vector obtained by splicing the multiple relative position vectors corresponding to the same video frame is 8 dimensions. Correspondingly, if the number of multiple video frames is 4, and each video frame corresponds to an 8-dimensional first relative position vector, then the dimension of the second relative position vector obtained by splicing the first relative position vectors corresponding to the 4 video frames corresponding to the same sample video is 32 dimensions. Optionally, the computer device splices the first relative position vectors corresponding to the multiple video frames in the time order of the multiple video frames to obtain the second relative position vector.

[0128] Optionally, if the multiple second action parts are limbs, and the computer device determines the first relative position vector based on the first center position of the first action part and the boundary positions of the multiple second action parts, and the multiple boundary positions include the top, bottom, leftmost and rightmost boundary positions of the second action part, then the dimension of the first relative position vector obtained by splicing 16 2-dimensional relative position vectors corresponding to the same video frame is 32 dimensions. Correspondingly, if the number of multiple video frames is 4, and each video frame corresponds to a 32-dimensional first relative position vector, then the dimension of the second relative position vector obtained by splicing the first relative position vectors corresponding to the 4 video frames corresponding to the same sample video is 128 dimensions. Optionally, the computer device splices the first relative position vectors corresponding to the multiple video frames in the time sequence of the multiple video frames to obtain the second relative position vector.

[0129] In an embodiment of the present application, a second relative position vector of a sample video is obtained by splicing together first relative position vectors of multiple video frames of the sample video. Since the second relative position vector includes the first relative position vectors of multiple video frames, the second relative position vector is more representative, i.e., more accurate. Furthermore, multiple sample videos are clustered based on the second relative position vectors corresponding to each of the multiple sample videos, so that multiple first video clusters with high accuracy can be obtained.

[0130] In some embodiments, the computer device determines the distance between any two second relative position vectors based on the second relative position vectors corresponding to multiple sample videos respectively; when the distance is not greater than a preset distance, the computer device aggregates the two sample videos corresponding to the two second relative position vectors into the same first video cluster.

[0131] In this embodiment, the computer device clusters multiple sample videos by a hierarchical clustering method, and the hierarchical clustering algorithm is one of the clustering algorithms. The computer device determines the distance between sample videos of different categories to determine the similarity between sample videos of different categories. Then a hierarchical nested clustering tree is constructed. In the clustering tree, each sample video is a category, which is the lowest layer of the clustering tree, and the top layer of the clustering tree is the root node of the cluster, that is, the final category of multiple sample videos. Among them, creating a clustering tree by a hierarchical clustering method includes two implementation methods: bottom-up merging and top-down classification. In an embodiment of the present application, multiple sample videos are clustered based on a bottom-up merging method. Optionally, the hierarchical clustering method does not need to specify the number of clusters, and a distance threshold for a given cluster is sufficient.

[0132] Optionally, the preset distance is the minimum value of the distance between any two second relative position vectors. The computer device aggregates the two sample videos corresponding to the two second position vectors with the smallest distance into the same first video cluster. Then, the computer device recalculates the distance between the second relative position vectors of the first video cluster and other sample videos excluding the two sample videos in the first video cluster, aggregates the first video cluster with the smallest distance and the sample videos corresponding to the second position vector into the same first video cluster, or aggregates the two sample videos corresponding to the two second position vectors with the smallest distance into the same first video cluster. The computer device iterates the above steps repeatedly until the distance between any two relative position vectors is greater than a preset threshold, stops clustering, and obtains multiple first video clusters.

[0133] In an embodiment of the present application, multiple sample videos are clustered based on the distance between any two second relative position vectors. Since the hierarchical clustering method does not require the pre-determination of the number of clusters, a multi-level clustering structure at different granularities can be obtained by setting different distance thresholds. The calculation speed is fast and the logic is simple, thereby improving the efficiency of clustering multiple sample videos.

[0134] In another implementation, the computer device clusters multiple sample videos based on the k-means clustering algorithm. Accordingly, the computer device randomly determines k sample videos as initial cluster centers, then determines the distances of other multiple sample videos from the multiple initial cluster centers, and assigns the other multiple sample videos to the first video cluster corresponding to the cluster center closest to them. Among them, the cluster center and the sample videos assigned to them represent a cluster; each time a sample video is assigned, the cluster center of the cluster is recalculated according to the existing sample videos in the cluster. This process will be repeated until the target termination condition is met. The target termination condition can be at least one of no or the minimum number of sample videos being reassigned to different clusters, no or the minimum number of cluster centers changing again, and the local minimum of the sum of squared errors. In another implementation, the computer device clusters multiple sample videos based on a representative density-based clustering algorithm (Density-Based Spatial Clustering of Applications with Noise, DBSCAN), which will not be repeated here.

[0135] In the embodiments of the present application, the problem of relying on labeled sample videos to train action recognition models in the prior art is solved by performing action division and clustering on sample videos in an unsupervised manner without the need for action labels.

[0136] 306. The computer device trains an action recognition model based on the multiple first video clusters.

[0137] In some embodiments, the step of training the action recognition model based on the plurality of first video clusters by the computer device includes: the computer device annotates the action labels of the plurality of sample videos based on the plurality of first video clusters. The computer device performs training based on the annotated sample videos to obtain the action recognition model.

[0138] In some embodiments, the step of performing action recognition by a computer device through an action recognition model includes: the computer device obtains a target video, inputs the target video into the action recognition model, and outputs the action category of the target video and the relative position vectors of multiple action parts of the target object in the target video. The target video is a video of the action category to be identified, and the relative position vector is used to explain why the target video is identified as the action category. If the target video includes multiple video frames, the computer device determines the average value of the fourth relative position vectors of the same action part corresponding to the multiple video frames, obtains the relative position vector of the same action part, and then obtains the relative position vectors of the multiple action parts of the target object in the target video.

[0139] Optionally, the action recognition model marks the relative position vectors of multiple action parts as follows: Figure 6 The schematic diagram of the relative position vector shown is output to explain why the target video is identified as the action category, avoiding the problem that the action recognition model is difficult to understand and apply as a black box model, that is, solving the problem of poor interpretability of the action recognition model in the prior art, so that the action recognition model has a wider range of application scenarios, such as smart education scenarios.

[0140] See also Figure 7 , Figure 7It is a flow chart of a training method for an action recognition model provided by an embodiment of the present application. It includes five stages: video division, video frame sampling, image segmentation, relative position determination, and action clustering. In the video division stage, the computer device divides the initial sample video into short videos at a certain time interval to obtain a sample video, and the action changes in the sample video are small. In the video frame sampling stage, the computer device randomly extracts images of n video frames in the sample video, and the action of the object in the image is used to represent the action of the object in the sample video. In the image segmentation stage, the computer device segments the image of the video frame to divide the object in the image into multiple action parts of the trunk and limbs. In the relative position determination stage, the computer device determines the relative position vectors of the trunk part and the limb part, and splices the relative position vectors in multiple video frames. In the action clustering stage, the computer device clusters multiple sample videos based on the k-means algorithm to obtain multiple first video clusters of different action categories, and then the computer device puts the multiple first video clusters into application and trains to obtain an action recognition model. It should be noted that the resource configuration of the computer device can be determined according to the data volume of the sample video to improve the flexibility of sample video processing; for example, when the data volume of the sample video is large, multiple background servers are used to process the sample video and train the action recognition model.

[0141] In an embodiment of the present application, by determining the relative position vectors between multiple action parts, since the relative position vectors between multiple action parts can effectively convey the action information of the object, multiple sample videos are clustered based on the relative position vectors corresponding to the multiple sample videos, and multiple first video clusters with accurate classification can be obtained. Subsequently, the action recognition model is trained based on the multiple first video clusters, which can improve the recognition effect of the obtained action recognition model.

[0142] Figure 8 1 is a flow chart of a method for training an action recognition model provided in an embodiment of the present application. The method for training an action recognition model comprises the following steps:

[0143] 801. A computer device obtains multiple sample videos.

[0144] 802. The computer device segments the target object in the sample video to obtain multiple action parts corresponding to the sample video.

[0145] 803. The computer device determines a first action part and multiple second action parts among the multiple action parts based on the part information of the multiple action parts corresponding to the sample video, wherein the first action part is a reference action part of the target object.

[0146] 804. The computer device determines relative position vectors of multiple second motion parts relative to the first motion part, and obtains a relative position vector corresponding to the sample video.

[0147] 805. The computer device clusters the multiple sample videos based on the relative position vectors respectively corresponding to the multiple sample videos to obtain multiple first video clusters.

[0148] Steps 801-805 are the same as steps 301-305 and will not be repeated here.

[0149] 806. The computer device determines a target action part from multiple second action parts.

[0150] Among them, the target action part contains the least pixels.

[0151] In some embodiments, the target action part may also be the second action part containing the most pixels. Alternatively, the pixels contained in the target action part are within a preset pixel range. In some embodiments, the number of target action parts is multiple, such as the target action part is the two second action parts containing the least pixels, or the target action part is the two second action parts containing the most pixels, which is not specifically limited here.

[0152] 807. The computer device clusters the multiple sample videos based on the first action parts and the target action parts of the multiple sample videos to obtain multiple second video clusters.

[0153] In some embodiments, the computer device clusters multiple sample videos based on the first action parts and target action parts of the multiple sample videos to obtain multiple second video clusters, including: the computer device determines the first similarity and the second similarity of any two sample videos, the first similarity is the similarity between the first action parts of the two sample videos, and the second similarity is the similarity between the target action parts of the two sample videos; the computer device determines the third similarity between the two sample videos based on the first similarity and the second similarity. When the third similarity is greater than the preset similarity, the computer device aggregates the two sample videos into the same second video cluster.

[0154] Optionally, the first similarity and the second similarity are both structural similarity index (SSIM). Accordingly, the step of determining the first similarity of any two sample videos by the computer device includes: the computer device extracts brightness features, contrast features and structural features of the first action parts of the two sample videos respectively, and obtains the first similarity between the two sample videos through the SSIM function based on the brightness features, contrast features and structural features. The SSIM function is a function of brightness features, contrast features and structural features. The step of determining the second similarity of any two sample videos by the computer device is the same as the step of determining the first similarity of any two sample videos by the computer device, and will not be repeated here.

[0155] In some embodiments, the sample video includes multiple video frames, and the first similarity corresponding to the sample video includes the first similarities corresponding to the multiple video frames. Optionally, the computer device determines the similarity of the first action part in the video frame at the same time in the two sample videos, and obtains the first similarity of the two sample videos based on the average value of the similarity of the first action part in the video frame corresponding to multiple moments in the two sample videos. Similarly, the computer device determines the similarity of the target action part in the video frame at the same time in the two sample videos, and obtains the second similarity of the two sample videos based on the average value of the similarity of the target action part in the video frame corresponding to multiple moments in the two sample videos. Optionally, the computer device determines the average value between the first similarity and the second similarity to obtain the third similarity between the two sample videos; or, the computer device determines the sum of the first similarity and the second similarity to obtain the third similarity between the two sample videos. Among them, the preset similarity can be set and changed as needed, and this is not specifically limited in the embodiments of the present application.

[0156] In the embodiment of the present application, since the similarity between the first action parts and the similarity between the target action parts can effectively represent the similarity between multiple sample videos, multiple sample videos are clustered by comprehensively analyzing the similarity between the first action parts and the similarity between the target action parts of multiple sample videos, and multiple second video clusters with accurate classification can be obtained.

[0157] In an embodiment of the present application, since the target action parts in multiple sample videos are all the second action parts containing the least pixels, the multiple sample videos are clustered by the first action part and the target action part, so that the multiple sample videos can be compared and clustered at the same level, and by clustering based on the target action parts in the second action part, the time-consuming problem of clustering based on multiple second action parts is avoided, the efficiency of clustering multiple sample videos is improved, and the training efficiency of the action recognition model is improved.

[0158] It should be noted that, in some embodiments, the computer device can also only perform steps 801-803 and steps 806-807 to obtain multiple second video clusters, and then train the action recognition model based on the multiple second video clusters. The specific training steps of the action recognition model are the same as step 306 and will not be repeated here.

[0159] It should be noted that the embodiment of the present application does not limit the order of steps 804-805 and steps 806-807. The computer device can first execute steps 804-805 and then execute steps 806-807; or, the computer device can first execute steps 806-807 and then execute steps 804-805; or, the computer device can also execute steps 804-805 and steps 806-807 at the same time; the embodiment of the present application does not specifically limit this.

[0160] 808. The computer device trains an action recognition model based on the multiple first video clusters and the multiple second video clusters.

[0161] In some embodiments, the computer device determines whether the multiple first video clusters are the same as the multiple second video clusters; if the multiple first video clusters are the same as the multiple second video clusters, the computer device trains the action recognition model based on the multiple first video clusters or the multiple second video clusters. If the multiple first video clusters are not the same as the multiple second video clusters, it is manually determined whether there are inaccurately clustered video clusters in the multiple first video clusters or the multiple second video clusters, and then the inaccurately clustered video clusters are corrected; the computer device trains the action recognition model based on the corrected multiple first video clusters or the corrected multiple second video clusters. The specific training steps of the action recognition model are the same as step 306 and will not be repeated here.

[0162] See also Fig. 9 , Fig. 9 This is a schematic diagram of the application of an action recognition model in an intelligent education scenario provided by an embodiment of the present application. In this scenario, the target object is a teacher, who is standing and teaching. There is a whiteboard or projection for the teacher to display possible content. The scene includes the teacher's half body or full body. The action recognition model identifies the action category in the scene as explanation. Therefore, the action recognition model in the embodiment of the present application can analyze the teacher's teaching behavior, thereby supporting AI teaching evaluation.

[0163] In an embodiment of the present application, multiple sample videos are clustered respectively based on two factors, namely, relative position vector and similarity. In this way, multiple first video clusters and multiple second video clusters are clustered respectively based on the relative position vector and the similarity. It is then possible to determine whether there are inaccurately clustered video clusters among the clustered video clusters through the comparison between the multiple first video clusters and the multiple second video clusters. Furthermore, if there are inaccurately clustered video clusters among the clustered video clusters, the video clusters can be corrected. In this way, the action recognition model is trained based on the corrected video clusters, which can improve the accuracy of the trained action recognition model.

[0164] Fig.10 is a block diagram of a training device for an action recognition model provided according to an embodiment of the present application, the device is used to execute the training method for the action recognition model, see Fig.10 , the device comprises:

[0165] A first acquisition module 1001 is used to acquire multiple sample videos;

[0166] The segmentation module 1002 is used to segment the target object in the sample video to obtain multiple action parts corresponding to the sample video, where the multiple action parts are action parts of the target object, and the target object is an object performing an action;

[0167] A determination module 1003 is used to determine the relative position vectors between the multiple action parts, and obtain the relative position vectors corresponding to the sample video, where the relative position vectors are used to represent the positional relationship between the multiple action parts;

[0168] A clustering module 1004 is used to cluster the multiple sample videos based on the position information of the action parts respectively corresponding to the multiple sample videos to obtain multiple first video clusters, wherein the first video cluster includes at least one sample video, and the sample videos in the same first video cluster have the same action;

[0169] The training module 1005 is used to train the action recognition model based on multiple first video clusters.

[0170] In some embodiments, the determination module 1003 is used to determine a first action part and multiple second action parts among multiple action parts based on the part information of multiple action parts corresponding to the sample video, where the first action part is the benchmark action part of the target object; determine the relative position vectors of the multiple second action parts relative to the first action part, and obtain the relative position vector corresponding to the sample video.

[0171] In some embodiments, the determination module 1003 is used to respectively determine the first center position and the second center position, the first center position being the center position of the first action part, and the second center position being the center position of the second action part; the vector between the first center position and the second center position is determined as the relative position vector of the second action part relative to the first action part; and the relative position vectors of multiple second action parts relative to the first action part are combined to form a relative position vector corresponding to the sample video.

[0172] In some embodiments, the determination module 1003 is used to respectively determine a first center position and multiple boundary positions, the first center position being the center position of the first action part, and the multiple boundary positions being the boundary positions of the second action part; determine the vectors between the first center position and the multiple boundary positions respectively; concatenate the vectors between the first center position and the multiple boundary positions respectively to obtain a relative position vector of the second action part relative to the first action part; and combine the relative position vectors of the multiple second action parts relative to the first action part to form a relative position vector corresponding to the sample video.

[0173] In some embodiments, the sample video includes multiple video frames, and the relative position vector corresponding to the sample video includes relative position vectors corresponding to multiple video frames respectively; the clustering module 1004 is used to splice multiple relative position vectors corresponding to the same video frame to obtain a first relative position vector corresponding to the same video frame; splice the first relative position vectors corresponding to multiple video frames of the same sample video to obtain a second relative position vector corresponding to the sample video; cluster the multiple sample videos based on the second relative position vectors corresponding to the multiple sample videos respectively to obtain multiple first video clusters.

[0174] In some embodiments, the clustering module 1004 is used to determine the distance between any two second relative position vectors based on the second relative position vectors respectively corresponding to the plurality of sample videos;

[0175] When the distance is not greater than the preset distance, the two sample videos corresponding to the two second relative position vectors are aggregated into the same first video cluster.

[0176] In some embodiments, the apparatus further comprises:

[0177] The second acquisition module is used to acquire a target video, where the target video is a video of the action category to be identified;

[0178] The input-output module is used to input the target video into the action recognition model, output the action category of the target video and the relative position vectors of multiple action parts of the target object in the target video, and the relative position vectors are used to explain why the target video is identified as an action category.

[0179] In some embodiments, the training module 1005 is used to determine a first action part and multiple second action parts among multiple action parts based on part information of multiple action parts, wherein the first action part is a reference action part of the target object; determine a target action part from multiple second action parts, wherein the target action part contains the least pixels; cluster multiple sample videos based on the first action parts and the target action parts of multiple sample videos to obtain multiple second video clusters, wherein the second video cluster includes at least one sample video, and the actions of the sample videos in the same second video cluster are the same; and train an action recognition model based on the multiple first video clusters and the multiple second video clusters.

[0180] In some embodiments, the training module 1005 is used to determine a first similarity and a second similarity between any two sample videos, where the first similarity is the similarity between the first action parts of the two sample videos, and the second similarity is the similarity between the target action parts of the two sample videos; based on the first similarity and the second similarity, determine a third similarity between the two sample videos; when the third similarity is greater than a preset similarity, aggregate the two sample videos into the same second video cluster.

[0181] In some embodiments, the segmentation module 1002 is used to extract multiple video frames from the sample video; segment the target object included in the video frame to obtain multiple action parts corresponding to the video frame; and combine the action parts corresponding to the multiple video frames into multiple action parts corresponding to the sample video.

[0182] In the embodiments of the present application, the computer device can be configured as a terminal or a server. When the computer device is configured as a terminal, the terminal serves as the execution subject to implement the technical solution provided in the embodiments of the present application; when the computer device is configured as a server, the server serves as the execution subject to implement the technical solution provided in the embodiments of the present application; or, the technical solution provided in the present application is implemented through interaction between the terminal and the server, which is not limited in the embodiments of the present application.

[0183] Fig.11 The structure block diagram of a terminal 1100 provided by an exemplary embodiment of the present application is shown. The terminal 1100 may be a portable mobile terminal, such as a smart phone, a tablet computer, an MP3 player (Movi12g Picture Experts Group Audio Layer III, dynamic image expert compression standard audio layer 3), an MP4 (Movi12g Picture Experts Group Audio Layer IV, dynamic image expert compression standard audio layer 4) player, a notebook computer or a desktop computer. The terminal 1100 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0184] Typically, the terminal 1100 includes a processor 1101 and a memory 1102 .

[0185] The processor 1101 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1101 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1101 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1101 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1101 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0186] The memory 1102 may include one or more computer-readable storage media, which may be non-transitory. The memory 1102 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1102 is used to store at least one program code, which is used to be executed by the processor 1101 to implement the training method of the action recognition model provided in the method embodiment of the present application.

[0187] In some embodiments, the terminal 1100 may further optionally include: a peripheral device interface 1103 and at least one peripheral device. The processor 1101, the memory 1102 and the peripheral device interface 1103 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1103 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1104, a display screen 1105, a camera assembly 1106, an audio circuit 1107 and a power supply 1108.

[0188] The peripheral device interface 1103 may be used to connect at least one peripheral device related to I / O (I12put / Output, input / output) to the processor 1101 and the memory 1102. In some embodiments, the processor 1101, the memory 1102, and the peripheral device interface 1103 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1101, the memory 1102, and the peripheral device interface 1103 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0189] The radio frequency circuit 1104 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1104 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1104 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1104 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 1104 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1104 may also include circuits related to 12FC (12ear Field Communication, short-range wireless communication), which is not limited in this application.

[0190] The display screen 1105 is used to display the UI (User I12interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1105 is a touch display screen, the display screen 1105 also has the ability to collect touch signals on the surface or above the surface of the display screen 1105. The touch signal can be input to the processor 1101 as a control signal for processing. At this time, the display screen 1105 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1105 can be one, set on the front panel of the terminal 1100; in other embodiments, the display screen 1105 can be at least two, respectively set on different surfaces of the terminal 1100 or in a folding design; in other embodiments, the display screen 1105 can be a flexible display screen, set on a curved surface or a folding surface of the terminal 1100. Even, the display screen 1105 can also be set to a non-rectangular irregular figure, that is, a special-shaped screen. The display screen 1105 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitti12gDiode).

[0191] The camera assembly 1106 is used to capture images or videos. Optionally, the camera assembly 1106 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1106 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0192] The audio circuit 1107 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1101 for processing, or input them into the radio frequency circuit 1104 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different action parts of the terminal 1100. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1101 or the radio frequency circuit 1104 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1107 may also include a headphone jack.

[0193] The power supply 1108 is used to power various components in the terminal 1100. The power supply 1108 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1108 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0194] In some embodiments, the terminal 1100 further includes one or more sensors 1109 , including but not limited to: an acceleration sensor 1110 , a gyroscope sensor 1111 , a pressure sensor 1112 , an optical sensor 1113 , and a proximity sensor 1114 .

[0195] The acceleration sensor 1110 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 1100. For example, the acceleration sensor 1110 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 1101 can control the display screen 1105 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1110. The acceleration sensor 1110 can also be used for collecting game or user motion data.

[0196] The gyro sensor 1111 can detect the body direction and rotation angle of the terminal 1100, and the gyro sensor 1111 can cooperate with the acceleration sensor 1110 to collect the user's 3D actions on the terminal 1100. The processor 1101 can implement the following functions based on the data collected by the gyro sensor 1111: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0197] The pressure sensor 1112 may be provided at the side frame of the terminal 1100 and / or the lower layer of the display screen 1105. When the pressure sensor 1112 is provided at the side frame of the terminal 1100, it may detect the user's holding signal of the terminal 1100, and the processor 1101 may perform left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 1112. When the pressure sensor 1112 is provided at the lower layer of the display screen 1105, the processor 1101 may control the operability control on the UI interface according to the user's pressure operation on the display screen 1105. The operability control includes at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0198] The optical sensor 1113 is used to collect the ambient light intensity. In one embodiment, the processor 1101 can control the display brightness of the display screen 1105 according to the ambient light intensity collected by the optical sensor 1113. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1105 is increased; when the ambient light intensity is low, the display brightness of the display screen 1105 is reduced. In another embodiment, the processor 1101 can also dynamically adjust the shooting parameters of the camera assembly 1106 according to the ambient light intensity collected by the optical sensor 1113.

[0199] The proximity sensor 1114, also called a distance sensor, is usually arranged on the front panel of the terminal 1100. The proximity sensor 1114 is used to collect the distance between the user and the front of the terminal 1100. In one embodiment, when the proximity sensor 1114 detects that the distance between the user and the front of the terminal 1100 is gradually decreasing, the processor 1101 controls the display screen 1105 to switch from the screen-on state to the screen-off state; when the proximity sensor 1114 detects that the distance between the user and the front of the terminal 1100 is gradually increasing, the processor 1101 controls the display screen 1105 to switch from the screen-off state to the screen-on state.

[0200] Those skilled in the art will understand that Fig.11 The structure shown in the figure does not constitute a limitation on the terminal 1100, and the terminal 1100 may include more or fewer components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0201] Fig.12It is a structural diagram of a server provided according to an embodiment of the present application. The server 1200 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1201 and one or more memories 1202, wherein the memory 1202 stores at least one computer program, and the at least one computer program is loaded and executed by the processor 1201 to implement the training method of the action recognition model provided by the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server may also include other components for implementing device functions, which will not be repeated here.

[0202] An embodiment of the present application also provides a computer-readable storage medium, in which at least one program code is stored, and the at least one program code is loaded and executed by a processor to implement a training method for an action recognition model of any of the above-mentioned implementation methods.

[0203] An embodiment of the present application also provides a computer program product, which includes at least one program code, and the at least one program code is loaded and executed by a processor to implement the training method of the action recognition model described in any of the above implementation methods.

[0204] In some embodiments, the computer program product involved in the embodiments of the present application may be deployed and executed on a computer device, or on multiple computer devices located at one location, or on multiple computer devices distributed at multiple locations and interconnected by a communication network. Multiple computer devices distributed at multiple locations and interconnected by a communication network may constitute a blockchain system.

[0205] The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application. It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the sample videos involved in the present application were obtained with full authorization.

Claims

1. A method for training an action recognition model, characterized in that: The method comprises: Get multiple sample videos; Segmenting the target object in the sample video to obtain a plurality of action parts corresponding to the sample video, wherein the plurality of action parts are action parts of the target object, and the target object is an object performing an action; Determine the relative position vectors between the multiple action parts to obtain the relative position vector corresponding to the sample video, wherein the relative position vector is used to represent the positional relationship between the multiple action parts; Splicing multiple relative position vectors corresponding to the same video frame to obtain a first relative position vector corresponding to the same video frame; splicing the first relative position vectors corresponding to multiple same video frames of the same sample video to obtain a second relative position vector corresponding to the sample video; clustering the multiple sample videos based on the second relative position vectors respectively corresponding to the multiple sample videos to obtain multiple first video clusters, wherein the first video cluster includes at least one sample video, and the sample videos in the same first video cluster have the same actions, the sample video includes multiple video frames, and the relative position vectors corresponding to the sample videos include the relative position vectors respectively corresponding to the multiple video frames; Based on the multiple first video clusters, an action recognition model is trained.

2. The method according to claim 1, characterized in that: The determining the relative position vectors between the plurality of action parts to obtain the relative position vector corresponding to the sample video includes: Based on the part information of the multiple action parts corresponding to the sample video, determining a first action part and multiple second action parts among the multiple action parts, wherein the first action part is a reference action part of the target object; Determine the relative position vectors of the multiple second action parts relative to the first action part, and obtain the relative position vector corresponding to the sample video.

3. The method according to claim 2, characterized in that The determining the relative position vectors of the plurality of second motion parts relative to the first motion part to obtain the relative position vector corresponding to the sample video includes: Determine a first center position and a second center position respectively, the first center position being the center position of the first action part, and the second center position being the center position of the second action part; Determine a vector between the first center position and the second center position as a relative position vector of the second action part relative to the first action part; The relative position vectors of the plurality of second motion parts relative to the first motion part are combined into a relative position vector corresponding to the sample video.

4. The method according to claim 2, characterized in that: The determining the relative position vectors of the plurality of second motion parts relative to the first motion part to obtain the relative position vector corresponding to the sample video includes: Determine a first center position and a plurality of boundary positions respectively, wherein the first center position is the center position of the first action part, and the plurality of boundary positions are the boundary positions of the second action part respectively; Determine vectors between the first center position and the plurality of boundary positions respectively; splicing the vectors between the first center position and the plurality of boundary positions respectively to obtain a relative position vector of the second action part relative to the first action part; The relative position vectors of the plurality of second motion parts relative to the first motion part are combined into a relative position vector corresponding to the sample video.

5. The method according to claim 1, characterized in that The clustering of the plurality of sample videos based on the second relative position vectors respectively corresponding to the plurality of sample videos to obtain a plurality of first video clusters includes: Determine the distance between any two second relative position vectors based on the second relative position vectors respectively corresponding to the multiple sample videos; When the distance is not greater than a preset distance, the two sample videos corresponding to the two second relative position vectors are aggregated into the same first video cluster.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Obtain a target video, where the target video is a video of a category of action to be identified; The target video is input into the action recognition model, and the action category of the target video and the relative position vectors of multiple action parts of the target object in the target video are output, wherein the relative position vectors are used to explain why the target video is identified as the action category.

7. The method according to claim 1, characterized in that The step of training the action recognition model based on the plurality of first video clusters includes: Based on the part information of the plurality of action parts, determining a first action part and a plurality of second action parts among the plurality of action parts, wherein the first action part is a reference action part of the target object; Determine a target action part from the plurality of second action parts, wherein the target action part contains the least number of pixels; Clustering the plurality of sample videos based on the first action parts and the target action parts of the plurality of sample videos to obtain a plurality of second video clusters, wherein the second video cluster includes at least one sample video, and the actions of the sample videos in the same second video cluster are the same; The action recognition model is trained based on the plurality of first video clusters and the plurality of second video clusters.

8. The method according to claim 7, characterized in that The clustering of the plurality of sample videos based on the first action parts and the target action parts of the plurality of sample videos to obtain a plurality of second video clusters includes: Determine a first similarity and a second similarity between any two sample videos, wherein the first similarity is a similarity between first action parts of the two sample videos, and the second similarity is a similarity between target action parts of the two sample videos; Determining a third similarity between the two sample videos based on the first similarity and the second similarity; When the third similarity is greater than a preset similarity, the two sample videos are aggregated into the same second video cluster.

9. The method according to claim 1, characterized in that: The step of segmenting the target object in the sample video to obtain a plurality of action parts corresponding to the sample video includes: Extracting a plurality of video frames from the sample video; Segmenting the target object included in the video frame to obtain multiple action parts corresponding to the video frame; The action parts corresponding to the multiple video frames are respectively combined into multiple action parts corresponding to the sample video.

10. A training device for an action recognition model, characterized in that: The device comprises: A first acquisition module, used to acquire multiple sample videos; a segmentation module, configured to segment the target object in the sample video to obtain a plurality of action parts corresponding to the sample video, wherein the plurality of action parts are action parts of the target object, and the target object is an object performing an action; A determination module, used to determine a relative position vector between the plurality of action parts, wherein the relative position vector is used to represent a positional relationship between the plurality of action parts; A clustering module is used to splice multiple relative position vectors corresponding to the same video frame to obtain a first relative position vector corresponding to the same video frame; splice the first relative position vectors corresponding to multiple same video frames of the same sample video to obtain a second relative position vector corresponding to the sample video; cluster the multiple sample videos based on the second relative position vectors respectively corresponding to the multiple sample videos to obtain multiple first video clusters, wherein the first video cluster includes at least one sample video, and the sample videos in the same first video cluster have the same action, the sample video includes multiple video frames, and the relative position vectors corresponding to the sample video include the relative position vectors respectively corresponding to the multiple video frames; A training module is used to train an action recognition model based on the multiple first video clusters.

11. The device according to claim 10, characterized in that The determining module is used to: Based on the part information of the multiple action parts corresponding to the sample video, determining a first action part and multiple second action parts among the multiple action parts, wherein the first action part is a reference action part of the target object; Determine the relative position vectors of the multiple second action parts relative to the first action part, and obtain the relative position vector corresponding to the sample video.

12. The device according to claim 10, characterized in that The determining module is used to: Determine a first center position and a second center position respectively, the first center position being the center position of the first action part, and the second center position being the center position of the second action part; Determine a vector between the first center position and the second center position as a relative position vector of the second action part relative to the first action part; The relative position vectors of the plurality of second motion parts relative to the first motion part are combined into a relative position vector corresponding to the sample video.

13. The device according to claim 11, characterized in that The determining module is used to: Determine a first center position and a plurality of boundary positions respectively, wherein the first center position is the center position of the first action part, and the plurality of boundary positions are the boundary positions of the second action part respectively; Determine vectors between the first center position and the plurality of boundary positions respectively; splicing the vectors between the first center position and the plurality of boundary positions respectively to obtain a relative position vector of the second action part relative to the first action part; The relative position vectors of the plurality of second motion parts relative to the first motion part are combined into a relative position vector corresponding to the sample video.

14. The device according to claim 10, characterized in that The clustering module is used to: Determine the distance between any two second relative position vectors based on the second relative position vectors respectively corresponding to the multiple sample videos; When the distance is not greater than a preset distance, the two sample videos corresponding to the two second relative position vectors are aggregated into the same first video cluster.

15. The device according to any one of claims 10 to 14, characterized in that: The device also includes: A second acquisition module is used to acquire a target video, where the target video is a video of a to-be-identified action category; The input-output module is used to input the target video into the action recognition model, output the action category of the target video and the relative position vectors of multiple action parts of the target object in the target video, and the relative position vectors are used to explain why the target video is identified as the action category.

16. The device according to claim 10, characterized in that The training module is used to: Based on the part information of the plurality of action parts, determining a first action part and a plurality of second action parts among the plurality of action parts, wherein the first action part is a reference action part of the target object; Determine a target action part from the plurality of second action parts, wherein the target action part contains the least number of pixels; Clustering the plurality of sample videos based on the first action parts and the target action parts of the plurality of sample videos to obtain a plurality of second video clusters, wherein the second video cluster includes at least one sample video, and the actions of the sample videos in the same second video cluster are the same; The action recognition model is trained based on the plurality of first video clusters and the plurality of second video clusters.

17. The device according to claim 16, characterized in that The training module is used to: Determine a first similarity and a second similarity between any two sample videos, wherein the first similarity is a similarity between first action parts of the two sample videos, and the second similarity is a similarity between target action parts of the two sample videos; Determining a third similarity between the two sample videos based on the first similarity and the second similarity; When the third similarity is greater than a preset similarity, the two sample videos are aggregated into the same second video cluster.

18. The device according to claim 10, characterized in that The segmentation module is used for: Extracting a plurality of video frames from the sample video; The target object included in the video frame is segmented to obtain a plurality of action parts corresponding to the video frame; and the action parts corresponding to the plurality of video frames are combined into a plurality of action parts corresponding to the sample video.

19. A computer device, characterized in that: The computer device includes a processor and a memory, the memory is used to store at least one computer program, and the at least one computer program is loaded by the processor and executes the training method of the action recognition model described in any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store at least one computer program, and the at least one computer program is used to execute the training method of the action recognition model described in any one of claims 1 to 9.

21. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the training method of the action recognition model as claimed in any one of claims 1 to 9 is implemented.

Images