Behavior recognition method and device

Abstract

The invention discloses a behavior recognition method and device, and relates to the technical field of computers. A specific embodiment of the method comprises the following steps: acquiring multiple pieces of video data and a behavior category corresponding to each frame in each piece of video data; for each frame in each piece of video data, extracting human visual features, object visual features and human skeleton features from the frame; obtaining a behavior semantic feature corresponding to the behavior category according to the behavior category of the frame; constructing visual motion features according to the human visual features and the object visual features, and constructing skeleton motion features according to the human skeleton features and the object visual features; and according to a behavior semantic feature corresponding to each frame in each piece of video data, constraining the visual motion feature and the skeleton motion feature, and training to obtain a behavior recognition model. According to the embodiment, the accuracy and reliability of operation behavior recognition can be improved, the working efficiency and quality are improved, and the operation normalization and safety are guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a behavior recognition method and device. Background Art

[0002] In many industry fields, by identifying the operating behaviors of employees, the standardization of operating behaviors can be ensured. For example, in the process of vehicle maintenance operations, by identifying the operating behaviors of employees, it is possible to judge whether the employee behaviors are standardized and whether the operating procedures are correct.

[0003] In the related technology, the behavior recognition methods mainly include action detection methods based on spatiotemporal features, action recognition methods based on skeletons, and action detection methods based on language information enhancement. However, the above methods have the problem of difficulty in feature extraction, ignoring the interactive features in the maintenance process and the constraints between features, resulting in low accuracy of behavior recognition. Summary of the invention

[0004] In view of this, the embodiments of the present invention provide a behavior recognition method and device, which can improve the accuracy and reliability of operation behavior recognition, improve work efficiency and quality, and ensure operation standardization and safety.

[0005] To achieve the above object, according to one aspect of an embodiment of the present invention, a method for training a behavior recognition model is provided, comprising:

[0006] Acquire multiple video data and the behavior category corresponding to each frame in each video data;

[0007] For each frame in each video data, human visual features, object visual features and human skeleton features are extracted from the frame; and behavior semantic features corresponding to the behavior category are obtained according to the behavior category of the frame;

[0008] Constructing visual motion features according to the human visual features and the object visual features, and constructing skeletal motion features according to the human skeletal features and the object visual features;

[0009] The visual motion features and the skeletal motion features are constrained according to the behavior semantic features corresponding to each frame in each video data, and a behavior recognition model is obtained by training.

[0010] Optionally, extracting human visual features, object visual features and human skeleton features from the frame includes:

[0011] Extracting human frame coordinates and object frame coordinates from the frame, and extracting the time sequence features of the frame from the video data; obtaining the human visual features and the object visual features respectively according to the human frame coordinates, the object frame coordinates and the time sequence features;

[0012] The coordinates of each joint point of the human body are located according to the human body frame coordinates; and the human body skeleton features corresponding to the human body frame coordinates are extracted from the coordinates of each joint point.

[0013] Optionally, before obtaining the behavior semantic feature corresponding to the behavior category according to the behavior category of the frame, the method further includes:

[0014] Construct a mapping relationship between behavior categories and behavior semantic features;

[0015] Among them, the mapping relationship between behavior categories and behavior semantic features is constructed, including:

[0016] For each behavior category in the plurality of behavior categories, obtaining an action category semantic text, an action attribute semantic text, and a human body semantic text corresponding to the behavior category;

[0017] Extracting action category semantic features, action attribute semantic features and human body trunk semantic features from the action category semantic text, action attribute semantic text and human body trunk semantic text respectively;

[0018] The action category semantic features, action attribute semantic features and human body trunk semantic features are feature concatenated to obtain the behavior semantic features corresponding to the behavior category.

[0019] Optionally, constructing a visual motion feature according to the human visual feature and the object visual feature includes:

[0020] Obtaining a first interaction feature between a person and an object according to the visual features of the person and the visual features of the object;

[0021] Obtaining a second interaction feature between people according to human visual features of at least two people;

[0022] Obtaining a third interaction feature between the person and the at least two objects according to the human visual feature and the object visual features of the at least two objects;

[0023] The first interaction feature, the second interaction feature and the third interaction feature are cascaded to obtain the visual motion feature.

[0024] Optionally, constraining the visual motion feature and the skeletal motion feature according to the behavior semantic feature corresponding to each frame in each video data, and training to obtain a behavior recognition model, including:

[0025] Constructing a first symmetric cross entropy loss according to the visual motion feature and the behavioral semantic feature;

[0026] Constructing a second symmetric cross entropy loss according to the skeletal motion feature and the behavioral semantic feature;

[0027] Constructing a total loss according to the first symmetric cross entropy loss and the second symmetric cross entropy loss;

[0028] Iterative training is performed according to the total loss to obtain the behavior recognition model.

[0029] According to another aspect of an embodiment of the present invention, a behavior recognition method is provided, including:

[0030] Get target video data;

[0031] For each target frame in the target video data, extracting human visual features, object visual features and human skeleton features from the target frame;

[0032] Constructing visual motion features according to the human visual features and the object data features, and constructing skeletal motion features according to the human skeletal features and the object data features; constructing corresponding multiple behavioral semantic features based on multiple behavioral categories;

[0033] Inputting the visual motion feature, the skeletal motion feature and the multiple behavior semantic features into a behavior recognition model to obtain a target behavior category corresponding to the target frame;

[0034] The behavior recognition model is trained according to the training method of the behavior recognition model of the embodiment of the present invention.

[0035] Optionally, the visual motion feature, the skeletal motion feature and the plurality of behavior semantic features are input into a behavior recognition model to obtain a target behavior category corresponding to the target frame, including:

[0036] For the behavior semantic feature, calculating a first score result of the visual motion feature and the behavior semantic feature, and calculating a second score result of the skeletal motion feature and the behavior semantic feature; performing a weighted summation of the first score result and the second score result to obtain a score result corresponding to the behavior semantic feature;

[0037] The behavior category corresponding to the behavior semantic feature with the highest score result is used as the target behavior category.

[0038] According to another aspect of an embodiment of the present invention, a training device for a behavior recognition model is provided, comprising:

[0039] A first acquisition module acquires a plurality of video data and a behavior category corresponding to each frame in each video data;

[0040] The first extraction module extracts human visual features, object visual features and human skeleton features from each frame in each video data; and obtains behavior semantic features corresponding to the behavior category according to the behavior category of the frame;

[0041] A first construction module constructs visual motion features according to the human visual features and the object visual features, and constructs skeletal motion features according to the human skeletal features and the object visual features;

[0042] The training module trains a behavior recognition model based on the visual motion features, skeletal motion features and behavior semantic features corresponding to each frame in each video data.

[0043] According to another aspect of an embodiment of the present invention, there is provided a behavior recognition device, comprising:

[0044] The second acquisition module acquires target video data;

[0045] A second extraction module, for each target frame in the target video data, extracts human visual features, object visual features and human skeleton features from the target frame;

[0046] The second construction module constructs visual motion features according to the human visual features and the object data features, constructs skeletal motion features according to the human skeletal features and the object data features; and constructs corresponding multiple behavioral semantic features based on multiple behavior categories;

[0047] A recognition module, inputting the visual motion feature, the skeletal motion feature and the multiple behavior semantic features into a behavior recognition model to obtain a target behavior category corresponding to the target frame;

[0048] The behavior recognition model is trained according to the training method of the behavior recognition model of the embodiment of the present invention.

[0049] According to another aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0050] one or more processors;

[0051] a storage device for storing one or more programs,

[0052] When the one or more programs are executed by the one or more processors, the one or more processors implement the behavior recognition model training method or behavior recognition method provided by the present invention.

[0053] According to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the training method of the behavior recognition model or the behavior recognition method provided by the present invention is implemented.

[0054] One embodiment of the above invention has the following advantages or beneficial effects: the training method of the behavior recognition model of the embodiment of the present invention first obtains multiple video data and the behavior category of each frame in each video data, extracts human visual features and human skeleton features from each frame, and obtains the behavior semantic features corresponding to the behavior category; then constructs visual motion features according to human visual features and object visual features, constructs skeleton motion features according to human skeleton features and object visual features, constrains visual motion features and skeleton motion features according to behavior semantic features, and trains to obtain a behavior recognition model. The training method constrains the feature extraction and model training of visual motion features and skeleton motion features through behavior semantic features, thereby improving the detection accuracy and reliability of the behavior recognition model; the training method is based on the extraction of human visual features and human skeleton features, and constructs visual motion features and skeleton motion features to achieve action representation, improve the effectiveness of action modeling, and enable the model to understand the behavior action more accurately.

[0055] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0057] Figure 1 is a schematic diagram of the main process of a method for training a behavior recognition model according to an embodiment of the present invention;

[0058] Figure 2 is a schematic diagram of the main process of a behavior recognition method according to an embodiment of the present invention;

[0059] Figure 3 is a flow chart of a behavior recognition method according to an embodiment of the present invention;

[0060] Figure 4 is a schematic diagram of main modules of a training device for a behavior recognition model according to an embodiment of the present invention;

[0061] Figure 5 is a schematic diagram of main modules of a behavior recognition device according to an embodiment of the present invention;

[0062] Figure 6 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0063] Figure 7 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0065] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0066] At present, the existing technologies for identifying employee operation behaviors are mainly divided into motion detection methods based on spatiotemporal features, motion recognition methods based on skeletons, and motion detection methods based on language information enhancement. Among them, the motion detection method based on spatiotemporal features ignores the importance of posture features and does not pay attention to the fine-grained information of postures during the process of employees performing vehicle maintenance operations, because the employee's own body parts, the tools used, and the vehicle parts often occlude each other. The method based on visual features ignores the importance of posture features and does not pay attention to the fine-grained information of postures, resulting in difficulty in accurately judging whether the employee's actions are standardized and whether the operation process is correct in complex operation scenarios. The motion recognition method based on skeletons ignores the importance of interaction modeling and does not pay attention to the interaction features between staff and different people and objects during a series of vehicle maintenance operations, because it involves a variety of complex situations such as individual human behavior, human interaction, human interaction with a single object, and human interaction with two objects. The method based on skeleton features ignores the importance of interaction modeling and does not pay attention to the interaction features between staff and different people and objects, resulting in difficulty in accurately identifying and analyzing employee operation behaviors in actual vehicle maintenance scenarios. Although the action detection method based on language information enhancement has shown that collaborative training can learn powerful representation capabilities for behavior recognition tasks, current research only focuses on a single vision-language or skeleton-language aspect. How to achieve comprehensive enhancement and constraints among the three features of vision, language and skeleton has not been fully explored.

[0067] Based on the above technical problems, an embodiment of the present invention provides a behavior recognition method, which can improve the accuracy and reliability of behavior recognition, thereby improving work efficiency and quality.

[0068] Figure 1 is a schematic diagram of the main process of a method for training a behavior recognition model according to an embodiment of the present invention, such as Figure 1 As shown, the training method includes the following steps:

[0069] Step S101: Acquire multiple video data and the behavior category corresponding to each frame in each video data;

[0070] Step S102: for each frame in each video data, extracting human visual features, object visual features and human skeleton features from the frame; obtaining behavior semantic features corresponding to the behavior category according to the behavior category of the frame;

[0071] Step S103: constructing visual motion features according to human visual features and object visual features, and constructing skeletal motion features according to human skeletal features and object visual features;

[0072] Step S104: constraining the visual motion features and the skeletal motion features according to the behavior semantic features corresponding to each frame in each video data, and training to obtain a behavior recognition model.

[0073] In an embodiment of the present invention, during the training of the behavior recognition model, a plurality of video data are first obtained. The video data may be a vehicle maintenance monitoring video. Each video data is composed of a plurality of continuous frames, and the behavior category of each frame in each video data is obtained. The behavior category may be manually labeled. The behavior category is set according to specific business needs. For example, for vehicle maintenance operation behaviors, the behavior category may include a person scrubbing a vehicle, a person washing a car with a water gun, a person wiping a car body with a detergent, etc.

[0074] In the vehicle maintenance process, the visual features of people and objects describe the form, position and state changes of actions, which plays an important role in improving the accuracy of behavior recognition. In addition, in order to enhance the robustness of the model in the environment of monitoring perspective change, background noise, changing lighting conditions, etc., it is realized by extracting skeleton features. Therefore, for each frame in each video data, multimodal feature extraction is performed on the frame, including extracting human visual features, object visual features and human skeleton features.

[0075] In an embodiment of the present invention, extracting human visual features, object visual features and human skeleton features from a frame includes:

[0076] Extracting human frame coordinates and object frame coordinates from the frame, and extracting frame temporal features from the video data; obtaining human visual features and object visual features respectively according to the human frame coordinates, object frame coordinates and temporal features;

[0077] The coordinates of each joint point of the human body are located according to the human body frame coordinates; and the human body skeleton features corresponding to the human body frame coordinates are extracted from the coordinates of each joint point.

[0078] In an embodiment of the present invention, for each frame in the video data, the target detector YOLO can be first used to obtain the human frame coordinates and the object frame coordinates in the frame, and the number of the human frame coordinates and the object frame coordinates is one or more. Then, the video feature extraction algorithm VIT is used to extract the temporal features of the frame, and then the ROIAlign algorithm (Region of Interest Align, an improved regional feature extraction method) is used to obtain the human visual features according to the human frame coordinates and the temporal features, and the object visual features are obtained according to the object frame coordinates and the temporal features.

[0079] Then, based on the obtained human frame coordinates, the pose estimation model HRNet is used to obtain the coordinates of each joint point of the human body, so as to achieve accurate positioning of each joint of the human body. After the coordinates of each joint point of the human body corresponding to each human frame coordinate are input into the bone feature extraction algorithm GCN, the human bone features corresponding to each human frame coordinate in the frame are generated.

[0080] In the embodiment of the present invention, before obtaining the behavior semantic feature corresponding to the behavior category according to the behavior category of the frame, the method further includes:

[0081] Construct a mapping relationship between behavior categories and behavior semantic features;

[0082] Among them, the mapping relationship between behavior categories and behavior semantic features is constructed, including:

[0083] For each behavior category in the multiple behavior categories, obtaining an action category semantic text, an action attribute semantic text, and a human body semantic text corresponding to the behavior category;

[0084] Extracting action category semantic features, action attribute semantic features and human body trunk semantic features from action category semantic text, action attribute semantic text and human body trunk semantic text respectively;

[0085] The action category semantic features, action attribute semantic features and human body trunk semantic features are concatenated to obtain the behavior semantic features corresponding to the behavior category.

[0086] In an embodiment of the present invention, before obtaining the behavior semantic features corresponding to the behavior categories according to the behavior categories of the frame, a mapping relationship between the behavior categories and the behavior semantic features is constructed, so that the behavior semantic features corresponding to the behavior categories of the frame can be obtained according to the mapping relationship. Specifically, first, for each behavior category, different semantic descriptions can be generated for each behavior category through GPT-3, such as generating action category semantic texts, action attribute semantic texts, and human body trunk semantic texts corresponding to the behavior categories. Then, the pre-trained Transformer-based language model CLIP is used to extract action category semantic features, action attribute semantic features, and human body trunk semantic features from the action category semantic texts, action attribute semantic texts, and human body trunk semantic texts, respectively.

[0087] For example, the semantic feature extraction for action categories is as follows:

[0088] ,

[0089] in, is the description of the action category, which describes the meaning of the action category. For example, the semantic description of the action category of a person scrubbing a vehicle is: Worker uses water guns to scrub vehicle. C is the number of employee operation action categories to be detected. It is a model used to extract semantic features of action categories.

[0090] The specific extraction of action attribute semantic features is as follows:

[0091] ,

[0092] in, is the description of the action attributes, which describes the total number of people and objects involved in the action, whether assistance is needed, and the three key steps to complete the operation. For example, the semantic description of the action attribute of a person scrubbing a vehicle is: 2entities, no assistance needed, pick up water guns, scrub the vehicles surface, check the cleaning effect. C is the number of employee operation action categories to be detected. It is a model used to extract semantic features of action attributes.

[0093] The specific semantic feature extraction of human torso is as follows:

[0094] ,

[0095] in, is the description of the human torso, which describes the three main body movements involved in the action, such as the semantic description of the human torso of a person scrubbing a car is: Standing, bending over, arm extension and swinging. C is the number of employee operation action categories to be detected. A model for extracting semantic features of the human torso.

[0096] After obtaining the semantic features of the action category, the semantic features of the action attribute, and the semantic features of the human body, feature cascading is performed to obtain the behavior semantic features corresponding to the behavior category. For example, the behavior semantic features of the behavior category obtained are extracted as follows:

[0097]

[0098] in, , as well as are the action category semantic features, action attribute semantic features and human body semantic features captured above. C is the number of employee operation action categories to be detected.

[0099] In the embodiment of the present invention, the behavior semantic features corresponding to each behavior category can be obtained according to the mapping relationship between the behavior category and the behavior semantic features.

[0100] In the embodiment of the present invention, constructing visual motion features according to human visual features and object visual features includes:

[0101] Obtaining the first interaction feature between a person and an object according to the visual features of the human body and the visual features of the object;

[0102] Obtaining a second interaction feature between people according to human visual features of at least two people;

[0103] Obtaining a third interaction feature between the person and the at least two objects according to the human visual feature and the object visual features of the at least two objects;

[0104] The first interaction feature, the second interaction feature and the third interaction feature are cascaded to obtain the visual motion feature.

[0105] In the embodiment of the present invention, during the vehicle maintenance process, many interactive behaviors are involved, among which the key ones are mainly human-to-human interactions (such as communication between people and customers, collaboration between workers), human-to-object interactions (such as people opening car doors, people operating jacks), and simultaneous interactions between people and objects 1 and 2 (such as people washing cars with water guns, people wiping car bodies with detergents). Through a single action modeling method, a single action modeling method, such as a human interaction modeling method, cannot fully and comprehensively characterize various behaviors. Therefore, the features of different interactive behaviors can be extracted and modeled according to human visual features and object visual features, and feature cascading can be performed to obtain visual motion features.

[0106] Specifically, human-object interaction relationship extraction is mainly used to identify employees' behaviors in operating objects. The model needs to focus on and extract specific objects and their states that are closely related to employees' operations from the many objects in the current scene, so as to effectively understand the interaction patterns between employees and objects.

[0107] The Non-Local Block model can more efficiently capture the dependencies between features and select other features that have a high activation effect on the target feature by means of the cross attention mechanism, and fuse them to achieve the combination of two sets of features. In addition, a large amount of computing resources will not be consumed in the process. Therefore, the embodiment of the present invention adopts the Non-LocalBlock model to extract interactive features.

[0108] The human visual features and object visual features are input into the Non-Local Block model to obtain the first interaction features between humans and objects, wherein the human visual features are the input of the query vector of the Non-Local Block model, the object visual features are the input of the key and value vector of the Non-Local Block model, and the first interaction features between humans and objects are the output of the Non-LocalBlock model.

[0109] Interpersonal interaction relationship extraction is mainly used to accurately identify various types of interactive behaviors between employees and other people. In this process, the model not only needs to accurately identify the specific people involved in the interaction, but also needs to clearly extract the discriminative interaction features between the two people to enhance the effect of action recognition.

[0110] The inputs of the query vector, key and value vectors of the Non-Local Block model are all human visual features, and the output is the second interaction feature between people.

[0111] The extraction of the interaction between a person and two objects at the same time mainly focuses on the situation where the employee uses the operation behavior of object 1 to realize the operation of object 2. This requires the model to pay attention to the two objects being operated at the same time and to extract the continuous interaction features between the person and the two objects.

[0112] At this time, two Non-Local Block models are used to extract the object visual features of the person and at least two objects in parallel, that is, the human visual features and the object visual features of object 1 are input into the Non-Local Block model to obtain the interaction features between the person and object 1; the human visual features and the object visual features of object 2 are input into the Non-Local Block model to obtain the interaction features between the person and object 2. Then, a set of learnable convolutions is used to realize the fusion of the two extracted human interaction features, thereby obtaining the third interaction features between the person and at least two objects.

[0113] The third interaction feature obtained is shown as follows:

[0114]

[0115] Among them, j represents the two objects to be interacted, W is a set of learnable weights used to learn the fusion weights between two human-object interaction features, and VisFea_HOO represents the extracted interaction features between human and object 1 and object 2, that is, the third interaction feature.

[0116] After obtaining the first interaction feature, the second interaction feature and the third interaction feature, the three features are cascaded, that is, feature splicing is performed to obtain the visual motion feature.

[0117] Similarly, skeleton motion features are constructed based on human skeleton features and object visual features. Specifically, a first interaction feature between people and objects is obtained based on human skeleton features and object visual features, a second interaction feature between people is obtained based on human skeleton features of at least two people, and a third interaction feature between people and at least two objects is obtained based on human skeleton features and object visual features of at least two objects. These three features are cascaded to extract skeleton motion features.

[0118] The human skeleton features and object visual features are input into the Non-Local Block model to obtain the first interaction features between people and objects, wherein the human skeleton features are the input of the query vector of the Non-Local Block model, the object visual features are the input of the key and value vector of the Non-Local Block model, and the first interaction features between people and objects are the output of the Non-LocalBlock model.

[0119] The inputs of the query vector, key and value vectors of the Non-Local Block model are all human skeleton features, and the output is the second interaction feature between people.

[0120] Two Non-Local Block models are used to extract the object visual features of a person and at least two objects in parallel, that is, the human skeleton features and the object visual features of object 1 are input into the Non-Local Block model to obtain the interaction features between the person and object 1; the human skeleton features and the object visual features of object 2 are input into the Non-Local Block model to obtain the interaction features between the person and object 2. Then, a set of learnable convolutions is used to realize the fusion of the two extracted human interaction features, so as to obtain the third interaction features between the person and at least two objects.

[0121] After obtaining the first interaction feature, the second interaction feature and the third interaction feature, the three features are cascaded, that is, feature splicing is performed to obtain the skeletal motion feature.

[0122] The language description of maintenance operations by employees contains the knowledge of the gradual changes in the states of people and objects involved in the maintenance operations. These knowledge features can enable the model to learn in a fine-grained manner, thereby more accurately extracting discriminative behavior classification features. Bidirectional learning between vision and skeleton and language can constrain visual motion features and skeleton motion features through the behavioral semantic features corresponding to each frame in each video data. The behavior recognition model is trained.

[0123] In the embodiment of the present invention, visual motion features and skeletal motion features are constrained according to the behavior semantic features corresponding to each frame in each video data, and a behavior recognition model is obtained by training, including:

[0124] The first symmetric cross entropy loss is constructed based on visual motion features and behavioral semantic features;

[0125] Construct the second symmetric cross entropy loss based on the skeletal motion features and behavioral semantic features;

[0126] Construct the total loss based on the first symmetric cross entropy loss and the second symmetric cross entropy loss;

[0127] Iterative training is performed according to the total loss to obtain the behavior recognition model.

[0128] In an embodiment of the present invention, a first symmetric cross entropy loss is constructed according to the visual motion feature and the behavioral semantic feature. The first symmetric cross entropy loss is used to maximize the similarity between the visual motion feature and the matching behavioral semantic feature and minimize the similarity between the visual motion feature and the unmatched behavioral semantic feature. The first symmetric cross entropy loss is constructed by calculating the KL divergence, for example,

[0129]

[0130]

[0131] Where LossVis is the first symmetric cross entropy loss between visual motion features and behavioral semantic features, B is the size of a batch in training. s() represents the cosine similarity. Tem is the temperature hyperparameter used for scaling. k∈K(i) ={k|k ∈ [1,B], yk = yi}.

[0132] The second symmetric cross entropy loss is constructed based on the skeletal motion features and the behavioral semantic features. The second symmetric cross entropy loss is used to maximize the similarity between the skeletal motion features and the matching behavioral semantic features, and minimize the similarity between the skeletal motion features and the unmatched behavioral semantic features. The second symmetric cross entropy loss is constructed by calculating the KL divergence. For example,

[0133]

[0134] Where B is the size of a batch in training. s() represents the cosine similarity. Tem is the temperature hyperparameter used for scaling. k∈K(i) = {k|k ∈ [1,B], yk = yi}. LossSke is the second symmetric cross entropy loss between human skeleton features and behavioral semantic features.

[0135] During the training process, in order to obtain the synchronous constraints of behavioral semantic features on visual motion features and skeletal motion features, the first symmetric cross entropy loss and the second symmetric cross entropy loss are fused to train the behavior recognition model. For example, the two can be directly added to obtain the total loss, and the behavior recognition model is obtained by iteratively training the total loss.

[0136] The training method of the behavior recognition model of the embodiment of the present invention first obtains multiple video data and the behavior category of each frame in each video data, extracts human visual features and human skeleton features from each frame, and obtains the behavior semantic features corresponding to the behavior category; then constructs visual motion features according to human visual features and object visual features, constructs skeleton motion features according to human skeleton features and object visual features, constrains the visual motion features and skeleton motion features according to the behavior semantic features, and trains to obtain the behavior recognition model. The training method constrains the feature extraction and model training of visual motion features and skeleton motion features by using the behavior semantic features to improve the detection accuracy and reliability of the behavior recognition model; the training method is based on the extraction of human visual features and human skeleton features, and constructs visual motion features and skeleton motion features to achieve action representation, improve the effectiveness of action modeling, and enable the model to understand the behavior action more accurately.

[0137] According to another aspect of the embodiment of the present invention, Figure 2 As shown, a behavior recognition method is provided, including:

[0138] Step S201: Acquire target video data;

[0139] Step S202: for each target frame in the target video data, extracting human visual features, object visual features and human skeleton features from the target frame;

[0140] Step S203: constructing visual motion features according to human visual features and object data features, and constructing skeletal motion features according to human skeletal features and object data features; and constructing corresponding multiple behavior semantic features based on multiple behavior categories respectively;

[0141] Step S204: inputting the visual motion features, the skeletal motion features and the multiple behavior semantic features into the behavior recognition model to obtain the target behavior category corresponding to the target frame;

[0142] The behavior recognition model is trained according to the training method of the behavior recognition model of the embodiment of the present invention.

[0143] In an embodiment of the present invention, the behavior recognition model obtained by training can be used to identify the behavior in the target visual data to obtain the behavior category of each target frame in the target video data. First, multimodal feature extraction is performed on the target frame to extract human visual features, object visual features and human skeletal features. Specifically, the human frame coordinates and object frame coordinates in the target frame are obtained, such as by using the target detector YOLO; then the temporal features of the target frame are extracted, such as by using the video feature extraction algorithm VIT; then the human visual features are obtained according to the human frame coordinates and the temporal features, and the object visual features are obtained according to the object frame coordinates and the temporal features, which can be achieved by the ROIAlign algorithm. The coordinates of each joint point of the human body are located according to the human frame coordinates, which can be achieved by the posture estimation model HRNet; then the human skeletal features are obtained according to the coordinates of each joint point of the human body, and the human skeletal features can be obtained by inputting the joint point coordinates into the skeletal feature extraction algorithm GCN.

[0144] Then, visual motion features are constructed according to the visual features of the human body and the visual features of the object. Specifically, a first interaction feature between a person and an object is obtained according to the visual features of the human body and the visual features of the object, and a second interaction feature between a person and a person is obtained according to the visual features of the human body of at least two people; a third interaction feature between a person and at least two objects is obtained according to the visual features of the human body and the visual features of at least two objects; the first interaction feature, the second interaction feature and the third interaction feature are feature cascaded to obtain a visual motion feature.

[0145] Skeletal motion features are constructed based on human skeleton features and object visual features. Specifically, a first interaction feature between a person and an object is obtained based on the human skeleton features and object visual features, and a second interaction feature between a person and a person is obtained based on the human skeleton features of at least two people; a third interaction feature between a person and at least two objects is obtained based on the human skeleton features and object visual features of at least two objects; and the first interaction feature, the second interaction feature and the third interaction feature are cascaded to obtain a skeleton motion feature.

[0146] In an embodiment of the present invention, visual motion features, skeletal motion features and multiple behavior semantic features are input into a behavior recognition model to obtain a target behavior category corresponding to a target frame, including:

[0147] For the behavior semantic feature, a first score result of the visual motion feature and the behavior semantic feature is calculated, and a second score result of the skeletal motion feature and the behavior semantic feature is calculated; the first score result and the second score result are weighted summed to obtain a score result corresponding to the behavior semantic feature;

[0148] The behavior category corresponding to the behavior semantic feature with the highest score is taken as the target behavior category.

[0149] In an embodiment of the present invention, after obtaining the visual motion feature and the skeletal motion feature, for each behavioral semantic feature, a first score result of the visual motion feature and the behavioral semantic feature is calculated, and the first score result can be the cosine similarity of the visual motion feature and the behavioral semantic feature; a second score result of the skeletal motion feature and the behavioral semantic feature is calculated, and the second score result can be the cosine similarity of the skeletal motion feature and the behavioral semantic feature; the first score result and the second score result are weighted and summed to obtain the score result of the behavioral semantic feature. The calculation formula is as follows:

[0150]

[0151] Among them, Rrsule is the scoring result, s(FeaVis, LangAct) is the cosine similarity between visual motion features and behavioral semantic features, which is the first scoring result, s(FeaSke, LangAct) is the cosine similarity between skeletal motion features and behavioral semantic features, which is the second scoring result, and α is the weight.

[0152] The behavior recognition method of the embodiment of the present invention first obtains target video data, and for each target frame in the target video data, extracts human visual features, object visual features, and human skeletal features from the target frame, then constructs visual motion features based on human visual features and object visual features, constructs skeletal motion features based on human skeletal features and object visual features, constructs behavior semantic features corresponding to each behavior category based on multiple behavior categories, and then inputs the visual motion features, skeletal motion features, and multiple behavior semantic features into a behavior recognition model to obtain the behavior category of the target frame. The behavior recognition method performs behavior recognition based on the behavior recognition model, and can solve the problem of the difficulty of behavior recognition under the condition of cluttered information and difficult feature extraction. Through intelligent behavior recognition and analysis, the efficiency and accuracy of behavior recognition are improved, thereby improving work efficiency and quality, standardizing operation behaviors and work processes, and preventing mistakes and accidents.

[0153] Figure 3A schematic diagram of the framework of a behavior recognition method according to an embodiment of the present invention. Obtain a vehicle maintenance monitoring video; perform multimodal feature extraction on the video, specifically, extract the temporal features of the video, extract the human frame coordinates and object frame coordinates of each frame, and obtain human visual features, object visual features, and human skeletal features based on the temporal features, human frame coordinates, and object frame coordinates through the RoiAlign algorithm; perform semantic feature extraction for each behavior category, extract action category semantic features using a text encoder based on category description, extract action attribute semantic features using a text encoder based on attribute description, extract human torso semantic features using a text encoder based on human body part description, and then perform feature cascading of action category semantic features, action attribute semantic features, and human torso semantic features to obtain maintenance behavior semantic features; a visual-level maintenance behavior capture module based on semantic constraints, according to the human body Visual features and object visual features are used to extract visual-level human-object interaction features, human-to-human interaction features, and human-to-two-object simultaneous interaction features. After feature cascading, visual-level action features, namely visual motion features, are obtained, and the visual motion features are constrained by behavioral semantic features. Similarly, a skeletal-level maintenance behavior capture module based on semantic constraints extracts skeletal-level human-object interaction features, human-to-human interaction features, and human-to-two-object simultaneous interaction features based on human skeletal features and object visual features. After feature cascading, skeletal-level action features, namely skeletal motion features, are obtained, and the skeletal motion features are constrained by behavioral semantic features. Behavior recognition is performed based on the behavior recognition model trained with semantic-based visual feature constraints and semantic-based skeletal feature constraints to obtain the behavior category of the current frame.

[0154] According to another aspect of the embodiment of the present invention, Figure 4 As shown, a training device 400 for a behavior recognition model is provided, comprising:

[0155] A first acquisition module 401 acquires a plurality of video data and a behavior category corresponding to each frame in each video data;

[0156] The first extraction module 402 extracts human visual features, object visual features and human skeleton features from each frame in each video data; and obtains behavior semantic features corresponding to the behavior category according to the behavior category of the frame;

[0157] A first construction module 403 constructs visual motion features according to human visual features and object visual features, and constructs skeletal motion features according to human skeletal features and object visual features;

[0158] The training module 404 trains a behavior recognition model according to the visual motion features, skeletal motion features and behavior semantic features corresponding to each frame in each video data.

[0159] In the embodiment of the present invention, the first extraction module 402 is further used to:

[0160] Extracting human frame coordinates and object frame coordinates from the frame, and extracting frame temporal features from the video data; obtaining human visual features and object visual features respectively according to the human frame coordinates, object frame coordinates and temporal features;

[0161] The coordinates of each joint point of the human body are located according to the human body frame coordinates; and the human body skeleton features corresponding to the human body frame coordinates are extracted from the coordinates of each joint point.

[0162] In the embodiment of the present invention, the first extraction module 402 is further configured to: before obtaining the behavior semantic features corresponding to the behavior category according to the behavior category of the frame,

[0163] Construct a mapping relationship between behavior categories and behavior semantic features;

[0164] Among them, constructing the mapping relationship between behavior categories and behavior semantic features includes:

[0165] For each behavior category in the multiple behavior categories, obtaining an action category semantic text, an action attribute semantic text, and a human body semantic text corresponding to the behavior category;

[0166] Extracting action category semantic features, action attribute semantic features and human body trunk semantic features from action category semantic text, action attribute semantic text and human body trunk semantic text respectively;

[0167] The action category semantic features, action attribute semantic features and human body trunk semantic features are concatenated to obtain the behavior semantic features corresponding to the behavior category.

[0168] In the embodiment of the present invention, the first construction module 403 is further used to: obtain a first interaction feature between a person and an object according to a visual feature of a human body and a visual feature of an object;

[0169] Obtaining a second interaction feature between people according to human visual features of at least two people;

[0170] Obtaining a third interaction feature between the person and the at least two objects according to the human visual feature and the object visual features of the at least two objects;

[0171] The first interaction feature, the second interaction feature and the third interaction feature are cascaded to obtain the visual motion feature.

[0172] In this embodiment of the present invention, the training module 404 is further used to: construct a first symmetric cross entropy loss based on visual motion features and behavioral semantic features; construct a second symmetric cross entropy loss based on skeletal motion features and behavioral semantic features; construct a total loss based on the first symmetric cross entropy loss and the second symmetric cross entropy loss; perform iterative training based on the total loss to obtain a behavior recognition model.

[0173] According to another aspect of the embodiment of the present invention, Figure 5 As shown, a behavior recognition device 500 is provided, comprising:

[0174] The second acquisition module 501 acquires target video data;

[0175] The second extraction module 502 extracts human visual features, object visual features and human skeleton features from each target frame in the target video data;

[0176] The second construction module 503 constructs visual motion features according to human visual features and object data features, constructs skeletal motion features according to human skeletal features and object data features; and constructs corresponding multiple behavior semantic features based on multiple behavior categories;

[0177] The recognition module 504 inputs the visual motion features, the skeletal motion features and the multiple behavior semantic features into the behavior recognition model to obtain the target behavior category corresponding to the target frame;

[0178] The behavior recognition module is trained according to the training method of the behavior recognition model of the embodiment of the present invention.

[0179] In the embodiment of the present invention, the identification module 504 is further configured to:

[0180] For the behavior semantic feature, a first score result of the visual motion feature and the behavior semantic feature is calculated, and a second score result of the skeletal motion feature and the behavior semantic feature is calculated; the first score result and the second score result are weighted summed to obtain a score result corresponding to the behavior semantic feature;

[0181] The behavior category corresponding to the behavior semantic feature with the highest score is taken as the target behavior category.

[0182] According to another aspect of an embodiment of the present invention, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the training method of the behavior recognition model or the behavior recognition method provided by the present invention.

[0183] According to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the training method of the behavior recognition model or the behavior recognition method provided by the present invention is implemented.

[0184] Figure 6 An exemplary system architecture 600 of a behavior recognition model training method or behavior recognition model training, as well as a behavior recognition method or a behavior recognition device to which the embodiments of the present invention may be applied is shown.

[0185] like Figure 6 As shown, system architecture 600 may include terminal devices 601, 602, 603, network 604 and server 605. Network 604 is used to provide a medium for communication links between terminal devices 601, 602, 603 and server 605. Network 604 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0186] The user can use the terminal devices 601, 602, 603 to interact with the server 605 through the network 604 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 601, 602, 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0187] The terminal devices 601 , 602 , and 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0188] The server 605 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 601, 602, and 603. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only an example) to the terminal device.

[0189] It should be noted that the training method or behavior recognition method of the behavior recognition model provided in the embodiment of the present invention is generally executed by the server 605 , and accordingly, the training device or behavior recognition device of the behavior recognition model is generally arranged in the server 605 .

[0190] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0191] Reference below Figure 7, which shows a schematic diagram of the structure of a computer system 700 of a terminal device suitable for implementing an embodiment of the present invention. Figure 7 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0192] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0193] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0194] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0195] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0196] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0197] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be arranged in a processor. For example, they may be described as follows: a processor includes a first acquisition module, a first extraction module, a first construction module, and a training module. The names of these modules do not, in some cases, constitute limitations on the modules themselves. For example, the first acquisition module may also be described as a "module for acquiring multiple video data and a behavior category corresponding to each frame in each video data."

[0198] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: obtaining multiple video data and the behavior category corresponding to each frame in each video data; for each frame in each video data, extracting human visual features, object visual features and human skeletal features from the frame; obtaining behavior semantic features corresponding to the behavior category according to the behavior category of the frame; constructing visual motion features according to human visual features and object visual features, and constructing skeletal motion features according to human skeletal features and object visual features; constraining visual motion features and skeletal motion features according to the behavior semantic features corresponding to each frame in each video data, and training to obtain a behavior recognition model.

[0199] According to the technical solution of the embodiment of the present invention, the training method of the behavior recognition model of the embodiment of the present invention first obtains multiple video data and the behavior category of each frame in each video data, extracts human visual features and human skeleton features from each frame, and obtains the behavior semantic features corresponding to the behavior category; then constructs visual motion features according to human visual features and object visual features, constructs skeleton motion features according to human skeleton features and object visual features, constrains visual motion features and skeleton motion features according to behavior semantic features, and trains to obtain a behavior recognition model. The training method constrains the feature extraction and model training of visual motion features and skeleton motion features by using behavior semantic features to improve the detection accuracy and reliability of the behavior recognition model; the training method is based on the extraction of human visual features and human skeleton features, and constructs visual motion features and skeleton motion features to achieve action representation, improve the effectiveness of action modeling, and enable the model to understand the behavior action more accurately.

[0200] The behavior recognition method of the embodiment of the present invention first obtains target video data, and for each target frame in the target video data, extracts human visual features, object visual features, and human skeletal features from the target frame, then constructs visual motion features based on human visual features and object visual features, constructs skeletal motion features based on human skeletal features and object visual features, constructs behavior semantic features corresponding to each behavior category based on multiple behavior categories, and then inputs the visual motion features, skeletal motion features, and multiple behavior semantic features into a behavior recognition model to obtain the behavior category of the target frame. The behavior recognition method performs behavior recognition based on the behavior recognition model, and can solve the problem of the difficulty of behavior recognition under the condition of cluttered information and difficult feature extraction. Through intelligent behavior recognition and analysis, the efficiency and accuracy of behavior recognition are improved, thereby improving work efficiency and quality, standardizing operation behaviors and work processes, and preventing mistakes and accidents.

[0201] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for training a behavior recognition model, characterized in that: include: Acquire multiple video data and the behavior category corresponding to each frame in each video data; For each frame in each video data, human visual features, object visual features and human skeleton features are extracted from the frame; and behavior semantic features corresponding to the behavior category are obtained according to the behavior category of the frame; Constructing visual motion features according to the human visual features and the object visual features, and constructing skeletal motion features according to the human skeletal features and the object visual features; The visual motion features and the skeletal motion features are constrained according to the behavior semantic features corresponding to each frame in each video data, and a behavior recognition model is obtained by training.

2. The method according to claim 1, characterized in that Extracting human visual features, object visual features and human skeleton features from the frame, including: Extracting human frame coordinates and object frame coordinates from the frame, and extracting the time sequence features of the frame from the video data; obtaining the human visual features and the object visual features respectively according to the human frame coordinates, the object frame coordinates and the time sequence features; The coordinates of each joint point of the human body are located according to the human body frame coordinates; and the human body skeleton features corresponding to the human body frame coordinates are extracted from the coordinates of each joint point.

3. The method according to claim 1, characterized in that: Before obtaining the behavior semantic feature corresponding to the behavior category according to the behavior category of the frame, the method further includes: Construct a mapping relationship between behavior categories and behavior semantic features; Among them, constructing the mapping relationship between behavior categories and behavior semantic features includes: For each behavior category in the plurality of behavior categories, obtaining an action category semantic text, an action attribute semantic text, and a human body semantic text corresponding to the behavior category; Extracting action category semantic features, action attribute semantic features and human body trunk semantic features from the action category semantic text, action attribute semantic text and human body trunk semantic text respectively; The action category semantic features, action attribute semantic features and human body trunk semantic features are feature concatenated to obtain the behavior semantic features corresponding to the behavior category.

4. The method according to claim 1, characterized in that: Constructing visual motion features according to the human visual features and the object visual features, including: Obtaining a first interaction feature between a person and an object according to the visual features of the person and the visual features of the object; Obtaining a second interaction feature between people according to human visual features of at least two people; Obtaining a third interaction feature between the person and the at least two objects according to the human visual feature and the object visual features of the at least two objects; The first interaction feature, the second interaction feature and the third interaction feature are cascaded to obtain the visual motion feature.

5. The method according to claim 1, characterized in that: The visual motion feature and the skeletal motion feature are constrained according to the behavior semantic feature corresponding to each frame in each video data, and a behavior recognition model is obtained by training, including: Constructing a first symmetric cross entropy loss according to the visual motion feature and the behavioral semantic feature; Constructing a second symmetric cross entropy loss according to the skeletal motion feature and the behavioral semantic feature; Constructing a total loss according to the first symmetric cross entropy loss and the second symmetric cross entropy loss; Iterative training is performed according to the total loss to obtain the behavior recognition model.

6. A behavior recognition method, characterized in that: include: Get target video data; For each target frame in the target video data, extracting human visual features, object visual features and human skeleton features from the target frame; Constructing visual motion features according to the human visual features and the object data features, and constructing skeletal motion features according to the human skeletal features and the object visual features; Based on multiple behavior categories, corresponding multiple behavior semantic features are constructed respectively; Inputting the visual motion feature, the skeletal motion feature and the multiple behavior semantic features into a behavior recognition model to obtain a target behavior category corresponding to the target frame; Wherein, the behavior recognition model is trained according to any one of the methods of claims 1-5.

7. The method according to claim 6, characterized in that Inputting the visual motion feature, the skeletal motion feature and the multiple behavior semantic features into a behavior recognition model to obtain a target behavior category corresponding to the target frame includes: For the behavior semantic feature, calculating a first score result of the visual motion feature and the behavior semantic feature, and calculating a second score result of the skeletal motion feature and the behavior semantic feature; performing a weighted summation of the first score result and the second score result to obtain a score result corresponding to the behavior semantic feature; The behavior category corresponding to the behavior semantic feature with the highest score result is used as the target behavior category.

8. A training device for a behavior recognition model, characterized in that: include: A first acquisition module acquires a plurality of video data and a behavior category corresponding to each frame in each video data; The first extraction module extracts human visual features, object visual features and human skeleton features from each frame in each video data; and obtains behavior semantic features corresponding to the behavior category according to the behavior category of the frame; A first construction module constructs visual motion features according to the human visual features and the object visual features, and constructs skeletal motion features according to the human skeletal features and the object visual features; The training module trains a behavior recognition model based on the visual motion features, skeletal motion features and behavior semantic features corresponding to each frame in each video data.

9. A behavior recognition device, characterized in that: include: The second acquisition module acquires target video data; A second extraction module, for each target frame in the target video data, extracts human visual features, object visual features and human skeleton features from the target frame; A second construction module constructs visual motion features according to the human visual features and the object data features, and constructs skeletal motion features according to the human skeletal features and the object data features; Based on multiple behavior categories, corresponding multiple behavior semantic features are constructed respectively; A recognition module, inputting the visual motion feature, the skeletal motion feature and the multiple behavior semantic features into a behavior recognition model to obtain a target behavior category corresponding to the target frame; Wherein, the behavior recognition model is trained according to any one of the methods of claims 1-5.

10. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

11. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images