Human behavior perception method, device and program product based on query

Through the query-based human behavior perception method, the pre-trained model and action decoder are used to process video data, and the problems of low versatility and scalability in the existing technology are solved, achieving unified processing and performance improvement of multi-tasks.

CN117994852BActive Publication Date: 2024-11-12BEIJING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410229510.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-02-29
Publication Date
2024-11-12
Estimated Expiration
2044-02-29

AI Technical Summary

Technical Problem

The existing human behavior perception methods have low versatility and scalability when dealing with multitasking, resulting in the need to design and develop models separately for each task, which increases R&D costs and limits the universality and applicability of the models. .

Method used

Using query-based human behavior perception method, by obtaining a pre-trained human behavior perception model, inputting video data into the model, using a video encoder to extract visual features, and correlating it with preset query through action decoder, optimizing different task goals and realizing Joint optimization across tasks to improve the universality and scalability of the model.

Benefits of technology

The unified processing of multiple tasks in the field of human behavior perception is realized, which improves the universality and scalability of the model, reduces the number of parameters, reduces the risk of overfitting, and improves the performance of the model on each task.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117994852B_ABST
    Figure CN117994852B_ABST
Patent Text Reader

Abstract

The present invention provides a method, device and program product for human behavior perception based on query, the method comprising: obtaining a pre-trained human behavior perception model, the human behavior perception model comprising a video encoder and an action decoder; the video encoder is used to extract visual features corresponding to video data; the action decoder is used to associate the visual features with at least one preset query as a query key-value pair, optimize at least one preset query and output the result corresponding to the query task; the video data is input into the human behavior perception model to obtain the human behavior perception result. The present invention can realize cross-task joint optimization, complete multiple tasks in the field of human behavior perception, and improve versatility and scalability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a query-based human behavior perception method, device and program product. Background Art

[0002] Human behavior perception is an important topic in computer vision. Usually, human behavior perception aims to locate, segment, classify and evaluate human behavior in a given video to achieve understanding and recognition of human behavior, involving action recognition, action segmentation, action location and action evaluation. By accurately analyzing and understanding human postures, actions and behaviors in video data, it can be applied to many fields, such as video surveillance, human-computer interaction, video content analysis, etc., and has important application prospects.

[0003] Existing methods for perceiving human behavior, including two-stage methods or single-stage methods for action localization, algorithms based on Temporal Convolutional Network (TCN), algorithms based on Graph Neural Network (GNN), and algorithms based on Transformer for action segmentation, can only handle one task for different human behavior tasks, and use different processing methods for different tasks. One model can only complete a specific human behavior perception, and the degree of unification and standardization is relatively general. Although it can cope with diverse task requirements to a certain extent, it cannot achieve effective integration and collaborative work between multiple tasks. In practical applications, designing and developing corresponding models and algorithms for each task separately not only increases the R&D cost, but also limits the versatility and scalability of the model, and there is a problem of low versatility and scalability of the algorithm. Summary of the invention

[0004] In view of this, the embodiments of the present invention provide a method, device and program product for human behavior perception based on query to eliminate or improve one or more defects existing in the prior art. It can realize cross-task joint optimization, complete multiple tasks in the field of human behavior perception, and improve versatility and scalability.

[0005] One aspect of the present invention provides a method for perceiving human behavior based on a query, the method comprising the following steps:

[0006] Obtain a pre-trained human behavior perception model, the human behavior perception model comprising a video encoder and an action decoder; the video encoder is used to extract visual features corresponding to video data; the action decoder is used to associate the visual features with at least one preset query as a query key-value pair, optimize the at least one preset query and output a result corresponding to the query task;

[0007] The video data is input into the human behavior perception model to obtain the human behavior perception results.

[0008] Optionally, the video encoder includes a convolutional network layer, a transformer network layer and an encoder layer; the video data includes a video sequence or a feature of the video sequence; when the video data is a feature of the video sequence, extracting visual features corresponding to the video data includes:

[0009] Expand each feature into multiple dimensions through convolutional network layers;

[0010] The features processed by the convolutional network layer are mapped to the output feature pyramid through the transformer network layer;

[0011] The feature pyramid is interactively processed through the encoding layer to obtain visual features.

[0012] Optionally, when the video data is a video sequence, before expanding each feature into multiple dimensions through a convolutional network layer, it also includes: extracting features corresponding to the video sequence through a convolutional network layer.

[0013] Optionally, at least one preset query includes a category query, a boundary query and / or a mask query; the query task includes an action category query task, an action location query task, an action segmentation query task and / or an action evaluation query task;

[0014] The action category query task refers to the interaction between the category query and the visual features in the decoding layer to obtain the predicted action category;

[0015] The action localization query task refers to the interaction between category query, boundary query and visual features in the decoding layer to obtain the predicted time segment and corresponding action category;

[0016] The action segmentation query task refers to the predicted action segmentation fragments obtained by interacting boundary query, mask query, category query and visual features in the decoding layer;

[0017] The action evaluation query task refers to the interaction between category query and visual features in the decoding layer to obtain the predicted action performance score.

[0018] Optionally, the action decoder includes a preset number of decoder layers, takes the visual features as query key-value pairs, associates them with preset queries, optimizes the preset queries and outputs human action perception results, including:

[0019] Cross-attending visual features in each decoding layer;

[0020] Self-attention is performed between category queries, boundary queries and mask queries, and associations and interactions are established between category queries, boundary queries and mask queries to output human behavior perception results.

[0021] Optionally, before inputting the features corresponding to the video sequence into the human behavior perception model to obtain the human behavior perception result, it also includes: shielding category query, boundary query and / or mask query based on the query task.

[0022] Optionally, obtain a pre-trained human behavior perception model, including:

[0023] Obtaining training data corresponding to each query task; the training data includes: sample video data and standard output results of each query task corresponding to the sample video data;

[0024] The visual features of the sample video data are extracted by the backbone network corresponding to each query task to obtain the sample visual features; the sample visual features extracted by the backbone networks corresponding to different query tasks have the same shape;

[0025] The sample visual features are input into the initial human perception model, and the action decoder and each query task are jointly pre-trained to obtain a joint training result; the initial human perception model has the same model structure as the human perception model;

[0026] Input the joint training results and the standard output results into the preset loss function corresponding to each query task to obtain the loss function value;

[0027] The initial human perception model is iteratively trained using the loss function value to obtain the human perception model.

[0028] Another aspect of the present invention provides a query-based human behavior perception device, including a processor, a memory, and a computer program / instructions stored in the memory, characterized in that the processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the device implements the steps of the above-mentioned query-based human behavior perception method.

[0029] Another aspect of the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon, wherein the computer program / instruction, when executed by a processor, implements the steps of the above-mentioned query-based human behavior perception method.

[0030] Another aspect of the present invention provides a computer program product, including a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the steps of the above-mentioned query-based human behavior perception method are implemented.

[0031] The query-based human behavior perception method, device and program product of the present invention can obtain a human behavior perception result by obtaining a pre-trained human behavior perception model and inputting video data into the human behavior perception model, thereby solving the problem of low versatility and scalability of existing human behavior perception algorithms. By abstracting different task objectives into different types of queries, a unified decoder is used to obtain corresponding outputs according to the task objectives, and the action decoder is unified through the codec, that is, the common parts of different tasks are shared to the maximum extent, while the remaining parts of a single task are kept unchanged, ensuring that different tasks have more accurate weights, and outputting different results for specific tasks, thereby achieving cross-task joint optimization, and finally being able to complete multiple tasks in the field of human behavior perception, thereby improving versatility and scalability.

[0032] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0033] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings:

[0035] Figure 1 A flowchart of a human behavior perception method based on query provided by an embodiment of the present invention;

[0036] Figure 2 A schematic diagram of the structure of a query-based human behavior perception model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0038] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0039] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0040] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0041] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0042] First, several terms involved in the embodiments of the present application are introduced.

[0043] Video sequence: A data set consisting of a series of consecutive video frames.

[0044] Backbone: In deep learning, backbone refers to a convolutional neural network (CNN) or its variants, which is used to extract high-level feature representations from input data. Backbone is usually the basis for building more complex models, such as object detection, semantic segmentation, human pose estimation, and other tasks that require a global understanding of the entire image and the fusion of features at different scales and levels.

[0045] Actionformer: Actionformer is a fictional term that combines two concepts: Action and Transformer. In this fictional concept, Actionformer may refer to an imaginary model or architecture for handling action-related tasks, such as action recognition, action generation, etc. In general, Actionformer can be understood as a model based on the Transformer architecture, specifically designed to handle action-related tasks. It may combine the Transformer's self-attention mechanism and multi-head attention mechanism to capture the timing information and dependencies in the action sequence.

[0046] Action localization: Action localization aims to locate the start and end frames of actions in untrimmed videos, which is a challenging task due to the similarity of adjacent frames and the fuzziness of action boundaries. There are generally two types of algorithms for action localization: Two-stage methods generally generate single or multiple proposals based on action instances in the video in the first stage. First, candidate segments are generated and a binary classifier is trained that associates each segment with a confidence score to generate proposals. The second stage refines the generated proposals, and those with high scores are fed to a multi-class classifier to classify the action.

[0047] Action Recognition: Action recognition aims to identify action categories in trimmed videos. This is a challenging task due to the complexity of individual actions and the similarities between them. Action recognition algorithms generally rely on backbones to extract features of video clips and are relatively simple in classification modules. On the other hand, many attempts have been made in previous work on how to design effective networks to extract spatiotemporal features from videos. These attempts can be divided into two categories, namely 3D CNN-based and Transformer-based.

[0048] Action segmentation and action evaluation: Action segmentation aims to segment actions in long uncut videos. According to the type of network used, mainstream segmentation methods can be divided into three categories: TCN-based methods, GNN-based methods, and Transformer-based methods. Action evaluation aims to evaluate the performance of human body in completing actions and give them scores. Mainstream methods mainly formulate action evaluation as a regression problem and directly predict action scores.

[0049] Human behavior perception: Human behavior perception is an important part of the computer vision field. It covers multiple subtasks that work together to deeply understand and analyze human movements and behaviors in videos. These tasks mainly include basic positioning and classification, and are further extended to detailed evaluation of human behavior.

[0050] The query-based human behavior perception method provided by this application is introduced in detail below.

[0051] Optionally, the execution subject of the query-based human behavior perception method provided in the present application is an electronic device, which may be a terminal such as a computer, a mobile phone, a tablet computer, or a server. This embodiment does not limit the implementation method of the electronic device.

[0052] This embodiment provides a human behavior perception method based on query, such as Figure 1 As shown, the method at least includes step S101 to step S102:

[0053] Step S101, obtain a pre-trained human behavior perception model, which includes a video encoder and an action decoder; the video encoder is used to extract visual features corresponding to video data; the action decoder is used to use the visual features as query key-value pairs, associate them with at least one preset query, optimize at least one preset query and output the results corresponding to the query task.

[0054] refer to Figure 2 In the human behavior perception model shown, the input of the video encoder 21 is video data, so that the video encoder extracts visual features corresponding to the video data based on the video data.

[0055] In this embodiment, the video data includes a video sequence or a feature of a video sequence. The video sequence refers to a sequence consisting of a series of continuous human behavior image frames, including those recorded and generated by various devices or systems, such as cameras, mobile phones, computer screen recording software, etc., and can also be obtained from the Internet. This embodiment does not limit the method for obtaining the video sequence.

[0056] In this embodiment, the video encoder is composed of a projection function using a convolutional network. The human behavior perception model first uses a video encoder to encode the input video data into a multi-scale feature representation. The video encoder includes a convolutional neural network layer (CNN), a transformer network layer (Transformer Network) and an encoder layer (Encoder). The projection function is expressed by the following formula:

[0057] X={x 1 , x 2 ,……,x T}→Z={z 1 , z 2 ,……,z L}

[0058] Where X represents the video sequence; x T represents the Tth feature; Z represents the feature pyramid, z L represents the Lth valid visual feature.

[0059] When the video data is the feature of the video sequence, the convolutional network layer expands each feature into multiple dimensions and maps it to the transformer network of the output feature pyramid. It is then input into the decoding layer of the Transformer architecture for feature interaction to extract effective visual features.

[0060] Specifically, when the video data is a feature of a video sequence, the basis extracts visual features corresponding to the video data, including: expanding each feature into multiple dimensions through a convolutional network layer; mapping the features processed by the convolutional network layer to an output feature pyramid through a transformer network layer; and performing feature interaction processing on the feature pyramid through a coding layer to obtain visual features.

[0061] When the video data is a video sequence, the convolutional network layer is also used to extract features corresponding to the video sequence.

[0062] Specifically, when the video data is a video sequence, before expanding each feature into multiple dimensions through a convolutional network layer, it also includes: extracting features corresponding to the video sequence through a convolutional network layer.

[0063] In this embodiment, the input of the action decoder 22 is the visual features extracted by the video encoder 21, so that the action decoder 22 can use the visual features as query key-value pairs, associate them with at least one preset query, optimize at least one preset query and output the results corresponding to the query task.

[0064] Among them, at least one preset query includes a category query, a boundary query and / or a mask query; the query task includes an action category query task, an action location query task, an action segmentation query task and / or an action evaluation query task. The output combination results of different preset queries are the results of different query tasks. Specifically, it is expressed by the following formula:

[0065] <O c , O l , O s , O a >=ActionDec( c , Q b , O m >; Z)

[0066] In the formula, O c Represents the predicted action category; O l Indicates the predicted start and end time of human behavior; O s represents the predicted action segment; O a represents the predicted action performance score; Q c Indicates category query; Q b Indicates boundary query; O m represents mask query; Z represents feature pyramid.

[0067] ​Among them, the action category query task refers to the action category predicted by the interaction of category query and visual features in the decoding layer; the action location query task refers to the time segment and corresponding action category predicted by the interaction of category query, boundary query and visual features in the decoding layer; the action segmentation query task refers to the action segmentation segment predicted by the interaction of boundary query, mask query, category query and visual features in the decoding layer; the action evaluation query task refers to the action performance score predicted by the interaction of category query and visual features in the decoding layer.

[0068] For different query tasks, the cross-attention mechanism is shielded according to the query tasks. Different query tasks are achieved by setting mask queries to mask and unmask different parts of the video data. Specifically, the features corresponding to the video sequence are input into the human behavior perception model, and before obtaining the human behavior perception results, it also includes: shielding category queries, boundary queries and / or mask queries based on the query tasks.

[0069] In this embodiment, the action category query task aims to predict the category of human behavior action instances in video data. For this task, the action decoder does not need additional text queries as input, only category queries are needed. Category queries and visual features interact in the action decoder to obtain the predicted action category. Specifically, the action category query task can be expressed by the following formula:

[0070] <O c > = ActionDec(Q c ; Z)

[0071] In the formula, O c represents the predicted action category; Q c represents category query; Z represents feature pyramid.

[0072] Existing action classification algorithms include two-stage algorithms and single-stage algorithms. Among them, the two-stage algorithm refers to the first stage that generates single or multiple proposals based on human action action instances in the video. First, candidate segments are generated and a binary classifier is trained that associates each segment with a confidence score to generate suggestions. The second stage further refines the generated proposals, and those with high scores are fed to a multi-class classifier to classify the action. The single-stage algorithm refers to directly predicting the temporal boundaries and corresponding categories of human behavior action instances in the video.

[0073] However, the two-stage algorithm is generally more complicated in terms of process, relatively slow in algorithm speed, and less flexible. The single-stage algorithm generally has an optimization module to refine the predicted errors or inaccurate boundaries and action categories.

[0074] In this embodiment, the predicted action category can be obtained by interacting the category query and the visual features in the action decoder, which simplifies the entire action recognition process, reduces cumbersome steps, and improves the algorithm speed. The way of interaction between category query and visual features can also better adapt to different action categories and scene changes, improve the flexibility and generalization ability of the algorithm, and make the model more adaptable; at the same time, through the interaction of category query and visual features, the accuracy and precision of action recognition can be improved.

[0075] The goal of the action localization query task is to measure the start and end time of human behavior action instances in video data and the category of the corresponding action instances. For this task, the action decoder does not require additional text queries as input, but requires category queries and boundary queries. Category queries, boundary queries, and visual features interact in the action decoder to obtain the predicted time segment and the corresponding action category to achieve action localization. Specifically, the action localization query task can be expressed by the following formula:

[0076] <O c , O l > = ActionDec(Q c ;Q b ; Z)

[0077] In the formula, O c Represents the predicted action category; O l Indicates the predicted start and end time of human behavior; Q c Indicates category query; Q b represents boundary query; Z represents feature pyramid.

[0078] Existing action localization algorithms include two-stage algorithms and single-stage algorithms. Among them, the two-stage algorithm refers to the first stage that generates single or multiple proposals based on human action action instances in the video. First, candidate segments are generated and a binary classifier is trained that associates each segment with a confidence score to generate suggestions. The second stage further refines the generated proposals, and those with high scores are fed into a multi-class classifier to classify the action. The single-stage algorithm refers to directly predicting the temporal boundaries and corresponding category action instances in the video.

[0079] However, the two-stage algorithm is generally more complicated in terms of process, relatively slow in algorithm speed, and less flexible. The single-stage algorithm generally has an optimization module to refine the predicted errors or inaccurate boundaries and action categories.

[0080] In this embodiment, the predicted time segment and the category of the corresponding action are obtained through the interaction of category query, boundary query and visual features in the action decoder to achieve action positioning, which simplifies the entire action recognition process, reduces cumbersome steps, and improves the algorithm speed. The interaction between category query and visual features can also better adapt to different action categories and scene changes, improve the flexibility and generalization ability of the algorithm, and make the model more adaptable. At the same time, action positioning is achieved through the interaction of category query, boundary query and visual features in the action decoder, which can improve the accuracy and precision of action positioning.

[0081] The goal of the action segmentation query task is to divide and segment the action instances and background of different human behaviors in a video data. For this task, the action decoder does not need additional text queries as input, but requires boundary queries and their masks. The mask refers to the mask that covers different parts through mask queries. In the action segmentation query task, the background and action of each frame in the mask judgment task are judged. For example, taking the mask as "1001", "1" and "0" represent action, background, background and action respectively. The boundary query and visual features interact in the action decoder to obtain the predicted action segmentation fragments. Specifically, the action segmentation query task can be expressed by the following formula:

[0082] <O s > = ActionDec(Q b ; Z)

[0083] In the formula, O s represents the predicted action segment; Q b represents boundary query; Z represents feature pyramid.

[0084] Existing action segmentation algorithms include algorithms based on Temporal Convolutional Network (TCN), algorithms based on Graph Neural Network (GNN), and algorithms based on Transformer.

[0085] The goal of the action evaluation query task is to score a human action instance in a video data to evaluate its performance. For this task, the action decoder does not require additional text queries as input, but requires category queries. Category queries and visual features interact in the action decoder to obtain the predicted action performance score. Specifically, the action evaluation query task can be expressed as follows:

[0086] <O a > = ActionDec(Q c ; Z)

[0087] In the formula, Oa represents the predicted action performance score; O c represents the predicted action category; Z represents the feature pyramid.

[0088] Existing action evaluation algorithms have great constraints in feature extraction. Most traditional methods formulate action evaluation as a regression problem and directly predict action scores, but their performance is limited by the potential ambiguity of action score labels.

[0089] In this embodiment, the predicted action performance score is obtained by interacting the category query and the visual features in the action decoder, which can better consider the influence of the action category on the action performance, reduce the impact of the potential ambiguity of the action score label to a certain extent, and improve the pertinence and accuracy of the prediction.

[0090] In this embodiment, an action form decoder is used to construct an action decoder architecture. Given visual features, the action decoder uses a Transformer multi-layer architecture to optimize the preset query and present the output.

[0091] In one example, the Transformer multi-layer architecture includes a preset number of decoders in the Transformer architecture, wherein the preset number includes but is not limited to 4, 5 or 6. In actual implementation, the value of the preset number can be modulated according to actual conditions, and this embodiment does not limit the value of the preset number.

[0092] Specifically, the action decoder includes a preset number of decoder layers, takes visual features as query key-value pairs, associates them with preset queries, optimizes the preset queries and outputs human behavior perception results, including: cross-attention to visual features in each decoding layer; self-attention between category queries, boundary queries and mask queries, and associations and interactions established between category queries, boundary queries and mask queries, and outputs human behavior perception results.

[0093] In another example, the Transformer multi-layer architecture also includes a deformed Transformer multi-layer architecture, such as Video Transformer or Temporal Shift Module (TSM).

[0094] In actual implementation, the Transformer multi-layer architecture can be optimized and modulated according to actual conditions. This embodiment does not limit the implementation method of the Transformer multi-layer architecture.

[0095] Optionally, the human behavior perception model is pre-stored in the electronic device, or is obtained by training the electronic device. Specifically, obtaining a pre-trained human behavior perception model includes: obtaining training data corresponding to each query task; the training data includes: sample video data and standard output results of each query task corresponding to the sample video data; extracting visual features of the sample video data through the backbone network corresponding to each query task to obtain sample visual features; the sample visual features extracted by the backbone network corresponding to different query tasks have the same shape; inputting the sample visual features into the initial human perception model, and jointly pre-training the action decoder and each query task to obtain a joint training result; the initial human perception model has the same model structure as the human perception model; inputting the joint training result and the standard output result into the preset loss function corresponding to each query task to obtain the loss function value; using the loss function value to iteratively train the initial human perception model to obtain the human perception model.

[0096] Step S102, inputting the features corresponding to the video data into a human behavior perception model to obtain a human behavior perception result.

[0097] Specifically, in the case where the video data is the features corresponding to the video sequence, each feature is expanded into multiple dimensions through the convolutional network layer; the features processed by the convolutional network layer are mapped to the output feature pyramid through the transformer network layer; the feature pyramid is interactively processed through the encoding layer to obtain visual features; based on the current query task, at least one query is selected from the category query, boundary query and mask query, and interacts with the visual features in the action decoder to obtain the human behavior perception result; in the case where the video data is a video sequence, before each feature is expanded into multiple dimensions through the convolutional network layer, it is also necessary to extract the features corresponding to the video sequence through the convolutional network layer.

[0098] In summary, the query-based human behavior perception method provided in this embodiment obtains a pre-trained human behavior perception model, inputs video data into the human behavior perception model, and obtains human behavior perception results, which can solve the problem of low versatility and scalability of existing human behavior perception algorithms. By abstracting different task objectives into different types of queries, a unified decoder is used to obtain corresponding outputs according to the task objectives, and the action decoder is unified through the codec, that is, the common parts of different tasks are shared to the maximum extent, while keeping the remaining parts of a single task unchanged, ensuring that different tasks enjoy more accurate weights, and outputting different results for specific tasks, thereby achieving cross-task joint optimization, and ultimately being able to complete multiple tasks in the field of human behavior perception, thereby improving versatility and scalability.

[0099] In addition, by training the human behavior perception model in a joint training manner, it is possible to share the underlying feature representations and weights, and optimize these features between different tasks, reducing the number of parameters of the human behavior perception model and reducing the risk of overfitting. At the same time, it also helps to generalize the human behavior perception model. Through the mutual promotion between multiple tasks, the human behavior perception model can achieve better performance in each task.

[0100] Corresponding to the above method, the present invention also provides a query-based human behavior perception device, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps of the above-mentioned query-based human behavior perception method.

[0101] The embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned query-based human behavior perception method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0102] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0103] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the aforementioned query-based human behavior perception method when executed by a processor.

[0104] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0105] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0106] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A human behavior perception method based on query, characterized in that: The method comprises the following steps: Obtain a pre-trained human behavior perception model, the human behavior perception model comprising a video encoder and an action decoder; the video encoder is used to extract visual features corresponding to video data; the action decoder is used to associate the visual features with at least one preset query as a query key-value pair, optimize the at least one preset query and output a result corresponding to the query task; the at least one preset query comprises a category query, a boundary query and / or a mask query; Based on the query task, the category query, the boundary query and / or the mask query are shielded; the query task includes an action category query task, an action location query task, an action segmentation query task and / or an action evaluation query task; the action category query task refers to the action category predicted by the interaction of the category query and the visual feature in the action decoder; the action location query task refers to the time segment and the corresponding action category predicted by the interaction of the category query, the boundary query and the visual feature in the action decoder; the action segmentation query task refers to the action segmentation segment predicted by the interaction of the boundary query, the mask query, the category query and the visual feature in the action decoder; the action evaluation query task refers to the action performance score predicted by the interaction of the category query and the visual feature in the action decoder; The video data is input into the human behavior perception model to obtain a human behavior perception result.

2. The method according to claim 1, characterized in that The video encoder includes a convolutional network layer, a transformer network layer and an encoder layer; the video data includes a video sequence or a feature of a video sequence; In the case where the video data is a feature of the video sequence, extracting the visual feature corresponding to the video data includes: Expanding each feature into multiple dimensions through the convolutional network layer; Mapping the features processed by the convolutional network layer to an output feature pyramid through the transformer network layer; The feature pyramid is subjected to feature interaction processing by the encoder layer to obtain the visual feature.

3. The method according to claim 2, characterized in that In the case where the video data is the video sequence, before expanding each feature into multiple dimensions through the convolutional network layer, it also includes: extracting features corresponding to the video sequence through the convolutional network layer.

4. The method according to claim 1, characterized in that: The action decoder includes a preset number of decoder layers, and the visual features are used as query key-value pairs, associated with preset queries, and the preset queries are optimized and output human behavior perception results, including: Cross-attending the visual features in each decoding layer; Self-attention is performed between the category query, the boundary query and the mask query, associations and interactions are established between the category query, the boundary query and the mask query, and the human behavior perception result is output.

5. The method according to claim 1, characterized in that The obtaining of a pre-trained human behavior perception model comprises: Acquire training data corresponding to each query task; the training data includes: sample video data and standard output results of each query task corresponding to the sample video data; Extracting visual features of the sample video data through the backbone network corresponding to each query task to obtain sample visual features; the sample visual features extracted by the backbone networks corresponding to different query tasks have the same shape; Inputting the sample visual features into an initial human perception model, performing joint pre-training on the action decoder and each query task, and obtaining a joint training result; the initial human perception model has the same model structure as the human perception model; Inputting the joint training result and the standard output result into the preset loss function corresponding to each query task to obtain a loss function value; The initial human perception model is iteratively trained using the loss function value to obtain the human perception model.

6. A human behavior perception device based on query, comprising a processor, a memory, and a computer program / instruction stored in the memory, characterized in that: The processor is used to execute the computer program / instructions. When the computer program / instructions are executed, the device implements the steps of the method according to any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method as claimed in any one of claims 1 to 5 are implemented.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.