A human-object interaction detection method based on multi-scale deformable Transformer network
Through the multi-scale deformable Transformer network, combined with the deformable Transformer encoder and decoder, the problem of missing scale information in HOI detection is solved, more accurate and robust HOI detection is achieved, and the interaction prediction ability in complex environments is improved.
Patent Information
- Application Number
- CN202211447645.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-18
- Publication Date
- 2025-09-23
- Estimated Expiration
- 2042-11-18
AI Technical Summary
Existing HOI detection methods usually rely on a single architecture, ignore the intrinsic connection between localization and interaction recognition tasks, and lack scale information, resulting in unsatisfactory results.
A multi-scale deformable Transformer network is adopted to extract image features through a multi-scale backbone network. It is combined with a deformable Transformer encoder and decoder, and uses deformable multi-head self-attention and multi-scale deformable self-attention modules to extract visual context features, realize person and object decoding and action decoding, and finally filter out repeated predictions through non-maximum suppression.
The accuracy and robustness of HOI detection are improved in complex environments, and the interaction between people and objects can be consistently predicted. This solves the problem of missing scale information in existing methods and improves the detection effect.
Smart Images
Figure CN116758298B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the technical field of human-object interaction detection, and in particular relates to a human-object interaction detection method based on a multi-scale deformable Transformer network. Background Art
[0002] Human-object interaction (HOI) detection is a crucial component of intelligent robotics, focusing on interactions between humans and objects. Compared to object detection, it focuses more on action and scene understanding, which is crucial for machines to understand human activities. Specifically, HOI detection predicts a set of triplets from an input image. These triplets include the locations of the interacting humans and objects, as well as the interaction category. Intelligently and efficiently locating human-object pairs and predicting their interactions in complex and ever-changing environments is a challenging task.
[0003] Localization and interaction recognition are two key tasks in HOI (human-object interaction) detection, requiring advanced image and scene understanding. Existing HOI detection methods typically rely on a single architecture to address both tasks. However, such strategies often overlook the inherent connection between the two distinct tasks. Furthermore, previous methods are limited by the lack of scale information, resulting in suboptimal results. Summary of the Invention
[0004] The purpose of this application is to provide a human-object interaction detection method based on a multi-scale deformable Transformer network to overcome the technical problems pointed out in background counting.
[0005] A method for detecting human-object interactions based on a multi-scale deformable Transformer network, including:
[0006] The image to be detected is input into the multi-scale backbone network to extract features, the extracted features of each scale are flattened into a one-dimensional vector, and then input into the deformable Transformer encoder to complete the encoding of the multi-scale features and obtain the visual context features;
[0007] Input the visual context features into the deformable Transformer decoder of the cascade architecture to obtain the human and object decoding features and action decoding features;
[0008] The decoded features of people and objects are passed through an embedding layer and the FFN network to detect the bounding boxes and object categories of people and objects. The decoded features of actions are passed through the FFN network to detect the action categories.
[0009] The bounding boxes of people and objects, object categories, and action categories detected are filtered out through non-maximum suppression to remove duplicate predictions, and the final results are drawn and displayed in the image to be detected.
[0010] Furthermore, the deformable Transformer decoder of the cascade architecture includes a cascaded person and object decoder and an action decoder.
[0011] Furthermore, the human and object decoder and the action decoder both include several layers of deformable decoding layers, each of which includes a deformable multi-head self-attention module and a multi-scale deformable self-attention module.
[0012] Furthermore, the human and object decoder performs the following operations:
[0013] Initialize the query vector, split the query vector into a person interaction query vector and a position query vector of equal length, and apply a linear layer to the position query vector to obtain an anchor box variable;
[0014] The person interaction query vector and the position query vector are reconstructed to obtain the person and object query vector, which are used as the Q, V, and K vectors of the deformable multi-head self-attention module in the first deformable decoding layer, respectively. After passing through the deformable multi-head self-attention module, the multi-head attention vector is output;
[0015] After adding the multi-head attention vector to the anchor box variable, it is input into the multi-scale deformable self-attention module together with the anchor box variable and visual context features to obtain a multi-scale deformable vector;
[0016] The multi-scale deformable vector is input to the next deformable decoding layer as the Q, V, and K vectors of the deformable multi-head self-attention module in the next deformable decoding layer. The same operation as the first deformable decoding layer is performed, and finally the output of the last deformable decoding layer is used as the decoding feature of people and objects.
[0017] Furthermore, the action decoder performs the following operations:
[0018] The output of each deformable decoding layer of the human and object decoders is passed through the action embedding function to obtain the action query vector, which is input into the corresponding deformable decoding layer of the action decoder;
[0019] In each deformable decoding layer of the action decoder, the action query vector is used as the Q, V, and K vectors of the deformable multi-head self-attention module, and the multi-head attention vector is output after passing through the deformable multi-head self-attention module;
[0020] In each deformable decoding layer of the action decoder, the multi-head attention vector, visual context features, and the output of the previous deformable decoding layer are input into the multi-scale deformable self-attention module to obtain a multi-scale deformable vector;
[0021] The output of the last deformable decoding layer of the action decoder is used as the action decoding feature.
[0022] This application proposes a method for detecting human-object interactions based on a multi-scale deformable Transformer network. This method uses a multi-scale backbone network to extract image features and combines it with a deformable Transformer encoder to extract contextual multi-scale features. This incorporates position and size variations of different objects across spatial scales, resulting in more accurate and robust visual feature representations. Furthermore, this method introduces a cascaded deformable decoder structure to consistently predict interactions in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 This is a flow chart of the applicant's method for detecting interaction with an object;
[0024] Figure 2 Schematic diagram of the multi-scale deformable Transformer network for this application;
[0025] Figure 3 Schematic diagram of the deformable Transformer decoder of this application. DETAILED DESCRIPTION
[0026] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.
[0027] In one embodiment, the present application provides a method for detecting human-object interaction based on a multi-scale deformable Transformer network, including:
[0028] Step S1: Input the image to be detected into the multi-scale backbone network to extract features, flatten the extracted features of each scale into a one-dimensional vector, and input it into the deformable Transformer encoder to complete the encoding of the multi-scale features and obtain the visual context features.
[0029] This application uses the multi-scale backbone network of Swin Ttransformer to extract preliminary hierarchical image visual features. Given an image After inputting into the multi-scale backbone network, we get These four scale feature maps, C s is the conversion dimension obtained after calculation through the multi-layer backbone network, C l =256, and concatenate them into the deformable Transformer encoder to generate visual context features V m .
[0030] The network structure of the cascade decoding in this application is as follows Figure 2As shown in Figure 2, after the backbone network extracts multi-scale information and obtains visual context features through the visual encoder, the visual context features are input into the human and object decoder and the action decoder in a shared manner. Figure 2 In [1], the entire encoding process of the Deformable Transformer encoder is represented as the encoding of visual information, and the overall architecture of this block is called the Visual Encoder. The Deformable Transformer encoder uses position encoding Multi-scale feature maps and position encoding are fed into a deformable encoder to generate visual context features. Among them C d It is C s It is obtained by 1×1 convolution operation transformation, N m It represents the sum of pixels of a single channel feature map, and the visual context feature is input to the subsequent deformable decoder to decode the HOI feature.
[0031] Step S2: Input the visual context features into the deformable Transformer decoder of the cascade architecture to obtain the human and object decoding features and action decoding features.
[0032] The cascaded deformable Transformer decoder (Deformable Transformer Decoder) of this embodiment includes a cascaded human and object decoder and an action decoder.
[0033] Step S1 obtains the visual context feature V through a backbone network and a deformable Transformer encoder. m After this step, V m Input to the person and object decoder and action decoder respectively.
[0034] like Figure 2 As shown in the figure, both the human and object decoder and the action decoder contain several layers of deformable decoding layers (Deformable Transformer Decoder Layer). Each deformable decoding layer includes two self-attention modules: Deformable Multi-Head Attention (Deformable Multi-Head Self-Attention Module) and Multi-Scale Deformable Attention (Multi-Scale Deformable Self-Attention Module).
[0035] In the human-object decoder, a query vector of length N (set to 300 in this example) is first initialized. In order to strengthen the fusion of visual information during training, this application divides the query vector into a human-object interaction query vector and a position query vector. The position query vector is passed through the linear layer to generate a learnable anchor box variable, which is input into the deformable decoding layer together with the two query vectors. The specific decoding details are as follows: Figure 3 As shown. Figure 3 In the example, the two query vectors are concatenated as the parameter input of the multi-head self-attention operation and then fed into the multi-scale deformable self-attention module, where they are combined with the learnable anchor box variables and visual context features to obtain the final person and object decoding feature F. ho .like Figure 3 As shown, the human and object decoder of this embodiment performs the following operations:
[0036] Step S2.1: Initialize the query vector, divide the query vector into a person interaction query vector and a position query vector of equal length, and apply a linear layer to the position query vector to obtain an anchor box variable.
[0037] The deformable Transformer decoder of this embodiment first initializes a query vector with a length of N, which is set to 300 in this application, and divides it into two parts of equal length, namely the human object interaction query vector (human object interaction) and position query vector And the position query vector is passed through the linear layer to generate a learnable anchor box variable The deformable anchor box variables are specifically expressed as (x,y) coordinates.
[0038] Step S2.2: Reconstruct the person interaction query vector and the position query vector to obtain the person and object query vector, which are used as the Q, V, and K vectors of the deformable multi-head self-attention module in the first deformable decoding layer, respectively. After passing through the deformable multi-head self-attention module, the multi-head attention vector is output.
[0039] The architecture of the human and object decoder is as follows Figure 3 As shown, the human-object decoder includes l layers of deformable decoding layers. After obtaining the learnable anchor box variables, the human interaction query vector Q is transformed into hoi With the position query vector Q pos Reassemble the query vectors of people and objects, which are specifically expressed as:
[0040] Q ho =[Q hoi ,Q pos ];
[0041] in,
[0042] Next, Q ho is fed into a multi-scale deformable head self-attention module, and then the query vector and visual context features V m Perform multi-scale deformable attention operations between them.
[0043] Specifically, the character interaction query vector Q hoi With the position query vector Q pos Reassemble them as the Q, V, and K vectors of the deformable multi-head self-attention module, and output the multi-head attention vector M after passing through the deformable multi-head self-attention module. ha , expressed as:
[0044] M ha =MultiHeadAttn(Q,V,K).
[0045] Step S2.3: After adding the multi-head attention vector to the anchor box variable, it is input into the multi-scale deformable self-attention module together with the anchor box variable and the visual context feature to obtain a multi-scale deformable vector.
[0046] In this step, the multi-head attention vector M ha After adding the anchor box variable, it is input into the multi-scale deformable self-attention module together with the anchor box variable and the visual context feature to obtain the multi-scale deformable vector M sda , the process of obtaining multi-scale deformable vectors can be specifically expressed as:
[0047]
[0048] Among them, Q da =M ha +a,ref=a.
[0049] Step S2.4: Input the multi-scale deformable vector to the next deformable decoding layer as the Q, V, and K vectors of the deformable multi-head self-attention module in the next deformable decoding layer. Perform the same operation as the first deformable decoding layer, and finally use the output of the last deformable decoding layer as the person and object decoding features.
[0050] In each deformable decoder layer, the decoded person and object query vectors are output, represented as i represents the current layer number,
[0051] For the i-th (i>1) deformable decoding layer, the multi-head attention vector M ha It can be expressed as
[0052] M ha =MultiHeadAttn(Q,V,K)
[0053] in
[0054] For the i-th (i>1) deformable decoding layer, the multi-scale deformable vector M sda It can be expressed as:
[0055]
[0056] Among them, Q da =M ha +a,ref=a.
[0057] This application updates the output of the last layer Decoded person and object features F ho , the decoded human and object features are expressed as: bs is the batch size set for the model, which is set to 4 in this example. Different from directly extracting human interaction features from visual feature sequences, this embodiment uses a separate deformable decoder to obtain significant human and object features with position information.
[0058] In order to better combine the features of people and objects to predict interaction information, this application specifically constructs a deformable action decoder to construct action features separately. The action decoder is based on the visual context feature V m And the updated person and object query vector Q ho Embedded action query Q a To complete the decoding operation.
[0059] like Figure 2 , the action decoder proposed in this application will update the query vector Q ho Input into the action embedding function to obtain the action query Q a Then use the action decoder to obtain the action decoding feature F act The action decoder of this embodiment performs the following operations:
[0060] Step S3.1: Apply the output of each deformable decoding layer of the human and object decoders to the action embedding function to obtain an action query vector, which is then input into the deformable decoding layer corresponding to the action decoder.
[0061] In this embodiment, the output of each deformable decoding layer of the human and object decoder is applied to the action embedding function to obtain the action query vector, which is specifically expressed as:
[0062]
[0063] Among them, Embedding (a) It is the defined action embedding function, which is mainly obtained by multiple embedding layers.
[0064] Step S3.2: In each deformable decoding layer of the action decoder, the action query vector is used as the Q, V, and K vectors of the deformable multi-head self-attention module, and the multi-head attention vector is output after passing through the deformable multi-head self-attention module.
[0065] In the deformable decoding layer of the action decoder, the multi-head attention vector M ha It can be expressed as:
[0066] M ha =MultiHeadAttn(Q, V, K)
[0067] in i represents the current layer number.
[0068] Step S3.3: In each deformable decoding layer of the action decoder, the multi-head attention vector, visual context features, and the output of the previous deformable decoding layer are input into the multi-scale deformable self-attention module to obtain a multi-scale deformable vector.
[0069] In this embodiment, the multi-scale deformable vector M sda It can be expressed as:
[0070]
[0071] in, (When i=1, ).
[0072] Step S3.3: The output of the last deformable decoding layer of the action decoder is used as the action decoding feature.
[0073] In this embodiment, the last layer updates the output of the person and object query vector Decoded action feature F is represented as act ,in In this way, interaction consistency is further maintained.
[0074] Step S3: The decoded features of people and objects are passed through an embedding layer and the FFN network to detect the bounding boxes and object categories of people and objects, and the decoded features of actions are passed through the FFN network to detect the action categories.
[0075] In this embodiment, the decoded person and object decoding features F ho and action decoding feature F actAfter that, the decoded features of people and objects are embedded into The embedding process is expressed as:
[0076]
[0077] Among them, Embedding (ho) It is the defined embedding function of people and objects, which is mainly obtained by multiple embedding layers.
[0078] and F act Use FFN (feedforward network) in the interactive head network to predict {h box , o bbox , o cls , a cls}, as shown below:
[0079]
[0080]
[0081]
[0082] a cls =FFN act (F act ),
[0083] in, Represent object categories and action categories respectively, C o , C a Represents the number of categories of objects and actions respectively. Represent the bounding boxes of people and objects, respectively, which are predicted by the center point and the normalized reference point.
[0084] Step S4: The bounding boxes of people and objects, object categories, and action categories detected are filtered out through non-maximum suppression to remove duplicate predictions, and the final results are drawn and displayed in the image to be detected.
[0085] As mentioned above, a HOI prediction result is represented by the following four parts: person bounding box, object bounding box, object category and action category.
[0086] In step S3, the person and object decoder branch predicts the bounding box pairs of people and objects and the corresponding object categories, and then the action decoder branch predicts the action category separately. Then the interactive head network uses the FFN network to predict the results based on different feature information. Formally, the prediction result corresponding to the i-th query vector is expressed as follows:
[0087]
[0088] The prediction result is a representation after a non-maximum (NMS) suppression process, where after processing repeated boxes using the NMS algorithm, the top K results with the highest confidence scores are taken as the final prediction.
[0089] In the NMS algorithm, the main task is to calculate the IoU value between different HOI instances, which can be expressed as:
[0090]
[0091] Among them, i and j represent the indexes of different people or objects. After calculating the IoU value, the threshold δ is used to eliminate the HOI instances with lower scores and return the final result. After obtaining the final prediction result, it can be drawn and displayed in the original image.
[0092] This application uses a query vector-based training method, and uses the Hungarian matching algorithm to match each true value with the predicted value during the training phase. The overall loss function of the network consists of three parts. The first part is the bounding box loss function, part 2 For the cross-overlap merging loss, the third part is the classification loss:
[0093]
[0094] Among them, λ1, λ2, λ3 are parameters used to adjust the weight of each loss.
[0095] Among them, In the loss function, x∈(h,o) represents the loss function statistics of the bounding box loss of human and object, y∈(h,o) represents the statistical intersection and overlap loss of human and object, and z∈(o,a) represents the statistical classification loss of object and action.
[0096] It should be noted that during training, the applicant and the object decoder can initialize the query vector by randomly generating it. In actual applications, the query vector can be initialized using trained network parameters. Initializing the query vector is a relatively mature method in the field and will not be further described here.
[0097] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.
Claims
1. A method for detecting human-object interaction based on a multi-scale deformable Transformer network, characterized in that: The method for detecting human-object interaction based on a multi-scale deformable Transformer network includes: The image to be detected is input into the multi-scale backbone network to extract features, the extracted features of each scale are flattened into a one-dimensional vector, and then input into the deformable Transformer encoder to complete the encoding of the multi-scale features and obtain the visual context features; Input the visual context features into the deformable Transformer decoder of the cascade architecture to obtain the human and object decoding features and action decoding features; The decoded features of people and objects are passed through an embedding layer and the FFN network to detect the bounding boxes and object categories of people and objects. The decoded features of actions are passed through the FFN network to detect the action categories. The bounding boxes of people and objects, object categories, and action categories detected are filtered out through non-maximum suppression to remove duplicate predictions, and the final results are drawn in the image to be detected. The deformable Transformer decoder of the cascade architecture includes a cascaded human and object decoder and an action decoder; The human and object decoder and the action decoder both include several layers of deformable decoding layers, each of which includes a deformable multi-head self-attention module and a multi-scale deformable self-attention module.
2. The method for detecting human-object interaction based on a multi-scale deformable Transformer network according to claim 1, characterized in that: The human and object decoder performs the following operations: Initialize the query vector, split the query vector into a person interaction query vector and a position query vector of equal length, and apply a linear layer to the position query vector to obtain an anchor box variable; The person interaction query vector and the position query vector are reconstructed to obtain the person and object query vector, which are used as the Q, V, and K vectors of the deformable multi-head self-attention module in the first deformable decoding layer, respectively. After passing through the deformable multi-head self-attention module, the multi-head attention vector is output; After adding the multi-head attention vector to the anchor box variable, it is input into the multi-scale deformable self-attention module together with the anchor box variable and visual context features to obtain a multi-scale deformable vector; The multi-scale deformable vector is input to the next deformable decoding layer as the Q, V, and K vectors of the deformable multi-head self-attention module in the next deformable decoding layer. The same operation as the first deformable decoding layer is performed, and finally the output of the last deformable decoding layer is used as the decoding feature of people and objects.
3. The method for detecting human-object interaction based on a multi-scale deformable Transformer network according to claim 1, characterized in that: The action decoder performs the following operations: The output of each deformable decoding layer of the human and object decoders is passed through the action embedding function to obtain the action query vector, which is input into the corresponding deformable decoding layer of the action decoder; In each deformable decoding layer of the action decoder, the action query vector is used as the Q, V, and K vectors of the deformable multi-head self-attention module, and the multi-head attention vector is output after passing through the deformable multi-head self-attention module; In each deformable decoding layer of the action decoder, the multi-head attention vector, visual context features, and the output of the previous deformable decoding layer are input into the multi-scale deformable self-attention module to obtain a multi-scale deformable vector; The output of the last deformable decoding layer of the action decoder is used as the action decoding feature.