A method for detecting human interaction based on dynamic anchors

Abstract

This paper proposes a person interaction detection algorithm based on dynamic anchors. It uses dynamic anchors to guide the model to extract multi-scale features of instances, uses the dynamic anchor position distribution to match dynamic anchors and query embeddings, adds the semantic information extracted from the multi-scale features to the query embedding, generates a person interaction detection embedding, and predicts all elements.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence image processing, and in particular to a human interaction detection method based on dynamic anchor points. Background Art

[0002] Human interaction detection primarily consists of two parts: human-object detection and human interaction action recognition. Existing methods fall into two main categories: two-stage and single-stage. Two-stage methods first use existing detection frameworks to identify and locate instances (people and objects) in the image. Then, they use visual features extracted from the localized regions to identify action classes. Single-stage methods use a Transformer to detect human-object pairs and then use the interaction points between these pairs to simultaneously predict the human-object offset and action class.

[0003] However, these methods are susceptible to interference from ambiguous semantic features when people and objects in the image are far apart. In addition, existing methods often fail to extract the required features when there is a noisy background in the image, affecting the detection accuracy. Summary of the Invention

[0004] The technical problem to be solved by the present invention is that the existing human interaction detection method has excessive noise background and fuzzy semantic feature interference, resulting in low detection accuracy.

[0005] The present invention proposes a human interaction detection method based on dynamic anchor points.

[0006] A method for detecting human interaction based on dynamic anchor points, comprising:

[0007] Step 1: Divide the acquired image into image blocks;

[0008] Step 2: Calculate global attention and construct an interaction module consisting of layer normalization, global attention, and a multi-layer neural network. Apply the interaction module to the image patch to extract anchor points and multi-scale features.

[0009] Step 3: Calculate the central anchor point based on all the anchor points extracted in step 2. Rearrange all anchor points based on their distance from the central anchor point and divide them into intervals based on the distance. Calculate the distribution frequency based on the number of anchor points in the interval. Sample the anchor points in each distance interval based on the distribution frequency to obtain sampled anchor points. Match the sampled anchor points with the query embedding in the sampling order.

[0010] In step 4, the acquired image and query embedding are combined using the matched sampling anchors, and the semantic information in the multi-scale features is added to the query embedding to generate a person interaction detector.

[0011] Preferably, in step 1, the image is divided into non-overlapping image blocks.

[0012] Preferably, in step 2, the step of calculating the global attention is: using the block function to divide the image blocks into blocks of size feature blocks, each feature block is mapped to the query After pooling, concatenate and map to keys Sum According to the formula Calculate the global attention, where B represents the deviation predicted by the deep convolutional layer, d is a constant, Softmax() represents the normalized exponential function, i represents the i-th feature block, H and W represent the height and width of the token respectively, and k h and k w Represent the height and width of the feature block respectively.

[0013] Preferably, the formula for the central anchor point is:

[0014]

[0015] Among them, num D Represents the total number of anchor points. Represents the coordinates of the i-th anchor point among all anchor points, {x Centre ,y Centre} represents the coordinates of the center anchor point.

[0016] Preferably, the formula for dividing the intervals is:

[0017]

[0018] Among them, M i represents the i-th interval, L max Represents the distance from the farthest anchor point to the central anchor point, N represents the number of intervals, N is a positive integer, i = 0, 1, 2…N.

[0019] Preferably, when sampling the anchor points within each distance interval, equidistant sampling is performed, and the sampling formula is:

[0020]

[0021] in, Represents the number of anchor points that need to be sampled within the interval, N q Represents the total number of anchor points that need to be sampled, P i Represents the Mth i The frequency of distribution within a range;

[0022]

[0023] in, Represents the number of anchor points in the Mi interval, num D Represents the total number of anchor points;

[0024] Preferably, in step 4, the formula performed by combining the acquired image and query embedding is:

[0025]

[0026] Among them, MSDeformAttn represents multi-scale deformable attention, z q Represents the characterization features of the query, represents the coordinates of the dynamic anchor point for each query, represents a multi-scale feature map, m, l, k represent the index of the attention head, input feature scale, and sampling anchor point, respectively, ΔD mlqk and A mlqk They represent the sampling offset and attention weight of the kth dynamic anchor in the lth feature layer and the mth attention head, W m represents the connection matrix of m attention heads, x l and φ l Represents the feature map at the lth feature layer and the regression to the coordinates.

[0027] Preferably, the Among them F q ,F k ,F v Both are linear mapping layers.

[0028] Preferably, the formula of the block function is:

[0029]

[0030]

[0031] in, represents the feature block, Sw() represents the linear layer, and y represents the image block.

[0032] Compared with the prior art, the present invention has the following advantages and effects:

[0033] 1. The interactive module of the present invention can effectively alleviate the interference problem of fuzzy semantic features when the distance between people and objects in the image is far.

[0034] 2. The anchor points extracted by the present invention are dynamic anchor points. According to the dynamic anchor point radar sampling strategy, the problem of noise background affecting feature extraction is solved. In addition, the semantic information extracted under the guidance of dynamic anchor points is combined with query embedding to improve the accuracy of the HOI detection task. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0036] Figure 1 Schematic diagram of the interaction module in Example 1. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0038] Example 1:

[0039] A method for detecting human interaction based on dynamic anchor points, the steps are as follows:

[0040] Step 1: Split the image: Split the input RGB image into non-overlapping image blocks. For example, use swin_transformer or resnet to extract the backbone and obtain the feature map y (i.e., image block).

[0041] Step 2.1, calculate global attention: use the partition function Partition() to divide the feature map y into blocks, and the feature blocks after block The size is Then the feature block Pooling obtains a new feature block x, where H and W represent the height and width of the feature map y, and k h 、k w Represents the height and width of the window when partitioning. The formula is as follows:

[0042]

[0043]

[0044] This formula acts as a linear layer. Sw() is a simple linear layer, and the feature block x size is Among them, H, W, k h 、k w For the above.

[0045] Each feature block x is mapped to a query After pooling, concatenate and map to keys Sum According to the formula Calculate the global attention, where B represents the deviation predicted by the deep convolutional layer, d is a constant, Softmax() represents the normalized exponential function, i represents the i-th feature block, and F q ,F k ,F v Both are linear mapping layers.

[0046] Step 2.2, construct an interaction module consisting of layer normalization, global attention, and a multi-layer neural network. For example, construct a global interaction module consisting of LayerNorm, global attention, and MLP (Multilayer Perceptron), where LayerNorm and MLP use existing algorithms, such as Figure 1 As shown in the figure, the interaction module is applied to the tokens of the patch, maintaining the same number of tokens. Together with the linear embedding, this is called a stage. The interaction module adopts a hierarchical design and consists of four stages. Each stage reduces the resolution of the input feature map, scaling the image to different scales, extracting anchor points and features at the four scales mentioned above. These four features at different scales are called multi-scale features.

[0047] In this solution, the anchor points are calculated based on the input photo's features through a combination of steps 2.1 and 2.2. Since the anchor points vary for different photos, they can be called dynamic anchor points. In existing techniques, the anchor points are set from the outset and remain the same for all images, thus being considered static anchor points.

[0048] Step 3.1, generate the central anchor point: generate the central anchor point by taking the weighted average of all anchor points. The formula for calculating the coordinates of the central anchor point is as follows:

[0049]

[0050] Among them, num D Represents the total number of anchor points. Represents the coordinates of the i-th anchor point among all anchor points, {x Centre ,y Centre} represents the coordinates of the center anchor point.

[0051] Step 3.2, divide the anchor point arrangement interval: All anchor points are rearranged according to their distance from the central anchor point, and the interval [0, Lmax] is evenly divided into N intervals M, the formula is as follows:

[0052]

[0053] Among them, M i represents the i-th interval, L maxRepresents the distance from the farthest anchor point to the central anchor point, N represents the number of intervals, N is a positive integer, i = 0, 1, 2…N.

[0054] Step 3.3, calculate the distribution frequency: Calculate the number of anchor points in each distance interval and get the distribution frequency. The formula is as follows:

[0055]

[0056] in, Represents the number of anchor points in the Mi interval, num D Represents the total number of anchor points.

[0057] Step 3.4, equidistant sampling anchor points: perform equidistant sampling on each interval according to the distribution frequency. The formula is as follows:

[0058]

[0059] in, Represents the number of anchor points that need to be sampled within the interval, N q Represents the total number of anchor points that need to be sampled, P i Represents the Mth i The frequency distribution within an interval.

[0060] Step 3.5, matching sampling anchors and query embeddings: Match the sampling anchors and query embeddings according to the sampling order.

[0061] Step 4.1: Combine input image and query embedding: Use anchor points to effectively combine the input scene and query embedding. The calculation formula is as follows:

[0062]

[0063] Among them, MSDeformAttn represents multi-scale deformable attention, z q Represents the characterization features of the query, represents the coordinates of each query anchor point, represents a multi-scale feature map, m, l, k represent the index of the attention head, input feature scale, and sampling anchor point, respectively, ΔD mlqk and A mlqk They represent the sampling offset and attention weight of the kth dynamic anchor in the lth feature layer and the mth attention head, W m represents the connection matrix of m attention heads, x l and φ l Represents the feature map at the lth feature layer and the regression to the coordinates.

[0064] Step 4.2, generate a person interaction detection detector: the deformable transformer connects the multi-scale features with the query embedding to generate a person interaction detection detector;

[0065] Step 4.3, use the feedforward network to predict: the feedforward network includes a linear layer to process the human interaction detection embedding, a stacked decoding layer to calculate self-attention and multi-scale attention, and the last layer outputs the interaction detection head to predict the human interaction detection instance (object category, action category, human bounding box and object bounding box).

[0066] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for detecting human interaction based on dynamic anchor points, characterized in that: include: Step 1: Divide the acquired image into image blocks; Step 2: Calculate global attention and construct an interaction module consisting of layer normalization, global attention, and a multi-layer neural network. Apply the interaction module to the image patch to extract anchor points and multi-scale features. Step 3: Calculate the central anchor point based on the anchor points extracted in step 2. Rearrange the anchor points according to their distance from the central anchor point and divide them into intervals according to the distance. Calculate the distribution frequency based on the number of anchor points in the interval. Sample the anchor points in each distance interval equidistantly based on the distribution frequency to obtain sampled anchor points. Match the sampled anchor points with the query embedding in the sampling order. The formula for the center anchor point is: in, Represents the total number of anchor points; Represents the first of all anchor points The coordinates of the anchor points, Represents the coordinates of the center anchor point; The formula for dividing the interval is: in, represents the i-th interval, Represents the distance from the farthest anchor point to the center anchor point, represents the number of intervals, is a positive integer, = ; The formula for the equidistant sampling is: in, Represents the number of anchor points that need to be sampled within the interval, Represents the total number of anchor points that need to be sampled, Representative The frequency of distribution within a range; in, Represents the number of anchor points in the Mi interval, Represents the total number of anchor points; Step 4: Combine the acquired image and query embedding using the matched sampling anchors, add multi-scale features to the query embedding, and generate a person interaction detector; The formula for combining the acquired image and query embedding is: in, represents multi-scale deformable attention, Represents the characterization features of the query, represents the coordinates of the dynamic anchor point for each query, represents a multi-scale feature map, Represent the index of attention head, input feature scale, and sampling anchor point respectively, and Representing the The dynamic anchor point is feature layer, the The sampling offset and attention weight of each attention head, represent The connection matrix of the attention heads, Representatives in the The feature map of each feature layer and the regression of coordinates.

2. The method for detecting human interaction based on dynamic anchor points according to claim 1, characterized in that: In step 1, the image is divided into non-overlapping image blocks.

3. The method for detecting human interaction based on dynamic anchor points according to claim 1, wherein: In step 2, the step of calculating global attention is: using the block function to divide the image blocks into blocks of size feature blocks, each feature block is mapped to the query , pooled and concatenated, and then mapped to the key Sum , according to the formula , calculate the global attention, where B represents the deviation predicted by the deep convolutional layer, d is a constant, represents the normalized exponential function, Representative feature blocks, Represents the height and width of the token respectively, Represent the height and width of the feature block respectively.

4. The method for detecting human interaction based on dynamic anchor points according to claim 3, wherein: described ,in Both are linear mapping layers.

5. The method for detecting human interaction based on dynamic anchor points according to claim 3, characterized in that: The formula of the block function is, in, represents a linear layer, and y represents an image patch.

Images