A human interaction detection method based on dynamic noise reduction distillation and prior purification

Abstract

The present application belongs to the field of human interaction relationship detection, and discloses a human interaction detection method based on dynamic noise reduction distillation and prior purification, wherein a global encoding feature is generated for an image sample through an encoder; a learnable query, a real query and a noise query are constructed, the real query and the noise query are added to obtain a joint query, and the learnable query and the joint query are spliced to generate a mixed query; the global encoding feature and the mixed query are input into a decoder, the output of the decoder is extracted to obtain a student prediction set and a teacher prediction set, and a total loss is calculated for optimization update. The method effectively balances the noise reduction strategy and the knowledge distillation, minimizes the influence of negative samples in query distillation, and retains positive samples at the same time.

Description

Technical Field

[0001] The present invention belongs to the field of character interaction relationship detection, and specifically relates to a character interaction detection method based on dynamic noise reduction distillation and prior purification. Background Art

[0002] Human interaction detection is a research task in the field of computer vision, which aims to detect the interaction between people and objects in images or videos. It not only identifies people and objects in the image, but also understands and judges the specific interaction behaviors between them. It is widely used in intelligent monitoring and human-computer interaction image understanding. Research on HOI (Human-Object Interaction, human interaction relationship detection) can be roughly divided into two types, one is a two-stage detection method and the other is a one-stage detection method. The two-stage detection method first uses the existing detector to detect the target, and then determines the association between the targets. The one-stage detection method defines HOI as a set prediction problem, and uses the transformer architecture to obtain a triple (<Human,Object,Interaction> ) collection.

[0003] However, due to the limited information in the initial query and the limited number of annotated person and object pairs in most training images in existing datasets, these methods, while leveraging knowledge distillation, have improved their detection capabilities, but they still suffer from shortcomings: 1) One-stage detection methods often require longer training cycles than two-stage methods, and the addition of a teacher network further increases the number of parameters, exacerbating this shortcoming. 2) Traditional knowledge distillation can be divided into two types: global distillation, which simply calculates the distillation loss between the results of the learnable query and the results without filtering the results generated by the real query. This approach is overly simplistic and may result in the loss of prior knowledge. After all, even if prior knowledge is incorporated into the real query, not every query will necessarily yield the correct prediction. Precise distillation, on the other hand, selects the result set with the highest match between the two queries. However, since each query only contains one triplet, this approach underutilizes prior knowledge. Summary of the Invention

[0004] The purpose of the present invention is to provide a person interaction detection method based on dynamic denoising distillation and prior purification, which includes denoising knowledge distillation, a priori knowledge purifier and dynamic distillation function, effectively balancing denoising strategy and knowledge distillation, minimizing the impact of negative samples in query distillation, and retaining positive samples.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] A method for detecting human interaction based on dynamic noise reduction distillation and prior purification, comprising:

[0007] Extract labeled image samples from the training dataset, preprocess the image samples and generate global encoding features through the Transformer encoder;

[0008] Constructing a learnable query, a real query, and a noise query, adding the real query and the noise query to obtain a joint query, and concatenating the learnable query and the joint query to generate a hybrid query;

[0009] Input the global encoding features and the mixed query into the Transformer decoder, extract the result set generated by the Transformer decoder based on the learnable query as the student prediction set, extract the result set generated by the Transformer decoder based on the joint query as the teacher prediction set, and calculate the basic loss based on the student prediction set;

[0010] The teacher prediction set is input into the prior knowledge purifier to purify the teacher prediction set, and the distillation loss is calculated based on the purified teacher prediction set and the student prediction set;

[0011] The distillation loss weight is dynamically adjusted according to the training stage. The total loss is obtained by combining the distillation loss and the basic loss based on the distillation loss weight. The Transformer encoder, learnable query and Transformer decoder are optimized and updated based on the total loss. The final trained Transformer encoder, learnable query and Transformer decoder are used to output the person interaction relationship detection results for the input image.

[0012] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution. They are merely further supplements or optimizations. Under the premise that there are no technical or logical contradictions, each optional method can be combined separately for the above-mentioned overall solution, or multiple optional methods can be combined.

[0013] Preferably, the constructing of learnable queries, real queries and noise queries includes:

[0014] Random initialization generates learnable queries;

[0015] Initialize a learnable embedding matrix, compose the labels of the image samples in the training dataset into a label sequence, and multiply the label sequence with the embedding matrix to obtain the real query;

[0016] The true labels in the label sequence are randomly replaced, and the replaced label sequence is multiplied by the embedding matrix to obtain a noise query.

[0017] Preferably, the step of inputting the global encoding features and the mixed query into the Transformer decoder, extracting the result set generated by the Transformer decoder according to the learnable query as the student prediction set, and extracting the result set generated by the Transformer decoder according to the joint query as the teacher prediction set comprises:

[0018] F h =Decoder h (Q mix ,M)

[0019] F o =Decoder o (Q mix ,M)

[0020] F i =Decoder i (Cross((F h +F o ),Embed Verb ),M)

[0021] Where, F h Indicates the information decoded by the human body, F o Indicates the information after object decoding, F i Decoder represents the information after interactive decoding. h Represents human body decoding operation, Decoder o Indicates object decoding operation, Decoder i represents the interactive decoding operation, Q mix represents mixed query, M represents global encoding feature, Cross represents cross attention operation, Embed Verb represents the interaction tag embedding query;

[0022] Box h =FFN(F h )

[0023] Box o =FFN(F o )

[0024] Score0=FFN(F o )

[0025] Score i =FFN(F i )

[0026] In the formula, Box h Indicates the position information of the human body, Box oIndicates the location information of the object, Score0 indicates the object classification score, Score i represents the interaction operation score, FFN refers to feedforward neural network;

[0027] Then the result set output by the Transformer decoder is (Box h ,Box o ,Score0,Score i );

[0028] According to the dimension from the result set (Box h ,Box o ,Score0,Score i ) extracts the prediction results corresponding to the learnable query as the student prediction set, and extracts the prediction results corresponding to the joint query as the teacher prediction set.

[0029] Preferably, the purification method adopted by the prior knowledge purifier is a Top-K mechanism, a score mechanism or a fusion mechanism;

[0030] The Top-K mechanism is as follows: the prediction results in the teacher prediction set are sorted in descending order according to the interaction operation score, and the top K prediction results with the highest prediction probability are extracted as the purified teacher prediction set;

[0031] The scoring mechanism is as follows: the prediction results with interaction operation scores lower than the execution threshold in the teacher prediction set are removed to obtain a purified teacher prediction set;

[0032] The fusion mechanism is: adding the corresponding position values ​​of all prediction results in the teacher prediction set, and averaging them to obtain a prediction result as the purified teacher prediction set.

[0033] Preferably, the basic loss is calculated as follows:

[0034] Loss base =λ b L bbox +λ u L u +λ o L o +λ i L i

[0035] Where, Loss base is the basic loss, L bbox is the bounding box regression loss calculated by the absolute value loss function, L u is the bounding box regression loss calculated by the Giou loss function, L ois the object classification loss calculated by the cross entropy loss function based on the object classification scores in the student prediction set, L i is the interaction classification loss calculated by the focus loss function based on the score of interaction operations in the student prediction set, λ b ,λ u ,λ o ,λ i All are hyperparameters;

[0036] The distillation loss is calculated as follows:

[0037] Loss KD =D KL (logsoftmax(Mean(R s )),softmax(Refine(R t )))

[0038] Where, Loss KD represents the distillation loss, R s represents the student prediction set, R t Represents the teacher prediction set, Refine represents the refinement operation, Refine(R t ) represents the purified teacher prediction set, Mean represents the average processing, D KL represents the KL divergence loss function, logsoftmax represents the logarithmic normalized exponential function, and softmax represents the normalized exponential function;

[0039] The total loss is calculated as follows:

[0040] Loss=Loss base +αλ k Loss KD

[0041] Where Loss is the total loss, λ k is a hyperparameter and α is a smoothing factor.

[0042] Preferably, the dynamic adjustment of the distillation loss weight according to the training stage is implemented based on a smoothing factor, and the implementation method is as follows:

[0043] During the training iteration, the smoothing factor α is kept constant before reducing the learning rate. After reducing the learning rate, the smoothing factor α is set to linear decay, exponential decay, or logarithmic decay.

[0044] This paper provides a method for detecting human interaction based on dynamic noise reduction distillation and prior purification. First, a dynamic distillation optimization strategy is proposed to balance noise reduction and knowledge distillation, effectively reducing computational overhead while ensuring the high quality and stability of the distillation process. Secondly, a prior knowledge purifier is designed, operating between global and precise distillation. By pre-filtering the prediction results generated by real queries, it effectively removes redundant information and noise, improving the model's prediction accuracy and reliability. Finally, a systematic experimental evaluation is conducted on a public dataset, fully demonstrating the technical advancement and practical application value of this algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flowchart of a method for detecting human interaction based on dynamic noise reduction distillation and prior purification according to the present invention;

[0046] Figure 2 This is a data frame diagram of a human interaction detection method based on dynamic noise reduction distillation and prior purification according to the present invention. DETAILED DESCRIPTION

[0047] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0049] In order to overcome the defects of the existing technology for human interaction detection, this embodiment provides a human interaction detection method based on dynamic noise reduction distillation and prior purification to improve the accuracy of interaction detection.

[0050] like Figure 1 As shown, the human interaction detection method based on dynamic noise reduction distillation and prior purification in this embodiment includes the following steps:

[0051] Step S1: extract labeled image samples from the training dataset, preprocess the image samples and generate global encoding features through the Transformer encoder.

[0052] This embodiment first uses the ResNet-50 network model as a feature extractor to preprocess the image, scales, crops, and normalizes the source image samples to obtain an overall feature map E, and divides the overall feature map into several image blocks of the same size. Each image block is linearly mapped to a vector, and then the Transformer encoder is used to process the vectors corresponding to all image blocks to obtain the global encoding feature M.

[0053] Step S2: Construct a learnable query Q l , real query Q gt and noise query Q n , and further generate the joint query Q gt&n and mixed query Q mix .

[0054] Learnable Query Q l is randomly initialized, and the real query Q gt 、Noise query Q n It encodes all real samples about the image by using a learnable embedding matrix (two-dimensional matrix). Specifically, a learnable embedding matrix is ​​initialized, the labels of the image samples in the training dataset are combined into a label sequence, and the label sequence is multiplied by the embedding matrix to obtain the real query Q gt Randomly replace the true label in the label sequence and multiply the replaced label sequence with the embedding matrix to obtain the noise query Q n Noise is added to the query by randomly replacing the true label. The process of adding noise means randomly replacing the original correct sequence with other labels, but not every query needs to be replaced. This process is random. gt&n By the real query Q gt and noise query Q n The sum of these two queries is then added to the randomly generated learnable query Q l Splicing, finally get the mixed query Q mix :

[0055] Q mix =Concat(Q l ,Q gt&n )

[0056] Where Concat represents the concatenation operation. l As well as the learnable embedding matrix, they are updated during training using back-gradient propagation.

[0057] Step S3: Use the tri-classification decoder to complete the prediction of the three types of information, and learn the query Q l and the joint query Q gt&nThe result sets generated by these two queries are divided into teacher prediction sets and student prediction sets, and the student prediction sets are used to calculate the basic loss.

[0058] Three independent Transformer decoders are used to predict the three types of information. The human body decoder and object decoder extract corresponding features from the feature maps extracted by the backbone network, and the interaction decoder uses these subject and object features to identify interaction verbs, thereby improving the accuracy and efficiency of HOI detection.

[0059] F h =Decoder h (Q mix ,M)

[0060] F o =Decoder o (Q mix ,M)

[0061] F i =Decoder i (Cross((F h +F o ),Embed Verb ),M)

[0062] Where, Decoder h,o,i Respectively represent human body, object, and interaction decoding operations, which are used to obtain different information from triples; Cross represents cross attention operation, Embed Verb represents the action label embedding query; F h,o,i They represent the human body, object, and interactively decoded information respectively.

[0063] Box h =FFN(F h )

[0064] Box o =FFN(F o )

[0065] Score0=FFN(F o )

[0066] Score i =FFN(F i )

[0067] Where FFN refers to feedforward neural network, Box h,o Respectively represents the location information of humans or objects, Score i,o represent the interaction operation score and object classification score respectively.

[0068] Then the result set output by the Transformer decoder is (Box h ,Box o ,Score0,Score i ); because the number of queries does not change, each query corresponds to a triple, for example Figure 2 As shown in the <person, ship, driving>, the order of the query will not change, so the final result set is divided according to the dimension. h ,Box o ,Score0,Score i ) extracts the prediction results corresponding to the learnable query as the student prediction set, and extracts the prediction results corresponding to the joint query as the teacher prediction set. It is important to emphasize that the difference between these two types of queries lies only in the construction and naming of the initial query. During network training, both queries undergo the same processing flow.

[0069] Since knowledge distillation is implemented by designing a teacher network and a student network, the number of model parameters may increase significantly, which may have an adverse impact on the convergence of the model. To alleviate this problem, the present invention clearly distinguishes the result sets generated by the two types of queries, treating them as the teacher prediction set and the student prediction set respectively. The difference between the two is only reflected in the construction and name of the initial query. During the network training process, both queries undergo the same processing.

[0070] Compute the base loss on the set of student predictions:

[0071] Loss base =λ b L bbox +λ u L u +λ o L o +λ i L i

[0072] Where, Loss base is the basic loss, L bbox is the bounding box regression loss calculated by the absolute value loss function, L u is the bounding box regression loss calculated by the Giou loss function, L o is the object classification loss calculated by the cross entropy loss function based on the score Score0, L i The focus loss function is based on the score Score i The calculated interaction classification loss, λ b ,λ u ,λ o ,λ iare all hyperparameters, and their values ​​are set to 2.5, 1, 1, and 1 respectively.

[0073] Step S4: Pre-filter the results generated by the real query through the prior knowledge purifier, simplify the prediction results in the teacher prediction set, and calculate the distillation loss in combination with the student prediction set.

[0074] This implementation designs three different optimization strategies and selects one of them to simplify the prediction results in the teacher prediction set and minimize the distillation overhead:

[0075] 1) Top-K Mechanism: The Top-K mechanism selects the top K predictions with the highest probability from the teacher's prediction set. The specific steps are as follows: sort the predictions in the teacher's prediction set in descending order according to the interaction operation score, and extract the top K predictions with the highest prediction probability as the purified teacher's prediction set.

[0076] 2) Score Mechanism: A confidence threshold is set for the teacher prediction set to eliminate predictions that do not meet the required probability. The specific steps are as follows: Predictions with interaction scores below the execution threshold are removed from the teacher prediction set to obtain a purified teacher prediction set.

[0077] 3) Fusion Mechanism: This method uses mean fusion to merge all predictions in the teacher's prediction set, thus removing any restrictions on the correspondence between the student and teacher sets. The specific steps are as follows: All corresponding position values ​​of the predictions in the teacher's prediction set are added together and averaged to obtain a single prediction result, which serves as the purified teacher's prediction set.

[0078] After purifying the teacher prediction set, combine it with the student prediction set to calculate the distillation loss Loss KD :

[0079] Loss KD =D KL (logsoftmax(Mean(R s )),softmax(Refine(R t )))

[0080] Where, Loss KD represents the distillation loss, R s represents the student prediction set, R t Represents the teacher prediction set, Refine represents the refinement operation, Refine(R t ) represents the purified teacher prediction set, Mean represents the average processing, D KL represents the KL divergence loss function, logsoftmax represents the logarithmic normalized exponential function, and softmax represents the normalized exponential function.

[0081] S5. Dynamically adjust the distillation loss weights based on the training phase. The total loss is calculated by combining the distillation loss weights with the base loss. The Transformer encoder, learnable query, and Transformer decoder are optimized and updated based on the total loss. After training, the resulting Transformer encoder, optimal learnable query, and Transformer decoder are used to detect person interactions in the input image. In the inference application, the person interaction detection results are output for the input image.

[0082] Since the noise added to the model gradually increases, while the distillation process is stable, we improve the distillation loss by designing a dynamic smoothing factor to control the distillation process and precisely control the impact of the distillation process. Specifically, the weight of the distillation loss is continuously adjusted throughout the training process to adapt to the noise reduction strategy. Therefore, a stage-by-stage distillation optimization strategy is proposed. The final loss can be expressed as:

[0083] Loss=Loss base +αλ k Loss KD

[0084] Where Loss is the total loss, λ k is a hyperparameter set to 1, and α is the final designed balance solution. During training iterations, keeping the smoothing factor α constant before reducing the learning rate can make the model performance more stable. After reducing the learning rate, setting the smoothing factor α to linearly, exponentially, or logarithmically decay gradually frees the learnable queries from their dependence on real queries, thereby improving the performance of the final model and accelerating the model's convergence.

[0085] This embodiment provides a method for detecting human interaction based on dynamic noise reduction distillation and prior purification. First, a dynamic distillation optimization strategy is proposed to balance the relationship between noise reduction and knowledge distillation, effectively reducing computational overhead while ensuring the high quality and stability of the distillation process. Second, a prior knowledge purifier is designed that operates between global and precise distillation. By pre-filtering the prediction results generated by real queries, it effectively removes redundant information and noise, thereby improving the prediction accuracy and reliability of the model. Finally, a systematic experimental evaluation is conducted on a public dataset, fully confirming the technical advancement and practical application value of this algorithm.

[0086] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0087] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A method for detecting human interaction based on dynamic noise reduction distillation and prior purification, characterized in that: The human interaction detection method based on dynamic noise reduction distillation and prior purification includes: Extract labeled image samples from the training dataset, preprocess the image samples and generate global encoding features through the Transformer encoder; Constructing a learnable query, a real query, and a noise query, adding the real query and the noise query to obtain a joint query, and concatenating the learnable query and the joint query to generate a hybrid query; Input the global encoding features and the mixed query into the Transformer decoder, extract the result set generated by the Transformer decoder based on the learnable query as the student prediction set, extract the result set generated by the Transformer decoder based on the joint query as the teacher prediction set, and calculate the basic loss based on the student prediction set; The teacher prediction set is input into the prior knowledge purifier to purify the teacher prediction set, and the distillation loss is calculated based on the purified teacher prediction set and the student prediction set; The distillation loss weight is dynamically adjusted according to the training stage. The total loss is obtained by combining the distillation loss and the basic loss based on the distillation loss weight. The Transformer encoder, learnable query and Transformer decoder are optimized and updated based on the total loss. The final trained Transformer encoder, learnable query and Transformer decoder are used to output the person interaction relationship detection results for the input image.

2. The method for detecting human interaction based on dynamic noise reduction distillation and prior purification according to claim 1, characterized in that: The constructing of learnable queries, real queries and noise queries includes: Random initialization generates learnable queries; Initialize a learnable embedding matrix, compose the labels of the image samples in the training dataset into a label sequence, and multiply the label sequence with the embedding matrix to obtain the real query; The true labels in the label sequence are randomly replaced, and the replaced label sequence is multiplied by the embedding matrix to obtain a noise query.

3. The method for detecting human interaction based on dynamic noise reduction distillation and prior purification according to claim 1, characterized in that: The method of inputting the global encoding features and the mixed query into the Transformer decoder, extracting the result set generated by the Transformer decoder according to the learnable query as the student prediction set, and extracting the result set generated by the Transformer decoder according to the joint query as the teacher prediction set includes: F h =Decoder h (Q mix ,M) F o =Decoder o (Q mix ,M) F i =Decoder i (Cross((F h +F o ),Embed Verb ),M) Where, F h Indicates the information decoded by the human body, F o Indicates the information after object decoding, F i Decoder represents the information after interactive decoding. h Represents human body decoding operation, Decoder o Indicates object decoding operation, Decoder i represents the interactive decoding operation, Q mix represents mixed query, M represents global encoding feature, Cross represents cross attention operation, Embed Verb represents the interaction tag embedding query; Box h =FFN(F h ) Box o =FFN(F o ) Score0=FFN(F o ) Score i =FFN(F i ) In the formula, Box h Indicates the position information of the human body, Box o Indicates the location information of the object, Score0 indicates the object classification score, Score i represents the interaction operation score, FFN refers to feedforward neural network; Then the result set output by the Transformer decoder is (Box h , Box o , Score0, Score i ); According to the dimension from the result set (Box h , Box o , Score0, Score i ) extracts the prediction results corresponding to the learnable query as the student prediction set, and extracts the prediction results corresponding to the joint query as the teacher prediction set.

4. The method for detecting human interaction based on dynamic noise reduction distillation and prior purification according to claim 1, characterized in that: The purification method adopted by the prior knowledge purifier is a Top-K mechanism, a score mechanism or a fusion mechanism; The Top-K mechanism is as follows: the prediction results in the teacher prediction set are sorted in descending order according to the interaction operation score, and the top K prediction results with the highest prediction probability are extracted as the purified teacher prediction set; The scoring mechanism is as follows: the prediction results with interaction operation scores lower than the execution threshold in the teacher prediction set are removed to obtain a purified teacher prediction set; The fusion mechanism is: adding the corresponding position values ​​of all prediction results in the teacher prediction set, and averaging them to obtain a prediction result as the purified teacher prediction set.

5. The method for detecting human interaction based on dynamic noise reduction distillation and prior purification according to claim 1, characterized in that: The basic loss is calculated as follows: Loss base =λ b L bbox +λ u L u +λ o L o +λ i L i Where, Loss base is the basic loss, L bbox is the bounding box regression loss calculated by the absolute value loss function, L u is the bounding box regression loss calculated by the Giou loss function, L o is the object classification loss calculated by the cross entropy loss function based on the object classification scores in the student prediction set, L i is the interaction classification loss calculated by the focus loss function based on the score of interaction operations in the student prediction set, λ b ,λ u ,λ o ,λ i All are hyperparameters; The distillation loss is calculated as follows: Loss KD =D KL (logsoftmax(Mean(R s )),softmax(Refine(R t ))) Where, Loss KD represents the distillation loss, R s represents the student prediction set, R t Represents the teacher prediction set, Refine represents the refinement operation, Refine(R t ) represents the purified teacher prediction set, Mean represents the average processing, D KL represents the KL divergence loss function, logsoftmax represents the logarithmic normalized exponential function, and softmax represents the normalized exponential function; The total loss is calculated as follows: Loss=Loss base +αλ k Loss KD Where Loss is the total loss, λ k is a hyperparameter and α is a smoothing factor.

6. The method for detecting human interaction based on dynamic noise reduction distillation and prior purification according to claim 5, characterized in that: The dynamic adjustment of the distillation loss weight according to the training stage is implemented based on the smoothing factor. The implementation method is as follows: During the training iteration, the smoothing factor α is kept constant before reducing the learning rate. After reducing the learning rate, the smoothing factor α is set to linear decay, exponential decay, or logarithmic decay.

Images