Character interaction detection method based on dynamic noise reduction distillation and prior purification

Abstract

The invention belongs to the field of character interaction relationship detection, and discloses a character interaction detection method based on dynamic noise reduction distillation and prior purification, which comprises the following steps: generating global coding features for an image sample through an encoder; constructing a learnable query, a real query and a noise query, adding the real query and the noise query to obtain a joint query, and splicing the learnable query and the joint query to generate a mixed query; and inputting the global coding features and the mixed query into a decoder, extracting the output of the decoder to obtain a student prediction set and a teacher prediction set, and calculating the total loss for optimization and updating. According to the method, a noise reduction strategy and knowledge distillation are effectively balanced, the influence of negative samples in query distillation is minimized, and meanwhile, positive samples are reserved.

Description

Technical Field

[0001] The present invention belongs to the field of character interaction relationship detection, and specifically relates to a character interaction detection method based on dynamic noise reduction distillation and prior purification. Background Art

[0002] Human interaction detection is a research task in the field of computer vision, which aims to detect the interaction between people and objects in images or videos. It not only identifies people and objects in the image, but also understands and judges the specific interaction behaviors between them. It is widely used in intelligent monitoring and human-computer interaction image understanding. Research on HOI (Human-Object Interaction) can be roughly divided into two types, one is a two-stage detection method and the other is a one-stage detection method. The two-stage detection method first uses existing detectors to detect targets, and then determines the association between targets. The one-stage detection method defines HOI as a set prediction problem, and uses the transformer architecture to obtain triples (<Human,Object,Interaction> )'s collection.

[0003] However, on the one hand, the information in the initial query is limited; on the other hand, the number of annotated pairs of people and objects in most training images in the existing data sets is limited. Even though the detection capabilities of the methods have been improved by relying on knowledge distillation, these methods still have shortcomings: 1) The training process cycle of the one-stage detection method is often longer than that of the two-stage detection method, and the addition of the teacher network undoubtedly further increases the number of parameters, exacerbating this shortcoming. 2) In the past, there were two types of knowledge distillation: global distillation refers to not screening the results generated by the real query, and simply and roughly calculating the distillation loss of the results of the learnable query and its distillation loss. This method is too simple and may lead to the loss of prior knowledge. After all, even if prior knowledge is incorporated into the real query, not every query will necessarily get the correct prediction result. Precise distillation selects the calculation with the highest matching degree in the two query result sets, but each query only contains one triple information after all, this method leads to the prior knowledge not being fully utilized. Summary of the invention

[0004] The purpose of the present invention is to provide a person interaction detection method based on dynamic denoising distillation and prior purification, which includes denoising knowledge distillation, prior knowledge purifier and dynamic distillation function, effectively balances denoising strategy and knowledge distillation, minimizes the impact of negative samples in query distillation, and retains positive samples.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] A method for detecting human interaction based on dynamic noise reduction distillation and prior purification, the method comprising:

[0007] Extract labeled image samples from the training data set, preprocess the image samples and generate global encoding features through the Transformer encoder;

[0008] Constructing a learnable query, a real query and a noise query, adding the real query and the noise query to obtain a joint query, and concatenating the learnable query and the joint query to generate a hybrid query;

[0009] Input the global encoding features and the mixed query into the Transformer decoder, extract the result set generated by the Transformer decoder based on the learnable query as the student prediction set, extract the result set generated by the Transformer decoder based on the joint query as the teacher prediction set, and calculate the basic loss based on the student prediction set;

[0010] Input the teacher prediction set into the prior knowledge purifier, purify the teacher prediction set, and calculate the distillation loss based on the purified teacher prediction set and the student prediction set;

[0011] The distillation loss weight is dynamically adjusted according to the training stage, and the total loss is obtained by combining the distillation loss and the basic loss based on the distillation loss weight. The Transformer encoder, learnable query and Transformer decoder are optimized and updated based on the total loss. The Transformer encoder, learnable query and Transformer decoder obtained by the final training are used to output the character interaction relationship detection results for the input image.

[0012] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution, but are merely further supplements or preferences. Under the premise that there are no technical or logical contradictions, each optional method can be combined with the above-mentioned overall solution separately, and multiple optional methods can also be combined.

[0013] Preferably, the constructing of learnable queries, real queries and noise queries comprises:

[0014] Random initialization generates learnable queries;

[0015] Initialize a learnable embedding matrix, combine the labels of the image samples in the training dataset into a label sequence, and multiply the label sequence with the embedding matrix to obtain the real query;

[0016] The true labels in the label sequence are randomly replaced, and the replaced label sequence is multiplied by the embedding matrix to obtain a noise query.

[0017] Preferably, the step of inputting the global encoding features and the mixed query into the Transformer decoder, extracting the result set generated by the Transformer decoder according to the learnable query as the student prediction set, and extracting the result set generated by the Transformer decoder according to the joint query as the teacher prediction set comprises:

[0018] F h =Decoder h (Q mix ,M)

[0019] F o =Decoder o (Q mix ,M)

[0020] F i =Decoder i (Cross((F h +F o ),Embed Verb ),M)

[0021] In the formula, F h Indicates the information decoded by the human body, F o Indicates the decoded information of the object, F i Indicates the information after interactive decoding, Decoder h Indicates human body decoding operation, Decoder o Indicates object decoding operation, Decoder i represents the interactive decoding operation, Q mix represents a mixed query, M represents the global encoding feature, Cross represents the cross attention operation, and Embed Verb represents the interactive label embedding query;

[0022] Box h =FFN(F h )

[0023] Box o =FFN(F o )

[0024] Score0=FFN(F o )

[0025] Score i =FFN(F i )

[0026] In the formula, Box h Indicates the position information of the human body, Box oIndicates the location information of the object, Score0 indicates the object classification score, Score i represents the interactive operation score, FFN refers to feed-forward neural network;

[0027] The result set output by the Transformer decoder is (Box h ,Box o ,Score0,Score i );

[0028] According to the dimension from the result set (Box h ,Box o ,Score0,Score i ) extracts the prediction results corresponding to the learnable queries as the student prediction set, and extracts the prediction results corresponding to the joint queries as the teacher prediction set.

[0029] Preferably, the purification method adopted by the prior knowledge purifier is a Top-K mechanism, a score mechanism or a fusion mechanism;

[0030] The Top-K mechanism is: sorting the prediction results in the teacher prediction set in descending order according to the interaction operation score, and extracting the top K prediction results with the highest prediction probability as the purified teacher prediction set;

[0031] The score mechanism is as follows: the prediction results whose interactive operation scores are lower than the execution threshold in the teacher prediction set are removed to obtain a purified teacher prediction set;

[0032] The fusion mechanism is: adding the corresponding position values ​​of all prediction results in the teacher prediction set, and averaging to obtain a prediction result as the purified teacher prediction set.

[0033] Preferably, the basic loss is calculated as follows:

[0034] Loss base =λ b L bbox +λ u L u +λ o L o +λ i L i

[0035] In the formula, Loss base is the basic loss, L bbox is the bounding box regression loss calculated by the absolute value loss function, L u is the bounding box regression loss calculated by the Giou loss function, L ois the object classification loss calculated by the cross entropy loss function based on the object classification scores in the student prediction set, L i is the interactive classification loss calculated by the focus loss function based on the score of interactive operations in the student prediction set, λ b , u , o , i All are hyperparameters;

[0036] The distillation loss is calculated as follows:

[0037] Loss KD =D KL (logsoftmax(Mean(R s )),softmax(Refine(R t )))

[0038] In the formula, Loss KD represents the distillation loss, R s represents the set of student predictions, R t Represents the teacher's prediction set, Refine represents the refinement operation, Refine(R t ) represents the purified teacher prediction set, Mean represents the average processing, D KL represents the KL divergence loss function, logsoftmax represents the logarithmic normalized exponential function, and softmax represents the normalized exponential function;

[0039] The total loss is calculated as follows:

[0040] Loss=Loss base +αλ k Loss KD

[0041] In the formula, Loss is the total loss, λ k is a hyperparameter and α is a smoothing factor.

[0042] Preferably, the dynamic adjustment of the distillation loss weight according to the training stage is implemented based on the smoothing factor, and the implementation method is as follows:

[0043] During the training iteration, the smoothing factor α is kept constant before reducing the learning rate, and after reducing the learning rate, the smoothing factor α is set to linearly decay, exponentially decay, or logarithmically decay.

[0044] The present invention provides a method for detecting human interaction based on dynamic noise reduction distillation and prior purification. First, a dynamic distillation optimization strategy is proposed to balance the relationship between noise reduction and knowledge distillation, effectively reducing the computational overhead while ensuring the high quality and stability of the distillation process. Secondly, a prior knowledge purifier is designed, which operates between global and precise distillation. By pre-filtering the prediction results generated by real queries, redundant information and noise are effectively eliminated, thereby improving the prediction accuracy and reliability of the model. Finally, a systematic experimental evaluation is carried out on a public dataset to fully confirm the technical advancement and practical application value of this algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flow chart of a method for detecting human interaction based on dynamic noise reduction distillation and prior purification of the present invention;

[0046] Figure 2 This is a data frame diagram of a method for detecting human interaction based on dynamic noise reduction distillation and prior purification according to the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0049] In order to overcome the defects in the prior art for character interaction detection, this embodiment provides a character interaction detection method based on dynamic noise reduction distillation and prior purification to improve the accuracy of interaction detection.

[0050] like Figure 1 As shown, the human interaction detection method based on dynamic noise reduction distillation and prior purification in this embodiment includes the following steps:

[0051] Step S1: extract labeled image samples from the training data set, preprocess the image samples, and generate global encoding features through the Transformer encoder.

[0052] This embodiment first uses the ResNet-50 network model as a feature extractor to preprocess the image, scales, crops and normalizes the source image samples to obtain an overall feature map E, and divides the overall feature map into several image blocks of the same size, linearly maps each image block to a vector, and then uses the Transformer encoder to process the vectors corresponding to all image blocks to obtain the global encoding feature M.

[0053] Step S2: Construct a learnable query Q l 、Real query Q gt and noise query Q n , and further generate the joint query Q gt&n and mixed query Q mix .

[0054] Learnable Query Q l is randomly initialized, and the real query Q gt 、Noise query Q n It encodes all real samples about the image by using a learnable embedding matrix (two-dimensional matrix). Specifically, a learnable embedding matrix is ​​initialized, the labels of the image samples in the training data set are combined into a label sequence, and the label sequence is multiplied by the embedding matrix to obtain the real query Q gt Randomly replace the true label in the label sequence and multiply the replaced label sequence with the embedding matrix to obtain the noise query Q n Noise is added to the query by randomly replacing the true label. The process of adding noise means randomly replacing the original correct sequence with other labels, but not every query needs to be replaced. This process is random. gt&n By the real query Q gt and noise query Q n The sum of these two queries is then combined with the randomly generated learnable query Q l Finally, we get the mixed query Q mix :

[0055] Q mix =Concat(Q l ,Q gt&n )

[0056] In the formula, Concat represents the concatenation operation. l As well as the learnable embedding matrix, they are updated during training according to the back gradient propagation.

[0057] Step S3: Use the three-division decoder to complete the prediction of three types of information, and learn the query Q l and the joint query Q gt&nThe result sets generated by these two queries are divided into teacher prediction sets and student prediction sets, and the student prediction sets are used to calculate the basic loss.

[0058] Three independent Transformer decoders are used to predict the three types of information. The human decoder and object decoder extract corresponding features from the feature maps extracted from the backbone network, and the interaction decoder uses these subject and object features to identify interaction verbs, thereby improving the accuracy and efficiency of HOI detection.

[0059] F h =Decoder h (Q mix ,M)

[0060] F o =Decoder o (Q mix ,M)

[0061] F i =Decoder i (Cross((F h +F o ),Embed Verb ),M)

[0062] In the formula, Decoder h,o,i Respectively represent human body, object, and interactive decoding operations, which are used to obtain different information from triples; Cross represents cross attention operation, Embed Verb represents the action label embedding query; F h,o,i They represent the human body, objects, and interactively decoded information respectively.

[0063] Box h =FFN(F h )

[0064] Box o =FFN(F o )

[0065] Score0=FFN(F o )

[0066] Score i =FFN(F i )

[0067] In the formula, FFN refers to feedforward neural network, Box h,o Respectively represent the location information of humans or objects, Score i,o represent the interaction operation score and object classification score respectively.

[0068] The result set output by the Transformer decoder is (Box h ,Box o ,Score0,Score i ); because the number of queries does not change, each query corresponds to a triple, for example Figure 2 As shown in the figure <person, ship, driver>, the order of the query will not change, so the final result set is divided according to the dimension. h ,Box o ,Score0,Score i ) extracts the prediction results corresponding to the learnable query as the student prediction set, and extracts the prediction results corresponding to the joint query as the teacher prediction set. It should be emphasized that the difference between the two types of queries is only reflected in the construction and naming of the initial query. During the network training process, both queries go through the same processing flow.

[0069] Since knowledge distillation is achieved by designing a teacher network and a student network, the number of model parameters may increase significantly, which may have an adverse effect on the convergence of the model. In order to alleviate this problem, the present invention clearly distinguishes the result sets generated by the two types of queries, which are regarded as teacher prediction sets and student prediction sets respectively. The difference between the two is only reflected in the construction and name of the initial query. During the network training process, the two queries undergo the same processing.

[0070] Find the base loss on the set of student predictions:

[0071] Loss base =λ b L bbox +λ u L u +λ o L o +λ i L i

[0072] In the formula, Loss base is the basic loss, L bbox is the bounding box regression loss calculated by the absolute value loss function, L u is the bounding box regression loss calculated by the Giou loss function, L o is the object classification loss calculated by the cross entropy loss function based on the score Score0, L i The focus loss function is based on the score Score i The calculated interaction classification loss, λ b , u , o , iare all hyperparameters, and their values ​​are set to 2.5, 1, 1, and 1 respectively.

[0073] Step S4: Pre-filter the results generated by the real query through the prior knowledge purifier, simplify the prediction results in the teacher prediction set, and calculate the distillation loss in combination with the student prediction set.

[0074] This implementation designs three different optimization strategies and selects one of them to simplify the prediction results in the teacher prediction set and minimize the distillation overhead:

[0075] 1) Top-K mechanism: The Top-K mechanism refers to selecting the top K predictions with the highest probability from the teacher prediction set. The specific steps are as follows: sort the prediction results in the teacher prediction set in descending order according to the interaction operation score, and extract the top K prediction results with the highest prediction probability as the purified teacher prediction set.

[0076] 2) Score mechanism: A confidence threshold is set for the teacher prediction set to eliminate predictions that do not meet the required probability. The specific steps are as follows: The prediction results with interaction operation scores lower than the execution threshold in the teacher prediction set are removed to obtain the purified teacher prediction set.

[0077] 3) Fusion mechanism: The mean fusion method is used to merge all the prediction results in the teacher prediction set, so as not to restrict the correspondence between the student and teacher sets. The specific steps are as follows: add the corresponding position values ​​of all prediction results in the teacher prediction set, and average them to obtain a prediction result as the purified teacher prediction set.

[0078] After purifying the teacher prediction set, combine it with the student prediction set to calculate the distillation loss Loss KD :

[0079] Loss KD =D KL (logsoftmax(Mean(R s )),softmax(Refine(R t )))

[0080] In the formula, Loss KD represents the distillation loss, R s represents the set of student predictions, R t Represents the teacher's prediction set, Refine represents the refinement operation, Refine(R t ) represents the purified teacher prediction set, Mean represents the average processing, D KL represents the KL divergence loss function, logsoftmax represents the logarithmic normalized exponential function, and softmax represents the normalized exponential function.

[0081] S5. Dynamically adjust the distillation loss weight according to the training stage, combine the distillation loss with the basic loss to obtain the total loss based on the distillation loss weight, and optimize and update the Transformer encoder, learnable query, and Transformer decoder based on the total loss. After the training is completed, the final trained Transformer encoder, optimal learnable query, and Transformer decoder are used to obtain the character interaction relationship for the input image. In the reasoning application, the character interaction relationship detection result is output for the input image.

[0082] Since the noise added to the model increases gradually, while the distillation process is stable, a dynamic smoothing factor is designed to control the distillation process to improve the distillation loss in order to precisely control the impact of the distillation process. Specifically, the weight of the distillation loss is continuously adjusted throughout the training process to accommodate the inclusion of the noise reduction strategy. Therefore, a stage-level distillation optimization strategy is proposed. The final loss can be expressed as:

[0083] Loss=Loss base +αλ k Loss KD

[0084] In the formula, Loss is the total loss, λ k is a hyperparameter, and its value is set to 1. α is the final designed balance solution, that is, in the training iteration process, keeping the smoothing factor α constant before reducing the learning rate can make the performance of the model more stable. After reducing the learning rate, set the smoothing factor α to decay linearly, exponentially, or logarithmically, gradually making the learnable queries free from the dependence on the real queries, thereby improving the performance of the final model and accelerating the convergence of the model.

[0085] This embodiment provides a method for detecting human interaction based on dynamic noise reduction distillation and prior purification. First, a dynamic distillation optimization strategy is proposed to balance the relationship between noise reduction and knowledge distillation, while effectively reducing the computational overhead and ensuring the high quality and stability of the distillation process. Secondly, a prior knowledge purifier is designed, which operates between global and precise distillation, and effectively removes redundant information and noise by pre-filtering the prediction results generated by real queries, thereby improving the prediction accuracy and reliability of the model. Finally, a systematic experimental evaluation is carried out on a public dataset to fully confirm the technical advancement and practical application value of this algorithm.

[0086] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0087] The above-mentioned embodiments only express several implementation modes of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A method for detecting human interaction based on dynamic noise reduction distillation and prior purification, characterized in that: The method for detecting human interaction based on dynamic noise reduction distillation and prior purification includes: Extract labeled image samples from the training data set, preprocess the image samples and generate global encoding features through the Transformer encoder; Constructing a learnable query, a real query and a noise query, adding the real query and the noise query to obtain a joint query, and concatenating the learnable query and the joint query to generate a hybrid query; Input the global encoding features and the mixed query into the Transformer decoder, extract the result set generated by the Transformer decoder based on the learnable query as the student prediction set, extract the result set generated by the Transformer decoder based on the joint query as the teacher prediction set, and calculate the basic loss based on the student prediction set; Input the teacher prediction set into the prior knowledge purifier, purify the teacher prediction set, and calculate the distillation loss based on the purified teacher prediction set and the student prediction set; The distillation loss weight is dynamically adjusted according to the training stage, and the total loss is obtained by combining the distillation loss and the basic loss based on the distillation loss weight. The Transformer encoder, learnable query and Transformer decoder are optimized and updated based on the total loss. The Transformer encoder, learnable query and Transformer decoder obtained by the final training are used to output the character interaction relationship detection results for the input image.

2. The method for detecting human interaction based on dynamic noise reduction distillation and prior purification according to claim 1, characterized in that: The constructing of learnable queries, real queries and noise queries includes: Random initialization generates learnable queries; Initialize a learnable embedding matrix, combine the labels of the image samples in the training dataset into a label sequence, and multiply the label sequence with the embedding matrix to obtain the real query; The true labels in the label sequence are randomly replaced, and the replaced label sequence is multiplied by the embedding matrix to obtain a noise query.

3. The method for detecting human interaction based on dynamic noise reduction distillation and prior purification according to claim 1, characterized in that: The method of inputting the global encoding features and the mixed query into the Transformer decoder, extracting the result set generated by the Transformer decoder according to the learnable query as the student prediction set, and extracting the result set generated by the Transformer decoder according to the joint query as the teacher prediction set includes: F h =Decoder h (Q mix ,M) F o =Decoder o (Q mix ,M) F i =Decoder i (Cross((F h +F o ),Embed Verb ),M) In the formula, F h Indicates the information decoded by the human body, F o Indicates the decoded information of the object, F i Indicates the information after interactive decoding, Decoder h Indicates human body decoding operation, Decoder o Indicates object decoding operation, Decoder i represents the interactive decoding operation, Q mix represents a mixed query, M represents the global encoding feature, Cross represents the cross attention operation, and Embed Verb represents the interactive label embedding query; Box h =FFN(F h ) Box o =FFN(F o ) Score0=FFN(F o ) Score i =FFN(F i ) In the formula, Box h Indicates the position information of the human body, Box o Indicates the location information of the object, Score0 indicates the object classification score, Score i represents the interactive operation score, FFN refers to feed-forward neural network; The result set output by the Transformer decoder is (Box h , Box o , Score0, Score i ); According to the dimension from the result set (Box h , Box o , Score0, Score i ) extracts the prediction results corresponding to the learnable queries as the student prediction set, and extracts the prediction results corresponding to the joint queries as the teacher prediction set.

4. The method for detecting human interaction based on dynamic noise reduction distillation and prior purification according to claim 1, characterized in that: The purification method adopted by the prior knowledge purifier is a Top-K mechanism, a score mechanism or a fusion mechanism; The Top-K mechanism is: sorting the prediction results in the teacher prediction set in descending order according to the interaction operation score, and extracting the top K prediction results with the highest prediction probability as the purified teacher prediction set; The score mechanism is as follows: the prediction results whose interactive operation scores are lower than the execution threshold in the teacher prediction set are removed to obtain a purified teacher prediction set; The fusion mechanism is: adding the corresponding position values ​​of all prediction results in the teacher prediction set, and averaging to obtain a prediction result as the purified teacher prediction set.

5. The method for detecting human interaction based on dynamic noise reduction distillation and prior purification according to claim 1, characterized in that: The basic loss is calculated as follows: Loss base =λ b L bbox +λ u L u +λ o L o +λ i L i In the formula, Loss base is the basic loss, L bbox is the bounding box regression loss calculated by the absolute value loss function, L u is the bounding box regression loss calculated by the Giou loss function, L o is the object classification loss calculated by the cross entropy loss function based on the object classification scores in the student prediction set, L i is the interactive classification loss calculated by the focus loss function based on the score of interactive operations in the student prediction set, λ b , u , o , i All are hyperparameters; The distillation loss is calculated as follows: Loss KD =D KL (logsoftmax(Mean(R s )),softmax(Refine(R t ))) In the formula, Loss KD represents the distillation loss, R s represents the set of student predictions, R t Represents the teacher's prediction set, Refine represents the refinement operation, Refine(R t ) represents the purified teacher prediction set, Mean represents the average processing, D KL represents the KL divergence loss function, logsoftmax represents the logarithmic normalized exponential function, and softmax represents the normalized exponential function; The total loss is calculated as follows: Loss=Loss base +αλ k Loss KD In the formula, Loss is the total loss, λ k is a hyperparameter and α is a smoothing factor.

6. The method for detecting human interaction based on dynamic noise reduction distillation and prior purification according to claim 5, characterized in that: The dynamic adjustment of the distillation loss weight according to the training stage is implemented based on the smoothing factor, and the implementation method is as follows: During the training iteration, the smoothing factor α is kept constant before reducing the learning rate, and after reducing the learning rate, the smoothing factor α is set to linearly decay, exponentially decay, or logarithmically decay.

Images