Interaction detection method based on encoder-decoder structure to enhance samples and queries

Abstract

The present invention discloses an interaction detection method based on enhanced samples and enhanced queries of an encoder-decoder structure, comprising the steps of inputting a labeled interaction image into a feature extraction module to obtain image features output by the feature extraction module; inputting the image features into the dynamic language information generation query module and the mask truth denoising module respectively to obtain two groups of queries for querying people, objects and interactions; inputting the two groups of queries into the decoder to obtain two groups of features for predicting interaction triplets; and then inputting the two groups of features into the interaction prediction module to obtain prediction results. Based on the multi-label Focal loss and interaction loss, the prediction results are supervised and the network parameters of the deep learning model are updated; the optimal deep learning model after training is taken to output the prediction results for the interaction image to be detected. The present invention introduces the dynamic language information generation query module and the mask truth denoising module to overcome the technical problems raised in the above background technology.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method for detecting interaction between a secondary path and a primary path in learning based on an encoder-decoder. Background Art

[0002] Human-object interaction detection is a multi-task learning process that holds significant significance for image semantic understanding and human-centric scene understanding. In factories and workshops, interaction detection can identify as many safety issues as possible among workers in the environment, thereby preventing accidents. Therefore, the interaction detection task essentially involves detecting as many interaction triplets (person, object, interaction) as possible in an image, which requires both recognition and positioning.

[0003] Currently, interaction detection is primarily categorized as convolutional network-based interaction detection and encoder-decoder-based interaction detection. Convolutional network-based detection can be further categorized into one-stage and two-stage detection methods. Two-stage detection relies on an advanced target detection model to identify and locate people and objects, then pairs them in a specific manner and feeds them into interaction detection to generate interaction triplets. Two-stage detection, on the other hand, generates interaction triplets in a single step, eliminating the pairing process. Both one-stage and two-stage methods are lightweight because they are based on convolutional networks. However, the local nature of convolution can lead to inaccurate or even inability to recognize interactions over long distances.

[0004] Interaction detection based on the encoder-decoder can be further divided into single-query detection and three-query detection. Single-query detection uses only one query to obtain interaction triplets, while three-query detection uses three queries to obtain the person, object, and interaction respectively, and then forms the interaction triplets. However, they both predict the interactions that appear in the image based on the concept of collection and use the bipartite matching algorithm to achieve supervision of the prediction by the true value. At the same time, due to the self-attention and cross-attention in the encoder-decoder, not only short-distance interactions can be detected, but also long-distance dependencies. More importantly, the object detection effect under the encoder-decoder structure is also excellent, which means that the pairing of people and objects will be more meaningful and also based on reference.

[0005] However, encoder-decoder-based detection uses a fixed, identical query for all inputs, requiring the model to perform case-by-case analysis with significant capabilities. Furthermore, current detection with this structure is limited by the discrete nature of binary matching. This means that for the same input, the model's predictions and true values ​​may not match consistently in each training round, preventing the predictions from learning a single true value. This leads to learning dispersion. These two factors significantly hinder interactive detection based on encoder-decoders. Summary of the Invention

[0006] The present invention aims to overcome the above-mentioned problems of the prior art and provides an interactive detection method for enhanced samples and enhanced queries based on an encoder-decoder structure. The present invention overcomes the technical problems raised in the above-mentioned background technology by introducing a dynamic language information generation query module and a masked true value denoising module.

[0007] To achieve the above object, the technical solution adopted by the present invention is:

[0008] The interaction detection method of enhanced samples and enhanced queries based on the encoder-decoder structure is performed by constructing a deep learning model. The deep learning model includes a feature extraction module, a dynamic language information query generation module, a masked true value denoising module, and an interaction prediction module. The interaction detection method of enhanced samples and enhanced queries based on the encoder-decoder structure includes:

[0009] Taking the labeled interaction picture and inputting it into the feature extraction module to obtain the picture features output by the feature extraction module, the feature extraction module includes a deep residual network and an encoder;

[0010] The image features are simultaneously input into the dynamic language information generation query module and the mask truth denoising module, and the two outputs are respectively input into the decoder to obtain features for people, objects and interactions;

[0011] The features of people, objects and interactions obtained by the decoder of the two modules are input into the interaction prediction module to obtain the corresponding prediction results Pr l ,Pr g ∈{people, things, interactions};

[0012] Supervising the dynamic language information generation query module and the interaction detection module based on multi-label Focal loss and interaction loss, and updating the network parameters of the deep learning model;

[0013] The best deep learning model after training is used to output the prediction results for the interaction image to be detected.

[0014] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution. They are merely further supplements or optimizations. Under the premise that there are no technical or logical contradictions, each optional method can be combined separately for the above-mentioned overall solution, or multiple optional methods can be combined.

[0015] Preferably, the deep residual network is a deep residual network ResNet-50.

[0016] Preferably, the processing of the decoder unit includes:

[0017] The input image features are used as query library E, and then query library E inputs dynamic language information to generate query module, generating query Q guided by language information l The detailed process is as follows:

[0018] Enter the query library E into FFN cand (·) Get the prediction set pred of generated language information hoi , and then get the candidate set Cand for generating language information based on Topk(·) hoi .

[0019]

[0020] in Represents the jth sentence in the interaction description statement. According to the candidate set Cand hoi and Sem2Vis(·), generating a query Q guided by language information l .

[0021] Q l =Sem2Vis(Cand hoi ) (2)

[0022] According to the true value GT input mask true value denoising module, generate a query Q with true value guidance g , the detailed process is as follows:

[0023] gt item =[box k ,c k ,cdist,area k ],k∈{h,o} (3)

[0024] According to the true value gt item , we can get a B×B binary mask map G, where box, c, area, and cdist represent the box, center point, area, and center distance of a person or object, respectively, and h and o represent people and objects, respectively.

[0025] Mask=Thres(P,T)&G (4)

[0026] Then use the Laplace probability function Generate a probability map P of the same size as the mask map, and use the probability T as the dividing point to divide the probability according to the threshold function Thres(·,·) Figure 2 Finally, the binary probability map and the true value gt item The binary mask image obtained is subjected to an AND operation to obtain a Mask containing a true value sample of noise.

[0027] Q g =FFN mask(Mask) (5)

[0028] Using true value samples containing noise and FFN mask (·) Get the truth query Q g .

[0029] Preferably, the prediction results are as follows:

[0030]

[0031] Among them, Pr l ,Pr g They represent the prediction result outputs of the dynamic language information generation query module and the mask truth denoising module respectively. Represent the regression or classification functions for people on the main road and the secondary road respectively, Represent the regression or classification functions for objects on the main road and the secondary road respectively, Represent the regression or classification functions used for interaction on the main path and the secondary path respectively.

[0032] Preferably, the supervised dynamic language information generation query module includes:

[0033] Generate pred generated by the dynamic language information query module hoi and true value label hoi The multi-label Focal loss L of the supervised dynamic language information generation query module is calculated by adaptive Focal loss MFL , the mathematical expression is as follows:

[0034]

[0035] where num pos Indicates the number of positive samples and negative samples, inds pos and inds neg Through label hoi The obtained positive and negative sample subscripts, γ is a hyperparameter. The multi-label Focal loss can measure the pred generated by the dynamic language information generation query module hoi For the true value label hoi coverage situation.

[0036] Preferably, the supervisory interaction detection module includes:

[0037] The general interaction loss supervision includes the mean absolute error of the bounding box position L b , the bounding box size generalization intersection-union ratio L u , object classification cross entropy L c and interactive classification cross entropy L a, so the general interaction loss function L hoi The mathematical expression is as follows:

[0038] L hoi =λ b L b +λ u L u +λ c L c +λ a L a (8)

[0039] where λ b ,λ u ,L c and L a All are super parameters.

[0040] Since the dynamic language information generation query module and the mask truth value denoising module will both pass through the interactive detection module to generate prediction results, they each need to use L hoi get and Supervise the prediction results of each party, and then according to the L MFL , and the total loss L is obtained total , mathematically expressed as follows:

[0041]

[0042] Where α is a hyperparameter.

[0043] The interactive detection method of enhanced samples and enhanced queries based on the encoder-decoder structure provided by the present invention introduces a dynamic language information generation query module, so that the model can adapt to the input image and generate specific search areas and appearance features; introduces a masked true value denoising module, so that the model can skip bipartite matching by using true value queries with trace noise, and achieve complete consistency between the prediction results and true value matching of the same input in different rounds of training, thereby solving the problem of learning dispersion. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Flowchart of the interactive detection method of enhanced samples and enhanced queries based on the encoder-decoder structure of the present invention;

[0045] Figure 2 Schematic diagram of the structure of the deep learning model of the present invention. DETAILED DESCRIPTION

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0048] In one embodiment, Figure 1 As shown, a method for interaction detection based on enhanced samples and enhanced queries of an encoder-decoder structure is proposed. Interaction detection is performed by constructing a deep learning model. The deep learning model includes a feature extraction module, a dynamic language information query generation module, a masked truth denoising module, and an interaction prediction module. The method for interaction detection based on enhanced samples and enhanced queries includes:

[0049] Step S1: Take the labeled interactive image and input it into the feature extraction module to obtain the image features output by the feature extraction module. The feature extraction module includes a deep residual network and an encoder.

[0050] For images that need interaction detection, this implementation first randomly enlarges and randomly crops the image and its corresponding annotation to obtain input I∈R 3×H×W , and then input I into the network.

[0051] For example, after enlarging the image by 1 times, the size and coordinates of the bounding boxes of the corresponding people and objects must also increase by a multiple; if the image is randomly cropped, the size and coordinates of the bounding boxes must be based on the cropping area, and the intersection of the two must be taken to correct the size and coordinates of the bounding boxes.

[0052] The feature extraction module of this embodiment uses the pre-trained deep residual network ResNet-50 and the encoder of the encoder-decoder to extract the image features E∈R from the processed image I. D×H×W .

[0053] Step S2: Input the image features into the dynamic language information generation query module and the mask truth value denoising module respectively to obtain two instructive queries.

[0054] like Figure 2 As shown, the detailed process of using the query library E to generate a query containing language information guidance is as follows: Input the query library E into FFN cand(·) Get the prediction set pred of generated language information hoi , and then get the candidate set Cand for generating language information based on Topk(·) hoi .

[0055]

[0056] According to the candidate set Cand hoi and Sem2Vis(·), generating a query Q guided by language information l .

[0057] Q l =Sem2Vis(Cand hoi ) (2)

[0058] The detailed process of generating a query containing truth value guidance in this embodiment is as follows:

[0059] According to the true value GT input mask true value denoising module, generate a query Q with true value guidance g The detailed process is as follows:

[0060] gt item =[box k ,c k ,cdist,area k ],k∈{h,o} (3)

[0061] According to the true value gt item , we can get a B×B binary mask map G, where box, c, area, and cdist represent the box, center point, area, and center distance of a person or object, respectively, and h and o represent people and objects, respectively.

[0062] Mask=Thres(P,T)&G (4)

[0063] Then use the Laplace probability function Generate a probability map P of the same size as the mask map, and use the probability T as the dividing point to divide the probability according to the threshold function Thres(·,·) Figure 2 Finally, the binary probability map and the true value gt item The binary mask image obtained is subjected to an AND operation to obtain a Mask containing a true value sample of noise.

[0064] Q g =FFN mask (Mask) (5)

[0065] Using true value samples containing noise and FFN mask (·) Get the truth query Q g .

[0066] Step S3: Input two instructive queries into the decoder to obtain two sets of features for predicting interaction triples. The detailed steps are as follows:

[0067] like Figure 2 As shown, this embodiment uses a decoder to respectively decode the query Q containing language information guidance. l and the query Q with truth value guidance g Get the corresponding feature e l and e g , the expression is as follows:

[0068]

[0069] Where decoder represents a decoder.

[0070] Step S4: Input the two sets of features for people, objects and interactions into the interaction prediction module to obtain the prediction results Pr corresponding to the main path and the secondary path respectively. l ,Pr g ∈{people, things, interactions}

[0071] After obtaining two sets of features, the features are sent to the interactive prediction module to obtain their respective prediction results. The prediction process of the two groups is shown as follows:

[0072]

[0073] Among them, Pr l ,Pr g They represent the prediction result outputs of the dynamic language information generation query module and the mask truth denoising module respectively. Represent the regression or classification functions for people on the main road and the secondary road respectively, Represent the regression or classification functions for objects on the main road and the secondary road respectively, Represent the regression or classification functions used for interaction on the main path and the secondary path respectively.

[0074] Furthermore, the regression or classification function in this embodiment adopts an existing function, such as logistic regression, quantile regression, least squares regression, etc.

[0075] Step S5: Based on the multi-label focal loss and interaction loss, the prediction results are supervised and the network parameters of the deep learning model are updated.

[0076] The multi-label Focal loss is used to supervise the dynamic language information generation query module, as follows:

[0077] Generate pred generated by the dynamic language information query module hoi and true value labelhoi The multi-label Focal loss L of the supervised dynamic language information generation query module is calculated by adaptive Focal loss MFL , the mathematical expression is as follows:

[0078]

[0079] where num pos Indicates the number of positive samples and negative samples, inds pos and inds neg Through label hoi The obtained positive and negative sample subscripts, γ is a hyperparameter. The multi-label Focal loss can measure the pred generated by the dynamic language information generation query module hoi For the true value label hoi coverage situation.

[0080] The interaction loss is used to supervise the correctness of the interaction triples. The specific process is as follows: The general interaction loss supervision includes the mean absolute error L of the bounding box position b , the bounding box size generalization intersection-union ratio L u , object classification cross entropy L c and interactive classification cross entropy L a , so the general interaction loss function L hoi The mathematical expression is as follows:

[0081] L hoi =λ b L b +λ u L u +λ c L c +λ a L a (8)

[0082] where λ b ,λ u ,L c and L a All are super parameters.

[0083] Since the dynamic language information generation query module and the mask truth value denoising module will both pass through the interactive detection module to generate prediction results, they each need to use L hoi get and Supervise the prediction results of each party, and then according to the L MFL , and the total loss L is obtained total , mathematically expressed as follows:

[0084]

[0085] Where α is a hyperparameter.

[0086] According to the total loss L total , gradient backpropagation, optimize the model, and complete the training.

[0087] During the training process, a total of 80 rounds of training were conducted with a batch size of 12. The learning rates of ResNet and encoder-decoder were initialized to 0.00001 and 0.0001, respectively. The AdamW algorithm was used for 10 iterative trainings and the results of each iteration were saved. In the 50th round, the learning rates were reduced to one tenth of the original values.

[0088] Step S6: Take the optimal deep learning model after training and output the prediction results for the interaction image to be detected. Consistency learning is only used during training. In actual application, only the prediction results output by the main path are used.

[0089] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0090] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A method for interaction detection based on enhanced samples and enhanced queries of an encoder-decoder structure, which performs interaction detection by constructing a deep learning model, characterized in that: The deep learning model includes a feature extraction module, a dynamic language information generation query module, a mask truth denoising module and an interaction prediction module. The method based on enhanced samples and enhanced query interaction detection includes: Taking the labeled interaction picture and inputting it into the feature extraction module to obtain the picture features output by the feature extraction module, the feature extraction module includes a deep residual network and an encoder; The image features are simultaneously input into the dynamic language information generation query module and the mask truth value denoising module, and the two outputs are respectively input into the decoder to obtain features for people, objects and interactions; the processing process of the dynamic language information generation query module and the mask truth value denoising module includes: The input image features are used as query library E, and then query library E inputs dynamic language information to generate query module, generating query Q guided by language information l , the detailed process is as follows: Enter the query library E into FFN cand (·) Get the prediction set pred of generated language information hoi , and then get the candidate set Cand for generating language information based on Topk(·) hoi ; According to the candidate set Cand hoi and Sem2Vis(·), generating a query Q guided by language information l ; Q l =Sem2Vis(When hoi ) (2) According to the true value GT input mask true value denoising module, generate a query Q with true value guidance g , the detailed process is as follows: gt item =[box k ,c k ,cdist,area k ],k∈{h,o} (3) According to the true value gt item , we can get a B×B binary mask image G, where box, c, area, cdist represent the box, center point, area and center distance of a person or object respectively, and h, o represent a person and an object respectively; Mask=Thres(P,T)&G (4) Then use the Laplace probability function Generate a probability map P of the same size as the mask map, and binarize the probability map with probability T as the dividing point according to the threshold function Thres(·,·); finally, the binarized probability map and the true value gt are compared. item The binary mask image obtained is subjected to AND operation to obtain the Mask containing the true value sample of noise; Q g =FFN mask (Mask) (5) Using true value samples containing noise and FFN mask (·) Get the truth query Q g ; The features of people, objects and interactions obtained by the decoder of the two modules are input into the interaction prediction module to obtain the corresponding prediction results Pr l ,Pr g ∈{people, things, interactions}; Supervising the dynamic language information generation query module and the interaction detection module based on multi-label Focal loss and interaction loss, and updating the network parameters of the deep learning model; The best deep learning model after training is used to output the prediction results for the interaction image to be detected.

2. The method for interactive detection of enhanced samples and enhanced queries based on an encoder-decoder structure according to claim 1, wherein: The deep residual network is the deep residual network ResNet-50.

3. The interactive detection method of enhanced samples and enhanced queries based on the encoder-decoder structure according to claim 1, characterized in that The prediction results are as follows: Among them, Pr l ,Pr g They represent the prediction result outputs of the dynamic language information generation query module and the mask truth denoising module respectively. Represent the regression or classification functions for people on the main road and the secondary road respectively, Represent the regression or classification functions for objects on the main road and the secondary road respectively, Represent the regression or classification functions used for interaction on the main path and the secondary path respectively.

4. The interactive detection method of enhanced samples and enhanced queries based on an encoder-decoder structure according to claim 1, characterized in that: The supervised dynamic language information generation query module includes: Generate pred generated by the dynamic language information query module hoi and true value label hoi The multi-label Focal loss L of the supervised dynamic language information generation query module is calculated by adaptive Focal loss MFL , the mathematical expression is as follows: where num pos Indicates the number of positive samples and negative samples, inds pos and inds neg Through label hoi The obtained positive and negative sample subscripts, γ is a hyperparameter; the multi-label Focal loss can measure the pred generated by the dynamic language information generation query module hoi For the true value label hoi coverage situation.

5. The method for interactive detection of enhanced samples and enhanced queries based on an encoder-decoder structure according to claim 4, wherein: Supervised interaction detection module, including: The general interaction loss supervision includes the mean absolute error of the bounding box position L b , the bounding box size generalization intersection-union ratio L u , object classification cross entropy L c and interactive classification cross entropy L a , so the general interaction loss function L hoi The mathematical expression is as follows: L hoi =λ b L b +λ u L u +λ c L c +λ a L a (8) where λ b ,λ u ,L c and L a All are super parameters; Since the dynamic language information generation query module and the mask truth value denoising module will both pass through the interactive detection module to generate prediction results, they each need to use L hoi get and Supervise the prediction results of each party, and then according to the L MFL , and the total loss L is obtained total , mathematically expressed as follows: Where α is a hyperparameter.

Images