Cascade constraint decoding method for human interaction detection based on suggestive guided query

Abstract

The present invention discloses a cascade constraint decoding method for detecting human interaction based on suggestive guided queries. A human interaction detection network model is constructed and trained. The human interaction detection network model includes a target detector, a vector fusion module, and a cascade constraint interaction decoder. The image to be detected is then input into the target detector to obtain a target frame, target category, and confidence score. In the vector fusion module, the obtained visual query vector, spatial query vector, and semantic query vector are concatenated to obtain a final fused query vector. The fused query vector is then decoded using a main branch decoder and a suppression branch decoder through a cascade constraint interaction decoder. The outputs of the two branches are interactively perceived and fused, and then the interaction detection result is obtained through a multi-layer perceptron. The present invention can tap deeper interactive representations and improve the generalization performance of the model, thereby improving the accuracy of human-object interaction detection and achieving leading performance.

Description

Technical Field

[0001] The present application belongs to the technical field of human-object interaction detection, and relates to a human interaction detection method based on cascade constraint decoding of suggestive guided queries. Background Art

[0002] Human-object interaction (HOI) detection is a key component of computer vision and interaction understanding. It focuses on the interactions between people and objects, which is crucial for machines to understand human activities. Effectively locating human-object pairs and predicting the corresponding interaction categories in complex and changing environments has always been a challenge for HOI detection.

[0003] With the introduction of the DETR (End-to-end Object Detection with Transformers) framework, end-to-end human-object interaction detection methods based on Transformers have become the mainstream of existing technologies. The QPIC method used Transformers for the first time to solve the problem of human-object interaction detection, and the CDN method used a dual-branch decoding structure for the first time to achieve more effective interaction prediction. Although these methods utilize more contextual information in various situations and improve prediction performance by introducing transformers, these HOI detectors use randomly initialized query vectors for prediction, which leads to insufficient prior knowledge.

[0004] To more effectively distinguish and represent localization and interaction, some end-to-end models combine a dual-branch decoder with a predictive matching method. However, using a unified architecture for multi-task prediction (i.e., localization and interaction recognition) is unintuitive and has limitations in propagating the human-object context to the interaction context.

[0005] In addition, the traditional multi-branch decoder structure still has limitations in the fine-grained representation of interaction features. Summary of the Invention

[0006] The purpose of this application is to provide a cascade constraint decoding human interaction detection method based on suggestive guided query to overcome the above-mentioned problems existing in the prior art in human-object interaction detection.

[0007] In order to achieve the above objectives, the technical solutions of this application are as follows:

[0008] A human interaction detection method based on cascade constraint decoding and suggestive guided query, comprising:

[0009] Constructing and training a person interaction detection network model, the person interaction detection network model includes a target detector, a vector fusion module and a cascade constraint interaction decoder, the vector fusion module includes a candidate region establishment unit, a spatial vector generation unit and a prompt guidance vector generation unit;

[0010] Input the image to be detected into the target detector to obtain the target box, target category and confidence score;

[0011] In the candidate region establishment unit, valid person and object indexes are obtained based on the confidence scores, and a person pair matrix is ​​established. The person pair matrix is ​​multiplied by the feature points output by the object detector decoder and flattened. The visual query vector is obtained through a feedforward neural network.

[0012] In the spatial vector generation unit, valid human or object target frames are screened to form human target frame interaction pairs, and the corresponding spatial vectors are calculated and spliced, and then input into the feedforward neural network to obtain the spatial query vector;

[0013] Obtaining a prompt of the image to be detected and inputting the prompt into a prompt guidance vector generation unit to generate a semantic query vector;

[0014] The generated visual query vector, spatial query vector and semantic query vector are concatenated to obtain the final fused query vector;

[0015] In the cascade constrained interaction decoder, the main branch decoder and the suppression branch decoder are used to decode the fused query vector to obtain the output of the two branches. The outputs of the two branches are then input into the interaction perception fusion unit, and then the interaction detection results are obtained through the multi-layer perceptron.

[0016] Furthermore, obtaining valid person and object indexes based on the confidence scores and establishing a person pair matrix includes:

[0017] Filter out the indexes of target boxes whose target category is human and whose confidence is greater than the first threshold, thus obtaining valid human indexes;

[0018] Filter out the indexes of target boxes whose target category is object and whose confidence is greater than the first threshold, thus obtaining valid object indexes;

[0019] A person pair matrix is ​​established with the number of target boxes as the dimensions of rows and columns. The elements corresponding to the valid person and object indexes in the matrix are set to 1, and the other elements are set to 0 to obtain the person pair matrix.

[0020] Furthermore, the effective human or object target frames are screened out to form a person target frame interaction pair, and the corresponding spatial vector is calculated, where the spatial vector includes: a center point vector, a distance vector, a covering value vector, an angle vector and an area vector.

[0021] Furthermore, the step of obtaining a prompt of the image to be detected and inputting the prompt into a prompt guidance vector generating unit to generate a semantic query vector includes:

[0022] The prompts of the image to be detected are passed through the pre-trained CLIP text decoder to obtain the semantic features representing people and objects;

[0023] Then, the representations of people and objects are concatenated and input into the feedforward neural network to obtain the semantic query vector.

[0024] Furthermore, in the cascade constrained interaction decoder, the main branch decoder and the suppression branch decoder are used to decode the fused query vector to obtain the output of two branches, and then the output of the two branches are input to the interaction perception fusion unit, and then the interaction detection result is obtained through the multi-layer perceptron, including:

[0025] The feature map extracted by the target detector backbone network, as well as the position vector and the fused query vector are input into the main branch decoder to obtain the main branch decoding features;

[0026] Generate a random mask, input the feature map extracted by the target detector backbone network, as well as the position vector and the fused query vector into the suppression branch decoder to obtain the suppression branch decoding feature;

[0027] Input the main branch decoding features and the suppression branch decoding features into the interaction perception fusion unit to obtain the interaction features;

[0028] The interaction features are input into the multi-layer perceptron to obtain the predicted interaction category.

[0029] Furthermore, the operation performed by the interactive perception fusion unit is expressed as follows:

[0030] A e =λ1*h master +λ2*h sup

[0031] Among them, λ1 is the weight of the main branch, λ2 is the weight of the inhibition branch, and A e is the interaction feature, h master is the main branch decoding feature, h sup Decode features for the suppression branch.

[0032] Furthermore, both the main branch decoder and the suppression branch decoder adopt Transformer decoder, and in the suppression branch decoder, a random mask is used to randomly filter the input fusion query vector.

[0033] This application proposes a cascaded constraint decoding method for human interaction detection based on suggestive guided queries, which is used for suggestive guided queries for human-object interaction detection. It effectively establishes interaction semantic information and improves prior knowledge in interaction scenarios. At the same time, a cascaded constraint decoder architecture is proposed to effectively predict interaction categories, and a random mask matrix is ​​introduced to construct a two-branch constraint decoding structure. In addition, an interaction-aware merging module is established to better balance and integrate different branch features. The human interaction detection network model proposed in this application frees the network from learning unnecessary embedding transformations to explore deeper interactive representations and improve the generalization performance of the model. This application improves the accuracy of human-object interaction detection and achieves leading performance over recent methods on two popular datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flow chart of the method for detecting human interaction using cascaded constraint decoding based on suggestive guided queries in this application;

[0035] Figure 2 This is a schematic diagram of the human interaction detection network model structure of this application;

[0036] Figure 3 This is a schematic diagram of the character target frame interaction;

[0037] Figure 4 This is a schematic diagram of a cascaded constrained interactive decoder according to an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0039] In one embodiment, Figure 1 As shown in the figure, a cascade constraint decoding method for human interaction detection based on suggestive guided query is proposed, which includes:

[0040] Step S1: construct and train a human interaction detection network model, wherein the human interaction detection network model includes a target detector, a vector fusion module and a cascade constraint interaction decoder, and the vector fusion module includes a candidate region establishment unit, a spatial vector generation unit and a prompt guidance vector generation unit.

[0041] like Figure 2 As shown, the human interaction detection network model constructed in this application includes a target detector, a vector fusion module and a cascade constraint interaction decoder. The vector fusion module includes a candidate region establishment unit, a spatial vector generation unit and a prompt guidance vector generation unit.

[0042] Step S2: Input the image to be detected into the target detector to obtain the target box, target category and confidence score.

[0043] The target detector in this embodiment uses the popular ResNet50 in the field of vision as the backbone network, and the required input image is The 3 represents the number of RGB channels, H and W are the height and width of the image respectively, the image is input into the backbone network to obtain the feature map H1 and W1 are the height and width of the feature map, C f is the dimension after the backbone network conversion. Then after 1×1 convolution, the dimension of the feature map is C f Transferred to C d Finally, the feature maps are flattened and concatenated into a length N m , dimension C d The eigenvector of N m Represented as the length of the concatenated vector.

[0044] Next, generate the position code, Combine it with S q Send them together to the Transformer encoder to obtain the encoded features:

[0045] E f =f enc (S q ,P e );

[0046] where f enc (-,-) represents the encoder function.

[0047] Next, E f , P e , Q (Q represents a randomly initialized vector in the detector) is input into the Transformer decoder, and the calculation process can be expressed as:

[0048] D f =f dec (E f ,P e ,Q);

[0049] where f dec (-,-,-) represents the decoder function, D f That is the decoded output feature of the target detector.

[0050] The final person and object bounding box (collectively referred to as the target box) B h,o , confidence score S and target category Calculated by the multi-layer perceptron MLP:

[0051] B h,o ,S=Sigmoid(MLP(D f ));

[0052] C o =FFN(D f );

[0053] Among them, N obj Represents the total category of the target, and FFN represents a single-layer fully connected layer network. h,o is the target frame, including the target frame of people and the target frame of objects; S represents the confidence score; C o Indicates the target category.

[0054] It should be noted that there are currently many relatively mature detection technologies for target detection. This application is not limited to the specific form of target detectors, which will not be described in detail below.

[0055] Step S3: In the candidate region establishment unit, valid person and object indexes are obtained based on the confidence scores, a person pair matrix is ​​established, the person pair matrix is ​​multiplied by the feature points decoded and output by the target detector, and then flattened, and a visual query vector is obtained through a feedforward neural network.

[0056] Specifically, valid person and object indexes are obtained according to the confidence score. In this application, the confidence score is set to 0.5, that is, when the confidence score exceeds 0.5, it is considered valid.

[0057] Among them, the people index Expressed as:

[0058]

[0059] Object Index Expressed as:

[0060]

[0061] In this embodiment, all detected target frames are treated as an array, each target frame corresponds to an element in the array, and the number of each target frame in the array is its corresponding index. score>0.5 (S) i Indicates filtering indexes with confidence greater than 0.5, c i Indicates the category, in this example, c i =1 means the category is human, c i ≠1 means the category is an object. That is, the index of the target box whose target category is human and whose confidence is greater than 0.5 is filtered out, and the valid human index is obtained. Filter out the indexes of target boxes whose target category is object and whose confidence is greater than 0.5, and get the valid object index

[0062] Then, a person pair matrix is ​​established with the number of target boxes as the dimensions of rows and columns. The elements corresponding to the valid person and object indexes in the matrix are set to 1, and the other elements are set to 0 to obtain the person pair matrix.

[0063] Specifically, establish a character pair matrix The rows of the matrix represent the human index, the columns represent the object index, and N q Indicates the number of detected target boxes. When there is a valid index pair, the corresponding value in the person pair matrix is ​​set to 1, otherwise it is 0.

[0064]

[0065] For example, the people index Contains index 3, object index contains index 5, then the character pair matrix The element in the third row and fifth column is 1, and so on.

[0066] Then, based on the obtained character pair matrix, this application will Do a dot multiplication with the decoded output features of the target detector and input it into the feedforward neural network FFN to obtain the visual query vector Q a .

[0067]

[0068] in, Decodes the output features of the target detector, representing the output result of the last layer of the decoder in the target detector, f flatten Represents the vector flattening function, which flattens the obtained matrix result into a vector.

[0069] Step S4: In the spatial vector generation unit, valid human or object target frames are screened to form a person target frame interaction pair, and the corresponding spatial vectors are calculated and spliced, and then input into the feedforward neural network to obtain a spatial query vector.

[0070] Specifically, the human target frames and object target frames with a confidence score greater than 0.5 are screened as valid human or object target frames, and each human target frame is paired with each object target frame one by one to obtain human target frame interaction pairs.

[0071] Subsequently, the spatial variables corresponding to each person-target frame interaction pair can be calculated through the coordinates of each vertex of the target frame, including: the middle point of the intersection area, distance, cover value, angle and area.

[0072] Among them, for the i-th person-target frame interaction pair, its intersection area is expressed as

[0073]

[0074] Among them, (x i1 ,y i1 ) represents the upper left coordinate of the intersection area, (x i2 ,y i2 ) represents the upper right coordinate of the intersection area, such as Figure 3 As shown, the intersection area is represented by the upper left coordinate and the lower right coordinate of the intersection area.

[0075] Next, calculate the middle point of the intersection area based on the upper left and lower right coordinates of the intersection area

[0076]

[0077] Among them, the distance between the person target frame and the object target frame in the i-th person target frame interaction is i , the calculation formula is as follows:

[0078] distance i =[(x i1 -x i2 ) 2 +(y i1 -y i2 ) 2 ] 1 / 2 ;

[0079] Among them, the overlap value corresponding to the interaction of the i-th person target frame is i , represents the area of ​​interest containing the target box of people and objects, and the calculation formula is as follows:

[0080] overlap i =|x lt -x rb |*|y lt -y rb |;

[0081] Among them, (x lt ,y lt ) represents the upper left corner of the target box of people and objects, (x rb ,y rb ) represents the lower right corner of the target box of people and objects, |x lt -x rb | represents the absolute horizontal distance of the area of ​​interest, |y lt -y rb |Indicates the absolute vertical distance of the area of ​​interest.

[0082] Among them, the angle theta corresponding to the interaction of the i-th person target frame i, represents the offset angle of the target frame coordinate point of people and objects, and the calculation formula is as follows:

[0083]

[0084] Where atan2 is the four-quadrant inverse tangent function for finding angles.

[0085] Among them, the total area corresponding to the interaction of the i-th person target frame is area i , the calculation formula is as follows:

[0086]

[0087] in, represents the area of ​​the human target box, Indicates the area of ​​the object frame, Represents the area of ​​the intersection.

[0088] This embodiment calculates different space vectors based on the interactive human and object frames, center p The center point vector of the target box. Determining the center point allows the network to learn more information about the interacting entities. distance represents a distance vector, which indicates the distance between interacting entities. Distance can reflect interaction propensity, allowing the network to learn better interaction features. The theta, overlap, and area vectors represent the angle, area of ​​the focus region, and area of ​​the intersection region, respectively. Introducing spatial feature information can better help the network detect interaction results in complex spatial scenarios.

[0089] Finally, N p The target boxes interact with each other to combine the same spatial variables into corresponding vectors, that is, the center point vector center is obtained p , distance vector distance, cover value vector overlap, angle vector theta and area vector area, concatenate the above spatial vectors and input them into FFN to obtain the spatial query vector Q s :

[0090] Q s =FFN(f cat (center p ,distance,overlap,theta,area)).

[0091] Step S5: Obtain the prompt of the image to be detected, and input it into the prompt guidance vector generation unit to generate a semantic query vector.

[0092] In this embodiment, the object category C of the image to be detected is o , generate a prompt P containing semantic information l, for example (a photo of human / object), and then feed the prompt into the pre-trained CLIP text decoder Clip e To obtain the semantic feature f l :

[0093] f l =Clip e (P l );

[0094] This embodiment constructs different prompts according to people and objects, and uses the pre-trained CLIP text decoder Clip e Get the semantic features f representing people and objects l (h),f l (o) Then, the semantic features representing people and objects are concatenated and input into the feedforward neural network FFN to obtain the semantic query vector:

[0095] Q l =FFN(f cat (f l (h),f l (o))).

[0096] Step S6: Concatenate the generated visual query vector, spatial query vector, and semantic query vector to obtain a final fused query vector.

[0097] The generated visual query vector Q a , spatial query vector Q s With the semantic query vector Q l Splicing to get the final fusion query vector Q all Finally, the present application fuses the query vectors obtained in S201, S202, and S203 to obtain a fused query vector:

[0098] Q all =f cat (Q a ,Q s ,Q l ).

[0099] Step S7: In the cascade constrained interaction decoder, the main branch decoder and the suppression branch decoder are used to decode the fused query vector to obtain the output of the two branches, and then the output of the two branches are input to the interaction perception fusion unit, and then the interaction detection result is obtained through the multi-layer perceptron.

[0100] In this embodiment, the cascade constraint interaction decoder is as follows Figure 4 As shown in the figure, it includes two branches, namely the main branch decoder and the suppression branch decoder, both of which use Transformer decoders. The inputs of the two branch decoders pass through the interactive perception fusion unit to obtain the detection results.

[0101] Specifically, the query vector Q is fused all Input the main branch decoder to obtain the output feature h of the main branch master , the fused query vector Q all Input the suppression branch decoder to obtain the output feature h of the suppression branch sup , h master and h sup Input into the interaction perception fusion module to obtain the interaction detection results.

[0102] The main branch decoder and the suppression branch decoder are used to decode the fused query vector to obtain the outputs of the two branches, and then the outputs of the two branches are input to the interaction perception fusion unit to obtain the interaction detection result, including:

[0103] Step S701: The feature map F extracted by the target detector backbone network f , and the position vector P e and fused query vector Q all Input the main branch decoder to obtain the main branch decoding feature h master .

[0104] The main branch decoder includes a self-attention module, a residual network, a regularization module, and an FFN network layer, all of which are relatively mature technologies in the Transformer decoder and will not be described in detail here.

[0105] The entire main branch decoder can be expressed as:

[0106] h master =f master (F f ,P e ,Q all )

[0107] Among them, f master It represents the main branch decoder operation. After the main branch decoder, the decoding feature h can be obtained. master .

[0108] Step S702: Generate a random mask R m , the feature map F extracted by the target detector backbone network f , and the position vector P e and fused query vector Q all Input the suppression branch decoder to obtain the suppression branch decoding feature h sup .

[0109] This example constructs a random mask to cover the learnable fusion vector to enhance the generalization performance of the model. A floating point number η in the range of [-1, 1] is randomly generated using the uniform distribution function μ:

[0110] η=μ(-1,1)

[0111] Construct a random matrix based on the generated floating-point numbers It is expressed as follows:

[0112]

[0113] The random mask matrix R m , feature map F f , position vector P e and fused query vector Q all Input the suppression branch and obtain the decoding features of the suppression branch:

[0114]

[0115] Among them, f sup Denotes the decoding operation of suppressing branches, Indicates the use of random mask matrix R m To randomly filter the input fusion query vector Q all .

[0116] In this embodiment, the feature map F f , position vector P e and fused query vector Q all Input into the Transformer decoder together can better learn features.

[0117] Step S703: decode the main branch feature h master and the suppression branch decoding feature h sup Input the interactive perception fusion unit to obtain the interactive feature A e .

[0118] The interactive perception fusion unit is specifically expressed as:

[0119] A e =λ1*h master +λ2*h sup

[0120] Where λ1 is the weight of the main branch, and λ2 is the weight of the suppression branch. In this example, λ1 = 0.6 and λ2 = 0.4.

[0121] Step S704: Interaction feature A e Input the multilayer perceptron to get the predicted interaction category C a .

[0122] Specifically expressed as:

[0123] C a =MLP(A e).

[0124] Multilayer Perceptron (MLP) is a relatively mature technology in this field and will not be described in detail here. h,o , object category C o , interaction category C a By drawing it on the original image, we can get the visual human-object interaction detection results.

[0125] This application also verifies the above method through experiments. The experimental data show that the method of this application has better interaction detection accuracy than other methods in the prior art. The experimental data are shown in Tables 1 and 2 below:

[0126] Table 1

[0127]

[0128] Table 1 shows the performance comparison of the method of the present application with other methods on the HICO-DET dataset. "Default" and "Known" represent two scenarios on the HICO-DET dataset. The former represents the default setting, and the latter represents the setting where no samples appear in the result evaluation in the dataset. The accuracy of the prediction results is expressed in mAP, which represents the average accuracy of the human-object interaction prediction results. "Full" represents all samples, "Rare" represents the situation where there are few samples in the evaluation process, and "Non-Rare" represents the situation where there are no few samples in the evaluation process. The results show that the method of the present application can more effectively utilize prior knowledge to achieve more stable interaction detection.

[0129] Table 2

[0130]

[0131] As shown in Table 2, on the V-COCO dataset, the method of this application has two indicators of the dataset: It achieved an mAP (human-object interaction prediction accuracy) of 67.2 and 72.5, which is at least 1.5% better than the previous state-of-the-art human-object interaction detection method.

[0132] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A cascade constraint decoding method for detecting human interaction based on suggestive guided queries, characterized in that: The method for detecting human interaction by cascade constraint decoding based on suggestive guided query includes: Constructing and training a person interaction detection network model, the person interaction detection network model includes a target detector, a vector fusion module and a cascade constraint interaction decoder, the vector fusion module includes a candidate region establishment unit, a spatial vector generation unit and a prompt guidance vector generation unit; Input the image to be detected into the target detector to obtain the target box, target category and confidence score; In the candidate region establishment unit, valid person and object indexes are obtained based on the confidence scores, and a person pair matrix is ​​established. The person pair matrix is ​​multiplied by the feature points output by the object detector decoder and flattened. The visual query vector is obtained through a feedforward neural network. In the spatial vector generation unit, valid human or object target frames are screened to form human target frame interaction pairs, and the corresponding spatial vectors are calculated and spliced, and then input into the feedforward neural network to obtain the spatial query vector; Obtaining a prompt of the image to be detected and inputting the prompt into a prompt guidance vector generation unit to generate a semantic query vector; The generated visual query vector, spatial query vector and semantic query vector are concatenated to obtain the final fused query vector; In the cascade constrained interaction decoder, the feature map extracted by the target detector backbone network, as well as the position vector and the fused query vector are input into the main branch decoder to obtain the main branch decoding feature; a random mask is generated, and the feature map extracted by the target detector backbone network, as well as the position vector and the fused query vector randomly screened by the random mask are input into the inhibition branch decoder to obtain the inhibition branch decoding feature; the main branch decoding feature and the inhibition branch decoding feature are input into the interaction perception fusion unit to obtain the interaction feature; the interaction feature is input into the multi-layer perceptron to obtain the predicted interaction category.

2. The method for detecting human interaction by cascade constraint decoding based on suggestive guided query according to claim 1, characterized in that: The process of obtaining valid person and object indexes based on the confidence scores and establishing a person pair matrix includes: Filter out the indexes of target boxes whose target category is human and whose confidence is greater than the first threshold, thus obtaining valid human indexes; Filter out the indexes of target boxes whose target category is object and whose confidence is greater than the first threshold, thus obtaining valid object indexes; A person pair matrix is ​​established with the number of target boxes as the dimensions of rows and columns. The elements corresponding to the valid person and object indexes in the matrix are set to 1, and the other elements are set to 0 to obtain the person pair matrix.

3. The method for detecting human interaction by cascading constraint decoding based on suggestive guided query according to claim 1, characterized in that: The effective human or object target frames are screened out to form a human target frame interaction pair, and the corresponding spatial vector is calculated, where the spatial vector includes: a center point vector, a distance vector, a covering value vector, an angle vector and an area vector.

4. The method for detecting human interaction by cascading constraint decoding based on suggestive guided query according to claim 1, characterized in that: The step of obtaining a prompt of the image to be detected and inputting the prompt into a prompt guidance vector generating unit to generate a semantic query vector includes: The prompts of the image to be detected are passed through the pre-trained CLIP text decoder to obtain the semantic features representing people and objects; Then, the representations of people and objects are concatenated and input into the feedforward neural network to obtain the semantic query vector.

5. The method for detecting human interaction by cascading constraint decoding based on suggestive guided query according to claim 1, characterized in that: The operations performed by the interactive perception fusion unit are expressed as follows: ； in The weight of the main branch, To suppress the weight of the branch, is the interactive feature, Decoding features for the main branch, Decode features for the suppression branch.

6. The method for detecting human interaction by cascading constraint decoding based on suggestive guided query according to claim 1, characterized in that: The main branch decoder and the suppression branch decoder both use Transformer decoders. In the suppression branch decoder, a random mask is used to randomly filter the input fusion query vector.

Images