Parallel character interaction relationship detection method based on feature fusion and noise suppression

Abstract

This invention relates to the field of character interaction relationship detection, and discloses a parallel character interaction relationship detection method based on feature fusion and noise suppression. The invention first constructs a three-branch network model, dividing the original HOI task into three branches for parallel processing. During the detection process, three completely separate parallel detectors are used for separate detection. To establish communication between independent branches and facilitate the exchange of contextual information, this invention proposes an auxiliary feature fusion module, utilizing the relationships between different information to more accurately understand and analyze the image. To reduce the interference of noise that may be generated during the fusion process on the original features, this invention proposes a fusion noise suppression module, which can improve the accuracy of interaction detection. Detection results on public datasets show that the implemented algorithm achieves an interaction detection accuracy superior to current mainstream algorithms, demonstrating excellent detection performance.

Description

Technical Field

[0001] This invention relates to the field of character interaction relationship detection, and specifically to a one-stage character interaction relationship detection method based on Transformer. Background Technology

[0002] Human Object Interaction Detection (HOI) is a technique that aims to detect the location of a human body in a given photograph, locate and classify the objects interacting with it, and determine the nature of the interaction. The prediction results are often defined as a triple: <human body location, (object location, object classification), interaction classification>. This technology is widely used in various scenarios such as video surveillance, smart cockpits, human-computer interaction, and intelligent robots; its research results can also support higher-level visual content understanding tasks, such as human motion recognition, image captioning, and scene understanding.

[0003] Supervised learning for human interaction detection has achieved relatively ideal results. Existing one-stage methods often divide the original task into multiple sub-tasks, and then build a multi-branch deep network based on Transformer, using different branches to complete each sub-task. Specifically, training data is input into the model, prediction results are calculated, and parameters are updated according to the loss function and optimization algorithm. This process is repeated multiple times until the model converges or reaches a predetermined number of training epochs. However, such methods are often accompanied by two problems: 1) Information exchange between multiple branches is not given sufficient attention, the cascaded structure reasoning method is simplistic, and the fully parallel method severs the connections between different pieces of information. 2) Noise interference may also occur during the fusion of different features, affecting the detection effect. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a parallel method for detecting character interaction relationships based on feature fusion and noise suppression. This invention includes an auxiliary feature fusion module and a fusion noise suppression module to solve the problems of insufficient information exchange between multiple branches and noise generated during feature fusion.

[0005] To achieve the above objectives, the technical solution adopted in this application is as follows:

[0006] A parallel character interaction relationship detection method based on feature fusion and noise suppression, comprising:

[0007] Obtain labeled image samples from the training dataset, and use the Transformer encoder to process the image samples to obtain global encoded features;

[0008] The global encoded features are input into the HOI detection module to obtain human features, object features and interaction features. The HOI detection module consists of three branches: human branch, object branch and interaction branch. Each branch contains a Transformer decoder.

[0009] Take the output of each branch of the HOI detection module and input it into the corresponding auxiliary feature fusion module. Each auxiliary feature fusion module takes the output of the corresponding branch as the main feature and the outputs of the other two branches as auxiliary features. After fusing the main feature and the auxiliary features, the corresponding fused feature is obtained.

[0010] The fusion features corresponding to the interaction branches are input into the fusion noise suppression module, which obtains the early prediction classification result and the early interaction loss based on the early prediction classification result.

[0011] The fusion features corresponding to the three branches are input into the corresponding task classifiers respectively. The triplet prediction results are obtained based on the task classifiers, and the basic loss is calculated based on the triplet prediction results and the labels of the image samples.

[0012] The total loss is obtained by combining the pre-interaction loss and the basic loss. The parameters of the detection network model are updated according to the total loss. The trained detection network model is used to obtain the interaction relationship between people in the input image. The detection network model includes a Transformer encoder, a Transformer decoder, an auxiliary feature fusion module, a fusion noise suppression module, and a task classifier.

[0013] Several alternative methods are provided below, but they are not intended as additional limitations on the overall solution above. They are merely further additions or optimizations. Provided there are no technical or logical contradictions, each alternative method can be combined individually with respect to the overall solution above, or multiple alternative methods can be combined with each other.

[0014] Preferably, the step of processing image samples using a Transformer encoder to obtain global encoded features includes:

[0015] The ResNet-50 network model is used to preprocess the image samples to obtain the overall feature map;

[0016] The overall feature map is divided into several image blocks of the same size, and the image blocks are linearly mapped into vectors;

[0017] The Transformer encoder is used to process vectors to obtain global encoded features.

[0018] Preferably, the Transformer encoder includes multiple coding layers, where the encoding result output by the previous coding layer is used as the input to the next coding layer.

[0019] Preferably, the Transformer decoder in the human body branch uses the human body query vector. h The Transformer decoder in the object branch uses the object query vector Query. o The Transformer decoder in the interactive branch uses a relational query vector Query. i .

[0020] Preferably, each auxiliary feature fusion module takes the output of the corresponding branch as the main feature and the outputs of the other two branches as auxiliary features. After fusing the main feature and the auxiliary features, the corresponding fused feature is obtained, including:

[0021] 1) Use an MLP network to process the two auxiliary features into a hybrid feature:

[0022]

[0023] In the formula, m represents the mixed features, self_att represents the self-attention mechanism, and MLP is a multilayer perceptron network. and For auxiliary features, a1∈{h, o, i}, a2∈{h, o, i}, a1≠a2;

[0024] 2) The main feature F t t∈{h,o,i},t≠a1≠a2 serves as the query in the attention mechanism, and is successively fused with hybrid features and global encoding features through multiple cross-attention mechanisms:

[0025] f t =cross_att(cross_att(F t ,m),M)

[0026] In the formula, f t The intermediate features are fused, cross_att(q, kv) is the cross attention mechanism, q is the query, k and v are the key and value respectively, and M is the global encoded feature;

[0027] 3) Pass the main features and intermediate features through a channel attention fusion layer to obtain the fused features corresponding to each branch:

[0028]

[0029] In the formula, For feature fusion, channel_att represents the channel attention mechanism, channel_att = σ(MLP(F t f t)), ⊙ represents the Hadamard product operation, and σ is the sigmoid activation function.

[0030] Preferably, the step of inputting the fusion features corresponding to the interaction branches into the fusion noise suppression module, obtaining the early predicted classification result from the fusion noise suppression module, and obtaining the early interaction loss based on the early predicted classification result includes:

[0031] 1) The fusion features corresponding to the interaction branches are fed into the MLP network of the fusion noise suppression module to obtain the suppression features F. p ;

[0032] 2) Suppression feature F p After averaging, the data is fed into a feedforward neural network to obtain the predicted classification result P, where P = [p1, p2, ..., p...]. k , ..., p K ], where K represents the total number of interaction categories in the training dataset, p k This represents the probability of predicting the k-th interaction in the classification result P in advance;

[0033] 3) The aforementioned early interaction loss L e Obtained from the asymmetric loss function:

[0034]

[0035] Where, p′ k It is the offset probability, p' k =max(p k -u), where u is the threshold, y k For interactive labels of image samples, y k =1 indicates that the k-th interaction exists in the image sample, y k =0 indicates that the k-th interaction does not exist in the image sample, and γ+ and γ- are both hyperparameters.

[0036] Preferably, the step of inputting the fused features corresponding to the three branches into the corresponding task classifiers, obtaining triplet prediction results based on the task classifiers, and calculating the basic loss based on the triplet prediction results and the labels of the image samples includes:

[0037]

[0038]

[0039]

[0040]

[0041] In the formula, Box h The bounding box represents the predicted human body shape, and FFN represents the feedforward neural network. Box represents the fusion feature corresponding to the human body branch. o Represents the predicted object bounding box. The fusion feature corresponding to the object branch, Score o This represents the score for each object query across all object categories. For the fusion features corresponding to the interaction branches, FFN proj For linear adjustment layers, Score interact For each interaction query, the score across all interaction categories is used to determine the final triplet prediction result: <Box h (Box) o Score o Score interact >;

[0042] The basic loss is calculated as follows:

[0043] L base =λ b *L bbox +λ u *L u +λ o *L o +λ i *L i

[0044] In the formula, L base For the basic loss, L bbox The bounding box regression loss, L, is calculated using the absolute value loss function. u The bounding box regression loss is calculated using the Giou loss function. The predicted values ​​for both bounding box regression losses are... Box = Concat(Box) h Box o ), N q For the number of queries, the four dimensions represent the x-coordinate, y-coordinate, width, and length of the bounding box, respectively. o It is determined by the cross-entropy loss function based on the score. o The calculated object classification loss, L i It is determined by the focus loss function based on the score. interact The calculated interaction classification loss, λ b , λ u , λ o and λ i All of these are hyperparameters.

[0045] This invention provides a parallel method for detecting human interaction relationships based on feature fusion and noise suppression. First, a three-branch network model is constructed, dividing the original HOI (Homologous Identity Inquiry) task into three branches for parallel processing. During detection, three completely separate parallel detectors are used. To establish communication between independent branches and facilitate the exchange of contextual information, this invention proposes an auxiliary feature fusion module, utilizing the relationships between different pieces of information to more accurately understand and analyze the image. To reduce the interference of noise that may be generated during the fusion process on the original features, this invention proposes a fusion noise suppression module, which can improve the accuracy of interaction detection. Detection results on public datasets show that the implemented algorithm achieves an interaction detection accuracy superior to current mainstream algorithms, demonstrating excellent detection performance. Attached Figure Description

[0046] Figure 1 The flowchart shows the parallel character interaction relationship detection method based on feature fusion and noise suppression proposed in this application.

[0047] Figure 2 This is a schematic diagram of the detection network model of this application. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application.

[0050] Please see Figure 1 As shown, this invention is a parallel method for detecting human interaction relationships based on feature fusion and noise suppression, comprising the following steps:

[0051] S1. Obtain labeled image samples from the training dataset, preprocess the source image samples, and use the Transformer encoder to process the source image samples into global encoded features.

[0052] First, a ResNet-50 network model is used as a feature extractor to preprocess the image. The source image samples are scaled, cropped, and normalized to obtain an overall feature map E with a size of 2048×27×28. The overall feature map E is divided into several image patches of the same size, and the image patches are linearly mapped to vectors. A Transformer encoder is then used to further obtain global encoded features M with a size of 756×4×256. The processing of the overall feature map E is expressed by the following formula:

[0053] E′=MutiHeadAttention(LayerNorm(E))+E

[0054] Where E′ represents the attention feature, MultiHeadAttention represents the multi-head attention mechanism, and LayerNorm represents the layer normalization operation.

[0055] M = FFN(LayerNorm(E′)) + E′

[0056] FFN stands for Feedforward Neural Network.

[0057] To improve the encoding effect, this embodiment uses multiple encoding layers continuously when using the Transformer encoder, with the encoding result of the previous layer serving as the input of the next layer, and this process is repeated three times.

[0058] S2. Construct a HOI detection model with three branches of parallel inference based on Transformer deep networks, such as... Figure 2 As shown, three independent queries are used - Human Body Query. n Object Query o Relationship Query i Human features, object features, and interaction features are obtained from the global encoded features obtained in step S1.

[0059] Three independent queries are learnable vector embeddings of size 64×256, all randomly initialized. In traditional Transformer models, queries are used to compute attention weights. However, in the vision domain, queries represent object category and location information in object detection. Each query represents a potential object, and the model determines the object's classification and location by performing self-attention computation on these queries and the feature maps output by the encoder. Therefore, queries play a crucial role in object detection tasks, acting as equivalent to candidate boxes in traditional object detection. Applied to the HOI detection task, the model performs end-to-end multi-task detection using these three learnable queries, making the HOI task simpler and more flexible.

[0060] Attention mechanisms are a widely used method in modern deep learning models, which are good at capturing the internal correlations of data or features; the Transformer model is based on the self-attention mechanism to build deep networks; in specific tasks, the Transformer outperforms neural network models, and its greatest advantage comes from the fact that the Transformer is suitable for parallelization.

[0061] S3. After the three branches in step S2, an auxiliary feature fusion module is set up. This module serves as the intersection of the three branches. After obtaining the corresponding features for each branch, the auxiliary feature fusion module is used to fuse the features of other branches into the main features of the original branch as auxiliary features. This step can be repeated multiple times along with the layer iterations of the Transformer decoder, with the output of the previous layer initializing the query of the next layer. That is, if multiple decoding layers are used consecutively when using the Transformer decoder, an auxiliary feature fusion module is connected after each decoding layer, and the output of the auxiliary feature fusion module of the previous layer is used as the input of the next decoding layer.

[0062] The specific processing procedure of the auxiliary feature fusion module is as follows:

[0063] 1) First, use an MLP network to process the two auxiliary features into a hybrid feature m:

[0064]

[0065] In the formula, m represents the mixed features, self_att represents the self-attention mechanism, and MLP is a multilayer perceptron network. and For auxiliary features, a1∈{h,o,i}, a2∈{h,o,i}, a1≠a2; the self-attention mechanism can filter out irrelevant information, calculate attention weights in parallel, and improve computational efficiency; it can also filter out irrelevant information in the sequence through the masking mechanism, which improves the robustness and generalization ability of the model.

[0066] 2) The main feature F t t∈{h,o,i},t≠a1≠a2 serves as the query in the attention mechanism, and is successively fused with hybrid features and global encoded features through multiple cross-attention mechanisms:

[0067] f t =cross_att(cross_att(F t ,m),M)

[0068] In the formula, f tThe intermediate features are fused together. `cross_att(q, kv)` represents the cross-attention mechanism, where `q` is the query, `k` and `v` are the key and value, respectively, and `M` represents the global encoded features. The cross-attention mechanism enhances the model's expressive power by fusing different feature information, capturing more global semantic and contextual information. The primary and secondary features originate from different sources in different branches, which enriches the relational reasoning results.

[0069] 3) The main features and intermediate features are passed through a channel attention fusion layer to obtain the fused features corresponding to each branch, with a size of 64×256:

[0070]

[0071] In the formula, These are fusion features, corresponding to three branches, which are respectively called human fusion features. Object fusion features and interactive integration features channel_att represents the channel attention mechanism, channel_att = σ(MLP(F) t f t )), ⊙ represents the Hadamard product operation, and σ is the sigmoid activation function. Channel attention is an attention mechanism used in neural networks to weight information from different channels (or feature maps). In channel attention, a set of weights is learned to adjust the response values ​​of different channels, highlighting important channel features and suppressing irrelevant channel features.

[0072] S4. Input the interaction fusion features produced by the interaction branch into the fusion noise suppression module, and predict the classification results in advance in the module. During training, extract the interaction action type information of the sample labels, use the loss function to calculate the loss of the predicted and acquired interaction information, and summarize the loss into the total loss to suppress the noise that may be generated in the original features.

[0073] 1) Integrating interactive features The suppression feature F is obtained by feeding it into the MLP network of the fusion noise suppression module. p Its dimensions are 64×600.

[0074]

[0075] 2) Using the inhibition feature F obtained in 1) p Obtain the predicted classification results in advance:

[0076] P = FFN(Mean(F) p ))

[0077] Where FFN is a feedforward neural network, Mean is the mean operation, P is the pre-prediction of the classification result, and P = [p1, p2, ..., p...]. k , ..., p K ], where K represents the total number of interaction categories in the training dataset, p k This represents the probability of predicting the k-th interaction in the classification result P in advance.

[0078] 3) Early interaction loss L e Obtained from the asymmetric loss function:

[0079]

[0080] Where, p′ k It is the offset probability, p′ k =max(p k -u), where u is the threshold value, which can be set to 0.05, and y k The interaction label for the image sample indicates whether a certain interaction exists in the image sample. k =1 indicates that the k-th interaction exists in the image sample, y k =0 indicates that the k-th interaction does not exist in the image sample. γ+ and γ- are hyperparameters, and their values ​​are set to 0 and 4, respectively.

[0081] Asymmetric loss is a loss function used for classification problems, specifically addressing classification issues in imbalanced datasets. In asymmetric loss, different weights are assigned to samples of different classes. Generally, samples from the minority classes are given heavier weights to focus more attention on their classification. This effectively addresses the problems caused by class imbalance.

[0082] S5. Input the outputs of the three branches into their respective task classifiers, and make triplet predictions based on the three classification results.

[0083]

[0084]

[0085]

[0086]

[0087] In the formula, Box h This represents the predicted human bounding box localization, with dimensions of 64×4, where 64 represents the number of queries, 4 represents the vector dimensions (x-axis, y-axis, width, height), and FFN represents a feedforward neural network. Box represents the fusion feature corresponding to the human body branch.o This represents the predicted object bounding box location, with dimensions of 64×4, where 64 represents the number of queries and 4 represents the vector dimensions (x-axis, y-axis, width, height). The fusion feature corresponding to the object branch, Score o This represents the score for each object query across all object categories. For the fusion features corresponding to the interaction branches, FFN proj This is a linear adjustment layer, designed to map object classification scores to interaction classification scores. (Score) interact For each interaction query, a score is assigned across all interaction categories. The size of both scores is determined by the number of object categories and interaction categories present in the training dataset. The final triplet prediction result is as follows: <Box h (Box) o Score o Score interact >

[0088] S6. Calculate the basic loss and total loss of the triplet prediction and the real sample, and update the detection network model parameters using the inverse algorithm.

[0089] 1) Basic loss:

[0090] L base =λ b *L bbox +λ u *L u +λ o *L o +λ i *L i

[0091] In the formula, L base For the basic loss, L bbox The bounding box regression loss, L, is calculated using the absolute value loss function. u The bounding box regression loss is calculated using the Giou loss function. The predicted values ​​for both bounding box regression losses are... Box = Concat(Box) h Box o ), N q For the number of queries, the four dimensions represent the x-coordinate, y-coordinate, width, and length of the bounding box, respectively. o It is determined by the cross-entropy loss function based on the score. o The calculated object classification loss, L i It is determined by the focus loss function based on the score. interact The calculated interaction classification loss, λ b , λ u, λ o and λ i These are all hyperparameters, and their values ​​are set to 2.5, 1, 1, and 1 respectively.

[0092] The L1 loss function is a commonly used regression loss function. It measures the prediction error by calculating the absolute difference between the predicted and the true values. A key characteristic of the L1 loss function is its high robustness and insensitivity to outliers. This is because the L1 loss function considers only the absolute value of the error, rather than the square of the error, when calculating the error.

[0093] GIoU loss is a loss function used in object detection tasks. It measures the overlap between the predicted and ground truth bounding boxes based on the IoU (Intersection over Union) metric. GIoU loss introduces an improvement when calculating IoU: it calculates the minimum closure between the ground truth and predicted boxes (the smallest box containing both boxes) and then calculates the IoU. This improvement more effectively accounts for differences in the shape and size of the target boxes, allowing GIoU loss to better penalize the discrepancies between the predicted and ground truth boxes during optimization.

[0094] Cross-entropy loss is a commonly used loss function for classification tasks, widely applied in machine learning and deep learning. It evaluates model performance by measuring the difference between the model's predictions and the true labels. Specifically, for classification tasks, cross-entropy loss measures the difference between the probability distributions of the predicted labels and the probability distributions of the true labels by calculating the cross-entropy between the two.

[0095] Focal loss is a loss function that improves model performance by mitigating the problems caused by class imbalance in classification problems. Originally designed to address class imbalance in object detection, it has since been widely applied to other classification tasks. Traditional cross-entropy loss has some limitations when dealing with class imbalance, such as a smaller contribution from positive samples when the proportion of negative samples is high. Focal loss addresses this issue by introducing a moderating factor that reduces the contribution of easily classified samples to the loss function, focusing more on difficult samples.

[0096] 2) Total losses:

[0097] Final Loss = L base +λ e *L e

[0098] Where Final Loss is the total loss, λ eThis is a hyperparameter, and its value is set to 1.

[0099] For ease of explanation, this embodiment refers to the Transformer encoder, Transformer decoder, auxiliary feature fusion module, fusion noise suppression module, and task classifier involved as the detection network model. During training, all parameters of the detection network model are updated. After training, the detection network model is used to obtain the interaction relationships between people from the input image. In inference applications, the input image, after being processed by the Transformer encoder and its three branches, is directly predicted by the corresponding task classifier.

[0100] This embodiment provides a parallel character interaction relationship detection method based on feature fusion and noise suppression. First, a three-branch network model is constructed, dividing the original HOI task into three branches for parallel processing. During detection, three completely separate parallel detectors are used for separate detection. To establish communication between independent branches and facilitate the exchange of contextual information, this invention proposes an auxiliary feature fusion module, utilizing the relationships between different pieces of information to more accurately understand and analyze the image. To reduce the interference of noise that may be generated during the fusion process on the original features, this invention proposes a fusion noise suppression module, which can improve the accuracy of interaction detection.

[0101] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0102] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A parallel method for detecting character interaction relationships based on feature fusion and noise suppression, characterized in that, The parallel character interaction relationship detection method based on feature fusion and noise suppression includes: Obtain labeled image samples from the training dataset, and use the Transformer encoder to process the image samples to obtain global encoded features; The global encoded features are input into the HOI detection module to obtain human features, object features and interaction features. The HOI detection module consists of three branches: human branch, object branch and interaction branch. Each branch contains a Transformer decoder. Take the output of each branch of the HOI detection module and input it into the corresponding auxiliary feature fusion module. Each auxiliary feature fusion module takes the output of the corresponding branch as the main feature and the outputs of the other two branches as auxiliary features. After fusing the main feature and the auxiliary features, the corresponding fused feature is obtained. The fusion features corresponding to the interaction branches are input into the fusion noise suppression module, which obtains the early prediction classification result and the early interaction loss based on the early prediction classification result. The fusion features corresponding to the three branches are input into the corresponding task classifiers respectively. The triplet prediction results are obtained based on the task classifiers, and the basic loss is calculated based on the triplet prediction results and the labels of the image samples. The total loss is obtained by combining the pre-interaction loss and the basic loss. The parameters of the detection network model are updated according to the total loss. The trained detection network model is used to obtain the interaction relationship between people in the input image. The detection network model includes a Transformer encoder, a Transformer decoder, an auxiliary feature fusion module, a fusion noise suppression module, and a task classifier.

2. The parallel character interaction relationship detection method based on feature fusion and noise suppression according to claim 1, characterized in that, The process of using a Transformer encoder to process image samples to obtain global encoded features includes: The ResNet-50 network model is used to preprocess the image samples to obtain the overall feature map; The overall feature map is divided into several image blocks of the same size, and the image blocks are linearly mapped into vectors; The Transformer encoder is used to process vectors to obtain global encoded features.

3. The parallel character interaction relationship detection method based on feature fusion and noise suppression according to claim 1, characterized in that, The Transformer encoder contains multiple coding layers, with the output of the previous coding layer serving as the input to the next coding layer.

4. The parallel character interaction relationship detection method based on feature fusion and noise suppression according to claim 1, characterized in that, The Transformer decoder in the human body branch uses the human body query vector. h The Transformer decoder in the object branch uses the object query vector Query. o The Transformer decoder in the interactive branch uses a relational query vector Query. i .

5. The parallel character interaction relationship detection method based on feature fusion and noise suppression according to claim 1, characterized in that, Each auxiliary feature fusion module takes the output of the corresponding branch as the main feature and the outputs of the other two branches as auxiliary features. After fusing the main feature and the auxiliary features, the corresponding fused feature is obtained, including: 1) Use an MLP network to process the two auxiliary features into a hybrid feature: In the formula, m represents the mixed features, self_att represents the self-attention mechanism, and MLP is a multilayer perceptron network. and For auxiliary features, a1∈{h,o,i}, a2∈{h,o,i}, a1≠a2; 2) The main feature F t t∈{h,o,i},t≠a1≠a2 serves as the query in the attention mechanism, and is successively fused with hybrid features and global encoding features through multiple cross-attention mechanisms: f t =cross_att(cross_att(F t ,m),M) In the formula, f t The intermediate features are fused, aross_att(q,kv) is the cross-attention mechanism, q is the query, k and v are the key and value respectively, and M is the global encoded feature; 3) Pass the main features and intermediate features through a channel attention fusion layer to obtain the fused features corresponding to each branch: In the formula, For feature fusion, channel_att represents the channel attention mechanism, channel_att = σ(MLP(F t ,f t )), ⊙ represents the Hadamard product operation, and σ is the sigmoid activation function.

6. The parallel character interaction relationship detection method based on feature fusion and noise suppression according to claim 1, characterized in that, The step involves inputting the fusion features corresponding to the interaction branches into the fusion noise suppression module, obtaining the early predicted classification result from the fusion noise suppression module, and obtaining the early interaction loss based on the early predicted classification result, including: 1) The fusion features corresponding to the interaction branches are fed into the MLP network of the fusion noise suppression module to obtain the suppression features F. p ; 2) Suppression feature F p After averaging, the data is fed into a feedforward neural network to obtain the predicted classification result P, where P = [p1, p2, ..., p...]. k ,…,p K ], where K represents the total number of interaction categories in the training dataset, p k This represents the probability of predicting the k-th interaction in the classification result P in advance; 3) The aforementioned early interaction loss L e Obtained from the asymmetric loss function: Where, p′ k It is the offset probability, p' k =max(p k -u), where u is the threshold, y k For interactive labels of image samples, y k =1 indicates that the k-th interaction exists in the image sample, y k =0 indicates that the k-th interaction does not exist in the image sample, and γ+ and γ- are both hyperparameters.

7. The parallel character interaction relationship detection method based on feature fusion and noise suppression according to claim 1, characterized in that, The process involves inputting the fused features corresponding to the three branches into the corresponding task classifiers, obtaining triplet prediction results based on the task classifiers, and calculating the basic loss based on the triplet prediction results and the labels of the image samples, including: In the formula, Box h The bounding box represents the predicted human body shape, and FFN represents the feedforward neural network. Box represents the fusion feature corresponding to the human body branch. o Represents the predicted object bounding box. The fusion feature corresponding to the object branch, Score o This represents the score for each object query across all object categories. For the fusion features corresponding to the interaction branches, FFN proj For linear adjustment layers, Score interact For each interaction query, the score across all interaction categories is used to determine the final triplet prediction result: <Box h (Box) o Score o Score interact >; The basic loss is calculated as follows: L base =λ b *L bbox +λ u *L u +λ o *L o +λ i *L i In the formula, L base For the basic loss, L bbox The bounding box regression loss, L, is calculated using the absolute value loss function. u The bounding box regression loss is calculated using the Giou loss function. The predicted values ​​for both bounding box regression losses are... Box = Concat(Box) h Box o ), N q For the number of queries, the four dimensions represent the x-coordinate, y-coordinate, width, and length of the bounding box, respectively. o It is determined by the cross-entropy loss function based on the score. o The calculated object classification loss, L i It is determined by the focus loss function based on the score. interact The calculated interaction classification loss, λ b , λ u , λ o and λ i All of these are hyperparameters.

Images