A target relation recognition method based on self-evolution domain migration

Abstract

The application provides a target relationship identification method based on self-evolution domain migration, generates data by using a generation model, and fills the domain difference between the generated data and the original data by using a feature alignment method, so as to improve the performance of a target relationship identification benchmark model. This method significantly reduces the dependence of target relationship identification on large-scale high-quality real data, fills the data shortage problem by generating images, and provides a new idea for using generated data for actual training and target relationship identification. The application proposes a scene alignment module and an instance alignment module to align context scene features and human-object pair instance features, which provides a new idea for using generated data for actual training and target relationship identification, and the new model framework makes progress in precision and availability.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more specifically to a method for target relationship recognition. Background Technology

[0002] The goal of target relation recognition is to locate humans and objects in images and identify their interactions. Relationship interactions are represented by a triple <human, relation, object>. Target relation recognition allows machines to understand human activities in images at a fine-grained level, representing a significant step towards advanced visual reasoning and scene understanding. Although many models have been proposed in recent years, the long-tailed distribution of objects and relations in datasets often results in models trained on such datasets performing well on non-rare classes but only moderately on rare classes.

[0003] Previous methods for addressing the long-tail problem can be categorized into two paradigms. The first is training strategy-based methods, which alleviate the long-tail problem by employing strategies such as resampling and retraining, specifically by altering the sampling strategy for long-tail data or adjusting the loss function. The second approach is based on ensemble learning, which utilizes the concept of ensemble learning to reconstruct the interactions between person-object pairs and different HOI instances as new training samples. In this way, they promote knowledge generalization between HOI classes and form new triples to alleviate the long-tail problem. While these methods have achieved significant progress, they haven't fundamentally solved the long-tail problem caused by data imbalance. Recently, AI-generated content has made significant progress in image generation, opening up the possibility of replacing real data with AI-generated data. Therefore, a direct idea is to use generative models to synthesize rare classes in target relation recognition datasets, which can provide more training data and address the long-tail distribution problem in target relation recognition from the ground up. However, simply using target relation recognition labels as language-guided images and merging them into the original dataset for training does not directly improve model performance. Analysis shows this is due to the significant domain gap between the original and generated data. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a target relationship identification method based on self-evolving domain migration.

[0005] This invention utilizes generative models to generate data and employs feature alignment methods to bridge the domain differences between generated and original data, thereby improving the performance of benchmark models for target relationship recognition. This approach significantly reduces the reliance on large-scale, high-quality real-world data for target relationship recognition, filling the data gap by generating images and providing a new approach for using generated data in practical training and target relationship recognition.

[0006] The technical solution adopted in this invention mainly includes the following parts: a data generation part, a pseudo-label generation part, and a feature alignment part. First, in the data generation part, existing generative models, such as blip-diffusion, are used to generate the required data by using the target relation category as language control. Second, in the pseudo-label generation part, a teacher-student model-based method is adopted. The generated images, after weak enhancement processing, are fed into the teacher model to generate pseudo-label data that can be used for training. Finally, in the feature alignment part, a scene feature alignment module and an instance feature alignment module are introduced. These modules align features in terms of scene and instance for images in the original domain and generated domain, respectively, to reduce the gap between the two domains. By combining these parts, this invention realizes a target relation recognition method based on self-evolving domain transfer.

[0007] The technical solution adopted by this invention to solve its technical problem includes the following steps:

[0008] Step 1: First, collect labels from the existing person-relationship recognition dataset. Each label is a triple <person, relation, object>. For example, the person-relationship of a person riding a horse is represented by the triple <person, riding, horse>. Then, represent each triple as a text description of "an image of a person + relation + object," where relation represents the relation category in the triple, and object represents the object category. For example, the triple <person, relation, object> represents the text description of "an image of a person riding a horse." Input the text into the blip-diffusion model to generate the corresponding image I. gen ;

[0009] Step 2: The teacher-student model consists of two target relationship recognition baseline models with the same architecture and initial weights. The teacher model takes a weakly enhanced generated image as input and generates pseudo-labels. The student model takes a strongly enhanced original image and the generated image as input and updates itself using backpropagation with the pseudo-labels generated by the teacher model. The parameters of the teacher model are updated by an exponential moving average of the parameters of the student model, as shown in the following formula:

[0010] θ t =αθ t +(1-θ)θ s (1)

[0011] Where θ t These are the parameters of the teacher model, θ s For the student model, α is a hyperparameter with a value range of 0-1; for the labels generated by the teacher model, a predefined threshold is set with a value range of 0-2.

[0012] Since the generated images cannot be trained without corresponding labels, the generated images are first input into the teacher model. The teacher model outputs a label and its corresponding prediction score. If the prediction score is higher than the threshold, the label generated by the teacher model is used as a pseudo-label for supervising the generated images; if the prediction score output by the teacher model is lower than the threshold, the corresponding prediction is discarded.

[0013] Step 3: For the baseline model of target relationship recognition, visual feature extraction is achieved using CNN and Transformer encoder;

[0014] Step 4: In order to align the scene information of the image, a conditionally controlled image reconstruction method is proposed. Some regions of the image are generated by random occlusion, and under the conditional control of the original image features of the same category, the encoder and auxiliary decoder of the student model are used to reconstruct the occluded regions. This helps to improve the model's learning ability of the generated domain features and improve the ability of the generated domain features to transfer to the original domain features.

[0015] Step 5: The operation in Step 4 effectively aligned the contextual scene features, but the target relationship recognition task is to locate person-object pairs and identify their relationship. Therefore, instance-level features play a very important role. Thus, an instance feature alignment module is proposed to align the person-object pair instance features.

[0016] Step 6: Calculate the overall network loss;

[0017] Backpropagation is performed using a loss function, and multiple models are obtained through training. The model with a map value of 0.34 or higher on the test set is selected for application. Finally, the relationships between people in the image can be obtained based on the model's output.

[0018] In step 3, an image I with shape (H, W, C) is fed into the CNN, and the CNN outputs features. Where H′ represents the length of the feature, W′ represents the width of the feature, and D b The dimension representing the feature; then the feature X b The feature dimension is transformed into D through a linear mapping layer. c Next, the flattening operator is used to compress the spatial dimension to one dimension, generating the flattened feature. Then X b ′ and position encoding These features are fed into the Transformer encoder, where, through a multi-head self-attention mechanism, the Transformer encoder learns global information and generates features with richer contextual information. After obtaining the Transformer encoder features, the Transformer encoder features Xe and instance query The input instance decoder detects the location of people and objects in an image and classifies the objects, where N insq D represents the number of queries. insd Represents the feature dimension of the query; Transformer encoder feature X e Relationship query The input relation decoder predicts the relation category corresponding to the person-object pair, where N intq D represents the number of relation queries. insd The feature dimensions represent the relation query; finally, the relation decoder outputs the relation features. Where N intq D represents the number of relation queries. d This represents the feature dimension of the decoder.

[0019] In step 4, firstly, the generated image I is scaled up by a ratio L. gen Random masking is performed, and in this invention, L is set to 0.8 to obtain the mask image I. mask Then the original image I src and mask image I mask The original image features X are obtained through the encoder of the model. src and mask image features X mask To improve the ability to learn the features of the original image, X src and X mask The input is fed into the Condition controller, which is a Transformer decoder. Within the Condition controller, X... src For querying, X mask The value serves as both a key and a value; the output of the conditional controller is represented as Sign. Considering that not all image patches in the original image contribute to the reconstructed image, a gating mechanism is used to select Sign values ​​greater than 0.5. The final original image feature X used for conditional control is... src_ond By Sign and X src Element-wise multiplication is used to mask out some original image features that are irrelevant to the reconstruction. Sign is expressed as:

[0020] Sign = Gate(Condition(X) src X mask (2)

[0021] X src_ond =Sign*X src (3)

[0022] Where Condition and Gate represent condition controller and gate mechanism respectively, and * represents element-wise multiplication;

[0023] The original image features X used for conditional control src_ond Features X of the masked image after masking mask The features are stitched together to obtain the fused features, which are then fed into an auxiliary decoder to reconstruct the generated image, resulting in the reconstructed image I. recon :

[0024] I recon =Decoder(Concate(X) src_ond ,X mask (4)

[0025] Where Decoder represents the auxiliary decoder, and Concate represents the concatenation operation;

[0026] Then reconstruct image I recon With generated image I gen By comparison, the scene reconstruction loss L is calculated using the mean squared error loss. scn The calculation formula is as follows:

[0027]

[0028] in and These represent the reconstructed i-th image patch and the generated i-th image patch, respectively.

[0029] In step 5, the specific steps are as follows:

[0030] Many previous domain alignment methods employed an adversarial learning framework to align features, with most methods feeding all relation queries into the discriminator for alignment. However, due to the presence of redundant or even erroneous queries, this inevitably leads to the model failing to focus on the desired relation queries, resulting in suboptimal results. To address this issue, a prototype instance alignment method based on graph networks is proposed.

[0031] For the output of the relation decoder, due to the existence of redundant queries, and considering that there are an average of about 6 sets of relationships between people in an image in the dataset, the 6 relation queries with the highest scores are selected as instance prototypes, and the remaining queries are treated as ordinary queries. To better aggregate instance information, ordinary queries can transmit instance feature information to the prototypes. Considering the effectiveness of graph neural networks in information transmission, a graph neural network is used to aggregate instance information. Specifically, a graph G = (A, X) is constructed, where X represents node features, specifically the output of the relation decoder. The prototype instances in the graph network are called prototype nodes, and the ordinary queries are called ordinary nodes. A is the adjacency matrix. To aggregate instance information to the prototypes, edges are defined to connect prototype nodes with ordinary nodes and other prototype nodes, while ordinary nodes are not connected to each other. Then, a graph convolutional neural network is used for node feature propagation, as shown in the following formula:

[0032] X = GCN(A, X) (6)

[0033] X p ={X i |i∈P} (7)

[0034] Where GCN represents Graph Convolutional Neural Network, P represents the set of indices of prototype nodes, A is the adjacency matrix, and X represents node features. i X represents the feature of the i-th node. p Representative prototype node features;

[0035] Extract the prototype node features X p The obtained instance features are used as input to the decoding feature discriminator for feature alignment, while the global feature X is also used. b and encoder features X e The data are also fed into the global feature discriminator and the encoded feature domain discriminator for feature alignment, as shown in the formula:

[0036]

[0037] Where D bac It is a global feature domain discriminator, D enc If is the coded feature domain discriminator, src represents the original image, and gen represents the generated image, then the instance loss is defined as follows:

[0038]

[0039] Where D dec This is a discriminator for the decoding feature domain.

[0040] In step 6, the network receives two inputs: the original image and the generated image, using L... supL represents the original image target relation recognition loss calculated using the labels. unsup The loss for object relationship recognition in generated images, calculated using pseudo-labels, is represented by the following formula for discriminator optimization:

[0041]

[0042] Where T and D represent the relation recognition detector and discriminator, respectively, and L inst For the instance loss obtained from formula (9), L adv For the discriminator loss; L is the loss of the original image. src It includes the original image target relationship recognition loss and the original image discriminator loss, and is defined as:

[0043] L src =L sup +L advs (11)

[0044] Where L sup For the target relationship recognition loss of the original image, L advs The original image is fed into the model and the original image discriminator loss is obtained using formula (10);

[0045] Loss L in generating the image gen The loss includes target relationship recognition loss, discriminator loss, and context reconstruction loss, defined as:

[0046] L gen =L unsup +L scn +L advg (12)

[0047] Where L unsup To generate the image target relationship recognition loss, L scn L is the scene reconstruction loss in formula (5). advg The generated image is fed into the model using the loss of the generated image discriminator obtained by formula (10);

[0048] The overall loss function is:

[0049] L loss =L src +L gen (13)

[0050] After obtaining the loss function, backpropagation is performed using the loss function. Then, multiple models are trained, and the model with a map value of 0.34 or higher on the test set is selected for application. Finally, the relationships between people in the image can be obtained based on the output of the model.

[0051] The beneficial effect of this invention lies in its adoption of a target relationship recognition method based on self-evolving domain transfer to address the long-tail problem in current target relationship recognition. Specifically, it proposes using a generative model to handle the long-tail problem. Furthermore, to bridge the domain gap between generated and original data, it proposes a scene alignment module and an instance alignment module to align contextual scene features and person-object pair instance features. This provides a new approach for using generated data for practical training and target relationship recognition. Experimental results show that the new model framework achieves improvements in both accuracy and usability. The overall structure of the model is as follows: Figure 1 As shown. Attached Figure Description

[0052] Figure 1 This is a model framework diagram of the target relationship recognition method based on self-evolutionary domain migration of the present invention.

[0053] Figure 2 This is the actual predicted effect of the present invention. Detailed Implementation

[0054] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0055] The following is a specific implementation example of the present invention.

[0056] This invention provides a target relationship identification method based on self-evolving domain migration, the specific process of which is as follows:

[0057] 1. Image generation;

[0058] The target relationship category information is extracted, such as "person riding a horse", and then the category is expanded to "a picture of a person riding a horse" and fed into the blip-diffusion model to produce an image of that category.

[0059] 2. Image feature extraction;

[0060] Given an original image I src and generate image I gen The image is resized to 512×512 and then fed into a ResNet-50 convolutional neural network to obtain the image features X to be detected. b ∈R (16×16)×2048 Then, it is passed through a linear mapping layer to make its feature dimension 256, and the new feature is X. b ′∈R (16×16)×256 This is then fed into the Transformer encoder, where, through a self-attention mechanism, the encoder outputs a feature X. e ∈R 256×256 .

[0061] 3. Target category and location prediction;

[0062] Encoder feature X e The data is fed into the instance decoder, which is a Transforemer decoder. The query value is set to 256. The output of the instance decoder is then X. instd ∈R 256×256 For each query, three fully connected layers are connected to predict the bounding box of a person, the bounding box of an object, and the category of the object, respectively.

[0063] 4. Target relationship prediction;

[0064] Encoder feature X e and instance decoder feature X instd The input is fed into a relational decoder, specifically a Transforemer decoder. With the query set to 64, the output of the relational decoder is X. d ∈R 64×256 For each query, a fully connected layer is added to predict the relationship corresponding to the query.

[0065] 5. Scene feature alignment;

[0066] Generate image I gen After being fed into the student model, I is first randomly masked at a ratio of 0.8 to obtain I. mask Then the original image I src and the masked image I mask The features X are obtained through the encoder of the model. src ∈R 256×256 and X mask ∈R 256×256 Then X src and X mask The signal is fed into the condition controller, and then passes through a gating unit to obtain the control signal Sign∈R. 256×1 Then, after the control signal expands the dimension, it interacts with X. src Element-by-element multiplication yields the conditional control signal X. src_cond ∈R 256 ×256 The conditional control signal is concatenated with the occluded features to obtain the fused features, which are then fed into the auxiliary decoder to reconstruct the generated image as shown in formula (4). The reconstructed image is then compared with the generated image according to formula (5) to obtain the contextual scene reconstruction loss L. scn .

[0067] 6. Encoder feature alignment;

[0068] Then the ResNet-50 feature X bThe data is fed into a global feature discriminator, which specifically consists of a two-dimensional convolution, and finally outputs the discriminator result D. b ∈R (16×16)×1 Similarly, the encoder feature X e The input is fed into a fully connected coded feature discriminator, and the final discriminator result D is output. e ∈R 256×1 .

[0069] 7. Instance feature alignment;

[0070] For the output of the relation decoder, the predicted relation scores are sorted, and the six highest-scoring predictions are selected as prototype nodes, with the rest as ordinary nodes. Then, an adjacency matrix is ​​constructed as described above, thereby constructing a graph G. The constructed graph is then fed into a two-layer graph convolutional neural network to obtain X. d ′∈R 64×256 The output is then processed, and the features of the prototype node are extracted to obtain the prototype feature X. p ′∈R 6×256 Then, it is fed into a fully connected decoding feature discriminator, and finally the discriminator result D is output. inst ∈R 6×1 D e and D inst The discriminator loss is calculated according to formula (10), and then the overall loss is calculated according to formula (13) for backpropagation.

[0071] 8. Model training;

[0072] The entire training process was end-to-end. Experiments were primarily conducted on two different HOI datasets: HICO-Det and V-COCO. The pseudo-label threshold was 1.4 on the GEN-VLKT and HOICLIP benchmark models, and 0.4 on the CDN benchmark model. Fine-tuning was performed on the pre-trained weights of the HOI benchmark model. The AdamW optimizer was used, with a weight decay of 1e-4 and a batch size of 16 generated images. All models were trained for 5 epochs on 8 DCUs. For CDN, the RestNet learning rate was 1e-5, and the rest were 1e-4. For GEN-VLKT and HOICLIP, the RestNet learning rate was 1e-6, and the rest were 1e-4.

[0073] 9. Model application;

[0074] After the above training process, multiple models can be obtained. The optimal model (those with the best performance on the test set) is selected for application. The parameters of the entire network model remain fixed; only the input image data needs to be propagated forward. This sequentially yields the image and language feature vectors X.e and instance decoder X instd The relation decoder outputs X d Then, by using a fully connected layer to classify and regress the person's location, the target's category, and location, the prediction results can be directly obtained. The actual effect is shown in the image below. Figure 2 As shown, this method can significantly improve the performance of target relationship recognition.

[0075] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are within the protection scope of the present invention.

Claims

1. A target relation identification method based on self-evolving domain migration, characterized in that... Includes the following steps: Step 1: First, collect labels from existing person-relationship recognition datasets. Each label is a triple <person, relationship, object>, where relationship represents the relationship category in the triple and object represents the object category in the triple. Input the text into the blip-diffusion model to generate the corresponding image. ; Step 2: The teacher-student model consists of two target relationship recognition baseline models with the same architecture and initial weights. The teacher model takes a weakly enhanced generated image as input and generates pseudo-labels. The student model takes a strongly enhanced original image and the generated image as input and updates itself using backpropagation with the pseudo-labels generated by the teacher model. The parameters of the teacher model are updated by an exponential moving average of the parameters of the student model, as shown in the following formula: （1）； in These are the parameters of the teacher model. For the student model, α is a hyperparameter with a value range of 0-1; for the labels generated by the teacher model, a predefined threshold is set with a value range of 0-2. Since the generated images cannot be trained without corresponding labels, the generated images are first input into the teacher model. The teacher model outputs a label and its corresponding prediction score. If the prediction score is higher than the threshold, the label generated by the teacher model is used as a pseudo-label for supervising the generated images; if the prediction score output by the teacher model is lower than the threshold, the corresponding prediction is discarded. Step 3: For the baseline model of target relationship recognition, visual feature extraction is achieved using CNN and Transformer encoder; Step 4: In order to align the scene information of the image, a conditionally controlled image reconstruction method is proposed. Some regions of the image are generated by random occlusion, and under the conditional control of the original image features of the same category, the encoder and auxiliary decoder of the student model are used to reconstruct the occluded regions. This helps to improve the model's learning ability of the generated domain features and improve the ability of the generated domain features to transfer to the original domain features. Step 5: The operations in Step 4 effectively aligned the contextual scene features, and an instance feature alignment module was proposed to align the human-object instance features; Step 6: Calculate the overall network loss; Backpropagation is performed using a loss function, and multiple models are obtained through training. The model with a map value of 0.34 or higher on the test set is selected for application. Finally, the relationships between people in the image can be obtained based on the model's output.

2. The target relationship identification method based on self-evolving domain migration according to claim 1, characterized in that: In step 3, a shape of... pictures The CNN outputs features. ,in The length of the representative feature The width representing the feature, The dimension representing the feature; then the feature The feature dimensions are transformed through a linear mapping layer. Next, the flattening operator is used to compress the spatial dimension to one dimension, generating the flattened feature. ,Then and position encoding These features are fed into the Transformer encoder, where, through a multi-head self-attention mechanism, the Transformer encoder learns global information and generates features with richer contextual information. After obtaining the Transformer encoder features, the Transformer encoder features are... and instance query The input instance decoder detects the location of people and objects in an image and classifies the objects accordingly. Represents the number of queries. Represents the feature dimensions of the query; Transformer encoder features Relationship query The input relation decoder predicts the relation category corresponding to pairs of people and objects. The number of relation queries. The feature dimensions represent the relation query; finally, the relation decoder outputs the relation features. ,in The number of relation queries. This represents the feature dimension of the decoder.

3. The target relationship identification method based on self-evolving domain migration according to claim 1, characterized in that: In step 4, firstly, the generated image is scaled up to a ratio L. Perform random masking to obtain the mask image. Then the original image and mask image The original image features are obtained through the encoder of the model. and mask image features To improve the ability to learn the features of the original image, and Input Condition Controller The Condition controller is a Transformer decoder; within the Condition controller, For query, The value serves as both a key and a value; the output of the conditional controller is represented as Sign. Considering that not all image patches in the original image contribute to the reconstructed image, a gating mechanism is used to select Sign values ​​greater than 0.

5. These Sign values ​​are ultimately used for conditional control. By Sign and X src Element-wise multiplication is used to mask out some original image features that are irrelevant to the reconstruction. Sign is expressed as: （2）； （3）； Here, Condition and Gate represent condition controller and gating mechanism, respectively. Represents element-wise dot product; The original image features used for conditional control Features of the masked image after masking The features are stitched together to obtain the fused features, which are then fed into an auxiliary decoder to reconstruct the generated image, resulting in the reconstructed image I. recon : （4）； in Represents an auxiliary decoder. This represents a splicing operation; Then the image will be reconstructed. With generated image By comparison, the scene reconstruction loss is calculated using the mean squared error loss. The calculation formula is as follows: （5）； in These represent the reconstructed i-th image patch and the generated i-th image patch, respectively.

4. The target relationship identification method based on self-evolving domain migration according to claim 3, characterized in that: The value of L is 0.

8.

5. The target relationship identification method based on self-evolving domain migration according to claim 1, characterized in that: In step 5, the specific steps are as follows: Build a graph The node features are represented by the output of the relation decoder. Prototype instances in the graph network are called prototype nodes, and ordinary queries are called ordinary nodes. A is the adjacency matrix. To aggregate instance information to prototypes, edges are defined to connect prototype nodes with ordinary nodes and other prototype nodes, while ordinary nodes are not connected to each other. Then, a graph convolutional neural network is used for node feature propagation, as shown in the following formula: （6）； （7）； in Representative graph convolutional neural network, A represents the set of indices of the prototype nodes, and A is the adjacency matrix. Representative node characteristics, Representing the Each node's characteristics Representative prototype node features; Extracted prototype node features The obtained instance features are used as input to the decoding feature discriminator for feature alignment, while the global features are also used. and encoder features The data are also fed into the global feature discriminator and the encoded feature domain discriminator for feature alignment, as shown in the formula: （8）； in It is a global feature domain discriminator. It is a discriminator for the encoded feature domain. Represents the original image. If we represent the generated image, then the instance loss is defined as follows: （9）； in This is a discriminator for the decoding feature domain.

6. The target relationship identification method based on self-evolving domain migration according to claim 1, characterized in that: In step 6, the network receives two inputs: the original image and the generated image, and uses... The original image target relationship recognition loss is calculated using the labels on the original image. The loss for object relationship recognition in generated images, calculated using pseudo-labels, is represented by the following formula for discriminator optimization: （10）； Where T and D represent the relation recognition detector and discriminator, respectively. The instance loss is obtained from formula (9). For discriminator loss; for loss of the original image. It includes the original image target relationship recognition loss and the original image discriminator loss, and is defined as: （11）； in The loss for target relationship recognition in the original image. The original image is fed into the model and the original image discriminator loss is obtained using formula (10); Loss of generating images The loss includes target relationship recognition loss, discriminator loss, and context reconstruction loss, defined as: （12）； in To generate the image target relationship recognition loss, The scene reconstruction loss is given by formula (5). The generated image is fed into the model using the loss of the generated image discriminator obtained by formula (10); The overall loss function is: （13）； After obtaining the loss function, backpropagation is performed using the loss function. Then, multiple models are trained, and the model with a map value of 0.34 or higher on the test set is selected for application. Finally, the relationships between people in the image can be obtained based on the output of the model.

Images