A semi-supervised object detection method based on attention relationship and neighborhood feature constraint

Abstract

The application belongs to the field of image semi-supervised object detection of computer vision, and particularly relates to a semi-supervised object detection method based on attention relationship and neighborhood feature constraint, which has the following characteristics and includes the following steps: step 1, inputting a labeled data set and an unlabeled data set into a training network and loading pre-training weights; step 2, performing deep feature extraction on the input image, performing supervised training on the labeled data sample image, and updating the network parameters through direction propagation; after deep feature extraction is performed on the unlabeled original image and the image after enhancement disturbance, a certain position is selected as a center domain, and the similarity of the center domain and the surrounding neighborhood sample position is calculated; step 3, if the similarity of the selected center domain position and the neighborhood sample position exceeds a set threshold, then the center domain and the neighborhood are subjected to smoothness learning, and the distance between the two is narrowed; step 4, when the neighborhood feature constraint is performed, the attention relationship between the center domain and each neighborhood position is considered, and attention weights are added to each neighborhood position to achieve more targeted constraint learning; step 5, after semi-supervised training, the network model weight is obtained, the model weight file is loaded, and the test image sample is input into the model to obtain the test sample detection result. In addition, the neighborhood feature constraint algorithm model of the application learns the feature relationship between the center domain and the neighborhood, and improves the prediction accuracy of the traditional semi-supervised consistency constraint model.

Description

Technical Field

[0001] This invention belongs to the field of semi-supervised object detection in computer vision, specifically relating to a semi-supervised object detection method based on attention relationships and neighborhood feature constraints. Background Technology

[0002] Object detection, a fundamental task in computer vision, has always been a hot and rapidly developing area. With the advancement of deep learning technology, object detection research has been further propelled, evolving from traditional methods of training detectors using hand-crafted features to the emergence of various neural network models, continuously improving the speed and accuracy of object detection. In the real world, a large number of unlabeled images exist, but labeling an image for object detection requires a significant amount of manpower. Moreover, object detection labeling is more time-consuming and labor-intensive than image classification labeling, and is prone to errors, resulting in low data utilization. Therefore, to address the labeling problem, semi-supervised learning techniques have recently attracted widespread attention from researchers. Semi-supervised techniques can improve the utilization efficiency of large amounts of unlabeled samples, enhancing object detection performance with only a small amount of labeled data.

[0003] Semi-supervised learning lies between supervised and unsupervised learning. It requires only a portion of the training data to be labeled, with the majority of samples remaining unlabeled. Semi-supervised learning can improve model accuracy by utilizing information from unlabeled samples. It alleviates the reliance on manual annotation, thus reducing annotation costs. Therefore, introducing semi-supervised learning techniques into object detection, especially for small objects or difficult-to-label images, is of great research significance. As early as the 1990s, researchers attempted to improve classifier performance using unlabeled samples during training. In recent years, with the rise of deep learning, the need for large datasets has increased, making the reliance on labeled samples more severe. Researching how to use unlabeled samples to improve model performance has become increasingly important and has thus attracted researchers' attention. In 2016, semi-supervised research methods using consistency constraints emerged, such as the π model, Temporalensembling Model, and Mean Teacher algorithms. Google's MixMatch and Fixmatch algorithms are hybrid methods combining consistency regularization and self-training. For a given input, even with some perturbation, the system's predictions should remain consistent, thus improving the model's generalization performance and avoiding overfitting to a small number of labeled datasets. These methods are semi-supervised approaches based on classification tasks, and have gained more attention in the field of semi-supervised object detection in recent years. In the last two years, Seoul National University's CSD and ISD are both semi-supervised object detection methods based on consistency constraints, and both rely on data augmentation techniques for consistency constraints, but they are more suitable for single-stage object detection methods. PLSD, on the other hand, is a semi-supervised algorithm for two-stage object detection, integrating self-supervision methods on top of consistency constraints. Summary of the Invention

[0004] The main technical problem addressed by this invention is the insufficient utilization of feature relationships between samples due to the inadequacy of traditional consistency constraints, as well as issues related to attention relationships between samples in the center and neighboring regions. The technical solution provided by this invention is as follows:

[0005] A semi-supervised object detection method based on attention relationships and neighborhood feature constraints includes the following steps:

[0006] Deep features are extracted from the original image and the perturbed image, and the perturbed image is enhanced by hybrid interpolation of the intermediate hidden layer;

[0007] Calculate the similarity between the center and neighborhood locations. If the similarity exceeds the threshold, perform smooth learning on the center and neighborhood to shorten the distance between the two locations.

[0008] Considering the attention relationship between the central region and the neighboring regions, attention weights are added to each neighboring region to achieve more targeted constraint learning;

[0009] After training, the weights of the semi-supervised network model are obtained. The model weight file is then loaded to test sample images to obtain the detection results.

[0010] Furthermore, the step of extracting deep features from the original image and the perturbed image, and performing hybrid interpolation enhancement on the perturbed image using intermediate hidden layers, specifically includes:

[0011] Forward propagation of the original image extracts features to obtain the intermediate hidden layer feature map. ;

[0012] After horizontally flipping the original image, forward propagation is performed to extract features, which also yields the intermediate hidden layer feature maps. ;

[0013] Using intermediate hidden layer feature maps and intermediate hidden layer feature maps Hybrid interpolation enhancement yields the intermediate hidden layer feature map after hybrid interpolation. ;

[0014] The formula for calculating the intermediate hidden layer hybrid interpolation enhancement is as follows:

[0015] ,

[0016] in The number is randomly sampled from the beta distribution. ;

[0017] Feature maps of intermediate hidden layers , , Subsequent training was conducted separately, and the results were predicted.

[0018] if and If the target is predicted at a certain position, the formula for the hybrid interpolation constraint that needs to be added is as follows:

[0019] ,

[0020] in express Divergence is used to measure the distance between two distributions. Indicates intermediate hidden layer and The result after mixing the predicted corresponding positions. Indicates intermediate hidden layer Predict the category confidence score for the corresponding location. Representation of feature map , Hybrid interpolation consistency constraint when all targets are predicted;

[0021] if The target was predicted at the corresponding location, but If the target is not predicted at the corresponding location, the hybrid interpolation constraint formula to be added is expressed as follows:

[0022] ,

[0023] ,

[0024] in express Divergence is used to measure the distance between two biased distributions. and These represent the intermediate hidden layers. The corresponding category prediction vector and coordinate prediction vector, Representation of feature map , There is only one category prediction constraint when the target is predicted; while and These represent the intermediate hybrid interpolation hidden layers. The corresponding category prediction vector and coordinate prediction vector, Representation of feature map , There is only one position prediction constraint when the target is predicted;

[0025] Similarly, if The target was predicted at the corresponding location, and If the target is not predicted at the corresponding position, then the hybrid interpolation constraint formula will... Replace with Therefore, the overall hybrid interpolation constraint formula is expressed as follows:

[0026] ,

[0027] ,

[0028] in Representation of feature map , There is only one hybrid interpolation consistency constraint when the target is predicted. This represents the constraint for the overall intermediate hidden layer hybrid interpolation;

[0029] Furthermore, the similarity between the central region and its neighboring regions is calculated. If the similarity exceeds a threshold, smoothing learning is performed on the central region and its neighboring regions to shorten the distance between them. Specifically, this includes: first, calculating the similarity between the central region and its neighboring regions using the following formula:

[0030] ,

[0031] ,

[0032] Since the class confidence vector is distributed, the Bach coefficient is used to calculate the similarity of the class confidence vectors between the center and the neighborhood. and Let represent the class confidence vectors obtained after forward propagation of the center region and the neighborhood region, respectively. The value represents the similarity obtained from the Bach coefficient, which ranges from [0,1] and means the similarity of the distribution overlap between the central region and the neighborhood.

[0033] Since the location confidence vector is in numerical form, L2 distance is used to calculate the similarity of the location confidence vectors between the center region and the neighborhood region. and Let represent the position confidence vectors obtained after forward propagation of the center region and the neighborhood region, respectively. This represents the calculated positional similarity, which means the similarity of the overlapping positional distributions between the central region and its neighboring regions.

[0034] Set the threshold to If the neighborhood similarity is greater than the threshold Then, smooth learning is performed on the center region and the neighborhood region to shorten the distance between them and improve the network's generalization performance. The specific formula is as follows: ,

[0035] ,

[0036] in This represents the class confidence vector obtained after forward propagation of the center region. This represents the class confidence vector obtained from forward propagation prediction after intermediate hidden layer hybrid interpolation enhancement is performed on the neighborhood. The loss function representing the neighborhood feature constraint with respect to the class confidence;

[0037] and This represents the position confidence vector obtained after forward propagation of the central region. This represents the position confidence vector obtained from forward propagation prediction after intermediate hidden layer hybrid interpolation enhancement is performed on the neighborhood. The loss function representing the neighborhood feature constraints with respect to location confidence;

[0038] By combining the neighborhood feature constraint loss function of the category and the neighborhood feature constraint loss function of the location, the total neighborhood feature constraint loss function is obtained. The calculation formula is as follows:

[0039] ,

[0040] Furthermore, considering the attention relationship between the central region and its neighboring locations, attention weights are added to each neighboring location to achieve more targeted constraint learning. The calculation formula is as follows:

[0041] ,

[0042] ,

[0043] in Feature bars representing a certain location, This represents a multilayer perceptron. This indicates a global average pooling operation. This represents the global max pooling operation, using and These are used to describe global average feature information and local maximum feature information, respectively.

[0044] A normalization operation similar to softmax is used to calculate the attention relationship between the center region and the neighborhood region. The calculation formula is as follows:

[0045] ,

[0046] in Indicates the first The feature information of each neighborhood after global max pooling and MLP. Indicates the first Feature information of each central region after global average pooling and MLP. Representing the neighborhood and central area The closer the attention relationship, the greater the weight should be when implementing neighborhood feature constraints. Specifically, the formula is as follows:

[0047] ,

[0048] in Indicates the first The neighborhood and the first Attention relationships within each central region This represents the feature constraint loss function between them. This represents the total neighborhood feature constraint loss;

[0049] Furthermore, after training, the semi-supervised network model weights are obtained. The model weight file is then loaded with test sample images to obtain the detection results, specifically including the following:

[0050] Calculate the map value using the test sample data. Attached Figure Description

[0051] Figure 1 The flowchart shows the semi-supervised target detection method based on attention relationship and neighborhood feature constraints constructed in this invention.

[0052] Figure 2 The feature constraints of the central domain and neighborhood constructed for this invention, and the smooth learning process diagram;

[0053] Figure 3 This is a flowchart illustrating the attention relationship in this invention;

[0054] Figure 4 This is a verification diagram of the neighborhood feature constraint effect in this invention. Detailed Implementation

[0055] The technical solutions in the embodiments of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. The described embodiments are merely some embodiments of the present invention.

[0056] The technical solution of the present invention to solve the above-mentioned technical problems is:

[0057] Example 1:

[0058] like Figure 1 As shown, the semi-supervised target detection method of the present invention, based on attention relationships and neighborhood feature constraints, includes the following steps:

[0059] Step 1: Extract deep features from the original image and the perturbed image, and perform hybrid interpolation enhancement on the perturbed image using intermediate hidden layers;

[0060] Forward propagation of the original image extracts features to obtain the intermediate hidden layer feature map. ;

[0061] After horizontally flipping the original image, forward propagation is performed to extract features, which also yields the intermediate hidden layer feature maps. ;

[0062] Using intermediate hidden layer feature maps and intermediate hidden layer feature maps Hybrid interpolation enhancement yields the intermediate hidden layer feature map after hybrid interpolation. ;

[0063] The formula for calculating the intermediate hidden layer hybrid interpolation enhancement is as follows:

[0064] ,

[0065] in The number is randomly sampled from the beta distribution. ∈[0,1];

[0066] Feature maps of intermediate hidden layers , , Subsequent training was conducted separately, and the results were predicted.

[0067] Step 2: Calculate the similarity between the center and neighborhood locations. If the similarity exceeds the threshold, perform smooth learning on the center and neighborhood to shorten the distance between them.

[0068] First, calculate the similarity between the center region and its neighborhood. If the neighborhood similarity is greater than a threshold... Then, smooth learning is performed on the center region and the neighborhood region to shorten the distance between them and improve the network's generalization performance. The specific formula is as follows:

[0069] ,

[0070] ,

[0071] in This represents the class confidence vector obtained after forward propagation of the center region. This represents the class confidence vector obtained from forward propagation prediction after intermediate hidden layer hybrid interpolation enhancement is performed on the neighborhood. Let represent the loss function of neighborhood feature constraints with respect to class confidence, and This represents the position confidence vector obtained after forward propagation of the central region. This represents the position confidence vector obtained from forward propagation prediction after intermediate hidden layer hybrid interpolation enhancement is performed on the neighborhood. The loss function representing the neighborhood feature constraints with respect to location confidence;

[0072] Step 3: Consider the attention relationship between the central region and the neighboring regions, and add attention weights to each neighboring region to achieve more targeted constraint learning;

[0073] We construct the attention relationship between the center region and its neighborhood, and add attention weights to each neighborhood position to achieve more targeted constraint learning. The calculation formula is as follows:

[0074] ,

[0075] ,

[0076] in Feature bars representing a certain location, This represents a multilayer perceptron. This indicates a global average pooling operation. This represents the global max pooling operation, using and These are used to describe global average feature information and local maximum feature information, respectively.

[0077] A normalization operation similar to softmax is used to calculate the attention relationship between the center region and the neighborhood region. The calculation formula is as follows:

[0078] ,

[0079] in Indicates the first The feature information of each neighborhood after global max pooling and MLP. Indicates the first Feature information of each central region after global average pooling and MLP. Representing the neighborhood and central area The closer the attention relationship, the greater the weight should be when implementing neighborhood feature constraints.

[0080] Step 4: After training, obtain the weights of the semi-supervised network model, load the model weight file, test sample images, and obtain the detection results.

Claims

1. A semi-supervised target detection method based on attention relationships and neighborhood feature constraints, characterized in that, Includes the following steps: Deep features are extracted from the original image and the perturbed image, and the perturbed image is enhanced by hybrid interpolation of the intermediate hidden layer; The deep features obtained from the forward propagation of the unlabeled image are obtained. The deep features predict the category and location vectors. The center location is selected, and the similarity between the center location and the neighborhood location is calculated. If the similarity exceeds the threshold, smoothness learning is performed on the two locations. The feature vector prediction results of the central region and the neighborhood after perturbation enhancement obtained above are used to calculate the distance and add a loss function to train and update the network parameters. Deep features are extracted from the central and neighboring regions, and attention relationships between them are calculated. Attention relationship weights are added to each neighboring location to achieve more targeted constraint learning. After training, the weights of the semi-supervised network model are obtained. The model weight file is then loaded to test sample images to obtain the detection results. The perturbated image undergoes hybrid interpolation enhancement in the intermediate hidden layer, specifically including the following: Forward propagation of the original image extracts features to obtain the intermediate hidden layer feature map. ; After horizontally flipping the original image, forward propagation is performed to extract features, which also yields the intermediate hidden layer feature maps. ; Using intermediate hidden layer feature maps and intermediate hidden layer feature maps Hybrid interpolation enhancement yields the intermediate hidden layer feature map after hybrid interpolation. ; The formula for calculating the intermediate hidden layer hybrid interpolation enhancement is as follows: , in The number is randomly sampled from the beta distribution. ∈[0,1]; Feature maps of intermediate hidden layers , , Subsequent training was conducted separately, and the results were predicted. if and If the target is predicted at a certain position, the formula for the hybrid interpolation constraint that needs to be added is as follows: ， in express Divergence is used to measure the distance between two distributions. Indicates intermediate hidden layer and The result after mixing the predicted corresponding positions. Indicates intermediate hidden layer Predict the category confidence score for the corresponding location. Representation of feature map , Hybrid interpolation consistency constraint when all targets are predicted; if The target was predicted at the corresponding location, but If the target is not predicted at the corresponding location, the hybrid interpolation constraint formula to be added is expressed as follows: ， ， in express Divergence is used to measure the distance between two biased distributions. and These represent the intermediate hidden layers. The corresponding category prediction vector and coordinate prediction vector, Representation of feature map , There is only one category prediction constraint when the target is predicted; while and These represent the intermediate hybrid interpolation hidden layers. The corresponding category prediction vector and coordinate prediction vector, Representation of feature map , There is only one position prediction constraint when the target is predicted; Similarly, if The target was predicted at the corresponding location, and If the target is not predicted at the corresponding position, then the hybrid interpolation constraint formula will... Replace with Therefore, the overall hybrid interpolation constraint formula is expressed as follows: ， ， in Representation of feature map , There is only one hybrid interpolation consistency constraint when the target is predicted. This represents the constraint for the overall intermediate hidden layer hybrid interpolation; The formula for calculating the similarity between the central region and the neighborhood positions is as follows: ， ， Since the class confidence vector is distributed, the Bach coefficient is used to calculate the similarity of the class confidence vectors between the center and the neighborhood. and Let represent the class confidence vectors obtained after forward propagation of the center region and the neighborhood region, respectively. The value represents the similarity obtained from the Bach coefficient, which ranges from [0,1] and means the similarity of the distribution overlap between the central region and the neighborhood. Since the location confidence vector is in numerical form, L2 distance is used to calculate the similarity of the location confidence vectors between the center region and the neighborhood region. and Let represent the position confidence vectors obtained after forward propagation of the center region and the neighborhood region, respectively. This represents the obtained positional similarity, which means the similarity of the overlapping positional distributions between the central region and its neighboring regions.

2. The semi-supervised object detection method based on attention relationship and neighborhood feature constraints according to claim 1, wherein calculating the similarity between the center region and the neighborhood, and performing smoothness learning on the two locations if the similarity exceeds a threshold, specifically includes the following: Set the threshold to Calculate the similarity between the center region and its neighbors. If the neighborhood similarity is greater than a threshold... Then, smooth learning is performed on the center region and the neighborhood region to shorten the distance between them and improve the network's generalization performance. The specific formula is as follows: ， ， in This represents the class confidence vector obtained after forward propagation of the center region. This represents the class confidence vector obtained from forward propagation prediction after intermediate hidden layer hybrid interpolation enhancement is performed on the neighborhood. The loss function representing the neighborhood feature constraint with respect to the class confidence; and This represents the position confidence vector obtained after forward propagation of the central region. This represents the position confidence vector obtained from forward propagation prediction after intermediate hidden layer hybrid interpolation enhancement is performed on the neighborhood. The loss function representing the neighborhood feature constraints with respect to location confidence; By combining the neighborhood feature constraint loss function of the category and the neighborhood feature constraint loss function of the location, the total neighborhood feature constraint loss function is obtained. The calculation formula is as follows: 。 3. The semi-supervised target detection method based on attention relationships and neighborhood feature constraints according to claim 1, characterized in that... We construct the attention relationship between the center region and its neighborhood, and add attention weights to each neighborhood position to achieve more targeted constraint learning. The calculation formula is as follows: ， ， in Feature bars representing a certain location, This represents a multilayer perceptron. This indicates a global average pooling operation. This represents the global max pooling operation, using and These are used to describe global average feature information and local maximum feature information, respectively. A normalization operation similar to softmax is used to calculate the attention relationship between the center region and the neighborhood region. The calculation formula is as follows: ， in Indicates the first The feature information of each neighborhood after global max pooling and MLP. Indicates the first Feature information of each central region after global average pooling and MLP. Representing the neighborhood and central area The closer the attention relationship, the greater the weight should be when implementing neighborhood feature constraints. Specifically, the formula is as follows: ， in Show the first The neighborhood and the first Attention relationships within each central region This represents the feature constraint loss function between them. This represents the total neighborhood feature constraint loss.

4. The semi-supervised object detection method based on attention relationships and neighborhood feature constraints according to claim 1, wherein after training, the weights of the semi-supervised network model are obtained, and the detection results are obtained by loading the model weight file and test sample images, specifically including the following: During training, the batch size is set to 32, including labeled and unlabeled sample images. Forward propagation extracts features and obtains prediction vectors. Backpropagation of the loss function updates the network parameters, and the parameter weights are saved every thousand iterations. When testing the model, the final weight parameters are loaded and the detection boxes are predicted. The target detection map value is calculated.

Images