A target detection method based on parallel interactive architecture model to achieve multi-dimensional feature fusion

Abstract

The present invention discloses a target detection method based on a parallel interactive architecture model to achieve multi-dimensional feature fusion, comprising the following steps: data acquisition and processing, building a model training environment, feeding data in the data set into the built model for training, and final model verification and testing. The present invention solves the problems of slow convergence speed and long training time in traditional target detection tasks, thereby improving the detection accuracy and speed of target detection tasks.

Description

Technical Field

[0001] The present invention belongs to the field of target detection in computer vision, and specifically proposes a target detection method based on a parallel interactive architecture model to achieve feature fusion. Background Art

[0002] Object detection has long been a fundamental task in computer vision. Its primary goal is to predict the location and category of instances in an image. As the foundation of many visual tasks, including instance segmentation and object tracking, object detection holds significant research significance in the field of image vision. With the recent rise in interest in practical applications such as autonomous driving and industrial defect detection, object detection has also received increasing attention from the industry. The core challenge of object detection is how to enable the detection network to fully learn the spatial and semantic information of the image from input features and how to use this information to accurately locate and classify instances. Object detectors require strong feature fusion capabilities and sufficient spatial sensitivity. Traditional deep learning detection models are mostly based on convolutional neural networks (CNNs). CNNs use convolution operations to fully fuse local features in an image. Their sensitive local spatial perception makes them one of the most suitable networks for object detection. However, CNNs also have certain limitations, such as their lack of global spatial feature fusion capabilities. Traditional CNN-based object detection models are generally divided into anchor-based and anchor-free models based on how they localize objects. The former uses anchors to predict potential objects, while the latter typically detect objects based on center points. Anchor-based models can be further divided into one-stage and two-stage models based on the detection steps. Classic models of the former include the YOLO series, SSD, and RetinaNet, while the latter is represented by the R-CNN series. The two-stage method first searches for potential target areas and then calculates the category scores for these areas in the second step, that is, first locates and then classifies; while the one-stage method directly generates a detection box in a single step to predict the category and position of the object. CNN-based models have two key issues: how to assign anchor and groundtruth labels and how to enable the model to effectively learn key semantic information from features. Models designed to solve these two problems also have significant shortcomings. For example, they require manual design of knowledge based on certain prior conditions. In fact, designing these priors, such as appropriate anchor points and thresholds, for different detection methods is a very difficult task. On the other hand, due to the size limitations of the convolution kernel, the global feature interaction ability of CNN is relatively weak.

[0003] In recent years, the emergence of the Visual Transformer (ViT), the Detection Transformer (DETR), and its variants have sparked a surge in the application of Transformers to object detection. These new object detection paradigms abandon traditional CNNs in favor of carefully designed multi-layer encoder-decoder architectures. The encoder fuses features, while the decoder uses object queries to decouple the rich semantics within the features. Compared to CNNs, ViT emphasizes global spatial semantic connections, integrating global spatial features through a global self-attention mechanism. DETR approaches object detection as a collective prediction task. During training, a certain number of object queries are matched with groundtruth. This process eliminates the label assignment required by traditional models; during inference, the network directly predicts objects based on the object queries. Furthermore, DETR uses positional embeddings to enhance the model's positional sensitivity for object localization. However, DETR-like detectors suffer from slow network convergence and high computational overhead. Summary of the Invention

[0004] To address the above-mentioned issues, the present invention combines the latest ideas from other fields of deep learning and proposes a method for feature fusion based on a parallel interactive architecture model, aiming to provide the model with advanced feature fusion capabilities. First, in terms of feature extraction methods, the present invention introduces 3D feature space window sampling, which is different from traditional CNN, and fully extracts local and global spatial features. Subsequently, the present invention proposes a multi-dimensional feature fusion network (CFFN), which enables the model to deeply fuse image features in spatial and channel dimensions, thereby better allowing the model to learn semantic information, thereby achieving better detection results and higher detection accuracy.

[0005] To achieve the above objectives, the present invention proposes the following technical solution: a target detection method based on a parallel interactive architecture model to achieve multi-dimensional feature fusion, the method comprising the following steps:

[0006] Step 1: Prepare the COCO2017 dataset required for model training; configure the COCO2017 dataset on the server and put it into the training folder in the required format;

[0007] Step 2: Build a model under the mmdetection framework and configure the PyTorch deep learning environment required for training;

[0008] Step 3: Set the training hyperparameters and input the dataset into the parallel interactive architecture end-to-end object detection model for training;

[0009] Step 4: The model feeds the input image into ResNet50 for feature extraction, outputs a multi-scale feature map, and constructs a 3D feature sampling space from the multi-scale feature map;

[0010] Step 5: Generate a set of object queries containing prediction vectors, including content vectors and position vectors. For each object query, a feedforward neural network generates sampling offsets, using its position vector as the initial coordinates. The initial sampling points of the model are generated in combination with the sampling offsets.

[0011] Step 6: Use the initial sampling point and its eight neighboring points in the sampling space to form a local sampling window, interpolate the points in the window to obtain the window features, and then flatten the window;

[0012] Step 7: The obtained feature matrix is fed into the feature fusion network CFFN. CFFN consists of a unidirectional parallel interaction structure (PSUI) and an inter-group self-attention layer, which fully integrates the features in the spatial and channel dimensions.

[0013] Step 8: The fully fused features are sent to the AdaptiveMixing decoding layer for feature decoupling;

[0014] Step 9: The final output of the decoding layer passes through two feed-forward neural networks (FFNs) to update the content vector and position vector of the object query. The content vector and position vector then pass through two FFNs to predict the category and position of the target to be detected.

[0015] Step 10: After the model training is completed, its accuracy can be verified, or the trained model file can be used to generate a detection frame based on the input test image to detect the category and position of the object to be detected in the test image. In step 4, the model uses the classic CNN backbone network ResNet50 to pre-extract features from the input image and obtain feature maps of four different scales. If the input image dimension is The output multi-scale feature maps are When constructing the 3D feature space, the number of channels of each scale feature map is normalized to the value d feat In step 5, the predicted vector object query consists of two parts: the content vector and the position vector, where the content vector is the coordinates (x, y, z, r) of the object query.

[0016] Step 6: Figure 2As shown, in the feature sampling space, eight neighboring points of the initial sampling point in step 5 and the initial sampling point itself are taken to form a local sampling window, and then the sampling points in the window are interpolated to obtain the sampling feature matrix x∈R G*W*P*C (Take an objectquery as an example.) Where G represents the sampling group, W and P represent the number of sampling windows and the number of points in the sampling window respectively, and C represents the number of feature channels. The formula is as follows:

[0017]

[0018] In the above formula, S is the size of the local window, i is the sampling point in the local window, Coordinate refers to the coordinate of the sampling point, and Interpolation is the interpolation operation.

[0019] In step 7, the CFFN contains a unidirectional interactive parallel structure (PSUI) and inter-group self-attention, where the PSUI consists of two left and right branches and a right-to-left unidirectional interactive network connecting them. The details of the PSUI are as follows: Figure 3 Shown are:

[0020] (1) The left branch performs inter-window self-attention to achieve local feature fusion. The V w The result of obtaining the semantic weight of the channel after the dot product of the feature matrix and factor, and Q w ,K w The feature matrix is obtained by the feedforward neural network; the local self-attention operation on different points in the window can fully integrate the local features between different neighboring points in the same window. The specific operation is as follows:

[0021] Q w ,K w =FFN1(x),FFN2(x),

[0022] V w =x⊙factor,

[0023]

[0024] In the above formula, Q w ,K w ,V w They are the three matrices in the self-attention operation, factor is the interaction factor containing channel weights generated by the single interaction network, d k is the scaling factor.

[0025] (2) The right branch first transforms the dimension of the local window feature matrix, and the transformed matrix is x∈R G *C*W*PThe next two dimensions form a feature map; the horizontal dimension represents the interior of a window, and the vertical dimension represents different windows. The right branch first performs a depthwise convolution with a 9x5 kernel to fuse features between windows and achieve global feature interaction. Pointwise convolution then achieves semantic fusion in the channel dimension. The final output of the right branch is converted to its original dimensions to facilitate concatenation with the left branch.

[0026] (3) The direction of the single interaction connection is from right to left. The depthwise convolution output of the right branch is operated by the interaction network to obtain a factor containing the channel semantic weight, which is input to the left branch to participate in the self-attention operation in the left branch.

[0027] (4) The final results of the left and right branches only retain the features of the initial sampling points in the P dimension, so the matrix dimension is transformed into x∈R G*W*C , after concat, it passes through a layer of FFN and the dimension remains x∈R G*W*C .

[0028] In step 7, CFFN includes a unidirectional interactive parallel structure (PSUI) and inter-group self-attention. The details of the inter-group self-attention are as follows: In order to reduce the amount of network computation and speed up network detection and training, this model will d feat Divide the matrix into four groups, and perform interpolation sampling on each group to reduce the size of one dimension. To compensate for the lack of interaction between channels in different groups caused by this operation, the model designs component self-attention to fuse the channel features between different groups. The formula is as follows:

[0029] Q g ,K g =FFN3(x),FFN4(x),

[0030]

[0031] In the above formula, Q g ,K g are the Query and Key matrices in the self-attention operation, d k is the scaling factor. In step 9, the decoder output is first converted to the same dimension as the object query's content vector through a FFN, completing the content vector update. It is then converted to the same dimension as the position vector through another FFN, completing the position vector update. The detection head uses the content vector and position vector to predict the candidate box's category and position through different FFNs.

[0032] During the training process of the present invention, the model predicts candidate boxes of the same size from the initial image of the training input with a set of fixed number (N) of object queries, where N is usually much larger than the actual number of objects of interest in the image, so an additional special class label is used to indicate that no object was detected.

[0033] During the entire training process of this invention, the parallel interactive architecture object detection model adopts a one-to-one label assignment method. It is necessary to match each prediction box with the bounding box. The model uses the Hungarian algorithm to achieve the best bipartite matching between the real object and the predicted object, that is, to find the optimal matching method. Let the total matching cost L matc h Minimum:

[0034]

[0035] The above formula σ is the matching rule between ground truth and prediction box, θN represents the possible matching method, y is the groundtruth set, Is a set of N prediction boxes. If the number of boxes in y is less than N, use Fill. L match is the matching cost between the true value and a prediction with index σ(i), which includes the classification loss L cls And the prediction box loss L box , L box Also includes IoU loss L iou and l1 loss L1.

[0036] Each element y in the groundtruth set y i All by c i with b i Composition, where c i is the category of the target in the box, b i Is the position vector, which defines the center coordinates and size of the real box. For the prediction of index σ(i), the present invention defines it as belonging to category c i The probability of The predicted box is Then L matc h Expressed as:

[0037]

[0038] The loss function in the training process of the present invention is the Hungarian loss of all pairs in the above matching:

[0039]

[0040] in For the best match.

[0041] After the 12 epochs in step 10 are completed, the trained model is saved as a pt file. The trained model file can be used to verify the model accuracy and images.

[0042] The specific method for detecting images mentioned in step 10 is:

[0043] You can use the network to load a trained model file to detect objects in an image. Run the detect code, set the detection model to the trained pt file, and set the input image directory to the folder containing the images to be detected. Start detection by feeding the data to be detected into the trained model for image recognition and location. The model will output several predicted boxes containing the locations and categories of potential objects in the image.

[0044] Compared with the existing technology, the present invention has the following advantages. At the method level: First, the present invention does not use traditional CNN to further extract features, but instead obtains spatial features by constructing a 3D sampling space and then performing window sampling in it, thereby improving the richness of feature extraction in the spatial dimension and strengthening the positioning ability of the model; second, in order to improve the quality of feature fusion, the present invention designs a CFFN network structure to fuse the extracted features in the spatial and channel dimensions. The CFFN network contains convolution and self-attention operations, and different methods are used for feature fusion in different dimensions. This design greatly enriches the semantic information and contributes to improving the accuracy of the model; at the application level, the present invention obtains an AP accuracy of 43.0 in a training cycle of 12 epochs in the example, which is better than many detection methods. At the same time, the present invention abandons the prior knowledge of the traditional CNN detection network, and the training speed is also improved, thereby solving the problems of slow convergence speed and long training time in the target detection task, and improving the accuracy and detection speed of the target detection task. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a network structure diagram of the present invention;

[0046] Figure 2 Schematic diagram of the sampling local window of the present invention;

[0047] Figure 3 It is a schematic diagram of the PSUI structure of the present invention. DETAILED DESCRIPTION

[0048] The specific embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and embodiments of the present invention. The following embodiments are used to illustrate the present invention but are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative work are within the scope of protection of the present invention.

[0049] Example 1: In response to the current traditional two-stage target detection method, the present invention proposes a 2D image target detection method that realizes multi-dimensional feature fusion based on a parallel interactive architecture model.

[0050] This example uses the COCO2017 dataset as experimental data. With the help of data augmentation technology, it utilizes an end-to-end parallel interactive target detection model based on the backbone network ResNet50, 3D feature space window sampling method, feature fusion network CFFN, and a detection head consisting of two feedforward networks to achieve object positioning and classification.

[0051] Step 1: Prepare the COCO2017 dataset required for model training; configure the COCO2017 dataset on the server and put it into the training folder in the required format;

[0052] Step 1.1: Obtain the public dataset COCO2017 from the COCO official website and download the images and annotations of the official training and validation sets.

[0053] Step 1.2. The COCO2017 dataset consists of training set images and annotation files, and validation set images and annotation files. The training set images, validation set images, and annotations are placed in the train2017, val2017, and annotations folders, respectively.

[0054] Step 2: Build the model and configure the PyTorch deep learning environment required for training;

[0055] Step 2.1: Create a virtual environment for the project in Anaconda and install key packages required for the training environment, such as PyTorch 1.11.0. The training server uses an NVIDIA RTX 3090 GPU, an Ubuntu 20.04 operating system, CUDA version 11.3, and Python 3.8 as the compiler language.

[0056] Step 2.2: Install and configure the mmdetection framework and the required mmcv compilation package. In addition, install other dependent packages required for the training script.

[0057] Step 3: Set the training hyperparameters and input the dataset into the parallel interactive architecture end-to-end object detection model for training;

[0058] Step 3.1: The training hyperparameters are: feature extraction backbone is ResNet50, initial learning rate is 0.000125, batch size is 4, and number of epochs is 12.

[0059] Step 4: The model feeds the input image into ResNet50 for feature extraction, outputs a multi-scale feature map, and then constructs a 3D feature sampling space from the multi-scale feature map;

[0060] Step 5: Generate a set of object queries containing prediction vectors, including content vectors and position vectors. For each object query, a feedforward neural network generates sampling offsets, using its position vector as the initial coordinates. The initial sampling points of the model are generated in combination with the sampling offsets.

[0061] Step 6: Use the initial sampling point and its eight neighboring points in the sampling space to form a local sampling window, interpolate the points in the window to obtain the window features, and then flatten the window;

[0062] Step 7: The obtained feature matrix is fed into the feature fusion network CFFN. CFFN consists of a unidirectional parallel interaction structure (PSUI) and an inter-group self-attention layer, which fully integrates the features in the spatial and channel dimensions.

[0063] Step 8: The fully fused features are sent to the Adaptive Mixing decoding layer for feature decoupling;

[0064] Step 9: The final output of the decoding layer passes through two feed-forward neural networks (FFNs) to update the content vector and position vector of the object query. The content vector and position vector then pass through two FFNs to predict the category and position of the target to be detected.

[0065] Step 10: After the model is trained, you can verify its accuracy. You can also use the trained model file to generate a detection frame based on the input test image to detect the category and location of the object to be detected in the test image.

[0066] After step 10.1 and 12 epochs are completed, the trained model is saved as a pt file. You can use val.py to verify the accuracy of the model based on the training weights. Input the val2017 dataset mentioned in step 1 into the model to evaluate the accuracy of the trained model. The accuracy indicators are AP, AP 50 、AP 75 、AP S 、AP M and AP L6. The final verification result of this embodiment has an AP of 43.0, which is better than most methods under the same experimental conditions.

[0067] In step 10.2, you can use the detection network to load the trained model file to detect objects in the image. Run the detect code, set the detection model to the trained pt file, and set the input image directory to the folder containing the images to be detected. Once these settings are complete, you can start detection. Input the data to be detected into the trained model for image recognition and location. The model will output several predicted boxes containing the location and category of potential objects in the image.

Claims

1. A target detection method based on a parallel interactive architecture model to achieve multi-dimensional feature fusion, characterized in that: The following steps are involved: Step 1: Prepare the COCO2017 dataset required for model training; Configure the COCO2017 dataset on the server and put it into the training folder in the required format; Step 2: Build a model under the mmdetection framework and configure the PyTorch deep learning environment required for training; Step 3: Set the training hyperparameters and input the dataset into the parallel interactive architecture end-to-end object detection model for training; Step 4: The model feeds the input image into ResNet50 for feature extraction, outputs a multi-scale feature map, and then constructs a 3D feature sampling space from the multi-scale feature map; Step 5: Generate a set of object queries containing prediction vectors, including content vectors and position vectors. For each object query, the object query generates sampling offsets through a feedforward neural network, using its position vector as the initial coordinates. Combined with the sampling offsets, the initial sampling points of the model are generated. Step 6: Use the initial sampling point and its eight neighboring points in the sampling space to form a local sampling window, interpolate the points in the window to obtain the window features, and then flatten the window; Step 7: The obtained feature matrix is fed into the feature fusion network CFFN. CFFN consists of a unidirectional parallel interaction structure (PSUI) and an inter-group self-attention layer, which fully integrates the features in the spatial and channel dimensions. Step 8: The fully fused features are sent to the AdaptiveMixing decoding layer for feature decoupling; Step 9: The final output of the decoding layer passes through two feed-forward neural networks (FFNs) to update the content vector and position vector of the object query. The content vector and position vector then pass through two FFNs to predict the category and position of the target to be detected. Step 10: After the model is trained, you can verify its accuracy. You can also use the trained model file to generate a detection frame based on the input test image to detect the category and location of the object to be detected in the test image.

2. The target detection method for realizing multi-dimensional feature fusion based on a parallel interactive architecture model according to claim 1, characterized in that: Step 4: This target detection model uses the classic CNN backbone network ResNet50 to pre-extract features from the input image and obtains feature maps of four different scales. If the input image dimension is The output multi-scale feature maps are In addition, when constructing the 3D feature space, the number of channels of each scale feature map is normalized to a unified value d feat , the above D i is the number of feature channels of the feature map of the i-th layer, H0 and W0 are the height and width of the input image.

3. The target detection method based on the parallel interactive architecture model for realizing multi-dimensional feature fusion according to claim 1, characterized in that: Step 5: The predicted vector object query consists of two parts: a content vector and a position vector, wherein the content vector represents the initial coordinates (x, y, z, r) of the object query, where r is the aspect ratio.

4. The target detection method for realizing multi-dimensional feature fusion based on a parallel interactive architecture model according to claim 1, characterized in that: Step 6: In the feature sampling space, eight neighboring points of the initial sampling point in step 5 and the initial sampling point itself are taken to form a local sampling window, and then the sampling points in the window are interpolated to obtain the sampling feature matrix x∈R G*W*P*C , where G represents the sampling group, W and P represent the number of sampling windows and the number of points in the sampling window respectively, and C represents the number of feature channels.

5. The target detection method for realizing multi-dimensional feature fusion based on a parallel interactive architecture model according to claim 1, characterized in that: Step 7: CFFN includes a unidirectional interactive parallel structure (PSUI) and inter-group self-attention, the details of which are as follows: (1) CFFN contains a unidirectional interactive parallel structure (PSUI) and inter-group self-attention, where the PSUI consists of two left and right branches and a right-to-left unidirectional interactive network connecting them; (2) The left branch of CFFN uses the self-attention operation between windows to realize local feature fusion. The V w The result of obtaining the semantic weight of the channel after the dot product of the window feature matrix and factor, and Q w ,K w The feature matrix is obtained by passing different feedforward neural networks; the local self-attention operation on different points in the window can fully integrate the local features between different neighboring points in the same window; (3) The right branch first transforms the dimension of the local window feature matrix, and the transformed matrix is x∈R G*C*W*P , and the next two dimensions form a feature map; the horizontal dimension represents the interior of a window, and the vertical dimension represents different windows. The right branch first performs depthwise convolution on it using a 9X5 convolution kernel to fuse the features between windows and realize global feature interaction; then the semantic fusion in the channel dimension is realized through pointwise convolution. The final output of the right branch will be converted to the original dimension to be concat with the left branch; (4) The direction of the single interaction connection is from right to left. The depthwise convolution output of the right branch is processed by the interaction network to obtain a factor containing the channel semantic weight, which is input to the left branch to participate in the self-attention operation in the left branch. (5) The final results of the left and right branches only retain the features of the initial sampling points in the P dimension, so the matrix dimension is transformed into x∈R G*W*C , after concat, it passes through a layer of FFN and the dimension remains x∈R G*W*C ; (6) In order to reduce the amount of network computing power and speed up network detection and training, this model will use d feat The model is divided into four groups, and each group performs interpolation sampling to reduce the size of the matrix in a certain dimension. In order to compensate for the lack of interaction between channels of different groups caused by this operation, the model designs component self-attention to fuse the channel features between different groups.

6. The target detection method for realizing multi-dimensional feature fusion based on a parallel interactive architecture model according to claim 1, characterized in that: Step 9: (1) The decoder output is first converted to the same dimension as the object query’s content vector through an FFN, completing the update of the content vector; then it is converted to the same dimension as the position vector through an FFN, completing the update of the position vector; (2) The detection head uses the content vector and position vector to predict the category and position of the candidate box through different FFNs.

Images