A progressive assisted target detection method and system based on CNN and ViT fusion

Abstract

The present invention provides a progressive auxiliary target detection method and system based on the fusion of CNN and ViT. The method includes: fusing the CNN and ViT structures in series; Pao detection consists of a feature extraction network (CNN), an RPN structure for separating foreground and background, and three ViT structures for inter-target modeling. The CNN first extracts target features and reduces the resolution of the feature map; the RPN structure then extracts regions of interest at different scale levels from the feature pyramid output by the CNN, and inputs their corresponding features into the three layers of ViT in a progressive structure for inter-target modeling. The correlation attention map is used to achieve feature interaction and supplementation, thereby improving feature effectiveness and thus improving detection effect; and a relaxed boundary calculation method is used to dynamically calculate the target scale clustering of different images in each batch, adding a soft boundary for target partitioning for supervised learning. The present invention solves the technical problems of poor fusion effect, increased network load, and false detection and missed detection.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a progressive auxiliary target detection method and system based on CNN and ViT fusion. Background Art

[0002] Object detection is a key area of computer vision research and can serve a wide range of applications, such as autonomous driving, intelligent video surveillance, and remote sensing. In recent years, due to the vigorous development of deep convolutional networks, a number of excellent detectors have been proposed, such as SSD, Faster R-CNN, Retinanet, FPN, Mask R-CNN, and Cascade R-CNN.

[0003] The use of transformers for object detection in computer vision originated with ViT (vision transformer). ViT, such as the Swin Transformer, is a new object detection method that can model global pixels in a relatively large area to separate the target area for object detection. However, the computational complexity is huge and too many background pixels are introduced. Naturally, the present invention conceived that if VIT only calculated the target area, the computational complexity would be greatly reduced, and the interference of background pixels would be removed, thereby improving the accuracy of object detection. The mainstream method for object detection, CNN, extracts and collects local features in a layered manner, which can separate foreground (target) and background pixels within a certain image range. The obtained region of interest features have a high degree of semantic information. Not surprisingly, feeding the target features extracted by CNN into VIT can greatly reduce the computational cost of VIT, thereby achieving the effect of improving performance. Therefore, various networks combining CNN and transformer have been proposed. For example, Conformer improves detection capabilities by utilizing the interaction of complementary detection features of transformer and CNN. Mobile-Former (University of Science and Technology of China and Huawei) achieves bidirectional fusion of local and global features, significantly outperforming lightweight networks such as MobileNetV3 in classification and downstream tasks. (Microsoft) CoTNet innovatively integrates the dynamic contextual information aggregation of the Transformer's self-attention mechanism with the static contextual information aggregation of convolutions, proposing a novel Transformer-style "plug-and-play" CoT module that can directly replace the 3×3 convolutions in the existing ResNet architecture's Bottleneck, achieving significant performance improvements. (JD AI) VitDet removes the PFN portion of detection and directly uses a single-scale feature map, upsampling and downsampling it to obtain four scales, achieving the same performance with minimal computation. Encouragingly, these detection networks all show varying degrees of improvement over their baselines. However, most fusion networks bidirectionally fuse local and global features, which in turn increases the load on both networks, making this fusion approach suboptimal. Furthermore, CNNs at different levels extract different receptive fields and target features, making direct fusion inappropriate.The existing invention patent application document with publication number CN114359756A, "A method for rapid and intelligent detection of damaged houses in post-earthquake drone remote sensing images," includes the following steps: A. Collecting drone remote sensing image samples and establishing a damaged house sample set; B. Based on the YOLOv5 deep learning model, firstly, introducing the Vision Transformer structure to optimize the Backbone network and improve detection accuracy; secondly, replacing the PANet network with BiFPN based on the multi-scale characteristics of the target to enhance the multi-scale information fusion capability; and then constructing the YOLOv5s-ViT-BiFPN model; C. Crop the input drone remote sensing image and record its geographic location coding information; D. Based on the model, perform damaged house detection on the cropped image to obtain a target detection frame; E. Restore and vectorize the geographic location information of the target detection frame to obtain a vector diagram of the damaged house detection frame. The specification of this existing document indicates that this solution replaces PANet with BiFPN for multi-scale fusion. However, this technology utilizes global feature learning and multi-scale information fusion, which increases the overall network load and reduces the robustness of the algorithm. Furthermore, this existing technology utilizes vectorization technology, which increases the algorithmic complexity. The existing invention patent application, publication number CN114842316A, entitled "A Real-Time Object Detection Method Combining Convolutional Neural Networks and Transformer Networks," includes the following steps: S1: Input image data; S2: Pass the image through a convolutional neural network backbone, imparting inductive bias to the extracted features; S3: Design a neck detection network to transition between the backbone and head detection networks, providing high-resolution and semantic features for the head detection network; S4: Design a head detection network, introducing a Transformer into the head network to construct multiple long-range dependencies between the generated local features and characterize the object categories and coordinates present in the image; S5: Design a nonlinear combination method to reduce false negatives and improve the detection model's ability to capture objects; and S6: Perform detection on a natural dataset. It can be seen from the specific embodiments of the prior art that the existing solution splices the feature data after difference with the feature data of the second-order network layer in the channel dimension; the second-order network layer provides high-resolution underlying information, and the feature fusion part completes the feature fusion of the spliced data through depth-wise separable convolution, that is, the second-order, third-order and fourth-order network layer data features are fused. Since in the actual application of the network model, the receptive fields extracted by CNN networks at different levels are different in size, and the target features are also different, the method of directly fusing the features of each layer, like the aforementioned existing solution, is likely to reduce the prediction effect of the overall model.

[0004] like Figure 1a and Figure 1b As shown, for the prior art of single shooting angle ( Figure 1a), smaller targets ( Figure 1b ) results in limited feature content, and problems such as false detection and missed detection often occur.

[0005] In summary, the existing technology has technical problems such as poor fusion effect, increased network load, and false detection and missed detection. Summary of the Invention

[0006] The technical problem to be solved by the present invention is how to solve the technical problems of poor fusion effect, increased network load, and false detection and missed detection in the prior art.

[0007] The present invention solves the above technical problems by adopting the following technical solutions: a progressive auxiliary target detection method based on CNN and ViT fusion includes:

[0008] S1. Construct a Pao detection model, wherein the Pao detection model includes: a feature extraction backbone network backbone, a detection neck, and a detection head. The feature extraction backbone network backbone is used to extract target features. The detection neck fuses the difference layer feature map up and down according to the target features to obtain a semantic position information difference resolution feature map. The difference layer feature map is binary classified and regressed to obtain no less than 2 (several sparse) regions of interest. Inter-target modeling is performed on regions of interest at different scale levels to interact and supplement target features to obtain ViT output results. The detection head classifies and regresses the ViT output data to obtain ViT prediction results.

[0009] S2. Construct a progressive inter-target auxiliary detection model to perform inter-target modeling and feature alignment on the large-resolution RoI features using the ViT-based region of interest modeling method to obtain aligned large-resolution RoI features and aligned medium-resolution RoI features, and perform inter-target modeling and feature alignment based on them. Use the aligned large-resolution RoI features and aligned medium-resolution RoI features with the small-resolution RoI features to perform inter-target modeling and feature alignment;

[0010] S3, the foreground target output by each layer of the RPN network of the detection neck in the Pao detection model and the ViT output results are trained according to the simOTA dynamic positive and negative sample allocation strategy;

[0011] S4. Calculate the classification loss and regression loss of each layer of RPN network output and ViT output data using the Pao detection model to obtain the joint probability loss;

[0012] S5. Design an area scaling function to cluster the target areas in the preset data set according to the length-width scale factor, and divide all targets into at least two parts according to the scaled target area to obtain at least two cluster centers to obtain dynamic clustering results. Based on this, the target scale boundaries of each image are reallocated to use large targets to assist in the detection of small and medium targets.

[0013] The present invention uses a CNN network to extract target features, and the obtained feature pyramid is separated from the foreground and background through the RPN to obtain the region of interest and its features; finally, the ViT network is used to perform inter-target modeling on the regions of interest of different scales in turn, thereby achieving high-precision detection with low computational complexity.

[0014] The present invention extracts primary features and separates regions of interest (ROIs) in the early stages of CNN. ViT then progressively stitches and models these regions of interest at different scales, enabling large targets to assist in the detection of medium and small targets. This progressive modeling approach, which prioritizes large targets over small ones, interacts and enhances features between targets, enabling large targets to assist in the detection of small ones. This approach is highly versatile and integrates CNN and ViT architectures in series, taking into account global information while simultaneously addressing local foreground responses. This enhances the model's target detection capabilities, making it particularly effective for small targets.

[0015] The present invention finds strong correlations between targets of the same scale and targets of different scales in the same scene. It can establish an attention heat map between targets within the entire image range and use their mutual relationships to supplement features and assist in recognition, which is of great significance for some fuzzy targets and small targets with limited features.

[0016] In a more specific technical solution, step S4 includes:

[0017] S41. Using the Focal loss function, a loss function balancing the difficulty and ease of samples, the difference between the joint probability of the RPN predicted category and the joint probability of the ViT predicted category and the target true label value is calculated to obtain the classification loss.

[0018] S42. Use the angle loss function SIOU to obtain and select the angle loss, redefine the distance loss and shape loss according to the angle loss, and use the preset logic to obtain the IoU loss. Use the first layer RPN and ViT of the Pao detection model to classify and regress the minimum resolution feature map to predict the large target information, obtain the regression result and the true value of the large target, and calculate the SIOU loss based on them.

[0019] S43. Use the second-layer RPN regression to process the medium-resolution feature map to obtain the medium-target foreground feature and splice it with the first-layer ViT output. The foreground feature is input into the second-layer ViT module for target-level modeling to obtain the small-medium target prediction result, and the regression loss is obtained by using the true value calculation.

[0020] S44. Use the high-resolution feature map in the third-layer RPN regression feature pyramid to obtain the foreground features in front of the small target. Concatenate the foreground features in front of the small target with the second-layer ViT output and input them into the third-layer ViT module for target-level modeling to obtain all target prediction results, the classification loss and regression loss of each layer output, and use the following logic to sum them to obtain the total loss of the entire network training:

[0021] L=W box L box +W cls L cls .

[0022] Where, L cls With L box Represents classification and regression losses respectively, w cls With w box is the weight factor of the two losses (which can be set in a ratio of 1:4, for example).

[0023] In a more specific technical solution, the classification loss in step S41 uses the following logic to calculate the difference between the joint probability of the RPN predicted category, the joint probability of the ViT predicted category, and the target true label value:

[0024] FL(p t )=-(1-p t ) γ log(p t )

[0025]

[0026] Among them, p t is the network prediction probability, (1-p t ) γ is the modular factor. For the accurately classified sample p t →1, the modularity factor approaches 0, and y represents the true category.

[0027] The focal loss used in this invention is equivalent to increasing the weight of difficult-to-distinguish samples in the loss function, making the loss function tend to favor difficult-to-distinguish samples, which helps improve the accuracy of difficult-to-distinguish samples. For positive samples (y=1), the present invention will tilt resources toward difficult-to-distinguish samples. The confidence of negative samples is low, and this weighting will further reduce it, thereby more completely isolating the interference of an excessive number of negative samples (most of the image is background).

[0028] In a more specific technical solution, step S42 includes:

[0029] S421, using the angle loss function SIOU to obtain and select the angle loss according to the following logic:

[0030]

[0031] Among them, IoU is the intersection-over-union loss, Δ is the distance loss, and Ω is the shape loss;

[0032] S422. Redefine the distance loss based on the angle loss using the following logic:

[0033]

[0034]

[0035]

[0036]

[0037]

[0038] in: γ is the relaxation factor; Λ is the angle loss, which is calculated from x, where x is the sine of the angle between the line connecting the midpoints of the prediction box and the target box and the horizontal line, α is the angle, σ is the distance between the two boxes, and c h is the height difference between the two frames. Δ is the distance loss, ρ x is the horizontal normalized Euclidean distance between the two boxes, ρ y The normalized Euclidean distance between the two boxes is calculated vertically. The goal of angle loss is to increase the loss the closer the two boxes are to a diagonal line, and decrease the loss the closer they are to a straight line. The Λ function precisely meets this requirement. Distance loss aims to increase the loss the further the distance. A relaxation factor is added to account for angle, so the closer the boxes are to a diagonal line, the greater the loss. The greater the horizontal or vertical distance, the greater the distance loss.

[0039] S423. Redefine the distance loss based on the angle loss using the following logic:

[0040]

[0041] in:

[0042]

[0043] θ is the shape attention parameter;

[0044] S424. Define the IoU loss using the following logic, where B is the predicted box, B GT is the true label box:

[0045]

[0046] S425. Use the following logic to classify and regress the minimum resolution feature map using the first layer RPN and ViT of the Pao detection model to predict large target information, obtain the regression result and the true value of the large target, and calculate the GIoU loss based on them:

[0047] L 1(reg) =L large .

[0048] In a more specific technical solution, step S43 uses the following logic to calculate the regression loss. large 、L middle 、L small Represents the regression losses of three types of targets: large, medium and small:

[0049] L 2(reg) =L large +L middle

[0050] L 3(reg) =L large +L middle +L small .

[0051] The three-layer ViT architecture of the present invention does not model large, medium, and small objects separately. Instead, it integrates the features of the next-level scale objects based on the modeling and alignment of the previous step to perform target modeling and feature alignment. The progressive fusion detection method adopted by the present invention ultimately helps the model achieve auxiliary detection between objects of the same scale and between objects of different scales, improving the applicability of the system.

[0052] In a more specific technical solution, step S5 includes:

[0053] S51, designing an area scaling function with a preset logic;

[0054] S52, clustering the targets in the preset data set according to the scaled target areas, dividing all targets into at least two parts according to the scaled target areas using a K-means algorithm, so as to obtain at least two cluster centers to obtain a dynamic clustering result;

[0055] S53. Reassign the object scale division boundary of each image according to the dynamic clustering result, so as to utilize the large object to assist in detecting small and medium objects.

[0056] The present invention dynamically adjusts the targets in each image so that the thresholds for dividing large, medium and small targets are in a relatively small range of dynamic changes, allowing the feature maps corresponding to the two adjacent categories to predict them, enhancing the model's response to boundary values and being more conducive to training convergence.

[0057] In a more specific technical solution, in step S51, the following logic design area scaling function is used:

[0058]

[0059] Among them, S ori is the original area, S is the area of the target after scaling, w is the width, and h is the height.

[0060] In a more specific technical solution, in step S52, the K-means algorithm is used to divide all objects into at least two parts according to the scaled object area using the following logic:

[0061]

[0062] Where n is the number of targets in the dataset.

[0063] In a more specific technical solution, in step S53, based on the dynamic clustering results, the target scale boundaries are reallocated for each image using the following logic:

[0064]

[0065] Among them, γ is the relaxation factor. In each batch, three clustering is performed separately for a single image, and the relaxation factor is added to the preset threshold according to the dynamic clustering result.

[0066] In each batch during training, the present invention will perform tri-clustering for a single image separately, and add a relaxation variable to the pre-calculated threshold according to the dynamic clustering result. By relaxing the boundary division, the target at the boundary point can be unrestricted to a certain fixed scale range, thereby relaxing the boundary restriction.

[0067] The feature maps of a neural network layer can fit objects at adjacent scales outside the scale that layer is responsible for predicting, thus achieving better generalization performance. Repeated predictions are removed in the subsequent NMS operation, thus ensuring high precision and high recall.

[0068] In a more specific technical solution, a progressive assisted object detection system based on CNN and ViT fusion includes:

[0069] Pao detection model construction module, used to construct Pao detection model, wherein the Pao detection model includes: feature extraction backbone network backbone, detection neck, detection head, using the feature extraction backbone network backbone to extract target features, using the detection neck to fuse the difference layer feature map up and down according to the target features to obtain the semantic position information difference resolution feature map, binary classification and regression processing of the difference layer feature map, thereby obtaining no less than 2 regions of interest, and performing target inter-target modeling on regions of interest at different scale levels to interact and supplement target features to obtain ViT output results, and using the detection head to classify and regress ViT output data to obtain ViT prediction results;

[0070] A progressive inter-target auxiliary detection module is used to construct a progressive inter-target auxiliary detection model, to use the ViT-based region of interest modeling method to perform inter-target modeling and feature alignment on the large-resolution RoI features, so as to obtain aligned large-resolution RoI features and aligned medium-resolution RoI features, and to perform inter-target modeling and feature alignment based on them, and to use the aligned large-resolution RoI features and aligned medium-resolution RoI features with small-resolution RoI features to perform inter-target modeling and feature alignment, so as to use CSPNet to extract the region of interest from the difference-scale feature map, and the progressive inter-target auxiliary detection module is connected to the Paodetection model construction module;

[0071] The training module is used to train the foreground targets and ViT output results of each layer of the RPN network in the Pao detection model according to the simOTA dynamic positive and negative sample allocation strategy. The training module is connected to the Pao detection model construction module;

[0072] The joint probability loss processing module is used to calculate the classification loss and regression loss of each layer of RPN network output and ViT output data using the Pao detection model to obtain the joint probability loss. The joint probability loss processing module is connected to the training module and the Pao detection model construction module;

[0073] The label scale dynamic division module is used to design an area scaling function, cluster the targets in the preset data set according to the scaled target area, divide all targets into no less than two parts according to the scaled target area, and obtain no less than two cluster centers to obtain dynamic clustering results. Based on this, the target scale division boundaries of each image are reallocated to use large targets to assist in the detection of small and medium targets. The label scale dynamic division module is connected to the Pao detection model construction module and the progressive inter-target auxiliary detection module.

[0074] Compared with the existing technology, the present invention has the following advantages: the present invention uses the CNN network to extract target features, and the obtained feature pyramid is separated from the foreground and background through the RPN to obtain the region of interest and its features; finally, the ViT network is used to sequentially perform inter-target modeling on regions of interest of different scales, thereby achieving high detection accuracy and low computational complexity.

[0075] The present invention extracts primary features and separates regions of interest (ROIs) in the early stages of CNN. ViT then progressively stitches and models these regions of interest at different scales, enabling large targets to assist in the detection of medium and small targets. This progressive modeling approach, which prioritizes large targets over small ones, interacts and enhances features between targets, enabling large targets to assist in the detection of small ones. This approach is highly versatile and integrates CNN and ViT architectures in series, taking into account global information while simultaneously addressing local foreground responses. This enhances the model's target detection capabilities, making it particularly effective for small targets.

[0076] The present invention finds strong correlations between targets of the same scale and targets of different scales in the same scene. It can establish an attention heat map between targets within the entire image range and use their mutual relationships to supplement features and assist in recognition, which is of great significance for some fuzzy targets and small targets with limited features.

[0077] The focal loss used in this invention is equivalent to increasing the weight of difficult-to-distinguish samples in the loss function, making the loss function tend to favor difficult-to-distinguish samples, which helps improve the accuracy of difficult-to-distinguish samples. For positive samples (y=1), the present invention will tilt resources toward difficult-to-distinguish samples. The confidence of negative samples is low, and this weighting will further reduce it, thereby more completely isolating the interference of an excessive number of negative samples (most of the image is background).

[0078] The three-layer ViT architecture of the present invention does not model large, medium, and small objects separately. Instead, it integrates the features of the next-level scale objects based on the modeling and alignment of the previous step to perform target modeling and feature alignment. The progressive fusion detection method adopted by the present invention ultimately helps the model achieve auxiliary detection between objects of the same scale and between objects of different scales, improving the applicability of the system.

[0079] The present invention dynamically adjusts the targets in each image so that the thresholds for dividing large, medium and small targets are in a relatively small range of dynamic changes, allowing the feature maps corresponding to the two adjacent categories to predict them, enhancing the model's response to boundary values and being more conducive to training convergence.

[0080] In each batch during training, the present invention will perform tri-clustering for a single image separately, and add a relaxation variable to the pre-calculated threshold according to the dynamic clustering result. By relaxing the boundary division, the target at the boundary point can be unrestricted to a certain fixed scale range, thereby relaxing the boundary restriction.

[0081] The feature maps of a neural network layer in this invention can fit objects at adjacent scales outside the scale predicted by that layer, thereby improving generalization performance. Duplicate predictions are removed in subsequent NMS operations, ensuring high precision and recall. This invention addresses the technical issues of poor fusion, increased network load, and false positives and missed detections in existing technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1a This is a schematic diagram of the first error detection and missed detection effect of the prior art;

[0083] Figure 1b This is a schematic diagram of the second error detection and missed detection effect of the prior art;

[0084] Figure 2 Schematic diagram of the structure of Pao detection in a progressive auxiliary target detection method based on CNN and ViT fusion according to Example 1 of the present invention;

[0085] Figure 3 The original image of the large target assisting the small target in Example 1 of the present invention;

[0086] Figure 4 This is a schematic diagram of the anchor change process of SimOTA in Example 1 of the present invention;

[0087] Figure 5a This is a schematic diagram of selecting the first anchor point in Example 1 of the present invention;

[0088] Figure 5b This is a schematic diagram of selecting the second anchor point in Example 1 of the present invention;

[0089] Figure 5c This is a schematic diagram of selecting the third anchor point in Example 1 of the present invention;

[0090] Figure 6 Schematic diagram of the prediction process and training process of Example 1 of the present invention;

[0091] Figure 7 This is a schematic diagram of the angle loss selection process in Example 1 of the present invention;

[0092] Figure 8 This is a function curve diagram of Angle cost in Example 1 of the present invention;

[0093] Figure 9 This is a schematic diagram of the area scaling function trend of Example 1 of the present invention;

[0094] Figure 10 This is a schematic diagram of adding a relaxation factor to the classification standard according to Example 1 of the present invention;

[0095] Figure 11a This is a schematic diagram of the attention layer of ViT in Example 2 of the present invention;

[0096] Figure 11b Schematic diagram of the attention layer 2 of ViT according to embodiment 2 of the present invention;

[0097] Figure 11c This is a schematic diagram of the attention layer 3 of ViT according to Example 2 of the present invention;

[0098] Figure 12 This is a diagram showing the results of Pao detection on the UAVDT dataset according to Example 2 of the present invention;

[0099] Figure 13 This is a graph showing the change in target density of ViT and Pao detection on the UAVDiT dataset according to Example 2 of the present invention;

[0100] Figure 14a This is a first comparison effect diagram of YOLOX and YOLOX-Pao detection according to Example 2 of the present invention;

[0101] Figure 14b This is a second comparison diagram of YOLOX and YOLOX-Pao detection according to Example 2 of the present invention;

[0102] Figure 14c This is a third comparison effect diagram of YOLOX and YOLOX-Pao detection according to Example 2 of the present invention;

[0103] Figure 14d This is the fourth comparison effect diagram of YOLOX and YOLOX-Pao detection in Example 2 of the present invention. DETAILED DESCRIPTION

[0104] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0105] Example 1

[0106] The present invention provides a progressive auxiliary target detection method based on the fusion of CNN and ViT to build a target detection network in series: CNN extracts primary features in the early stage and separates the region of interest; ViT performs progressive splicing and inter-target modeling on regions of interest of different scales to achieve large targets to assist the detection of medium and small targets. On the widely tested MS-COCO dataset, the present invention uses various backbones to combine with the Pao detection of the present invention and compare with its baseline network. The results all achieved an increase of 2%-4%. Finally, after using the YOLOX backbone, a new result was obtained on the COCO dataset, that is, the Map was 55.7%.

[0107] 1. Model Introduction

[0108] This section explains Pao detection. First, we describe its architecture in Section 1.1, and then describe the inter-object auxiliary detection model in Section 1.2. In Section 1.3, we describe the loss function of ViT.

[0109] 1.1. Architecture of Pao detection

[0110] like Figure 2As shown in the figure, the Pao detection architecture consists of three parts: a feature extraction backbone network, a detection neck, and a detection head. The feature extraction backbone network can adopt a classic CNN such as ResNet or CSPNet. Due to its parameter sharing and residual linking characteristics, it can quickly extract target features from shallow to deep. The detection neck includes two parts: a feature pyramid network (FPN), a RPN, and a ViT module. The FPN is responsible for fusing feature maps from different layers, thereby obtaining feature maps of different resolutions that take into account both semantics and positional information. The RPN is used to perform binary classification and regression on the multi-layer feature maps to obtain several regions of interest. The ViT module is responsible for modeling the inter-object regions of interest at different scales, realizing feature interaction and complementation. The detection head performs the final classification and regression operations on the ViT module output to obtain the final prediction result. The present invention believes that there is a strong correlation between objects of the same scale and objects of different scales in the same scene. It is possible to establish an attention heat map between objects across the entire image, and use these relationships to supplement features and assist in recognition. This is of great significance for fuzzy objects with limited features and small objects.

[0111] 2.2. Progressive Inter-object Auxiliary Detection Model from Large to Small

[0112] like Figure 3 As shown in the figure, small target detection is often a key challenge in previous detection models. Compared to large targets, small targets have fewer features and limited information available for identification and localization. Therefore, it is imperative to utilize surrounding and global targets for auxiliary identification and localization to improve detection effectiveness. In target detection, large targets often have higher detection accuracy, and in the same scene, large and small targets are highly correlated. Highly accurate large target features can be used to supplement small target features and assist in small target detection.

[0113] Commonly used object detectors can only detect the main body of the woman, leaving the person on the beach behind her unrecognizable. This paper designs a progressive inter-object auxiliary detection model that uses CSPNet to extract regions of interest (ROIs) from feature maps at different scales. First, a ViT-based ROI modeling approach is used to perform inter-object modeling and feature alignment on large-resolution RoI features. Subsequently, the aligned large-resolution RoI features are used with medium-resolution RoI features for further inter-object modeling and feature alignment. Finally, the aligned large- and medium-resolution RoI features are used with the small-resolution RoI features for further inter-object modeling and feature alignment. The specific forward (inference) process is as follows: first, use the backbone of CNN to extract features and obtain a feature pyramid; then use the RPN network to perform the first classification (binary classification of foreground and background) and regression, extract the regions of interest in feature maps of different resolutions, and extract the corresponding feature blocks from the corresponding feature maps for normalization and scaling, and finally obtain the region of interest features of the same dimension and their foreground scores; then, use the ViT structure to perform inter-target modeling and feature interaction on the region of interest of the small-resolution feature map (corresponding to the large target), and splice the output result with the region of interest of the medium-resolution feature map (corresponding to the medium target), and input the second layer of ViT structure for inter-target modeling and feature interaction to achieve feature enhancement of large targets for medium targets. The output result is also spliced with the region of interest corresponding to the large-resolution feature map (corresponding to the small target) and input the third layer of ViT structure, thereby achieving progressive feature supplementation, allowing large-scale targets with higher detection accuracy to assist small target detection. Finally, the three-stage ViT output features are classified and regressed for the second time, and the classification score is multiplied by the first foreground and background classification to obtain the final target category and location information. The process is as follows Figure 2 Through this progressive alignment and modeling approach, large objects with rich features are used to assist in the recognition of small and medium-sized objects, achieving good results.

[0114] 2.3 Training Phase

[0115] like Figure 4 As shown in the figure, during the training phase, the foreground target output by each layer of the RPN network and the ViT output prediction results are allocated according to the simOTA dynamic positive and negative sample strategy.

[0116] As shown in Figure 5, the specific process is as follows: based on the existing anchor and gt, gt is defined as a supplier that can provide positive samples, anchor is defined as a demander who needs a sample, and background is defined as a supplier that provides negative samples. The distance between them is loss. After continuous iteration, the shortest path is found to select the true anchor.

[0117] like Figure 6As shown, the loss is calculated and finally, the three-layer loss is added together to get the total loss, and the network parameters are updated according to the gradient descent method.

[0118] 2.4 Loss Function

[0119] The model calculates the joint probability loss for the RPN output and the ViT output. The loss is composed of the weighted classification and regression losses, which are expressed as follows:

[0120] L=W box L box +W cls L cls

[0121] The classification loss uses the Focal loss, which is difficult to balance samples, as the loss function to calculate the difference between the joint probability of the predicted category output by RPN and ViT and the target true label value. The formula is as follows:

[0122] FL(p t )=-(1-p t ) γ log(p t )

[0123] where (1-p t ) γ It is called modulating factor. For the sample p that is accurately classified t →1, the modulating factor approaches 0. For samples with inaccurate classification (1-p t )→1, the modulating factor approaches 1. Compared with the cross entropy loss, the focal loss does not change the loss for inaccurately classified samples, but reduces the loss for accurately classified samples. Overall, it is equivalent to increasing the weight of inaccurately classified samples in the loss function. At the same time, p t It also reflects the difficulty of classification, p t The larger the value, the higher the confidence of the classification, the more similar it is to the true category y, that is, the ground truth, which means that the sample is easier to classify and the classification is more accurate; t The smaller it is, the lower the confidence of the classification is, which means the sample is more difficult to classify, where y is an adjustable factor and y>0. Therefore, focal loss is equivalent to increasing the weight of difficult samples in the loss function, making the loss function tend to difficult samples, which helps to improve the accuracy of difficult samples. The main goal of focal loss is to increase the importance of difficult-to-classify samples. The judgment standard of difficult-to-classify samples is that for positive samples, the smaller the confidence, the more difficult it is to classify. Increasing the importance means increasing the weight of the loss, which increases the calculation. t The expression is as follows:

[0124]

[0125] For positive samples (y = 1 ) Focal loss will tilt resources towards difficult-to-distinguish samples, and the confidence of negative samples themselves is small, which will be further reduced after this weighting, thereby more completely isolating the interference of too many negative samples (most of the image is background).

[0126] The regression loss function uses the SIOU loss function with angle loss. The goal of the regression loss function is to ideally match the predicted box with the ground-truth box. Mismatch is judged based on three criteria: the distance between box centers, the overlap area, and the aspect ratio. Traditional penalty metrics include distance, shape, and IoU. To accelerate network convergence and improve performance, the matching direction is considered, ultimately resulting in the SIOU loss function:

[0127]

[0128] Where IoU is the intersection-over-union loss, which promotes the complete overlap of the predicted box with the ground-truth box; Δ is the distance loss, which is also affected by the angle loss; and Ω is the shape loss, which constrains the shape of the predicted box under different movement conditions. The purpose of using the angle loss is to minimize the number of singular variables related to distance.

[0129] In this embodiment, the predicted box is first moved to the x-axis or y-axis of the true box (whichever is closer), and then continues to move along the relevant axis.

[0130] like Figure 7 and Figure 8 As shown, if α≤π / 4, then minimize α, otherwise minimize β. The angle loss is defined as follows:

[0131]

[0132] in:

[0133]

[0134]

[0135]

[0136] Taking into account the above angle loss, the distance loss is redefined as follows:

[0137]

[0138] in:

[0139]

[0140] In this embodiment, the purpose of angle loss is to make the loss larger the closer the two target boxes are to the diagonal, and the loss smaller the closer they are to a straight line. The Λ function just meets the requirement. The purpose of distance loss is that the farther the distance is, the greater the distance loss is. The relaxation factor is added to take the angle into account, so the closer to the diagonal, the greater the loss is, and the farther the horizontal or vertical distance is, the greater the distance loss is. Figure 7 and Figure 8 It can be seen that when α→0, the contribution of Distance cost becomes greatly reduced. On the contrary, the closer α is to The greater the contribution of distance cost. As the angle increases, the problem becomes more and more difficult. Therefore, the larger the angle, the greater the effort to calculate the distance. The shape loss is defined as:

[0141]

[0142] in:

[0143]

[0144] The value of θ defines each shape cost and its value is unique. The value of θ is a very important part of this equation, which controls the degree of attention to the shape cost. If θ is 1, it will immediately optimize a shape, thereby damaging the free movement of the shape. The definition of IoU loss is:

[0145]

[0146] For the three-layer Pao detection network, the first layer RPN and ViT are responsible for classification and regression operations on the minimum resolution feature map to predict the corresponding large target information. The regression results and the large target groundtruth are used to calculate the GIoU loss:

[0147] L 1(reg) =L large

[0148] The second layer RPN of the network is responsible for regressing the medium-resolution feature map, obtaining the foreground features of the medium target and splicing them with the output of the first layer ViT, which is then input into the second layer ViT module for target-level modeling. Finally, the network's prediction results for small and medium targets are obtained, and then the regression loss is calculated using the groundtruth. The third layer RPN of the network regresses the high-resolution feature map in the feature pyramid, and obtains the foreground features in front of the small target. It is spliced with the output of the second layer ViT and input into the third layer ViT module for target-level modeling. Finally, the prediction results of all targets are obtained, and then the regression loss is calculated. The formula is as follows:

[0149] L 2(reg) =L large +L middle

[0150] L 3(reg) =L large +L middle +L small

[0151] Therefore, the three-layer ViT structure does not model large, medium, and small objects separately. Instead, it builds on the modeling and alignment of the previous step by integrating the features of the next-level scale objects. This progressive fusion detection method ultimately enables the model to achieve auxiliary detection between objects of the same scale and between objects of different scales.

[0152] The total loss of the entire network training is the sum of the classification and regression losses of the three-layer RPN and ViT outputs.

[0153] 3. Dynamic division method of label scale

[0154] like Figure 9 As shown in the figure, commonly used datasets and detectors generally divide targets into large, medium and small types based on the ratio of the target area to the image. However, under the same area, targets with large width and height differences have larger anchor spans, which makes the network predict them to be larger than the scale range (layer) in which they are located. Therefore, the present invention designs an area scaling function based on the aspect ratio of the target to measure the degree of influence of the width and height factors on its scale category. The area scaling function is defined as follows:

[0155]

[0156] Among them S ori is the original area, S is the area value of the target after scaling, which is used as the final standard for judging the size of the target. In the area scaling function, the input is the width and height of any target (w is width, h is height), and the function trend is as follows Figure 9 As shown, the function starts with an aspect ratio of 1 and increases monotonically along the positive and negative x-axis, with a range of [1, 1.6]. This indicates that the greater the difference in target width and height, the greater the gain (scaling) in area, making it more likely to move closer to the next level of scale (for example, a small target with a large difference in width and height may be classified as a medium target due to its excessive width or height). However, the scaling ratio is controllable, and the increase in target area does not exceed 1.6 times, which is consistent with the target scale distribution.

[0157] Then, the targets in the data set are clustered according to large, medium and small. First, the entire data set is clustered using the scaled target area: the K-means algorithm is used to divide all targets into three parts according to target size, and the cluster centers of the three areas are obtained as follows: SL ,S M ,S S , the calculation process is:

[0158]

[0159] Where n is the number of objects in the dataset. After calculation, objects of different sizes can be assigned to their respective classes. However, during actual training, the scale distribution of objects in different images varies: some objects have large scale differences, while others are smaller, and some contain only large or only small objects. Therefore, such a hard boundary demarcation method will impair the model's fit to the specific image: for some ground truth values that exist at the boundary, strictly following the hard boundary demarcation will cause the supervisory signal for their movement prediction in adjacent feature maps to be zero, which is not conducive to model generalization.

[0160] like Figure 10 As shown, for these boundary values, the best approach is to have the feature maps corresponding to the two adjacent classes predict them, enhancing the model's response to boundary values and facilitating training convergence. To address this issue, during training, the present invention performs tri-clustering on each image in each batch and adds a relaxation variable to the pre-calculated threshold based on the dynamic clustering results, thereby relaxing the boundary constraints.

[0161] The specific calculation process is: for each picture, the target is calculated as its cluster center S′ S ,S′ M ,S′ L , each class can use the mean and variance According to the dynamic clustering results, the target scale division boundary can be reassigned to each image:

[0162]

[0163] Where γ is the relaxation factor, representing the degree of movement from the fixed boundary to the dynamic boundary, and can be set to 0.1 in the experiment. Based on the relaxation formula, the present invention can dynamically adjust the objects in each image so that the thresholds for dividing large, medium and small objects are in a relatively small dynamic range.

[0164] The relaxed boundary partitioning method can make the targets at the boundary points not restricted to a certain fixed scale range. Because the "size" of the target size is a relative concept, the rigid division of its judgment criteria will restrict each layer to a certain type and "cannot move freely". After adopting the relaxed boundary, the feature map of a certain layer of the neural network can fit the adjacent scale targets outside the scale that the layer is responsible for predicting, thus having better generalization performance. The results of repeated predictions will be removed in the subsequent NMS operation, thereby ensuring high precision and high recall rate (for example: if the feature maps of two adjacent layers simultaneously predict the same target on the boundary, only the prediction box with higher confidence will be retained in the NMS operation to remove duplicate predictions).

[0165] Example 2

[0166] 4. Experimental Results

[0167] The experimental hardware used was an Intel 10900k i9 PC equipped with a 24GB NVIDIA 3090ti GPU and Ubuntu 20.04 as the operating system. The algorithm model was implemented in Python using the PyTorch deep learning framework. The experimental environment was based on PyTorch 1.9.1 and Python 3.8. The CNNs used the ResNet backbone from CenterNet2 and the DarkNet from TOLOX. The input sequence encoding dimension of the ViT branch was 256. Due to the small number of input sequences, a multi-head attention architecture was not used. During output post-processing, a non-maximum suppression (NMS) threshold of 0.8 was used, and a confidence threshold of 0.3 was used for visualization of test results. In the early stages of training, the CNN branch was pre-trained on the ImageNet classification dataset, while the ViT branch was pre-trained on ImageNet using the IboT mask self-supervision method. Some parameters during training were set: a learning rate of 4*10^(-3) and an iterative SGD gradient descent with momentum decay. The momentum is set to 0.9 and the weight decay is 0.0005. The entire training process is carried out on two GPUs, with a total of 400 epochs and 32 images (batch size = 32) trained simultaneously.

[0168] Table 1. Test results of this method and other detection networks on the COCO dataset

[0169]

[0170] Table 2. Test results of the proposed method and other detection networks on the UAVDT dataset

[0171]

[0172] like Figure 11a 、 Figure 11b and Figure 11cAs shown in the experimental results, the improved method adopted in this paper improves mAP by 5.1% on CenterNet2 and 4.8% on YOLOX, while increasing the number of parameters by no more than 10%. Its mAP values exceed those of mainstream CNN and ViT-based detectors.

[0173] Combine Figure 11a layer1 (first layer) in Figure 11b layer3 (middle layer) and Figure 11c The layer5 (final layer) image in shows the changing process of attention in each layer of ViT.

[0174] like Figure 12 As shown in the figure, we can see that at the beginning, self-attention only focuses on a part that is exactly the same as itself. As the number of training layers increases, it eventually interacts with the features of the entire image, proving the effectiveness of the added ViT.

[0175] like Figure 13 As shown in the figure, in order to compare with the common self-attention mechanism, the present invention compares the target heat map of ViT and Paodetection. Figure 13 In the figure, the first row shows the original image, and the second row shows the feature extraction results of traditional ViT (using the Swin Transformer in the experiment). Due to the use of Transformer to extract image features, the model contains many noise points in the background area. Since the Pao detection method in this paper only models the foreground objects pre-extracted by CNN, the feature map response is sparser, less noisy, and the response at the target location is more obvious, as shown in the third row of the figure.

[0176] like Figures 14a to 14d As shown in the figure, in order to test the experimental effect on small targets, the YOLOX and YOLOX-Pao detection that performed well in the above experiments were used to conduct experiments. The results are shown in the comparison figure above.

[0177] Experimental results show that the detection effect of all targets is improved, especially small targets.

[0178] V. Summary

[0179] This paper proposes a progressively assisted object detection method (Pao detection), which combines a CNN with a ViT architecture in series. This method improves the model's object detection capabilities by taking into account both local foreground responses and global information, making it particularly effective for small objects. Pao detection consists of a feature extraction network (CNN), an RPN architecture for separating foreground and background, and three ViT architectures for inter-object modeling. Specifically, the CNN first extracts target features and reduces the resolution of the feature map. Subsequently, the RPN architecture extracts regions of interest at different scales from the CNN output feature pyramid. These features are then progressively fed into the three ViT layers for inter-object modeling. A correlation attention map is used to enable feature interaction and complementation, improving feature validity and, consequently, detection performance. A relaxed boundary calculation method is employed to dynamically compute the scale clustering of objects across images in each batch, adding a "soft boundary" to supervised learning. Extensive experimental results demonstrate that Pao detection improves the object detection performance of feature extraction networks, such as ResNet and CSPNet. More specifically, the map of the above network on the COCO dataset is improved by 1.5%-3%. At the same time, experimental results show that it is more effective in detecting small targets. In the UAVDT dataset, which is mostly a small target dataset, the accuracy of the ResNet network is improved by 4%.

[0180] In summary, the present invention uses the CNN network to extract target features, and the obtained feature pyramid is separated from the foreground and background by the RPN to obtain the region of interest and its features; finally, the ViT network is used to perform inter-target modeling on the regions of interest of different scales in turn, thereby achieving high-precision detection with low computational complexity.

[0181] The present invention extracts primary features and separates regions of interest (ROIs) in the early stages of CNN. ViT then progressively stitches and models these regions of interest at different scales, enabling large targets to assist in the detection of medium and small targets. This progressive modeling approach, which prioritizes large targets over small ones, interacts and enhances features between targets, enabling large targets to assist in the detection of small ones. This approach is highly versatile and integrates CNN and ViT architectures in series, taking into account global information while simultaneously addressing local foreground responses. This enhances the model's target detection capabilities, making it particularly effective for small targets.

[0182] The present invention finds strong correlations between targets of the same scale and targets of different scales in the same scene. It can establish an attention heat map between targets within the entire image range and use their mutual relationships to supplement features and assist in recognition, which is of great significance for some fuzzy targets and small targets with limited features.

[0183] The focal loss used in this invention is equivalent to increasing the weight of difficult-to-distinguish samples in the loss function, making the loss function tend to favor difficult-to-distinguish samples, which helps improve the accuracy of difficult-to-distinguish samples. For positive samples (y=1), the present invention will tilt resources toward difficult-to-distinguish samples. The confidence of negative samples is low, and this weighting will further reduce it, thereby more completely isolating the interference of an excessive number of negative samples (most of the image is background).

[0184] The three-layer ViT architecture of the present invention does not model large, medium, and small objects separately. Instead, it integrates the features of the next-level scale objects based on the modeling and alignment of the previous step to perform target modeling and feature alignment. The progressive fusion detection method adopted by the present invention ultimately helps the model achieve auxiliary detection between objects of the same scale and between objects of different scales, improving the applicability of the system.

[0185] The present invention dynamically adjusts the targets in each image so that the thresholds for dividing large, medium and small targets are in a relatively small range of dynamic changes, allowing the feature maps corresponding to the two adjacent categories to predict them, enhancing the model's response to boundary values and being more conducive to training convergence.

[0186] In each batch during training, the present invention will perform tri-clustering for a single image separately, and add a relaxation variable to the pre-calculated threshold according to the dynamic clustering result. By relaxing the boundary division, the target at the boundary point can be unrestricted to a certain fixed scale range, thereby relaxing the boundary restriction.

[0187] The feature maps of a neural network layer in this invention can fit objects at adjacent scales outside the scale predicted by that layer, thereby improving generalization performance. Duplicate predictions are removed in subsequent NMS operations, ensuring high precision and recall. This invention addresses the technical issues of poor fusion, increased network load, and false positives and missed detections in existing technologies.

[0188] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A progressive auxiliary target detection method based on CNN and ViT fusion, characterized in that: The method comprises: S1. Construct a Pao detection model, wherein the Pao detection model includes: a feature extraction backbone network backbone, a detection neck, and a detection head. The feature extraction backbone network backbone is used to extract target features. The detection neck is used to fuse the difference layer feature map up and down according to the target features to obtain a semantic position information difference resolution feature map. The difference layer feature map is binary classified and regressed to obtain no less than two regions of interest. Inter-target modeling is performed on the regions of interest at different scale levels to interact and supplement the target features to obtain ViT output results. The detection head is used to classify and regress the ViT output data to obtain ViT prediction results. S2. Construct a progressive inter-target auxiliary detection model to perform inter-target modeling and feature alignment on the large-resolution RoI features using the ViT-based region of interest modeling method to obtain aligned large-resolution RoI features and aligned medium-resolution RoI features, and perform inter-target modeling and feature alignment based on the aligned large-resolution RoI features and the aligned medium-resolution RoI features and small-resolution RoI features, and perform inter-target modeling and feature alignment to extract the region of interest from the difference-scale feature map using CSPNet; S3, training the foreground target output by each RPN network layer of the detection neck in the Pao detection model and the ViT output result according to the simOTA dynamic positive and negative sample allocation strategy; S4. Calculate the classification loss and regression loss of each layer of the RPN network output and the ViT output data using the Pao detection model to obtain a joint probability loss. S5. Design an area scaling function to cluster the targets in the preset data set according to the scaled target area based on the length and width ratio factor, and divide all the targets into no less than two parts according to the scaled target area to obtain no less than two cluster centers to obtain dynamic clustering results, and redistribute the target scale division boundaries for each image accordingly, so as to use large targets to assist in the detection of small and medium targets.

2. The progressive auxiliary target detection method based on CNN and ViT fusion according to claim 1 is characterized in that: The step S4 comprises: S41, using the Focal loss function to balance the difficulty and ease of samples, calculate the difference between the joint probability of the RPN predicted category, the joint probability of the ViT predicted category, and the target true label value, thereby obtaining the classification loss; S42. Use the angle loss function SIOU to obtain and select the angle loss, redefine the distance loss and shape loss according to the angle loss, and obtain the IoU loss by using the preset logic. Use the first layer RPN and ViT of the Pao detection model to classify and regress the minimum resolution feature map to predict the large target information, obtain the regression result and the true value of the large target, and calculate the SIOU loss based on them. S43, using the second layer RPN regression to process the medium-resolution feature map to obtain the medium target foreground feature and splicing it with the first layer ViT output, and inputting the result into the second layer ViT module for target-level modeling to obtain the small medium target prediction result, and using the true value calculation to obtain the regression loss; S44. Use the high-resolution feature map in the third-layer RPN regression feature pyramid to obtain the foreground features in front of the small target, concatenate the foreground features in front of the small target with the second-layer ViT output, and input them into the third-layer ViT module for target-level modeling to obtain all target prediction results, the classification loss and the regression loss output by each layer, and use the following logic to sum them to obtain the total loss of the entire network training: L=W box L box +W cls L cls Where, L cls With L box Represents classification and regression losses respectively, w cls With w box is the weight factor of the two losses.

3. The progressive auxiliary target detection method based on CNN and ViT fusion according to claim 2 is characterized in that: In step S41, the following logic is used to calculate the difference between the joint probability of the RPN predicted category, the joint probability of the ViT predicted category, and the target true label value: FL(p t )=-(1-p t ) γ log(p t ) Among them, (1-p t ) γ is the modular factor, for the accurately classified sample p t →1, the modularity factor approaches 0, p t Indicates the difficulty of classification, and y indicates the true category.

4. The progressive auxiliary target detection method based on CNN and ViT fusion according to claim 2 is characterized in that: The step S42 includes: S421. Obtain and select the angle loss using the angle loss function SIOU according to the following logic: Among them, IoU is the intersection-over-union loss, Δ is the distance loss, and Ω is the shape loss; S422. Redefine the distance loss according to the angle loss using the following logic: in: γ=2-Λ, γ is the relaxation factor, Λ is the angle loss, which is calculated from x, x is the sine of the angle between the line connecting the midpoints of the prediction box and the target box and the horizontal line, α is the angle, σ is the distance between the two boxes, and c h is the height difference between the two frames, Δ is the distance loss, ρ x is the horizontal normalized Euclidean distance between the two boxes, ρ y is the vertical normalized Euclidean distance between the two frames; S423. Redefine the distance loss according to the angle loss using the following logic: in: θ is the shape attention parameter; S424. Define the IoU loss using the following logic: Among them, B is the prediction box, B GT is the true label box; S425. Use the following logic to classify and regress the minimum resolution feature map using the first layer RPN and ViT of the Pao detection model to predict large object information, obtain the regression result and the true value of the large object, and calculate the GIoU loss based on them: L 1(reg) =L large 。 5. The progressive auxiliary target detection method based on CNN and ViT fusion according to claim 2 is characterized in that: The step S43 calculates the regression loss using the following logic: L 2(reg) =L large +L middle L 3(reg) =L large +L middle +L small Among them, L large 、L middle 、L small Represent the regression losses of three types of targets: large, medium and small.

6. The progressive auxiliary target detection method based on CNN and ViT fusion according to claim 1 is characterized in that: The step S5 comprises: S51, designing the area scaling function using preset logic; S52, clustering the targets in the preset data set according to the scaled target areas, dividing all the targets into at least two parts according to the scaled target areas using a K-means algorithm, so as to obtain at least two cluster centers to obtain a dynamic clustering result; S53: reallocate the object scale division boundaries for each image according to the dynamic clustering result, so as to utilize large objects to assist in detecting small and medium objects.

7. The progressive auxiliary target detection method based on CNN and ViT fusion according to claim 6 is characterized in that: In step S51, the area scaling function is designed using the following logic: Among them, S ori is the original area, S is the area of the target after scaling, w is the width, and h is the height.

8. The progressive auxiliary target detection method based on CNN and ViT fusion according to claim 6 is characterized in that: In step S52, the K-means algorithm is used to divide all the objects into at least two parts according to the scaled object areas using the following logic: Where n is the number of targets in the dataset.

9. The progressive auxiliary target detection method based on CNN and ViT fusion according to claim 6, characterized in that: In step S53, based on the dynamic clustering results, the target scale boundaries are reallocated for each image using the following logic: Wherein, γ is a relaxation factor. In each batch, three clustering is performed for a single image separately, and the relaxation factor is added to the preset threshold according to the dynamic clustering result.

10. A progressive assisted target detection system based on CNN and ViT fusion, characterized in that: The system comprises: A Pao detection model construction module is used to construct a Pao detection model, wherein the Pao detection model includes: a feature extraction backbone network backbone, a detection neck, and a detection head. The feature extraction backbone network backbone is used to extract target features. The detection neck is used to fuse the difference layer feature map up and down according to the target features to obtain a semantic position information difference resolution feature map. The difference layer feature map is binary classified and regressed to obtain no less than two regions of interest. Inter-target modeling is performed on the regions of interest at different scale levels to interact and supplement the target features to obtain ViT output results. The detection head is used to classify and regress the ViT output data to obtain ViT prediction results. A progressive inter-target auxiliary detection module is used to construct a progressive inter-target auxiliary detection model, to use the ViT-based region of interest modeling method to perform inter-target modeling and feature alignment on the large-resolution RoI features, so as to obtain aligned large-resolution RoI features and aligned medium-resolution RoI features, and perform inter-target modeling and feature alignment based on them, and to use the aligned large-resolution RoI features and the aligned medium-resolution RoI features and small-resolution RoI features to perform inter-target modeling and feature alignment, so as to use CSPNet to extract the region of interest from the difference-scale feature map, and the progressive inter-target auxiliary detection module is connected to the Pao detection model construction module; A training module is used to train the foreground targets output by each RPN network layer of the detection neck in the Pao detection model and the ViT output results according to the simOTA dynamic positive and negative sample allocation strategy. The training module is connected to the Pao detection model construction module; A joint probability loss processing module, configured to calculate the classification loss and regression loss of the RPN network output of each layer and the ViT output data using the Pao detection model to obtain a joint probability loss. The joint probability loss processing module is connected to the training module and the Pao detection model construction module; The label scale dynamic division module is used to design an area scaling function, cluster the targets in the preset data set according to the scaled target area, divide all the targets into no less than two parts according to the scaled target area, and obtain no less than two cluster centers to obtain dynamic clustering results, thereby reallocating the target scale division boundaries for each image to use large targets to assist in the detection of small and medium targets. The label scale dynamic division module is connected to the Pao detection model construction module and the progressive inter-target auxiliary detection module.

Images