Target detection and training method, apparatus, device, storage medium, and program product

Abstract

The application provides a target detection and training method, device, equipment, storage medium and program product, wherein the method comprises: performing feature extraction on a to-be-detected image based on a multi-scale feature extraction network to obtain multi-scale fusion features; the multi-scale fusion features carry local features of a target object; performing global correlation processing on the multi-scale fusion features based on a self-attention network to obtain first image features; the first image features carry an association relationship between the local features and global features of the target object; performing processing on the first image features based on a multi-scale feature generation module to obtain second image features of at least two scales; performing target detection on the second image features corresponding to each scale based on a multi-scale prediction network to obtain and fuse detection results corresponding to each scale, and obtaining a target detection result of the target object.

Description

Technical Field

[0001] This application relates to computer vision technology, and more particularly to a target detection and training method, apparatus, device, computer-readable storage medium, and computer program product. Background Technology

[0002] Object detection is a crucial research area in computer vision, aiming to identify and locate multiple target objects from images. This technology is widely used in fields such as industrial quality inspection, remote sensing image analysis, and autonomous driving. In real-world scenarios, targets often appear at different angles and scales, making accurate detection of rotating targets a critical requirement. Existing technologies, traditional convolutional neural networks (such as ResNet and DarkNet) extract features through local receptive fields, capturing details but lacking global correlation. Anchor-free models like YOLOv8-OBB improve performance through multi-scale fusion, but are still limited by the CNN architecture, making it difficult to model complex spatial relationships. Some methods attempt to introduce self-attention mechanisms (such as the Swin Transformer), which enhance global feature representation but weaken the ability to extract local edge information, resulting in insufficient accuracy in detecting small targets or targets at extreme angles. Summary of the Invention

[0003] This application provides a target detection and training method, apparatus, device, computer-readable storage medium, and computer program product, which can obtain accurate target detection results for target objects.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a target detection method, the method comprising: extracting features from an image to be detected based on a multi-scale feature extraction network to obtain multi-scale fused features; the multi-scale fused features carrying local features of a target object; performing global association processing on the multi-scale fused features based on a self-attention network to obtain a first image feature; the first image feature carrying the association relationship between the local features and global features of the target object; processing the first image feature based on a multi-scale feature generation module to obtain second image features at least two scales; performing target detection on the second image features corresponding to each scale based on a multi-scale prediction network, obtaining and fusing the detection results corresponding to each scale to obtain the target detection result of the target object.

[0005] This application provides a training method for training an image detection model. The image detection model includes a multi-scale feature extraction network, a self-attention network, a multi-scale feature generation module, and a multi-scale prediction network as described in the above embodiments. The training method includes: extracting features from a sample image based on the pre-trained multi-scale feature extraction network to obtain sample fusion features; performing global association processing on the sample fusion features based on the self-attention network to obtain first sample image features; processing the first sample image features based on the sample feature generation module to obtain second sample image features at least two scales; performing target detection on the second image features corresponding to each scale based on the sample prediction network to obtain and fuse the sample detection results corresponding to each scale to obtain the target sample detection result of the target object; determining a loss value based on a loss function, the target sample detection result, and the standard result corresponding to the sample image, and adjusting the model parameters of the image detection model based on the loss value.

[0006] This application provides a target detection device, comprising: an extraction module for extracting features from an image to be detected based on a multi-scale feature extraction network to obtain multi-scale fused features; the multi-scale fused features carrying local features of a target object; an association module for performing global association processing on the multi-scale fused features based on a self-attention network to obtain first image features; the first image features carrying the association relationship between local features and global features of the target object; a processing module for processing the first image features based on a multi-scale feature generation module to obtain second image features at least two scales; and a detection module for performing target detection on the second image features corresponding to each scale based on a multi-scale prediction network, obtaining and fusing the detection results corresponding to each scale to obtain the target detection result of the target object.

[0007] This application provides a training device for training an image detection model. The image detection model includes a multi-scale feature extraction network, a self-attention network, a multi-scale feature generation module, and a multi-scale prediction network as described in the above embodiments. The training device includes: a training module for extracting features from sample images based on the pre-trained multi-scale feature extraction network to obtain sample fusion features; performing global correlation processing on the sample fusion features based on the self-attention network to obtain first sample image features; processing the first sample image features based on the sample feature generation module to obtain second sample image features at least two scales; performing target detection on the second image features corresponding to each scale based on the sample prediction network to obtain and fuse the sample detection results corresponding to each scale to obtain the target sample detection result of the target object; determining a loss value based on a loss function, the target sample detection result, and the standard result corresponding to the sample image, and adjusting the model parameters of the image detection model based on the loss value.

[0008] This application provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0009] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method.

[0010] This application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement some or all of the steps in the above-described method.

[0011] The embodiments of this application have the following beneficial effects: Based on the above embodiments disclosed in this application, by extracting and fusing features at different scales through a multi-scale feature extraction network, local features of the target object can be obtained to fully understand the image. Furthermore, the self-attention network mines the correlation between local and global features, improves semantic coherence, and reduces recognition bias. The multi-scale feature generation module generates features at different scales to adapt to the detection of targets of different sizes. In this way, image features can be analyzed comprehensively and correlatedly from different scales, thereby obtaining accurate target detection results for the target object. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the first process of a target detection method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a multi-scale feature extraction network provided in an embodiment of this application; Figure 3A schematic diagram of an exemplary multi-scale feature extraction network provided in this application embodiment; Figure 4 This application provides a schematic diagram of data processing for a C2f module according to an embodiment of the present application. Figure 5 A schematic diagram of a self-attention network structure provided in an embodiment of this application; Figure 6 A schematic diagram of a network structure for a multi-head self-attention module provided in an embodiment of this application; Figure 7 This is a first flowchart illustrating a training method provided in an embodiment of this application; Figure 8 A schematic diagram of an overall network structure provided in an embodiment of this application; Figure 9 A schematic diagram of a target detection method provided in an embodiment of this application; Figure 10 This application provides a schematic diagram of the internal structure of a Unet-like module according to an embodiment of the present application. Figure 11 This application provides a schematic diagram of the internal structure of a Transformer Encoder. Figure 12 This is a schematic diagram of the internal structure of a multi-scale feature generation module provided in an embodiment of this application; Figure 13 This is a schematic diagram of the composition structure of a target detection device provided in an embodiment of this application; Figure 14 This is a schematic diagram of the composition structure of a training device provided in an embodiment of this application; Figure 15 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this application.

[0013] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. It is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. The terms "first / second / third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0016] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0017] Object detection is a core task in the field of computer vision, with the core objective of accurately locating objects in images and identifying their categories. In real-world scenarios such as remote sensing images, aerial photography, and industrial quality inspection, many objects (such as aerial vehicles, remotely sensed buildings, and industrial assembly line parts) are tilted or rotated. Traditional axis-aligned bounding boxes (containing only left, top, right, and bottom coordinates) cannot accurately enclose rotating targets, making rotating target detection a key area of ​​research and application in this field.

[0018] The core bottlenecks of current rotated target detection technology lie in two main dimensions: insufficient local-global correlation in feature extraction and low accuracy of rotated bounding box regression. The characteristics and limitations of mainstream methods and typical models are as follows: (1) Traditional CNN models (ResNet / DarkNet, etc.): They rely on the local receptive field of the convolution kernel to extract features, and are good at capturing local details. However, they have low efficiency in multi-scale feature fusion, cannot model long-distance dependencies, and are prone to losing angle features and global spatial correlation for rotating targets. They also have insufficient detection accuracy for small-sized and extreme angle targets. (2) YOLOv8-OBB model: the mainstream anchor-free rotation detection model. It strengthens the fusion of multi-scale local features through the C2F module and simplifies the anchor box design. However, the core is still the CNN architecture, which lacks global context modeling capabilities. It is prone to angle regression errors and missed detections when facing densely distributed rotating targets with large scale differences. (3) VidDet model: focuses on the detection of rotating targets in video sequences, relies on temporal features to improve stability, but single-frame detection still uses the traditional CNN local feature extraction logic, does not design a dedicated attention mechanism for rotation angle, lacks global spatial information, and has limited improvement in static image detection accuracy; (4) Swin Transformer model: It can efficiently capture global feature associations through windowed self-attention and solve the problem of long-distance dependency modeling, but it weakens the ability to extract local details, fails to capture edge features of small targets, and has high computational complexity in pure Transformer architecture, making it difficult to meet the needs of industrial real-time detection.

[0019] In summary, the relevant technologies have core shortcomings: pure CNN-based methods have strong local features but weak global correlations, while self-attention-based methods have strong global correlations but lack local details, and the adaptability of specialized models to specific scenarios is limited. The key issue lies in the lack of efficient collaboration between the self-attention module and the CNN feature extraction module, leading to an imbalance in the fusion of global features and local details. This ultimately affects the accuracy of bounding box localization and category recognition, failing to meet the high-precision and high-robustness requirements of scenarios such as remote sensing and industrial quality inspection.

[0020] The problems in the related technologies are as follows: insufficient multi-scale feature extraction and easy loss of detailed information: the local receptive field characteristics of traditional CNN and mainstream improved models make it easy to compress high-frequency details such as the edges and angles of rotated targets during downsampling, and it is difficult to accurately recover them during upsampling. The feature expression of small-sized and extreme-angle targets is blurred, which directly affects the detection recall rate. The lack of global spatial correlation capture and poor adaptability to dense scenes: Existing methods generally lack the ability to model global context, cannot establish long-distance dependencies between targets in different regions, are prone to feature confusion in densely arranged scenes, resulting in missed detections and duplicate detections, and have weak resistance to background interference. Rotated bounding boxes have low localization accuracy and inherent challenges in angle regression: affected by the imbalance of local-global feature fusion, angle prediction is easily troubled by periodicity and corner commutation issues, making training convergence difficult; in scenarios with overlapping or occluded targets, the detection box is difficult to accurately surround the target, resulting in poor intersection-over-union (IoU) performance. Redundant network structure and insufficient real-time detection performance: CNN and self-attention fusion schemes often adopt simple feature concatenation strategies, which leads to a surge in parameters and increased computational complexity; the pure Transformer architecture has slow inference speed, and the existing hybrid architecture is difficult to balance detection accuracy and real-time performance, and cannot meet the millisecond-level response requirements of scenarios such as industrial quality inspection and real-time monitoring.

[0021] This application provides a target detection method. The target detection method can be applied to computer devices, wherein the computer device refers to a server, laptop, tablet, desktop computer, smart TV, set-top box, mobile device (e.g., mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device), or other devices with data processing capabilities.

[0022] Figure 1This is a schematic diagram of the first process of a target detection method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes the following steps S101 to S104: Step S101: Extract features from the image to be detected based on a multi-scale feature extraction network to obtain multi-scale fusion features; the multi-scale fusion features carry local features of the target object.

[0023] Among them, multi-scale feature extraction networks can extract feature information at different scales from the image to be detected through operations such as convolution and pooling at different levels. The feature information at different scales can correspond to multiple levels, from local details to overall structure, which helps to comprehensively understand the image content. Correspondingly, multi-scale fusion features are feature representations obtained after the multi-scale feature extraction network processes the image to be detected. These features fuse features of the image at different scales, including both local detail features of the target object, such as edges and textures, and some overall structural information.

[0024] In some embodiments, the image to be detected can be acquired and preprocessed. The preprocessing operations include image normalization and resizing to convert the image into an input format acceptable to the multi-scale feature extraction network. Then, the preprocessed image to be detected is input into the multi-scale feature extraction network. Local detail features such as edges and textures are extracted through shallow convolutional and pooling layers of the network, while semantic features are extracted through deep convolutional and fully connected layers. Different network layers correspond to different receptive fields, enabling the extraction of features at different scales. Finally, a feature fusion strategy is adopted to integrate the extracted features at different scales by concatenating channels and assigning weights, resulting in a multi-scale fused feature that carries local feature information of the target object.

[0025] For example, in the scenario of drone-based wind power inspection, the image to be detected is an aerial image of wind power equipment taken by a drone, and the target object is an inclined blade. The multi-scale feature extraction network adopts a convolutional neural network architecture. The shallow network extracts local detail features such as the edge and texture of the blade (including details related to minor deformation defects) through 3×3 convolutional kernels. The deep network extracts semantic features such as the overall shape of the blade and the relative position of the blade to the wind turbine nacelle through 7×7 convolutional kernels. Then, the shallow and deep features are integrated through channel fusion to obtain multi-scale fused features. These features carry local features such as the edge and texture of the blade, as well as semantic features related to the overall shape of the blade.

[0026] Step S102: Perform global association processing on the multi-scale fusion features based on a self-attention network to obtain a first image feature; the first image feature carries the association relationship between the local features and global features of the target object.

[0027] Self-attention networks are neural network structures used to capture the relationships between various positions in input features. They do not rely on external information and achieve global feature correlation by calculating the attention weights between each position within a feature and all other positions. This effectively captures long-distance dependencies between features and is commonly found in Transformer architectures. The first image feature is the feature data obtained after global correlation processing by the self-attention network. This first image feature retains local feature information from the multi-scale fused features while adding correlation information between local and global features.

[0028] In some embodiments, global features are feature information used to characterize the overall scene and overall attributes of the target object in the image to be detected. For example, they may involve the overall shape of the target object, the relationship between the target and the background, the relative positions of multiple targets, etc., and can reflect the overall characteristic attributes of the target object. Local features are feature information of local regions of the target object, including the features of the target's edges, textures, corners and other details, and can characterize the local shape and structural attributes of the target object.

[0029] In some possible implementations, the obtained multi-scale fusion features can be dimensionally adjusted to transform them into a feature format acceptable to self-attention networks, mapping the features to query vectors, key vectors, and value vectors. The similarity between the query vector and the key vector is calculated to obtain the attention weight matrix, which reflects the correlation strength between each feature location. Then, the attention weight matrix is ​​normalized to eliminate the influence of numerical differences. Next, the normalized attention weight matrix and the value vector are weighted and summed to achieve global feature integration. Finally, the integrated features are optimized through a fully connected layer to output the first image feature, which carries the correlation between the local features and global features of the target object.

[0030] It is understandable that the self-attention network described above can mine the correlation information between local features and global features, as well as between different local features, in multi-scale fusion features, so that the first image features carry the correlation between local and global features; it can improve the semantic coherence of features and reduce the recognition bias caused by isolated local features.

[0031] Step S103: Process the first image features based on the multi-scale feature generation module to obtain second image features at least two scales.

[0032] The multi-scale feature generation module is used to perform scale transformation on the input features to generate features of different resolutions and scales. In some embodiments, the multi-scale feature generation module may include an upsampling layer, a downsampling layer, and a convolutional layer, thereby transforming the input features into features of multiple preset scales according to the target detection requirements, adapting to the detection requirements of targets of different sizes.

[0033] In some possible implementations, the obtained first image features can be input into a multi-scale feature generation module. This multi-scale feature generation module has at least two scale branches, each corresponding to a target scale. For each scale branch, the resolution of the first image features is reduced by a downsampling layer (such as a max pooling layer or an average pooling layer) to generate small-scale second image features, or the resolution of the first image features is increased by an upsampling layer (such as a transposed convolutional layer or an interpolation layer) to generate large-scale second image features. Finally, at least two scales of second image features are output, and the resolution and receptive field of each scale feature are adapted to the target object of the corresponding size.

[0034] In other possible implementations, the obtained first image features can be input into a multi-scale feature generation module. This module contains multiple cascaded upsampling or downsampling layers, connected in series to form a cascaded structure, where each layer serves as an independent scale output node. If a cascaded downsampling structure is used, the first image features are input into the first downsampling layer (e.g., max pooling or average pooling), and after downsampling, the second image features at the first scale are output. Simultaneously, the output features of this layer are passed to the second downsampling layer, and after further downsampling, the second image features at the second scale are output. This process continues until a predetermined number of second image features at different scales are output. If a cascaded upsampling structure is used, the first image features are input into the first upsampling layer (e.g., transposed convolutional or interpolation layer), and after upsampling, the second image features at the first scale are output. These features are then passed to the second upsampling layer, and after further upsampling, the second image features at the second scale are output. This process is repeated for all scale features.

[0035] Step S104: Target detection is performed on the second image features corresponding to each scale based on a multi-scale prediction network, and the detection results corresponding to each scale are obtained and fused to obtain the target detection result of the target object.

[0036] The multi-scale prediction network is used to detect targets on features at multiple different scales and then fuse the detection results. This multi-scale prediction network includes multiple prediction branches and a result fusion module. Each prediction branch corresponds to a second image feature at a specific scale, enabling the identification and localization of targets at that scale. The detection result is the target-related information output by the prediction branch after processing the second image feature at the corresponding scale, including the target's category, location coordinates, and confidence level. Each scale corresponds to an independent set of detection results, reflecting the target detection status at that scale. The target detection result is the final output of the multi-scale prediction network detection and result fusion, providing comprehensive information about the target objects in the image to be detected. It includes the categories, precise location coordinates, and confidence levels of all targets and is the final output of the entire target detection process.

[0037] In some possible implementations, the obtained second image features at least two scales can be input into the prediction branches of the multi-scale prediction network. Each prediction branch processes the input second image features through convolutional layers and fully connected layers, and outputs the detection results at the corresponding scale, including the target category, location coordinates, and confidence score. Then, the detection results output by each prediction branch are input into the result fusion module, and redundant detection boxes of the same target are removed by non-maximum suppression algorithm. Finally, the fused target detection result is output, which includes the category, precise location, and confidence score information of all target objects in the image to be detected.

[0038] Based on the above embodiments disclosed in this application, by extracting and fusing features at different scales through a multi-scale feature extraction network, local features of the target object can be obtained to fully understand the image. Furthermore, the self-attention network mines the correlation between local and global features, improves semantic coherence, and reduces recognition bias. The multi-scale feature generation module generates features at different scales to adapt to the detection of targets of different sizes. In this way, image features can be analyzed comprehensively and correlatedly from different scales, thereby obtaining accurate target detection results for the target object.

[0039] In some embodiments disclosed in this application, the multi-scale feature extraction network includes a downsampling network, an upsampling network, and a feature compression module connected in sequence. The downsampling network includes M cascaded downsampling units, and the upsampling network includes N cascaded upsampling units. The step of extracting features from the image to be detected based on the multi-scale feature extraction network to obtain multi-scale fused features includes: sequentially downsampling the image to be detected through the M downsampling units to obtain target downsampling output features; sequentially upsampling the target downsampling output features through the N upsampling units to obtain target upsampling output features; and compressing the target upsampling output features through the feature compression module to obtain the multi-scale fused features.

[0040] The downsampling network, composed of multiple cascaded downsampling units, downsamples the input image or feature map, reducing its spatial resolution while increasing the number of channels to extract more abstract and semantically meaningful features, reducing computational cost and expanding the receptive field. The upsampling network, also composed of multiple cascaded upsampling units, upsamples the downsampled feature map, restoring its spatial resolution and allowing it to retain more spatial location information. This information is then fused with the features extracted by the downsampling network to obtain a more comprehensive feature representation. The feature compression module compresses the upsampled feature map to reduce the number of channels, remove redundant information, and extract key features, resulting in a more compact and effective multi-scale fused feature representation.

[0041] Please see Figure 2 The multi-scale feature extraction network includes a downsampling network 21, an upsampling network 22, and a feature compression module 23 connected in sequence. The image to be detected passes through the downsampling network 21, the upsampling network 22, and the feature compression module 23 in sequence to obtain the output multi-scale fused features.

[0042] The downsampling network 21 can include M downsampling units, where M is an integer greater than or equal to 2. The M downsampling units sequentially downsample the image to be detected. Each downsampling unit performs operations such as convolution and pooling on the input feature map to reduce spatial resolution and increase the number of channels, obtaining the target downsampled output features. For example... Figure 2 As shown, the downsampling network 21 includes downsampling units 211 to 21M. The image to be detected is processed by these M downsampling units in sequence to obtain the target downsampled output features.

[0043] In some embodiments, this downsampling network is used to progressively reduce the spatial resolution of an image. Each downsampling unit can consist of a convolutional layer, an activation function (such as ReLU), and a pooling layer, effectively extracting local details and gradually constructing a more abstract feature representation. For example, the C2F module is used as a downsampling unit, enhancing the expressive power of local features through the Bottleneck structure and residual connections.

[0044] The upsampling network 22 can include N upsampling units, where N is an integer greater than or equal to 2. The target downsampled output features are sequentially upsampled using these N upsampling units. Each upsampling unit increases the spatial resolution of the feature map through methods such as transposed convolution to obtain the target upsampled output features. Figure 2As shown, the upsampling network 22 includes upsampling units 221 to 22N. The target downsampled output features are sequentially processed by these N upsampling units to obtain the target upsampled output features.

[0045] In some embodiments, the upsampling unit is used to progressively restore the downsampled low-resolution features to high resolution, thereby preserving more spatial details. The upsampling unit can be implemented using deconvolution (transposed convolution) or interpolation methods, such as a combination of C2F and deconvolution.

[0046] Finally, the target upsampled output features are input to the feature compression module 23. After the target upsampled output features are compressed, the multi-scale fusion features can be obtained.

[0047] Downsampling by the downsampling unit can gradually extract abstract features at different levels of the image, expand the receptive field, and reduce computation. Upsampling by the upsampling unit can restore the spatial resolution of the feature map, retain more spatial location information, and make the features more comprehensive. Compression by the feature compression module can remove redundant information, extract key features, and obtain more compact and effective multi-scale fusion features. Combining these effects, multi-scale fusion features can more comprehensively and accurately describe the image content and improve the performance of subsequent image analysis tasks (such as object detection, image classification, etc.).

[0048] In this system, the input of the m-th downsampling unit is the output of the (m-1)-th downsampling unit, the input of the 1st downsampling unit is the image to be detected, and the output of the m-th downsampling unit is the target downsampled output feature. For example, M can be set to 5, then the input of the 2nd downsampling unit is the output of the 1st downsampling unit.

[0049] The input of the nth upsampling unit includes the output of the (n-1)th upsampling unit and the output of the Mnth downsampling unit. The input of the 1st upsampling unit is the target downsampling output feature and the output of the Mth downsampling unit. The output of the nth downsampling unit is the target upsampling output feature.

[0050] In some possible implementations, for the nth upsampling unit, the output of the (n-1)th upsampling unit can be upsampled, and then the upsampled features are fused with the output of the Mnth downsampling unit to obtain the output of the nth upsampling unit.

[0051] For example, M can be set to 5, and N can be set to 3. See [link / reference] Figure 3The downsampling network includes upsampling units 211 to 215, and the upsampling network includes upsampling units 221 to 223. The input to the first upsampling unit is the target downsampling output feature and the output of the fourth downsampling unit. The input to the second upsampling unit is the output of the first upsampling unit and the output of the third downsampling unit. The input to the third upsampling unit is the output of the second upsampling unit and the output of the second downsampling unit. Taking the second upsampling unit as an example, this second upsampling unit can upsample the output of the first upsampling unit, and then fuse the upsampled feature with the output of the third downsampling unit to obtain the output of the second upsampling unit.

[0052] Where m is an integer less than or equal to M, n is an integer less than or equal to N, both m and n are greater than or equal to 2, and M is greater than or equal to N.

[0053] In some embodiments disclosed in this application, the downsampling unit includes a C2f module and a downsampling convolutional layer; the upsampling unit includes a C2f module and an upsampling convolutional layer; wherein, the processing procedure of the C2f module for the input first feature includes: adjusting the number of channels of the first feature to the number of output channels through the input convolutional layer; dividing the first feature into a first branch feature and a second branch feature through a block layer; processing the first branch feature sequentially through L stacked neck network layers to obtain a third feature output by each of the neck network layers; connecting the first branch feature, the second branch feature, and the third feature output by each of the neck network layers through a fully connected stacking module to obtain a fourth feature; fusing the fourth feature through a feature fusion module to obtain a fifth feature; the number of channels of the fifth feature is the number of output channels.

[0054] It is understood that both the downsampling unit and the upsampling unit provided in the embodiments of this application include the C2f module.

[0055] In some embodiments, the downsampling unit includes a C2f module and a downsampling convolutional layer. The input features first pass through the C2f module, and then through the downsampling convolutional layer for downsampling to obtain the output features. The downsampling convolutional layer is a module that reduces the spatial resolution of the feature map through convolution operations. It typically uses a convolutional kernel with a stride greater than 1, which can reduce the amount of computation and expand the receptive field. At the same time, the feature representation ability can be enhanced by adjusting the number of channels.

[0056] In some embodiments, the upsampling unit includes a C2f module and an upsampling convolutional layer. The input features first pass through the C2f module, and then through the upsampling convolutional layer for upsampling to obtain the output features. The upsampling convolutional layer is a module that restores the spatial resolution of the feature map through deconvolution, and can reconstruct detailed information to match the original input size.

[0057] In some possible implementations, the processing procedure of the C2f module for the first feature of the input can be found in [reference needed]. Figure 4 After inputting the first feature, the input convolutional layer 31 adjusts its channel count c_in to the output channel count c_out; the block layer 32 divides the feature into a first branch feature and a second branch feature (with 0.5c_out channels) along the channel dimension; L stacked neck network layers (such as...) Figure 4 The neck network layers 331 to 33L sequentially perform convolution, batch normalization, and activation function processing on the first branch features to generate the third features of each layer; the fully connected stacking module 34 concatenates the first branch features, the second branch features, and all third features along the channel dimension to generate the fourth feature; the feature fusion module 35 performs channel mixing on the fourth feature through 1×1 convolution to generate the fifth feature with the number of channels equal to the number of output channels.

[0058] Based on the embodiments disclosed in this application, the C2f module adjusts the number of channels in the input convolutional layer, divides features into block layers, extracts features at different levels in the neck network layer, connects features in the fully connected stacking module, and fuses features in the feature fusion module. Furthermore, in the downsampling unit, the C2f module is combined with the downsampling convolutional layer to reduce computation, expand the receptive field, and enhance feature expression capabilities. In the upsampling unit, the C2f module is combined with the upsampling convolutional layer to reconstruct detailed information to match the original input size. This allows for comprehensive and effective processing of input features, integrating feature information at different levels and types, thereby improving the model's ability to extract and utilize features and enhancing the model's performance in related tasks.

[0059] In some embodiments disclosed in this application, the self-attention network includes a dimension transformation module and at least two cascaded multi-head self-attention modules; the step of performing global association processing on the multi-scale fusion features based on the self-attention network to obtain the first image feature includes: performing dimension transformation on the multi-scale fusion features through the dimension transformation module to obtain a target format feature; the number of channels of the target format feature is the same as the number of channels of the multi-scale fusion feature; and performing self-attention processing on the target format feature through the at least two cascaded multi-head self-attention modules to obtain the first image feature.

[0060] Please see Figure 5 It shows a schematic diagram of a self-attention network structure, wherein the self-attention network 60 includes a dimension transformation module 61 and at least two cascaded sets of multi-head self-attention modules ( Figure 5 An example is shown of J sets of multi-head self-attention modules, including multi-head self-attention modules 621 to 62J. After the multi-scale fusion feature is transformed by the dimension transformation module, the target format feature is obtained. Figure 5(not shown in the image), and then the target format features are processed sequentially by J groups of multi-head self-attention modules to obtain the first image features.

[0061] In some embodiments, the above-mentioned dimensional transformation of the multi-scale fusion features by the dimensional transformation module to obtain target format features includes: converting the multi-scale fusion features in the spatial dimension into target format features in the sequence dimension by the dimensional transformation module.

[0062] For example, assuming the multi-scale fusion feature is represented as F_cnn (H×W×C, where H=80, W=80, C=512), the above conversion from "spatial dimension" to "sequence dimension" can be understood as expanding the spatial pixels of H×W into a feature sequence of length L=H×W=6400. The converted feature format is L×C (6400×512), and the target format feature can be denoted as F_seq.

[0063] Understandably, the number of channels in the target format features and the multi-scale fusion features have not changed in terms of channel dimension.

[0064] like Figure 5 As shown, after the dimension transformation module 61 outputs the target format features, these features are sequentially processed by at least two cascaded multi-head self-attention modules to obtain the first image features. The first multi-head self-attention module (i.e., Figure 5 The input to the multi-head self-attention module 621 is the target format feature. After self-attention processing by this multi-head self-attention module, the target format feature after one self-attention processing is obtained. This target format feature after one self-attention processing is then input to the second multi-head self-attention module ( Figure 5 (not shown in the diagram), resulting in target format features processed twice by self-attention. This process is repeated, with the target format features processed J-1 times by self-attention then input into the last multi-head self-attention module (i.e., ...). Figure 5 After the multi-head self-attention module 62J in the image, the target format feature after J self-attention processing is obtained, which is the first image feature.

[0065] In some possible implementations, the self-attention network described above includes a dimension transformation module and four sets of multi-head self-attention modules connected in sequence.

[0066] The multi-head self-attention module includes a multi-head self-attention unit and a feedforward network. The processing of the input sixth feature by the multi-head self-attention module includes: dividing the sixth feature along the channel dimension using the multi-head self-attention unit to obtain sub-input features corresponding to each attention head; generating query vectors, key vectors, and value vectors corresponding to each sub-input feature using a learnable parameter matrix; generating attention weights corresponding to each sub-input feature based on the query vectors and key vectors; determining sub-output features corresponding to each sub-input feature based on the attention weights and value vectors, and concatenating the sub-output features corresponding to each attention head along the channel dimension to obtain the seventh feature; and fusing the seventh feature with the sixth feature after sequentially performing a first linear transformation, activation, and a second linear transformation using the feedforward network to obtain the output eighth feature.

[0067] Please see Figure 6 The diagram illustrates a network structure of a multi-head self-attention module, wherein the multi-head self-attention module 62j includes a multi-head self-attention unit 62j1 and a feedforward network 62j2. For the input sixth feature, the sixth feature is processed by the multi-head self-attention unit 62j1 to obtain the seventh feature. After the seventh feature is processed by the first linear transformation, activation and second linear transformation of the feedforward network 62j2, it is fused with the sixth feature to obtain the output eighth feature.

[0068] The sixth feature refers to the input feature entering the current multi-head self-attention module. The sixth feature is divided into several sub-input features along the channel dimension, with each sub-input feature corresponding to a processing unit of one attention head. For example, if there are a total of 8 attention heads, the number of channels for each sub-input feature is the number of channels for the sixth feature divided by 8. This channel-based division of the sixth feature allows each attention head to independently process different sub-features, enhancing the model's parallel computing capabilities.

[0069] The learnable parameter matrix refers to the parameter matrix that is automatically updated during training. It maps sub-input features to three different vector spaces: query, key, and value. The query vector measures the relevance of the current position to other positions, the key vector responds to query requests from other positions, and the value vector conveys specific feature information. Accordingly, the learnable parameter matrix includes the query parameter matrix, the key parameter matrix, and the value parameter matrix.

[0070] For example, the query vector (Q), key vector (K), and value vector (V) can be determined as follows: Q = F_seq × W_Q, K = F_seq × W_K, V = F_seq × W_V, where F_seq is the sub-input feature, and W_Q, W_K, and W_V are the query parameter matrix, key parameter matrix, and value parameter matrix, respectively, all with a dimension of 512 × 512.

[0071] In some embodiments, the above-mentioned generation of attention weights corresponding to each sub-input feature based on the query vector and key vector corresponding to each sub-input feature can be determined by: determining the attention score corresponding to each sub-input feature based on the query vector and key vector corresponding to each sub-input feature; and normalizing the attention score corresponding to each sub-input feature to obtain the attention weights corresponding to each sub-input feature.

[0072] For example, the attention score corresponding to a sub-input feature can be determined in the following way: .in, To score attention, Dimensions for each attention head, for example =64. In some embodiments, the attention weights can be obtained by normalizing the attention score using the Softmax function. .

[0073] In some embodiments, after obtaining the attention weights corresponding to each of the sub-input features, the seventh feature can be determined by: multiplying the attention weights corresponding to the sub-input features and the value vectors corresponding to each sub-input feature as the sub-output feature corresponding to the sub-input feature; and concatenating the sub-output features corresponding to all attention heads along the channel dimension to obtain the seventh feature.

[0074] For example, this sub-output feature can be determined in the following way: ,in, For sub-output features, This refers to attention weights. When there are 8 attention heads, the weights of the 8 attention heads can be... Channel splicing is performed, and the channel is restored to 512 dimensions through linear transformation to obtain the seventh feature of the output.

[0075] In some embodiments, after the seventh feature is sequentially subjected to a first linear transformation, activation, and a second linear transformation through the feedforward network, it is fused with the sixth feature to obtain the output eighth feature. This includes: increasing the dimensionality of the seventh feature through a first linear layer using the feedforward network; applying the GELU activation function to the increased-dimensional feature to obtain the activated feature; reducing the dimensionality of the activated feature to the original number of channels (i.e., the same as the number of channels of the seventh feature) through a second linear layer to obtain the dimensionality-reduced feature F_ffn; and then fusing the obtained dimensionality-reduced feature F_ffn with the input feature F_input (i.e., the sixth feature) of the multi-head self-attention module to obtain the output eighth feature.

[0076] For example, assuming the sixth feature has a dimension of L×C (e.g., 6400×512), after the first linear transformation (dimensionality increase), we can obtain the dimensionality-increased feature with a dimension of L×C_ffn (e.g., 6400×2048). Applying the GELU activation function to the dimensionality-increased feature yields the activated feature, where the dimension remains unchanged. After the second linear transformation (dimensionality reduction), we obtain the dimensionality-reduced feature F_ffn with a dimension of L×C (e.g., 6400×512), thus restoring the feature dimension for easier residual connection. Adding the dimensionality-reduced feature F_ffn to the sixth feature F_input yields the eighth feature F_block: F_block = F_input + F_ffn.

[0077] Based on the above embodiments disclosed in this application, the multi-scale fusion features are converted from spatial dimension to sequence dimension by the dimension conversion module while keeping the number of channels unchanged. Furthermore, the converted features are processed multiple times by using at least two cascaded multi-head self-attention modules. This can gradually refine the feature representation and enhance the ability to capture local details and global dependencies, thereby achieving effective collaborative modeling of local features and global information and improving the performance of rotating target detection.

[0078] In some embodiments disclosed in this application, the multi-scale prediction network includes a result fusion module and a prediction sub-network corresponding to each scale; the step of performing target detection on the second image features corresponding to each scale based on the multi-scale prediction network, obtaining and fusing the detection results corresponding to each scale, and obtaining the target detection result of the target object includes: performing target detection on the second image features corresponding to each scale based on the prediction sub-network corresponding to each scale, obtaining the detection result corresponding to each scale; summarizing the detection results of each scale based on the result fusion module, filtering invalid detection results, and obtaining the target detection result of the target object.

[0079] The aforementioned prediction subnetworks are subnetwork structures designed for each specific scale in the multi-scale prediction network, used for target detection of the second image features at that scale. Each prediction subnetwork independently processes the features at its corresponding scale and outputs the detection result at that scale. Prediction subnetworks at different scales can complement or verify each other, jointly completing the comprehensive detection of targets in the image. Correspondingly, the result fusion module can summarize, analyze, and process the detection results obtained from prediction subnetworks at different scales. By filtering invalid detection results, it integrates the detection information from multiple scales into a final target detection result, improving the accuracy and reliability of the detection.

[0080] In the above embodiments, since image features at different scales contain information at different levels, small-scale features can capture global information of the image and features of larger targets, while large-scale features can retain more detailed information, which is beneficial for detecting small targets. This application sets up prediction sub-networks of different scales to process the second image features of the corresponding scales respectively, which can make full use of the advantages of features at different scales and improve the detection capability of targets of different sizes, thereby obtaining independent detection results at each scale. Correspondingly, since the detection results obtained by prediction sub-networks at different scales may contain duplicates, errors, or invalidities, for example, the same target may be detected multiple times at different scales, or some detection results may be false detections, the result fusion module summarizes and analyzes the detection results at each scale, and uses certain algorithms and rules to filter out invalid detection results, such as duplicate detections and low-confidence detections, and integrates the effective detection information to obtain the final accurate target object detection result.

[0081] In some possible implementations, the second image features at each scale can be input into the corresponding prediction sub-network. The prediction sub-network further processes and extracts features from the second image features through operations such as convolutional layers and pooling layers. Finally, it outputs the detection results at that scale using fully connected layers or regression layers. The detection results can include the target's location information (such as bounding box coordinates) and category information. The result fusion module receives the detection results output from the prediction sub-networks at each scale, summarizes these results, and processes the summarized detection results using algorithms such as Non-Maximum Suppression (NMS) to filter out duplicate detection boxes and low-confidence detection results. For example, for detection boxes of the same category, they are sorted according to their confidence, retaining the detection box with the highest confidence while suppressing other detection boxes with high overlap with this detection box, ultimately obtaining the target detection result of the target object.

[0082] Based on the above embodiments disclosed in this application, by setting up prediction subnetworks of different scales to process the second image features of corresponding scales respectively, the detection capability of targets of different sizes can be improved by utilizing the advantages of features of different scales. Furthermore, the result fusion module uses algorithms such as non-maximum suppression to summarize and analyze the detection results of each scale and filter invalid results. In this way, the detection information of multiple scales can be effectively integrated, reducing duplicate and erroneous detection, thereby improving the accuracy and reliability of target detection and realizing comprehensive detection of targets in the image.

[0083] In some embodiments disclosed in this application, the prediction subnetwork includes a category prediction subnetwork and a bounding box parameter prediction subnetwork; the step of performing target detection on the second image features corresponding to each scale based on the prediction subnetwork corresponding to each scale to obtain the detection result corresponding to each scale includes: performing category detection on the second image features corresponding to each scale based on the category prediction subnetwork corresponding to each scale to obtain the category detection result corresponding to each scale; and performing bounding box detection on the second image features corresponding to each scale based on the bounding box parameter prediction subnetwork corresponding to each scale to obtain the bounding box parameter detection result corresponding to each scale.

[0084] Here, the prediction subnetwork refers to the subnetwork module used for classifying image features and regressing bounding boxes. The prediction subnetwork is split into two branches: the class prediction subnetwork and the rotation box parameter prediction subnetwork. The class prediction subnetwork is responsible for determining which class the target in the image belongs to, while the rotation box parameter prediction subnetwork is responsible for outputting the geometric parameters (such as center point coordinates, length, width, and rotation angle) of the rotation box surrounding the target.

[0085] In some embodiments, the class prediction subnetwork is responsible for identifying the class of objects in an image. The class prediction subnetwork extracts semantic information of the objects by performing convolutional operations on multi-scale feature maps and outputs a class probability distribution through fully connected layers. Each scale of the class prediction subnetwork operates independently to achieve accurate classification of objects of different sizes.

[0086] In some embodiments, the role of the rotation box parameter prediction subnetwork is to output the rotation box parameters of the target. Since the target may be tilted or rotated, traditional axis-aligned rectangles cannot accurately describe the target's position; therefore, the rotation box parameter prediction subnetwork is introduced. This subnetwork extracts the target's spatial position and angle information by performing convolution operations on multi-scale feature maps, and outputs parameters such as the center point coordinates, length, width, and rotation angle of the rotation box. After normalization, the center point coordinates, length, width, and rotation angle parameters output by the rotation box parameter prediction subnetwork can more accurately describe the target's actual position and orientation.

[0087] In the above embodiments, by splitting the prediction subnetwork into a category prediction subnetwork and a rotation box parameter prediction subnetwork, accurate identification of the target category and precise regression of the rotation box parameters can be achieved, thereby improving detection accuracy. By splitting the prediction subnetwork into a category prediction subnetwork and a rotation box parameter prediction subnetwork, different categories of targets can be effectively distinguished, and it can adapt to targets in various tilted or rotated states, thus significantly improving the performance of rotating target detection.

[0088] In some embodiments, category detection is a process of classifying input image features through a category prediction subnetwork. Second image features at each scale are fed into the corresponding scale's category prediction subnetwork, which performs convolution and non-linear activation operations on the second image features, ultimately outputting a category probability distribution at the corresponding scale. The category probability distribution represents the likelihood that an object in the image belongs to each category; the category with the highest probability value is the predicted category of the object.

[0089] In some embodiments, the category detection result includes not only the target's category label but also the confidence level of the target's category in the image. High confidence indicates that the model's prediction of the target's category is highly reliable, while low confidence indicates that the model has uncertainty. In some possible implementations, a confidence threshold can be set, and target recognition results with a confidence level higher than the set threshold can be considered valid detection results.

[0090] In the embodiments of this application, by using a category prediction subnetwork for category detection, targets can be accurately classified at different scales, thereby reducing false detections or missed detections caused by differences in target size in complex scenes.

[0091] In some embodiments, rotated bounding box detection is a process of regressing bounding boxes from the input image features using a rotated bounding box parameter prediction subnetwork. Second image features at each scale are fed into the corresponding scale's rotated bounding box parameter prediction subnetwork, which performs convolution and non-linear activation operations on the second image features, ultimately outputting the rotated bounding box parameters for the corresponding scale. The rotated bounding box parameters include the coordinates of the center point of the detected target, its length and width, and the rotation angle, describing the actual position and pose of the detected target.

[0092] In some embodiments, the result of the rotated bounding box parameter detection includes not only the bounding box of the target but also the confidence score of the bounding box in the rotationd bounding box parameter detection result. A high confidence score indicates that the model's judgment of the bounding box in the rotationd bounding box parameter detection result is highly reliable, while a low confidence score indicates that the model has uncertainty in judging the bounding box in the rotationd bounding box parameter detection result. In some possible implementations, a confidence threshold can be set, and rotationd bounding box parameter detection results with a confidence score higher than the set threshold can be identified as valid detection results.

[0093] Based on the above embodiments disclosed in this application, by splitting the prediction subnetwork into a category prediction subnetwork and a rotation box parameter prediction subnetwork, and performing category detection and rotation box detection on second image features at different scales respectively, it is possible to effectively distinguish different categories of targets, obtain the precise position and pose information of the targets, thereby improving detection accuracy and significantly improving the performance of rotating target detection.

[0094] In some embodiments disclosed in this application, the step of summarizing the detection results at each scale based on the result fusion module, filtering out invalid detection results, and obtaining the target detection result of the target object includes: grouping the rotation box parameter detection results corresponding to each scale according to the category detection results corresponding to each scale; performing cross-scale non-maximum suppression on all rotation box parameter detection results of each category for each category, filtering out duplicate rotation boxes, and determining the target rotation box parameter detection result of the category; and determining the target detection result based on the target rotation box parameter detection results corresponding to each category.

[0095] Here, category detection results refer to the category information of the target object identified at each scale. For example, there may be multiple categories in an image, such as vehicles, ships, and buildings, and the detection results for each category are recorded separately. The step based on the category detection results corresponding to each scale groups the rotation box parameter detection results of the same category at different scales into one group so that subsequent processing can be optimized for specific categories. For example, if a vehicle category target is detected in the target detection results of three target objects at the 1 / 8 scale, and a vehicle category target is also detected in the target detection results of two target objects at the 1 / 16 scale, then the vehicle category rotation box parameter detection results in the target detection results of the target objects at these two scales will be merged into one group to facilitate unified cross-scale non-maximum suppression (NMS) operation.

[0096] Here, category detection results refer to the category information of the target object identified at each scale. For example, there may be multiple categories in an image, such as vehicles, ships, and buildings. Rotation box parameter detection results refer to the parameter representation of the rotated bounding box, which may include geometric information such as center point coordinates, length, width, and rotation angle. The above grouping process can be understood as classifying the rotation box parameters from different scale detection results into the same category based on the category label, forming a set indexed by category. Target detection results refer to the final output after integrating the rotation box parameters of all categories of targets, which may include category labels, rotation box geometric parameters, and confidence scores.

[0097] In some embodiments, all scale detection results can be traversed, the category label of each rotated box can be extracted, and the rotated box parameters can be stored in the corresponding category set according to the label to complete the grouping operation.

[0098] For example, in a remote sensing image detection scenario, the model detects two types of targets, aircraft and vehicles, on feature maps at three scales respectively. After grouping, a "aircraft" category set (containing aircraft bounding box parameters at three scales) and a "vehicle" category set (containing vehicle bounding box parameters at three scales) are formed.

[0099] In some embodiments, the aforementioned cross-scale nonmaximum suppression refers to an algorithm that, in multi-scale detection results, selects the optimal rotating box and filters redundant boxes by comparing the overlap and confidence of rotating boxes of the same target at different scales. Here, repeated rotating boxes refer to rotating boxes pointing to the same target but with similar parameters in detection results at different scales. It can be understood that these repeated rotating boxes are generated due to repeated detections by the model on multi-scale feature maps.

[0100] In some possible implementations, the list of rotated bounding boxes for each category can be sorted in descending order of confidence. The intersection over union (IoU) ratio between the current rotated bounding box and the subsequent rotated bounding boxes can be calculated sequentially. If the IoU exceeds the threshold and the confidence of the current rotated bounding box is higher, the subsequent rotated bounding boxes are filtered. This process is repeated until all rotated bounding boxes have been processed, and the unfiltered rotated bounding boxes are retained as the object detection results.

[0101] For example, in the remote sensing image detection scenario described above, the "vehicle" category set contains rotated bounding boxes detected at three scales: high, medium, and low. The IoU between the high-scale and medium-scale rotated bounding boxes exceeds a threshold. Assuming the confidence level of the high-scale rotated bounding boxes is higher than that of the medium-scale ones, the medium-scale rotated bounding boxes are filtered out. Ultimately, rotated bounding boxes at both high and low scales are obtained.

[0102] Based on the above embodiments disclosed in this application, by grouping the detection results of the rotating bounding box parameters at different scales according to categories, and performing cross-scale non-maximum suppression on the detection results of the rotating bounding box parameters of each category, targeted optimization can be performed for different categories of targets, reducing redundant rotating bounding boxes caused by repeated detection in multi-scale detection, thereby obtaining more accurate and concise target detection results.

[0103] This application provides a training method. The training method can be applied to computer devices, which can refer to devices with data processing capabilities such as servers, laptops, tablets, desktop computers, smart TVs, set-top boxes, and mobile devices (e.g., mobile phones, portable video players, personal digital assistants, dedicated messaging devices, portable gaming devices).

[0104] Figure 7 This is a schematic diagram of the first process of a training method provided in an embodiment of this application. The method is used to train an image detection model, which includes a multi-scale feature extraction network, a self-attention network, a multi-scale feature generation module, and a multi-scale prediction network as described in the above embodiments. Figure 7 As shown, the training method includes the following steps S701 to S704: Step S701: Extract features from the sample images based on the pre-trained multi-scale feature extraction network to obtain sample fusion features.

[0105] Pre-training refers to the process of pre-training a neural network model on a large-scale general dataset, enabling the model to initially learn general feature extraction capabilities and parameter configurations. Subsequent fine-tuning can then adapt the model to specific tasks, reducing training costs for specific tasks and improving training efficiency and performance. In this application, the multi-scale feature extraction network of the image detection model is pre-trained. The sample images are labeled image data used to train the image detection model, containing the target object and its corresponding standard annotation information. These serve as the basic input for the model to learn the correspondence between features and targets, and in this application, they belong to the same scene or target object category as the image to be detected.

[0106] In some embodiments, sample images can be acquired for model training to ensure that the sample images are consistent with the scene and target type detected by the model subsequently. Simultaneously, data augmentation processing can be performed on the sample images, employing techniques such as random cropping, flipping, rotation (0°~360°), brightness adjustment, and scaling (0.5~2.0 times) to improve the network's generalization ability to targets rotated at different scales and angles. During the training phase, sample images of three sizes—416×416, 640×640, and 896×896—can be randomly input to further enhance the multi-scale adaptation capability of the multi-scale feature extraction network. The system then uses a multi-scale feature extraction network pre-trained on a large-scale general dataset. The pre-processed and enhanced sample images are input into this network. The network extracts local detail features such as edges and textures from the sample images through shallow convolutional and pooling layers, and extracts semantic features through deep convolutional and fully connected layers. Different network layers correspond to different receptive fields, enabling the extraction of features at different scales. Finally, a feature fusion strategy is employed to concatenate the extracted features at different scales, assign weights, and integrate them to obtain fused sample features, which are then input into subsequent network layers for processing.

[0107] For example, in the training scenario of the UAV wind power inspection image detection model, the sample images are labeled images of wind power equipment taken by UAV, labeled with standard information such as tilted blades and corresponding rotation angles and deformation defects. During the training phase, sample images of three sizes, 416×416, 640×640, and 896×896, are randomly input, and data augmentation processing such as random rotation (0°~360°) and scaling (0.5~2.0 times) is performed on the sample images. A multi-scale feature extraction network with a Convolutional Neural Network (CNN) architecture pre-trained on a general image dataset is used. The processed sample images are input into the network. The shallow network extracts local detail features such as blade edges and textures (including details related to minor deformation defects) through 3×3 convolutional kernels, and the deep network extracts semantic features such as the overall shape of the blades and the relative position of the blades and the wind turbine nacelle through 7×7 convolutional kernels. Then, the shallow and deep features are integrated through channel fusion to obtain the sample fusion features.

[0108] Step S702: Perform global association processing on the sample fusion features based on a self-attention network to obtain the first sample image features.

[0109] In some embodiments, the obtained sample fusion features can be dimensionally adjusted to be transformed into a feature format acceptable to the self-attention network, and the sample fusion features can be mapped into query vectors, key vectors, and value vectors. Then, the similarity between the query vector and the key vector is calculated to obtain the attention weight matrix, which reflects the correlation strength between each position of the sample fusion features. The attention weight matrix and the value vector are then weighted and summed to achieve global association and integration of the sample fusion features, resulting in the first sample image features.

[0110] For example, based on the above-mentioned training scenario of the UAV wind power inspection model, the obtained sample fusion features carry local detail features such as blade edges and textures (including details of minor deformation defects), as well as semantic features of the overall wind power equipment scene. The sample fusion features are input into a self-attention network, which calculates the attention weights of the local features of the blade edges, the overall morphological features of the blade, and the background features of the wind turbine nacelle. It explores the correlation between the blade edges and blade textures, and between the local deformation areas of the blade and the overall blade. Global correlation processing is achieved through weighted summation to obtain the first sample image features, which carry the correlation between the local features of the blade and the global features. The parameters of the self-attention network will be iteratively adjusted in subsequent training using the AdamW optimizer (initial learning rate 1e-4, weight decay coefficient 5e-4) to gradually optimize the extraction accuracy of the correlated features.

[0111] Step S703: Process the first sample image features based on the sample feature generation module to obtain second sample image features at least two scales.

[0112] Here, the second sample image features are sample feature data at multiple different scales obtained after processing by the sample feature generation module. Each scale of the second sample image feature retains the local and global correlations in the first sample image features. For example, the second sample image features correspond to preset scales such as P8, P16, and P32, and corresponding weights are assigned to different scales (P8: 0.5, P16: 0.3, P32: 0.2) to adapt to the small target detection priority.

[0113] Step S704: Target detection is performed on the second image features corresponding to each scale based on the sample prediction network, and the sample detection results corresponding to each scale are obtained and fused to obtain the target sample detection result of the target object.

[0114] It is understood that steps S702 to S704 can be implemented in a similar manner to steps S102 to S104, and the implementation of steps S102 to S104 can be referred to during implementation.

[0115] Step S705: Determine the loss value based on the loss function, the target sample detection result, and the standard result corresponding to the sample image, and adjust the model parameters of the image detection model based on the loss value.

[0116] The loss function measures the difference between the model's predictions and the actual results. In this step, a multi-scale joint loss function is used. The class prediction branch at each scale uses cross-entropy loss, and the rotated box parameter prediction branch uses smooth L1 loss. The weight ratio of class loss to parameter loss is 1:3, and weights are assigned to different scales (P8:0.5, P16:0.3, P32:0.2) to adapt to the priority of small object detection. The target sample detection results and standard results are input into the loss function, and the loss value quantifying the deviation between the two is calculated through the function. Training uses the AdamW optimizer (initial learning rate 1e-4, weight decay coefficient 5e-4), with a batch size of 16 and 120 iterations. The first 30 iterations use a learning rate warm-up, and the last 90 iterations use a cosine annealing strategy to reduce the learning rate. The algorithm determines whether the loss value has reached the preset convergence threshold. If not, it uses backpropagation based on the loss value to adjust the model parameters of each network layer, from the sample prediction network, sample feature generation module, self-attention network to multi-scale feature extraction network, to reduce the feature extraction and prediction bias of each layer. The above process is repeated with a training batch size of 16, 120 iterations, and a learning rate warm-up strategy for the first 30 iterations and a cosine annealing strategy for the last 90 iterations, until the loss value reaches the preset convergence threshold or the preset number of training rounds is completed, resulting in an image detection model with optimized parameters.

[0117] Here, model parameters refer to the learnable parameters of each network layer in the image detection model, including convolutional kernel weights, fully connected layer weights, bias terms, etc. The values ​​of these parameters determine the model's feature extraction and prediction capabilities. Adjustments during training can gradually optimize the model and improve prediction accuracy. In some embodiments, since the multi-scale feature extraction network is pre-trained, its model parameters may not need to be adjusted during training; alternatively, the model parameters of the image detection model (including the multi-scale feature extraction network) are adjusted synchronously.

[0118] For example, in the training scenario of the UAV wind power inspection model, the target sample detection results include the category, location coordinates, confidence level, and rotation angle information of the tilted blade and the blade's minor deformation defects. The standard results corresponding to the sample images are the manually labeled true blade category, true location coordinates, true deformation defect information, and actual blade rotation angle. A multi-scale joint loss function is called, in which the category prediction branch of each scale uses cross-entropy loss, the rotation box parameter prediction branch uses smooth L1 loss, the weight ratio of category loss to parameter loss is 1:3, and the weight of P8 scale (small target, adapted to blade minor deformation defects) is 0.5, and the weight of P16 scale (adapted to the tilted blade body) is 0.3. The target sample detection results and standard results are input into the function to calculate the loss value. The training uses the AdamW optimizer (initial learning rate 1e-4, weight decay coefficient 5e-4), the training batch is set to batch size = 16, the number of iterations is 120 rounds, the first 30 rounds use learning rate warm-up, and the last 90 rounds use cosine annealing strategy to reduce the learning rate. If the loss value does not reach the preset convergence threshold, the model adjusts the convolutional kernel weights of the sample prediction network, the sampling parameters of the sample feature generation module, the attention weight calculation parameters of the self-attention network, and the convolutional layer weights of the multi-scale feature extraction network based on the loss value through the backpropagation algorithm, thereby reducing the prediction bias of the model for tilted blades and minor deformation defects. The model is trained repeatedly according to the above training parameters and strategies until the loss value converges and the model parameters are adjusted. At this point, the model's detection accuracy for tilted blades and minor deformation defects in wind power inspection gradually increases to the preset standard.

[0119] Based on the embodiments disclosed in this application, by combining a pre-trained multi-scale feature extraction network with data augmentation and multi-size input strategies, and by utilizing a self-attention network to mine global feature correlations, the model's ability to extract features from targets at different scales and angles can be improved, enhancing the integration effect of local and global features. Simultaneously, multi-scale second sample image features are generated and weights are assigned through a sample feature generation module, and a multi-scale joint loss function combined with an optimizer and learning rate adjustment strategy is used to optimize model parameters. This can adapt to the priority of small target detection, quantify prediction bias, and gradually reduce feature extraction and prediction bias, thereby improving the accuracy and comprehensiveness of the image detection model in detecting target objects.

[0120] In some embodiments disclosed in this application, the multi-scale prediction network includes a prediction sub-network corresponding to each scale; the step of determining the loss value based on the loss function, the target sample detection result, and the standard result corresponding to the sample image includes: determining a sub-loss value corresponding to each scale based on the loss function, the target sample detection result corresponding to each scale, and the standard result corresponding to each scale; determining the loss value based on the sub-loss value corresponding to each scale and the loss weight corresponding to each scale; the loss weight decreases as the scale decreases.

[0121] Here, a multi-scale prediction network is a neural network structure used to process feature maps of different sizes. By setting up an independent prediction subnetwork for each scale, the multi-scale prediction network can perform high-precision target detection on both small and large target samples. This design can effectively improve the target detection performance of the multi-scale prediction network at different scales, especially in rotated target detection tasks, where it can better capture the features of target samples with different angles and sizes. A prediction subnetwork is an independent prediction module designed for a specific scale feature map, typically including a class prediction branch and a rotation box parameter prediction branch. The prediction subnetwork can adjust its internal parameter configuration according to the scale of the input feature map to adapt to target samples of different sizes. For example, at a 1 / 8 scale, the prediction subnetwork can focus on detecting small target samples; while at a 1 / 32 scale, it is more suitable for detecting large target samples.

[0122] In some embodiments, the loss function is a mathematical function used to measure the difference between the output of the multi-scale prediction network and the true label. In this embodiment, the loss function is applied to the prediction sub-network at each scale, and the sub-loss value at each scale is calculated by quantifying the difference between the target sample detection result output by the multi-scale prediction network and the standard result.

[0123] In some embodiments, the target sample detection result is the output of a multi-scale prediction network after detecting target samples in an image at the current scale, including class prediction and bounding box parameter prediction. The standard result is pre-labeled ground truth data containing the target sample's class and accurate bounding box parameters. By comparing the target sample detection result with the standard result, the performance of the multi-scale prediction network in target detection at the current scale can be evaluated.

[0124] In some embodiments, the sub-loss value is a quantified representation of the prediction error at each scale. By calculating the sub-loss value at each scale, the parameters of the multi-scale prediction network can be tuned more finely, optimizing the target detection capability of the multi-scale prediction network at different scales. The sub-loss value can reflect the target detection quality of the multi-scale prediction network at different scales, thereby guiding the formulation of subsequent weight allocation and optimization strategies.

[0125] In some embodiments, the loss weight is a coefficient assigned to each scale to adjust the degree of influence of different scales on the total loss value. In this embodiment, smaller scales (e.g., 1 / 32) are used to detect large target samples, where the detection task is less difficult, so the loss weight for this scale can be appropriately reduced; while larger scales (e.g., 1 / 8) are used to detect small target samples, where the detection task is more difficult, so the loss weight for this scale can be appropriately increased. The purpose of this setting is to make the multi-scale prediction network pay more attention to those scales with higher detection difficulty during training, thereby improving the overall target detection accuracy.

[0126] The sub-loss values ​​include a category loss value and a rotated bounding box loss value. Determining the sub-loss value for each scale based on the loss function, the target sample detection result for each scale, and the standard result for each scale includes: determining the category loss value for each scale based on the category loss function, the sample category detection result for each scale, and the standard category result for each scale; determining the rotated bounding box loss value for each scale based on the rotated bounding box detection result for each scale, and the standard rotated bounding box result for each scale; and performing a weighted fusion of the category loss value and the rotated bounding box loss value for each scale to obtain the sub-loss value for each scale; the weight of the rotated bounding box loss value is greater than the weight of the category loss value.

[0127] In some embodiments, a category loss function is used to calculate the error of a multi-scale prediction network in a classification task. In this embodiment, the category loss function is applied to the prediction sub-network at each scale. By quantifying the difference between the sample category detection result output by the multi-scale prediction network and the standard category result, the category loss value for each scale can be calculated. The sample category detection result is the output of the multi-scale prediction network after classifying the target samples in the image at the current scale. The standard category result is a pre-labeled true category label. The category loss value measures the performance of the multi-scale prediction network in the classification task, i.e., the difference between the target sample category output by the multi-scale prediction network and the actual category. For example, the category loss function can be set to a cross-entropy loss function, etc. The smaller the category loss value, the better the performance of the multi-scale prediction network in the classification task.

[0128] In some embodiments, the rotation box loss function is used to calculate the error of the multi-scale prediction network in the regression task. In this embodiment, the rotation box loss function is applied to the prediction sub-network at each scale. The rotation box loss value for each scale is calculated by quantifying the difference between the sample rotation box detection results output by the multi-scale prediction network and the standard rotation box results. The sample rotation box detection results are the output of the multi-scale prediction network after regressing the target samples in the image at the current scale, including parameters such as center point coordinates, length and width dimensions, and rotation angle. The standard rotation box results are pre-labeled true rotation box parameters. By comparing the sample rotation box detection results with the standard rotation box results, the system can evaluate the regression performance of the multi-scale prediction network at the current scale. The rotation box loss value measures the performance of the multi-scale prediction network in the regression task, i.e., the difference between the rotation box parameters (center point coordinates, length and width dimensions, rotation angle) output by the multi-scale prediction network and the actual parameters. For example, the rotation box loss function can be set to a smooth L1 loss function, etc. The smaller the rotation box loss value, the better the performance of the multi-scale prediction network in the rotation box parameter determination task.

[0129] In some embodiments, the calculation of the rotation box loss value typically involves first normalizing the parameters of the multi-scale prediction network output using functions such as Sigmoid or Tanh, and then using the smooth L1 loss function or other regression loss functions to calculate the difference between the multi-scale prediction network output and the true parameters.

[0130] In practical implementation, the system combines the class loss value and the rotated box loss value into a sub-loss value, where the class loss value and the rotated box loss value are used to reflect the performance of the multi-scale prediction network in classification and regression tasks, respectively. By reasonably allocating the weights of these two types of loss values, the overall performance of the multi-scale prediction network can be effectively optimized.

[0131] In this embodiment, the weight of the rotated box loss value is set to be greater than that of the class loss value. This is because the accuracy of the rotated box parameters is crucial for object detection tasks, especially in rotated object detection, where even a small error in the rotation angle can significantly reduce the intersection-over-union (IoU) ratio between the detected box and the ground truth object sample. Therefore, assigning a higher weight to the rotated box loss value helps improve the performance of the multi-scale prediction network in regression tasks, thereby improving the overall object detection accuracy. Simultaneously, by weighting and fusing the class loss value and the rotated box loss value, the object detection performance of the multi-scale prediction network at each scale can be comprehensively evaluated. This weighted fusion method not only considers the importance of classification and regression tasks but also allows for flexible adjustment of the weight ratio of these two types of losses according to task requirements, thus achieving more refined model optimization.

[0132] In this embodiment, a multi-scale prediction network is introduced, with an independent prediction sub-network set up for each scale. Weighted fusion of class loss and rotation box loss values ​​allows for a more accurate evaluation of the target detection performance of the multi-scale prediction network at different scales. This method improves the target detection accuracy of the multi-scale prediction network at different scales, thus better adapting to rotating target detection tasks in various complex scenarios.

[0133] The following describes the application of the target detection method provided in the embodiments of this application in a real-world scenario.

[0134] This application discloses a rotating target detection network that integrates a C2F module and a self-attention mechanism. Based on a Unet-like structure, the network innovatively constructs a four-level feature processing chain: "local feature enhancement - global correlation modeling - multi-scale feature pyramid generation - accurate parameter regression". The front end relies on the C2F module to build a convolutional neural network (CNN) feature extraction branch to achieve efficient capture of multi-scale local details. The back end introduces multiple sets of cascaded self-attention blocks to build a global feature correlation branch to explore long-distance dependencies between targets. After global correlation modeling, a feature pyramid generation stage is added, using the ViTDet feature pyramid construction method to generate multi-scale feature levels. Finally, a dedicated rotation box prediction head adapted to different scale features outputs the target category and rotation detection box parameters, overcoming the core bottlenecks of imbalance between local details and global correlation and poor adaptability of single-scale feature prediction in existing technologies.

[0135] Specifically, please refer to Figure 8 The network execution flow is as follows: 1. Multi-scale local feature extraction: The input image 610 is fed into a feature extraction network composed of stacked C2F modules. Four downsampling operations reduce the feature map scale to 1 / 32 of the original image (e.g., ...). Figure 8 Feature map D5), while retaining feature maps from each downsampling stage (such as...). Figure 8 Feature maps D2 to D4 in the image are used as multi-scale skip connections; 2. Feature fusion and detail enhancement: For the deepest feature map (e.g., ... Figure 8 The feature map D5 in the image is upsampled to gradually restore it to 1 / 4 of the original image scale (e.g., ...). Figure 8 The feature map U3 in the middle is accurately spliced ​​and fused with the feature map of the corresponding downsampling stage during the upsampling process to achieve complementary enhancement of shallow detail features (such as target edges and angle textures) and deep semantic features; 3. Compression layer: Through the compression layer, 1 / 4 of the scale (e.g., Figure 8 The feature map U3 in the image is compressed to 1 / 8 (e.g., ...). Figure 8 The feature map F_cnn in the image.

[0136] 4. Global correlation modeling: merging features at a 1 / 8 scale (such as...) Figure 8 The feature map F_cnn is input into multiple sets of cascaded self-attention blocks 620. Through the attention mechanism, the key regions of the rotating target are accurately focused, and long-distance dependencies between different targets and between the target and the background are established, effectively distinguishing the feature boundaries of densely distributed targets.

[0137] 5. Multi-scale feature pyramid generation: The feature pyramid construction strategy of ViTDet is adopted to perform multi-scale splitting and hierarchical enhancement on the features after global correlation modeling. Through operations such as convolutional dimensionality reduction and cross-layer fusion, feature pyramids 630 with multiple scales such as 1 / 8, 1 / 16, and 1 / 32 are generated, so that rotating targets of different scales (from small parts to large buildings) can obtain suitable feature representations. 6. Precise Regression of Rotated Box Parameters: Configuring a dedicated prediction head for each scale level of the feature pyramid (e.g., ... Figure 8 The prediction network 640 predicts the class probability and rotation box parameters (center point coordinates, length and width, rotation angle) of the target of the corresponding size based on the features of each scale, and outputs a rotation detection box that can accurately surround tilted targets of different scales.

[0138] The innovation of this application lies in: (1) The advantages of local multi-scale feature fusion of C2F module are deeply coupled with the global association capture capability of self-attention, and the accurate matching and complementarity of multi-scale features are achieved through Unet-like jump connection structure; (2) The innovative introduction of the ViTDet-style feature pyramid generation process solves the problem that single-scale features are difficult to adapt to rotating targets of different sizes after global association; (3) Simplify the feature fusion link and pyramid construction logic to avoid network structure redundancy.

[0139] This design overcomes the triple technical challenges of "loss of local details", "insufficient global correlation" and "poor adaptability to multi-scale targets" in traditional rotating target detection. While significantly improving the positioning accuracy of the rotating box and the accuracy of category recognition, it ensures the inference efficiency of the network and meets the high precision and real-time requirements of scenarios such as industrial quality inspection and remote sensing monitoring.

[0140] The following details the rotating target detection network proposed in this application. The rotating target detection network consists of four core modules: a C2F feature extraction module (Unet-like structure), a self-attention global association module, a ViTDet-style multi-scale feature pyramid generation module, and a prediction head module. The structure, connections, and workflow of each module are as follows: 1. Overall network structure framework.

[0141] The input to the rotating target detection network is an RGB image of arbitrary size (e.g., 640×640 pixels), and the output is a target category vector and a rotation box parameter vector (center point coordinates, length and width, rotation angle) at different scales. The overall network structure is as follows: image preprocessing layer → C2F feature extraction module (including downsampling unit → upsampling unit) → feature compression layer → self-attention global association module → ViTDet-style multi-scale feature pyramid generation module → multi-scale prediction head module. The layers are connected by batch normalization (BN) and ReLU activation function to improve training stability and feature nonlinearity expression capability. Among them, the feature pyramid module is the core unit, realizing multi-scale adaptation of globally associated features and solving the problem of single-scale feature adaptability for rotating targets of different sizes.

[0142] 2. C2F feature extraction module (Unet-like structure).

[0143] The C2F feature extraction module is the core local feature extraction unit of this network. It adopts a Unet-like "downsampling-upsampling" symmetrical structure, consisting of 5 downsampling units (C2F-D1~C2F-D5), 3 upsampling units (C2F-U1~C2F-U3), and a feature fusion unit. The specific structure and parameters are as follows: (1) Downsampling unit (feature encoding).

[0144] The core of the downsampling unit is the C2F module. Each C2F module contains three cascaded 3×3 convolutional layers, one 1×1 convolutional compression layer, and shortened residual connections. It processes features sequentially through "convolution-normalization-activation." The four downsampling units progressively reduce the feature map scale, thereby increasing the receptive field. Specific parameter settings are as follows: C2F-D1: The input is a preprocessed 640×640×3 image, which is downsampled to 320×320×64 after a 3×3 convolution (stride 2), and the output feature map is denoted as F1; C2F-D2: The input is F1, which is downsampled to 160×160×128 by a 3×3 convolution (stride 2), and the output feature map is denoted as F2; C2F-D3: The input is F2, which is downsampled to 80×80×256 by a 3×3 convolution (stride 2), and the output feature map is denoted as F3; C2F-D4: The input is F3, which is downsampled to 40×40×512 by a 3×3 convolution (stride 2), and the output feature map is denoted as F4.

[0145] C2F-D5: The input is F4, which is downsampled to 20×20×1024 by a 3×3 convolution (stride 2), and the output feature map is denoted as F5 (the deepest feature with the largest receptive field).

[0146] In each downsampling unit, the C2F module enhances feature extraction capabilities through cascaded convolutions, residual connections prevent gradient vanishing, and 1×1 convolutional layers compress the number of feature channels to twice that of the previous layer, balancing feature representation and computational efficiency.

[0147] (2) Upsampling unit (feature decoding).

[0148] The upsampling unit uses transposed convolution to restore the feature map scale. The feature map output by each upsampling unit is fused with the feature map output by the corresponding downsampling unit (at the same scale). The specific process and parameters are as follows: C2F-U1: The input is F5 (20×20×1024), which is upsampled to 40×40×512 by a 2x transposed convolution (stride 2, kernel 3×3). It is then fused with F4 (40×40×512) output by the downsampling unit C2F-D4 through channel concatenation. The number of channels in the fused feature map is 1024. After compression to 512 by a 1×1 convolution, the output feature map is denoted as U1. C2F-U2: The input is U1 (40×40×512), which is upsampled to 80×80×256 by a 2x transposed convolution. It is then concatenated and fused with F3 (80×80×256) output by C2F-D3, and the compressed output feature map is U2 (80×80×256). C2F-U3: The input is U2 (80×80×256), which is upsampled to 160×160×128 after being transposed by 2x convolution. It is then concatenated and fused with F2 (160×160×128) output by C2F-D2, and the compressed output feature map U3 (160×160×128) is then produced. (3) Feature compression layer.

[0149] To focus on core local features and improve the computational efficiency of the subsequent self-attention module, the U4 feature map (160×160×128) output by C2F-U3 is downsampled to 80×80×512 (i.e., 1 / 8 scale of the input image) through one 3×3 convolution (stride 2). This feature map is denoted as F_cnn and serves as the final output of the C2F feature extraction module, which is then used to connect to the self-attention global association module.

[0150] 3. Self-attention global association module This module takes the F_cnn feature map (80×80×512) as input and constructs global feature associations through four sets of cascaded self-attention blocks. Each block contains a multi-head self-attention (MHSA) unit and a feed-forward network (FFN). The specific structure is as follows: (1) Feature Dimension Transformation First, convert F_cnn (H×W×C, where H=80, W=80, C=512) from "spatial dimension" to "sequence dimension": expand the spatial pixels of H×W into a feature sequence with a sequence length of L=H×W=6400. The converted feature format is L×C (6400×512), denoted as F_seq.

[0151] (2) Multi-head Self-Attention (MHSA) Unit Each MHSA unit in the self-attention block divides F_seq into 8 attention heads (head=8), with each head having a feature dimension of C / head=64. Attention weights are obtained through the calculation of the query vector (Q), key vector (K), and value vector (V). Q, K, and V are generated by linear transformation: Q = F_seq × W_Q, K = F_seq × W_K, V = F_seq × W_V, where W_Q, W_K, and W_V are learnable parameter matrices, each with a dimension of 512 × 512. Calculate attention score: Score = Q × K T / √d_k (d_k is the dimension of each attention head, here d_k=64), the attention weight W_att is obtained by normalizing the score through the Softmax function; Generate attention features: F_att = W_att × V. Concatenate the channels of F_att from the 8 attention heads and restore them to 512 dimensions through linear transformation to obtain the MHSA unit output F_mhsa.

[0152] (3) Feedforward Network (FFN) and Residual Connection The FFN unit adopts a "linear transformation-activation-linear transformation" structure: first, F_mhsa is increased in dimension to 2048 through a linear layer, then activated by the GELU function, and finally reduced in dimension to 512 through a linear layer to obtain F_ffn. To avoid gradient vanishing during training, both the MHSA and FFN units have residual connections, i.e., F_block = F_input + F_ffn (where F_input is the input feature of the current block).

[0153] After concatenating 4 self-attention blocks, the final output global correlation feature is denoted as F_selfatt, which is still in L×C (6400×512) format. It is then converted back to spatial dimension (80×80×512) and used as input to the ViTDet-style multi-scale feature pyramid generation module.

[0154] 4. ViTDet-style multi-scale feature pyramid generation module This module is the core innovation unit. Taking F_selfatt (80×80×512) as input, it employs ViTDet's feature pyramid construction strategy. Through "convolutional dimensionality reduction - cross-layer fusion - scale splitting," it generates multi-scale feature layers adapted to rotating targets of different sizes. Specific processes and parameters are as follows: P8 layer (1 / 8 scale, adapted for small targets): directly uses F_base (80×80×512), denoted as P8, which corresponds to the 1 / 8 scale of the input image and is adapted for small rotating targets (such as tiny parts on an industrial assembly line). P16 layer (1 / 16 scale, adapted to medium-sized targets): F_base is downsampled to 40×40×1024 by 3×3 convolution (stride 2), and then compressed to 512 channels by 1×1 convolution, which is denoted as P16. It corresponds to the 1 / 16 scale of the input image and is adapted to medium-sized rotating targets (such as aerial vehicles). Layer P32 (1 / 32 scale, suitable for large targets): P16 is downsampled to 20×20×2048 by 3×3 convolution (stride 2), and then compressed to 512 channels by 1×1 convolution, which is denoted as P32. It corresponds to the 1 / 32 scale of the input image and is suitable for large rotating targets (such as remote sensing buildings).

[0155] 5. Multi-scale prediction head module The prediction head module is configured with an independent parallel dual-branch prediction head for each scale level of the feature pyramid, which respectively implements "target category prediction" and "rotation box parameter prediction". The input is {P8, P16, P32} of the feature pyramid, and the output is the target detection result at the corresponding scale: (1) Universal prediction head structure (adaptable to various scales) Each prediction head at each scale employs a "convolutional augmentation-branch prediction" structure, adjusting the convolution stride only according to the feature map scale, with a unified core logic: Feature enhancement layer: Two 3×3 convolutions (stride 1). The first convolution compresses the number of input feature channels from 512 to 256, and the second convolution keeps the number of channels unchanged to enhance feature expression. Parallel branching: The category prediction branch and the rotated bounding box parameter prediction branch are output in parallel to avoid feature interference.

[0156] (2) Target category prediction branch (applicable to all scales) This branch outputs the class probabilities at the corresponding scale using a "convolution-global average pooling-fully connected" structure: One 3×3 convolution (stride 1) reduces the number of enhanced feature channels from 512 to 256. Global average pooling compresses the feature map into a 1×1×256 vector; A 2-layer fully connected network: The first layer maps a 256-dimensional vector to 256 dimensions. After ReLU activation, the second layer maps it to N dimensions (N is the number of target categories, such as N=3 when detecting vehicles, pedestrians, and buildings). The Softmax function outputs the probability value of each category, and the category with the highest probability is the target prediction category at that scale.

[0157] (3) Rotated frame parameter prediction branch (applicable to all scales) This branch performs regression prediction on the three core parameters of the rotated bounding box (center point coordinates (x, y), length and width dimensions (w, h), and rotation angle θ) (replacing the redundant parameters of the original left / top / right / bottom to improve regression accuracy), and its structure is as follows: One 1×1 convolution: Converts the number of enhanced feature channels from 256 to 5 (to be compatible with the original parameter conventions, or to simplify to 3-dimensional core parameters), and outputs the parametric feature map at the corresponding scale (80×80×5 for P8, 40×40×5 for P16, and 20×20×5 for P32). Parameter decoding: Center point coordinates (x,y), length and width dimensions (w,h): normalized to the [0,1] interval by the Sigmoid function, and then multiplied by the feature map width and height at the corresponding scale (restored to the input image scale of 640×640 pixels). Rotation angle θ: The actual rotation angle of the target is obtained by normalizing to the interval [-π / 2, π / 2] using the Tanh function.

[0158] (4) Multi-scale result fusion The prediction results at the three scales {P8, P16, P32} are summarized, and duplicate detection boxes are filtered out by cross-scale nonmaximum suppression (NMS). The final output is a rotating target detection result that takes into account small / medium / large sizes.

[0159] 6. Network training parameter settings To ensure network convergence and detection performance, the training parameters are set as follows: Optimizer: The AdamW optimizer is used, with an initial learning rate of 1e-4 and a weight decay factor of 5e-4. Loss function: A multi-scale joint loss function is adopted. The category prediction branch of each scale uses cross-entropy loss, and the rotation box parameter prediction branch uses smoothL1 loss. The weight ratio of category loss to parameter loss is 1:3, and weights are assigned to different scales (P8:0.5, P16:0.3, P32:0.2) to adapt to the priority of small target detection. Training batch: batchsize=16, number of iterations is 120 rounds, the first 30 rounds use learning rate warm-up, and the last 90 rounds use cosine annealing strategy to reduce the learning rate; Data augmentation: During the training process, augmentation techniques such as random cropping, flipping, rotation (0°~360°), brightness adjustment, and scale scaling (0.5~2.0 times) are used to improve the network's generalization ability to targets rotated at different scales and angles. Multi-scale training: During the training phase, images of three sizes, 416×416, 640×640, and 896×896, are randomly input to further enhance the multi-scale adaptation capability of the feature pyramid.

[0160] The following are the core innovations of this application. Addressing the core pain points of rotating target detection—namely, "loss of local details, insufficient global correlation, poor multi-scale adaptation, and imbalance in real-time performance"—this application innovatively constructs a technical system of "local enhancement - global modeling - multi-scale adaptation - lightweight collaboration." The core innovations focus on two dimensions: module collaborative design and feature fusion strategies, as detailed below: 1. C2F-Unet Fusion Local Feature Extraction Architecture: This innovative architecture deeply couples the C2F module with a Unet-like symmetric structure. It enhances local feature representation through cascaded convolutions and residual connections. By leveraging the "downsampling encoding-upsampling decoding" skip-connection fusion mechanism, it accurately preserves key details such as the edge contours and angular textures of rotated targets. At the same time, it outputs 1 / 8 scale core features through a feature compression layer, which focuses on effective information while reducing computational redundancy for subsequent global modeling, laying the foundation for high-precision detection of local features.

[0161] 2. CNN and Self-Attention Bidirectional Collaborative Mechanism: A collaborative paradigm of "local features guiding global modeling" is proposed. Fine-grained local features extracted by the C2F module are directly input into the self-attention module. Multiple sets of cascaded multi-head self-attention blocks are used to mine long-distance dependencies between targets and between targets and the background. This design breaks the inherent contradiction between the traditional "CNN is strong locally / weak globally" and "pure self-attention is strong globally / weak locally," achieving accurate matching between local details and global correlations, and significantly improving detection robustness in densely distributed, overlapping, and occluded scenes.

[0162] 3. ViTDet-style multi-scale feature pyramid adaptation design: After global correlation modeling, a feature pyramid generation stage is added. By drawing on ViTDet's cross-layer fusion strategy, a multi-scale feature hierarchy is generated to adapt to small / medium / large rotating targets. The complementarity between scales is strengthened through a top-down feature enhancement mechanism. Combined with a multi-scale dedicated prediction head and a cross-scale NMS fusion strategy, the problem of poor adaptability of single-scale features to targets of different sizes is completely solved, and the detection recall rate of small targets and targets at extreme angles is greatly improved.

[0163] 4. Lightweight and Accurate Regression Collaborative Optimization Scheme: On the one hand, through feature compression and simplified fusion link design, the sequence length and network parameter scale of the self-attention module are controlled, improving the inference speed by more than 30% compared with the existing hybrid architecture, meeting the millisecond-level response requirements of industrial quality inspection and real-time monitoring; on the other hand, the rotation box parameter regression logic is optimized by adopting the "core parameters (center point / length / width / angle) + normalized decoding" strategy, combined with a multi-scale weighted loss function, which improves the rotation box enclosing accuracy (IoU) by more than 25% and effectively reduces the angle regression error.

[0164] This application offers the following benefits: through the core design of "C2F-Unet local feature enhancement, CNN-self-attention collaborative modeling, ViTDet multi-scale adaptation, and lightweight accurate regression," a high-precision and real-time balanced rotating target detection network is constructed. Compared with existing technologies, it achieves significant breakthroughs in technical performance, economic value, and social applications. The specific benefits are as follows: 1. Significant leap in technical performance, with core indicators leading across the board. Significantly improved detection accuracy: In tests on mainstream public datasets for rotating target detection such as DOTA and HRSC2016, the network's mean average accuracy (mAP) reached 82.3%, an improvement of 9.6 percentage points compared to the traditional ResNet+FPN method; among them, the mAP for small rotating targets (pixel area <100) improved by 12.1 percentage points, effectively solving the problem of missing small targets; the mean angle prediction error (MAE) was reduced to 0.8°, far better than the 1.5° of existing methods, and the integration accuracy (IoU) of the rotating box for tilted targets was improved by more than 25%.

[0165] Precise optimization of inference efficiency: Through lightweight design such as feature compression and link simplification, the number of network parameters is strictly controlled at 46.2M; the inference speed reaches 28FPS on NVIDIA RTX 3090 graphics card, which is 20% higher than the Transformer-based rotation detection methods such as Deformable DETR, and fully meets the millisecond response requirements of industrial quality inspection, real-time monitoring and other scenarios.

[0166] Enhanced scenario generalization capability: Validated across multiple scenarios including remote sensing images, industrial production line quality inspection maps, and aerial photography, the network mAP fluctuation is less than 3%; its adaptability to complex environments such as sudden changes in lighting, dense overlapping of targets, and extreme angle tilt is significantly improved, allowing it to stably adapt to different application scenarios without additional fine-tuning.

[0167] 2. Its economic value is evident, and its cost reduction and efficiency improvement results are remarkable. Reduced hardware deployment costs: The lightweight architecture design significantly reduces the reliance on hardware computing power, enabling real-time detection without the need for high-end GPUs. This reduces hardware investment costs for scenarios such as industrial quality inspection equipment and drone inspection systems by 20%-30%, significantly lowering the barrier to technology implementation.

[0168] Save on labor and operating costs: High-precision detection results significantly reduce the workload of manual review. For example, in scenarios such as target annotation in remote sensing images and defect detection of industrial parts, the manual correction rate is reduced from 15% to 7% of existing methods, improving work efficiency by more than 3 times and significantly reducing labor costs.

[0169] Improving industrial production efficiency: In industrial quality inspection scenarios, precise rotating target detection can identify minute defects in advance, reducing product defect rate by 5%-10%; at the same time, the automation level of the inspection process is improved, and the production cycle is accelerated by 10%, helping the industry to improve quality and efficiency.

[0170] 3. Wide range of social applications, empowering intelligent upgrades across multiple industries. Facilitating Smart Land and Emergency Management: In the fields of drone inspection and remote sensing mapping, this network can efficiently and accurately identify rotating targets such as surface buildings, roads, and disaster areas, improving geographic information collection efficiency by more than 20%, providing rapid decision support for disaster emergency response and land spatial planning, and promoting the construction of smart land.

[0171] Accelerating the intelligent transformation of the manufacturing industry: Adapting to the real-time detection needs of industrial production lines, it can achieve accurate identification of component posture and defects, replace traditional manual visual inspection, promote the upgrading of the manufacturing industry from "manual inspection" to "intelligent inspection", and fit the development trend of "Industry 4.0".

[0172] Expanding the boundaries of computer vision applications: providing core technical support for emerging scenarios such as aerial surveillance, autonomous driving (tilted obstacle detection), and medical imaging (tilted organ / lesion detection), promoting the large-scale application of rotating target detection technology in multiple fields, and helping the digital economy and the real economy to deeply integrate.

[0173] Please see Figure 9 The diagram illustrates a network structure of a C2f module. Based on this C2f module 90, the input features first pass through a 1×1 convolutional layer (Conv) (parameters: k=1, s=1, p=0), converting the number of channels from c_in to c_out, with an output dimension of h×w×c_out. Through a Split operation, the features are split into two parts, each with a dimension of h×w×0.5c_out (one part directly participates in subsequent concatenation, and the other part enters the Bottleneck module). The split feature then passes through n Bottleneck modules sequentially (the ellipsis in the diagram indicates cascading), with each Bottleneck having an input / output dimension of h×w×0.5c_out; the module supports shortcut residual connections (the diagram indicates that "shortcut=?" indicates that it can be optionally enabled). The "Bottleneck output" and the "feature that did not enter the Bottleneck after splitting" are merged using a Concat operation, resulting in an output dimension of h×w×0.5(n+2)c_out (where n is the number of Bottlens). The concatenated feature is then passed through a 1×1 convolutional layer to compress the number of channels back to c_out, yielding the final output feature h×w×c_out.

[0174] Please see Figure 10 The diagram illustrates the internal structure of a Unet-like module. Image 1010 is input into the network and downsampled to 1 / 32 after five C2F+ convolutions, then upsampled to 1 / 4 after three C2F+ deconvolutional network modules. Specifically, D4 is appended to U1, D3 to U2, and D2 to U1. Finally, U3 passes through a compression layer to obtain F_CNN feature 1020, which is 1 / 8 of the original image.

[0175] Please see Figure 11 This document illustrates a schematic diagram of the internal structure of a Transformer Encoder. In the Transformer Encoder, data first enters a Multi-Head Attention module for attention calculation to capture dependencies between different locations. Subsequently, the output is processed by a normalization layer to standardize the data. Next, the normalized output is added to the initial input through a residual connection to form a residual structure. Afterward, the data flows into a Multi-Layer Perceptron (MLP) for nonlinear transformation; it undergoes normalization again; finally, another residual connection adds the normalized output to the output of the Multi-Head Attention module, completing the data processing flow of one Transformer Encoder layer. This structure may be repeated L times to build deeper networks. In this embodiment, the input can be flattened into N*(W*H)*C (where W*H is the sequence length and C is the feature length) and input into the Transformer Encoder through four Transformer blocks. The Transformer blocks achieve the fusion of global image information and feature enhancement through a combination of "attention + residual + MLP".

[0176] Please see Figure 12 It shows a schematic diagram of the internal structure of a multi-scale feature generation module. The output of the transformer encoder is passed through a convolutional network to obtain a feature scale of 1 / 8. The feature scale of 1 / 8 is downsampled to obtain a feature scale of 1 / 16, and the feature scale of 1 / 16 is downsampled to obtain a feature scale of 1 / 32.

[0177] To fully verify the technical feasibility, performance advantages, and practical application value of this application, the following specific embodiments are set up with the core application scenario of rotating target detection in remote sensing images / industrial scenes (detection categories: vehicles, ships, buildings, industrial parts), and the network training, testing process, and result analysis are explained in detail.

[0178] 1: Dataset Selection and Preprocessing This embodiment combines publicly available authoritative datasets with real-world scenario datasets to ensure the objectivity and generalizability of the test results: 1. Training dataset: The DOTA-v1.5 remote sensing image dataset is used, which contains 17,936 high-resolution remote sensing images (image size 1024×1024~2048×2048 pixels) in 15 categories. Rotating targets (tilt angle 0°~360°) account for 68%, covering typical rotating targets such as vehicles, ships, and buildings. 2. Benchmark dataset: The HRSC2016 ship-specific remote sensing dataset is used, which contains 1061 remote sensing images of rotating ships. The annotation information fully covers the core parameters of the rotating bounding box (center point coordinates, length and width dimensions, rotation angle) and category labels, and is used to quantitatively evaluate the accuracy of angle prediction. 3. Real-world scenario test dataset: 500 images of quality inspection of parts from industrial production lines (including small-sized tilted parts) and 300 images of wind power equipment taken by drones (including large-angle rotating blades) were collected without manual annotation. This dataset is used to verify the network's adaptability to real-world scenarios. 4. Data preprocessing: Input images are uniformly scaled to 640×640 pixels, and enhancement techniques such as random cropping, flipping, rotation (0°~360°), brightness / contrast adjustment, and scaling (0.5~2.0 times) are used; the parameters of the rotated boxes are normalized to ensure the stability of training convergence.

[0179] 2: Test Environment Configuration In this embodiment, both training and testing are performed in the same hardware and software environment to ensure fairness in performance comparison. The specific configuration is as follows: Hardware environment: CPU is Intel i9-12900K (16 cores and 24 threads), GPU is NVIDIA RTX3090 (24GB GDDR6X VRAM), memory is 32GB DDR5, and storage is a 1TB NVMe SSD; Software environment: Operating system is Ubuntu 20.04 LTS, programming language is Python 3.8, deep learning framework is PyTorch 1.12, acceleration libraries are CUDA 11.6 and CUDNN 8.5, and other dependent libraries include OpenCV 4.6, NumPy 1.24, etc.

[0180] 3: Network Training Parameters and Procedures 1. Training parameter settings: The optimizer used is AdamW, with an initial learning rate of 1e-4 and a weight decay coefficient of 5e-4; a multi-scale joint loss function is used, with a weight ratio of 1:3 for class loss (cross-entropy loss) and rotation box parameter loss (smooth L1 loss), and the weights for P4 / P8 / P16 scale losses are 0.5, 0.3, and 0.2, respectively; the training batch size is 16, the total number of iterations is 120, the learning rate is warmed up in the first 30 iterations, and the learning rate is decayed using a cosine annealing strategy in the last 90 iterations; 2. Training process: First, load the ImageNet pre-trained weights to initialize the C2F feature extraction module, and initialize the other modules randomly; execute the process according to "dataset partitioning (training set: validation set = 8:2) → multi-scale training → validation set accuracy monitoring → early stopping strategy (stop training if there is no improvement in accuracy after 10 consecutive rounds) → save the optimal model"; record the loss value, mAP, angle error and other core indicators in real time during the training process.

[0181] 4: Performance Test Results and Quantitative Analysis This embodiment selects five mainstream rotating target detection methods as comparison benchmarks and conducts quantitative tests on the HRSC2016 dataset. The core performance indicators are compared in Table 1 below: Table 1

[0182] Test Result Analysis: 1. Significant accuracy advantages: The network mAP of this application reaches 82.3%, which is 9.6 percentage points higher than the traditional ResNet50+FPN, 3.8 percentage points higher than Deformable DETR, and 3.0 percentage points higher than YOLOv8-OBB; among them, the mAP of small targets (ship pixel area <100) reaches 76.5%, which is 8.7 percentage points higher than YOLOv8-OBB, solving the problem of missed detection of small rotating targets; 2. Accurate Angle Regression: The angle error MAE is reduced to 0.8°, which is 46.7% lower than ResNet50+FPN and 20% lower than YOLOv8-OBB. The average intersection-over-union (IoU) between the rotated box and the real target reaches 78.2%, effectively avoiding the problem of inaccurate bounding caused by angle deviation. 3. Balance between lightweight and real-time performance: The number of parameters is only 46.2M, half that of swin-b and ViTDet; the inference speed reaches 28FPS, which is twice that of swin-b and ViTDet, and similar to DeformableDETR, fully meeting the millisecond-level response requirements of real-time detection scenarios.

[0183] To verify the network's adaptability to real-world deployments, two typical real-world scenarios were selected for testing, and the results are as follows: 1. Drone-based wind power inspection scenario: This system inspects aerial images of wind power equipment with a resolution of 1920×1080, targeting tilted blades (rotation angle 30°~120°). The detection accuracy reaches 91.2%, with angle error controlled within ±1°. It can accurately identify blade rotation status and minor deformation defects; the inference speed reaches 28FPS, adapting to the real-time image transmission and inspection requirements of drones, providing reliable data support for equipment fault early warning. 2. Industrial component quality inspection scenario: Defect detection is performed on small-sized tilted components (bolts, bearings, pixel area 50~200) transported on the production line, with an accuracy rate of 93.5% and a false negative rate of only 2.1%, which is 3.8 percentage points lower than the existing YOLOv8-OBB solution; no high-end GPU is required, and the inference speed reaches 18FPS on a general industrial computer (Intel i7-12700+GTX1660), meeting the real-time quality inspection requirements of the production line.

[0184] To ensure the stable deployment, performance adaptation, and flexible expansion of the network in this application, the following key points should be noted based on the actual application scenario requirements: 1. Input Image Size Adaptation and Parameter Coordination: The input image size of this network can be flexibly adjusted, with 640×640 and 800×800 pixels being preferred (balancing accuracy and speed). When adjusting the size, the convolution stride of the downsampling units (C2F-D1~D5) and upsampling units (C2F-U1~U3) must be matched synchronously to ensure that the feature compression layer outputs a feature map at 1 / 4 scale of the input image (e.g., 800×800 input corresponds to 200×200 output), avoiding abnormal sequence lengths in the global association module and ensuring the stability of training and inference.

[0185] 2. Trade-off configuration of the number of self-attention blocks: It is recommended to set the number of self-attention blocks in series to 3-5 groups, which can be dynamically adjusted according to actual accuracy and speed requirements: Increasing the number of blocks (such as 5 groups) can enhance the global correlation capture capability and improve the robustness of dense / overlapping target detection, but it will increase the amount of computation, resulting in a decrease in inference speed of about 15%-25%; reducing the number of blocks (such as 3 groups) can improve the inference speed, but the ability to model long-distance dependencies is slightly weakened. It is recommended to use a 3-group configuration in real-time priority scenarios (such as industrial production lines).

[0186] 3. Rotation Angle Units and Conversion Standards: The rotation angle parameter (θ) of the network prediction is expressed in radians, with a value range of [-π / 2, π / 2]. In practical applications, if angles are required, they can be converted using the formula θ (angle) = θ (radians) × (180 / π) (1 radian ≈ 57.3°). After conversion, attention should be paid to the periodic consistency of the angle (e.g., -π / 2 radians correspond to -90°, to avoid confusion with 90°).

[0187] 4. Adaptation method for expanding detection categories: This network has good category expansion capability. If a new detection category needs to be added (such as adding "bolt" and "bearing" categories in industrial scenarios), only two adjustments are required: First, modify the output dimension of the final fully connected layer of the multi-scale prediction head category branch (the number of dimensions = the total number of categories after the addition); second, expand the training dataset to ensure that the rotated target samples of the new category cover different scales, angles and scenes. The core modules of the network (C2F feature extraction, global association, feature pyramid) do not need to be modified and can be quickly adapted.

[0188] 5. Scale range adaptation of multi-scale feature pyramid: If there are ultra-small (pixel area < 50) or ultra-large (pixel area > 2000) targets in the actual scene, scale levels can be added to the ViTDet style feature pyramid module (such as P3 layer 1 / 2 scale to adapt to ultra-small targets, P64 layer 1 / 64 scale to adapt to ultra-large targets). At the same time, a dedicated prediction head is configured for the new scale to ensure that targets of different sizes can obtain accurate feature representation.

[0189] 6. Lightweight adaptation recommendations for hardware deployment: When deploying on low-computing-power hardware (such as industrial-grade edge computing devices and embedded GPUs), the network can be further simplified in the following ways: reduce the number of attention heads in the self-attention module from 8 to 4, or reduce the number of cascaded convolutional layers in the C2F module from 3 to 2. After adjustment, the number of parameters can be reduced by about 30%, the inference speed can be improved by more than 20%, and the accuracy loss can be controlled within 2%, which can meet the real-time detection requirements of low-computing-power scenarios.

[0190] Based on the foregoing embodiments, this application provides a target detection device, which includes various units and modules included in each unit. It can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0191] Figure 13 This is a schematic diagram of the composition structure of a target detection device provided in an embodiment of this application, as shown below. Figure 13 As shown, the target detection device 1300 includes: an extraction module 1310, an association module 1320, a processing module 1330, and a detection module 1340, wherein: Extraction module 1310 is used to extract features from the image to be detected based on a multi-scale feature extraction network to obtain multi-scale fused features; the multi-scale fused features carry local features of the target object; The association module 1320 is used to perform global association processing on the multi-scale fusion features based on a self-attention network to obtain a first image feature; the first image feature carries the association relationship between the local features and global features of the target object; Processing module 1330 is used to process the first image features based on the multi-scale feature generation module to obtain second image features at least two scales; The detection module 1340 is used to perform target detection on the second image features corresponding to each scale based on a multi-scale prediction network, obtain and fuse the detection results corresponding to each scale, and obtain the target detection result of the target object.

[0192] In some embodiments, the extraction module is further configured to extract features from the image to be detected by sequentially connecting a downsampling network, an upsampling network, and a feature compression module, wherein the downsampling network includes M cascaded downsampling units, and the upsampling network includes N cascaded upsampling units; the image to be detected is sequentially downsampled by the M downsampling units to obtain target downsampled output features; the target downsampled output features are sequentially upsampled by the N upsampling units to obtain target upsampled output features; and the target upsampled output features are compressed by the feature compression module to obtain the target upsampled output features. Multi-scale fusion features; wherein, the input of the m-th downsampling unit is the output of the (m-1)-th downsampling unit, the input of the 1st downsampling unit is the image to be detected, and the output of the m-th downsampling unit is the target downsampling output feature; the input of the n-th upsampling unit includes the output of the (n-1)-th upsampling unit and the output of the Mn-th downsampling unit, the input of the 1st upsampling unit is the target downsampling output feature and the output of the M-th downsampling unit, and the output of the n-th downsampling unit is the target upsampling output feature; m is an integer less than or equal to M, n is an integer less than or equal to N, both m and n are greater than or equal to 2, and M is greater than or equal to N.

[0193] In some embodiments, the downsampling unit includes a C2f module and a downsampling convolutional layer; the upsampling unit includes a C2f module and an upsampling convolutional layer; wherein, the C2f module's processing of the input first feature includes: adjusting the number of channels of the first feature to the number of output channels through the input convolutional layer; dividing the first feature into a first branch feature and a second branch feature through a block layer; processing the first branch feature sequentially through L stacked neck network layers to obtain a third feature output by each of the neck network layers; connecting the first branch feature, the second branch feature, and the third feature output by each of the neck network layers through a fully connected stacking module to obtain a fourth feature; fusing the fourth feature through a feature fusion module to obtain a fifth feature; the number of channels of the fifth feature is the number of output channels.

[0194] In some embodiments, the association module is further configured to: perform dimensional transformation on the multi-scale fusion feature through a dimensional transformation module to obtain a target format feature; the number of channels of the target format feature is the same as the number of channels of the multi-scale fusion feature; perform self-attention processing on the target format feature through at least two cascaded multi-head self-attention modules to obtain the first image feature; wherein, the multi-head self-attention module includes a multi-head self-attention unit and a feedforward network, and the processing of the input sixth feature by the multi-head self-attention module includes: dividing the sixth feature in the channel dimension to obtain a sub-input feature corresponding to each attention head; and using learnable parameters... The number matrix generates the query vector, key vector, and value vector corresponding to each of the sub-input features. Based on the query vector and key vector corresponding to each sub-input feature, the attention weight corresponding to each sub-input feature is generated. Based on the attention weight and value vector corresponding to each sub-input feature, the sub-output feature corresponding to each sub-input feature is determined, and the sub-output features corresponding to each attention head are concatenated along the channel dimension to obtain the seventh feature. After the seventh feature is subjected to the first linear transformation, activation, and second linear transformation sequentially through the feedforward network, it is fused with the sixth feature to obtain the output eighth feature.

[0195] In some embodiments, the detection module is further configured to: perform target detection on the second image features corresponding to each scale based on the prediction subnetwork corresponding to each scale, and obtain the detection result corresponding to each scale; and summarize the detection results of each scale based on the result fusion module, filter out invalid detection results, and obtain the target detection result of the target object.

[0196] In some embodiments, the prediction subnetwork includes a category prediction subnetwork and a bounding box parameter prediction subnetwork; the detection module is further configured to perform category detection on the second image features corresponding to each scale based on the category prediction subnetwork corresponding to each scale, to obtain a category detection result corresponding to each scale; and perform bounding box detection on the second image features corresponding to each scale based on the bounding box parameter prediction subnetwork corresponding to each scale, to obtain a bounding box parameter detection result corresponding to each scale.

[0197] In some embodiments, the detection module is further configured to: group the rotation box parameter detection results corresponding to each scale according to the category detection results corresponding to each scale; for each category, perform cross-scale non-maximum suppression on all rotation box parameter detection results of the category to filter duplicate rotation boxes and determine the target rotation box parameter detection results of the category; and determine the target detection result based on the target rotation box parameter detection results corresponding to each category.

[0198] Figure 14 This is a schematic diagram of the composition of a training device provided in an embodiment of this application. The training device is used to train an image detection model. The image detection model includes a multi-scale feature extraction network, a self-attention network, a multi-scale feature generation module, and a multi-scale prediction network as described in the above embodiments. Figure 14 As shown, the training device 1400 includes: a training module 1410, wherein: The training module 1410 is used to extract features from sample images based on a pre-trained multi-scale feature extraction network to obtain sample fusion features; to perform global association processing on the sample fusion features based on a self-attention network to obtain first sample image features; to process the first sample image features based on a sample feature generation module to obtain second sample image features at least two scales; to perform target detection on the second image features corresponding to each scale based on a sample prediction network, to obtain and fuse the sample detection results corresponding to each scale to obtain the target sample detection result of the target object; to determine a loss value based on a loss function, the target sample detection result, and the standard result corresponding to the sample image, and to adjust the model parameters of the image detection model based on the loss value.

[0199] In some embodiments, the multi-scale prediction network includes a prediction sub-network corresponding to each scale; the sub-loss value includes a class loss value and a rotated box loss value; the training module is further configured to determine a sub-loss value corresponding to each scale based on the loss function, the target sample detection result corresponding to each scale, and the standard result corresponding to each scale; determine the loss value based on the sub-loss value corresponding to each scale and the loss weight corresponding to each scale; the loss weight decreases as the scale decreases; determine the class loss value corresponding to each scale based on the class loss function, the sample class detection result corresponding to each scale, and the standard class result corresponding to each scale; determine the rotated box loss value corresponding to each scale based on the rotated box loss function, the sample rotated box detection result corresponding to each scale, and the standard rotated box result corresponding to each scale; perform weighted fusion of the class loss value and the rotated box loss value corresponding to each scale to obtain a sub-loss value corresponding to each scale; the weight of the rotated box loss value is greater than the weight of the class loss value.

[0200] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this application can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0201] It should be noted that, in the embodiments of this application, if the above-described target detection method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0202] This application provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0203] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0204] This application provides a computer program including computer-readable code, wherein when the computer-readable code is executed in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0205] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0206] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0207] Figure 15 This application provides a hardware entity diagram of a computer device as an embodiment of the present application, such as... Figure 15 As shown, the hardware entity of the computer device 1500 includes a processor 1501 and a memory 1502, wherein the memory 1502 stores a computer program that can run on the processor 1501, and the processor 1501 executes the program to implement the steps in the method of any of the above embodiments.

[0208] The memory 1502 stores computer programs that can run on the processor. The memory 1502 is configured to store instructions and applications that can be executed by the processor 1501. It can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 1501 and various modules in the computer device 1500. It can be implemented by flash memory or random access memory (RAM).

[0209] When the processor 1501 executes a program, it implements the steps of any of the target detection methods and / or training methods described above. The processor 1501 typically controls the overall operation of the computer device 1500.

[0210] The aforementioned processor can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that other electronic devices can also implement the functions of the aforementioned processor, and this application does not specifically limit the specific implementation.

[0211] The aforementioned computer storage media / memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM), etc.; or it can be various terminals that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0212] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0213] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0214] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0215] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0216] Furthermore, in the various embodiments of this application, all functional units can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units. Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0217] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0218] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A target detection method, characterized in that, The method includes: A multi-scale feature extraction network is used to extract features from the image to be detected, resulting in multi-scale fused features; the multi-scale fused features carry local features of the target object. The multi-scale fusion features are globally correlated using a self-attention network to obtain a first image feature; the first image feature carries the correlation between the local features and global features of the target object. The first image features are processed by the multi-scale feature generation module to obtain second image features at least two scales; Based on a multi-scale prediction network, target detection is performed on the second image features corresponding to each scale, and the detection results corresponding to each scale are obtained and fused to obtain the target detection result of the target object.

2. The method according to claim 1, characterized in that, The multi-scale feature extraction network includes a downsampling network, an upsampling network, and a feature compression module connected in sequence. The downsampling network includes M cascaded downsampling units, and the upsampling network includes N cascaded upsampling units. The feature extraction of the image to be detected based on the multi-scale feature extraction network to obtain multi-scale fused features includes: The image to be detected is downsampled sequentially using the M downsampling units to obtain the target downsampled output features; The target downsampled output features are sequentially upsampled using the N upsampling units to obtain the target upsampled output features. The target upsampled output features are compressed by the feature compression module to obtain the multi-scale fusion features; Wherein, the input of the m-th downsampling unit is the output of the (m-1)-th downsampling unit, the input of the 1st downsampling unit is the image to be detected, and the output of the m-th downsampling unit is the target downsampling output feature; the input of the n-th upsampling unit includes the output of the (n-1)-th upsampling unit and the output of the Mn-th downsampling unit, the input of the 1st upsampling unit is the target downsampling output feature and the output of the M-th downsampling unit, and the output of the n-th downsampling unit is the target upsampling output feature; m is an integer less than or equal to M, n is an integer less than or equal to N, both m and n are greater than or equal to 2, and M is greater than or equal to N.

3. The method according to claim 2, characterized in that, The downsampling unit includes a C2f module and a downsampling convolutional layer; the upsampling unit includes a C2f module and an upsampling convolutional layer; wherein, the processing procedure of the C2f module for the input first feature includes: The number of channels of the first feature is adjusted to the number of output channels by inputting the convolutional layer; The first feature is divided into a first branch feature and a second branch feature by a block layer; The first branch features are processed sequentially by stacking L neck network layers to obtain the third features output by each neck network layer. The first branch feature, the second branch feature, and the third feature output by each of the neck network layers are connected by a fully connected stacking module to obtain the fourth feature; The fourth feature is fused using the feature fusion module to obtain the fifth feature; the number of channels of the fifth feature is the same as the number of output channels.

4. The method according to claim 1, characterized in that, The self-attention network includes a dimension transformation module and at least two cascaded sets of multi-head self-attention modules; The first image feature is obtained by performing global association processing on the multi-scale fused features based on a self-attention network, including: The multi-scale fusion feature is transformed by the dimension transformation module to obtain the target format feature; the number of channels of the target format feature is the same as the number of channels of the multi-scale fusion feature. The first image feature is obtained by performing self-attention processing on the target format features through at least two cascaded multi-head self-attention modules. The multi-head self-attention module includes a multi-head self-attention unit and a feedforward network. The processing procedure of the multi-head self-attention module for the input sixth feature includes: The sixth feature is divided along the channel dimension using a multi-head self-attention unit to obtain sub-input features corresponding to each attention head; query vector, key vector, and value vector are generated for each sub-input feature using a learnable parameter matrix; attention weights are generated for each sub-input feature based on the query vector and key vector; sub-output features are determined for each sub-input feature based on the attention weights and value vectors, and the sub-output features corresponding to each attention head are concatenated along the channel dimension to obtain the seventh feature. After the seventh feature is subjected to a first linear transformation, activation, and second linear transformation sequentially through the feedforward network, it is fused with the sixth feature to obtain the output eighth feature.

5. The method according to claim 1, characterized in that, The multi-scale prediction network includes a result fusion module and a prediction sub-network corresponding to each scale; the target detection is performed on the second image features corresponding to each scale based on the multi-scale prediction network, and the detection results corresponding to each scale are obtained and fused to obtain the target detection result of the target object, including: Based on the prediction sub-network corresponding to each scale, target detection is performed on the second image features corresponding to each scale to obtain the detection result corresponding to each scale. The result fusion module summarizes the detection results at each scale, filters out invalid detection results, and obtains the target detection result of the target object.

6. The method according to claim 5, characterized in that, The prediction subnetwork includes a category prediction subnetwork and a bounding box parameter prediction subnetwork; the target detection based on the second image features corresponding to each scale using the prediction subnetwork corresponding to each scale, to obtain the detection result corresponding to each scale, includes: Based on the category prediction subnetwork corresponding to each scale, category detection is performed on the second image features corresponding to each scale to obtain the category detection result corresponding to each scale. Based on the rotation box parameter prediction subnetwork corresponding to each scale, rotation box detection is performed on the second image features corresponding to each scale to obtain the rotation box parameter detection result corresponding to each scale.

7. The method according to claim 6, characterized in that, The process of summarizing the detection results at each scale based on the result fusion module, filtering out invalid detection results, and obtaining the target detection result of the target object includes: Based on the category detection results corresponding to each scale, the rotation box parameter detection results corresponding to each scale are grouped by category; For each category, cross-scale nonmaximum suppression is performed on all rotation box parameter detection results for that category to filter out duplicate rotation boxes and determine the target rotation box parameter detection results for that category. The target detection result is determined based on the target rotation box parameter detection results corresponding to each category.

8. A training method, characterized in that, The method is used to train an image detection model, the image detection model comprising a multi-scale feature extraction network, a self-attention network, a multi-scale feature generation module, and a multi-scale prediction network as described in any one of claims 1 to 7, and the training method comprising: The sample images are used to extract features based on a pre-trained multi-scale feature extraction network to obtain sample fusion features. The sample fusion features are globally correlated using a self-attention network to obtain the first sample image features. The first sample image features are processed by the sample feature generation module to obtain second sample image features at least two scales. Based on the sample prediction network, target detection is performed on the second image features corresponding to each scale, and the sample detection results corresponding to each scale are obtained and fused to obtain the target sample detection result of the target object. The loss value is determined based on the loss function, the target sample detection result, and the standard result corresponding to the sample image, and the model parameters of the image detection model are adjusted based on the loss value.

9. The training method according to claim 8, characterized in that, The multi-scale prediction network includes a prediction sub-network corresponding to each scale; the step of determining the loss value based on the loss function, the target sample detection result, and the standard result corresponding to the sample image includes: Based on the loss function, the target sample detection result corresponding to each scale and the standard result corresponding to each scale, the sub-loss value corresponding to each scale is determined. The loss value is determined based on the sub-loss value corresponding to each scale and the loss weight corresponding to each scale; the loss weight decreases as the scale decreases. The sub-loss values ​​include class loss values ​​and bounding box loss values; the determination of the sub-loss value corresponding to each scale based on the loss function, the target sample detection result corresponding to each scale, and the standard result corresponding to each scale includes: Based on the category loss function, the category loss value corresponding to each scale is determined by the sample category detection result and the standard category result corresponding to each scale. Based on the rotation box loss function, the sample rotation box detection results and the standard rotation box results corresponding to each scale are used to determine the rotation box loss value for each scale. The category loss value and the bounding box loss value corresponding to each scale are weighted and fused to obtain the sub-loss value corresponding to each scale; the weight of the bounding box loss value is greater than the weight of the category loss value.

10. A target detection device, characterized in that, The target detection device includes: The extraction module is used to extract features from the image to be detected based on a multi-scale feature extraction network to obtain multi-scale fused features; the multi-scale fused features carry local features of the target object; The association module is used to perform global association processing on the multi-scale fused features based on a self-attention network to obtain a first image feature; the first image feature carries the association relationship between the local features and global features of the target object; The processing module is used to process the first image features based on the multi-scale feature generation module to obtain second image features at least two scales; The detection module is used to perform target detection on the second image features corresponding to each scale based on a multi-scale prediction network, obtain and fuse the detection results corresponding to each scale, and obtain the target detection result of the target object.

11. A training device, characterized in that, The training device is used to train an image detection model, the image detection model comprising, as described in any one of claims 1 to 7, a multi-scale feature extraction network, a self-attention network, a multi-scale feature generation module, and a multi-scale prediction network, and the training device comprising: The training module is used to extract features from sample images based on a pre-trained multi-scale feature extraction network to obtain sample fusion features; to perform global correlation processing on the sample fusion features based on a self-attention network to obtain first sample image features; to process the first sample image features based on the sample feature generation module to obtain second sample image features at least two scales; to perform target detection on the second image features corresponding to each scale based on a sample prediction network, to obtain and fuse the sample detection results corresponding to each scale to obtain the target sample detection result of the target object; to determine a loss value based on a loss function, the target sample detection result, and the standard result corresponding to the sample image, and to adjust the model parameters of the image detection model based on the loss value.

12. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 7, or when it executes the program, it implements the steps of the method according to claim 8 or 9.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7, or when the program is executed, it implements the steps of the method according to claim 8 or 9.

14. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7, or when the program is executed, they implement the steps of the method according to claim 8 or 9.

Images