Target detection method and system based on self-attention and multi-scale adaptive fusion

Abstract

The invention discloses a target detection method and system based on self-attention and multi-scale adaptive fusion, and relates to the field of multi-scale target detection, and the method comprises the steps: splitting a to-be-recognized image through a multi-scale feature extraction module, and sequentially executing n single-scale feature extraction stages, obtaining (n-1) local enhanced feature maps of different scales of the to-be-recognized image; utilizing a multi-scale feature fusion module to construct a feature pyramid comprising n-1 local enhanced feature maps with different scales, and performing scale and space perception attention calculation on the local enhanced feature map on each feature layer of the feature pyramid to obtain n-1 layers of fusion feature maps with different scales; and respectively calculating classification loss and regression loss of the anchor-frame-free branches on each layer of fusion feature map by using a scale adaptive module, and carrying out target detection on the to-be-identified image by using the corresponding feature layer when the sum of the classification loss and the regression loss is minimum. The precision of target detection in the to-be-recognized image is improved.

Description

Technical Field

[0001] The present application relates to the technical field of multi-scale target detection, and in particular to a target detection method and system based on self-attention and multi-scale adaptive fusion. Background Art

[0002] In recent years, although the speed and accuracy of target detection have been greatly improved, the multi-scale problem has always been one of the core challenges affecting the accuracy of marine target detection tasks in complex scenes. Multi-scale target detection aims to accurately classify and locate targets of different scales in images. In the vast and complex marine environment, related group collaborative perception tasks such as maritime search, escort, and roundup require simultaneous detection of large warships and small unmanned vessels. However, the scale and shape of the detected targets are often not fixed, and the scale change problem has always been one of the most challenging problems in target detection.

[0003] Among the existing algorithms proposed for the multi-scale problem of targets, there are three main effective ones, namely image pyramid, SSD feature pyramid, and FPN feature pyramid. Their common idea is to use multi-scale features to detect objects of different sizes. Although the above algorithms use multi-scale information to extract target features and achieve good results, these multi-scale target detection algorithms only distinguish the size of the target and do not pay attention to other aspects, and ultimately the accuracy of target detection is also low. Summary of the invention

[0004] The purpose of this application is to provide a target detection method and system based on self-attention and multi-scale adaptive fusion, which improves the accuracy of target detection in the image to be identified.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a target detection method based on self-attention and multi-scale adaptive fusion, and the target detection method based on self-attention and multi-scale adaptive fusion includes:

[0007] Inputting the image to be identified into a pre-trained target detection model; the target detection model includes: a multi-scale feature extraction module, a multi-scale feature fusion module and a scale adaptation module;

[0008] The multi-scale feature extraction module is used to split the image to be identified, and n single-scale feature extraction stages are performed in sequence to obtain n-1 local enhanced feature maps of different scales of the image to be identified; wherein n>1, and each single-scale feature extraction stage is embedded with a plurality of sliding window visual converter blocks with improved self-attention; the improved self-attention is that a position offset is added in the process of calculating the self-attention matrix of the sliding window visual converter block;

[0009] A feature pyramid including n-1 local enhanced feature maps of different scales is constructed by using a multi-scale feature fusion module, and scale and space-aware attention calculations are performed on the local enhanced feature maps on each feature layer of the feature pyramid to obtain n-1 layers of fused feature maps of different scales; each feature layer of the feature pyramid corresponds to a local enhanced feature map and a fused feature map of one scale;

[0010] The scale adaptive module is used to calculate the classification loss and regression loss of the anchor-free branch on each layer of the fusion feature map, and the feature layer corresponding to the minimum sum of the classification loss and the regression loss is determined as the optimal scale feature layer; the optimal scale feature layer is used for target detection of the image to be recognized.

[0011] Optionally, the calculation formula of the self-attention matrix is:

[0012]

[0013] Attention(·,·,·) is the self-attention matrix, Q is the query vector, K is the key vector, V is the value vector, softmax(·) is the activation function, and d k is the dimension of the key vector, and DWConv(·) is the depthwise separable convolution.

[0014] Optionally, the multi-scale feature extraction module comprises at least: a block partitioning layer, a linear embedding layer and a plurality of the sliding window visual converter blocks; splitting the image to be identified by using the multi-scale feature extraction module, and sequentially performing n single-scale feature extraction stages to obtain n-1 local enhanced feature maps of different scales of the image to be identified, specifically comprising:

[0015] Using the block partitioning layer to split the image to be identified into a plurality of non-overlapping image blocks of equal size;

[0016] Projecting all image blocks to any dimension using the linear embedding layer to obtain feature representations of each image block;

[0017] Based on the feature representation of all image blocks, n single-scale feature extraction stages are executed in sequence. Except for the first single-scale feature extraction stage, the sliding window visual transformer blocks in the remaining n-1 single-scale feature extraction stages will extract n-1 local enhanced feature maps of different scales; among them, in the remaining n-1 single-scale feature extraction stages, one single-scale feature extraction stage corresponds to a local enhanced feature map of one scale.

[0018] Optionally, except for the first single-scale feature extraction stage, each of the remaining n-1 single-scale feature extraction stages is also embedded with an image block fusion layer; the image block fusion layer is used to fuse feature representations of different image blocks.

[0019] Optionally, the multi-scale feature fusion module includes: a pyramid module, a scale-aware attention module and a space-aware attention module; the multi-scale feature fusion module is used to construct a feature pyramid containing n-1 local enhanced feature maps of different scales, and the scale and space-aware attention calculations are performed on the local enhanced feature maps on each feature layer of the feature pyramid to obtain n-1 layers of fused feature maps of different scales, specifically including:

[0020] Using the pyramid module to construct a feature pyramid containing n-1 local enhanced feature maps of different scales;

[0021] The scale-aware attention module and the space-aware attention module are used to perform weighted attention fusion on the local enhanced feature maps on each feature layer of the feature pyramid to obtain n-1 layers of fused feature maps of different scales.

[0022] Optionally, each output layer of the feature pyramid introduces two additional convolutional layers; wherein one convolutional layer is used for classification prediction of the anchor-free branch, and the other convolutional layer is used for regression prediction of the anchor-free branch.

[0023] Optionally, the calculation formula of the classification loss is:

[0024]

[0025] In the formula, is the classification loss on the l-th layer fusion feature map, is the valid box in the l-th layer fusion feature map , i, j are the positions of pixels or feature points, and FL(·,·,·) is the FL loss function.

[0026] Optionally, the calculation formula of the regression loss is:

[0027]

[0028] In the formula, is the regression loss on the l-th layer fusion feature map, is the valid box in the l-th layer fusion feature map is the number of pixels or feature points in , i, j is the position of the pixel or feature point, and IOU(·,·,·) is the IOU loss function.

[0029] Optionally, the calculation formula of the optimal scale feature layer is:

[0030]

[0031] In the formula, l * is the optimal scale feature layer, is the classification loss on the l-th layer fusion feature map, is the regression loss on the l-th layer fusion feature map.

[0032] In a second aspect, the present application also provides a computer system, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the target detection method based on self-attention and multi-scale adaptive fusion.

[0033] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0034] In dealing with the problem of multi-scale target detection, this application first introduces a sliding window visual converter block with improved self-attention through a multi-scale feature extraction module, that is, by splitting the image to be identified and calculating self-attention, it continuously focuses on the extraction of local feature information of the target and strengthens the connection between features, further improving the ability to capture features of different scales. Then, the scale and space-aware attention mechanism is introduced in the multi-scale feature fusion module, which effectively integrates features from different levels of the feature pyramid and reduces the semantic differences between features. Finally, in the scale adaptation module, the classification loss and regression loss of each layer of the feature pyramid are calculated in an anchor-free manner, and the most appropriate feature layer is automatically selected according to the minimum classification and regression losses to perform high-precision detection on the target in the image to be identified. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0036] Figure 1 A flowchart of a target detection method based on self-attention and multi-scale adaptive fusion provided in an embodiment of the present application;

[0037] Figure 2 A schematic diagram of the structure of a scale-aware attention module provided in an embodiment of the present application;

[0038] Figure 3A schematic diagram of the structure of a spatial perception attention module provided in an embodiment of the present application;

[0039] Figure 4 An exemplary structural diagram of a multi-scale feature extraction module provided in an embodiment of the present application;

[0040] Figure 5 An exemplary structural diagram of a target detection model provided in an embodiment of the present application;

[0041] Figure 6 A visual detection result diagram obtained by the target detection model provided in the embodiment of the present application;

[0042] Figure 7 This is a diagram of the internal structure of a computer system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0044] The purpose of this application is to provide a target detection method and system based on self-attention and multi-scale adaptive fusion, which improves the accuracy of target detection in the image to be identified.

[0045] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0046] Example 1

[0047] This embodiment provides a target detection method based on self-attention and multi-scale adaptive fusion, such as Figure 1 As shown in FIG. 1 , the target detection method based on self-attention and multi-scale adaptive fusion specifically includes:

[0048] Step S1: inputting the image to be identified into a pre-trained target detection model; the target detection model includes: a multi-scale feature extraction module, a multi-scale feature fusion module and a scale adaptation module.

[0049] In this embodiment, the multi-scale feature extraction module includes: a block partitioning layer, a linear embedding layer, an image block fusion layer and a plurality of sliding window visual transformer blocks (Swin Transformer blocks).

[0050] The multi-scale feature fusion module includes: pyramid module, scale-aware attention module and space-aware attention module. Figure 2 As shown in Figure 1, the scale-aware attention module consists of an average pooling layer, a 1×1 convolutional layer, a Relu activation function, and a Hardsigmoid activation function. This module dynamically fuses features based on the semantic importance of different scales. Figure 3 As shown in Figure 1, the spatial-aware attention module consists of a deformable convolution index, a 3×3 convolution layer, a Sigmoid activation function, and an offset. This module mainly focuses on foreground objects at different spatial positions.

[0051] In order to realize the classification and regression prediction of multi-scale features, an anchor-free mechanism is introduced in the scale adaptation module, and two additional convolutional layers are introduced in the output layer of each feature layer of the feature pyramid, one convolutional layer is used for classification prediction of the anchor-free branch, and the other convolutional layer is used for regression prediction of the anchor-free branch.

[0052] Step S2: Split the image to be identified by using a multi-scale feature extraction module, and execute n single-scale feature extraction stages in sequence to obtain n-1 local enhanced feature maps of different scales of the image to be identified; wherein n>1, and each single-scale feature extraction stage is embedded with multiple sliding window visual converter blocks with improved self-attention; the improved self-attention is to add a position offset in the process of calculating the self-attention matrix of the sliding window visual converter block.

[0053] In this embodiment, step S2 specifically includes:

[0054] Step S21: using a block partitioning layer to split the image to be identified into a plurality of non-overlapping image blocks of equal size.

[0055] Step S22: Project all image blocks to any dimension using a linear embedding layer to obtain feature representations of each image block.

[0056] Step S23: Based on the feature representation of all image blocks, n single-scale feature extraction stages are executed in sequence. Except for the first single-scale feature extraction stage, the sliding window visual converter blocks in the remaining n-1 single-scale feature extraction stages will extract n-1 local enhanced feature maps of different scales; wherein, in the remaining n-1 single-scale feature extraction stages, one single-scale feature extraction stage corresponds to a local enhanced feature map of one scale.

[0057] As a preferred implementation mode, Figure 4As shown in the figure, the multi-scale feature extraction module needs to execute four single-scale feature extraction stages in sequence, namely Stage1, Stage2, Stage3 and Stage4. Among them, Stage1, Stage2 and Stage4 each contain 2 SwinTransformerblocks, and Stage3 contains 6 Swin Transformerblocks. Since Stage1 only obtains the initial features of the image through linear embedding and does not contain deep semantic information, it can be used as a feature layer. Therefore, the linear embedding layer is included in Stage1. Stage2, Stage3 and Stage4 also contain image block fusion layers.

[0058] Specifically, the H×W×3 (H and W are length and width) RGB image input block partition layer is split into non-overlapping N×(P 2 ×3) split images, which facilitates the subsequent Swin Transformer blocks to calculate self-attention in a small window, reducing the sequence length and computational complexity. 2 ×3 split images (P is the length and width of the split image) are regarded as a split token, and the RGB image is split into N two-dimensional split image sequences, where N is the effective input sequence length of SwinTransformerblocks.

[0059] Furthermore, the split image is fed as a whole into the linear embedding layer, and each split image is projected to an arbitrary dimension C to obtain feature representations of different split images (split tokens). Subsequently, the feature representations of these split tokens are fed into several Swin Transformerblocks with improved self-attention. Among them, the first Swin Transformerblocks keeps the number of split tokens of the input and output unchanged, and is designated as Stage 1 together with the linear embedding layer. In order to produce a hierarchical representation, as the Stage deepens, the number of split tokens is gradually reduced by the image block fusion layer, which is specifically manifested in that the size of the feature map is reduced by 1 / 2 of the original size and the number of channels is increased by 2 times the original size, thereby obtaining multi-size features (i.e., obtaining multiple local enhanced feature maps).

[0060] As another preferred implementation, Swin Transformer blocks with improved self-attention solves the problem that traditional transformer models in visual image tasks cannot handle multi-scale, retains their global information perception and sequence modeling capabilities, and combines the self-attention mechanism with the multi-stage hierarchical architecture design principle of the convolutional network. Swin Transformer blocks are used for feature extraction and downsampling operations to improve the multi-scale representation capability of the target detection model, so as to obtain local and global information and extract multi-scale features.

[0061] Although SwinTransformerblocks can provide more compact feature representation and richer semantic information, and its hierarchical structure can effectively extract multi-scale features, helping the subsequent network to better locate multi-scale targets. However, since the self-attention mechanism has permutation invariance (the output results of different permutations are the same), it is impossible to capture key position information in two-dimensional images. In order to make up for this defect, it is necessary to add position offset in the process of calculating the self-attention matrix of each SwinTransformerblocks, that is, this embodiment improves the window multi-head attention and moving window multi-head self-attention in the original structure of SwinTransformerblocks, introduces the method of relative position offset for position encoding when calculating attention, and the relative position offset is added to the self-add attention matrix to improve the overall accuracy.

[0062] Among them, the local enhanced relative offset method is used to add the position information to each Swin Transformerblocks, and directly operate on the attention result instead of the attention calculation. Specifically, the local enhanced relative offset method directly adds the position information to the V vector, and then adds it to the result of the self-attention calculation. Let the variable is the value element v i and v j The edge between them, E represents the position offset weight of the V vector, then the calculation of the self-attention matrix can be defined as follows:

[0063]

[0064] Where Attention(·,·,·) is the self-attention matrix, Q, K, and V are the core components for calculating self-attention, representing the three feature vectors of the image, namely the query vector, the key vector, and the value vector, respectively. Softmax(·) is the activation function, and d k is the dimension of the key vector.

[0065] However, if all connections in E are considered in the process of calculating the self-attention matrix, it will require huge computational costs. Therefore, in order to better process local position information and reduce the amount of calculation, the position information of V is learned by depthwise separable convolution (DWConv) instead of E, and then this depthwise separable convolution is added to V in a residual way, that is, the calculation formula of the final self-attention matrix is:

[0066]

[0067] By locally enhancing the relative offset method, relative position information is added and a local assumption bias is introduced, thereby improving the target detection model's ability to represent position information, ultimately obtaining local and global information and extracting multi-scale features.

[0068] Step S3: Use the multi-scale feature fusion module to construct a feature pyramid containing n-1 local enhanced feature maps of different scales, and perform scale and space-aware attention calculations on the local enhanced feature maps on each feature layer of the feature pyramid to obtain n-1 layers of fused feature maps of different scales; each feature layer of the feature pyramid corresponds to a local enhanced feature map and a fused feature map of one scale.

[0069] In this embodiment, step S3 is specifically as follows:

[0070] Step S31: Use the pyramid module to construct a feature pyramid containing n-1 local enhanced feature maps of different scales.

[0071] Step S32: Use the scale-aware attention module and the space-aware attention module to perform weighted attention fusion on the local enhanced feature maps on each feature layer of the feature pyramid to obtain n-1 layers of fused feature maps of different scales.

[0072] As a preferred implementation, among the multi-scale features at different levels, shallow features have higher resolution and can carry rich geometric detail information, but have a small receptive field and lack semantic information; deep features have a larger receptive field and rich semantic information, but have low resolution and are difficult to carry geometric detail information. These multi-scale features from different levels are extracted from different depths of the network, which creates an obvious semantic gap, and further fusion of multi-scale features is required.

[0073] The feature pyramid combined with a top-down structure with lateral connections can fuse feature information of different scales in adjacent layers of the feature pyramid, thereby reducing the semantic gap between feature layers of different scales. It is one of the effective methods for multi-scale target detection. However, this general method makes it difficult to highlight the key and important information in the feature map and suppress non-key redundant information. Therefore, in the process of feature fusion of the multi-scale feature pyramid, this embodiment introduces scale-aware attention and space-aware attention to features of different scales, respectively, and finally forms a multi-scale fusion feature map with weighted attention to obtain identifiable key features that need to be focused on, and ignore irrelevant redundant features, thereby optimizing the multi-scale feature fusion process and providing more accurate and effective feature representation for subsequent tasks.

[0074] Step S4: Use the scale adaptive module to calculate the classification loss and regression loss of the anchor-free branch on each layer of the fusion feature map, and determine the feature layer corresponding to the minimum sum of the classification loss and the regression loss as the optimal scale feature layer; the optimal scale feature layer is used for target detection of the image to be recognized.

[0075] Most prediction head networks of current target detection methods use different layers to predict target instances of different scales, using the anchor box mechanism to discretize all targets into candidate boxes of different sizes and aspect ratios, mapping small candidate boxes to shallow features and large candidate boxes to deep features, and then matching the real boxes to different prediction boxes based on the intersection-over-union ratio. However, this artificially designed anchor box assignment rule may not be optimal, resulting in inappropriate anchor box design that limits the selection of target multi-scale feature layers.

[0076] As a preferred implementation mode, this embodiment improves the prediction head network according to the FSAF model, learns the classification and regression parameters in an anchor-free manner, and finds the most suitable feature scale for targets of different scales.

[0077] Based on the RetinaNet prediction head network, this embodiment introduces two additional convolutional layers for each layer of the feature pyramid output layer, which are used for classification and regression prediction of the anchor-free frame branch respectively. Use the effective positive sample area to regress the true frame, and select the most appropriate feature layer for prediction based on the minimum loss function. Among them, the category of the true frame is defined as k, and the position prediction coordinates of the true frame are b = [x, y, w, h], where x, y, w and h represent the x-axis coordinate, y-axis coordinate, width and height of the center point of the frame respectively; the projection of the prediction frame on the lth feature layer can be defined as The valid frame can be defined as yes The scaling factor is 0.2; the ignore box can be defined as Too The scaling factor is 0.5.

[0078] For any image, the classification loss of the anchor-free branch is the sum of the focal losses of all non-ignored frame regions (normalized by the total number of features in all valid frame parts), and its regression loss is the average of the intersection-over-union losses of all valid frame regions (positive samples). Given an object I, the classification loss on its l-th layer fusion feature map is:

[0079]

[0080] In the formula, is the classification loss on the l-th layer fusion feature map, is the valid box in the l-th layer fusion feature map where i, j is the number of pixels or feature points in , i and j are the positions of pixels or feature points, and FL(·,·,·) is the FL (Focal Loss) loss function.

[0081] The regression loss on its l-th layer fusion feature map is:

[0082]

[0083] In the formula, is the regression loss on the l-th layer fusion feature map, and IOU(·,·,·) is the IOU (Intersection over Union Loss) loss function.

[0084] In the process of automatically assigning objects to feature layers of appropriate scales, the scale adaptation module allows learning using fused feature maps of all scale levels to select the most appropriate feature layer based on the semantics of the object. An object will pass through the scale adaptation module to calculate the classification loss on each layer of the feature pyramid. and regression loss The target detection model selects the feature layer l that minimizes the sum of the two losses. * As the optimal scale feature layer for target detection, the calculation formula of the optimal scale feature layer is:

[0085]

[0086] This embodiment learns the classification and regression parameters in an anchor-free manner. During the prediction process, the most suitable feature layer is determined by calculating and comparing the classification loss and regression loss on each feature layer for the final prediction of the target. This process ensures that the category and contour of each target are accurately identified and predicted, and the prediction result is optimized by selecting the feature layer with the smallest total loss, which contains the precise location, size, and category information of the target.

[0087] In addition, in order to verify the effectiveness of the above-mentioned target detection method based on self-attention and multi-scale adaptive fusion, this embodiment takes the multi-scale target detection tasks of single-class and six-class ships at sea as an example. Figure 5 The target detection model shown is used for validation analysis:

[0088] The data set of this embodiment comes from two types, one is the sea surface ship image data collected on the spot, and the other is the sea surface ship image data obtained by web crawling and other methods. For the quantitative distribution of targets of different scales, the six types of real ship data sets at sea divide the target size into three scales: large, medium, and small, to measure the performance of the target detection model in detecting targets of different scales. Specifically, large targets account for about 41% of the data set, medium targets account for about 36%, and small targets account for about 23%. The scale of ships actually photographed at sea is mainly small and medium, which makes detection difficult. The network-collected ship data contains a total of 8,301 ship images, and the distribution of ship scales is relatively uniform.

[0089] Figure 6 This is a visualization detection result obtained by using the target detection model in the target detection method based on self-attention and multi-scale adaptive fusion. From the visualization detection result, it can be seen that the target detection model in this embodiment can effectively detect marine ship targets of various scales and has high detection accuracy. Therefore, the target detection method based on self-attention and multi-scale adaptive fusion proposed in this embodiment can effectively improve the detection capability of multi-scale targets of unmanned ships in marine scenes and show better detection results.

[0090] As shown in Tables 1 and 2, the performance of the target detection model in this embodiment is much higher than other two-stage target detection models, such as the classic model FasterR-CNN, the model Sparse R-CNN based on sparse candidate targets, etc. At the same time, the performance of the target detection model in this embodiment is also higher than other single-stage target detection models, such as the anchor-free classic target detection method FCOS, the dense target detection model DDOD, etc. In addition, the performance of the target detection model in this embodiment is also higher than the latest target detection models that use the hierarchical visual transformer with a moving window as a high-precision backbone network, such as PVTv2, ConvNeXt, etc. On the dataset of six types of ships at sea, the target detection model in this embodiment can achieve 89.0% AP 50 , which is 1.9% higher than the benchmark model RetinaNet, and higher than other models such as FCOS, PVTv2, and ConvNeXt. The experimental data in the table show that this embodiment can effectively improve the multi-scale representation capability of the target detection model from the aspects of multi-scale feature extraction, multi-scale feature fusion, and scale adaptation, and further improve the multi-scale target detection accuracy of unmanned ships in complex sea environments.

[0091] Table 1 Comparative experimental table of single-class ship dataset at sea

[0092]

[0093] In Table 1 above, MAP is the mean average precision, which means the average precision of the categories. The average precision is first calculated for the detection results of each category, and then the average of these average precisions is taken.

[0094] AP 50 It represents the average precision when the IoU threshold is 0.50, which means that when the ratio of the intersection area of ​​the predicted box and the true box to the union area is greater than 50%, the prediction result is considered correct.

[0095] AP 75 It represents the average precision when the IoU threshold is 0.75, which means that when the ratio of the intersection area of ​​the predicted box and the true box to the union area is greater than 75%, the prediction result is considered correct.

[0096] AP S Represents the average accuracy for small-scale targets.

[0097] AP M Represents the average accuracy for mesoscale targets.

[0098] AP L Represents the average accuracy for large-scale objects.

[0099] Table 2 Comparative experimental table of six types of ship data sets at sea

[0100]

[0101] Example 2

[0102] This embodiment provides a computer system, which may be a server or a terminal, and its internal structure diagram may be as follows: Figure 7 As shown. The computer system includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer system is used to provide computing and control capabilities. The memory of the computer system includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer system is used to exchange information between the processor and an external device. The communication interface of the computer system is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a target detection method based on self-attention and multi-scale adaptive fusion is implemented.

[0103] Those skilled in the art will understand that Figure 7The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer system to which the scheme of the present application is applied. The specific computer system may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0104] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0105] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0106] In summary, this application effectively improves the performance of the target detection model in terms of multi-scale representation capabilities by introducing the self-attention mechanism and scale adaptation technology, so that the target detection model can better adapt to the detection needs of targets of different scales, and further improve the accuracy of multi-scale target detection of unmanned ships in complex sea environments. Through the introduction of the attention mechanism, the target detection model can focus more on important feature information, thereby improving efficiency while maintaining high accuracy. In addition, the application of scale adaptation technology further enhances the robustness of the target detection model in complex marine environments, enabling it to robustly cope with targets of different scales and shapes.

[0107] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0108] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A target detection method based on self-attention and multi-scale adaptive fusion, characterized in that: The target detection method based on self-attention and multi-scale adaptive fusion includes: Inputting the image to be identified into a pre-trained target detection model; the target detection model includes: a multi-scale feature extraction module, a multi-scale feature fusion module and a scale adaptation module; The multi-scale feature extraction module is used to split the image to be identified, and n single-scale feature extraction stages are performed in sequence to obtain n-1 local enhanced feature maps of different scales of the image to be identified; wherein n>1, and each single-scale feature extraction stage is embedded with a plurality of sliding window visual converter blocks with improved self-attention; the improved self-attention is that a position offset is added in the process of calculating the self-attention matrix of the sliding window visual converter block; A feature pyramid including n-1 local enhanced feature maps of different scales is constructed by using a multi-scale feature fusion module, and scale and space-aware attention calculations are performed on the local enhanced feature maps on each feature layer of the feature pyramid to obtain n-1 layers of fused feature maps of different scales; each feature layer of the feature pyramid corresponds to a local enhanced feature map and a fused feature map of one scale; The scale adaptive module is used to calculate the classification loss and regression loss of the anchor-free branch on each layer of the fusion feature map, and the feature layer corresponding to the minimum sum of the classification loss and the regression loss is determined as the optimal scale feature layer; the optimal scale feature layer is used for target detection of the image to be recognized.

2. The target detection method based on self-attention and multi-scale adaptive fusion according to claim 1, characterized in that: The calculation formula of the self-attention matrix is: Attention(·,·,·) is the self-attention matrix, Q is the query vector, K is the key vector, V is the value vector, softmax(·) is the activation function, and d k is the dimension of the key vector, and DWConv(·) is the depthwise separable convolution.

3. The target detection method based on self-attention and multi-scale adaptive fusion according to claim 1, characterized in that: The multi-scale feature extraction module at least includes: a block partitioning layer, a linear embedding layer and a plurality of sliding window visual converter blocks; the multi-scale feature extraction module is used to split the image to be identified, and n single-scale feature extraction stages are sequentially performed to obtain n-1 local enhanced feature maps of different scales of the image to be identified, specifically including: Using the block partitioning layer to split the image to be identified into a plurality of non-overlapping image blocks of equal size; Projecting all image blocks to any dimension using the linear embedding layer to obtain feature representations of each image block; Based on the feature representation of all image blocks, n single-scale feature extraction stages are executed in sequence. Except for the first single-scale feature extraction stage, the sliding window visual transformer blocks in the remaining n-1 single-scale feature extraction stages will extract n-1 local enhanced feature maps of different scales; among them, in the remaining n-1 single-scale feature extraction stages, one single-scale feature extraction stage corresponds to a local enhanced feature map of one scale.

4. The target detection method based on self-attention and multi-scale adaptive fusion according to claim 3 is characterized in that: Except for the first single-scale feature extraction stage, each of the remaining n-1 single-scale feature extraction stages is also embedded with an image block fusion layer; the image block fusion layer is used to fuse feature representations of different image blocks.

5. The target detection method based on self-attention and multi-scale adaptive fusion according to claim 1, characterized in that: The multi-scale feature fusion module includes: a pyramid module, a scale-aware attention module and a space-aware attention module; the multi-scale feature fusion module is used to construct a feature pyramid containing n-1 local enhanced feature maps of different scales, and the scale and space-aware attention calculations are performed on the local enhanced feature maps on each feature layer of the feature pyramid to obtain n-1 layers of fused feature maps of different scales, specifically including: Using the pyramid module to construct a feature pyramid containing n-1 local enhanced feature maps of different scales; The scale-aware attention module and the space-aware attention module are used to perform weighted attention fusion on the local enhanced feature maps on each feature layer of the feature pyramid to obtain n-1 layers of fused feature maps of different scales.

6. The target detection method based on self-attention and multi-scale adaptive fusion according to claim 5, characterized in that: Each output layer of the feature pyramid introduces two additional convolutional layers, wherein one convolutional layer is used for classification prediction of the anchor-free branch, and the other convolutional layer is used for regression prediction of the anchor-free branch.

7. The object detection method based on self-attention and multi-scale adaptive fusion according to claim 1, characterized in that: The calculation formula of the classification loss is: In the formula, is the classification loss on the l-th layer fusion feature map, is the valid box in the l-th layer fusion feature map , i, j are the positions of pixels or feature points, and FL(·,·,·) is the FL loss function.

8. The object detection method based on self-attention and multi-scale adaptive fusion according to claim 1, characterized in that: The calculation formula of the regression loss is: In the formula, is the regression loss on the l-th layer fusion feature map, is the valid box in the l-th layer fusion feature map is the number of pixels or feature points in , i, j is the position of the pixel or feature point, and IOU(·,·,·) is the IOU loss function.

9. The target detection method based on self-attention and multi-scale adaptive fusion according to claim 1, characterized in that: The calculation formula of the optimal scale feature layer is: In the formula, l * is the optimal scale feature layer, is the classification loss on the l-th layer fusion feature map, is the regression loss on the l-th layer fusion feature map.

10. A computer system comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the target detection method based on self-attention and multi-scale adaptive fusion as described in any one of claims 1 to 9.

Images