Method for determining the position of an occluded eye based on associative memory

Abstract

The application discloses a method for occluded object standard positioning based on associative memory, and belongs to the technical field of computer vision. The method comprises the following steps: inputting a to-be-detected image into an interference segmentation and target detection integrated network to obtain an initial detection frame of a candidate target, a target category, an initial detection confidence, a target region feature and interference occlusion information; determining an occluded candidate target according to the interference occlusion information; performing memory searching and comparison based on a pre-constructed visual memory library to obtain an associative memory feature corresponding to the occluded candidate target; and performing detection result correction based on the target region feature and the associative memory feature of the occluded candidate target to obtain a correct positioning result of the target in the to-be-detected image. The application can compensate for feature loss caused by occlusion by using complete memory of unoccluded targets, and improve target positioning accuracy and confidence reliability under complex occlusion conditions.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a method for accurate localization of occluded targets based on associative memory. Background Technology

[0002] Infrared target detection is widely used in scenarios such as missile-borne guidance, maritime surveillance, perimeter alerting, and unmanned platform perception. Compared with visible light imaging, infrared imaging can reflect the difference in thermal radiation between the target and the background, and has better target perception capabilities in nighttime, low-light, and some complex environmental conditions. However, in practical applications, targets are easily obscured or occluded by other objects in the scene, as well as factors such as smoke, clouds, and firelight, making it difficult to fully present their outline and structural information in infrared images. For example, in missile-borne guidance scenarios, when detecting and locating ship targets on the sea surface, the ship target may be obscured by clouds, smoke, or localized firelight, making only a portion of the target area clearly identifiable in the infrared image.

[0003] Because infrared images typically lack texture information and have blurred target boundaries, when a target is occluded, the effective visual information used to characterize the complete spatial range is further reduced, thereby increasing the difficulty of determining the target boundary and position, and leading to a decrease in positioning accuracy.

[0004] The impact of occlusion on target detection is not only manifested in the reduction of target feature information and the decrease in detection confidence, but more importantly, it directly destroys the complete target structure and boundary information required for bounding box localization. For a fully visible target, the detection network can determine the overall position of the target based on its contour, spatial structure, and internal features. However, when the target is occluded, the structural and boundary information within the occluded area cannot be directly obtained from the current image, and the detection network can only infer the complete position and scale of the target based on the remaining visible area. Therefore, even if the detection network can still determine the presence of the target based on local visible features, it may still be difficult to accurately estimate its complete spatial range, leading to phenomena such as positional shifts in the predicted bounding box, errors in width and height estimation, or shrinkage towards the visible area, ultimately reducing the degree of overlap between the predicted bounding box and the true complete boundary of the target. For applications such as missile-borne guidance, which have high requirements for the accuracy of target spatial position and boundary, the above-mentioned localization errors may further affect the accuracy of subsequent target tracking, guidance decisions, and state estimation.

[0005] To address the problem of target detection under occlusion conditions, existing methods primarily improve the detection capabilities of models when target information is incomplete by increasing occlusion training samples, constructing artificial occlusion data, enhancing local target features, or modeling the visible region, local components, and occlusion state of the target. Some methods also predict the visible region of the target, the local structure of the target, or use contextual information to help infer the complete target range. However, these methods generally still rely mainly on the target information observable in the current image and learn the mapping relationship from locally visible information to the complete target location through training data. When the boundaries of key parts of the target or a large area of ​​the target structure are occluded, the information used to determine the complete spatial range of the target in the image is already missing. Relying solely on limited visible features is still insufficient to reliably recover the complete structure and accurate boundaries of the target in an unoccluded state.

[0006] In human visual cognition, the identification and localization of occluded targets do not rely solely on directly observed visual information. Humans can combine visual memories formed from previous observations of similar targets, associating the target's possible complete structure based on currently visible local features, and supplementing the missing visual information with existing memories. For example, when part of a ship is obscured by clouds or smoke, a person can still associate the possible location and extent of the obscured part based on the currently visible local structure of the ship, combined with previously formed memories of the ship's overall shape, thus forming a judgment of the ship's complete spatial location. This process embodies an associative memory mechanism, where current local visual information triggers a complete memory representation related to the target, and the retrieved memory information supplements the target features missing in the current observation.

[0007] While some existing memory mechanisms in object detection can preserve and utilize historical features, sample representations, or prior information, these technologies are primarily used for tasks such as temporal information aggregation, context enhancement, feature matching, or semantic representation enhancement. They are not adequately designed to address the problem of missing complete target structure and boundary information caused by target occlusion in a single infrared image. For accurate localization of occluded targets, a key challenge remains: how to utilize currently visible target features to retrieve and associate memory information matching the current target from the complete target memory built during training, and how to use the retrieved memory representations to compensate for the loss of structural information caused by occlusion, thereby restoring the target's complete spatial extent.

[0008] Therefore, it is necessary to study a memory-enhanced localization method for occluded targets. This method would enable the detection system to move beyond relying solely on the limited visible target information in the current image. Instead, it would leverage the associative memory mechanism of human vision, retrieve representations matching the current target from the complete target memory formed during training based on the local observation features of the occluded target, and use related memories to compensate for the loss of structural and spatial information caused by occlusion. This would restore the complete spatial range of the target and achieve accurate localization of the detection box and reliable estimation of confidence. Consequently, it would improve the target localization accuracy under complex occlusion conditions such as clouds, fog, and fire, meeting the application requirements of scenarios such as missile-borne guidance and maritime surveillance. Summary of the Invention

[0009] This invention aims to address the problem of inaccurate detection bounding box localization and confidence scores due to incomplete target region features when a target is occluded. It provides an accurate target localization method based on associative memory. This method constructs a visual memory bank using target region features from unoccluded target samples. Based on the recognition results of occluded candidate targets, relevant memory features are retrieved. Furthermore, by incorporating interference occlusion information and feature differences, the associative memory features are continuously adjusted. This process supplements the target's prior information while preserving the current target region features, thereby correcting the detection bounding box and detection confidence score of candidate targets.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] This invention provides a method for accurate localization of occluded targets based on associative memory, comprising the following steps: The image to be detected is acquired and input into a trained interference segmentation and target detection integrated network to obtain the initial detection box, target category, initial detection confidence, target region features and corresponding interference occlusion information of the candidate target, which are used as the recognition result. Based on the interference and occlusion information, the occluded candidate target is determined from the candidate targets; For the occluded candidate target, memory lookup and comparison are performed based on a pre-constructed visual memory bank, which includes multiple memory groups. Each memory group stores the memory features of the unoccluded target sample and the corresponding target attribute information. Candidate memory groups are selected from the visual memory bank according to the target attribute information of the occluded candidate target. The associative memory features corresponding to the occluded candidate target are obtained by combining the target region features of the occluded candidate target with the memory features in the candidate memory groups. Based on the target region features of the occluded candidate target and the associative memory features, the detection results are corrected to obtain the accurate localization result of the target in the image to be detected.

[0012] In one embodiment, the construction of the visual memory bank includes: The training dataset is input into the integrated network for interference segmentation and target detection to obtain target region features and extract target attribute information including target category, target principal axis direction and target region feature response intensity. Memory units are established by screening the target region features and target attribute information of unoccluded target samples, using the target region features as memory features, and dividing the memory units into multiple memory groups according to the target attribute information to form the visual memory bank.

[0013] In one embodiment, determining the occluded candidate target from the candidate targets based on the interference occlusion information includes: Candidate target regions are determined based on the initial detection bounding boxes of candidate targets, and interference occlusion information of the corresponding regions is extracted from the interference occlusion probability map output by the integrated interference segmentation and target detection network. The degree of interference occlusion of candidate targets is determined based on the interference occlusion information, and candidate targets whose interference occlusion degree reaches a preset threshold are identified as occluded candidate targets.

[0014] In one embodiment, prior to memory retrieval and comparison based on the visual memory bank, the method further includes: Local groups are formed based on the initial detection box center position of the occluded candidate targets. Within each local group, the top K candidate targets are retained according to the initial detection confidence, where K is a preset positive integer. Perform the memory lookup and comparison on the retained candidate targets.

[0015] In one embodiment, the local grouping based on the center position of the initial detection bounding box of the occluded candidate target includes: Candidate targets that are occluded are sorted in descending order of initial detection confidence. The candidate target with the highest initial detection confidence is selected from the ungrouped candidate targets as the benchmark candidate target. The center of its initial detection box is used as the benchmark center. The product of the larger of the width and height of the initial detection box and the preset scale coefficient is used as the grouping radius. Candidate targets that are not yet grouped, have the same category as the benchmark candidate target, and whose Euclidean distance from the center of the initial detection box to the benchmark center is not greater than the grouping radius are assigned to the same local group as the benchmark candidate target. Repeat the grouping process described above until all occluded candidate targets have been grouped.

[0016] In one embodiment, the step of selecting a candidate memory group in the visual memory bank based on the target attribute information of the occluded candidate target, and combining the target region features of the occluded candidate target with the memory features within the candidate memory group to obtain the associative memory features corresponding to the occluded candidate target, includes: Based on the identification results of the retained candidate targets, the target category, the target principal axis direction, and the target region feature response intensity are determined, and a memory group with the same category as the candidate target is determined from the visual memory bank. Calculate the mean value of the target principal axis direction and the mean value of the characteristic response intensity of the target region for all memory cells within the memory group; Based on the difference between the target principal axis direction of the candidate target and the mean of the target principal axis direction of the memory group, and the difference between the target region feature response intensity of the candidate target and the mean of the target region feature response intensity of the memory group, the attribute distance is calculated according to a preset weight, and the memory group with the smallest attribute distance is selected as the candidate memory group based on the attribute distance. The target region features of the candidate target are used as query features. The query features and the memory features in the candidate memory group are normalized and the similarity is obtained by dot product calculation. The similarity is normalized to obtain the associative memory weights corresponding to each memory feature. The associative memory weights are then used to perform a weighted summation of the corresponding unnormalized memory features to obtain the associative memory features corresponding to the candidate target.

[0017] In one embodiment, the correction of the detection result based on the target region features of the occluded candidate target and the associative memory features includes: Based on the identification results of candidate targets and the corresponding associative memory features, continuous gating weights for each spatial location are obtained. The associative memory features are adjusted using the continuous gating weights and then concatenated with the target region features of the candidate targets to obtain target correction features; The target correction features are input into the trained detection correction module, which outputs detection box correction parameters and detection confidence correction parameters. Based on these parameters, the initial detection box and initial detection confidence of the candidate target are corrected to obtain the accurate target localization result.

[0018] In one embodiment, obtaining continuous gating weights for each spatial location based on the candidate target identification results and corresponding associative memory features includes: Obtain the corresponding interference occlusion information and target region features from the candidate target identification results; Calculate the feature difference map based on the target region features of the candidate targets and the corresponding associative memory features; After aligning the interference occlusion information and feature difference map in spatial location, they are input into a trained continuous gating weight prediction module to obtain the continuous gating weights for each spatial location.

[0019] In one embodiment, the continuous gating weight prediction module includes three convolutional layers and two nonlinear activation layers; The first non-linear activation layer is located after the first convolutional layer, and the second non-linear activation layer is located after the second convolutional layer. After adjusting the interference occlusion information and feature difference map to the same spatial size, they are stitched together along the channel dimension and used as the input of the first convolutional layer. The first and second convolutional layers are used for feature fusion, and the third convolutional layer outputs the gating response at each spatial location, which is then normalized to obtain continuous gating weights.

[0020] In one embodiment, adjusting the associative memory features using the continuous gating weights and concatenating them with the target region features of the candidate target to obtain target correction features includes: Align the continuous gating weights with the associative memory features in spatial position, and obtain the adjusted associative memory features by multiplying them element by element; Based on preserving the target region features of the candidate targets, the target region features and the adjusted associative memory features are concatenated along the channel dimension to obtain the target correction features.

[0021] In one embodiment, the step of inputting the target correction features into a trained detection correction module and outputting detection box correction parameters and detection confidence correction parameters, thereby correcting the initial detection box and initial detection confidence of the candidate target, includes: The target correction features are input into the trained detection correction module to obtain the detection box correction parameters and the predicted localization quality; the detection box correction parameters include the center position offset parameter and the scale offset parameters of width and height; Based on the center position offset parameter and the scale offset parameter, the center position, width and height of the initial detection box of the candidate target are corrected to obtain the corrected detection box; Based on the predicted positioning quality, a detection confidence correction parameter is determined, and the initial detection confidence of the candidate target is corrected accordingly to obtain the corrected detection confidence. Non-maximum suppression is performed based on the corrected detection bounding box and detection confidence to obtain accurate target localization results.

[0022] In one embodiment, the detection correction module includes: a detection box correction parameter prediction branch and a detection confidence correction parameter prediction branch; each branch includes: two fully connected layers and one non-linear activation layer; the non-linear activation layer is located after the first fully connected layer; the first fully connected layer is used to perform latent space mapping and feature fusion on the target correction features; The second fully connected layer in the detection box correction parameter prediction branch is used to output candidate detection box correction parameters; the detection box correction parameters include center point horizontal correction parameters, center point vertical correction parameters, width scale correction parameters, and height scale correction parameters. The second fully connected layer in the detection confidence correction parameter prediction branch is used to output the predicted localization quality.

[0023] In one embodiment, determining the detection confidence correction parameter based on the predicted positioning quality includes: Based on the difference between the predicted positioning quality and the preset reference positioning quality, a detection confidence correction parameter is determined; the detection confidence is then corrected within the Logit space of the initial detection confidence based on the detection confidence correction parameter to obtain the corrected detection confidence.

[0024] In one embodiment, the integrated network for interference segmentation and target detection includes: a feature extraction network, a target detection branch, and an interference segmentation branch; The feature extraction network is used to extract feature maps from the input image; The target detection branch obtains the initial detection box, target category, and initial detection confidence of the candidate target based on the feature map; The interference segmentation branch obtains an interference occlusion probability map corresponding to the interference occlusion region based on the feature map.

[0025] In one embodiment, the target category, the target principal axis direction, and the target region feature response intensity are determined as follows: For training images, the target category is determined based on the category label of the unoccluded target samples; for images to be detected, the target category is determined based on the predicted target category. For the training image, the horizontal bounding box is used as the target box; for the image to be detected, the detection box is used as the target box; principal component analysis is performed on the spatial distribution of significant pixels whose pixel intensity is greater than the mean pixel intensity in the region within the target box, and the direction of the eigenvector corresponding to the largest eigenvalue of the two-dimensional coordinate covariance matrix is ​​used as the target principal axis direction; the angle corresponding to the target principal axis direction is mapped to [0°, 180°); The feature response intensity of the target region is obtained by averaging the feature responses of the target region in the channel dimension and the spatial dimension.

[0026] In one embodiment, dividing the memory unit into multiple memory groups based on the target attribute information includes: The attribute labels of each memory unit are determined based on the target category, the preset direction interval to which the target principal axis direction belongs, and the preset intensity level interval to which the target region feature response intensity belongs. Memory units with the same attribute labels are grouped into the same memory group.

[0027] In one embodiment, the preset direction interval and the preset intensity level interval are determined in the following manner: The [0°, 180°) range is divided into multiple non-overlapping preset direction intervals, and the preset direction interval to which the target belongs is determined according to the angle corresponding to the main axis direction of the target. Based on the statistical distribution of target region feature response intensity of unoccluded target samples in the training dataset, the boundaries of multiple preset intensity level intervals are determined, and the preset intensity level interval to which the target region belongs is determined based on the target region feature response intensity.

[0028] In one embodiment, the construction of the visual memory bank and the training of the detection and correction module are carried out in a phased manner: First, a visual memory bank is constructed using complete target samples in the training dataset whose interference occlusion degree is no higher than the first threshold, and the integrated network for interference segmentation and target detection and the visual memory bank are fixed. Then, using the candidate targets before non-maximum suppression as training samples, the occluded candidate targets are selected according to the degree of interference occlusion in the candidate target area, and the detection correction module and the continuous gating weight prediction module are trained. During the inference phase, visual memory association and detection result correction are performed only on the K candidate targets that are occluded and enter the first set of the corresponding local group. The remaining candidate targets retain the initial detection box and initial detection confidence. Finally, non-maximum suppression is performed after merging all candidate targets.

[0029] As can be seen from the above technical solution, this invention provides an accurate target localization method based on associative memory. It draws upon the process in human visual cognition of associating existing memories with partially visible information to infer the overall shape of a target. By using the visual memory of the unoccluded target as a priori information and combining it with the current occlusion state of the target, it adaptively fuses target features and memory features to compensate for the loss of target information caused by occlusion, thereby achieving target localization and confidence level correction. Specific beneficial effects are as follows: (1) By using the target region features of the unoccluded target samples to construct a visual memory bank, a complete target feature prior is provided for the occluded target, which helps to supplement the target feature loss caused by occlusion and reduce the positioning deviation caused by the shrinkage of the detection box to the local visible area.

[0030] (2) Organize memory features according to target category, target axis direction and target region feature response intensity, and combine attribute matching and feature similarity to retrieve memory, so that the acquisition of associative memory features has clear target attribute basis and reduces the participation of irrelevant memory features.

[0031] (3) Before non-maximum suppression, the occluded candidate targets are locally grouped and the top-ranked candidate targets in each group are retained. This provides an opportunity to correct multiple candidate detection boxes in the same local area, while limiting the number of candidate targets participating in memory association and controlling additional processing overhead.

[0032] (4) Combine the interference occlusion information and the difference between the current target area features and the associative memory features to generate continuous gating weights for each spatial location and adjust the associative memory features; retain the current target area features through channel splicing so that the detection correction module can simultaneously use the current observation information and the memory prior information to jointly correct the detection box and the detection confidence. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0034] Figure 1 A flowchart of the method for accurate localization of occluded targets based on associative memory provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the integrated network structure for interference segmentation and target detection provided in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the construction of a visual memory library with target attribute information provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the continuous gating weight prediction module structure provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the detection and correction module structure provided in an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the correction effect provided in an embodiment of the present invention; Figure 7 This is a comparison chart of detection performance provided in an embodiment of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] This invention discloses a method for accurate localization of occluded targets based on associative memory. The overall process is as follows: Figure 1 As shown, it includes the following steps: S1. Obtain the image to be detected, input the image to be detected into the trained interference segmentation and target detection integrated network, and obtain the initial detection box of the candidate target, the target category, the initial detection confidence, the target region features and the corresponding interference occlusion information as the recognition result; S2. Determine the occluded candidate target from the candidate targets based on the interference occlusion information; S3. For the occluded candidate target, memory search and comparison are performed based on a pre-constructed visual memory bank. The visual memory bank includes multiple memory groups, each storing the memory features of the unoccluded target sample and the corresponding target attribute information. Candidate memory groups are selected in the visual memory bank according to the target attribute information of the occluded candidate target, and the associative memory features corresponding to the occluded candidate target are obtained by combining the target region features of the occluded candidate target with the memory features in the candidate memory group. S4. Based on the target region features of the occluded candidate target and the associative memory features, the detection results are corrected to obtain the accurate positioning result of the target in the image to be detected.

[0037] This invention can utilize the complete memory of an unobstructed target to compensate for feature loss caused by occlusion, thereby improving the accuracy and reliability of target localization under complex occlusion conditions.

[0038] The following is combined with Figures 1-7 The specific implementation methods of the embodiments of the present invention will be described in detail, which mainly involve an integrated network for interference segmentation and target detection, pre-construction of a visual memory bank, screening of occluded candidate targets, local grouping and retention of candidate targets, memory search and comparison based on the visual memory bank, detection result correction, and training methods.

[0039] (a) An integrated network for interference segmentation and target detection; Reference Figure 2As shown, this embodiment uses a trained interference segmentation and target detection integrated network to process the input image, so as to simultaneously obtain the initial detection results of the candidate target, the target region features corresponding to the candidate target, and the interference occlusion probability of each spatial location in the input image; the initial detection results include: initial detection box, target category, and initial detection confidence.

[0040] The integrated interference segmentation and target detection network includes a feature extraction network, a target detection branch, and an interference segmentation branch. The target detection branch and the interference segmentation branch are connected to the feature extraction network and share the intermediate features output by the feature extraction network.

[0041] The feature extraction network can employ a convolutional neural network-based backbone, such as ResNet, ResNeXt, and EfficientNet, or a Transformer-based backbone, such as Swin Transformer and ViT. The object detection branch can use an anchor-box-based detection head, such as Faster R-CNN and RetinaNet, or an anchor-box-free detection head, such as FCOS and CenterNet. The interference segmentation branch can use a semantic segmentation head, mapping the feature map to an interference occlusion probability map with the same size as the input image through convolutional layers and upsampling operations. The object detection branch and the interference segmentation branch share the multi-layer feature map output by the feature extraction network, ensuring feature consistency between the object detection and interference segmentation tasks, and enhancing the feature extraction network's ability to represent target structures and interference occlusion regions through multi-task learning.

[0042] (1) A feature extraction network is used to perform step-by-step feature extraction and multi-scale feature fusion on the input image. For example, it includes a five-level convolutional feature extraction unit, a four-level downsampling unit, a spatial receptive field merging unit, and a two-level top-down feature fusion unit. The number of output channels of the five-level convolutional feature extraction unit is 32, 64, 128, 256, and 512, respectively. Each level of the convolutional feature extraction unit includes a convolutional layer and a non-linear activation layer. The non-linear activation layer uses the LeakyReLU activation function. A downsampling unit is set before the second to fifth level convolutional feature extraction units to gradually reduce the spatial size of the feature map and increase the receptive field of the feature map.

[0043] The input image is first resized to a preset input size and then pixel-normalized. In one specific setting, for example, the preset input size is 512×640 pixels. The normalized input image is then passed through five levels of convolutional feature extraction units to obtain the third-layer feature T3, the fourth-layer feature T4, and the fifth-layer feature T5. The third-layer feature T3 contains more information about the target edges and local details, the fourth-layer feature T4 contains the target's local structure and semantic information, and the fifth-layer feature T5 has a large receptive field and contains high-level semantic information about the target.

[0044] Next, the fifth-layer feature T5 is input into the spatial receptive field merging unit. The spatial receptive field merging unit includes a first 1×1 convolutional layer, a multi-scale max pooling layer, and a second 1×1 convolutional layer. The first 1×1 convolutional layer is used to perform channel mapping on the fifth-layer feature T5; the multi-scale max pooling layer uses three pooling windows of different sizes to perform parallel pooling on the channel-mapped features to extract contextual information under different receptive fields; the pooling results corresponding to different pooling windows are concatenated with the unpooled features along the channel dimension, and then channel fusion is performed through the second 1×1 convolutional layer to obtain the fifth-layer target feature P5.

[0045] The fifth-layer target feature P5 is upsampled by a factor of two using nearest-neighbor interpolation, so that the upsampled feature has the same spatial size as the fourth-layer feature T4. The upsampled fifth-layer target feature P5 and the fourth-layer feature T4 are concatenated along the channel dimension and then fused through a convolutional layer to obtain the fourth-layer target feature P4. Subsequently, the fourth-layer target feature P4 is upsampled by a factor of two using nearest-neighbor interpolation, so that the upsampled feature has the same spatial size as the third-layer feature T3. The upsampled fourth-layer target feature P4 and the third-layer feature T3 are concatenated along the channel dimension and then fused through a convolutional layer to obtain the third-layer target feature P3.

[0046] With an input image size of 512×640 pixels, and downsampling factors of 8, 16, and 32 relative to the input image for the third-layer target feature P3, fourth-layer target feature P4, and fifth-layer target feature P5, the spatial dimensions of these features are 64×80, 32×40, and 16×20, respectively, and the number of channels are 128, 256, and 512. The third-layer target feature P3 is used to represent small-sized targets and local details, the fourth-layer target feature P4 is used to represent medium-sized targets, and the fifth-layer target feature P5 is used to represent larger-sized targets and high-level semantic information.

[0047] (2) Target detection branch, which receives the third-layer target features P3, the fourth-layer target features P4, and the fifth-layer target features P5, and detects targets of different sizes at the three feature levels respectively. This target detection branch includes a scale transformation convolutional layer, a feature fusion layer, and three prediction heads corresponding to the third-layer target features P3, the fourth-layer target features P4, and the fifth-layer target features P5 respectively.

[0048] Specifically, after the third-layer target feature P3 is processed by the first-scale transform convolutional layer, its spatial size is adjusted to be the same as that of the fourth-layer target feature P4, and then concatenated with the fourth-layer target feature P4 along the channel dimension to obtain the first fused feature. After the first fused feature is processed by the second-scale transform convolutional layer, its spatial size is adjusted to be the same as that of the fifth-layer target feature P5, and then concatenated with the fifth-layer target feature P5 along the channel dimension to obtain the second fused feature. The third-layer target feature P3, the first fused feature, and the second fused feature are respectively input into the corresponding prediction heads to complete the detection of targets of different sizes.

[0049] Each prediction head includes a feature fusion convolutional layer and an output convolutional layer. The feature fusion convolutional layer performs local feature fusion on the input features, and the output convolutional layer outputs the candidate target prediction parameters corresponding to each spatial location. For each spatial location with N anchor boxes and C target categories, the output convolutional layer has N×(C+5) output channels. Here, C parameters represent the predicted value for each target category, and the remaining 5 parameters represent the horizontal offset of the candidate target detection box center point, the vertical offset of the center point, the width scale parameter, the height scale parameter, and the target confidence level, respectively.

[0050] Based on the candidate target prediction parameters and the position and size of the corresponding anchor boxes, the candidate target detection boxes are decoded to obtain the initial detection boxes, target categories, and initial detection confidence scores of the candidate targets. This embodiment does not directly predict the target principal axis direction through the target detection branch. Instead, after obtaining the target region features of the candidate targets, the target principal axis direction is calculated based on the spatial distribution of significant pixel response positions within the target region, thus ensuring consistency between the target principal axis direction and the target attribute calculation method used in the subsequent visual memory database.

[0051] The number and size of anchor boxes are determined based on the size distribution of target bounding boxes in the training dataset. Specifically, the normalized width and height of each target bounding box in the training dataset are clustered, and the width and height of the anchor boxes are determined based on the cluster centers. Different anchor boxes are then assigned to the third-layer target feature P3, the fourth-layer target feature P4, and the fifth-layer target feature P5 according to the target size. The actual value of the number of anchor boxes N is 3, selected based on the average detection precision, target recall, and inference time on the validation set.

[0052] (3) The interference segmentation branch is a pixel-level binary classification semantic segmentation branch used to determine whether each pixel in the input image belongs to an interference occlusion region. The interference segmentation branch does not distinguish between different interference instances in the same category of interference regions, but outputs an interference occlusion probability map corresponding to the spatial location of the input image.

[0053] The interference segmentation branch takes the third-layer target feature P3 as input and includes four convolutional layers, three upsampling layers, and two channel weight correction units. The number of output channels for the first, second, third, and fourth convolutional layers are 128, 32, 8, and 2, respectively. All three upsampling layers use nearest-neighbor interpolation to double the length and width of the input feature map. The first channel weight correction unit is placed after the first upsampling layer, and the second channel weight correction unit is placed after the second upsampling layer.

[0054] Each channel weight correction unit includes a feature mapping convolutional layer, a global pooling layer, a channel mapping layer, and a weight activation layer. The feature mapping convolutional layer is used to extract local correlations between different channels; the global pooling layer is used to compress the two-dimensional features of each channel into corresponding channel descriptors; the channel mapping layer is used to generate weights for each channel based on the channel descriptors; and the weight activation layer is used to constrain the weights within a preset numerical range. Each channel weight is multiplied channel-by-channel with its corresponding input feature to obtain the channel-corrected features, thereby enhancing the channel responses related to interference regions.

[0055] The third-layer target feature P3, after being processed by convolutional layers, upsampling layers, and channel weight correction units, yields a dual-channel output feature with the same spatial size as the input image. The first channel of the dual-channel output feature represents the predicted value of a pixel belonging to a non-interference region, and the second channel represents the predicted value of a pixel belonging to an interference region. Softmax normalization is performed on the dual-channel output feature to obtain the probability that each pixel belongs to an interference region, and the interference probabilities of all pixels are used to construct an interference occlusion probability map O. For any spatial location (x, y), the interference occlusion probability O(x, y) ranges from [0, 1], with a larger value indicating a higher probability that the spatial location belongs to an interference occlusion region.

[0056] (4) Training method: In order to enable the target detection branch and the interference segmentation branch to form a substantial synergistic relationship, this embodiment adopts a training method that combines shared features and joint loss constraints. The target detection branch and the interference segmentation branch share the feature extraction network, so that both the detection loss and the segmentation loss can update the parameters of the feature extraction network through backpropagation.

[0057] The loss function in the object detection branch includes bounding box regression loss, object confidence loss, and object category loss. The bounding box regression loss, using the IoU loss function, constrains the positional and size differences between the predicted bounding boxes and the ground truth bounding boxes. The object confidence loss, using the binary cross-entropy loss function, constrains the presence of an object at a candidate location. The object category loss, using the Focal loss function, constrains the object category prediction result. The loss function in the interference segmentation branch is pixel-level binary cross-entropy loss.

[0058] The joint loss function of the object detection branch and the noise segmentation branch is expressed as:

[0059] in, This represents the joint loss of the integrated network. This represents the regression loss of the detection box. Indicates the target confidence loss. Indicates the target category loss. Indicates the interference segmentation loss. , , and These represent the weighting coefficients of the corresponding loss terms.

[0060] During training, the gradients of each parameter in the feature extraction network, object detection branch, and interference segmentation branch are calculated based on the joint loss function, and the parameters are updated through backpropagation and an optimizer. Thus, the supervisory information generated by the interference segmentation branch is applied to the object detection branch through shared features, enabling the feature extraction network to learn the spatial distribution characteristics of the interference region while maintaining its object discrimination ability.

[0061] The input image is processed by a feature extraction network to obtain a feature map. The object detection branch outputs the initial detection box, object category, and initial detection confidence of the candidate object based on the feature map; the interference segmentation branch outputs an interference occlusion probability map based on the feature map, where the value at each spatial location is used to represent the probability that interference occlusion exists at the corresponding location.

[0062] Target region features are extracted by mapping the feature map output by the detection network to the location coordinates of the target region. First, based on the coordinates of the target bounding box in the original image and the downsampling ratio of the feature map relative to the original image, the target bounding box coordinates are mapped onto the corresponding feature map, thereby determining the region range of the target in the feature map. Subsequently, the corresponding local features are extracted from this region, and the target regions of different sizes are unified into a fixed-size feature representation using the ROI Align region feature extraction method, thus obtaining the target region features corresponding to the target.

[0063] In the visual memory bank construction phase, the target region is determined based on the target bounding boxes of the training samples; in the image processing phase, the target region is determined based on the initial detection boxes of the candidate targets. This yields the initial detection boxes of the candidate targets, target categories, initial detection confidence, target region features, and interference / occlusion information, which are then used as inputs for subsequent occluded candidate target selection and visual memory association.

[0064] (ii) Pre-construction of the visual memory bank; The process of building a visual memory bank with target attributes is as follows: Figure 3 As shown.

[0065] The construction of the visual memory bank includes: inputting the training dataset into the integrated network of interference segmentation and target detection to obtain target region features, extracting target attribute information including target category, target principal axis direction and target region feature response intensity; selecting target region features and target attribute information of unoccluded target samples to establish memory units, using target region features as memory features, and dividing the memory units into multiple memory groups according to target attribute information to form a visual memory bank.

[0066] Specifically, the images in the training dataset are input into an integrated network for interference segmentation and object detection to obtain corresponding feature maps. The target region is determined based on the annotation information of the target samples, and the corresponding target region features are extracted from the feature maps.

[0067] Unoccluded target samples are selected from the training samples, and the target region features of these samples are used as memory features. For each selected target sample, its target category, target principal axis direction, and target region feature response intensity are determined, and the memory features are associated with the above target attribute information to establish a memory unit.

[0068] The target category is determined based on the category labels of the training samples. The target principal axis direction is used to represent the main extension direction of the target, and its direction angle is mapped to [0°, 180°), so that directions that differ by 180° correspond to the same principal axis direction.

[0069] The target region feature response intensity is obtained by averaging the response values ​​of all channels and spatial locations of the target region feature, and is used to represent the overall feature response level of the target region. Let the target region feature be... ,in, It is the set of real numbers; For the number of channels, and Let the height and width of the target region feature be the values, then the response intensity of the target region feature is... Represented as:

[0070] This involves averaging all response values ​​across the target region's features; c, h, and w represent the channel index, the row index of the current channel, and the column index of the current channel, respectively. The visual memory bank construction phase and the candidate target retrieval phase employ the same response intensity calculation method.

[0071] Furthermore, the response distribution is determined based on the pixel values ​​corresponding to the spatial locations within the target area, and a set of significant response locations is constructed based on the spatial distribution of response locations within the target area. Pixels with a value greater than the average pixel intensity within the area are considered significant responses.

[0072] Let the set of significant response locations contain the first... The coordinates of each position are The number of significant response locations is The spatial center coordinates of the significant response location are then expressed as: ,

[0073] Construct a two-dimensional covariance matrix based on the coordinate offset of the significant response location relative to the spatial center:

[0074] Principal component analysis is performed on the two-dimensional covariance matrix, and the eigenvector corresponding to the largest eigenvalue is taken as the first principal component of the spatial distribution of the significant response location, and the direction of the first principal component is determined as the target principal axis direction.

[0075] Furthermore, [0°, 180°) is divided into multiple non-overlapping preset direction intervals, and the division boundary of the preset intensity level interval is determined based on the statistical distribution of the feature response intensity of the unoccluded target samples in the training dataset.

[0076] Map the target principal axis angle to Within the range, the number of preset direction categories is set to... The angular interval between adjacent direction categories is

[0077] Set according to this angle interval A preset direction is defined, and the difference between the target's principal axis angle and the angles of each preset direction is calculated. The target is then classified into the direction category with the smallest angle difference. As one specific implementation method, when... At that time, the four directional categories correspond to and .

[0078] During the construction of the visual memory bank, the principal axis direction attribute of each unoccluded complete target sample is calculated in the manner described above. The attribute information corresponding to the target sample is determined based on the principal axis direction attribute. After the visual memory unit is formed, the memory group to which it belongs is determined based on the attribute information of the sample corresponding to the memory unit, so that subsequent candidate targets are preferentially matched with complete target memories that have the same direction attribute.

[0079] In this embodiment, attribute labels are determined for each memory unit based on the target category, the direction interval to which the target's main axis direction belongs, and the intensity level interval to which the target region's feature response intensity belongs.

[0080] Memory units with the same attribute labels are organized into the same memory group, and multiple memory groups constitute a visual memory bank. Each memory group stores target memory features that have the same category, similar directional attributes, and similar response intensity levels.

[0081] Each target memory in the visual memory bank includes at least key features for memory retrieval and value features for memory reading. Both key and value features are derived from the target region features of the corresponding complete target sample. Target region features are extracted from the corresponding region of the target bounding box on the feature map of the detection network and uniformly adjusted to a preset size, thus ensuring a consistent feature representation for different target samples. In one implementation, the target region features are normalized to obtain the corresponding key features, which are used to calculate similarity with the query features of the target to be detected; simultaneously, the target region features before normalization are retained as the corresponding value features for subsequent memory information reading. Therefore, key and value features can be obtained from the same target region features through different processing methods, without needing to repeatedly extract two sets of features from the original image.

[0082] In subsequent retrieval processes, normalized memory features are used to calculate feature similarity, while unnormalized memory features are used for weighted reading to preserve the original feature response information for target correction.

[0083] In the process of constructing the visual memory bank, target samples with an interference occlusion degree of no more than 1% are identified as unoccluded complete target samples, and target memory features are extracted using complete target samples to establish the visual memory bank.

[0084] (iii) Screening of occluded candidate targets; The image to be detected is input into an integrated network for interference segmentation and target detection to obtain candidate targets before non-maximum suppression, including the initial detection bounding box, target category and initial detection confidence of each candidate target, and the interference occlusion probability map of the image to be detected.

[0085] The target region is determined based on the initial detection bounding box of each candidate target. The target region features are extracted from the feature map output by the network, and the probability map of the corresponding region is extracted from the interference occlusion probability map as the interference occlusion information of the candidate target.

[0086] The initial detection bounding box, target category, initial detection confidence, target region features, and interference occlusion information of the candidate target are used together as the recognition result. The degree of interference occlusion is determined based on the interference occlusion information corresponding to the candidate target region and compared with a preset threshold. Candidate targets that reach the threshold are identified as occluded candidate targets. The interference occlusion probability map represents the probability of a pixel being occluded; therefore, the average occlusion probability within a target region reflects the degree of occlusion within that region. For example, a candidate ship target region contains 10,000 pixels. According to the interference occlusion probability map, the average interference occlusion probability of each pixel in this region is 0.42. Therefore, the interference occlusion degree of this candidate target is 0.42. If the preset interference occlusion threshold is 0.30, since 0.42 is greater than 0.30, the candidate ship target is identified as an occluded candidate target, and subsequent visual memory retrieval and localization correction are performed on it. If the interference occlusion degree of another candidate target is 0.15, it does not reach the preset threshold and is not identified as an occluded candidate target.

[0087] Specifically, for candidate targets generated by the detection network, the target region is extracted based on the corresponding position of the initial detection box of the candidate target in the interference occlusion probability map, and the average interference occlusion probability within the candidate target region is calculated.

[0088] Let the location within the candidate target area be... The corresponding interference occlusion probability is The number of spatial locations contained in the candidate target region is The degree of interference and occlusion of the candidate target Represented as:

[0089] in, The range of values ​​is The larger the value, the higher the degree of interference and occlusion of the candidate target area.

[0090] The degree of interference occlusion of candidate targets Compared with a preset occlusion threshold, when When the preset occlusion threshold is reached, the corresponding candidate target is identified as an occluded candidate target.

[0091] During the construction of the visual memory bank, target samples with an interference occlusion level of no more than 1% are defined as unoccluded complete target samples, and target memory features are extracted using complete target samples to establish the visual memory bank. In the subsequent training of the detection and correction module, candidate targets with an interference occlusion level of no less than 10% are used as occluded training samples.

[0092] Interference and occlusion level This is used only for screening occluded candidate targets. In subsequent visual memory association and feature fusion processes, the interference occlusion probability corresponding to each spatial location within the candidate target area is retained to form a spatial interference occlusion probability map. This describes the interference distribution at different locations within the target area and serves as the input for subsequent continuous gating weight calculations.

[0093] The above screening process is used to identify candidate targets that require memory association and detection correction, avoiding memory retrieval for all candidate targets.

[0094] Furthermore, target region features are extracted from the intermediate feature map output by the integrated interference segmentation and target detection network: For candidate targets, based on the mapping position of their initial detection boxes on the corresponding intermediate feature map, the target region is sampled through a region feature alignment operation to map candidate targets of different sizes into target region features with uniform spatial size.

[0095] Let the first The feature space size of the target region corresponding to the intermediate feature map of the layer is The number of channels is After region feature alignment, the resulting size is... The characteristics of the target region.

[0096] Based on the positional mapping relationship of the same candidate bounding boxes, corresponding regions are extracted from the interference occlusion probability map, and the same region feature alignment method is used to obtain a size of [missing information]. The spatial interference occlusion probability map is used to ensure that it maintains a spatial correspondence with the features of the target region.

[0097] For associative memory features retrieved from the visual memory bank, the spatial dimensions are also adjusted to match those of the current target region features. When the number of channels of the target region features and the associative memory features is inconsistent, channel projection operations are used to map them to the same feature dimension, and feature difference calculations and subsequent fusion are performed after unifying the feature space.

[0098] The visual memory bank construction stage and the candidate target retrieval stage use the same feature level, region feature alignment method, target box coordinate mapping rule and channel projection method to ensure that the visual memory features and the current candidate target features are in the same feature representation space.

[0099] (iv) Local grouping and retention of candidate targets; Before performing memory lookup and comparison based on the visual memory bank, the candidate targets are locally grouped according to the center position of the initial detection box of the occluded candidate targets. Within each local group, the top K candidate targets are retained according to the initial detection confidence, where K is a preset positive integer. Memory lookup and comparison are then performed on the retained candidate targets.

[0100] Specifically, candidate targets identified as occluded are sorted according to their initial detection confidence scores from highest to lowest. The candidate target with the highest initial detection confidence score is selected from the ungrouped candidate targets as the baseline candidate target. The center of its initial detection box is used as the baseline center, and the product of the larger of its width and height and a preset scale coefficient is used as the grouping radius. For example, among the ungrouped candidate targets, candidate target A has the highest initial detection confidence score, its initial detection box center coordinates are (100, 80), its width is 40, its height is 20, and its preset scale coefficient is 1.5. Since the larger of its width and height is 40, the grouping radius is... Using (100, 80) as the reference center, candidate targets whose center of the detection box is no more than 60 units away from the reference center are grouped into the same local group.

[0101] For the remaining ungrouped candidate targets, compare their target categories and the center position of their initial detection boxes. When a candidate target has the same category as the baseline candidate target, and the Euclidean distance from its initial detection box center to the baseline center is not greater than the grouping radius, it is assigned to the same local group as the baseline candidate target.

[0102] After completing the current grouping, a new baseline candidate target is selected from the remaining ungrouped candidate targets, and the above process is repeated until all occluded candidate targets are grouped.

[0103] Within each local group, the top K candidate targets are retained in descending order of initial detection confidence, where K is a preset positive integer; if the number of candidate targets in a group is less than K, all candidate targets in that group are retained.

[0104] Through the above processing, similar candidate targets that are spatially close are grouped into local groups, and the number of candidate targets entering the memory association stage in each local group is controlled.

[0105] (v) Memory retrieval and comparison based on visual memory bank: For occluded candidate targets, memory lookup and comparison are performed based on a pre-built visual memory bank. Specifically, the target category, target principal axis direction, and target region feature response intensity are determined based on the recognition results of the retained candidate targets, and memory groups with the same category as the candidate targets are identified from the visual memory bank.

[0106] First, memory groups of the same category are identified from the visual memory bank based on the candidate target categories. Then, based on the target principal axis direction and the target region feature response intensity of the memory units within each memory group, the mean value of the target principal axis direction and the mean value of the target region feature response intensity of that memory group are determined.

[0107] The directional difference and response intensity difference between the candidate target and each memory group of the same category are calculated respectively. The attribute distance is calculated according to the preset weight, and the memory group with the smallest attribute distance is selected as the candidate memory group.

[0108] Specifically, the mean of the target principal axis direction and the mean of the target region feature response intensity of all memory units in the memory group are calculated; based on the difference between the target principal axis direction of the candidate target and the mean of the target principal axis direction of the memory group, and the difference between the target region feature response intensity of the candidate target and the mean of the target region feature response intensity of the memory group, the attribute distance is calculated according to the preset weight, and the memory group with the smallest attribute distance is selected as the candidate memory group.

[0109] The aforementioned attribute matching is used to narrow down the feature retrieval scope, so that subsequent similarity calculations are performed within the memory group related to the current candidate target attribute.

[0110] Then, the target region features of the candidate target are used as query features. The same feature dimension processing and L2 normalization processing are applied to the query features and the memory features within the candidate memory group. Subsequently, the normalized query features are subjected to dot product operations with each memory feature to obtain the corresponding feature similarity. The dot product results are used to characterize the similarity between the current candidate target and each memory feature in the feature space, and the candidate memory features are ranked according to the feature similarity.

[0111] Similarity is normalized to obtain the associative memory weights corresponding to each memory feature. Similarity normalization is used to convert the matching results of different memory features into weighted reading coefficients. Using the associative memory weights, the corresponding unnormalized memory features are weighted and summed to obtain the associative memory features corresponding to the current candidate target. That is, for a candidate target in the image to be detected, the target region features are extracted from the corresponding feature map according to its initial detection box, and its ROI is aligned to the same preset size as the target region features in the visual memory bank to obtain the query features. After the query features are normalized in the same way as the key features, the similarity is calculated with each key feature in the candidate memory group; then the obtained similarity is normalized to obtain the associative memory weights corresponding to each memory. Finally, the associative memory weights are used to weight and sum the corresponding value features to obtain the associative memory features corresponding to the current candidate target.

[0112] Associative memory features are derived from the memory features of unoccluded targets and are used to provide relevant prior information about the current candidate target.

[0113] In subsequent processing, the target region features of the candidate target are denoted as R, and the corresponding associative memory features are denoted as M.

[0114] (vi) Correction of test results: The detection results are corrected based on the target region features and associative memory features of the occluded candidate targets. Specifically, based on the recognition results of the candidate targets and the corresponding associative memory features, continuous gating weights for each spatial location are obtained; the associative memory features are adjusted using the continuous gating weights and concatenated with the target region features of the candidate targets to obtain the target correction features; the target correction features are input into the trained detection correction module, which outputs detection box correction parameters and detection confidence correction parameters, thereby correcting the initial detection box and initial detection confidence of the candidate targets to obtain the accurate target localization result.

[0115] 1) Continuous Gated Weight Prediction Module: Obtain interference and occlusion information from the candidate target recognition results, and calculate the feature difference map based on the target region feature R and the associative memory feature M.

[0116] The interference occlusion information and feature difference map corresponding to the candidate target region are adjusted to the same spatial size so that their spatial positions in the target region correspond to each other. Then, they are stitched together along the channel dimension and used as the input of the continuous gating weight prediction module to obtain the continuous gating weights of each spatial position.

[0117] The feature difference map is calculated based on the target region features of the current candidate target and the associative memory features retrieved from the visual memory bank: First, the target region features and corresponding associative memory features of the current candidate target are adjusted to have the same spatial size and channel dimension. Let the unified target region features be... The characteristics of associative memory are Both have the same number of channels. .

[0118] To each and Normalization is performed along the channel dimension to eliminate the influence of different feature response scales on the difference calculation.

[0119] For spatial location The absolute differences between the normalized target region features and the associative memory features in each channel are calculated and aggregated along the channel dimension to obtain a single-channel feature difference map. :

[0120] in, and These represent the normalized target region features and associative memory features, respectively. , represents the channel index.

[0121] Feature difference map It is used to describe the degree of difference between the observed features of the current candidate target and the memory features of the complete target at different spatial locations, and serves as one of the inputs for subsequent continuous gating weight calculations.

[0122] This continuous gating weight prediction module, such as Figure 4 As shown, it includes three convolutional layers and two nonlinear activation layers, connected in the order of the first convolutional layer, the first nonlinear activation layer, the second convolutional layer, the second nonlinear activation layer, and the third convolutional layer.

[0123] like Figure 4 As shown, the interference occlusion information and feature difference map are adjusted to the same spatial size and then stitched along the channel dimension, serving as the input to the first convolutional layer. The first and second convolutional layers fuse the input interference occlusion information and feature difference information, and the third convolutional layer outputs the gating response at each spatial location. The gating response is then normalized to obtain the continuous gating weight G.

[0124] Specifically, the spatial interference occlusion probability map O and the feature difference map D are first adjusted to the same spatial size and then stitched together along the channel dimension to form gated input features.

[0125] The gated input features pass through a first convolutional layer, a first nonlinear activation layer, a second convolutional layer, a second nonlinear activation layer, and a third convolutional layer in sequence. The first and second convolutional layers jointly model the distribution of interference occlusion within the candidate target region and the feature differences between the currently observed features and the complete target memory. The third convolutional layer outputs the gated response corresponding to each spatial location. .

[0126] The continuous gating weight G is obtained by normalizing the gating response using the Sigmoid function:

[0127] The value range of G(x,y) is [0,1].

[0128] Gating weights are used to spatially adjust associative memory features, so that memory features introduced at different locations are affected by both the occlusion information of candidate targets and the feature difference information.

[0129] 2) Construction of target correction features: Align the continuous gating weights G and the associative memory features M in spatial position, and obtain the adjusted associative memory features by multiplying them element by element.

[0130] in, This indicates the characteristics of adjusted associative memory. This indicates element-wise multiplication.

[0131] Therefore, based on the interference and occlusion status of different spatial locations of the candidate target and the feature differences between the current observation features and the complete target memory, the degree of introducing associative memory features at the corresponding spatial locations is adaptively adjusted.

[0132] While retaining the target region features R of the candidate targets, the target region features R are combined with the adjusted associative memory features. By splicing along the channel dimension, the target correction feature Z is obtained.

[0133]

[0134] in, This indicates splicing along the channel dimension.

[0135] In this way, the target correction features simultaneously include the target region features provided by the current image and the gated associative memory features, which serve as input to the detection correction module.

[0136] 3) Detection and Correction Module: Reference Figure 5As shown, the target correction features are used as input to the detection correction module. The detection correction module includes a detection box correction parameter prediction branch and a detection confidence correction parameter prediction branch. Each branch includes two fully connected layers and one non-linear activation layer. The non-linear activation layer is located after the first fully connected layer. The first fully connected layer is used to perform latent space mapping and feature fusion on the target correction features. The second fully connected layer in the prediction branch for the detection box correction parameters is used to output the candidate detection box correction parameters. The second fully connected layer in the prediction branch of the detection confidence correction parameter is used to output the predicted localization quality.

[0137] The detection box correction parameters include the center point horizontal offset parameter, the center point vertical offset parameter, the width scale offset parameter, and the height scale offset parameter, which are denoted as follows: , , and The detection confidence correction parameter is denoted as... .

[0138] In this embodiment, the initial detection bounding box of the candidate target is represented as:

[0139] in, , The coordinates of the detection box center are, , The detection box width and height are defined. Based on the parameter decoding method in the original scheme, the corrected detection box is as follows:

[0140]

[0141]

[0142]

[0143] This yields the corrected detection box:

[0144] in, This indicates exponential operations. The center position offset parameter is used to adjust the center of the detection box, and the scale offset parameter is used to adjust the width and height of the detection box.

[0145] Adjust parameters based on detection confidence. The initial detection confidence of the candidate targets is corrected to obtain the corrected detection confidence. Then, non-maximum suppression is performed based on the corrected detection bounding box and the detection confidence to obtain the target localization result.

[0146] In another embodiment, the detection correction module obtains the predicted localization quality of the candidate target based on the target correction features, and corrects the initial detection confidence of the candidate target based on the predicted localization quality: The target correction features are expanded and input into the detection bounding box correction parameter prediction branch and the detection confidence correction parameter prediction branch, respectively. The detection bounding box correction parameter prediction branch outputs the detection bounding box correction parameters, while the detection confidence correction parameter prediction branch outputs the predicted localization quality of the candidate target after correction.

[0147] Let the predicted positioning quality after candidate target correction be... Then, the output of the localization quality prediction branch is mapped to the Sigmoid function. Within the range.

[0148] During training, the initial detection boxes of candidate targets are corrected based on the detection box correction parameters to obtain the corrected detection boxes. The corrected detection bounding box is compared with the corresponding ground truth bounding box. The intersection-union ratio between them is used as the target for quality supervision in positioning:

[0149] in, The range of values ​​is Positioning quality is predicted using positioning quality loss constraints. With positioning quality supervision objectives The difference between them allows the predicted localization quality to reflect the accuracy of the corrected detection box localization. As a specific implementation, the localization quality loss uses a binary cross-entropy loss:

[0150] During the inference process, let the initial detection confidence of the candidate target output by the original target detection network be . Predicted positioning quality is The preset reference positioning quality is The quality adjustment coefficient is Based on the difference between the predicted positioning quality and the reference positioning quality, the detection confidence correction parameter is determined. :

[0151] in, Represents the Logit transformation; The preset reference positioning quality is obtained from the positioning quality statistics of candidate targets during the training phase, and is used as a neutral reference with unchanged confidence. This is a preset quality adjustment coefficient used to adjust the degree of influence of positioning quality on detection confidence.

[0152] Based on the detection confidence correction parameters, the initial detection confidence is corrected within the Logit space to obtain the corrected detection confidence. :

[0153] Right now:

[0154] To avoid numerical anomalies in the Logit transformation when the input value is 0 or 1, the initial detection confidence, predicted positioning quality, and reference positioning quality can be limited to a preset effective range before the transformation is performed.

[0155] When predicting positioning quality Equal to reference positioning quality At that time, the confidence correction parameter was detected. If the value is 0, the corrected detection confidence remains the same as the initial detection confidence. When the predicted positioning quality is higher than the reference positioning quality, the detection confidence of the corresponding candidate target is increased; when the predicted positioning quality is lower than the reference positioning quality, the detection confidence of the corresponding candidate target is decreased.

[0156] Specifically, the quality logit of each valid candidate is first converted into a location quality probability using a sigmoid function. Then, the mean of the logit is calculated in the logit space and mapped back to the probability space. This represents the overall quality center predicted by the current model for the training candidates, rather than a manually set IoU threshold. During inference, the predicted quality of the candidate boxes is compared to this reference value: higher than this value indicates a higher quality prediction. Candidates with relatively more reliable localization are considered to have improved detection confidence; those with lower localization are considered to have higher detection confidence. Candidates are suppressed when the two are equal; the original confidence level remains unchanged. This design allows quality prediction to participate in candidate ranking with a relative calibration signal, avoiding the problem that traditional multiplicative fusion can only reduce the detection score.

[0157] Therefore, while retaining the original target detection network category judgment results, the confidence of candidate targets is adaptively calibrated based on the predicted localization quality of the corrected detection boxes, thereby improving the candidate box selection results in the subsequent candidate target ranking and non-maximum suppression process.

[0158] (vii) Training methods: The construction of the visual memory bank and the training of the detection and correction module are carried out in stages.

[0159] First, a visual memory bank is constructed using complete target samples in the training dataset whose interference occlusion level is no higher than a first threshold (e.g., 1%). For each complete target sample, target region features are extracted from the intermediate feature map of the interference segmentation and target detection integrated network, and then assigned to the corresponding visual memory group according to the target category, target principal axis direction, and target region feature response intensity attributes.

[0160] After the visual memory bank is established, the integrated network for interference segmentation and target detection, as well as the visual memory bank, are fixed. Based on this, the continuous gating weight prediction module and the detection correction module are trained.

[0161] The detection and correction training samples consist of initial detection boxes for candidate targets and their corresponding ground truth boxes. The candidate targets are generated by an integrated network for interference segmentation and target detection before non-maximum suppression, and occluded candidate targets are selected as training samples based on the degree of interference occlusion within their regions.

[0162] Let the initial detection box center coordinates of the candidate target be... The width and height are respectively and The corresponding coordinates of the center of the actual target bounding box are The width and height are respectively and The supervision target for the detection box correction parameters is represented as:

[0163] Let the detection frame correction parameters predicted by the detection correction module be as follows: The corrected detection box parameters are expressed as follows:

[0164]

[0165]

[0166]

[0167] The training loss for the detection correction module includes a detection box location regression loss and a detection confidence correction loss. The detection box location regression loss uses a smoothed L1 loss constraint to constrain the difference between the predicted correction parameters and the supervised target; the detection confidence correction loss is optimized based on the matching quality between the corrected detection box and the ground truth target box.

[0168] Through the above training method, the detection correction module corrects the position and detection confidence of the occluded candidate target based on the current observation features of the occluded candidate target and the retrieved complete target memory features, while keeping the detection network and visual memory bank fixed.

[0169] During the inference phase, furthermore, the visual memory association and detection result correction are not performed on all candidate targets generated by the detection network, but only on the candidate targets that are identified as occluded and enter the top K reserved set of the corresponding local group: First, from the candidate targets generated by the integrated interference segmentation and target detection network before non-maximum suppression, occluded candidate targets are selected based on the degree of interference occlusion within the candidate target region.

[0170] Based on the spatial relationship of the occluded candidate targets, they are locally grouped, and the candidate targets within the same local group are ranked according to the initial detection confidence of the candidate targets. Only the top K ranked candidate targets in each local group are retained for the visual memory association and detection correction process.

[0171] For candidate targets that do not enter the retention set, including candidate targets that are not identified as occluded and occluded candidate targets that do not enter the top K of the corresponding local groups, visual memory retrieval and detection correction are not performed, and the initial detection box and initial detection confidence output by the interference segmentation and target detection integrated network are directly retained.

[0172] For occluded candidate targets that enter the retention set, the corresponding complete target memory features are retrieved from the visual memory bank according to the target attributes of the candidate targets. The target correction features are obtained through continuous gating and feature fusion. The detection correction module then predicts the corrected detection box and detection confidence.

[0173] The candidate targets that have undergone detection correction are merged with those that have not, forming a unified candidate target set. The corrected detection bounding boxes and confidence scores are used for the corrected candidate targets, while the initial detection bounding boxes and confidence scores are used for the remaining candidate targets. Finally, non-maximum suppression is applied to the unified candidate target set to obtain the final target localization result.

[0174] The occlusion target accurate localization method based on associative memory provided by this invention mainly solves the detection and identification of infrared ship targets, but it is also applicable to other infrared targets or other modal targets. For these data, it is only necessary to rebuild the visual memory library and train the network model to complete the corresponding target detection and identification tasks.

[0175] Taking infrared ship targets obscured by smoke, fog, or other interference as an example, the infrared image to be detected is input into an integrated network for interference segmentation and target detection. First, the initial detection results of candidate ship targets and corresponding interference occlusion information are obtained. The initial detection results include the initial detection bounding box of the candidate ship target, the target category, the initial detection confidence score, and the target region features extracted from the candidate target region. The degree of interference occlusion is calculated based on the interference occlusion identification results within the candidate ship target region and compared with a preset occlusion threshold. When the degree of interference occlusion reaches the preset threshold, the candidate ship target is identified as an occluded candidate target, and a detection correction process based on associative memory is initiated.

[0176] For candidate ship targets identified as occluded, candidate memories are selected from a pre-established visual memory bank based on target attributes such as target category, target principal axis direction, and target region feature response intensity. The target region features of the current candidate ship target are used as query features, and similarity is calculated between them and the key features in the candidate memories. Associative memory weights are obtained based on the similarity between each candidate memory and the query features. These weights are then used to weight and read the corresponding value features to obtain associative memory features that match the currently occluded ship target. These associative memory features contain target structure and position representation information learned from complete ship target samples, thus providing supplementary information for ship targets whose structural information is missing due to interference and occlusion.

[0177] Subsequently, the original target region features of the current candidate ship targets are fused with the obtained associative memory features, and the fused features are input into the detection correction module to predict the position correction amount corresponding to the initial detection box and the positioning quality of the correction box. The initial detection box of the candidate ship targets is adjusted according to the position correction amount, and the initial detection confidence is calibrated according to the predicted positioning quality, thereby obtaining the corrected ship target detection result.

[0178] Figure 6 An example of detection and correction for a jammed ship target is given. For Figure 6 The ship target shown was initially located only in a localized area due to extensive interference obstructing the right side of the ship, resulting in an initial detection confidence level of 0.41. After visual memory retrieval, associative memory feature reading, and detection correction, the detection box covered a more complete area of ​​the ship target, increasing the corrected detection confidence level to 0.72. This demonstrates that when the observed features of a ship target are incomplete due to interference obstruction, this invention can utilize visual memory information formed by complete target samples to supplement the missing representation of the current target, thereby correcting the target location and detection confidence level of the initial detection result.

[0179] To verify the effectiveness of the proposed occlusion target localization correction method based on associative memory, under the same training data, test data, and detection network conditions, the interference segmentation and target detection integrated network without associative memory correction was used as the baseline method, and the detection results were compared with those after applying the method of this invention.

[0180] Table 1: Method Performance Indicators

[0181] Data is shown in Table 1 and Figure 7 As shown, experimental results indicate that the AP50-95 of the baseline method is 35.884%, which is improved to 39.460% using the method of this invention, an improvement of 3.576 percentage points; under the more stringent evaluation condition of IoU=0.75, AP75 increases from 25.249% to 31.558%, an improvement of 6.309 percentage points. Further comparison of detection results at different IoU thresholds reveals that, as Figure 7 As shown, the present invention improves AP by approximately 6.2, 6.3 and 7.0 percentage points when the IoU threshold is 0.70, 0.75 and 0.80, respectively, while the improvement is relatively small at lower IoU thresholds.

[0182] This indicates that the performance gains brought by this invention are mainly concentrated under high IoU evaluation conditions, that is, after associative memory retrieval and detection correction, a higher degree of spatial overlap can be achieved between the predicted bounding box and the real target region. In this embodiment, for ship targets whose structural information is missing due to interference such as smoke and fog, this invention retrieves relevant target memories from the visual memory bank constructed from complete ship targets and uses associative memory features to supplement the structural and positional representation information of the currently occluded target, thereby further correcting the position of the original detection box and improving the positioning accuracy of the interfered ship targets.

[0183] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0184] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for accurate localization of occluded targets based on associative memory, characterized in that, Includes the following steps: The image to be detected is acquired and input into a trained interference segmentation and target detection integrated network to obtain the initial detection box, target category, initial detection confidence, target region features and corresponding interference occlusion information of the candidate target, which are used as the recognition result. Based on the interference and occlusion information, the occluded candidate target is determined from the candidate targets; For the occluded candidate target, memory lookup and comparison are performed based on a pre-constructed visual memory bank, which includes multiple memory groups. Each memory group stores the memory features of the unoccluded target sample and the corresponding target attribute information. Candidate memory groups are selected from the visual memory bank according to the target attribute information of the occluded candidate target. The associative memory features corresponding to the occluded candidate target are obtained by combining the target region features of the occluded candidate target with the memory features in the candidate memory groups. Based on the target region features of the occluded candidate target and the associative memory features, the detection results are corrected to obtain the accurate localization result of the target in the image to be detected.

2. The method for accurate localization of occluded targets based on associative memory according to claim 1, characterized in that, The construction of the visual memory bank includes: The training dataset is input into the integrated network for interference segmentation and target detection to obtain target region features and extract target attribute information including target category, target principal axis direction and target region feature response intensity. Memory units are established by screening the target region features and target attribute information of unoccluded target samples, using the target region features as memory features, and dividing the memory units into multiple memory groups according to the target attribute information to form the visual memory bank.

3. The method for accurate localization of occluded targets based on associative memory according to claim 1, characterized in that, The step of determining the occluded candidate target from the candidate targets based on the interference occlusion information includes: Candidate target regions are determined based on the initial detection bounding boxes of candidate targets, and interference occlusion information of the corresponding regions is extracted from the interference occlusion probability map output by the integrated interference segmentation and target detection network. The degree of interference occlusion of candidate targets is determined based on the interference occlusion information, and candidate targets whose interference occlusion degree reaches a preset threshold are identified as occluded candidate targets.

4. The method for accurate localization of occluded targets based on associative memory according to claim 1, characterized in that, Before performing memory retrieval and comparison based on the visual memory bank, the process also includes: Local groups are formed based on the initial detection box center position of the occluded candidate targets. Within each local group, the top K candidate targets are retained according to the initial detection confidence, where K is a preset positive integer. Perform the memory lookup and comparison on the retained candidate targets.

5. The method for accurate localization of occluded targets based on associative memory according to claim 4, characterized in that, The step of performing local grouping based on the center position of the initial detection box of the occluded candidate target includes: Candidate targets that are occluded are sorted in descending order of initial detection confidence. The candidate target with the highest initial detection confidence is selected from the ungrouped candidate targets as the benchmark candidate target. The center of its initial detection box is used as the benchmark center. The product of the larger of the width and height of the initial detection box and the preset scale coefficient is used as the grouping radius. Candidate targets that are not yet grouped, have the same category as the benchmark candidate target, and whose Euclidean distance from the center of the initial detection box to the benchmark center is not greater than the grouping radius are assigned to the same local group as the benchmark candidate target. Repeat the grouping process described above until all occluded candidate targets have been grouped.

6. The method for accurate localization of occluded targets based on associative memory according to claim 1, characterized in that, The step of selecting candidate memory groups from the visual memory bank based on the target attribute information of the occluded candidate targets, and combining the target region features of the occluded candidate targets with the memory features within the candidate memory groups to obtain the associative memory features corresponding to the occluded candidate targets; includes: Based on the identification results of the retained candidate targets, the target category, the target principal axis direction, and the target region feature response intensity are determined, and a memory group with the same category as the candidate target is determined from the visual memory bank; Calculate the mean value of the target principal axis direction and the mean value of the characteristic response intensity of the target region for all memory cells within the memory group; Based on the difference between the target principal axis direction of the candidate target and the mean of the target principal axis direction of the memory group, and the difference between the target region feature response intensity of the candidate target and the mean of the target region feature response intensity of the memory group, the attribute distance is calculated according to a preset weight, and the memory group with the smallest attribute distance is selected as the candidate memory group based on the attribute distance. The target region features of the candidate target are used as query features. The query features and the memory features in the candidate memory group are normalized and the similarity is obtained by dot product calculation. The similarity is normalized to obtain the associative memory weights corresponding to each memory feature. The associative memory weights are then used to perform a weighted summation of the corresponding unnormalized memory features to obtain the associative memory features corresponding to the candidate target.

7. The method for accurate localization of occluded targets based on associative memory according to claim 1, characterized in that, The correction of the detection results based on the target region features of the occluded candidate target and the associative memory features includes: Based on the identification results of candidate targets and the corresponding associative memory features, continuous gating weights for each spatial location are obtained. The associative memory features are adjusted using the continuous gating weights and then concatenated with the target region features of the candidate targets to obtain target correction features; The target correction features are input into the trained detection correction module, which outputs detection box correction parameters and detection confidence correction parameters. Based on these parameters, the initial detection box and initial detection confidence of the candidate target are corrected to obtain the accurate target localization result.

8. The method for accurate localization of occluded targets based on associative memory according to claim 7, characterized in that, The method for obtaining continuous gating weights for each spatial location based on the candidate target recognition results and corresponding associative memory features includes: Obtain the corresponding interference occlusion information and target region features from the candidate target identification results; Calculate the feature difference map based on the target region features of the candidate targets and the corresponding associative memory features; After aligning the interference occlusion information and feature difference map in spatial location, they are input into the trained continuous gating weight prediction module to obtain the continuous gating weights for each spatial location.

9. The method for accurate localization of occluded targets based on associative memory according to claim 8, characterized in that, The continuous gating weight prediction module includes: three convolutional layers and two nonlinear activation layers; The first non-linear activation layer is located after the first convolutional layer, and the second non-linear activation layer is located after the second convolutional layer. After adjusting the interference occlusion information and feature difference map to the same spatial size, they are stitched together along the channel dimension and used as the input of the first convolutional layer. The first and second convolutional layers are used for feature fusion, and the third convolutional layer outputs the gating response at each spatial location, which is then normalized to obtain continuous gating weights.

10. The method for accurate localization of occluded targets based on associative memory according to claim 7, characterized in that, The step of adjusting the associative memory features using the continuous gating weights and concatenating them with the target region features of the candidate target to obtain target correction features includes: Align the continuous gating weights with the associative memory features in spatial position, and obtain the adjusted associative memory features by multiplying them element by element; Based on preserving the target region features of the candidate targets, the target region features and the adjusted associative memory features are concatenated along the channel dimension to obtain the target correction features.

11. The method for accurate localization of occluded targets based on associative memory according to claim 7, characterized in that, The step of inputting the target correction features into a trained detection correction module and outputting detection box correction parameters and detection confidence correction parameters, thereby correcting the initial detection box and initial detection confidence of the candidate target; includes: The target correction features are input into the trained detection correction module to obtain the detection box correction parameters and the predicted localization quality; the detection box correction parameters include the center position offset parameter and the scale offset parameters of width and height; Based on the center position offset parameter and the scale offset parameter, the center position, width and height of the initial detection box of the candidate target are corrected to obtain the corrected detection box; Based on the predicted positioning quality, a detection confidence correction parameter is determined, and the initial detection confidence of the candidate target is corrected accordingly to obtain the corrected detection confidence. Non-maximum suppression is performed based on the corrected detection bounding box and detection confidence to obtain accurate target localization results.

12. The method for accurate localization of occluded targets based on associative memory according to claim 7, characterized in that, The detection correction module includes: a detection box correction parameter prediction branch and a detection confidence correction parameter prediction branch; each branch includes: two fully connected layers and one non-linear activation layer; the non-linear activation layer is located after the first fully connected layer; the first fully connected layer is used to perform latent space mapping and feature fusion on the target correction features; The second fully connected layer in the detection box correction parameter prediction branch is used to output candidate detection box correction parameters; the detection box correction parameters include center point horizontal correction parameters, center point vertical correction parameters, width scale correction parameters, and height scale correction parameters. The second fully connected layer in the detection confidence correction parameter prediction branch is used to output the predicted localization quality.

13. The method for accurate localization of occluded targets based on associative memory according to claim 12, characterized in that, The detection confidence correction parameters are determined based on the predicted positioning quality, including: Based on the difference between the predicted positioning quality and the preset reference positioning quality, the detection confidence correction parameter is determined; The detection confidence is corrected within the Logit space of the initial detection confidence based on the detection confidence correction parameters to obtain the corrected detection confidence.

14. The method for accurate localization of occluded targets based on associative memory according to claim 1, characterized in that, The integrated network for interference segmentation and target detection includes: a feature extraction network, a target detection branch, and an interference segmentation branch; The feature extraction network is used to extract feature maps from the input image; The target detection branch obtains the initial detection box, target category, and initial detection confidence of the candidate target based on the feature map; The interference segmentation branch obtains an interference occlusion probability map corresponding to the interference occlusion region based on the feature map.

15. The method for accurate localization of occluded targets based on associative memory according to claim 2, characterized in that, The target category, target principal axis direction, and target region feature response intensity are determined as follows: For training images, the target category is determined based on the category label of the unoccluded target samples; for images to be detected, the target category is determined based on the predicted target category. For training images, the horizontal bounding boxes are used as target boxes; for images to be detected, the detection boxes are used as target boxes. Principal component analysis is performed on the spatial distribution of significant pixels whose pixel intensity is greater than the mean pixel intensity in the target bounding box. The direction of the eigenvector corresponding to the largest eigenvalue of the two-dimensional coordinate covariance matrix is ​​taken as the principal axis direction of the target. The angle corresponding to the principal axis direction of the target is mapped to [0°, 180°). The feature response intensity of the target region is obtained by averaging the feature responses of the target region in the channel dimension and the spatial dimension.

16. The method for accurate localization of occluded targets based on associative memory according to claim 2, characterized in that, The step of dividing the memory unit into multiple memory groups based on the target attribute information includes: The attribute labels of each memory unit are determined based on the target category, the preset direction interval to which the target principal axis direction belongs, and the preset intensity level interval to which the target region feature response intensity belongs. Memory units with the same attribute labels are grouped into the same memory group.

17. The method for accurate localization of occluded targets based on associative memory according to claim 16, characterized in that, The preset direction range and preset intensity level range are determined as follows: The [0°, 180°) area is divided into multiple non-overlapping preset direction intervals, and the preset direction interval to which the target belongs is determined according to the angle corresponding to the main axis direction of the target. Based on the statistical distribution of target region feature response intensity of unoccluded target samples in the training dataset, the boundaries of multiple preset intensity level intervals are determined, and the preset intensity level interval to which the target region belongs is determined based on the target region feature response intensity.

18. The method for accurate localization of occluded targets based on associative memory according to claim 8, characterized in that, The construction and training of the visual memory bank and the detection and correction module adopt a phased approach: First, a visual memory bank is constructed using complete target samples in the training dataset whose interference occlusion degree is no higher than the first threshold, and the integrated network for interference segmentation and target detection and the visual memory bank are fixed. Then, using the candidate targets before non-maximum suppression as training samples, the occluded candidate targets are selected according to the degree of interference occlusion in the candidate target area, and the detection correction module and the continuous gating weight prediction module are trained. During the inference phase, visual memory association and detection result correction are performed only on the K candidate targets that are occluded and enter the first set of the corresponding local group. The remaining candidate targets retain the initial detection box and initial detection confidence. Finally, non-maximum suppression is performed after merging all candidate targets.

Images