A multi-temporal remote sensing image time-sensitive target change detection method, system, computer and storage medium

Abstract

The application discloses a kind of multi-temporal remote sensing image time-sensitive target change detection method, system, computer and storage medium, comprising: using strong scattering point guided SAR target detection network to carry out target detection, and output target frame information;Using difference enhancement and unsupervised classification fusion strategy to extract change area, and output change area mask;The target frame is fused with change area mask, through spatial alignment, time correlation and change type determination, realize the quantitative detection of time-sensitive target addition, leave, no change.This application solves the problem of method fragmentation and low detection accuracy, and is suitable for dynamic monitoring of port, airport and other civil scenarios.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, specifically to a method, system, computer, and storage medium for detecting time-sensitive target changes in multi-temporal remote sensing images. Background Technology

[0002] With the dense networking of high-resolution SAR satellites (such as Gaofen-3, Sentinel-1, and the ICEYE proposal), multi-temporal remote sensing imagery, with its advantage of capturing the spatial distribution differences of targets in the time dimension, has become an important data support for the dynamic monitoring of time-sensitive targets. SAR technology has the characteristics of all-weather, all-time, and strong penetration, which can overcome the bottlenecks of optical remote sensing limited by illumination and meteorological conditions, and plays an irreplaceable role in time-sensitive target monitoring scenarios such as ship berthing in ports and aircraft parking at airports.

[0003] However, existing multi-temporal SAR image time-sensitive target change detection technologies have two major problems: First, the methods are fragmented. Traditional technologies treat target detection and identification and SAR change area extraction as independent processes. The former can only achieve single-temporal target localization and classification, and cannot associate the change logic of multi-temporal targets (such as whether a ship is newly added). The latter can only extract regional change information and cannot clarify the specific type and status of the changed target (such as whether the changed area contains a newly added aircraft or other targets). This results in a lack of target-level and regional-level linkage interpretation in the change detection results. Second, the inherent interference of SAR images affects the detection accuracy. SAR images are imaged by backscattering intensity. Time-sensitive targets are easily affected by resolution, azimuth angle, and incident angle, exhibiting variations in scattering characteristics. They can even be confused with port facilities and noise interference, resulting in a high false alarm rate for target detection.

[0004] Existing patent 1 [Miao Weixing, Liu Wei, Li Xiaoye, Target Detection Method, Device, Equipment and Storage Medium Based on High-Resolution SAR: CN202211115504.9] discloses a target detection method based on high-resolution SAR. This method calculates the average brightness and total row / column brightness of a slice image of a ship target on the sea surface using high-resolution SAR. It then selects the target row and column with the highest brightness to determine the boundary position of the scattering sidelobe suppression region. The method filters the target slice images to be processed according to preset conditions and obtains a binary target image through a target detection algorithm. Finally, it determines the minimum bounding rectangle of the ship target, thereby solving the trailing phenomenon caused by scattering sidelobes in high-resolution SAR images of ship targets on the sea surface and achieving the extraction of the minimum bounding rectangle of the target. This method uses brightness to filter the boundary of the sidelobe suppression region and achieve target confirmation. Existing patent 2 [Jia Meng, Wang Lei, Zhao Zhiqiang, Chen Hao, Qiu Yuan, et al., SAR Image Change Detection Method Based on Discontinuous Adaptive Nonlocal Mean: CN108389218B] discloses a SAR image change detection method based on discontinuous adaptive nonlocal mean. It first preprocesses two SAR images reflecting the same geographical area acquired at different times, then generates a difference image. This difference image is divided into overlapping image blocks, mapped to the PCA feature space, and then clustered to achieve automatic detection of environmental changes. This patent is a regional-level SAR change detection method and does not achieve coordination with target detection results. Existing patent 3 [Shanghai Mido Digital Technology Co., Ltd., Method, System, Device, and Medium for Simultaneous Target Detection and Semantic Segmentation: CN117934852B] performs target detection and semantic segmentation simultaneously on the same input image, capturing the relationship between different regions of the image. However, it only performs semantic segmentation, not change detection, and does not construct a target-region fusion change detection framework. Moreover, this patent is for non-SAR scenarios, non-multi-temporal scenarios, non-change detection, and does not introduce change region masks or cross-temporal target association. Summary of the Invention

[0005] The purpose of this invention is to propose a method, system, computer, and storage medium for detecting time-sensitive target changes in multi-temporal remote sensing images, in order to overcome the shortcomings of existing multi-temporal SAR image time-sensitive target change detection methods, such as fragmentation, weak anti-interference ability, and low detection accuracy.

[0006] The technical solution to achieve the objective of this invention is: a method for detecting time-sensitive target changes in multi-temporal remote sensing images, comprising the following steps:

[0007] Step S1: The SAR target detection module is used to perform target detection on the multi-temporal SAR images respectively, and outputs the target bounding box information of the time-sensitive targets in each temporal phase. The target bounding box information includes the location, category and confidence level. The SAR target detection module is implemented based on the SAR target detection network guided by strong scattering points. The network includes a region proposal network guided by strong scattering points and a region of interest contrast loss optimization branch.

[0008] Step S2: The SAR change region extraction module is used to perform unsupervised change detection on multi-temporal SAR images and output change region masks. The SAR change region extraction module adopts a difference enhancement-unsupervised classification fusion strategy, including: generating synthetic change samples through background fusion method to amplify samples, calculating multi-class difference maps, and performing hierarchical clustering segmentation on multi-class difference maps to obtain high-confidence change region masks.

[0009] Step S3: The target bounding box output in step S1 and the change area mask output in step S2 are fused using the fusion change determination module to determine the change type of the time-sensitive target; the fusion includes: spatial alignment, temporal correlation and change type determination; the change type includes new target, departing target, no change target, suspected change target and state change target.

[0010] Furthermore, in step S1, the specific implementation of the region proposal network guided by the strong scattering point includes:

[0011] After converting the ground view from a directional bounding box to a horizontal bounding box and cropping it, feature points reflecting the scattering distribution in each time-sensitive target ground view frame are extracted using the Harris angle detector, and feature points with response values ​​greater than 0.01 times the maximum response value are selected.

[0012] If the number of feature points exceeds 20, they are clustered into 9 classes using K-means, and the cluster center points are taken as strong scattering points; if the number of feature points is less than 20, the 9 points with the highest response values ​​are selected as strong scattering points.

[0013] Calculate the offsets of the nine strong scattering points relative to the true center point coordinates, and construct a strong scattering point location map;

[0014] Add branches consisting of 1×1 convolutions and 3×3 convolutions to the region proposal network to predict the location of strong scattering points in each anchor frame, and restore the predicted locations to the original size to obtain the final strong scattering point location map;

[0015] Earth Mover distance is used as the loss function to calculate the similarity between the predicted point set and the ground reality point set.

[0016] Furthermore, in step S1, the implementation of the region of interest contrast loss optimization branch includes:

[0017] The 1024-dimensional feature vector output by the region of interest head feature extractor is encoded into 128-dimensional contrastive features through two layers of multilayer perceptrons.

[0018] A supervised contrast loss function is adopted, and the supervised contrast loss is only proposed to be calculated for regions with an overlap rate of 0.5 or greater than or equal to the ground truth, so as to bring the feature distance of similar targets closer and widen the feature distance of dissimilar targets further apart;

[0019] The total network loss is composed of a weighted average of the region proposal network loss, classification loss, regression loss, and region of interest contrast loss, with the weight parameter of the contrast loss set to 0.2.

[0020] Furthermore, in step S2, the specific implementation of the background fusion method for generating synthetic variation samples includes:

[0021] Cropping background blocks that do not contain time-sensitive targets from SAR images;

[0022] Randomly embed time-sensitive target blocks, and adjust the grayscale value of the target blocks to match the scattering intensity range of the background blocks;

[0023] Generate two types of change labels: background to target simulation addition and target to background simulation departure, increasing the number of change samples by 3 to 5 times.

[0024] Furthermore, in step S2, the multi-class difference map includes:

[0025] Logarithmic ratio difference plot: generated by calculating the absolute difference of the logarithms of gray values ​​between two temporal SAR images;

[0026] Mean ratio difference map: The local mean ratio of two temporal images is calculated using a sliding window with a size of 512*512 pixels;

[0027] Variance difference plot: obtained by calculating the absolute difference in local variance between two temporal images.

[0028] Furthermore, in step S2, the specific implementation of the hierarchical clustering segmentation includes:

[0029] K-means clustering is used to perform hierarchical segmentation of the multi-class difference map. The first-level clustering K=3 yields the unchanged class, the suspected changed class, and the high-confidence changed class. The second-level clustering further subdivides the suspected changed class into the weakly changed class and the noise interference class by K=2. This generates an initial change region mask, which is then processed by morphological filtering, connected component screening, and target box constraint correction to output the high-confidence change region mask.

[0030] Furthermore, in step S3, the specific implementation methods for spatial alignment, temporal correlation, and change type determination include:

[0031] Spatial alignment: The oriented target bounding boxes and change region masks output by the target detection module are transformed to the same geographic coordinate system. Geometric distortion is eliminated through affine transformation, and the overlap ratio (IOU) between the oriented target bounding boxes and change region masks is calculated. For each target bounding box, the intersection-union ratio (IoU) with each change connected component is calculated, and the connected component with the largest IoU is taken as the associated change region. If there is no overlap, the associated change region is empty, and the IoU is recorded as 0.

[0032] Temporal correlation: For the target detection results of two time phases t1 and t2, target matching pairs are established based on the conditions that the positional distance is less than 10 pixels, the category is the same, and the confidence level is greater than 0.8;

[0033] Change type determination:

[0034] (1) Adding and leaving judgment

[0035] If a target is detected at time t2 and there is no matching target at time t1, and the IoU between the target bounding box and its associated changed region is greater than or equal to 0.6, it is determined to be a newly added target; if a target is detected at time t2 and there is no matching target at time t1, but the IoU between the target bounding box and its associated changed region is less than 0.6, it is determined to be a suspected changed target.

[0036] If a target is detected at time t1 and there is no matching target at time t2, and the IoU between the target bounding box and its associated changed region is greater than or equal to 0.6, then it is determined to be a target that has left the target; if a target is detected at time t1 and there is no matching target at time t2, but the IoU between the target bounding box and its associated changed region is less than 0.6, then it is determined to be a suspected changed target.

[0037] (2) Calculation of the change overlap rate and determination of IoUmax under the matching target

[0038] If there is a matching target in time phase t1 and time phase t2, calculate the first overlap rate IOU1 between the target box and its associated change region in time phase t1, and the second overlap rate IOU2 between the target box and its associated change region in time phase t2, and take the larger value between IOU1 and IOU2 as the target change overlap rate IOUmax.

[0039] If the target bounding box at time t1 is basically in the same position as the target bounding box at time t2, the Euclidean distance between the center points of the matching target bounding boxes at time t1 and time t2 is ≤10 pixels, and the target change overlap rate IOUmax is less than 0.3, then it is determined to be a target with no change.

[0040] If there is a significant positional offset between the target bounding box at time t1 and the target bounding box at time t2, the Euclidean distance between the center points of the matching targets at time t1 and time t2 is greater than 10 pixels, and the target change overlap rate IOUmax is less than 0.3, then it is judged as a suspected changed target and output to the manual review module or the subsequent multi-time phase confirmation module.

[0041] If the target change overlap rate IOUmax is greater than or equal to 0.3 and less than 0.6, or if the area of ​​pixels falling within the target bounding box in the changed connected region associated with the target accounts for more than or equal to 30% and less than 70% of the total area of ​​the changed connected region, then it is judged as a suspected changed target.

[0042] If the target change overlap rate IOUmax is greater than or equal to 0.6, and the change area is mainly located inside the target box in time phase t1 or time phase t2, that is, in the change connected region associated with the target, the pixel area falling within the target box range accounts for ≥70% of the total area of ​​the change connected region, then it is determined to be a target with a change in target state. The change in target state includes position change, shape change, attitude change or scattering characteristic change.

[0043] If the target change overlap rate IOUmax is greater than or equal to 0.6, and the change area is mainly distributed outside the target box, that is, in the change connected region associated with the target, the pixel area falling within the target box range accounts for less than 30% of the total area of ​​the change connected region, then it is judged as a suspected change target;

[0044] All targets identified as suspected changes are output to the manual review module or the subsequent multi-temporal confirmation module.

[0045] A multi-temporal remote sensing image time-sensitive target change detection system, used to implement the aforementioned multi-temporal remote sensing image time-sensitive target change detection method, comprising:

[0046] Data input module: used to load multi-temporal SAR satellite image data and perform radiometric correction and geometric registration preprocessing;

[0047] SAR target detection module: Employs a SAR target detection network guided by strong scattering points, outputting the location, category, and confidence information of time-sensitive targets;

[0048] SAR change region extraction module: Employs a difference enhancement-unsupervised classification fusion strategy to output a high-confidence change region mask;

[0049] The fusion change determination module uses spatial alignment, temporal correlation, and change type determination methods to output the change type of time-sensitive targets.

[0050] Results output module: Displays change detection results in a visual format, including annotations of changed areas and a list of target change types.

[0051] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the multi-temporal remote sensing image time-sensitive target change detection method.

[0052] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-temporal remote sensing image time-sensitive target change detection method.

[0053] Compared with the prior art, the present invention has the following significant advantages:

[0054] 1) Break the disconnect between target detection and change region extraction, construct a dual-engine collaborative perception framework to achieve target-region level linkage change detection, and solve the shortcomings of traditional technology that know the region change but not the target type;

[0055] 2) Introduce a region proposal network guided by strong scattering points, and optimize anchor frame generation by utilizing the scattering characteristics of SAR targets to improve the robustness of target detection under complex imaging conditions and reduce the false alarm rate caused by scattering characteristic variations;

[0056] 3) A background fusion method sample amplification strategy is proposed, which increases the number of change samples by 3-5 times in small sample scenarios, solving the problem of poor generalization ability caused by insufficient samples in traditional unsupervised change detection.

[0057] 4) By integrating multi-dimensional change information from three types of difference maps and combining hierarchical clustering segmentation to generate high-confidence change region masks, noise interference from SAR images is effectively filtered out, and the accuracy of change region extraction is improved.

[0058] 5) Through a three-step fusion logic of spatial alignment, temporal correlation and threshold determination, quantitative determination of the addition / departure / no change of time-sensitive targets is achieved, providing directly applicable change detection results for civilian scenarios (such as port scheduling and airport management). Attached Figure Description

[0059] Figure 1 This is a schematic diagram of the overall process of the present invention.

[0060] Figure 2 This is a schematic diagram of the target detection and recognition network of the present invention.

[0061] Figure 3 This is a schematic diagram of SAR scattering point extraction according to the present invention.

[0062] Figure 4 This is the SAR image feature offset of the present invention.

[0063] Figure 5 This is an example diagram of sample amplification according to the present invention.

[0064] Figure 6 This is an example diagram of the variation region detection method based on difference enhancement according to the present invention.

[0065] Figure 7This is a schematic diagram of the SAR target detection and recognition results of the present invention.

[0066] Figure 8 This is a schematic diagram of the change detection results of the present invention.

[0067] Figure 9 This is a schematic diagram of the detection results of time-sensitive target changes at a port according to the present invention.

[0068] Figure 10 This is a schematic diagram of the time-sensitive target change detection results at an airport according to the present invention. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0070] This invention first provides a method for detecting time-sensitive target changes in multi-temporal remote sensing images. The framework for detecting time-sensitive target changes in multi-temporal remote sensing images constructed by this method uses precise target-level localization, region-level change constraints, and quantitative determination of change types as its core logic. It integrates two major technical paths: target detection and SAR change detection, and is divided into three core modules: a SAR target detection module, a SAR change region extraction module, and a fusion change determination module. The overall process is as follows: Figure 1 As shown.

[0071] Step 1, SAR target detection (corresponding to) Figure 1 (S101 operation)

[0072] The image is processed using a SAR target detection and recognition network. The network is optimized based on the physical properties of strong scattering points and a supervised contrastive learning mechanism to achieve high-precision target detection.

[0073] This invention proposes a novel SAR network, such as Figure 2 As shown, strong scattering points are introduced into the Region Proposal Network (RPN) to improve the performance of anchor frames, thereby enabling the detection of directed targets in SAR images. Furthermore, supervised contrastive learning loss is applied to the network to mitigate ship misclassification in SAR images, making ships of the same class as close as possible and ships of different classes as far apart as possible.

[0074] (1) Strong scattering point guided region proposal network

[0075] SAR images are the result of a SAR system transforming ground points at a unit resolution scale and image points at a pixel scale. Image intensity reflects the structural scattering characteristics of the target. Therefore, scattering feature points can be used to guide the Resource Planning Network (RPN), enabling the network to better adapt to changes in SAR imaging conditions. Extracted SAR image scattering points are shown below. Figure 3 As shown.

[0076] The specific implementation is as follows: After converting the ground view from a directional bounding box to a horizontal bounding box and cropping the ground view from the image, feature points reflecting the scattering distribution of each time-sensitive target ground view frame are extracted using a Harris angle detector. Here, points with response values ​​greater than 0.01 times the maximum response value are selected as feature points. When the number of feature points in a ground truth dataset exceeds 20, the feature points are clustered into 9 classes using K-means, and the center point of each cluster is defined as a strong scattering point, whose location contains structural features of different targets and is unaffected by changes in SAR imaging conditions. If the number of feature points in a ground truth dataset is less than 20, the 9 points with the highest response values ​​are selected as strong scattering points.

[0077] like Figure 4 As shown, the offsets of the nine strong scattering points in each true value relative to the coordinates of the true value center point. Represented as Strong scattering point location map ( (The output step size is determined by...) The offset is assigned to the corresponding ground view frame for each anchor frame.

[0078] The offset required in this invention is relative to the center point of the anchor box to guide the training of the network. Therefore, for each offset... In China The offsets need to be updated so that they can be applied to the feature map using the following equation. ( The position of the center point of the anchor frame on the output channel. :

[0079] (1)

[0080] Strong scattering point location map By updated offset Composition. This invention adds a branch consisting of 1x1 and 3x3 convolutions to the RPN to predict the location of strong scattering points in each anchor frame. (In the map) Each position Offset Predict the locations of nine possible strong scattering points from the current location and predict the offset map. Since these positions were calculated at different scales, it is necessary to restore the position coordinates to their original size.

[0081] (2)

[0082] Where (W,H) and (w_s,h_s) represent the original size and the scaled size, respectively. and These represent the scaled coordinates and original coordinates of the strong scattering points in the offset map, respectively. The final predicted map shows the locations of the strong scattering points. It is obtained from the above formula.

[0083] To better measure the similarity between two point sets, this invention uses the EarthMover distance (EMD) as the loss to calculate the similarity between the predicted point set and the ground reality point set. To calculate the matching error between the predicted strong scattering points and the ground reality strong scattering points, the predicted strong scattering point location map is reconstructed into the predicted point set. 9 represents the number of strong scattering points corresponding to each anchor frame, and 2 represents the two-dimensional coordinates of each strong scattering point. Similarly, the ground-based strong scattering point location map is reconstructed into a ground-based point set. Subsequently, the EarthMover distance was used to calculate the predicted point set N and the ground reality point set. The minimum matching cost between them is used as the strong scattering point location constraint loss. :

[0084] (3)

[0085] This represents a one-to-one matching relationship between the predicted point set and the true point set.

[0086] (2) RoI contrast loss improves inter-class discrimination

[0087] Due to the similarity of different target categories in SAR images, detectors often fail to classify them. Different categories also have small margins in their Rol features, making it difficult to have a robust classifier that can distinguish between them. To improve classification accuracy, this invention proposes applying the Rol contrastive loss to the network to learn more discriminative object proposal embeddings.

[0088] During training, the Rol head feature extractor proposes regions as vector embeddings. Then, the contrastive branch, with two multilayer perceptron (MLP) heads, encodes the Rol features into contrastive features. By default The similarity between proposed representations of Rol features encoded in the contrast head is measured, and the contrast objective is optimized to maximize the consistency between feature vectors from the same class and promote the divergence of feature vectors from different classes.

[0089] For batch data It is the i-th region proposal The RoI features encoded by the comparison header, It is the IOU value of the matched truth value. The label is the truth value. Loss function. As shown in the following formula:

[0090] (4)

[0091] in It is the number of embeddings with the same tag. , It is a hyperparameter temperature. This represents the RoI contrastive feature obtained after the i-th region proposal is processed by the Region of Interest Head Feature Extractor and the Contrast Feature Encoding Branch.

[0092] During the training phase, the contrastive loss is added to the overall network loss, resulting in the following loss function:

[0093] (5)

[0094] L is the final loss function. The region proposal network loss includes foreground / background classification loss and anchor box regression loss. It is classification loss. For the target bounding box regression loss, smooth L1 loss is preferred. For scattering point loss, It is a balance parameter, which is set empirically in this invention. .

[0095] Step 2, SAR change region extraction (corresponding) Figure 1 (S102 operation)

[0096] A time-sensitive target change detection method based on multi-temporal remote sensing images is adopted. A high-precision change difference map is generated based on the log ratio difference map, mean ratio difference map, and variance difference map, and a high-confidence change area mask is output.

[0097] (1) Large-area change region extraction based on difference enhancement

[0098] In large-scale detection of changes in targets of interest using remote sensing images from different time phases over a large area, the area where the change occurred accounts for a small proportion of the complete observed image, there are few change labels, and the change itself is a low-probability event.

[0099] like Figure 5As shown, in order to amplify samples, this invention uses a background fusion method to generate synthetic change samples through artificial synthesis: background blocks without targets (such as port areas without ships or airport runway areas without aircraft) are cropped from SAR images, and target blocks such as ships and aircraft are randomly embedded. By adjusting the grayscale values ​​of the target blocks to match the scattering intensity range of the background blocks, the consistency of scattering characteristics is maintained, generating two types of change labels: background to target (simulated addition) and target to background (simulated departure). This method increases the number of change samples by 3-5 times, alleviating the problem of small sample size.

[0100] (2) Variation region detection method based on difference enhancement

[0101] like Figure 6 As shown, this invention proposes a change detection technique enhanced by SAR intensity image difference maps. This technique utilizes multiple difference maps at different scales, combines the hierarchical clustering segmentation results of the difference maps, generates pseudo-labels using the hierarchical clustering results, and optionally trains a semi-supervised classifier to perform secondary discrimination on suspected change areas. Using this segmentation data for semi-supervised classifier training overcomes the problem of insufficient training and labeled data, thereby obtaining high-confidence change detection results.

[0102] SAR image disturbances and changes in complex scenarios are multifaceted. By performing change detection through fine classification of the results of preceding and following time phases, the mask of the changed area and the type of change can be obtained, thus providing support for situational analysis.

[0103] Calculate the three types of difference maps for two temporal SAR images:

[0104] Logarithmic ratio difference map: generated by calculating the absolute difference of the logarithms of gray values ​​between two temporal SAR images. Since speckle noise exhibits additive noise characteristics with an approximately Gaussian distribution in the logarithmic domain, this processing method can effectively suppress multiplicative speckle noise interference in the original image, making the details in weakly varying areas clearer and more discernible. It is particularly suitable for scenes with small differences in backscattering intensity.

[0105] Mean Ratio Difference Map: A sliding window is used to calculate the local mean ratio of two temporal images, with a window size of 512*512 pixels. Local mean smoothing reduces the impact of single-point noise while preserving regional variation trends. This difference map significantly highlights changes in targets with continuous spatial distribution characteristics, such as ships and aircraft, enhancing the contrast between changing areas and the background, making regional changes more apparent.

[0106] Variance difference map: This is obtained by calculating the absolute difference between the local variances of two temporal images. Variance reflects the dispersion of local gray values ​​in an image. This difference map can keenly capture fluctuations in the scattering characteristics of a target, such as changes in the distribution of scattering points caused by the movement of large vehicles or fluctuations in background scattering caused by ocean wave interference. It effectively complements the detection capabilities of the first two difference maps for changes in scattering stability.

[0107] Hierarchical clustering segmentation is performed on the three types of difference maps: 1) Normalize the three types of difference maps; 2) Construct three-dimensional feature vectors by pixel; 3) First-level clustering: Perform K-means clustering on all pixel features F(p), K=3, to obtain the unchanged class, suspected changed class, and high-confidence changed class; 4) Second-level clustering: Perform K-means clustering again on the suspected changed class, K=2, to further subdivide it into the weakly changed class and the noise interference class; 5) Generate initial change region mask: The high-confidence changed class is directly assigned a value of 1; the weakly changed class is retained based on neighborhood connectivity, target detection box coverage relationship or area threshold; the noise interference class and the unchanged class are assigned a value of 0.

[0108] Step 3, fusion change determination (corresponding to) Figure 1 (S103 operation)

[0109] By using spatial alignment, temporal correlation, and change type determination, the synergy between target detection results and target change detection results is achieved. This step integrates the results of the two sub-modules through a feature alignment mechanism to quantitatively determine the change type (new / departed / no change) of time-sensitive targets. The steps are as follows:

[0110] (1) Spatial alignment

[0111] The directional target bounding box (6-dimensional directional bounding box, (x,y,w,h,Δα,Δβ), where (x,y) are the center coordinates, w and h are the width and height of the outer rectangle, and Δα and Δβ are the offsets) output by the SAR target detection module is transformed to the same coordinate system (based on image geographic coordinates) with the SAR change area mask. Geometric distortion is eliminated through affine transformation, the overlap ratio (IOU) between the target bounding box and the change area is calculated, and the spatial association between the target and the change area is established.

[0112] (2) Time correlation

[0113] For the target detection results of multiple time phases t1 and t2, target matching pairs are established based on the conditions that the Euclidean distance between the center points of the target boxes is less than 10 pixels, the categories are the same, and the confidence level is greater than 0.8, so as to identify the same target across time phases;

[0114] (3) Determination of change type

[0115] After completing spatial alignment and cross-temporal target matching, the change type is determined for each target based on the spatial overlap between the target bounding box and the changed region. Finally, five categories of results are output: newly added target, target left, unchanged target, target with changed state, and suspected change. The specific determination logic is as follows:

[0116] (1) Determination of the addition / departure of isolated targets

[0117] For isolated targets that have not formed matching pairs across time phases, two candidate scenarios are identified based on their time phase affiliation: adding and leaving.

[0118] For targets detected only at time t2 and without matching results at time t1, the associated change region obtained from the spatial alignment step is used as the criterion: when the intersection-union ratio (IOU) between the target bounding box and its associated change region is ≥0.6, it is determined to be a new target; when the IOU is <0.6, it is determined to be a suspected changed target and sent to the manual review or subsequent multi-time phase secondary confirmation stage.

[0119] For targets detected only at time t1 and with no matching results at time t2, the same principle applies: when the IOU between the target bounding box and its associated changed region is ≥0.6, it is determined to be a target that has left the target; when the IOU is <0.6, it is determined to be a suspected changed target.

[0120] (2) Target state subdivision determination

[0121] For target pairs that successfully match at time phases t1 and t2, first extract the overlap ratio (IOU1) of the associated change region corresponding to the target box at time phase t1 and the overlap ratio (IOU2) of the associated change region corresponding to the target box at time phase t2, and take the larger value as the target change overlap ratio (IOUmax) of the target pair. Then, combine the target position offset and the distribution position of the change region for further subdivision judgment.

[0122] Unchanging targets: When the Euclidean distance between the center points of two time-phase target frames is ≤10 pixels (adapting to the conventional positioning error range of 1m resolution SAR images), and the target change overlap rate IOUmax <0.3, it is determined to be an unchanging target, indicating that the target's dwell state and scattering characteristics are stable.

[0123] Positional shift type suspected change: When the Euclidean distance between the center points of the target boxes in two time phases is greater than 10 pixels, there is a significant positional shift, but IOUmax is less than 0.3. It is not directly classified as no change, but is uniformly judged as a suspected change target, and is handed over to manual review or further confirmation by multi-time phase data.

[0124] Weak change type suspected change: When the target change overlap rate IOUmax is in the range of 0.3 (inclusive) to 0.6, or the proportion of pixel area of ​​the associated change connected component falling within the target bounding box is in the range of 30% (inclusive) to 70%, the change boundary and intensity are uncertain, and it is judged as a suspected change target.

[0125] Target state change: When IOUmax≥0.6 and the area of ​​pixels whose associated connected components fall inside the target bounding box is ≥70%, that is, when the change is mainly concentrated within the target body, it is determined to be a target state change. Specifically, it can correspond to situations such as fine adjustment of target position, change of shape, attitude deflection or fluctuation of scattering characteristics.

[0126] External interference type suspected change: When IOUmax≥0.6, but the area of ​​pixels in the connected component of the associated change falling into the target box is <30%, that is, the change is mainly distributed in the area around the target, it is judged as a suspected change target, and the misjudgment caused by changes in the surrounding environment is excluded.

[0127] All targets identified as potentially altered are uniformly output to the manual review module, or subjected to secondary verification using observation data from more time periods, to ensure the reliability of the final detection results.

[0128] This mechanism provides target-level location and category information through target detection and provides region-level change constraints through SAR change detection, thus solving the problem of misjudgment by a single method.

[0129] Corresponding to the above method, the present invention also provides a multi-temporal remote sensing image time-sensitive target change detection system, comprising:

[0130] The data input module is used to load multi-temporal SAR satellite image data and perform radiometric correction and geometric registration preprocessing.

[0131] The SAR target detection module uses the aforementioned strong scattering point-guided SAR target detection network, specifically implemented as described in step (I) above, outputting the location, category, and confidence information of time-sensitive targets;

[0132] The SAR change region extraction module adopts the aforementioned difference enhancement-unsupervised classification fusion strategy, specifically implemented as described in step (II) above, and outputs a high-confidence change region mask;

[0133] The fusion change determination module adopts the aforementioned spatial alignment, temporal correlation and change type determination methods, and is specifically implemented as described in step (III) above, outputting the change type of the time-sensitive target;

[0134] The results output module displays the change detection results in a visual format, including the annotation of changed areas and a list of target change types.

[0135] The connection relationships between the above modules are as follows: the output of the data input module is connected to the input of the SAR target detection module and the SAR change region extraction module respectively; the output of the SAR target detection module and the SAR change region extraction module are connected to the input of the fusion change determination module; the output of the fusion change determination module is connected to the input of the result output module.

[0136] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned method for detecting time-sensitive target changes in multi-temporal remote sensing images.

[0137] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned method for detecting time-sensitive target changes in multi-temporal remote sensing images.

[0138] Example

[0139] To verify the effectiveness of the present invention, the following experiment was conducted.

[0140] In this implementation, the experimental data and hardware / software environment are as follows:

[0141] Experimental data: Sourced from the Gaofen-3 (GF-3) and Taijing-4 (TJ-04) SAR satellites, covering 3 typical regions and 5 types of time-sensitive targets. The data is in (SL) mode, with a resolution of 1m (range) × 1m (azimuth), polarization HH / VV, and an image size of 1024 × 1024 pixels.

[0142] Hardware environment: GPU NVIDIA A100 (40GB VRAM), CPU Intel Xeon 8375C (32 cores), RAM 128GB;

[0143] Software environment: PyTorch 1.12, CUDA 11.2, OpenCV 4.5.5, Scikit-learn 1.0.2;

[0144] Object detection results: The constructed dataset was tested, and the test results are shown in Table 1:

[0145] Table 1 Target Detection Performance

[0146]

[0147] like Figure 7As described in the schematic diagram of SAR target detection results, the near-shore background interference of ship targets is severe (such as port cranes and breakwater scattering), but thanks to the RoI contrast loss which expands the inter-class feature differences between ships and port facilities, their detection performance is also greatly improved, and other targets are also improved to varying degrees.

[0148] Regional change detection results: Regional change detection tests were conducted on multi-temporal SAR images of key areas. Specific results are as follows: Figure 8 As shown in the figure above, the first column is time series 1, the second column is time series 2, the third column is the inference result, and the fourth column is the true value. The inference results show that the change detection method used in this invention can correctly detect changes in targets such as ships, and can eliminate, to a certain extent, the false images caused by changes due to factors such as SAR imaging angle and interference noise.

[0149] Change detection fusion test results: Experiments were conducted in key areas, and the results are as follows: Figure 9 and Figure 10 As shown. The accuracy rate for both adding / leaving is over 98%, among which... Figure 10 Airports performed best because airport targets (aircraft) are large and have stable scattering, resulting in high consistency between target detection and change area extraction results. Figure 9 Due to the interference of ocean waves, the boundaries of some changed areas in the port are blurred, resulting in a slightly lower accuracy rate in the determination, but the index is still greater than 90%.

[0150] The test results of this invention on the source domain GF1_WHU dataset are shown in Table 2.

[0151] Table 2. Test results on the source domain GF1_WHU dataset.

[0152]

[0153] In the above-mentioned experimental area, the overall accuracy rate of change detection was 95.49%, and the accuracy rate of change type determination was 98.41%.

[0154] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0155] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for detecting time-sensitive target changes in multi-temporal remote sensing images, characterized in that, Includes the following steps: Step S1: The SAR target detection module is used to perform target detection on the multi-temporal SAR images respectively, and outputs the target bounding box information of the time-sensitive targets in each temporal phase. The target bounding box information includes the location, category and confidence level. The SAR target detection module is implemented based on the SAR target detection network guided by strong scattering points. The network includes a region proposal network guided by strong scattering points and a region of interest contrast loss optimization branch. Step S2: The SAR change region extraction module is used to perform unsupervised change detection on multi-temporal SAR images and output change region masks. The SAR change region extraction module adopts a difference enhancement-unsupervised classification fusion strategy, including: generating synthetic change samples through background fusion method to amplify samples, calculating multi-class difference maps, and performing hierarchical clustering segmentation on multi-class difference maps to obtain high-confidence change region masks. Step S3: The target bounding box output in step S1 and the change area mask output in step S2 are fused using the fusion change determination module to determine the change type of the time-sensitive target; the fusion includes: spatial alignment, temporal correlation and change type determination; the change type includes new target, departing target, no change target, suspected change target and state change target.

2. The method for detecting time-sensitive target changes in multi-temporal remote sensing images according to claim 1, characterized in that, In step S1, the specific implementation of the region proposal network guided by strong scattering points includes: After converting the ground view from a directional bounding box to a horizontal bounding box and cropping it, feature points reflecting the scattering distribution in each time-sensitive target ground view frame are extracted using the Harris angle detector, and feature points with response values ​​greater than 0.01 times the maximum response value are selected. If the number of feature points exceeds 20, they are clustered into 9 classes using K-means, and the cluster center points are taken as strong scattering points; if the number of feature points is less than 20, the 9 points with the highest response values ​​are selected as strong scattering points. Calculate the offsets of the nine strong scattering points relative to the true center point coordinates, and construct a strong scattering point location map; Add branches consisting of 1×1 convolutions and 3×3 convolutions to the region proposal network to predict the location of strong scattering points in each anchor frame, and restore the predicted locations to the original size to obtain the final strong scattering point location map; Earth Mover distance is used as the loss function to calculate the similarity between the predicted point set and the ground reality point set.

3. The method for detecting time-sensitive target changes in multi-temporal remote sensing images according to claim 1, characterized in that, In step S1, the implementation of the region of interest contrast loss optimization branch includes: The 1024-dimensional feature vector output by the region of interest head feature extractor is encoded into 128-dimensional contrastive features through two layers of multilayer perceptrons. A supervised contrast loss function is adopted, and the supervised contrast loss is only proposed to be calculated for regions with an overlap rate of greater than or equal to 0.5 with the ground truth, so as to bring the feature distance of similar targets closer and widen the feature distance of dissimilar targets further apart; The total network loss is composed of a weighted average of the region proposal network loss, classification loss, regression loss, and region of interest contrast loss, with the weight parameter of the contrast loss set to 0.

2.

4. The method for detecting time-sensitive target changes in multi-temporal remote sensing images according to claim 1, characterized in that, In step S2, the specific implementation of the background fusion method for generating synthetic variation samples includes: Cropping background blocks that do not contain time-sensitive targets from SAR images; Randomly embed time-sensitive target blocks, and adjust the grayscale value of the target blocks to match the scattering intensity range of the background blocks; Generate two types of change labels: background to target simulation addition and target to background simulation departure, increasing the number of change samples by 3 to 5 times.

5. The method for detecting time-sensitive target changes in multi-temporal remote sensing images according to claim 1, characterized in that, In step S2, the multi-class difference map includes: Logarithmic ratio difference plot: generated by calculating the absolute difference of the logarithms of gray values ​​between two temporal SAR images; Mean ratio difference map: The local mean ratio of two temporal images is calculated using a sliding window with a size of 512*512 pixels; Variance difference plot: obtained by calculating the absolute difference in local variance between two temporal images.

6. The method for detecting time-sensitive target changes in multi-temporal remote sensing images according to claim 1, characterized in that, In step S2, the specific implementation of the hierarchical clustering segmentation includes: K-means clustering is used to perform hierarchical segmentation of the multi-class difference map. The first-level clustering K=3 yields the unchanged class, the suspected changed class, and the high-confidence changed class. The second-level clustering further subdivides the suspected changed class into the weakly changed class and the noise interference class by K=2. This generates an initial change region mask, which is then processed by morphological filtering, connected component screening, and target box constraint correction to output the high-confidence change region mask.

7. The method for detecting time-sensitive target changes in multi-temporal remote sensing images according to claim 1, characterized in that, In step S3, the specific implementation methods for spatial alignment, temporal correlation, and change type determination include: Spatial alignment: The oriented target bounding boxes and change region masks output by the target detection module are transformed to the same geographic coordinate system. Geometric distortion is eliminated through affine transformation, and the overlap ratio (IOU) between the oriented target bounding boxes and change region masks is calculated. For each target bounding box, the intersection-union ratio (IoU) with each change connected component is calculated, and the connected component with the largest IoU is taken as the associated change region. If there is no overlap, the associated change region is empty, and the IoU is recorded as 0. Temporal correlation: For the target detection results of two time phases t1 and t2, target matching pairs are established based on the conditions that the positional distance is less than 10 pixels, the category is the same, and the confidence level is greater than 0.8; Change type determination: (1) Adding and leaving judgment If a target is detected at time t2 and there is no matching target at time t1, and the IoU between the target bounding box and its associated changed region is greater than or equal to 0.6, it is determined to be a newly added target; if a target is detected at time t2 and there is no matching target at time t1, but the IoU between the target bounding box and its associated changed region is less than 0.6, it is determined to be a suspected changed target. If a target is detected at time t1 and there is no matching target at time t2, and the IoU between the target bounding box and its associated changed region is greater than or equal to 0.6, then it is determined to be a target that has left the target; if a target is detected at time t1 and there is no matching target at time t2, but the IoU between the target bounding box and its associated changed region is less than 0.6, then it is determined to be a suspected changed target. (2) Calculation of the change overlap rate and determination of IoUmax under the matching target If there is a matching target in time phase t1 and time phase t2, calculate the first overlap rate IOU1 between the target box and its associated change region in time phase t1, and the second overlap rate IOU2 between the target box and its associated change region in time phase t2, and take the larger value between IOU1 and IOU2 as the target change overlap rate IOUmax. If the target bounding box at time t1 is basically in the same position as the target bounding box at time t2, the Euclidean distance between the center points of the matching target bounding boxes at time t1 and time t2 is ≤10 pixels, and the target change overlap rate IOUmax is less than 0.3, then it is determined to be a target with no change. If there is a significant positional offset between the target bounding box at time t1 and the target bounding box at time t2, the Euclidean distance between the center points of the matching targets at time t1 and time t2 is greater than 10 pixels, and the target change overlap rate IOUmax is less than 0.3, then it is judged as a suspected changed target and output to the manual review module or the subsequent multi-time phase confirmation module. If the target change overlap rate IOUmax is greater than or equal to 0.3 and less than 0.6, or if the area of ​​pixels falling within the target bounding box in the changed connected region associated with the target accounts for more than or equal to 30% and less than 70% of the total area of ​​the changed connected region, then it is judged as a suspected changed target. If the target change overlap rate IOUmax is greater than or equal to 0.6, and the change area is mainly located inside the target box in time phase t1 or time phase t2, that is, in the change connected region associated with the target, the pixel area falling within the target box range accounts for ≥70% of the total area of ​​the change connected region, then it is determined to be a target with a change in target state. The change in target state includes position change, shape change, attitude change or scattering characteristic change. If the target change overlap rate IOUmax is greater than or equal to 0.6, and the change area is mainly distributed outside the target box, that is, in the change connected region associated with the target, the pixel area falling within the target box range accounts for less than 30% of the total area of ​​the change connected region, then it is judged as a suspected change target; All targets identified as suspected changes are output to the manual review module or the subsequent multi-temporal confirmation module.

8. A multi-temporal remote sensing image time-sensitive target change detection system, used to implement the multi-temporal remote sensing image time-sensitive target change detection method according to any one of claims 1-7, characterized in that, include: Data input module: used to load multi-temporal SAR satellite image data and perform radiometric correction and geometric registration preprocessing; SAR target detection module: Employs a SAR target detection network guided by strong scattering points, outputting the location, category, and confidence information of time-sensitive targets; SAR change region extraction module: Employs a difference enhancement-unsupervised classification fusion strategy to output a high-confidence change region mask; The fusion change determination module uses spatial alignment, temporal correlation, and change type determination methods to output the change type of time-sensitive targets. Results output module: Displays change detection results in a visual format, including annotations of changed areas and a list of target change types.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multi-temporal remote sensing image time-sensitive target change detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-temporal remote sensing image time-sensitive target change detection method according to any one of claims 1 to 7.

Images