Self-supervised scene change detection method, system and computer readable medium

Abstract

A computer-implemented network for performing a self-supervised change detection method, wherein at least one pair of images in a pair of images captured at different time instances is processed to detect structural changes due to appearance or disappearance of objects in the pair of images, a self-supervised pre-training method is employed that learns a representation for change detection using one or more pairs of images that are unlabelled, and the aligned pair of images is trained using a differential-based self-supervised pre-training method to maximize the correlation between the changed regions in the images, the changed regions providing the structural changes appearing in the pair of images.

Description

Technical Field

[0001] Embodiments of the present invention relate to a computer-implemented network that performs a self-supervised scene change detection method, wherein at least one image pair in images captured at different time instances is processed to detect structural changes due to the presence or absence of objects in the image pair, and a self-supervised pre-training method is employed that utilizes one or more unlabeled image pairs to learn a representation for scene change detection. Background Technology

[0002] Scene change detection (SCD) is a key perceptual task in computer vision, helping to identify changes between images of a scene captured at different times. It plays a crucial role in real-world applications such as ecosystem monitoring, urban sprawl, remote sensing, damage assessment, and high-resolution automotive mapping. Scene change detection methods based on deep neural networks (DNNs) rely on the availability of large-scale labeled images. The large-scale acquisition and labeling of these images and SCD datasets are difficult to obtain due to their labor-intensive and time-consuming nature.

[0003] To address the dependence on labeled data, various existing SCD methods initially pre-train their models on a large-scale ImageNet in a supervised manner, and then fine-tune them using extensive pixel-level annotations on domain-specific datasets. ImageNet is known from Deng, J., Dong, W., Socher, R., Li, LJ, Li, K. and Fei-Fei, L., June 2009. ImageNet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition (pp. 384 248-255) IEEE.

[0004] However, due to the significant difference in distribution between ImageNet data and the SCD dataset, the domain offset problem persists. This domain offset leads to a decrease in change detection performance in existing SCD methods. Summary of the Invention

[0005] Therefore, the purpose of this invention is to reduce the reliance on current scene change detection methods based on large-scale labeled data and to avoid the need for pre-training with additional large-scale labeled data.

[0006] According to embodiments of the present invention, a computer-implemented network for performing a self-supervised scene change detection method is proposed, wherein at least one image pair in images captured at different time instances is processed to detect structural changes due to the presence or absence of objects in the image pair. A self-supervised pre-training method is employed, which utilizes one or more unlabeled aligned image pairs to learn a representation for scene change detection. The aligned image pairs are trained using a difference-based self-supervised deep training method to maximize the correlation between changing regions in the images, which provide information about the structural changes occurring in the image pair. Therefore, the present invention relaxes the requirements for large-scale labeled datasets and the need for pre-training on additional large-scale labeled data.

[0007] To improve the effectiveness of the method of the present invention, random transformations are applied to image pairs to obtain two pairs of enhanced image pairs. These two pairs of enhanced image pairs are trained using a self-supervised training method to maximize the correlation between the changed regions in the images.

[0008] Very different transformations can be applied; for example, you can choose from a group that includes: color distortion, Gaussian blur, random cropping, shot noise, impulse noise, defocus blur, glass blur, motion blur, zoom blur, snow, frost, fog, brightness, contrast, elastic transformation, pixelation, and JPEG compression.

[0009] Preferably, the image pairs are processed to provide an output of corresponding feature representations related to structural changes in the image pairs.

[0010] Appropriately, representations of structural changes can be derived by applying absolute feature differences to the output to provide features of the changes.

[0011] Preferably, embodiments of the present invention employ feature differencing to obtain varying features. However, other distance metrics such as Euclidean distance, cosine distance, Manhattan distance, or others can be used to obtain varying features. Therefore, the present invention also covers the use of other distance metrics to obtain varying features.

[0012] Ideally, the cross-correlation of changing features should be maximized to learn the structural changes between image pairs.

[0013] It is important to note that the step of maximizing the cross-correlation of changing features employs a loss function selected from a group including contrastive learning loss functions and Barlow twin loss functions. However, it has been found that the Barlow twin loss function is preferred for maximizing the cross-correlation of changing features.

[0014] In one embodiment, a non-transitory computer-readable medium is provided that stores instructions executable by one or more processors to cause one or more processors to perform a self-supervised scene change detection method.

[0015] In one embodiment, a self-supervised scene change detection system is provided, including one or more processors; and a memory for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to execute a self-supervised scene change detection method. Attached Figure Description

[0016] One or more embodiments of the invention are illustrated in conjunction with the accompanying drawings, which are included and form part of this specification, and serve to explain the principles of the invention. The drawings are for illustrative purposes only and should not be construed as limiting the invention. In the drawings:

[0017] Figure 1 This paper presents a self-supervised pre-training scheme that learns representations of invariant regions by maximizing the cross-correlation between two images in the feature space; and

[0018] Figure 2 A difference-based scheme is shown, which represents a computer-implemented network for performing a self-supervised pre-training method for scene change detection according to the present invention. Detailed Implementation

[0019] Scene change detection (SCD) aims to identify regions of change between image pairs captured at different times. To achieve this, alignment between the radiative space of the image pairs and the low-level features of the change detection network is necessary. This alignment is challenging because regions of change are susceptible to noise variations caused by seasonal changes and viewpoint differences.

[0020] To facilitate alignment of low-level features between two images, the network of this invention employs a self-supervised pre-training method. Preferably, this method utilizes the Barlow twin objective function, which implicitly minimizes the difference between image pairs in the feature space by maximizing the cross-correlation of invariant regions. While the inventors recognize that the Barlow twin objective function is preferred due to its ability to learn robust representations without requiring a large number of negative samples, it is explicitly stated that other contrastive methods may also be used.

[0021] Although the image pairs captured at different times (T0 and T1) are semantically different from each other, they both represent the same scene at two different times. Therefore, in this invention, T0 is regarded as an enhanced version of the T1 image, and vice versa. Figure 1 (The top image) shows these input image pairs being fed into the Siamese encoder (f θ In this encoder, a multi-layered convolutional neural network (CNN) shares parameters θ to generate feature vectors f0′ and f1′. Then, a nonlinear projection head (g...) θ The nonlinear projection head (g) is applied to the encoded feature vectors to obtain representations z0′ and z1′. θ It consists of small multi-level perceptron (MLP) layers with non-linear activation functions.

[0022] The model is trained in a self-supervised manner using an objective function.

[0023]

[0024]

[0025] Where λ is the trade-off constant, C is the cross-correlation matrix calculated along the batch of samples b based on the representations of the input image pairs (z0′ and z1′), and i and j are the vector dimensions of the network output.

[0026] The objective function can be derived from the preprint arXiv: 2103.03230, 2021 (Zbontar,J.,Jing,L.,Misra,I.,LeCun,Y.and Deny,S.,Barlow twins:Self-supervised learning via redundancy reduction.arXiv preprint arXiv:2103.03230).

[0027] The objective function consists of two parts:

[0028] -(1) Invariant terms, by maximizing the diagonal components of the cross-correlation matrix that approximates the identity matrix, ensure that the representation of the input image pair (T0, T1) remains invariant to the presence of noise variations (e.g., seasonal variations), and

[0029] -(2) Redundancy reduction term, which attempts to decorrelate the off-diagonal components of the cross-correlation matrix, thereby aligning the representations of input image pairs to similarity. Therefore, this method provides a temporally consistent representation useful for downstream tasks of scene change detection.

[0030] Figure 2 (Bottom) This involves maximizing the correlation between two images. Since the representations of the two images, along with the representations of the changed regions, are forced to be closer together, this may affect the model's discriminative ability in downstream SCD tasks. Therefore, contrary to the above, in the case of increased correlation between the image pair (T0, T1), in subsequent steps, this invention employs a difference-based self-supervised pre-training called D-SSCD, which maximizes the correlation of the changed regions to learn discriminative representations that are crucial for SCD.

[0031] In this step, image pairs (T0, T1) obtained from different time instances are used as input. Random transformations such as color distortion and Gaussian blur are applied to these input image pairs to obtain two pairs of enhanced image pairs. Figure 2 Bottom image.

[0032] The enhanced image pairs are passed to the Siamese encoder (f θ ) and projection head (g θ In this context, the output is represented by corresponding features. Here, the encoder refers to a multi-layered convolutional neural network (CNN) that helps extract information from images, and the projector head refers to a multi-level perceptron (MLP) layer with non-linear activation functions in between. The model parameters (θ) are shared. To learn representations of the features that vary between image pairs, absolute feature difference is applied to the projected output – see [link to documentation]. Figure 1 Bottom image.

[0033]

[0034] Then, the Barlow twin objective function is applied to the difference representations d1 and d2 to maximize the cross-correlation of the changing features. In this way, the model will strive to learn non-redundant information about the correlated changes that occur between image pairs.

[0035] Finally, the encoder representation learned from the unlabeled image pairs in this way is fed into the downstream task of scene change detection to identify structural changes between image pairs. In the downstream task of scene change detection, the encoder representation learned from differentially-based self-supervised pre-training (D-SSCD) is fed into an existing SCD method, where the same encoder and its learned representation are deeply retrained with a finite number of labeled image pairs.

[0036] In one embodiment, a non-transitory computer-readable medium is provided that stores instructions executable by one or more processors to cause one or more processors to perform a self-supervised scene change detection method.

[0037] In one embodiment, a self-supervised scene change detection system is provided, including one or more processors; and a memory for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to execute a self-supervised scene change detection method.

[0038] Embodiments of the present invention may include every combination of the mutually independent features disclosed herein. Although the invention has been discussed above with reference to exemplary embodiments of the methods of the invention, the invention is not limited to this specific embodiment, which may be varied in many ways without departing from the invention. Therefore, the exemplary embodiments discussed should not be interpreted strictly in relation to the appended claims. Rather, these embodiments are intended only to interpret the wording of the appended claims and not to limit the claims to this exemplary embodiment. Therefore, the scope of protection of the invention should be interpreted solely according to the appended claims, wherein the exemplary embodiments should be used to resolve any possible ambiguities in the wording of the claims. Variations and modifications of the invention will be apparent to those skilled in the art, and it is intended that all such modifications and equivalents be covered in the appended claims. The full disclosure of all references, applications, patents, and publications cited above is incorporated herein by reference. Unless specifically stated above as “essential,” the various components or their interrelationships are not essential to the operation of the invention. Rather, the desired results can be achieved by substituting the various components and / or reconfiguring the relationships between them.

[0039] Optionally, embodiments of the present invention may include a general-purpose or special-purpose computer or distributed system programmed with computer software to implement the above steps. This computer software may employ any suitable computer language, including but not limited to C++, FORTRAN, BASIC, Java, Python, Linux, assembly language, microcode, distributed programming languages, etc. The device may also include multiple such computers / distributed systems (e.g., connected via the Internet and / or one or more intranets) with various hardware implementations. For example, data processing may be performed by a suitable programmed microprocessor, computing cloud, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), etc., combined with suitable memory, network, and bus elements. One or more processors and / or microcontrollers may operate via computer code instructions; preferably, the software is stored on one or more tangible, non-transitive memory storage devices.

Claims

1. A self-supervised scene change detection method executed by one or more computer processors in a computer-implemented network, the method comprising: Capture images at different time instances to obtain at least one unlabeled aligned image pair; The image pairs are trained using a difference-based self-supervised deep training method to detect structural changes caused by the presence or absence of objects in the image pairs; The training includes: Using the image pair as input, a feature representation corresponding to the image pair is obtained; absolute feature difference is applied to the feature representation to obtain a difference representation of the feature representation; based on the difference representation, the correlation between the changing regions in the image is maximized, and the changing regions provide the structural changes that occur in the image pair.

2. The method according to claim 1, further comprising: A random transformation is applied to the image pair to obtain two pairs of enhanced image pairs. The two pairs of enhanced image pairs are then trained using the self-supervised deep training method to maximize the correlation between the changed regions in the image.

3. The method according to claim 2, further comprising: The random transformation is applied to the image pair, and the random transformation is selected from the group consisting of: color distortion, random cropping, Gaussian blur, shot noise, impulse noise, defocus blur, glass blur, motion blur, zoom blur, snow, frost, fog, brightness, contrast, elastic transformation, pixelation, and JPEG compression.

4. The method according to claim 2, further comprising: The enhanced image pairs are processed to provide an output of corresponding feature representations related to the structural changes in the enhanced image pairs.

5. The method according to claim 4, further comprising: The representation of the structural change is derived by applying absolute feature difference to the output to provide the changing features.

6. The method according to claim 5, further comprising: Maximize the cross-correlation of the features of the changes to learn the structural changes between the image pairs.

7. The method according to claim 6, wherein, The step of maximizing the cross-correlation of the features of the change employs Euclidean distance, cosine distance, Manhattan distance, or other distance metrics that can be used to obtain the features of the change.

8. The method according to claim 6, wherein, The step of maximizing the cross-correlation of the features of the change employs a loss function selected from a group including contrastive learning loss function and Barlow twin loss function.

9. A non-transitory computer-readable medium storing instructions executable by one or more processors to cause the one or more processors to perform operations including the method of any one of claims 1-8.

10. A self-supervised scene change detection system, comprising one or more processors; and a memory for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to perform operations including the method of any one of claims 1-8.

Images