Image difference recognition

Abstract

1. A computer-implemented method for identifying one or more changes across a plurality of images, the method comprising: at a convolutional neural network, CNN, encoder, receiving CNN input data including data associated with each pixel of the plurality of images; and propagating the CNN input data through the CNN encoder to generate a plurality of feature maps, each feature map including a feature classification for each pixel of a respective image of the plurality of images according to a feature classification scheme, the feature classification scheme including a plurality of classifications, generated by the CNN encoder based on training data; at a ConvLSTM network, receiving ConvLSTM input data including the plurality of feature maps generated by the ConvLSTM encoder; and propagating the ConvLSTM input data through the ConvLSTM network to generate a change map, the change map including change data indicative of the one or more changes across the plurality of images.

Description

Technical Field

[0001] The present invention relates to identifying differences across multiple images. In one possible embodiment, some embodiments of the present invention can be used to identify changes over time in a target or environment by identifying differences between multiple images of the target or environment.

Background Art

[0002] In many fields of image analysis, it is desirable not only to analyze an image alone, but also to identify differences across multiple similar or linked images to obtain additional information not available from simply analyzing each image individually. For example, identifying changes in a series of images of the same object captured at different time steps can provide the owner of the images with information about how the object of the images changes over time. This can find applications in many technical fields, one of which is the environmental analysis of images generated from data acquired by one or more satellites. In other contexts, the same or similar images may be collected using different ranges of technologies. For example, the images may be captured using imaging signals of different wavelengths to capture one or more of optical images, radar images, microwave images, infrared images, UV radiation images, or X-ray images. Differences between such images can provide information about the characteristics of the imaging object, such as spectral characteristics such as density, chemical composition, refractive index, and / or absorption and / or reflection coefficients.

[0003] When attempting to identify differences across multiple images, there are two types of resolution to consider. Firstly, it may be desirable to resolve features within each image, which can be considered analogous to spatial resolution. Additionally or alternatively, it may be desirable to resolve variations between images, which can be considered analogous to temporal resolution. Achieving both high spatial and temporal resolution simultaneously leads to very large memory requirements. As a result, methods for identifying differences across multiple images with high spatiotemporal resolution requirements can place an excessive burden on the user's computing resources, both in terms of the amount of memory storage required and the amount of processing time required to implement the method.

[0004] The embodiments described below are not limited to those that resolve any or all of the shortcomings of the known methods described above. [Overview of the project]

[0005] This summary is provided to illustrate in a simplified manner the selection of concepts that will be further described in the detailed description below. This summary is not intended to identify the main or necessary features of the subject matter of the claims. Modifications and alternative features used to facilitate the implementation of the invention and / or to achieve substantially similar technical effects are deemed to fall within the scope of the invention disclosed herein.

[0006] The present invention is defined as described in the appendix of the claims.

[0007] In a general sense, this disclosure provides a method for identifying changes across multiple images using a system of convolutional neural network (CNN) encoders connected to a convolutional long-short-term memory ("ConvLSTM") network. In this way, the method and system described herein achieve a method for identifying changes at both high resolution within individual images and high resolution between different images, while reducing memory requirements.

[0008] ConvLSTM networks are well-known in this field, and examples are described in Xingjian SHI et al.'s "A Machine Learning Approach for Precipitation Nowcasting" in "Advances in Neural Information Processing Systems" 28 (NIPS 2015) ISBN:9781510825024, and in Elsayed et al.'s "Effects of Different Activation Functions for Unsupervised Convolutional LSTM Spatiotemporal Learning" in the April 2019 issue of "Advances in Science, Technology and Engineering Systems".

[0009] In a first embodiment, a computer implementation is provided for identifying one or more changes across a plurality of images, the method comprising: receiving CNN input data in a convolutional neural network (CNN) encoder, which includes data associated with each pixel of the plurality of images; propagating the CNN input data through the CNN encoder to generate a plurality of feature maps, each feature map including a feature classification scheme for each pixel of each of the plurality of images, which is generated by the CNN encoder based on training data; receiving ConvLSTM input data in a ConvLSTM network, which includes the plurality of feature maps generated by the CNN; and propagating the ConvLSTM input data through the ConvLSTM network to generate a change map, the change map including change data indicating one or more changes across the plurality of images.

[0010] In this way, one or more changes across multiple images can be identified based on a change map.

[0011] In another embodiment, a computational system is provided configured to identify one or more changes across a plurality of images by performing any of the methods described herein, the system comprising a Convolutional Neural Network (CNN) encoder configured to receive CNN input data, which includes data associated with a plurality of images, at the input of a CNN encoder, and to propagate the CNN input data through the CNN encoder to generate a plurality of feature maps, each feature map including a feature classification for each pixel of each of the plurality of images according to a feature classification scheme, the feature classification scheme including a plurality of classifications, generated by the CNN encoder based on training data, a ConvLSTM network, and a data connection link between the CNN encoder and the ConvLSTM network, the ConvLSTM network configured to receive ConvLSTM input data, which includes a plurality of feature maps generated by the ConvLSTM encoder, at the input of the ConvLSTM network, via the data connection link, and to propagate the ConvLSTM input data through the ConvLSTM network to generate a change map, the change map including change data indicating one or more changes across the plurality of images.

[0012] In other words, the computing system may be configured to implement any of the methods described herein. In some embodiments, the computing system may be contained within a single computing device and stored, for example, as computer-executable instructions on a computer-readable medium executed by a processor.

[0013] In another embodiment, a method is provided for training any of the computational networks and / or systems described herein. The method includes the steps of: providing training data including data representing a first and a second feature classification; pre-training a CNN encoder to generate a preliminary map based on the data representing the second feature classification and the output of the CNN encoder; and training both the CNN encoder and the ConvLSTM network based on the data representing the first feature classification and the output of the ConvLSTM network, wherein the trained CNN encoder is configured to generate a plurality of feature maps according to a feature classification scheme, and the trained ConvLSTM network is configured to generate a change map.

[0014] In another embodiment, a device is provided that includes a processor configured to perform one of the methods described herein. The processor may comprise any of the components of the computing network and / or system described herein.

[0015] In another embodiment, a computer program product is provided in which, when the program is executed by a computer, instructions are provided that cause the computer to perform one of the methods described herein.

[0016] In another embodiment, a computer-readable medium is provided that, when executed by a computer, contains instructions causing the computer to perform one of the methods described herein.

[0017] The method described herein can be executed by machine-readable software on a tangible storage medium, which includes, for example, a computer program code device adapted to perform any of the steps of the method described herein when the program is running on a computer, and the computer program can be embodied on the computer-readable medium. Examples of tangible (or non-temporary) storage mediums include magnetic disks, thumb drives, and memory cards, but do not contain propagating signals. The software may be adapted to run on a parallel or serial processor so that the method steps can be executed in any suitable order or simultaneously.

[0018] This application recognizes that firmware and software are valuable commodities that can be traded individually. This is designed to include software that is computed or controlled on “dumb” or standard hardware to perform the necessary functions. It also aims to include software that “describes” or defines hardware configurations, such as HDL (Hardware Description Language) software used to design silicon chips or configure general-purpose programmable chips to perform desired functions.

[0019] The features and embodiments discussed herein may be appropriately combined as will be apparent to those skilled in the art, and may be combined with any of the embodiments unless it is expressly provided that such combination is impossible or that those skilled in the art will understand that such combination is obviously impossible.

[0020] Embodiments of the present invention will be described by reference to the following drawings, as an example. [Brief explanation of the drawing]

[0021] [Figure 1] This diagram shows a schematic representation of satellites orbiting the Earth that collect SAR image data. [Figure 2a]A schematic diagram of a computing system configured to identify differences across multiple images, according to some embodiments of the present invention, is shown. [Figure 2b] A schematic diagram of the operation of the CNN encoder of the computing system of FIG. 2a is shown. [Figure 2c] A schematic diagram of the operation of the ConvLSTM network of the computing system of FIG. 2a is shown. [Figure 3] A method for identifying differences across multiple images, according to some embodiments of the present invention, is shown. [Figure 4] A method for training the computing system of FIG. 2a, according to some embodiments of the present invention, is shown. [Figure 5] Results of an example of identifying areas of deforestation in a forest environment imaged by SAR, according to some embodiments of the present invention, are shown. [Figure 6] A computer configured to execute the method of the invention according to the claims is shown.

[0022] Throughout the drawings, like reference numerals are used to represent like features.

Mode for Carrying Out the Invention

[0023] Hereinafter, embodiments of the present invention will be described as an example. These embodiments are not the only way to enable this, but represent the best patterns for carrying out the present invention that the applicant currently knows. In this description, the functions of the samples and a series of procedures for constructing and operating the samples will be described. However, the same or equivalent functions and sequences may be implemented by different examples.

[0024] The methods and systems described herein provide a means for identifying changes across multiple images with minimal loss of resolution (i.e., resolution within each image, or resolution of variations between images). For example, by generating a feature map for each image via a CNN encoder, the memory requirements of the method are significantly reduced while minimizing the loss of resolution within each image. This is because the CNN encoder can simplify image data, which may contain several data values ​​per pixel for the feature map. Such a feature map encodes a single value for each pixel, and the encoded value for each pixel represents the feature classification assigned to the pixel by the feature map.

[0025] In some embodiments, the change data encoded in the change includes quantitative data indicating the degree of one or more changes across multiple images. As described below, this allows the user to determine not only the presence of a change but also the degree of that change across multiple images. Due to the simplification of image data by the CNN encoder, this quantitative change analysis can be performed with relatively modest memory and processing requirements.

[0026] In some embodiments, the change data includes a change classification for each pixel in a selected image of a group of images, where, for a given pixel in a selected image, the change classification indicates whether the feature classification of the pixel is the same as or different from the feature classification of a corresponding pixel in another image of the group of images. In some embodiments, the change classification is a binary classification, i.e., a classification of whether a change has occurred. The memory requirements of the ConvLSTM can be further reduced by encoding the change maps generated by the ConvLSTM according to the change classification scheme. Importantly, in contrast to conventional neural network implementations, the methods and systems described herein focus on detecting whether a change has occurred across a group of images and do not rely on attempting to visualize and display the exact evolution of the changes across the images, even though these changes may be visible. Furthermore, creating change maps simplifies the process of change discrimination analysis. Instead of generating a stack of data detailing extensive information associated with each pixel in each image, the methods described herein can simply generate a dataset that identifies whether a change in the feature classification of each pixel has occurred across a group of images, and in some embodiments, regardless of the degree of the change, for each pixel in one of the images.

[0027] In practice, by reducing the change identification problem from a quantitative question (i.e., how much change is there between images) to a qualitative question (whether change has occurred between multiple images), the memory requirements of this method are significantly reduced, and the computer implementing this method can maintain high resolution in the multiple feature maps generated by the CNN encoder and the change maps generated by the ConvLSTM. The method described here leverages memory savings from the change map generation and change map classification schemes to provide high resolution with low memory and processing requirements. This would not be achievable without a specific configuration combining the CNN encoder with the ConvLSTM network, as provided in the method and system described here.

[0028] Figure 1 shows a schematic diagram of such a satellite 10 in orbit around the Earth 12 that collects SAR image data. The image data to be analyzed according to the method described herein may be SAR image data collected by satellite 10 in low Earth orbit (LOE) around the Earth.

[0029] Satellite 10 may image a target, for example, a region on Earth 12, such as an Arctic or Antarctic environment, a forest environment, or an urban environment, or any other landscape of interest. Satellite 10 is in a repeating orbit and images the same target environment at least once per orbital period. As Satellite 10 sequentially images the target in each pass of its orbit around Earth 12, each sequential image may be added to a stack that forms multiple images. As described below, each time a sequential image is added to the stack of images, change identification across the entire image can be applied to generate a sequential change map, allowing the evolution of the landscape of interest to be tracked. In some examples, this can be represented by a video or a series of images showing the sequential change map and their evolution over time.

[0030] As described above, in some embodiments, each of the multiple images is generated by synthetic aperture radar imaging (SAR). SAR is particularly well suited for use in satellite-based imaging systems because it can "see through" opaque atmospheric structures such as clouds and smoke from fires and directly image the Earth's surface.12

[0031] The method described here is applicable to a wide range of images. In particular, this method finds especially useful applications in SAR imaging. SAR can be used to create two-dimensional images and / or three-dimensional reconstructions of imaged objects such as landscapes.

[0032] As described above, in some embodiments, each of the multiple images is generated from data acquired by one or more satellites.

[0033] Satellite imagery provides an opportunity to collect image data on a wide variety of targets. For example, images generated from data acquired by satellites, such as satellite SAR imagery, can be used to image forest areas, urban environments, Arctic or Antarctic environments, or other landscapes. By applying the methods described herein to images generated from data acquired by satellites, users of the methods can easily identify the degree of environmental change in the imaged landscape. This may include, for example, the identification and detection of deforestation / forest regeneration, urbanization / de-urbanization, landslides, ice floe growth / contraction, and / or other environmental changes.

[0034] As described above, in some embodiments, each of the images generated from data acquired by the satellite is generated by a satellite in low Earth orbit.

[0035] Satellites in low Earth orbit may have short orbital periods of less than 1 hour, less than 90 minutes, less than 2 hours, less than 4 hours, less than 6 hours, less than 12 hours, less than 18 hours, or less than 1 day. Low Earth orbit is at an altitude of 160 to 1000 kilometers above the Earth's surface. Therefore, an example of an Earth-based observation satellite based on SAR can have an orbit of 450 to 650 kilometers above the Earth. In one example, satellite 10 may have an orbit of 550 kilometers above the Earth's surface. For example, in an orbit of 550 kilometers above the Earth, the satellite effectively traverses the ground at approximately 7.5 kilometers per second, or 27,000 kilometers per hour. Most satellites in this orbit traverse the Earth at a speed of 7-8 kilometers per second. Applying the method described herein to images generated from data acquired by one or more satellites in low Earth orbit provides users of the method with a means to achieve high temporal resolution of changes in the imaged object, in addition to high spatial resolution. This allows users to obtain more accurate detection and / or identification of spatiotemporal changes in imaged objects.

[0036] In some embodiments, each of the multiple images is an image of a common target captured at different times, such that identifying one or more differences across the multiple images is equivalent to identifying changes in the subject over time.

[0037] In this way, the resolution of features within a single image among multiple images can be considered as spatial resolution, and the resolution of changes across multiple images can be considered as temporal resolution. The method described herein enables the detection of temporal changes in an imaged object with high spatiotemporal resolution without imposing an excessive burden on the computer performing the required method in terms of memory or processing requirements. This allows any user of the method described herein to detect and identify changes in an imaged object with good spatial and temporal resolution, even on devices, networks, or systems with strict memory, storage, or processing limitations.

[0038] In some embodiments, each of the multiple images is consistent with each of the other images. Two images that are coherent with respect to each other are acquired by their respective imaging signals, which have a fixed relationship.

[0039] As will be explained in more detail below, phase information, particularly relative phase changes between images, can reveal additional information beyond the information encoded in the respective preliminary or characteristic graphs of each image. When images are coherent, the information encoded in phase changes between images is suitable for enhancing coherent analysis, including coherent change detection (CCD), digital elevation model (DEM) generation, or differential interferometry synthetic aperture radar imaging (InSAR).

[0040] CCDs can detect coherent changes between images that are virtually invisible to the human eye. This is because the sensitivity of a CCD is limited to a fraction of the wavelengths of light used to collect the image. For example, in radar imaging, a CCD can resolve changes of a few centimeters from satellite-acquired images. In synthetic aperture radar (SAR) imaging, a CCD provides users with the ability to see subtle differences between two SAR images with a resolution that exceeds "naked-eye" analysis.

[0041] The DEM utilizes subtle positional differences between two coherent images. Phase information associated with each pixel in the coherent images is then compared, highlighting variations relative to a reference plane. In other words, in SAR imaging, phase information can be used to estimate the height of features in the image relative to a reference "zero" height. DEM generation allows for the acquisition of this height data from phase variation information via phase unwrapping, enabling the formation of a three-dimensional digital elevation model of the region.

[0042] InSAR can be thought of as a combination of the CCD and DEM technologies mentioned above. In particular, InSAR facilitates the detection of very subtle changes in altitude over time. Images generated from data acquired by one or more satellites can be analyzed with InSAR to detect changes in the environment of about one millimeter over a month. This can be used to identify a variety of hazardous and emerging situations, from land landslides to infrastructure collapses, such as the collapse of dams and bridges.

[0043] In some embodiments, each of the multiple images may be an image of an area of ​​10 square kilometers or more, 50 square kilometers or more, 100 square kilometers or more, 1000 square kilometers or more, 10000 square kilometers or more, or faster, or 10000 square kilometers or more.

[0044] For example, each image may have an area of ​​5 kilometers x 5 kilometers or larger, 10 kilometers x 10 kilometers or larger, 50 kilometers x 50 kilometers or larger, or 100 kilometers x 100 kilometers or larger.

[0045] Even with such large area target sizes, the method described here enables spatial resolution of differences across multiple images representing features of sizes of 0.1 meters or less, 0.5 meters or less, 1 meter or less, 5 meters or less, 10 meters or less, or 50 meters or less.

[0046] The methods and systems described herein are particularly useful when applied to images that have undergone speckle patterns, such as when applied to data collected by satellites, especially SAR images. Speckle patterns arise from the coherence of a series of coherent wavefronts in the imaging signal. This physical phenomenon occurs when a coherent imaging signal is reflected from the object being imaged. Each of the many reflection points of the signal (based on diffraction theory) acts as a source of spherical waves. Multiple spherical waves returning to the imaging device, e.g., satellite 10, coherently coherent with each other, resulting in a characteristically "blurred" speckle pattern.

[0047] Analyzing images subject to speckle patterns can be extremely difficult because such images appear "noisy" to conventional algorithms used to identify objects contained within them. In practice, speckle patterns cause significant fluctuations in the data associated with adjacent pixels in an image, and therefore the informational content of a single pixel is very limited. As described below, the methods and systems disclosed herein provide means to mitigate the problems posed by such speckle patterns.

[0048] Figures 2a to 2c show schematic diagrams of the operation of a computing system 200 and its components, which may be configured to identify differences across multiple images.

[0049] Figure 2a shows a schematic diagram of a computing system 200 configured to identify differences across multiple images 210.

[0050] The data associated with multiple images 210 is input to the CNN encoder 220 to form CNN input data. The convolutional neural network of the CNN encoder may include multiple convolutional layers such that, as the data associated with the multiple images 210 propagates through the CNN encoder 220, the data associated with each of the multiple images 210 is convolved to generate multiple feature maps 230. Each feature map is associated with one of the multiple images 210.

[0051] In some embodiments, the data associated with each of the multiple images 210 includes amplitude data representing one or more amplitudes associated with each pixel of each of the multiple images 210. The amplitude data may include, for example, data values ​​representing the amplitudes associated with each of the R, G, and B channels in an RGB image, or data values ​​representing the amplitudes associated with each of the C, M, Y, and K channels in a CMYK channel, or data values ​​representing grayscale values ​​in a grayscale image.

[0052] In other words, amplitude data can be encoded as RGB values. That is, there may be data values ​​associated with each pixel, representing each of the R, G, and B values, and optionally each of the brightness values. Additionally or alternatively, amplitude data can be encoded as CMYK values. In other words, there may be data values ​​associated with each pixel, representing each of the C, M, Y, and K values, and optionally each of the brightness values. Additionally or alternatively, amplitude can be represented according to a grayscale. In other words, there may be data values ​​representing the hue of a pixel along a grayscale. Such data values ​​can be encoded as 8-bit integers or in another suitable format.

[0053] In some embodiments, the CNN input data further includes phase data indicating the respective phase values ​​of each pixel in each of a plurality of images 210, and the feature classification of each pixel in each image by its respective feature map 230 is at least partially based on the phase data, as will be described in more detail below.

[0054] When analyzing two or more images for differences between them, the relative phase changes between images can reveal further information beyond that encoded in the respective feature maps of each image. For example, in some cases, the phase value associated with each pixel of an image may indicate the phase of the signal used to acquire / generate the image at the location corresponding to that pixel. Thus, in reflection-based imaging, the phase value may indicate the distance between the target surface to which the signal is reflected and the pixel imaging detector to generate the image. On the other hand, in transmission-based imaging, the phase value may indicate the density of the target to which the signal is transmitted.

[0055] In other words, the phase data can indicate the optical path length of the imaging signal used to generate at least one of the multiple images 210. For example, in the scenario of the SAR satellite system 10 in Figure 1, the phase data indicates the relative height of the imaged target on the surface of the Earth 12.

[0056] The CNN encoder 220 is trained to generate feature maps 230 from data associated with each of the multiple images 210. Each feature map 230 contains a feature classification for each pixel of each of the multiple images 210, according to a feature classification scheme. The feature classification scheme is generated by the CNN encoder 220 based on its training with the training data. A further explanation of the training of the CNN encoder 220 can be found below in relation to Figure 4.

[0057] In some embodiments, the feature classification scheme is a binary classification scheme configured to classify identified objects as belonging to either a first feature classification or a second feature classification.

[0058] By simplifying the feature classification scheme to a binary classification scheme, the memory requirements of the method described herein are reduced. In particular, the binary classification scheme allows each of the feature maps 230 to be encoded in a series of one-hot encoding processes or similar.

[0059] In some embodiments, the feature classification scheme includes feature classifications of forested areas versus non-forested areas, glacial terrain versus non-glacial terrain, inhabited areas versus uninhabited terrain, buildings versus non-buildings in urban environments, land versus water, and / or any other suitable set of feature classifications. This allows users of the computing system 200 to detect, identify, and / or measure processes such as deforestation / forestation regeneration, ice (floor) growth / shrinkage, nomadic movement, (anti)urbanization, coastal erosion / development, and / or any other processes that can be detected, identified, and / or measured based on the suitable feature classification scheme.

[0060] In other words, in some embodiments, the feature classifications of the feature classification scheme include a forest classification indicating that the pixels thus classified represent forested areas and a non-forest classification indicating that the pixels thus classified represent land that is not forested, and the method further includes identifying areas of deforestation around forested areas based on differences identified across multiple images.

[0061] In this way, it may be possible to track deforestation or reforestation in areas of interest to users of the method. These areas may include rainforests such as the Amazon rainforest that are affected by logging operations or other deforestation projects. These areas may also include other forest areas affected by deforestation. Deforestation may include artificial deforestation, such as logging projects (permitted or unpermitted), and natural deforestation means, such as fires. For example, the method described here, applied to detect and identify areas of deforestation, may be able to track deforestation caused by wildfires, such as in forests in California, Australia, and Canada.

[0062] Furthermore, in some embodiments, the feature classifications of the feature classification scheme include ice classifications indicating that the thus classified pixels represent areas defined by or covered with ice, and non-ice classifications indicating that the thus classified pixels represent areas without ice, and the method further includes identifying areas of ice expansion or retreat in an Arctic or Antarctic environment over time based on differences identified across multiple images.

[0063] In this way, it may be possible to track the shrinking or growth of ice floes and / or glaciers in the Arctic and / or Antarctic circles. This could be used, for example, to provide information on the state of polar ice caps and other ice environments to governments, non-governmental organizations, climate scientists, and / or other users as a means of tracking the effects of climate change.

[0064] In some embodiments, the feature classifications of the feature classification scheme include building classifications indicating that the pixels thus classified represent buildings, and non-building classifications indicating that the pixels thus classified represent features that are not buildings, and the method further includes identifying construction and / or demolition sites in an urban environment over time based on differences identified across multiple images.

[0065] In this way, it may be possible to track the development or demolition of urban environments. This allows users of the methods described herein to provide a means for analyzing urban spread / sprawl. In other words, users applying the methods described herein to this context may be able to detect and identify urbanization and / or de-urbanization in areas of interest.

[0066] In some embodiments, the feature classifications of the feature classification scheme include a residential classification indicating that the thus classified pixels represent a residential location within the imaged area, and a non-residential classification indicating that the thus classified pixels represent an uninhabited location within the imaged area, and the method further includes identifying the construction and / or removal of residential locations in uninhabited or partially inhabited environments over time based on differences identified across multiple images.

[0067] In this way, by determining when and where their settlements or camps are built and dismantled, it may be possible to track the movements of nomadic peoples, or groups of otherwise mobile individuals.

[0068] In some embodiments, the feature classifications of the feature classification scheme include a land classification indicating that the thus classified pixels represent land, and a water classification indicating that the thus classified pixels represent a body of water, and the method further includes identifying coastal erosion of land or similar by body of water based on differences identified across multiple images.

[0069] In this way, it may be possible to track the progression of coastal erosion in the area of ​​interest. Additionally or alternatively, the effectiveness of coastal defenses in slowing the progression of coastal erosion may be determined by establishing the extent to which coastal erosion has slowed after the introduction of coastal defenses in the area of ​​interest.

[0070] The methods described above can be used in a variety of other situations in addition to those mentioned above. For example, the method applied to spatiotemporal resolution can be used to detect and identify landslides and / or other areas susceptible to avalanches in hills, cliffs, and mountains. This may be based, at least in part, on phase changes between images indicating a shift in ground elevation. In an alternative example, the method can be applied to a situation where multiple images are each taken simultaneously from the same object, but each image is captured by an imaging signal of a different wavelength. These imaging signals may be, for example, but not limited to, optical signals, microwave signals, radio signals, infrared signals, ultraviolet signals, and / or X-ray signals. Multiple images produced by imaging signals of different wavelengths can be analyzed according to the methods described herein to identify and / or detect spatial and spectral, or spatial spectral, variations across the multiple images. Identification and / or detection of spatial spectral differences can be used in many situations. For example, such a method may be applicable to the identification of coronal mass ejections from the sun or other solar activity.

[0071] In some embodiments, the computing system 200 further comprises a skip connection 240. The skip connection provides a propagation path for copies of data associated with multiple images, thereby effectively skipping the CNN encoder 220 for the copies.

[0072] In the operation of the CNN encoder 220, the CNN is trained to provide successive layers, increasing the number of operations performed by the neural network, and thus the CNN can infer additional information from each of multiple images. This can, in some situations, lead to feature map errors if the training is ineffective. Any errors introduced through the operation of the CNN encoder 220 can be mitigated by providing skip connections and convolving the data associated with each of the multiple images in its respective feature map. This improves the reliability and accuracy of the final change map generated by the ConvLSTM network 260.

[0073] Each of the outputs of the CNN encoder 220, i.e., the multiple feature maps 230, is convolved by the convolutional unit 250 with the data associated with each of the multiple images 210. The convolutional unit 250 convolves the data associated with each of the multiple images 210 with its respective feature map 230 to generate the ConvLSTM input data, i.e., the data input to the ConvLSTM network 260. The ConvLSTM input data can be thought of as multiple convolution maps. The ConvLSTM network 260 is a recurrent neural network configured to convolve the ConvLSTM input data to generate a single change map 270. In other words, the ConvLSTM network 260 convolves each of the multiple convolution maps with each other to generate the change map 270. Furthermore, the ConvLSTM network 260 convolves the data within each map of the ConvLSTM input data. In other words, the ConvLSTM network 260 is configured to convolve both between maps and within maps. For example, in a case where each of multiple images 210 represents an image of a common target captured at different times, the ConvLSTM network is configured to convolve the data both spatially and temporally. The change map 270 may include a change classification for each selected pixel among the multiple images 210. This change classification for each pixel may indicate whether the feature map classification for that pixel is the same as or different from the feature map classification for the corresponding pixel in another image among the multiple images 210. Additionally or alternatively, the change map 270 may include quantitative change data indicating the degree of one or more changes across the multiple images. For example, in the context of SAR imaging, the quantitative change data may indicate the degree to which the relative height of each captured pixel has changed across the multiple images 210.

[0074] In some embodiments, the change classification is binary classification. As described above, binary classification can be particularly efficient because it allows each pixel to be encoded in its respective change classification by one-hot encoding or a similar rapid process.

[0075] In some embodiments, the ConvLSTM input data further includes phase data indicating the respective phase values ​​of each pixel in each of a plurality of images 210, and propagating the ConvLSTM input data through the ConvLSTM network 260 includes convolving the phase data with a plurality of feature maps 230 to generate a change map 270.

[0076] As described above, phase information, particularly relative phase changes between images, can provide additional information beyond the information encoded in the respective feature maps of each image. By convolving the phase data with multiple feature maps 230 to generate a change map 270, the phase information can be used in determining the change data associated with each pixel of the change map 270. The change data may optionally include change classifications associated with each pixel of the change map 270, and / or quantitative change data associated with each pixel of the change map 270. This increases the sensitivity of the ConvLSTM network 260, enabling it to detect and identify changes across multiple images 210 that are encoded only in the phase data, or partially.

[0077] Figure 2b shows a schematic diagram of the operation of the CNN encoder 220 in the computing system 200.

[0078] For a given image 212 of multiple images 210, the image data associated with image 212 is provided as CNN input data to the CNN encoder 220. The CNN input data is propagated through the layers of the CNN encoder 220, as described above, to generate a feature map 232. In the example shown in Figure 2b, there are nine data values ​​associated with each pixel of image 212. The CNN encoder 220 convolves the data associated with image 212 to generate a feature map 232 that has only one data value associated with each pixel. In the example shown in Figure 2b, the data values ​​associated with each pixel are part of a binary feature classification scheme. For example, in the context of a method for detecting deforestation in a forest, the binary feature classification scheme might indicate that a value of "1" in the feature map 232 indicates that the corresponding pixel of image 212 represents a forest area, while a value of "0" in the feature map 232 indicates that the corresponding pixel of image 212 represents a non-forest area.

[0079] Referring to Figure 2b, the operation of the CNN encoder in the manner described above provides a means for reducing the data content that must be processed in order to detect and identify changes across multiple images with minimal loss of (spatial) resolution within each image 212 of the multiple images 210.

[0080] Figure 2c shows a schematic diagram of the operation of the ConvLSTM network 260 in the computing system 200.

[0081] The convolutional unit 250 convolves the data associated with each of the multiple images 210 with its respective feature map 230 to generate a ConvLSTM network 260. For example, image 212 in Figure 2b is convolved with feature map 232 to generate its respective convolution map. Each of the convolved maps is then repeatedly convolved by propagation through the ConvLSTM network 260 to generate a change map 270. As described above, the change map 270 may, in some embodiments, include binary change classification of each pixel of selected images from the multiple images 210 to enable the identification, detection, and / or measurement of differences across the multiple images 210.

[0082] Additionally or alternatively, the change map 270 may, in some embodiments, include quantitative change data associated with each pixel of the selected image, indicating the degree of change between the selected image and one or more other images 210.

[0083] The computational network systems described herein are configured to implement a method for identifying changes across multiple images with minimal resolution loss, either in the resolution within each image or in the resolution of variations between images. For example, by generating a feature map 230 for each image via a CNN encoder 220, the system's memory requirements are significantly reduced, with only minimal loss of resolution within each image. This is because the CNN encoder is configured to simplify image data, which may contain several data values ​​per pixel for the feature map. Such a feature map encodes a single value for each pixel, and the encoded value for each pixel represents the feature classification assigned to the pixel by the feature map. Furthermore, the ConvLSTM network 260 may be configured to encode the change map with quantitative change data indicating the degree of one or more changes across multiple images. Due to the simplification of image data by the CNN encoder 220, this quantitative analysis can be performed with relatively modest memory and processing requirements.

[0084] Additionally or alternatively, the change maps 270 may be encoded according to a binary change classification scheme, at the option of choice. In this way, the memory requirements of the ConvLSTM network 260 can also be further reduced. Importantly, in contrast to conventional computing systems and neural networks, the computing system described herein is configured to detect whether a change has occurred across multiple images, and this evolution may, of course, be shown in some examples, but does not rely on visualizing and displaying the exact evolution of the change across the images. Furthermore, generating the change maps 270 simplifies the process of change discrimination analysis. Instead of generating a stack of data that details the information associated with each pixel in each image, the computing system described herein is simply configured to generate a single dataset for each change map 270 that identifies each pixel in one of the images, regardless of whether a change in the feature classification of each pixel has occurred across multiple images 210.

[0085] As described above, by reducing the change identification problem from a quantitative question to a qualitative one, the memory and processing power required for the system described herein to implement the method described herein can be significantly reduced without compromising the resolution of change detection and / or identification that can be achieved by the system. This would not be achievable without a specific configuration combining the CNN encoder 220 with the ConvLSTM network 260, as provided for the computational network described herein.

[0086] Furthermore, as described above, the system described in relation to Figure 2 and the method described herein are particularly advantageous for analyzing speckle-patterned images, such as SAR images, generated from data collected by a satellite 10 in low Earth orbit. Due to the large variation in data associated with adjacent pixels affected by speckle, it may be necessary to obtain a large amount of information associated with each pixel. This contextual information may be spatial or temporal. For example, temporal contextual information can be obtained by repeatedly collecting image data on the same target using high frequencies. In the case of a satellite in low Earth orbit, this may be difficult because satellite 10 can only image targets on Earth 12 as it passes overhead. An alternative to temporal contextual information is spatial contextual information. Spatial contextual information can be obtained by capturing a field of view within each image that is much larger than the size of the area of ​​one or more speckles affecting the image. For example, the area of ​​the field of view may be 5 times, 10 times, 50 times, or 100 times larger than the area of ​​one or more speckle patterns affecting the image. Such fields of view are much larger than those used in typical object recognition algorithms due to the perceived computational cost of analyzing such large fields of view.

[0087] However, the CNN encoder 220 enables the computing system 200 to capture and process large fields of view in a memory-efficient manner. In the context of a SAR imaging system, the CNN encoder 220 can compress large-area images 210 into smaller feature maps 230. For example, each of several images may have an area of ​​512 × 512 pixels, which is compressed by the CNN encoder 220 into its respective feature map 230 having an area of ​​64 × 64 pixels. This represents a surface pressure coefficient of 64. Those skilled in the art will understand that other image sizes and other surface compression ratios are possible. However, for the context, equivalent neural networks in state-of-the-art technology typically process images with smaller areas, such as 299 × 299 pixels. This smaller area may not provide a sufficiently wide field of view for the computing network 200 to mitigate the effects of speckle in high-resolution SAR images. In other words, the CNN encoder 220 is configured to process images containing more than three times the number of pixels of equivalent neural networks used in state-of-the-art technology. This broad perspective allows the CNN encoder 220 to generate feature maps that are robust against the detrimental effects of speckle patterns that affect conventional object recognition algorithms.

[0088] Figure 3 illustrates a method for identifying differences across multiple images according to several embodiments of the present invention.

[0089] In operation S300, the CNN encoder 220 receives CNN input data, which includes data associated with multiple images 210.

[0090] In operation S320, the CNN input data propagates through the CNN encoder 220, generating multiple feature maps 230. Each of the multiple feature maps 230 contains the feature classification of each pixel of each of the multiple images 210, according to the feature classification scheme generated by the CNN encoder based on the training data.

[0091] In some embodiments, propagating CNN input data through a CNN encoder 220 to generate multiple feature maps 230 includes compressing the CNN input data.

[0092] In this way, the memory storage requirements of a computer implementing any of the methods described herein are reduced without compromising the resolution within each image. For a given image of multiple images 210, the feature map 230 may include a feature classification for each pixel of the image. In contrast, the original image data may include multiple data values ​​for each pixel of the image. For example, in a conventional RGB image, there may be data values ​​associated with each pixel indicating the R, G, and B values, and optionally, the brightness. Or, for a CMYK image, there may be data values ​​associated with each pixel indicating the C, M, Y, and K values, and optionally, the brightness. In some images, there may be further data values ​​indicating the saturation of the image. In some images, there may be further data values ​​indicating the respective phase values ​​associated with each pixel of the image. By simplifying the representation of each image in the feature map, the memory requirements of a computer implementing any of the methods described herein are significantly reduced, thereby enabling the methods to be implemented on a wider variety of computer-based systems, particularly on systems where there are stringent and demanding memory requirements on computing devices, systems, and / or networks.

[0093] In some examples, the data associated with each of multiple images can be encoded as complex data. In other words, the data can be encoded as one or more complex numbers. Each complex number can be considered to have a quantity and an argument. The quantity of a complex number may represent the amplitude value, and the argument of a complex number may represent the phase value.

[0094] In some examples, the data associated with each of multiple images may include contextual data. This may include any type of metadata or other types of image data. For example, the metadata may include a timestamp. This may be beneficial because it may allow the methods and systems disclosed herein to be sensitive to time-induced image variations (e.g., images of the same target captured during the day and at night may look different) or seasonal variations (e.g., images of the same target may have different texture characteristics, being drier in summer months and wetter in spring months). The metadata may include information indicating the local incidence angle of the imaging signal. This may be beneficial, in particular in the context of satellite SAR, where there may be slight differences in the local incidence angle of the same imaging target between different images, especially in examples where the orbital period of satellite 10 is not perfectly regular. In the context of geographic feature imaging, the metadata / image data may include meteorological data, such as a rain map. This may be particularly beneficial when applied to SAR images, as the humidity of the imaging target can significantly affect the resulting SAR image.

[0095] In operation S322, the ConvLSTM network 260 receives multiple feature maps generated by the CNN encoder 220.

[0096] In some embodiments, during operation S330, a skip connection 240 is provided between the input of the CNN encoder 220 and the input of the ConvLSTM network 260.

[0097] In such an embodiment, in operation S332, a copy of the CNN input data is propagated to the input of the ConvLSTM network 260 via the skip connection 240.

[0098] Furthermore, in operation S340, the data associated with each of the multiple images 210 being copied into the CNN input data, along with their respective feature maps 230 (e.g., generated by the CNN encoder 220), are convolved to generate ConvLSTM data. This operation can be performed, for example, by the convolution unit 250.

[0099] In other words, in some embodiments, the method further includes providing a skip connection between the input of a CNN encoder and the input of a ConvLSTM network; propagating a copy of the CNN input data to the input of the ConvLSTM network via the skip connection; and convolving each of the multiple images in the copy of the CNN input data with their respective feature maps generated by the CNN encoder to generate a ConvLSTM input.

[0100] In operation S350, the ConvLSTM input data is propagated through the ConvLSTM network 260 to generate a change map 270. The change map 270 may optionally include binary change classifications for each pixel of a selected image from a plurality of images 210, as described above, where the change classification of the pixel indicates whether the feature classification of the pixel is the same as or different from the feature classification of a corresponding pixel in another image from the plurality of images 210. Additionally or alternatively, the change map 270 may include quantitative change data indicating the degree of one or more changes across the plurality of images 210.

[0101] Finally, in operation S360, one or more changes across multiple images are identified based on the change map 270.

[0102] As described above, the method shown in Figure 3 can be applied in a variety of situations. For example, this method can be used to identify the extent and progression of deforestation processes, ice (floor) growth / reduction, urbanization / de-urbanization, coastal erosion / development, nomadic movements, solar activity, and / or any other processes that can be appropriately identified by the method described herein.

[0103] Generally, each of the multiple images 210 may be an image of a geographical area, and the feature classification scheme may include a first feature classification indicating that a pixel classified in this way represents the presence of a given geographical feature, and a second feature classification indicating that a pixel classified in this way represents the absence of the given geographical feature, and this method further includes identifying areas where the presence / absence of a given geographical feature changes based on differences identified across the multiple images.

[0104] Figure 4 shows how to train the computing system 200 according to several methods according to the present invention, as shown in Figure 2a.

[0105] To train the computing system 200, in particular the CNN encoder 220, the CNN encoder can receive various inputs. The first input is the training data 40. The training data 40 can be divided into two categories 42 and 44. The first category is the data representing the first feature classification, and the second category is the data representing the second feature classification.

[0106] In some embodiments, the first category 42 is data representing a first feature classification and is relatively sparse with the second category 44 representing a relatively widespread second feature classification. For example, in the context of a method for identifying deforestation, the first category 42 may represent deforested land and the second category 44 may represent forested land, and it should be noted that in most cases of deforestation, in identifying the deforestation process, it is necessary to detect the expansion of a relatively less widespread area of ​​deforested land compared to the vast area of ​​forest being deforested.

[0107] In other words, in some embodiments, the training data used to train the CNN encoder 220 includes data representing both a first and a second feature classification, where the data representing the first feature classification in the training data is sparser than the data representing the second feature classification. For example, the ratio of the amount of data representing the second classification to the amount of data representing the first classification in the training data may be 1:5 or less, 1:10 or less, 1:50 or less, 1:100 or less, 1:500 or less, or 1:1000 or less.

[0108] The method described herein is a departure from conventional computational approaches, at least in that the method is particularly oriented towards detecting and identifying changes across multiple images 210, as opposed to identifying specific or concrete objects within an image. This means that in a particular situation, for example, in images of a forest area where the detected change indicates deforestation, it is possible to train a CNN encoder 220 to detect changes across multiple images 210 using training data 40 in which the data indicating deforested land is sparse compared to the data indicating deforested land. Those skilled in the art will recognize that this principle is equally applicable to many forms of image analysis directed towards detecting changes in very common features (or features) of an image, and that changes across an entire image are changes to relatively less common features (or features) of the image.

[0109] The second input to the training method is the output of the CNN encoder 46.

[0110] In some embodiments, as shown in operation S410, the CNN encoder 220 is pre-trained to generate a preliminary map based on data representing a more common second feature classification 44 and the CNN encoder output 46. This provides a rough pre-training of the CNN encoder 220, allowing the memory requirements for fine-tuning training of later operations to be met with fewer computational resources. The preliminary map generated by the pre-trained CNN encoder 220 can be any suitable mapping. For example, the preliminary map could be a semantic map or a regression map.

[0111] The third input to the training method is the output of the ConvLSTM network 48.

[0112] In operation S420, the overall model is refined by training both the CNN encoder 220 and the ConvLSTM network 260 on data representing the first, more sparse feature classification and the output of the ConvLSTM network 48. At the end of operation S420, the fully trained CNN encoder 220 is configured to generate feature maps 230 based on the feature classification scheme that the CNN encoder 220 learns throughout the training process of operations S410 and S420. Furthermore, at the end of operation S420, the fully trained ConvLSTM network 260 is configured to generate change maps. By implementing the training method shown in Figure 4, the CNN encoder is trained to learn image properties that represent both the first and second feature classifications without requiring a large amount of data representing the first, relatively sparse feature classification 42. The result of this training is a feature classification scheme based on the training data, on which the feature classifications contained in each feature map 230 generated by the CNN encoder 220 are based.

[0113] In other words, in some embodiments, the feature classification scheme is generated by training a CNN encoder 220, and training the CNN encoder 220 includes pre-training the CNN encoder 220 to generate a preliminary map based on the data of a second feature classification 44 and the output of the CNN encoder 46, and training both the CNN encoder 220 and the ConvLSTM network 260 based on the data of a first feature classification 42 and the output of the ConvLSTM network 48.

[0114] If the change to the image is based on a small number of pixels relative to the total number of pixels, the CNN encoder 220 can be pre-trained on features having a popular second feature classification 44 and the output of the CNN encoder 46, providing a rough training to the CNN encoder 220, and then refined by training the CNN encoder 220 on the output of the ConvLSTM network 48 and a first (target) feature classification 42 that is relatively less popular than the second feature classification. This multi-stage training improves the training efficiency of the overall computational network 200. The improved efficiency can be considered by analogy to a multi-stage rocket. Just as a multi-stage rocket launch is more efficient than a single-stage rocket launch because each subsequent stage forms an effective boost to the previous stage, multi-stage training in some embodiments of the method described herein is more efficient because the first stage of pre-training trains the CNN encoder 220 to an approximate level of accuracy which is then refined to a desired level of accuracy by the second stage of training. Simply training the CNN encoder 220 according to the second-stage method consumes a large amount of processing power and memory, and is therefore undesirably inefficient.

[0115] In some embodiments, during the operation of training the computational network 200, the weights of the CNN encoder 220 are frozen so that the pre-trained CNN encoder and the trained CNN encoder consist of the same weights. This may be preferable when there is a relatively small amount of training data, for example, less than 1000 samples of training data, less than 500 samples of training data, less than 100 samples of training data, or less than 10 samples of training data.

[0116] In such an example, freezing the weights of the CNN encoder while the ConvLSTM network 260 is being trained prevents the CNN encoder 220 from being "overfitted." This is a common problem in training neural networks with sparse training data. In such an example, the preliminary map generated by the pre-trained CNN encoder is in the same format as the feature map 230 generated by the fully trained CNN encoder 220.

[0117] In some embodiments, during the operation of training the computational network 200, the weights of the CNN encoder 220 are unfrozen so that the pre-trained CNN encoder and the trained CNN encoder can be composed of different weights. This may be suitable when there is a relatively large amount of training data, for example, more than 10 samples of training data, more than 100 samples of training data, more than 500 samples of training data, more than 1000 samples of training data, or more than 10000 samples of training data.

[0118] In such examples, due to the proliferation of training data, training of the CNN encoder can naturally be robust against overfitting. Furthermore, if a large amount of training data is available, the accuracy of the overall computing system 200 can be improved by allowing the weights of the CNN encoder to be adjusted collaboratively with the nodes and weights of the ConvLSTM network 260 during training. In such examples, if the weights of the fully trained CNN encoder 220 differ from the weights of the pre-trained CNN encoder, the preliminary map generated by the pre-trained CNN encoder will not be in the same format as the feature map 230 generated by the fully trained CNN encoder 220.

[0119] As described above, in some embodiments, as will be discussed later, a sequential change map can be generated by applying the identification of changes across the entire image each time a sequential image is added to the image stack, so that the evolution of the landscape of interest can be tracked. In some examples, this can be represented by a video or a series of images showing the sequential change map and their evolution over time.

[0120] In other words, in some embodiments, the plurality of images 210 include a sequence of images, and the method further includes propagating ConvLSTM input data through a ConvLSTM network 260 to generate a sequence of change maps 270, convolving the ConvLSTM input data associated with each of the sequence of images with the ConvLSTM input data associated with each preceding image, so that each sequence of change map 270 represents a change between one of the plurality of images 210 and the sequence of images.

[0121] Figure 5 shows the results of an example of applying the described method to identify areas of deforestation in a forest environment imaged by SAR.

[0122] SAR images 50a to 50n are collected by satellite 10 as it passes through the target forest environment. On each repeated orbit of Earth 12, satellite 10 adds another image to multiple images 50 to create a stack of coherent images of the forest environment.

[0123] Multiple images 50a to 50n and the data associated with each of these images are used as input to the computing system 200 in Figure 2a. The CNN encoder 220 convolves the data associated with each of the images 50a to 50n to generate multiple feature maps 230. Each feature map 230 contains data that shows the feature classification of its respective image according to a feature classification scheme determined based on the training of the CNN encoder according to the method shown in Figure 4. In the example shown in Figure 5, the feature classification scheme is a binary classification of either forested or non-forested areas. The multiple feature maps 230 and copies of the data associated with each of the multiple images 50a to 50n are convolved and propagated through the ConvLSTM network 260 to generate a change map 52. As shown in Figure 5, the output returned from the computing system 200 is a change map 52 overlaid on the final image 50n of the image stack. In other examples, the change map 52 may be overlaid on any of the other images among the multiple images 50a to 50n, for example, the change map 52 may be overlaid on the first image 50a.

[0124] Figure 6 shows a computer 60 or other suitable device configured to perform one of the methods described herein.

[0125] Computer 60 includes a processor 62 which includes an image input interface 61, memory 63, a CNN encoder module 64, a ConvLSTM network module 65, an image difference output interface 66, a plurality of data connection links 67a to c, and a skip connection link 68.

[0126] The computer receives image data corresponding to multiple images 210 via the image input interface 61. The data associated with the multiple images 210 is sent to the CNN encoder module 64 of the processor 62 via the data connection link 67a. The CNN encoder module 64 is configured to operate as a CNN encoder 220, as shown in Figures 2a and 2b. The output of the CNN encoder module 64, for example, multiple feature maps 230, is sent to the ConvLSTM network module 64 of the processor 62 via the data connection link 67b. The ConvLSTM network module 64 is configured to operate as a ConvLSTM network 260, as shown in Figures 2a and 2c. The CNN encoder module 64, and optionally the ConvLSTM network module 65, may be trained based on training data stored in the memory 63 of the processor 62.

[0127] In some embodiments, copies of the data associated with multiple images 210 are sent to the ConvLSTM module via a skip connection link 68. The skip connection link 68 is configured to provide a skip connection 240, as shown in Figure 2a.

[0128] The output of the ConvLSTM network module 65, for example, the change map 270, is transmitted to the image difference output interface 66 via the data connection link 67c. The image difference output interface 66 provides the user of the computer 60 with information that identifies the differences between multiple images 210 received at the image input interface 61.

[0129] In the embodiments described above, the computing system 200 may be implemented on a server. The server may be a single server or a network of servers. In some examples, the server's functions can be provided by a geographically distributed network of servers, for example, a globally distributed network of servers, and a user can connect to a suitable one of the server networks based on their location.

[0130] For clarity, the above description refers to embodiments of the present invention with reference to a single user. It should be understood that, in practice, system 200 may be shared by multiple users, and may be shared by many users simultaneously.

[0131] The above embodiments are fully automated. In some examples, the system user or operator can manually instruct some steps of the process to be carried out.

[0132] In embodiments described in the present invention, the system may be implemented as any form of computing and / or electronic device. Such a device may include one or more processors, which may be a microprocessor, a controller, or any other suitable type of processor for processing computer-executable instructions that control the operation of the device in order to collect and record routing information. In some examples, for example, when using a system-on-a-chip architecture, the processor may include one or more fixed-function blocks (also called accelerators) that implement part of the method in hardware (rather than software or firmware). Platform software, including an operating system or any other suitable platform software, may be provided in the compute-based device to enable application software to run on the device.

[0133] The various functions described herein may be implemented in hardware, software, or any combination thereof. If implemented in software, these functions may be stored or transmitted on a computer-readable medium as one or more instructions or codes. The computer-readable medium may include, for example, a computer-readable storage medium. The computer-readable storage medium may include volatile or non-volatile, removable or non-removable media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. The computer-readable storage medium can be any available storage medium accessible by a computer. Such a computer-readable storage medium may include, but not limited to, RAM, ROM, EEPROM, flash memory or other storage devices, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium accessible by a computer that is used to transport or store desired program code in the form of instructions or data structures. The optical discs and discs used herein include optical discs (CDs), laser discs, optical discs, digital multipurpose discs (DVDs), floppy disks (registered trademarks), and Blu-ray discs (BDs). Furthermore, propagated signals are not included within the scope of computer-readable storage media. Computer-readable media also include communication media, including any medium that facilitates the transmission of computer programs from one location to another. Connections may, for example, be communication media. For example, software transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave is included in the definition of communication media. Any combination of the above should also be included within the scope of computer-readable media.

[0134] Alternatively, or additionally, the functions described herein can be performed, at least in part, by one or more hardware logic components. For example, but are not limited to, usable hardware logic components may include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0135] Although the diagram shows a single system, it should be understood that the computing device 60 can be a distributed system. This would allow, for example, several devices to communicate via a network connection and collaboratively perform tasks described as being performed by the computing device.

[0136] Although shown as a local device, it should be understood that the computing device 60 may be located remotely and accessed via a network or other communication link (e.g., using a communication interface).

[0137] The term “computer” is used herein to refer to any device that has processing power to the extent that it can execute instructions. Those skilled in the art will recognize that such processing power is incorporated into many different devices, and therefore the term “computer” includes PCs, servers, mobile phones, personal digital assistants, and many other devices.

[0138] Those skilled in the art will recognize that the storage devices for storing program instructions may be distributed across a network. For example, a remote computer may store an example of a process written as software. A local computer or terminal computer may access the remote computer, download some or all of the software, and execute the program. Alternatively, the local computer may download software fragments as needed, or execute some software instructions on the local terminal, or execute some software instructions on the remote computer (or computer network). Those skilled in the art will also recognize that, by utilizing the prior art known to those skilled in the art, all or some of the software instructions may be executed by dedicated circuits such as DSPs, programmable logic arrays, etc.

[0139] It should be understood that the above advantages and benefits may relate to one embodiment or to several embodiments. The embodiments are not limited to those that solve any or all of the aforementioned problems, or that have any or all of the aforementioned advantages and benefits. Variants should be considered to fall within the scope of the invention.

[0140] A reference to "one" item means one or more of those items. The term "contains" is used here to mean including the identified method steps or elements, although these steps or elements do not include an exclusive list, and the method or device may include additional steps or elements.

[0141] As used herein, the terms “component” and “system” are intended to include a computer-readable data store consisting of computer-executable instructions that, when executed by a processor, perform a specific function. Computer-executable instructions may include routines, functions, and the like. It should also be understood that a component or system may reside on a single device or be distributed across multiple devices.

[0142] Furthermore, as used herein, the term “exemplary” is intended to mean “as an example or illustration of something.”

[0143] Furthermore, regarding the extent to which the term "inclusion" is used in the detailed description or claims, since the term "inclusion" is interpreted as a transitional term within the claims, it is intended to have the same inclusiveness as the term "inclusion."

[0144] Furthermore, the operations described herein may include computer-executable instructions that are performed by one or more processors and / or stored on one or more computer-readable media. Computer-executable instructions may include routines, subroutines, programs, execution threads, and the like. Furthermore, the results of operations performed by these methods may be stored in computer-readable media and displayed on a display device and / or on similar devices.

[0145] The order of steps in the methods described herein is illustrative, but these steps may be performed in any suitable order, or, where appropriate, simultaneously. Furthermore, steps may be added to or replaced by any method, or a single step may be removed from any method, without departing from the scope of the subject matter described herein. Aspects of any of the embodiments described above may be combined with aspects of any other embodiments described to form further embodiments without losing the desired effect.

[0146] The above description of preferred embodiments is presented for illustrative purposes only, and it should be understood that various modifications can be made by those skilled in the art. The above description includes examples of one or more embodiments. Of course, for the purpose of describing the above embodiments, it is impossible to describe each possible modification and variation of the above device or method, but those skilled in the art will recognize that many further modifications and arrangements of various embodiments are possible. Accordingly, the embodiments described are intended to include all such variations, modifications, and variations that fall within the scope of the appended claims.

Claims

1. A computer implementation for identifying one or more changes across multiple images using a convolutional neural network (CNN) encoder (220) and a computation network (200) including a ConvLSTM network, The CNN encoder (220) receives CNN input data including data related to each pixel of each of the multiple images (S310), The process involves propagating the CNN input data through the CNN encoder (220) to generate a plurality of feature maps (230), each feature map including the feature classification of each pixel in each of the plurality of images according to a feature classification scheme, the feature classification scheme being generated by the CNN encoder (220) based on training data, and the propagation (S320). (S330) provides a skip connection (240) between the input of the CNN encoder (220) and the input of the convolution unit (250), The skip connection (240) is used to propagate a copy of the CNN input data to the input of the convolution unit (250) (S332), For each of the multiple input images, the convolution unit (250) convolves (S340) the data associated with the input image in the copy of the CNN input data with the respective feature map generated by the CNN encoder, thereby generating the respective convolved feature map, and thereby generating ConvLSTM input data including multiple convolved feature maps. In the ConvLSTM network (260), the ConvLSTM input data is received (S322), and The ConvLSTM input data is propagated via the ConvLSTM network (S350) to generate a change map (270) which includes change data showing one or more changes across the plurality of images. A method that includes generating by [a certain method].

2. The computer implementation method according to claim 1, wherein the change data includes quantitative data indicating the degree of one or more changes across the plurality of images.

3. The computer implementation method according to claim 1 or 2, wherein the change data includes a change classification for each pixel of a selected image of the plurality of images, and for a given pixel of the selected image, the change classification of the pixel indicates whether the feature classification of the pixel is the same as or different from the feature classification of a corresponding pixel of another image of the plurality of images.

4. The computer implementation method according to claim 1, wherein the CNN input data includes amplitude data indicating one or more amplitude values ​​associated with each of the pixels of each of the plurality of images.

5. The computer implementation method according to claim 1, wherein the feature classification scheme is a binary classification scheme configured to classify identified objects as belonging to either a first feature classification or a second feature classification.

6. The computer implementation method according to claim 5, wherein the training data used to train the neural network includes data representing both the first feature classification and the second feature classification, and the data representing the first feature classification in the training data is sparser than the data representing the second feature classification.

7. The feature classification scheme is generated by training the CNN encoder, and training the CNN encoder (220) is The CNN encoder is pre-trained to generate a preliminary map based on the data of the second feature classification and the output of the CNN encoder (S410), Training the computation network by training both the CNN encoder and the ConvLSTM network based on the data of the first feature classification and the output of the ConvLSTM network, wherein the trained CNN encoder is configured to generate the plurality of feature maps according to the feature classification scheme, and the trained ConvLSTM network is configured to generate a change map (260), and the method includes pre-training the trained CNN encoder (S420), The computer implementation method according to claim 6, including the method described in claim 6.

8. The CNN input data and the ConvLSTM input data, one or both of which further include phase data indicating the respective phase values ​​of each pixel in each of the plurality of images, and the feature classification of each pixel in each of the images by their respective feature maps is at least partially based on the phase data and / or The computer implementation method according to claim 1, wherein propagating the ConvLSTM input data via the ConvLSTM network includes convolving the phase data with the plurality of feature maps to generate the change map.

9. The computer implementation method according to claim 1 or 2, wherein each of the plurality of images is an image of a common target taken at different times, and identifying one or more changes across the plurality of images is equivalent to identifying one or more changes of the target over time.

10. The plurality of images include a sequence of images, and the method is The computer implementation method according to claim 1, further comprising: propagating the ConvLSTM input data via the ConvLSTM network (260); convolving the ConvLSTM input data associated with each of the consecutive images with the ConvLSTM input data associated with the preceding image to generate a series of change maps (270), wherein each series of change maps represents a change between one of the plurality of images and a series of images.

11. Each of the aforementioned plurality of images is an image of a geographical region, and the feature classification scheme is, A first feature classification that indicates the presence of a predetermined geographical feature in the pixels classified in this way, A second feature classification indicating that the pixels classified in this way represent the absence of the predetermined geographical feature, The aforementioned method, The further includes identifying regions where the presence / absence of a predetermined geographic feature changes based on the identified changes across the plurality of images, and optionally, The first characteristic classification described above is a forest classification that indicates that the pixels classified in this way represent a forest area. The second characteristic classification described above is a non-forest classification, indicating that the pixels classified in this way represent land that is not covered by forests. The aforementioned method, The method according to claim 1, further comprising identifying changes in the size of the area of ​​deforestation around a forested area based on the identified differences across the plurality of images.

12. The computer implementation method according to claim 1, wherein each of the plurality of images is generated by synthetic aperture radar imaging.

13. A computing system configured to identify one or more changes across a plurality of images by performing the method described in claim 1, wherein the system A convolutional neural network (CNN) encoder (220), wherein the CNN encoder (220) is At the input of the CNN encoder (220), CNN input data including the data associated with the plurality of images is received. The CNN encoder (220) propagates the CNN input data via the CNN encoder (220) (S320) to generate a plurality of feature maps (230), wherein each feature map includes a feature classification of each pixel in each of the plurality of images according to a feature classification scheme, and the feature classification scheme includes a plurality of classifications and is generated by the CNN encoder based on training data, and is configured to propagate, Folding unit (250), ConvLSTM network (260), A first data connection link between the CNN encoder and the convolutional unit (250), and a second data connection link between the convolutional unit (250) and the ConvLSTM network, This includes a skip connection (240) between the input of the CNN encoder and the input of the convolutional unit (250), The aforementioned system processes the ConvLSTM input data. The skip connection (240) is used to propagate a copy of the CNN input data to the input of the convolution unit (250) (S332), and For each of the multiple input images, the convolution unit (250) convolves the data associated with the input image in the copy of the CNN input data with the respective feature map generated by the CNN encoder (S340), thereby generating the respective convolved feature map, and thereby generating ConvLSTM input data including multiple convolved feature maps. Configured to generate, The aforementioned ConvLSTM network (260) is At the input of the ConvLSTM network, the ConvLSTM input data is received via the data connection link (S322), A computing system configured to generate a change map (260) by propagating the ConvLSTM input data via the ConvLSTM network (S350), wherein the change map includes change data indicating one or more changes across the plurality of images.

14. A method for training the calculation system according to claim 13, wherein the method To provide training data (40) that includes data representing the first and second feature classifications, The CNN encoder is pre-trained (S410) based on the second feature classification and the data representing the output of the CNN encoder to generate a preliminary map. A method comprising training both the CNN encoder and the ConvLSTM network based on the data representing the first feature classification and the output of the ConvLSTM network (S420), wherein the trained CNN encoder is configured to generate the plurality of feature maps according to the feature classification scheme, and the trained ConvLSTM network is configured to generate change maps.

15. During the training of the computing system, the weights of the CNN encoder are frozen so that the pre-trained CNN encoder and the trained CNN encoder are composed of the same weights, and / or The method according to claim 14, wherein during the training of the computing system, the weights of the CNN encoder are unfrozen so that the pre-trained CNN encoder and the trained CNN encoder can be composed of different weights.

16. When executed by a computer, the computer A computer-readable medium including instructions for performing the method according to any one of claims 1 to 2, 4 to 8, 10 to 12, 14, or 15.

17. A computer-readable medium that, when executed by a computer, includes instructions causing the computer to perform the method described in Claim 3.

18. A computer-readable medium that, when executed by a computer, includes instructions causing the computer to perform the method described in claim 9.

Images