Image difference recognition

Abstract

A computer-implemented method for identifying one or more changes across multiple images, the method comprising: receiving, at a convolutional neural network (CNN) encoder, CNN input data comprising data associated with each pixel of each of a plurality of images; propagating the CNN input data through the CNN encoder to generate a plurality of feature maps, wherein each feature map comprises a feature classification for each pixel of a corresponding image in the plurality of images according to a feature classification scheme, wherein the feature classification scheme comprises a plurality of classifications and is generated by the CNN encoder based on training data; receiving, at a ConvLSTM network, ConvLSTM input data comprising the plurality of feature maps generated by the CNN encoder; and propagating the ConvLSTM input data through the ConvLSTM network to generate a change map, wherein the change map comprises change data indicative of one or more changes across the plurality of images.

Description

Technical Field

[0001] The present invention relates to identifying differences between multiple images. In one possible implementation, some embodiments of the present invention may be used to identify changes in an object or environment over time by identifying differences between multiple images of the object or environment. Background Art

[0002] In many areas of image analysis, it is desirable not only to analyze images in isolation but also to identify differences between multiple similar or linked images to obtain additional information that cannot be obtained from simply analyzing each image individually. For example, identifying changes in a series of images of the same object captured at different time steps can provide the image owner with information about how the imaged object has changed over time. This can find application in many technical fields, one of which is environmental analysis of imagery generated from data acquired by one or more satellites. In other cases, the same similar images can be collected using a range of different techniques. For example, the images can be captured using imaging signals of different wavelengths, such as optical, radar, microwave, infrared, UV emission, or X-ray images. The differences between these images can provide information about properties of the imaged object, such as density, chemical composition, refractive index, and / or spectral properties such as absorption and / or reflectance coefficients.

[0003] When attempting to identify differences between multiple images, there are two types of resolution to consider. First, one may wish to resolve features within each image, which can be considered analogous to spatial resolution. Additionally or alternatively, one may wish to resolve changes between images, which can be considered analogous to temporal resolution. Simultaneously achieving high spatial and temporal resolution results in very large memory requirements. Consequently, methods for identifying differences between multiple images with high spatiotemporal resolution requirements may place an undue burden on the user's computing resources in terms of both the amount of memory storage required and the amount of processing time required to implement the method.

[0004] The embodiments described below are not limited to implementations that solve any or all disadvantages of the known methods described above. Summary of the Invention

[0005] This summary is provided to introduce in simplified form some concepts that will be further described in the following detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter; variations and alternative features that aid in the operation of the invention and / or are used to achieve substantially similar technical effects should be considered to fall within the scope of the invention.

[0006] The present invention is defined as set forth in the appended claims.

[0007] In general, the present disclosure provides a method for identifying changes across multiple images using a system of convolutional neural network (CNN) encoders coupled with convolutional long short-term memory (ConvLSTM) networks. In this manner, the methods and systems presented herein achieve a method for identifying changes with high resolution within a single image and between different images while reducing memory requirements.

[0008] ConvLSTM networks are well known in the art, and examples are described in Xingjian SHI et al., A Machine Learning Approach for Precipitation Nowcasting, 28 (NIPS 2015) ISBN: 9781510825024, and Elsayed et al., Effects of Different Activation Functions for Unsupervised Convolutional LSTM Spatiotemporal Learning, Advances in Science Technology and Engineering Systems Journal, April 2019.

[0009] In a first aspect, a computer-implemented method for identifying one or more changes across multiple images is provided, the method comprising the following steps: receiving, at a convolutional neural network (CNN) encoder, CNN input data comprising data associated with each pixel of each of a plurality of images; propagating the CNN input data through the CNN encoder to generate a plurality of feature maps, wherein each feature map comprises a feature classification for each pixel of a corresponding image in the plurality of images according to a feature classification scheme, wherein the feature classification scheme is generated by the CNN encoder based on training data; receiving, at a ConvLSTM network, ConvLSTM input data comprising the plurality of feature maps generated by the CNN encoder; propagating the ConvLSTM input data through the ConvLSTM network to generate a change map, wherein the change map comprises change data indicative of one or more changes across the plurality of images.

[0010] In this manner, one or more changes across a plurality of images may be identified based on the change map.

[0011] In another aspect, a computing system is provided that is configured to identify one or more changes across multiple images by implementing any of the methods described herein, the system comprising: a convolutional neural network (CNN) encoder configured to receive CNN input data comprising data associated with multiple images at an input of the CNN encoder, and propagate the CNN input data through the CNN encoder to generate a plurality of feature maps, wherein each feature map comprises a feature classification for each pixel of a corresponding image in the multiple images according to a feature classification scheme, wherein the feature classification scheme comprises a plurality of classifications and is generated by the CNN encoder based on training data; a ConvLSTM network; and a data connection link between the CNN encoder and the ConvLSTM network, wherein the ConvLSTM network is configured to: receive the ConvLSTM input data comprising the plurality of feature maps generated by the CNN encoder at an input of the ConvLSTM network via the data connection link, and propagate the ConvLSTM input data through the ConvLSTM network to generate a change map, wherein the change map comprises change data indicating one or more changes across the multiple images.

[0012] In other words, the computing system may be configured to implement any of the methods described herein.In some embodiments, the computing system may be included in a computing device, for example, as computer-executable instructions stored on a computer-readable medium for execution by a processor.

[0013] In another aspect, a method for training any of the computing networks and / or systems described herein is provided. The method comprises providing training data comprising data representing a first feature classification and a second feature classification, respectively; pre-training a CNN encoder based on the data representing the second feature classification and an output of the CNN encoder to generate a preliminary map; and training both the CNN encoder and the ConvLSTM network based on the data representing the first feature classification and an output of the ConvLSTM network, wherein the trained CNN encoder is configured to generate a plurality of feature maps according to a feature classification scheme, and the trained ConvLSTM network is configured to generate a variation map.

[0014] In another aspect, an apparatus is provided comprising a processor configured to perform any of the methods described herein. The processor may be provided with components of any of the computing networks and / or systems described herein.

[0015] In another aspect, there is provided a computer program product comprising instructions which, when executed by a computer, cause the computer to perform any of the methods described herein.

[0016] In another aspect, a computer-readable medium comprising instructions that, when executed by a computer, causes the computer to perform any of the methods described herein is provided.

[0017] The methods described herein may be performed by software in a machine-readable form on a tangible storage medium, for example in the form of a computer program comprising computer program code means adapted to perform all the steps of any of the methods described herein when the program is run on a computer, and wherein the computer program may be embodied on a computer-readable medium. Examples of tangible (or non-transitory) storage media include disks, thumb drives, memory cards, and the like, and do not include propagating signals. The software may be suitable for execution on a parallel processor or a serial processor, such that the method steps may be performed in any suitable order or simultaneously.

[0018] This application recognizes that firmware and software can be valuable, separately tradable commodities. It is intended to cover software that runs on or controls "dumb" or standard hardware to perform a desired function. It is also intended to cover software that "describes" or defines the configuration of hardware, such as HDL (Hardware Description Language) software used to design silicon chips, or to configure general-purpose programmable chips to perform a desired function.

[0019] The features and embodiments discussed herein may be appropriately combined, as will be apparent to those skilled in the art, and may be combined with any aspect unless such a combination is explicitly stated as not possible or will be understood by those skilled in the art to be self-evidently impossible. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Embodiments of the present invention will now be described by way of example with reference to the following drawings.

[0021] Figure 1 Schematic diagram depicting a satellite collecting SAR imagery data in orbit around the Earth.

[0022] Figure 2a Depicted is a schematic diagram of a computing system configured to identify differences across multiple images, according to some embodiments of the present invention.

[0023] Figure 2b Depicts Figure 2a Schematic diagram of the operation of the CNN encoder of the computing system.

[0024] Figure 2c Depicts Figure 2a Schematic diagram of the operation of the ConvLSTM network of the computing system.

[0025] Figure 3 Methods of identifying differences across multiple images according to some embodiments of the present invention are described.

[0026] Figure 4 Describes training according to some embodiments of the present invention Figure 2a Methods of computing systems.

[0027] Figure 5 Depicted are results of an example of identifying deforestation areas in a forest environment imaged by SAR, according to some embodiments of the present invention.

[0028] Figure 6 A computer configured to perform the method of the claimed invention is depicted.

[0029] The same reference numbers are used throughout the drawings to identify the same or similar features. DETAILED DESCRIPTION

[0030] The following description of embodiments of the present invention is by way of example only. These examples represent the best modes currently known to the applicant for putting the present invention into practice, although they are not the only ways to implement the present invention. This description sets forth the functions of the examples and the sequence of steps for constructing and operating the examples. However, the same or equivalent functions and sequences may be implemented by different examples.

[0031] The methods and systems described herein provide a means of identifying changes across multiple images with minimal loss of resolution—either within each image or between images. For example, by generating a feature map for each image using a CNN encoder, the memory requirements of the method are greatly reduced while minimizing loss of resolution within each image. This is because the CNN encoder is able to reduce image data, which may include several data values ​​per pixel, to a feature map. Such a feature map encodes a single value to each pixel, where the value encoded to each pixel represents a classification based on the characteristics of the pixel as determined by the feature map.

[0032] In some embodiments, the change data encoded in the change includes quantitative data indicating the extent of one or more changes across multiple images. As described below, this allows the user to determine not only the presence of a change, but also the extent of the change across multiple images. Due to the simplification of the image data by the CNN encoder, this quantitative change analysis can be performed with relatively modest memory and processing requirements.

[0033] In some embodiments, the change data includes a change classification for each pixel of a selected image from the plurality of images, wherein, for a given pixel of the selected image, the change classification of the pixel indicates whether the feature classification of the pixel is the same as or different from the feature classification of a corresponding pixel in another of the plurality of images. In some embodiments, the change classification is a binary classification, i.e., a classification of whether a change has occurred. By encoding the change map generated by the ConvLSTM according to a change classification scheme, the memory requirements of the ConvLSTM can be further reduced. Importantly, in contrast to methods implementing traditional neural networks, the methods and systems described herein focus on detecting whether a change has occurred across the plurality of images and do not rely on attempting to visualize and display the precise evolution of the changes across the images, although such changes can be displayed. Furthermore, generating a change map simplifies the process of change identification analysis. Instead of generating a data stack detailing extensive information associated with each pixel of each image, the methods described herein can simply generate a dataset that identifies, for each pixel of one of the images, whether a change in the feature classification of each pixel has occurred across the plurality of images, and in some embodiments, identifies the extent of the change.

[0034] In fact, by reducing the problem of change identification from a quantitative problem (i.e., how much change is there between images) to a qualitative problem (whether a change has occurred across multiple images), the memory requirements of the method are greatly reduced, thereby allowing a computer implementing the method to maintain high resolution in the multiple feature maps generated by the CNN encoder and the change map generated by the ConvLSTM. The method described herein takes advantage of the memory saved from the generation of feature maps and the change classification scheme of the change map to utilize a method that provides high resolution at low memory and processing requirements. This would not be achievable without the specific configuration of combining the CNN encoder with the ConvLSTM network as provided in the methods and systems described herein.

[0035] Figure 1 A schematic diagram depicts such a satellite 10 collecting SAR image data in orbit around the Earth 12. The image data analyzed according to the methods described herein may be SAR image data collected by a satellite 10 in low Earth orbit (LOE) around the Earth.

[0036] Satellite 10 can image a target, such as an area on Earth 12, such as an Arctic or Antarctic environment, a forested environment, an urban environment, or any other landscape of interest. Satellite 10 is in a repeating orbit and images the same target environment at least once per orbital period. As satellite 10 continuously images the target during each pass of its orbit around Earth 12, it can add each successive image to a stack of multiple images. Identifying changes between images, as described below, can be applied each time a successive image is added to the image stack to generate a continuous change map, thereby tracking the evolution of the landscape of interest. In some examples, this can be represented in a video or series of images depicting the continuous change map and its evolution over time.

[0037] As described above, in some embodiments, each of the multiple images is generated by synthetic aperture radar imaging (SAR). SAR is particularly suitable for satellite-based imaging systems because it can "see through" opaque atmospheric structures, such as clouds and smoke from fires, to directly image the surface of the earth 12.

[0038] The methods described herein are applicable to a wide range of images. In particular, these methods have particularly advantageous application in the context of SAR imaging. SAR can be used to create two-dimensional images and / or three-dimensional reconstructions of imaged objects (such as landscapes).

[0039] As described above, in some embodiments, each of the plurality of images is generated from data acquired by one or more satellites.

[0040] Satellite imagery provides an opportunity to collect image data on a variety of targets. For example, images generated from satellite-acquired data, such as satellite SAR imagery, can be used to image forested areas, urban environments, Arctic or Antarctic environments, or other landscapes. By applying the methods described herein to images generated from satellite-acquired data, users of the methods can readily identify the extent of environmental change in the imaged landscape. This can include, for example, identifying and detecting deforestation / reforestation, urbanization / counter-urbanization, landslides, ice growth / shrinkage, and / or other environmental changes.

[0041] As described above, in some embodiments, each image generated from satellite-acquired data is generated by a satellite in low Earth orbit.

[0042] Satellites in low-Earth orbit (LEO) can have short orbital periods of 1 hour or less, 90 minutes or less, 2 hours or less, 4 hours or less, 6 hours or less, 12 hours or less, 18 hours or less, or 1 day or less. LEOs can range from 160 kilometers to 1000 kilometers above the Earth's surface. Thus, an example of a SAR-based Earth-based observation satellite can have an orbit between 450 kilometers and 650 kilometers above the Earth. In one example, satellite 10 can have an orbit at 550 kilometers above the Earth's surface. For example, in an orbit 550 kilometers above the Earth, the satellite actually travels across the Earth at approximately 7.5 kilometers per second, or 27,000 kilometers per hour. Most satellites in this orbit will travel across the Earth at speeds in the range of 7 to 8 kilometers per second. Applying the methods described herein to images generated from data acquired by one or more satellites in LEO provides users of the methods with a means for achieving high temporal resolution of changes in imaged objects in addition to high spatial resolution. This provides users with more accurate detection and / or identification of spatiotemporal changes in imaged objects.

[0043] In some embodiments, each of the plurality of images is an image of a common target captured at a respective different time, such that identifying one or more differences across the plurality of images is equivalent to identifying changes in the object over time.

[0044] In this way, the resolution of features within a single image in a plurality of images can be considered as spatial resolution, and the resolution of changes across the plurality of images can be considered as temporal resolution. The methods described herein allow for the detection of temporal changes in an imaged object, wherein high spatiotemporal resolution is achieved without unduly burdening a computer implementing the claimed methods in terms of memory or processing requirements. This enables users of any of the methods described herein to detect and identify changes in an imaged object with good spatial and temporal resolution, even in devices, networks, or systems with severe memory, storage, or processing limitations.

[0045] In some embodiments, each of the plurality of images is coherent with each of the other images.Two images that are coherent with each other are collected by respective imaging signals having a fixed relationship.

[0046] As discussed in more detail below, phase information, and in particular relative phase variations between images, can indicate additional information beyond that encoded in the corresponding preliminary or feature maps of each image. If the images are coherent, the information encoded in the phase variations between images is suitable for enhanced coherence analysis, including coherent change detection (CCD), digital elevation model (DEM) generation, or differential interferometric synthetic aperture radar imaging (InSAR).

[0047] CCDs detect changes between coherent images that are unlikely to be visible to the human eye. This is because the sensitivity of a CCD is only a fraction of the wavelength of the light used to collect the image. For example, in the case of radar imaging, a CCD can resolve centimeter-scale changes in imagery collected by satellites. In the case of synthetic aperture radar (SAR) imaging, CCDs provide users with the ability to see minute differences between two SAR images at resolutions exceeding those possible with "naked eye" analysis.

[0048] A DEM exploits subtle differences in position between two coherent images. The phase information associated with each pixel in each coherent image is then compared, and changes relative to a reference plane are highlighted. In other words, in SAR imaging, phase information can be used to infer the height of features in the image relative to a reference "zero" altitude. DEM generation allows this height data to be derived from the phase change information via phase unwrapping to form a three-dimensional digital elevation model of the area.

[0049] InSAR can be considered a combination of the CCD and DEM techniques described above. InSAR is particularly useful for detecting very subtle changes in elevation over time. Images generated from data acquired by one or more satellites can be analyzed with InSAR to detect millimeter-scale changes in the environment over a period of one month. This can be used to identify a range of hazards and emerging conditions, from landslides to infrastructure failures, such as the collapse of dams or bridges.

[0050] In some embodiments, each of the plurality of images may be an image of an area of ​​10 square kilometers or greater, 50 square kilometers or greater, 100 square kilometers or greater, 1,000 square kilometers or greater, 5,000 square kilometers or greater, or 10,000 square kilometers or greater.

[0051] For example, each of the images may be 5 km x 5 km in area or larger, 10 km x 10 km in area or larger, 50 km x 50 km in area or larger, or 100 km x 100 km in area or larger.

[0052] Even for such large area object sizes, the methods described herein allow for differential spatial resolution across multiple images representing features having sizes of 0.1 meters or less, 0.5 meters or less, 1 meter or less, 5 meters or less, 10 meters or less, or 50 meters or less.

[0053] The methods and systems described herein, such as when applied to data collected by satellites, particularly SAR images, are particularly beneficial when applied to images affected by speckle patterns. The speckle pattern is generated by the interference of a series of coherent wavefronts of an imaging signal. This physical phenomenon occurs when a coherent imaging signal reflects from an imaging object. Each of the signal's many reflection points (based on diffraction theory) acts as a source of spherical waves. The multiple spherical waves returning to the imaging device (e.g., satellite 10) interfere with each other to produce a characteristic "blurred" speckle pattern.

[0054] Analyzing images affected by speckle patterns can be very difficult because such images appear "noisy" to traditional algorithms used to identify objects contained in the image. In effect, speckle patterns cause significant variability in the data associated with adjacent pixels in the image, so the information content of a single pixel is very limited. As described below, the methods and systems disclosed herein provide a means of mitigating the problems introduced by such speckle patterns.

[0055] Figures 2a to 2c A schematic diagram depicting the operation of a computing system 200 and its components, which may be configured to identify differences across multiple images.

[0056] Figure 2a A schematic diagram of a computing system 200 configured to identify differences across a plurality of images 210 is depicted.

[0057] The data associated with the plurality of images 210 is input to the CNN encoder 220 and forms CNN input data. The convolutional neural network of the CNN encoder may include a plurality of convolutional layers, such that when the data associated with the plurality of images 210 is transmitted through the CNN encoder 220, the data associated with each of the plurality of images 210 is convolved to generate a plurality of feature maps 230. Each feature map is associated with one of the plurality of images 210.

[0058] In some embodiments, the data associated with each of the plurality of images 210 includes amplitude data indicating one or more amplitudes respectively associated with each pixel of each of the plurality of images 210. The amplitude data may, for example, include data values ​​indicating an amplitude associated with each of the R, G, and B channels of an RGB image; or indicating an amplitude associated with each of the C, M, Y, and K channels of a CMYK channel; or indicating grayscale values ​​in a grayscale image.

[0059] In other words, the amplitude data can be encoded as RGB values. That is, there can be data values ​​associated with each pixel that indicate each of the R, G, and B values, and optionally indicate brightness. Additionally or alternatively, the amplitude data can be encoded as CMYK values. In other words, there can be data values ​​associated with each pixel that indicate each of the C, M, Y, and K values, and optionally indicate brightness. Additionally or alternatively, the amplitude can be indicated based on grayscale. In other words, there can be data values ​​that indicate the shading of the pixel along the grayscale. Such data values ​​can be encoded as 8-bit integers or in another appropriate format.

[0060] In some embodiments, the CNN input data also includes phase data indicating a corresponding phase value for each pixel of each image of the plurality of images 210, and feature classification of each pixel of each image via its corresponding feature map 230 (as discussed in more detail below) is based at least in part on the phase data.

[0061] When analyzing the differences between two or more images, the relative phase changes between the images can indicate additional information beyond that encoded in the corresponding feature maps of each image. For example, in some instances, the phase value associated with each pixel of an image can indicate the phase of the signal used to collect / generate the image at the location corresponding to that pixel. In reflection-based imaging, the phase value can thus indicate the distance between the pixel's imaging detector and the target surface from which the signal reflected to generate the image. Meanwhile, in transmission-based imaging, the phase value can indicate the density of the target through which the signal was transmitted.

[0062] In other words, the phase data may indicate an optical path length of an imaging signal used to generate at least one of the plurality of images 210. Figure 1 In the case of the SAR satellite system 10 , the phase data indicates the relative height of the imaged target above the surface of the Earth 12 .

[0063] CNN encoder 220 is trained to generate feature maps 230 from data associated with each of the plurality of images 210. Each feature map 230 includes a feature classification for each pixel of a corresponding image of the plurality of images 210 according to a feature classification scheme. The feature classification scheme is generated by CNN encoder 220 based on its training using training data. Further discussion of the training of CNN encoder 220 can be found below with respect to Figure 4 turn up.

[0064] In some embodiments, the feature classification scheme is a binary classification scheme configured to classify identified objects as belonging to a first feature classification or a second feature classification.

[0065] By simplifying the feature classification scheme to a binary classification scheme, the memory requirements of the method described herein are reduced. In particular, the binary classification scheme allows each feature map 230 to be encoded using a series of one-hot encoding processes or similar processes.

[0066] In some embodiments, the feature classification scheme includes a feature classification of forested versus non-forested land, a feature classification of snowy versus non-snowy terrain, a feature classification of inhabited settlements versus uninhabited terrain, a feature classification of buildings versus non-buildings in urban environments, a feature classification of land versus water, and / or any other suitable set of feature classifications. This can enable a user of computing system 200 to detect, identify, and / or measure various processes, such as deforestation / reforestation, ice (floe) growth / shrinkage, nomadic migration, (de)urbanization, coastal erosion / development, and / or any other process that can be detected, identified, and / or measured based on a suitable feature classification scheme.

[0067] In other words, in some embodiments, the multiple feature classifications of the feature classification scheme include: a forest classification indicating that the pixels so classified represent forest land, and a non-forest classification indicating that the pixels so classified represent non-forest land, and the method also includes: identifying deforestation areas surrounding the forest land based on the identified differences across the multiple images.

[0068] In this way, it is possible to track deforestation or reforestation in areas of interest to users of the method. These areas may include rainforests, such as the Amazon rainforest, where logging operations and other deforestation projects have been ongoing. These areas may also include other forest areas that have been subject to deforestation. Deforestation may include artificial deforestation, such as logging projects (authorized or unauthorized), and / or may include natural means of deforestation, such as fire. For example, the methods described herein for detecting and identifying deforestation areas may track deforestation caused by wildfires, such as in forests in California, Australia, Canada, etc.

[0069] Furthermore, in some embodiments, the plurality of feature classifications of the feature classification scheme include an ice classification indicating that pixels so classified represent areas bounded by or covered by ice, and a non-ice classification indicating that pixels so classified represent areas free of ice, and the method further includes identifying areas of ice expansion or retreat over time in an Arctic or Antarctic environment based on the identified differences across the plurality of images.

[0070] In this way, the shrinkage or growth of ice floes and / or glaciers in the Arctic and / or Antarctic circles can be tracked. This can be used, for example, to provide information about the state of polar ice caps and other icy environments to governments, non-governmental organizations, climate scientists, and / or other users as a means of tracking the effects of climate change.

[0071] In some embodiments, the plurality of feature classifications of the feature classification scheme include: a building classification indicating that pixels so classified represent buildings, and a non-building classification indicating that pixels so classified represent features that are not buildings, and the method further includes: identifying construction and / or demolition sites within an urban environment over time based on the identified differences across the plurality of images.

[0072] In this way, the development or destruction of urban environments can be tracked. This can provide users of the methods described herein with a means of analyzing urban expansion / sprawl. In other words, users of the methods described herein applied in this context can detect and identify urbanization and / or counter-urbanization in areas of interest.

[0073] In some embodiments, the multiple feature classifications of the feature classification scheme include: a residential classification indicating that pixels so classified represent residential locations within the imaging area, and a non-residential classification indicating that pixels so classified represent non-residential locations within the imaging area, and the method further includes: identifying the construction and / or removal of residential locations within an uninhabited or partially inhabited environment over time based on the identified differences across the multiple images.

[0074] In this way, the movements of nomadic or otherwise mobile groups of individuals can be tracked by determining when and where their settlements or camps were erected and dismantled.

[0075] In some embodiments, the multiple feature classifications of the feature classification scheme include: a land classification indicating that the pixels so classified represent land, and a water classification indicating that the pixels so classified represent water bodies, and the method also includes: identifying coastal erosion of land by water bodies, etc. based on the identified differences on multiple images.

[0076] In this way, the progression of coastal erosion in the region of interest may be tracked. Additionally or alternatively, the effectiveness of coastal defences in slowing the progression of coastal erosion may be determined by determining the extent to which coastal erosion is slowed after coastal defences are employed in the region of interest.

[0077] In addition to those outlined above, the above-described methods can be applied to a variety of other situations. For example, methods applied to spatiotemporal resolution can be used to detect and identify landslide areas in hills, cliffs, mountains, and / or other areas susceptible to landslides and / or avalanches. This can be based at least in part on phase changes between images, which indicate that the height of the ground has shifted. In an alternative example, the method can be applied to a situation where each of multiple images is taken at the same time of the same object, but each image is imaged by an imaging signal of a different wavelength. These imaging signals can be, for example, but not limited to, optical signals, microwave signals, radio signals, infrared signals, ultraviolet signals, and / or x-ray signals. Multiple images generated by imaging signals of different wavelengths can be analyzed according to the methods described herein to identify and / or detect spatial and spectral or spatio-spectral variations across the multiple images. Spatio-spectral difference identification and / or detection can be used in many situations. For example, this method can be applied to identify coronal mass ejections from the sun or other solar events.

[0078] In some embodiments, computing system 200 further includes skip connections 240. Skip connections provide a propagation path for copies of data associated with multiple images so that the CNN encoder 220 for the copies can be effectively skipped.

[0079] In the operation of the CNN encoder 220, the CNN is trained so that providing successive layers increases the number of operations performed by the neural network, and thus the CNN is able to derive additional information from each of the multiple images. In some cases, if the training is ineffective, this may lead to errors in the feature map. By providing skip connections and convolving the data associated with each of the multiple images with its corresponding feature map, any errors introduced by the operation of the CNN encoder 220 can be mitigated. This improves the reliability and accuracy of the final change map generated by the ConvLSTM network 260.

[0080] The output of CNN encoder 220, namely, each of the plurality of feature maps 230, is convolved with data associated with each of the plurality of images 210 via convolution unit 250. Convolution unit 250 convolves the data associated with each of the plurality of images 210 with its corresponding feature map 230 to generate ConvLSTM input data, i.e., data to be input into ConvLSTM network 260. ConvLSTM input data can be considered as a plurality of convolutional maps. ConvLSTM network 260 is a recurrent neural network configured to convolve the ConvLSTM input data to generate a single variation map 270. In other words, ConvLSTM network 260 convolves each of the plurality of convolutional maps with one another to generate variation map 270. Furthermore, ConvLSTM network 260 convolves the data within each corresponding map of the ConvLSTM input data. In other words, ConvLSTM network 260 is configured to perform convolutions both within and between maps. For example, in an instance where each of the plurality of images 210 represents an image of a common target imaged at different times, the ConvLSTM network is configured to convolve data in space and time. Variation map 270 may include a variation classification for each pixel of a selected one of the plurality of images 210. This variation classification for each pixel may indicate whether the feature map classification of the pixel is the same as or different from the feature map classification of a corresponding pixel of another of the plurality of images 210. Additionally or alternatively, variation map 270 may include quantitative variation data indicating the extent of one or more variations across the plurality of images. For example, in the case of SAR imaging, the quantitative variation data may indicate the extent to which the relative height of each imaging pixel varies across the plurality of images 210.

[0081] In some embodiments, the variation classification is a binary classification. As described above, binary classifications can be particularly effective because they allow each pixel to be encoded with its corresponding variation classification via one-hot encoding or similar fast processing.

[0082] In some embodiments, the ConvLSTM input data also includes phase data indicating a corresponding phase value for each pixel of each image of the plurality of images 210 , and propagating the ConvLSTM input data through the ConvLSTM network 260 includes convolving the phase data with the plurality of feature maps 230 to generate a variation map 270 .

[0083] As described above, phase information, and in particular, relative phase changes between images, can indicate information other than that encoded in the corresponding feature maps of each image. By convolving the phase data with the plurality of feature maps 230 to generate the change map 270, the phase information can be used in determining the change data associated with each pixel of the change map 270. The change data can optionally include a binary change classification associated with each pixel of the change map 270 and / or quantitative change data associated with each pixel of the change map 270. This increases the sensitivity of the ConvLSTM network 260, allowing the ConvLSTM network 260 to detect and identify changes across the plurality of images 210 that are only or partially encoded in the phase data.

[0084] Figure 2b A schematic diagram depicting the operation of CNN encoder 220 of computing system 200 is shown.

[0085] For a given image 212 in the plurality of images 210, image data associated with the image 212 is provided as CNN input data to the CNN encoder 220. As described above, the CNN input data is propagated through the layers of the CNN encoder 220 to generate a feature map 232. Figure 2b In the example shown, there are nine data values ​​associated with each pixel of image 212. CNN encoder 220 convolves the data associated with image 212 to generate feature map 232 having only one data value associated with each pixel. Figure 2b In the example shown, the data value associated with each pixel is part of a binary feature classification scheme. For example, in the context of a method for detecting deforestation of a forest, the binary feature classification scheme may be such that a value of "1" in feature map 232 indicates that the corresponding pixel of image 212 represents forested land, while a value of "0" in feature map 232 indicates that the corresponding pixel of image 212 represents non-forested land.

[0086] CNN encoder referenced above Figure 2b Operation in the manner described provides a means for reducing the data content that must be processed to detect and identify changes across the plurality of images, with minimal loss of (spatial) resolution within each image 212 of the plurality of images 210 .

[0087] Figure 2c A schematic diagram depicting the operation of the ConvLSTM network 260 of the computing system 200 is shown.

[0088] The convolution unit 250 convolves the data associated with each of the plurality of images 210 with its corresponding feature map 230 to generate a ConvLSTM network 260. For example, Figure 2bThe image 212 is convolved with the feature map 232 to generate a corresponding convolution map. Each convolution map is then cyclically convolved by propagation through the ConvLSTM network 260 to generate a change map 270. As described above, in some embodiments, the change map 270 may include a binary change classification for each pixel of a selected image in the plurality of images 210 to enable identification, detection, and / or measurement of differences between the plurality of images 210.

[0089] Additionally or alternatively, in some embodiments, variation map 270 may include quantitative variation data associated with each pixel of the selected image, the quantitative variation data indicating a degree of variation between the selected image and another one or more of plurality of images 210 .

[0090] The computing network system described herein is configured to implement a method that can identify changes on multiple images with minimal resolution loss - the resolution within each image, or the resolution of changes between images. For example, by generating a feature map 230 for each image through the CNN encoder 220, the memory requirements of the system are greatly reduced, while the resolution loss within each image is minimal. This is because the CNN encoder is configured to simplify image data that can include several data values ​​per pixel into a feature map. This feature map encodes a single value to each pixel, wherein the value encoded to each pixel represents a classification based on the features of the pixel by the feature map. In addition, the ConvLSTM network 260 can be configured to encode the change map with quantitative change data indicating the degree of one or more changes on multiple images. Due to the simplification of image data by the CNN encoder 220, this quantitative analysis can be performed with relatively moderate memory and processing requirements.

[0091] Additionally or alternatively, the change map 270 can be encoded according to an optional binary change classification scheme. In this way, the memory requirements of the ConvLSTM network 260 can be further reduced. Importantly, in contrast to conventional computing systems and neural networks, the computing system described herein is configured to detect whether a change exists across multiple images and does not rely on the precise evolution of the changes being visualized and displayed across the images, although such evolution can certainly be displayed in some instances. Furthermore, generating the change map 270 simplifies the process of change identification analysis. Instead of generating a data stack detailing information associated with each pixel of each image, the computing system described herein is simply configured to generate a single data set for each change map 270 that identifies, for each pixel of one of the images, whether a change in a feature classification for each pixel has occurred across the multiple images 210.

[0092] As described above, by reducing the problem of change identification from a quantitative problem to a qualitative problem, the memory and processing power required by the system described herein to implement the method described herein can be greatly reduced without compromising the resolution of change detection and / or identification that can be achieved by the system. This would not be achievable without the specific configuration of combining the CNN encoder 220 with the ConvLSTM network 260 as provided in the computational network described herein.

[0093] Furthermore, as mentioned above, the system described with respect to FIG. 2 and the methods described herein are particularly advantageous for analyzing images affected by speckle patterns, such as SAR images generated from data collected by satellites 10 in low-Earth orbit. Due to the large variation in data associated with neighboring pixels affected by speckle, a large amount of information may need to be obtained for each pixel. This contextual information can be spatial or temporal. For example, temporal contextual information can be obtained by repeatedly collecting image data on the same target at a high frequency. This can be difficult in the case of satellites in low-Earth orbit, as satellites 10 can only image targets on Earth 12 as they pass overhead. An alternative to temporal contextual information is spatial contextual information. Spatial contextual information can be obtained by capturing a field of view in each image that is significantly larger than the area of ​​the speckle pattern or patterns affecting the image. For example, the area of ​​the field of view may be 5 times or more, 10 times or more, 50 times or more, or 100 times or more larger than the area of ​​the speckle pattern or patterns affecting the image. Due to the perceptual computational cost of analyzing such a large field of view, such a field of view is much larger than that used in typical object recognition algorithms.

[0094] However, the CNN encoder 220 enables the computing system 200 to capture and process a large field of view in a memory-efficient manner. In the context of a SAR imaging system, the CNN encoder 220 can compress a large area image 210 into a smaller feature map 230. For example, each of a plurality of images may have an area of ​​512x512 pixels, which is compressed by the CNN encoder 220 into a corresponding feature map 230 having an area of ​​64x64 pixels. This represents an area compression factor of 64. One of ordinary skill in the art will appreciate that other image sizes and other area compression ratios are possible. However, for context, comparable neural networks in the prior art typically process images having a smaller area (e.g., an area of ​​299x299 pixels). This smaller area may not provide the computing network 200 with a wide enough field of view to mitigate speckle effects in high-resolution SAR images. In other words, the CNN encoder 220 is configured to process images that include three times or more the number of pixels of comparable neural networks employed in the prior art. This large field of view allows the CNN encoder 220 to generate feature maps that are robust to the deleterious effects of speckle patterns that affect traditional object recognition algorithms.

[0095] Figure 3 Methods of identifying differences across multiple images according to some embodiments of the present invention are described.

[0096] In operation S300 , the CNN encoder 220 receives CNN input data including data associated with a plurality of images 210 .

[0097] In operation S320, the CNN input data is propagated through the CNN encoder 220 to generate a plurality of feature maps 230. Each of the plurality of feature maps 230 includes a feature classification for each pixel of its corresponding image in the plurality of images 210 according to a feature classification scheme generated by the CNN encoder based on the training data.

[0098] In some embodiments, propagating the CNN input data through the CNN encoder 220 to generate the plurality of feature maps 230 includes compressing the CNN input data.

[0099] In this manner, the memory storage requirements of a computer implementing any of the methods described herein are reduced without compromising resolution within each image. For a given image in the plurality of images 210, feature map 230 may include a feature classification for each pixel of the image. In contrast, the raw image data may include multiple data values ​​for each pixel of the image. For example, in a traditional RGB image, there may be a data value associated with each pixel indicating each of the R, G, and B values, and optionally, brightness. Alternatively, for a CMYK image, there may be a data value associated with each pixel indicating each of the C, M, Y, and K values, and optionally, brightness. In some images, there may also be a data value indicating the image's saturation. In some images, there may also be a data value indicating the corresponding phase value associated with each pixel of the image. By simplifying the representation of each image to a feature map, the memory requirements of a computer implementing any of the methods described herein are significantly reduced, thereby allowing the methods to be implemented on a wider variety of computer-based systems, particularly those with stringent and demanding memory requirements on computing devices, systems, and / or networks.

[0100] In some examples, the data associated with each of the multiple images can be encoded as complex data. In other words, the data can be encoded as one or more complex numbers. Each complex number can be considered to have a magnitude and an argument. The magnitude of the complex number can represent an amplitude value, while the argument of the complex number can represent a phase value.

[0101] In some instances, the data associated with each of the multiple images can include contextual data. This can include any type of metadata or other type of image data. For example, the metadata can include a timestamp. This can be beneficial because it can allow the methods and systems disclosed herein to be sensitive to changes in the imagery caused by time of day (e.g., images of the same target captured during the day and at night may appear different) or seasonal variations (e.g., images of the same target may have different textural properties—they may be drier in the summer months, wetter in the spring months, etc.). The metadata can include information indicating the local angle of incidence of the imaging signal. This can be beneficial, particularly in the context of satellite SAR, where the local angle of incidence of the same imaged target may vary slightly between different images if the orbital period of the satellite 10 is not perfectly regular. In the context of images of geographic features, the metadata / image data can include weather data, such as rain gauge maps. This can be particularly beneficial when applied to SAR imagery, as the moisture content of the imaged target can have a significant impact on the resulting SAR image.

[0102] In operation S322 , the ConvLSTM network 260 receives a plurality of feature maps generated by the CNN encoder 220 .

[0103] In some embodiments, in operation S330 , a skip connection 240 is provided between the input of the CNN encoder 220 and the input of the ConvLSTM network 260 .

[0104] In such an embodiment, in operation S332 , a copy of the CNN input data is propagated to the input of the ConvLSTM network 260 via the skip connection 240 .

[0105] Furthermore, in operation S340, the data associated with each of the plurality of images 210 in the copy of the CNN input data is convolved with its corresponding feature map 230 (generated by the CNN encoder 220) to generate ConvLSTM data. For example, this operation can be performed by the convolution unit 250.

[0106] In other words, in some embodiments, the method further includes: providing a skip connection between the input of the CNN encoder and the input of the ConvLSTM network; propagating a copy of the CNN input data to the input of the ConvLSTM network through the skip connection; and convolving each of the multiple images in the copy of the CNN input data with its corresponding feature map generated by the CNN encoder to generate the ConvLSTM input.

[0107] In operation S350, the ConvLSTM input data is propagated through the ConvLSTM network 260 to generate a variation map 270. As described above, the variation map 270 may include an optional binary variation classification for each pixel of a selected image in the plurality of images 210, the variation classification of the pixel indicating whether a feature classification of the pixel is the same as or different from a feature classification of a corresponding pixel in another one of the plurality of images 210. Additionally or alternatively, the variation map 270 may include quantitative variation data indicating the extent of one or more variations across the plurality of images 210.

[0108] Finally, in operation S360 , one or more changes across the plurality of images are identified based on the change map 270 .

[0109] As mentioned above, Figure 3 The methods described can be applied in a variety of contexts. For example, the methods can be used to identify the extent and progression of deforestation processes, ice (floe) growth / shrinkage, urbanization / counter-urbanization, coastal erosion / development, nomadic migration, solar activity, and / or any other process that can be appropriately identified using the methods described herein.

[0110] In summary, each of the plurality of images 210 may be an image of a geographic area, and the feature classification scheme may include: a first feature classification indicating that the pixels so classified represent the presence of a predetermined geographic feature; and a second feature classification indicating that the pixels so classified represent the absence of the predetermined geographic feature, wherein the method further includes: identifying an area where the presence / absence of the predetermined geographic feature changes based on the identified differences across the plurality of images.

[0111] Figure 4 Describes the training of some methods according to the present invention such as Figure 2a The method of the computing system 200 is shown.

[0112] To train the computing system 200, and in particular the CNN encoder 220, the CNN encoder may receive various inputs. The first input is training data 40. The training data 40 may be divided into two categories 42, 44. The first category is data representing a first feature classification, while the second category is data representing a second feature classification.

[0113] In some embodiments, the first category 42 is data representing a first characteristic classification and is relatively scarce compared to the second category 44 representing a second characteristic classification that is relatively more prevalent. For example, in the context of a method for identifying deforestation, the first category 42 may represent deforested land, while the second category 44 may represent forested land, noting that in most cases of deforestation, the process of identifying deforestation will require detecting relatively small extensions of deforestation compared to large areas of deforested forest.

[0114] In other words, in some embodiments, the training data used to train the CNN encoder 220 includes data representing a first feature class and a second feature class, and the data representing the first feature class is scarce in the training data relative to the data representing the second feature class. For example, the ratio of the amount of data representing the first class to the amount of data representing the second class in the training data can be 1:5 or less, 1:10 or less, 1:50 or less, 1:100 or less, 1:500 or less, or 1:1000 or less.

[0115] The method described herein differs from conventional computing methods in that it is specifically directed to detecting and identifying changes across multiple images 210, rather than identifying specific or particular objects within the images. This means that in some cases, such as images of forested land where detected changes indicate deforestation, a CNN encoder 220 can be trained to detect changes across multiple images 210 using training data 40 where data indicating deforestation is scarce relative to data indicating forested land. Those skilled in the art will recognize that this principle is equally applicable to many forms of image analysis directed to detecting changes in a highly common feature (or features) of an image, where the change in the image is toward a relatively less common feature (or features) of the image.

[0116] The second input to the training method is the output of the CNN encoder 46.

[0117] In some embodiments, as shown in operation S410, the CNN encoder 220 is pre-trained based on data representing the second more general feature classification 44 and the CNN encoder output 46 to generate a preliminary graph. This provides a rough pre-training for the CNN encoder 220 so that the memory requirements of the fine-tuning training for subsequent operations can be achieved with less computing resources. The preliminary graph generated by the pre-trained CNN encoder 220 can be any suitable mapping. For example, the preliminary graph can be a semantic graph or a regression graph.

[0118] The third input to the training method is the output of the ConvLSTM network 48.

[0119] In operation S420, the overall model is refined by training both the CNN encoder 220 and the ConvLSTM network 260 based on data representing the first, more scarce feature classification and the output of the ConvLSTM network 48. At the end of operation S420, the fully trained CNN encoder 220 is configured to generate feature maps 230 based on the feature classification scheme learned by the CNN encoder 220 through the training process of operations S410 and S420. In addition, at the end of operation S420, the fully trained ConvLSTM network 260 is configured to generate a variation map. By implementing Figure 4 In the training method shown, the CNN encoder is trained to learn features of an image that indicate a first feature classification and a second feature classification without requiring a large amount of data representing the first, relatively scarce feature classification 42. The result of this training is a feature classification scheme based on the training data, and the feature classification included in each feature map 230 generated by the CNN encoder 220 is based on this feature classification scheme.

[0120] In other words, in some embodiments, a feature classification scheme is generated by training the CNN encoder 220, wherein training the CNN encoder 220 includes: pre-training the CNN encoder 220 based on data representing the second feature classification 44 and the output of the CNN encoder 46 to generate a preliminary graph; and training both the CNN encoder 220 and the ConvLSTM network 260 based on data representing the first feature classification 42 and the output of the ConvLSTM network 48.

[0121] In the case where the changes in the image are based on changes in a small number of pixels relative to the total number of pixels, pre-training the CNN encoder 220 based on features having a prevalent second feature classification 44 and the output of the CNN encoder 46 can provide a rough training to the CNN encoder 220, which can then be refined by training the CNN encoder 220 based on the output of the ConvLSTM network 48 and a first (target) feature classification 42 that is less prevalent than the second feature classification. This multi-stage training improves the training efficiency of the entire computing network 200. The improvement in efficiency can be considered by analogy to a multi-stage rocket. In the same way that a multi-stage rocket launch is more efficient than a single-stage rocket launch because each subsequent stage forms an effective improvement to the previous stage, multi-stage training of some embodiments of the methods described herein is more efficient because the pre-trained first stage trains the CNN encoder 220 to an approximate level of accuracy, which is then refined to the desired level of accuracy by the trained second stage. Simply training the CNN encoder 220 according to the second stage method will consume a large amount of processing power and storage, and is therefore undesirably inefficient.

[0122] In some embodiments, during the operation of training the computing network 200, the weights of the CNN encoder 220 are frozen so that the pre-trained CNN encoder and the trained CNN encoder are configured with the same weights. This may be appropriate when there is a relatively small amount of training data, such as 10,000 training data samples or less, 1,000 training data samples or less, 500 training data samples or less, 100 training data samples or less, or 10 training data samples or less.

[0123] In this case, freezing the weights of the CNN encoder while training the ConvLSTM network 260 prevents the CNN encoder 220 from being "overfitted" - a common problem when training neural networks with scarce training data. In this case, the preliminary map generated by the pre-trained CNN encoder will have the same form as the feature map 230 generated by the fully trained CNN encoder 220.

[0124] In some embodiments, during the operation of training the computing network 200, the weights of the CNN encoder 220 are unfrozen so that the pre-trained CNN encoder and the trained CNN encoder can be configured with different weights. This may be appropriate when there is a relatively large amount of training data, such as 10 training data samples or more, 100 training data samples or more, 500 training data samples or more, 1000 training data samples or more, or 10,000 training data samples or more.

[0125] In this case, due to the universality of the training data, the training of the CNN encoder may naturally be robust against overfitting. In addition, when a large amount of training data is available, the accuracy of the entire computing system 200 can be improved by allowing the weights of the CNN encoder and the nodes and weights of the ConvLSTM network 260 to be adjusted collaboratively during training. In this case, if the weights of the fully trained CNN encoder 220 are different from the weights of the pre-trained CNN encoder, the preliminary map generated by the pre-trained CNN encoder will not have the same form as the feature map 230 generated by the fully trained CNN encoder 220.

[0126] As described above, in some embodiments, each time a successive image is added to the image stack to generate a continuous change map, the identification of changes between the images, as described below, can be applied to track the evolution of the landscape of interest. In some instances, this can be represented in a video or series of images depicting the continuous change map and its evolution over time.

[0127] In other words, in some embodiments, the plurality of images 210 includes consecutive images, and the method further includes: propagating ConvLSTM input data through the ConvLSTM network 260, and convolving the ConvLSTM input data respectively associated with each consecutive image with the ConvLSTM input data respectively associated with the previous image to generate continuous change maps 270, wherein each continuous change map 270 represents a change between one of the plurality of images 210 and the consecutive image.

[0128] Figure 5 Depicted are the results of applying the described method to an example of identifying deforestation areas in a forest environment imaged by SAR.

[0129] SAR images 50a through 50n are collected by satellite 10 as it passes over a target forest environment. On each repeated orbit of Earth 12, satellite 10 adds another image to the plurality of images 50 to create a stack of coherent images of the forest environment.

[0130] A plurality of images 50a to 50n and data associated with the images are used as Figure 2a The CNN encoder 220 convolves the data associated with each of the images 50a to 50n to generate a plurality of corresponding feature maps 230. Each feature map 230 includes data indicating a feature classification of its corresponding image according to a feature classification scheme, which is based on Figure 4 The method shown is based on the training of a CNN encoder. Figure 5 In the example shown, the feature classification scheme is a binary classification of forest land or non-forest land. The plurality of feature maps 230 and a copy of the data associated with each of the plurality of images 50a to 50n are convolved and propagated through the ConvLSTM network 260 to generate a variation map 52. Figure 5 As shown, the output returned from the computing system 200 is a change map 52 overlaid on the final image 50n of the image stack. In other examples, the change map 52 can be overlaid on any other image in the plurality of images 50a to 50n, for example, the change map 52 can be overlaid on the first image 50a.

[0131] Figure 6 Depicted is a computer 60 or other suitable apparatus configured to perform any of the methods described herein.

[0132] The computer 60 includes an image input interface 61 , a processor 62 including a memory 63 , a CNN encoder module 64 , a ConvLSTM network module 65 , an image difference output interface 66 , a plurality of data connection links 67 a - c and a skip connection link 68 .

[0133] The computer receives image data corresponding to the plurality of images 210 at the image input interface 61. The data associated with the plurality of images 210 is transmitted to the CNN encoder module 64 of the processor 62 via the data connection link 67a. The CNN encoder module 64 is configured as follows Figure 2a and Figure 2b The output of the CNN encoder module 64, such as the plurality of feature maps 230, is transmitted to the ConvLSTM network module 64 of the processor 62 via the data connection link 67b. The ConvLSTM network module 64 is configured as follows Figure 2a and Figure 2c The ConvLSTM network 260 shown operates as shown. The CNN encoder module 64 and optionally the ConvLSTM network module 65 can be trained based on training data stored in the memory 63 of the processor 62.

[0134] In some embodiments, copies of the data associated with the plurality of images 210 are transmitted to the ConvLSTM module via the skip connection links 68. The skip connection links 68 are configured to provide Figure 2a The skip connection 240 depicted in FIG.

[0135] The output of ConvLSTM network module 65, such as change map 270, is transmitted via data connection link 67c to image difference output interface 66. Image difference output interface 66 provides information identifying differences between multiple images 210 received at image input interface 61 to a user of computer 60.

[0136] In the above embodiments, the computing system 200 can be implemented on a server. The server can include a single server or a network of servers. In some instances, the functionality of the server can be provided by a network of servers distributed across a geographical area, such as a globally distributed server network, and a user can connect to an appropriate one of the server networks based on, for example, the user's location.

[0137] For clarity, the above description discusses embodiments of the present invention with reference to a single user. It should be understood that in practice, computing system 200 may be shared by multiple users, and potentially by many users simultaneously.

[0138] The above embodiments are fully automatic. In some instances, a user or operator of the system may manually instruct some steps or operations of the method to be performed.

[0139] In the described embodiments of the present invention, the system may be implemented as any form of computing and / or electronic device. Such a device may include one or more processors, which may be microprocessors, controllers, or any other suitable type of processor for processing computer-executable instructions to control the operation of the device in order to collect and record routing information. In some instances, for example, where a system-on-chip architecture is used, the processor may include one or more fixed function blocks (also known as accelerators) that implement a portion of the method in hardware (rather than software or firmware). Platform software including an operating system or any other suitable platform software may be provided at the computing-based device to enable execution of application software on the device.

[0140] The various functions described herein can be implemented in hardware, software, or any combination thereof. If implemented in software, these functions can be stored on or transmitted on a computer-readable medium as one or more instructions or codes. Computer-readable media may include, for example, computer-readable storage media. Computer-readable storage media may include volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media can be any available storage medium that can be accessed by a computer. As an example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, flash memory or other memory devices, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store the required program code in the form of instructions or data structures and can be accessed by a computer. The optical disks and discs used herein include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy discs, and Blu-ray discs (BDs). In addition, propagation signals are not included in the scope of computer-readable storage media. Computer-readable media also include communication media, which include any media that facilitates the transfer of computer programs from one place to another. For example, a connection can be a communications medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves, it is included in the definition of communications medium. Combinations of the above are also included within the scope of computer-readable media.

[0141] Alternatively or additionally, the functions described herein may be at least partially performed by one or more hardware logic components. For example, but not limited to, the hardware logic components that may be used may include a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), an application specific standard product (ASSP), a system on a chip (SoC), a complex programmable logic device (CPLD), etc.

[0142] Although illustrated as a single system, it should be understood that the computing device 60 may be a distributed system. Thus, for example, several devices may communicate via a network connection and may jointly perform the tasks described as being performed by the computing device.

[0143] Although illustrated as a local device, it will be appreciated that computing device 60 may be remotely located and accessed via a network or other communication link (eg, using a communication interface).

[0144] As used herein, the term "computer" refers to any device that has processing capabilities such that it can execute instructions. Those skilled in the art will recognize that such processing capabilities are incorporated into many different devices, and thus the term "computer" includes PCs, servers, mobile phones, personal digital assistants, and many other devices.

[0145] Those skilled in the art will recognize that the storage devices for storing program instructions can be distributed over a network. For example, a remote computer can store an instance of a process described as software. A local or terminal computer can access the remote computer and download some or all of the software to run the program. Alternatively, the local computer can download software fragments as needed, or execute some software instructions at the local terminal and some software instructions at the remote computer (or computer network). Those skilled in the art will also recognize that all or part of the software instructions can be executed by a dedicated circuit (such as a DSP, a programmable logic array, etc.) using conventional techniques known to those skilled in the art.

[0146] It should be understood that the benefits and advantages described above may relate to one embodiment or may relate to several embodiments. The embodiments are not limited to embodiments that solve any or all of the problems described or embodiments that have any or all of the benefits and advantages described. Variations should be considered to be within the scope of the present invention.

[0147] Any reference to "an" item refers to one or more of those items. The term "comprising" is used herein to mean including the identified method steps or elements, but such steps or elements do not comprise an exclusive list and the method or apparatus may contain additional steps or elements.

[0148] As used herein, the terms "component" and "system" are intended to encompass computer-readable data storage configured with computer-executable instructions that, when executed by a processor, cause certain functions to be performed. Computer-executable instructions may include routines, functions, and the like. It should also be understood that a component or system may be located on a single device or distributed across several devices.

[0149] Furthermore, as used herein, the term "exemplary" is intended to mean "serving as an illustration or example of something."

[0150] Furthermore, to the extent that the term "includes" is used in either the detailed description or the claims, this term is intended to be inclusive in a manner similar to how the term "comprising" is interpreted when "comprising" is used as a transition word in a claim.

[0151] Furthermore, the actions described herein may include computer-executable instructions that may be implemented by one or more processors and / or stored on one or more computer-readable media. Computer-executable instructions may include routines, subroutines, programs, execution threads, etc. Furthermore, the results of the actions of these methods may be stored on a computer-readable medium, displayed on a display device, etc.

[0152] The order of the steps of the methods described herein is exemplary, but the steps may be performed in any suitable order, or concurrently where appropriate. Additionally, steps may be added or substituted in any method, or individual steps may be deleted from any method, without departing from the scope of the subject matter described herein. Aspects of any of the examples described above may be combined with aspects of any of the other examples described to form additional examples without losing the intended effect.

[0153] It should be understood that the above description of the preferred embodiments is given by way of example only, and that various modifications may be made by those skilled in the art. The foregoing description includes examples of one or more embodiments. Of course, it is not possible to describe every possible modification and variation of the above apparatus or method for the purposes of describing the aforementioned aspects, but those skilled in the art will recognize that many further modifications and permutations of the various aspects are possible. Therefore, the described aspects are intended to cover all such variations, modifications, and variations that fall within the scope of the appended claims.

Claims

1. A computer-implemented method for identifying one or more changes across a plurality of images using a computational network comprising a convolutional neural network (CNN) encoder and a ConvLSTM network, the method comprising: receiving, at the CNN encoder, CNN input data comprising data associated with each pixel of each of the plurality of images; propagating the CNN input data through the CNN encoder to generate a plurality of feature maps, wherein each feature map comprises a feature classification for each pixel of a corresponding image of the plurality of images according to a feature classification scheme, wherein the feature classification scheme is generated by the CNN encoder based on training data; providing a skip connection between the input of the CNN encoder and the input of the ConvLSTM network; Propagating a copy of the CNN input data to the input of the ConvLSTM network through the skip connection; as well as convolving the data associated with each of the plurality of images in the copy of the CNN input data with a corresponding feature map generated by the CNN encoder to generate the ConvLSTM network input data, the ConvLSTM network input data comprising a plurality of convolved feature maps; Receiving the ConvLSTM network input data at the ConvLSTM network; as well as The ConvLSTM network input data is propagated through the ConvLSTM network to generate a variation map, wherein the ConvLSTM network is configured to perform convolutions between and within maps of the ConvLSTM network input data, and wherein the variation map includes variation data indicative of one or more variations across the plurality of images. 2 . The computer-implemented method of claim 1 , wherein the change data comprises quantitative data indicating an extent of the one or more changes across the plurality of images.

3. The computer-implemented method of claim 1 or 2, wherein the variation data comprises a variation classification for each pixel of a selected image of the plurality of images, Wherein, for a given pixel of the selected image, the change classification of the pixel indicates whether a feature classification of the pixel is the same as or different from a feature classification of a corresponding pixel of another of the plurality of images. The computer-implemented method of claim 3 , wherein the change classification is a binary classification.

5. The computer-implemented method of claim 1 , wherein the CNN input data comprises amplitude data indicative of one or more amplitude values ​​associated with each of the pixels of each of the plurality of images. 6 . The computer-implemented method of claim 1 , wherein the feature classification scheme is a binary classification scheme configured to classify the identified object as belonging to a first feature classification or a second feature classification.

7. The computer-implemented method of claim 6 , wherein the training data used to train the neural network includes data representing the first feature classification and the second feature classification, and wherein the data representing the first feature classification is scarce in the training data relative to the data representing the second feature classification.

8. The computer-implemented method of claim 7, wherein the feature classification scheme is generated by training the CNN encoder, wherein training the CNN encoder comprises: pre-training the CNN encoder based on the data representing the second feature classification and the output of the CNN encoder to generate a preliminary graph; as well as The computational network is trained by training both the CNN encoder and the ConvLSTM network based on the data representing the first feature classification and the output of the ConvLSTM network, wherein the trained CNN encoder is configured to generate the plurality of feature maps according to the feature classification scheme, and the trained ConvLSTM network is configured to generate a variation map.

9. The computer-implemented method of claim 1 , wherein propagating the CNN input data through the CNN encoder to generate the plurality of feature maps comprises compressing the CNN input data.

10. The computer-implemented method of claim 1 , wherein the CNN input data further comprises phase data indicating a respective phase value for each pixel of each image of the plurality of images, and wherein feature classification of each pixel of each image by its respective feature map is based at least in part on the phase data.

11. The computer-implemented method of claim 1 , wherein the ConvLSTM network input data further comprises phase data indicating a respective phase value for each pixel of each of the plurality of images, and wherein propagating the ConvLSTM network input data through the ConvLSTM network comprises: The phase data is convolved with the plurality of feature maps to generate the variation map.

12. The computer-implemented method of claim 1 , wherein each of the plurality of images is an image of a common target imaged at a different time, such that identifying one or more differences across the plurality of images is equivalent to identifying one or more changes in the common target over time.

13. The computer-implemented method of claim 1 , wherein the plurality of images comprises sequential images, and the method further comprises: The ConvLSTM network input data is propagated through the ConvLSTM network, and the ConvLSTM network input data respectively associated with each of the consecutive images is convolved with the ConvLSTM network input data respectively associated with the previous image to generate a continuous change map, wherein each continuous change map represents a change between one of the multiple images and the consecutive image.

14. The computer-implemented method of claim 1, wherein each of the plurality of images is coherent with each of the other images.

15. The computer-implemented method of claim 1, wherein each of the plurality of images is an image of an area of ​​10 square kilometers or greater, 50 square kilometers or greater, 100 square kilometers or greater, 1,000 square kilometers or greater, 5,000 square kilometers or greater, or 10,000 square kilometers or greater.

16. The computer-implemented method of claim 15, wherein the variation map is configured to resolve spatial features having a size of 50 meters or less, 10 meters or less, 5 meters or less, or 1 meter or less.

17. The computer-implemented method of claim 1 , wherein each of the plurality of images is an image of a geographic area, and the feature classification scheme comprises: a first feature classification indicating that the pixels so classified represent the presence of a predetermined geographic feature; as well as a second feature classification indicating that the pixels so classified represent the absence of said predetermined geographic feature, The method further comprises: Regions where the presence / absence of the predetermined geographic feature changes are identified based on the identified differences across the plurality of images.

18. The computer-implemented method of claim 17, wherein The first feature classification is a forest classification indicating that the pixels so classified represent forest land, and the second feature classification being a non-forest classification indicating that the pixels so classified represent non-forest land, The method further comprises: Changes in deforestation size surrounding a forested land are identified based on the identified differences across the plurality of images.

19. The computer-implemented method of claim 1, wherein each of the plurality of images is generated by synthetic aperture radar imaging.

20. The computer-implemented method of claim 1, wherein each of the plurality of images is generated from data acquired by a satellite.

21. The computer-implemented method of claim 20, wherein each of the images is generated from data acquired by a satellite in low Earth orbit.

22. A computing system configured to identify one or more changes across a plurality of images by implementing the method of any one of the preceding claims, the system comprising: Convolutional neural network CNN encoder, the convolutional neural network CNN encoder is configured to receiving, at an input of the CNN encoder, CNN input data comprising data associated with the plurality of images, and propagating the CNN input data through the CNN encoder to generate a plurality of feature maps, wherein each feature map comprises a feature classification for each pixel of a corresponding image of the plurality of images according to a feature classification scheme, wherein the feature classification scheme comprises a plurality of classifications and is generated by the CNN encoder based on training data; ConvLSTM network; a skip connection between the input of the CNN encoder and the input of the ConvLSTM network, wherein the system is configured to: propagating a copy of the CNN input data to the input of the ConvLSTM network through the skip connection, and convolving each of the plurality of images in the copy of the CNN input data with a corresponding feature map generated by the CNN encoder to generate the ConvLSTM network input data, the ConvLSTM network input data comprising a plurality of convolved feature maps; and A data connection link between the CNN encoder and the ConvLSTM network, wherein the ConvLSTM network is configured to: receiving the ConvLSTM network input data at an input of the ConvLSTM network via the data connection link, and The ConvLSTM network input data is propagated through the ConvLSTM network to generate a variation map, wherein the ConvLSTM network is configured to perform convolutions between and within maps of the ConvLSTM network input data, and wherein the variation map includes variation data indicative of one or more variations across the plurality of images.

23. An apparatus comprising a processor configured to perform the method according to any one of claims 1-21.

24. A computer program product comprising instructions which, when said program is executed by a computer, cause said computer to perform the method according to any one of claims 1 to 21.

25. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1-21.

Images