Method and device for locating a remotely captured image of an object

Description

[0001] The present invention relates to a method, a computer program and a device for locating an image of an object, in particular the Earth or another natural or artificial celestial body or a surface section thereof, taken by means of at least one imaging recording device from a distance, in particular from a great distance.

[0002] The recordings can in particular be aerial photographs or satellite images, so that the recording device can accordingly be located on or in an aircraft or spacecraft. Accordingly, recording distances between the recording device and the recorded object in the range of a few hundred meters or several kilometers up to more than 100 km are conceivable. The recordings can in particular form a series of images or another group of several images which together depict a surface area on the object that is too large to be fully depicted in a single recording. Such series of images or groups are particularly known in connection with surface mapping of celestial bodies, in particular the Earth, as used, for example, in today's common navigation applications for smartphones and computers.There, the overall image of the Earth's surface is composed of a large number of overlapping satellite-based individual images.

[0003] It is also known to create weather forecasts based on satellite photos, which can particularly depict clouds in the Earth's atmosphere.

[0004] In many applications, such as weather forecasts or the planning of infrastructure projects such as the construction or operation of railway lines, it is helpful or even necessary to position the images relative to the surface of the object as precisely as possible, so that image elements or even individual pixels shown in the images can be assigned as precisely as possible to a position on the surface of the object. However, precise spatial assignment of such images taken from a distance can be complicated by the fact that determining the orientation and position of the recording device in space and / or height differences in the recorded area on the surface of the object can be subject to non-negligible inaccuracies.

[0005] It is also known to use aerial photographs to control solar power plants, in particular to take cloud shadows into account. For example, it is known from US 2011 / 220091 A1 to use images depicting cloud shadows in relation to a field of heliostats to adjust the operation of a solar energy system. For example, images of a heliostat field and the shadows cast by the clouds can be obtained. Additionally or alternatively, images of the sky and clouds can also be taken. The images can be analyzed to determine a shadowing parameter. Based on the shadowing parameter, an operating parameter of the solar energy system can be changed or maintained.US 2014 / 083413 A1 also describes real-time monitoring of cloud shadowing of at least part of a solar field to enable more efficient operation of a solar energy system, since cloud shadowing can influence the flux on a heliostat, which in turn can impact the energy generated by the solar system. The invention is based on the object of providing an improved solution, particularly with regard to the achievable accuracy, for locating images of an object taken remotely, in particular aerial photographs or satellite images.

[0006] This object is achieved according to the teaching of the independent claims. Various embodiments and further developments of the invention are the subject of the dependent claims.

[0007] A first aspect of the invention relates to a method, in particular a computer-implemented method, for locating an image of an object taken remotely by means of at least one imaging device, according to claim 1.

[0008] As already mentioned at the beginning, the distance between the recording device and the object being recorded may, in particular, be greater than a few hundred metres or a few kilometres, as is usual in the case of aerial photographs, or even, as in the case of spacecraft-based photographs, such as satellite photographs, in particular greater than 100 km.

[0009] For the purposes of the invention, "capturing" data, such as image data or measurement data, is understood to mean, in particular, (i) that the data in question is received as such, for example, via a data interface of a device executing the method, or (ii) that the data is generated, in particular by means of suitable sensors, such as an image recording device (camera) in the case of image data, or (iii) that the data is generated artificially, in particular within the framework of a simulation or based on a mathematical model. Mixed forms of two or more of these possibilities are also conceivable, for example, those in which part of the data is received, while another part of the data is first generated by sensors or by simulation.

[0010] A "medium" within the meaning of the invention is understood in particular to mean a quantity of matter that, according to its type, quantity, and density, is capable of blocking or modifying electromagnetic radiation in the wavelength range underlying the measurement, in particular absorbing, scattering, or reflecting it, so that the presence of the medium in the beam path between the recording device and the sensor device can influence the intensity, phase, and / or wavelength of the radiation arriving at the sensor device. In particular, such a medium can be a cloud in the atmosphere of a celestial body serving as an object, in particular a planet.The cloud can be a conventional cloud of water vapor or water ice ("weather cloud"), or a smoke cloud, such as a smoke cloud resulting from combustion or a volcanic ash cloud or a dust or sand cloud (such as in a sandstorm).

[0011] For the purposes of the invention, an "image" of the medium in the image data is understood to mean, in particular, an image region of the image recorded by the image data that at least partially depicts the medium. For example, a medium in the form of a conventional weather cloud in the visible wavelength range can be represented on the image recorded by means of appropriate coloring, e.g., as a white or gray area. False-color images are also possible.

[0012] For the purposes of the invention, an "image" of the medium in the measurement data is understood to mean, in particular, a surface area on the surface section in which at least one of the sensor devices is located, whereby the measurement data provided by it indicates the measurement of an attenuation, in particular an intensity reduction, of the electromagnetic radiation incident on the sensor device, in particular solar radiation, caused by the medium. The surface area thus lies in a "shadow" caused by the medium.

[0013] In the sense of the invention, image portions of the two images represented in the image data or the measurement data "correspond" to each other with regard to their respective image content, in particular when the corresponding image portions (image sections or pixels of the image recording or measured values ​​of the sensor device(s)) each either both represent the presence or both the absence or the lack of a (partial) image of the medium in the respective image portion.

[0014] The location information can be defined, in particular, by means of suitable coordinates in a global or local coordinate system defined with respect to the object or its surface, or a portion thereof. In the case of the Earth as the object, conventional geocoordinate systems can be used for this purpose, preferably those that use elevation coordinates (e.g., height above sea level) in addition to angular coordinates (e.g., longitude and latitude coordinates) to localize a location on the object's surface.

[0015] In the method according to the first aspect, by comparing the image data and the measurement data, in particular an at least partial compensation of optical distortions, which typically arise in particular during the 2D image recording of a curved 3D object surface, such as the earth's surface, can be achieved in a simple manner and thus the achievable accuracy of the regulation of the image recording can be improved.

[0016] The acquired image data each comprise a digital image recording of at least one surface section of the object in which the sensor device is located for different recording times. The method is further carried out with a plurality of iterations in such a way that: (i) different iterations each correspond to a different one of the recording times (which can be done in particular such that the iterations are ordered according to the chronological sequence of the recording times); (ii) in each iteration, only those pixels of the digital image recording are retained for further processing in the next iteration for which, in the current iteration and, if applicable, all previous iterations, as part of the comparison of the image data and the measurement data for the associated recording time and the sensor device, there is a match with regard to the respective image contents with regard to the presence orthe absence of any medium present in the beam path at the respective time of recording was detected; and (iii) after a certain final iteration, the position of the sensor device in the image recording is determined on the basis of at least one of the pixels still remaining up to that point. The successive iterations thus act similarly to a filter with regard to the pixels, so that the number of pixels still remaining is gradually reduced to such an extent that, on the basis of the pixel(s) still remaining after the final iteration and the known location information for the sensor device, the sensor device can be located in the digital image recording and thus also the image recording as such can be located in a global or local coordinate system with a high degree of accuracy.

[0017] As a criterion for determining whether a specific pixel in the image data represents the presence or absence of an image of a medium, an intensity or brightness and / or color assigned to the pixel by the image data can be used, in particular. As a criterion for determining whether the measurement data indicate the presence or absence of the medium, an intensity or brightness and / or color or wavelength of the detected electromagnetic radiation (in particular in the visible spectral range) detected by the sensor device can be used, in particular. In particular, if dust, sand, or ash clouds are to serve as the medium, image data representing image recordings in the wavelength range of radar radiation (radio frequency range, e.g., millimeter waves or centimeter waves) can be used in addition to or instead of image data representing image recordings in the visible spectral range.

[0018] In the following, exemplary embodiments of the method are described, which can each be combined with each other as well as with the other aspects of the invention described, unless this is expressly excluded or is technically impossible.

[0019] In some of the iteration-based embodiments, the final iteration is determined as one of the following: (i) the last iteration after which at least m, in particular contiguous, pixels remain, where m ≥ 1; (ii) the k-th iteration after which at least m, in particular contiguous, pixels remain, where m and k are natural numbers and m ≥ 1, k > 1. In this way, it can be ensured that after the final iteration, at least a number of pixels defined by the parameter m is present, on the basis of which the location of the sensor device in the digital image recording and thus also of the image recording as such can be carried out.Option (ii) also ensures that the number of iterations is absolutely limited, which can be particularly relevant if, in several consecutive iterations, the reduction of pixels progresses only slowly during the iteration due to a configuration of recording device and medium that hardly changes over time. Thus, by appropriately choosing the parameters m and k, especially variable ones, a (currently) desired compromise between speed and accuracy of localization can be defined.

[0020] In some embodiments, if the presence of a radiation-attenuating medium in the beam path is determined based on the image data and / or the measurement data, the radiation incident on the sensor device is subjected to a spectral analysis in order to infer the type of matter present in the medium. Since different media (such as water vapor, water droplets, aircraft exhaust, volcanic ash, sand, or dust) typically produce different optical spectra, in particular absorption spectra, of the radiation detected at the sensor device, conclusions can be drawn about the type of medium, in particular about one or more types of matter contained therein, and preferably also about their relative abundances or concentrations.

[0021] In some embodiments, locating the image recording further comprises determining at least one further item of location information for a further selected position in the image recording as a function of the position determined for the sensor device and its associated known location information. In this way, locating can be performed for further positions in the image recording, in particular such that, across a set of several such additionally determined positions, coverage of the image recording is achieved in such a way that none of the pixels is further than a predetermined distance from at least one of the located pixels.

[0022] In some embodiments, the method further comprises: using data representing the located image recording including the associated location information as input data for one or more meteorological models in order to create a weather forecast for at least a partial area of ​​the surface section of the object covered by the image recording based thereon and to generate and provide weather forecast data representing this weather forecast.Due to the high accuracy achievable with this method in locating images, especially weather satellite images, the weather events detectable in the images, such as cloud types and formations, and when considering a temporal sequence of images, also the dynamics of such clouds, can be located with high accuracy. This can promote an improvement in the forecast quality and reliability of weather forecasts based on them. This can be particularly relevant for weather forecasts for regions with highly heterogeneous surface structures or distinctive surface features (e.g., mountains, rivers, lakes, sea coasts).

[0023] In particular, in some of these embodiments, the method may further comprise controlling or configuring a technical device or a technical system as a function of the weather forecast data. In particular, the controlling or configuring may take place with regard to one or more functionalities or configuration options of the following technical devices or systems: (i) a plant or a system for manufacturing products; (ii) a plant or a system for generating electrical energy (e.g. weather-dependent power plant control); a distribution network for energy (e.g. smart grid or canal system for hydroelectric power plants); (iii) a traffic route or traffic network (e.g. railway line or railway network or road orRoad network with controllable traffic infrastructure, such as signalling systems or information display devices); (iv) a vehicle or a group of vehicles to be moved in a coordinated manner (e.g. For example, the power consumption of electric locomotives is often temperature- and therefore weather-dependent.

[0024] In some embodiments, the comparison of the image data and the measurement data is carried out using a machine learning-based method with the image data and the measurement data as input data. In particular, one or more artificial (trained) neural networks can be used to classify pixels of the image recording(s) as to whether they (co-)image the location where the sensor device is located. The neural network(s) can be supplied with, in particular, the image data, the measurement data, and one or more criteria, based on which it can be determined for a pixel whether its pixel value indicates the presence or absence of a medium, as input data.

[0025] In some of these embodiments, the weather forecast data for a specific forecast period, together with corresponding actually measured weather data, are used as training data or validation data for further training or validation of the machine learning-based method. In particular, training in the sense of supervised learning can be performed in order to prepare or further optimize the at least one neural network in advance or continuously for its use.

[0026] In some embodiments, at least one classification criterion is provided to the machine learning-based method as a (further) input variable, based on which an image of the medium represented both in the image data and in the measurement data can be classified according to its type. The classification can relate in particular to the type and / or quantity and / or concentration of matter contained in the medium. For example, the classification criterion can be defined such that the reflection or absorption of electromagnetic radiation by the medium at different wavelength ranges is taken into account, so that it is thus possible to use the method to differentiate from one another, at least in pairs, clouds of matter characterized by different reflection or absorption properties, such as water vapor clouds, rain clouds, ice clouds, sand and dust clouds, or ash clouds.

[0027] In some embodiments, the method is carried out with respect to a plurality of sensor devices located at different locations on the object. Furthermore, the positioning of the image recording is carried out based on the determination of the positions in the image recording corresponding to the respective locations of the sensor devices by comparing and assigning the respective location information of the sensor devices to the respective position determined for them. In this way, the positioning of the image recording is based on multiple position determinations, so that the achievable accuracy for the positioning of the image recording, particularly with regard to its orientation, can be further increased.In particular, on the basis of the various determined positions, a location of the image recording can be achieved by applying an optimization method which can in particular be defined in such a way that the sum of the deviations (summed over the various sensor devices) which may arise between a respective pixel depicting the location of an associated sensor device and a (possibly different) pixel on the located image recording corresponding to the known location information for this sensor device is minimized.

[0028] In some embodiments, at least one of the sensor devices comprises a solar array, with which the respective associated measurement signal is generated depending on the electrical energy generated by the solar array at the respective recording time. In particular, the solar power generated by the solar array or a variable dependent thereon can thus serve as the measurement signal. Since solar arrays are distributed across many locations in many parts of the world, existing infrastructure can be used to generate the measurement data. The exact location of this infrastructure is usually already known in advance and often even officially registered.

[0029] In some embodiments, capturing the image data comprises capturing image material relating to the object, which image material was generated using a recording device carried by an aircraft, a spacecraft, or a satellite. The aircraft may, in particular, be a manned aircraft or an (unmanned) drone. Satellites may, in particular, be weather satellites, Earth or planetary observation satellites, or, in particular, manned space stations.

[0030] A second aspect of the invention relates to a device for locating an image of an object, in particular the Earth or another natural or artificial celestial body or a surface section thereof, taken remotely by means of at least one imaging device, wherein the device is configured to carry out the method according to the first aspect.

[0031] A third aspect of the invention relates to a computer program or a non-transitory computer-readable storage medium having instructions which, when executed on a computer or a multi-computer platform, in particular on the device according to the second aspect, cause it to carry out the method according to the first aspect.

[0032] The computer program can in particular be stored on a non-volatile data carrier. This is preferably a data carrier in the form of an optical data carrier or a flash memory module. This can be advantageous if the computer program as such is to be handled independently of a processor platform on which the one or more programs are to be executed. In another implementation, the computer program can be present as a file on a data processing unit, in particular on a server, and can be downloaded via a data connection, for example the Internet or a dedicated data connection, such as a proprietary or local network. In addition, the computer program can have a plurality of interacting individual program modules. The modules can in particular be configured or at least be usable in such a way that they can be used in the sense of distributed computing (DC)."Distributed computing" on different devices (computers or

[0033] Processor units that are geographically separated from each other and connected via a data network.

[0034] The computer or multi-computer platform or device according to the second aspect can accordingly have one or more program memories in which the computer program is stored. Alternatively, the computer or multi-computer platform or device can also be configured to access an externally available computer program, for example, on one or more servers or other data processing units, via a communications connection, in particular to exchange data with it that is used during the execution of the method or computer program or that represents outputs of the computer program.

[0035] The device according to the second aspect can thus be in particular a computer or a multi-computer platform, in particular with a plurality of networked computers, which is configured by means of one or more computer programs according to the second aspect to carry out the method according to the first aspect in order to locate one or more image recordings of an object taken by means of at least one imaging recording device from a distance, in particular from a great distance.

[0036] The features and advantages explained with respect to the first aspect of the invention also apply accordingly to the further aspects of the invention.

[0037] Further advantages, features and possible applications of the present invention will become apparent from the following detailed description in conjunction with the figures.

[0038] It shows: Fig. 1schematically shows an exemplary scenario for a sensory acquisition of image recordings and radiation measurement values ​​for use in the context of the inventive method for locating image recordings. Fig. 2 a flow chart illustrating a preferred embodiment of the method according to the invention; Fig. 3 a flow chart for a more detailed illustration of an iterative image evaluation within the scope of a preferred embodiment of the method according to the invention, in particular the method from Fig. 2 ; Fig. 4 schematically an exemplary temporal sequence of image recordings and their evaluation within the framework of an exemplary embodiment of the method according to the invention based on an iterative image evaluation, in particular the method from the Figures 2 and 3 ; Fig. 5schematically a diagram to illustrate an evaluation of an image recording and of corresponding measurement data acquired by means of a sensor device based on machine learning, in particular on the basis of an artificial neural network, for determining one or more pixels which represent the position of the sensor device in the image recording or (co-)image it.

[0039] Throughout the figures, the same reference numerals are used for the same or corresponding elements of the invention.

[0040] Fig. 1 illustrates an exemplary scenario 100, against the background of which exemplary embodiments of the method according to the invention will be explained below with reference to the other figures.

[0041] Scenario 100 is an earth observation scenario in which a surface section 115 of the earth's surface is photographed from a satellite- or aircraft-based recording device 105 in order to create a (in the lower part of the Fig. 1 to generate an image recording I = I(t) (shown offset for illustration purposes), which depicts the surface section 115. The image recording I is a digital two-dimensional image recording that has a matrix-shaped grid of image points (pixels) P.

[0042] Between the recording device 105 and the surface section 115 is a medium 110, which can in particular be a cloud in the Earth's atmosphere. The cloud can in particular be a conventional "weather cloud," which essentially consists of water droplets and / or ice crystals, or a dust cloud, a sand cloud (e.g., as a result of a sandstorm), or an ash cloud (e.g., as a result of a volcanic eruption). In the following, it is assumed that the medium is a conventional weather cloud. The image recording I accordingly shows an image 135 of the medium 110. Such image recordings are known, in particular, from satellite-generated weather images.

[0043] Due to its composition, the medium 110 has the property of attenuating or even completely blocking electromagnetic radiation 125, for example visible light, at least in certain wavelength ranges, in particular through absorption, scattering, and / or reflection. Therefore, in the presence of the medium under solar radiation 125, a shadow 130 results on the surface section 115. By way of example, sun rays 125a to 125d are identified here as part of the radiation 125, with the dashed sections of the sun rays intended to indicate the attenuation, in particular the intensity reduction, caused by the medium 110. Since the distance of the sun from the Earth's surface is very great, it can be assumed, as shown here, to a good approximation, that the sun rays are essentially parallel to one another (scattering effects in the Earth's atmosphere are neglected here).

[0044] Sensor devices 120a to 120d for electromagnetic radiation are located at various locations on the surface section 115, which may in particular be solar energy systems. In particular, the measured solar current of the respective solar energy system can then be used as a measurement signal for the intensity of the solar radiation 125 radiated onto the respective solar energy system. If a solar energy system is located in the shadow 130 of the medium 110, its solar current will be lower the greater the attenuation caused by the medium 110 (although not necessarily according to a linear relationship). On the other hand, in the absence of the medium 110 or when the solar energy system is not located in the shadow 130, the solar current will be higher, up to a design-related maximum value. Thus, the presence of the medium 110 in the light path (ray path) between the sun and the solar energy system can be inferred from the measurement of the solar current. In the example from Fig. 1Therefore, the solar arrays 120a and 120d not shaded by the medium 110 will measure a higher solar current relative to their respective maximum value than the solar arrays 120b and 120c located in the shadow 130. The measurement data of the sensor devices or solar arrays 120b and 120c located in the shadow 130 thus represent (at least in sections or points) the shadow 130 and thus an image of the medium 110.

[0045] Since the direction from which the image recording I is recorded by means of the recording device 105 does not usually coincide with the direction of incidence of the solar radiation 125, the position of the image 135 and the shadow 130 of the medium 110 usually do not coincide if the position of the shadow 130 is transferred to the image recording I as part of a comparison.

[0046] Within the scope of a transformation, the transformation parameters of which can be found in particular within the scope of an optimization problem, in which the image of the shadow 130 in the image recording and the image 135 of the medium 110 are optimally superimposed, are determined. It is also possible to calculate the transformation parameters geometrically, at least approximately, from a possibly previously known position of the recording device 105 at time t as well as the recording angle for generating the image recording relative to the surface section 115 and the position of the sun. Based on the determined transformation parameters, the respective position of the sensor devices 120a to 120d can then be transformed into the image recording. This is exemplary for the sensor device 120b in Fig. 1illustrated, whereby, since it lies in the shadow 130, it correspondingly lies in the image area of ​​the image recording I occupied by the image 135a of the medium. The corresponding pixels representing the image 135a are marked in black here for the purpose of illustration. Such a transformation can be carried out in particular within the framework of the Figures 2 and 3 the exemplary procedure described below.

[0047] Fig. 2 shows an exemplary embodiment 200 of a method according to the invention for locating one or more image recordings, which is described below, in particular with exemplary reference to Fig. 1 , is explained.

[0048] In the method 200, in a step 205, image data are received, which in particular can represent a plurality of images taken by the satellite- or aircraft-based recording device 105, each of which images the same surface section 115 of the Earth for different times ti with i = 1, 2, 3, ...

[0049] Furthermore, in the method 200, in a step 210, measurement data S i := S(ti ) from the sensor devices 120a to 120d, in particular solar systems, arranged at various locations in the surface section 115 are received for the various times ti. The reception of the image data I i := I(ti ) and the measurement data S i can be carried out, in particular, via a data interface with which the data I i and S i can be supplied to a computer or a multi-computer platform that executes or is intended to execute the method 200.

[0050] In order to prevent any subsequent failure (running idle) of the method 200 in the event of the absence of a medium 110 from the outset, a further step 215 checks whether an image 135 or shadow 130 of a medium 110 (for example, a cloud) is represented in both the image data I i and the measurement data S i. If this is not the case (215 - no), the method branches back to step 205. Otherwise (215 - yes), in a step 220, for each time ti, the respective image recording I i is compared with the respective corresponding measurement data S i of a specific sensor device, e.g., the sensor device 120b, in order to thereby determine which pixels P in the image recording I i correspond to the measurement data S i with regard to their image content or respective pixel value (cloud or no cloud depicted).

[0051] Such a correspondence occurs for a time t i when a match is determined regarding the presence or absence of a respective image, represented both in the image data I i and in the measurement data S i, of any medium 110 located in the beam path at the respective recording time. This is particularly the case when the image point just checked belongs to an image 135a of the medium 110 in the image recording I i and, at the same time, a reduced solar current attributable to the presence of the medium 110 was measured at the sensor device 120b, i.e., when the latter lies in the shadow 130 (i.e., in the image of the medium 110 represented by the measurement data S i). Conversely, it is also the case when the image point does not belong to the image 135 and the solar current does not indicate a position of the sensor device 120b in the shadow 130.

[0052] In order to determine such a correspondence, according to one possible embodiment, in particular the above-mentioned transformation of the position of the sensor device 120b under consideration into the image recording I i can be carried out, so that it can be checked directly whether the position of the sensor device 120b lies within the circumference of the image of the medium in the image recording I i.

[0053] With the help of the correspondence check, those pixels can be identified in the respective image recording I i that are valid candidates for (co-)image the location of the sensor device 120b on the surface section 115 in this image recording. The comparison can be carried out in particular iteratively on the basis of the image data I i and measurement data S i acquired at different times ti with i = 1, 2, 3, ... Such an iterative comparison 300 is described below with reference to the Figures 3 and 4 be explained in more detail.

[0054] Once the comparison has been completed on average 220, the images can be located relative to the Earth's surface based on the pixels identified during the comparison that correspond to the measured values. For this purpose, the previously known locations of the measuring equipment (or solar arrays) from the measurement data are used. For example, if in a simple case a single pixel in an image was identified as representing a specific solar array, the previously known location of this solar array can be assigned to this pixel. This can be done in particular with regard to different pixels and accordingly different assigned locations or solar arrays, so that the location can be determined based on different pixels. This is particularly advantageous for determining not only a position but also an orientation or alignment of the images relative to the Earth's surface with good accuracy.

[0055] Furthermore, a step 230 can be provided in which a spectral analysis is performed on the radiation incident on the sensor devices in order to determine the type of matter present in the medium. For example, based on the spectral analysis and a spectrum characteristic of the matter, it can be determined whether the medium is a conventional weather cloud (water vapor or water ice crystals), a (volcanic) dust cloud, a sand cloud, or, for example, smoke from a combustion process, e.g., in an industrial plant.

[0056] Based on the 225 image recordings located on average as input data, weather forecast data, in particular for the surface section 115 covered by the image recordings, can now be generated in a step 235 using one or more meteorological models. In a further step 240, this data can be used, in particular, to control a technical system, such as an energy distribution network, a production facility, or a transport network. For example, in a rail network, critical infrastructure, such as switches, can be heated based on temperature profiles predicted within the framework of the weather forecast data at the respective locations of such infrastructure.The power demand for train operation with electric locomotives or railcars, which is usually temperature-dependent, can also be predicted and the corresponding control of a railway-related power supply infrastructure can be planned and implemented accordingly.

[0057] In the Figures 3 and 4 A concrete embodiment of the interactive comparison process 300 from the method 200 is illustrated. In a step 305, the value of a parameter m > 1 is first determined to establish a termination criterion for the iteration.

[0058] In a step 310, which may also coincide with step 305, a starting value i:=1 is further defined for an index i for identifying various points in time, in particular successive ones with increasing values ​​of i, for generating the image recordings and measurements, and an initial pixel set M 0 is defined for a first digital image recording I 1 . The pixel set M 0 can, in particular according to the present example, be equal to the set M of all pixels P of the image recording I 1 .

[0059] In Fig. 4 a temporal sequence 400 of image recordings and their interactive image analysis is illustrated, wherein the image points P are each represented as pixels of a pixel matrix which represents the respective image recording I i for the respective one of three consecutive times ti with i = 1, 2 or 3.

[0060] Now, in the adjustment process 300, the first run of the iteration can take place to the starting value i = 1. In step 315, a comparison of the first digital image recording I 1 with the measurement data S 1 for the same (recording or measurement time t 1 ) is carried out, wherein a check is carried out to determine which of the pixels in the image recording I 1 have a pixel value that corresponds to the measurement data S 1 (see the previous explanations for Fig. 2 ). Those pixels P for which this is true form the set M 1 as a subset of M 0 and are in Fig. 4 colored black, since this illustrates the case where the sensor device 120b specifically considered here provides measurement data which indicate the presence of a medium 110 in the beam path 125. In Fig. 4The image 140 of the sensor device 120b is again shown transformed into the image recording, while the image of the medium 110 in this image recording I 1 is identified by the reference numeral 135a. The pixels shown in black in the image 135a are thus those pixels P that (co-)image the medium 110, which is shown here as a cloud, which can be done in particular by means of a respective image value that corresponds to a lower brightness than is the case with pixels that do not (co-)image the medium.

[0061] According to the flowchart from Fig. 3In a further step 320, it is now checked whether the remaining number of (black) pixels, ie, the strength |M 1 | of the set M 1 , is greater than or equal to the parameter m. If this is the case (320 - yes), the index i is incremented in step 325, and the next iteration is started by repeating step 315. The sequence of this and, if applicable, each subsequent iteration corresponds to the sequence described above for the first iteration.

[0062] If, however, the termination criterion is met after one of the iterations (320 - no), in step 330, the respective position of each sensor device in the recorded images is determined by averaging the positions of the pixels P remaining in the subset M i after the last iteration run. The position thus determined is then assigned to the sensor device as a position in the recorded image(s). Accordingly, in a further step 335, at least one of the recorded images I i can be located with respect to the earth's surface based on the location of the respective sensor device known in advance from the measurement data S i and the position determined and assigned to it in section 330.

[0063] Now referring again to Fig. 4, it can be seen in the image recording I 2 for time t 2 that at this time the medium 110 has migrated relative to the image recording and is partially represented by other pixels than image 135b. The sensor device 120b is now no longer in the shadow of the medium, i.e. it measures a correspondingly high solar current as a measurement signal. After the second iteration, only those pixels in the pixel matrix are considered as candidates for the position of the sensor device 120b (set M 2 ) that (co-)image the medium 110 in I 1 (set M 1 ) and do not (co-)image the medium 110 in the image recording I 2. In Fig. 4 These remaining pixel candidates in I 2 are colored black, while the candidates that were eliminated compared to I 1 are shown hatched there.

[0064] In the third in Fig. 4In the illustrated iteration, the medium 110 has migrated further, so that in image recording I 3 it is imaged as image 135c at yet another location in the pixel matrix. The measurement signal or the measurement data of the sensor device 120b now again provide a reduced solar current, which suggests coverage (shadowing) by the medium 110. Now, after the third iteration, only those pixels (colored black in I 3) remain as candidates (set M 3 ) that, on the one hand, remained as pixel candidates after the second iteration (set M 2 ) and, on the other hand, simultaneously (co-)image the image 135c in I 3 .

[0065] Since the third iteration in the present example also represents the last iteration (end iteration) after which the termination criterion in step 320 is met (320 - no), a position of the sensor device 120b in the image recording I 3 is now determined from the pixels remaining in the set M 3 by calculating the (in particular geometric) center of gravity 145 of this pixel distribution. It is located by assigning it the location K on the earth's surface, which is previously known for this sensor device from the measurement data and which is determined in particular by corresponding coordinates, for example a longitude ϕ, a latitude λ and a height h above sea level can be represented.

[0066] Fig. 5shows schematically a diagram 500 for the basic illustration of an evaluation based on machine learning, in particular on the basis of an (exemplary) artificial neural network 510, of a set {I i} of image recordings and of corresponding sets of measurement data {S i} recorded by means of a sensor device for a set {ti} of recording or measurement times ti for determining a set {P} from one or more image points P which are candidates for representing the position of the sensor device in the image recordings, ie for (co-)imaging this sensor device.

[0067] While at least one exemplary embodiment has been described above, it should be appreciated that a wide variety of variations exist. It should also be understood that the described exemplary embodiments are merely non-limiting examples and are not intended to limit the scope, applicability, or configuration of the devices and methods described herein. Rather, the foregoing description will provide one skilled in the art with guidance for implementing at least one exemplary embodiment, it being understood that various changes in the operation and arrangement of the elements described in an exemplary embodiment may be made without departing from the subject matter as defined in the appended claims, as well as their legal equivalents. LIST OF REFERENCE SYMBOLS

[0068] 100Scenario for the sensory capture of image recordings I and measurement data M 105Recording device, in particular satellite- or aircraft-based 110Medium, in particular cloud 115Surface section of a celestial body, in particular of the Earth, captured by image recording I 120a-dSensor devices, in particular solar systems 125Electromagnetic radiation, in particular solar radiation 125a-dVarious rays or bundles of rays of radiation 125 130Shadow on the surface section caused by the attenuation of the electromagnetic radiation by the medium 110 135Image of the medium in image recording I 135aImage of the medium in image recording I 1 135bImage of the medium in image recording I 2 135cImage of the medium in image recording I 3 140into image recording I orli transformed position of the sensor device 120b 145Center of gravity of the pixels remaining after the final iteration 200Method for locating one or more image recordings 205-240Procedural steps of the method 200 300Iterative comparison of image data and measurement data, 305-335Procedural steps of the iterative image evaluation 300 400Sequence of image recordings and their iterative comparison with measurement data 500Machine learning-based evaluation of image recordings and measurement data 510(artificial) neural network (schematic) IImage recording or image data I 1 , I 2 , I 3 Image recordings or. Image data at different points in time iIndex for identifying different points in time for generating the image recordings and measurements mParameter for defining a termination criterion for the iterative comparison t, ti Time variable or recording times KCoordinates of the previously known location of the sensor device 120b . ϕ Longitude λLatitude hHeight above mean sea level (NN) M i Pixel sets or subsets PPixel on image acquisition S i Measurement data

Claims

1. A method (200) of locating an image recording (I; I1, I2, I3) of an object, in particular of the Earth or of another natural or artificial celestial body or of a surface portion (115) thereof, recorded by means of at least one imaging recording device (105) from a distance, wherein the method (200) comprises: acquiring (205) image data (I; li) which represent the image recording (I; I1, I2, I3) and its point in time of recording (t; ti), wherein the recorded image data (I; li) comprise, for different points in time of recording (t; ti), a respective digital image recording (I; I1, I2, I3) of at least one surface portion (115) of the object in which at least one sensor device (120a-d) is located; acquiring (210) measurement data (Si) which represent, for the point in time of recording (t; ti), a respective measurement signal of the at least one sensor device (120a-d) which is arranged in a surface portion (115) of the object, captured by means of the image recording (I; I1, I2, I3), at a location with known location information (K) and which in this context is configured to detect electromagnetic radiation (125) which is incident on the sensor device (120a-d) from an optical path, which can in particular be located between the sun and the sensor device (120a-d), and to generate the measurement signal as a function of the radiation (125) detected in the course of this; matching (220) the image data (I; li) and the measurement data (Si), wherein an image of a medium (110) which is at least partially located in the optical path of the electromagnetic radiation (125), which image is represented by the image data (I; li), is compared with an image of the medium (110), which image is represented by the measurement data (Si), in order to identify, with respect to their respective image content, mutually corresponding image portions of the two images; and locating (225) the image recording (I; I1, I2, I3) with respect to the object by means of a determination of a position (145) in the image recording (I; I1, I2, I3) which corresponds to the location of the sensor device (120a-d) on the basis of the matching, and an associating of the location information (K) with this position, wherein the method (200) is carried out with multiple iterations in such a manner that: different iterations each correspond to a different one of the points in time of recording (t; ti); in each iteration, only those pixels (P) of the digital image recording (I; I1, I2, I3) are retained for the further processing in the respective next iteration for which, in the respective current iteration and, if applicable, in all preceding iterations, within the framework of the matching of the image data (I; li) and the measurement data (Si) for the associated point in time of recording (t; ti) and the sensor device (120a-d), a match has been established with regard to the respective image contents with respect to a presence or absence, respectively, of a medium (110), which may be present in the optical path at the respective point in time of recording (t; ti); and after a certain final iteration, the position of the sensor device (120a-d) in the image recording (I; I1, I2, I3) is determined on the basis of at least one of the pixels (P) still remaining up to that point in time.

2. The method (200) in accordance with claim 1, wherein the final iteration is determined as one of the following: - the last iteration after which at least m pixels (P) remain, where m ≥ 1; - the kth iteration after which at least m pixels (P) remain, where m and k are natural numbers and m ≥ 1, k > 1.

3. The method (200) in accordance with any one of the preceding claims, wherein, when, on the basis of the image data (I; li) and / or the measurement data (Si), the presence of a radiation attenuating medium (110) in the optical path is detected, the radiation (125) which is incident on the sensor device (120a-d) is subjected to a spectral analysis (230) in order to infer the type of the matter which is present in the medium (110).

4. The method (200) in accordance with any one of the preceding claims, wherein the locating (225) of the image recording (I; I1, I2, I3) further comprises: determining at least one further item of location information for a further selected position in the image recording (I; I1, I2, I3) as a function of the position (145) determined for the sensor device (120a-d) and its associated known location information (K).

5. The method (200) in accordance with any one of the preceding claims, further comprising: using data which represent the image recording which has been located (I; I1, I2, I3), including the associated location information (K), as input data for one or more meteorological models in order to generate a weather forecast based thereon for at least a partial area of the surface portion (115) of the object covered by the image recording (I; I1, I2, I3) and to generate, and make available (235), weather forecast data which represent this weather forecast.

6. The method (200) in accordance with claim 5, further comprising: controlling or configuring (240) a technical device or a technical system as a function of the weather forecast data.

7. The method (200) in accordance with claim 6, wherein the controlling or configuring (240) is carried out with respect to one or more functionalities or configuration options of the following technical devices or the following technical system: - a facility or a system for the manufacture of products; - a facility or a system for the generation or distribution of electrical energy; - a distribution network for energy; - a transport route or a transport network; - a vehicle or a group of vehicles which are to be moved together in a coordinated manner.

8. The method (200) in accordance with any one of the preceding claims, wherein the matching of the image data (I; li) and of the measurement data (Si) is carried out using a method which is based on machine learning (500), with the image data (I; li) and the measurement data (Si) being used as input data.

9. The method (200) in accordance with any one of the claims 5 to 7 and claim 8, wherein the weather forecast data for a specific forecast period together with actual measured weather data corresponding thereto are used as training data or validation data for the further training or validation of the method based on machine learning (500).

10. The method (200) in accordance with claim 8 or 9, wherein at least one classification criterion is made available to the method based on machine-learning (500) as an input variable, on the basis of which, if applicable, an image of the medium (110) represented in the image data (I; li) as well as in the measurement data (Si) can be classified in accordance with its type.

11. The method (200) in accordance with claim 1, wherein the method (200) in accordance with any one of the preceding claims is carried out with respect to a plurality of sensor devices which are located at different locations on the object, and the locating of the image recording (I; I1, I2, I3) is carried out in a corresponding manner on the basis of the determination of the positions in the image recording (I; I1, I2, I3) which correspond to the respective locations of the sensor devices (120a-d) by means of the matching and the associating of the respective location information of the sensor devices (120a-d) to the position respectively determined for them.

12. The method (200) in accordance with claim 11, wherein at least one of the sensor devices comprises a solar installation with which the respective associated measurement signal is generated as a function of the electrical energy which is generated by the solar installation at the respective point in time of recording (t; ti).

13. The method (200) in accordance with any one of the preceding claims, wherein the capturing of the image data (I; li) comprises the capturing of image material in relation to the object which image material has been generated by means of a recording device which is carried by an aerial vehicle, a space vehicle or a satellite.

14. A device for locating an image recording (I; I1, I2, I3) of an object, in particular of the Earth or of another natural or artificial celestial body or of a surface portion (115) thereof, recorded by means of at least one imaging recording device (105) from a distance, wherein the device is configured to carry out the method (200) in accordance with any one of the preceding claims.

15. A computer program or a non-volatile computer readable storage medium which comprises instructions which, when they are being executed on a computer or on a multi-computer platform, cause the computer or the multi-computer platform to carry out the method (200) in accordance with any one of the claims 1 to 13.