Computer programs, computer-implemented methods, and apparatus (using machine learning for modeling climate data)

Abstract

To provide techniques for using machine learning to model climatic data.SOLUTION: In one example, a computer-implemented method comprises receiving climate data comprising a plurality of spatial components and a plurality of temporal components, and masking a portion of the climate data. A machine learning model is trained, where the training is based at least in part on the masked portion of the climate data. A vector representation of the climate data is generated via the machine learning model. Also disclosed are an apparatus comprising a processing device, and a computer program.SELECTED DRAWING: Figure 7

Description

[Background technology]

[0001] Climate change and disruptive climate-related events, such as hurricanes or severe weather, can impact numerous applications, including those in the retail, financial, and utility spaces. Understanding climate trends and accurately predicting climate activity is critical to effectively forecasting public and private business activity. [Technical Field]

[0002] Embodiments of the present invention provide techniques that use machine learning to model climate data. Summary of the Invention [Problem to be solved by the invention]

[0003] Understanding climate trends and accurately predicting climate activity is important for effectively forecasting public and private business activity. [Means for solving the problem]

[0004] In one exemplary embodiment, a computer-implemented method includes receiving climate data including a plurality of spatial components and a plurality of temporal components, and masking a portion of the climate data. A machine learning model is trained, the training being based at least in part on the masked portion of the climate data. A vector representation of the climate data is generated via the machine learning model.

[0005] Further exemplary embodiments are provided in the form of a computer program product comprising a non-transitory computer readable executable program code embodied therein that, when executed by a processor, causes the processor to perform the computer-implemented method. Still further, exemplary embodiments comprise an apparatus or system having a processor and memory configured to perform the computer-implemented method.

[0006] These and other features and advantages of the embodiments described herein will become more apparent from the accompanying drawings and the following detailed description. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 illustrates a system for modeling climate data in accordance with an illustrative embodiment. [Figure 2] FIG. 1 illustrates a block diagram of training a machine learning model on climate data according to an exemplary embodiment.

[0008] [Figure 3] FIG. 1 illustrates a block diagram of spatiotemporal location embedding associated with climate data modeling according to an exemplary embodiment.

[0009] [Figure 4] FIG. 10 illustrates an example of a climate token used in connection with modeling climate data, according to an exemplary embodiment.

[0010] [Figure 5] FIG. 1 illustrates an operational flow for climate data modeling and task-specific fine-tuning according to an exemplary embodiment.

[0011] [Figure 6A] FIG. 10 illustrates a histogram of temperatures according to an exemplary embodiment.

[0012] [Figure 6B] FIG. 10 is an illustration of a histogram of temperature forecasts according to an exemplary embodiment;

[0013] [Figure 7] FIG. 1 illustrates a climate data modeling process flow according to an exemplary embodiment.

[0014] [Figure 8] FIG. 1 illustrates an exemplary information processing system, according to an exemplary embodiment.

[0015] [Figure 9] FIG. 1 illustrates a cloud computing environment in accordance with an exemplary embodiment.

[0016] [Figure 10] FIG. 1 illustrates abstraction model layers in accordance with an illustrative embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] Exemplary embodiments are described herein with reference to exemplary information processing systems and associated computers, servers, storage devices, and other processing devices. However, it is understood that embodiments are not limited to use with the particular exemplary system and device configurations shown. Accordingly, as used herein, the term "information processing system" is intended to be broadly interpreted to encompass a wide range of processing systems, by way of example, processing systems having cloud computing and storage systems, and other types of processing systems having various combinations of physical, virtual, or combinations of processing resources.

[0018] As mentioned in the background section above, understanding climate trends and accurately predicting climate activity is important to effectively forecasting business activities. For example, some retailers recognize the impact of weather on their demand forecasts and may rely on short-term weather forecasts to implement response plans for weather-related hazards such as floods, droughts, hurricanes, and severe weather. However, encoding medium- to long-term seasonal weather data into use-case machine learning models presents challenges. For example, uncertainties associated with medium- to long-term weather data based on different geographic areas and temporal variables create complexities in generating and training machine learning models to predict climate events and using the predictions in downstream use cases, such as, for example, supply and demand chain forecasts.

[0019] The present embodiments advantageously provide techniques for encoding spatiotemporal climate data into vector representations to efficiently solve climate-aware forecasting use cases across domains, regions, and / or time frames. The encoding is performed using climate masking techniques and upcoming climate forecast models to pre-train a climate data-to-vector ("climate2vec") machine learning model. The present embodiments further provide techniques for fine-tuning the climate2vec model to implement various downstream applications, such as, for example, climate-aware forecasting, including but not necessarily limited to, demand and lead-time forecasting at retail nodes in a supply chain.

[0020] FIG. 1 illustrates a system 100 for modeling climate data, according to an exemplary embodiment. As indicated by lines or arrows, or a combination thereof, in FIG. 1 , the components of system 100 are operatively connected to one another via physical connections, such as, for example, wires, direct electrical contact connections, or a combination thereof, and / or wireless connections, such as, for example, WiFi, Bluetooth, IEEE 802.11, a local area network (LAN), a wide area network (WAN), a cellular network, an ad hoc network (e.g., a wireless ad hoc network (WANET)), a satellite network, or other networks, or combinations thereof, including, but not limited to, the Internet. For example, a network may operatively link feature reconstruction engine 110 to climate-aware forecast engine 120 and its components.

[0021] The system 100 includes a feature reconstruction engine 110 having a transformer pre-training layer 111, a masked climate forecast layer 112, an uncertainty representation layer 113, a next-generation climate forecast prediction layer 114, and a spatiotemporal location embedding layer 115.

[0022] As shown in FIG. 1 , climate forecast data, including weather forecast data 102 and extreme event data 104, is input to a feature reconstruction engine 110. The weather forecast data 102 includes, for example, temperature, humidity, wind speed and direction, precipitation (e.g., snow, rain, etc.), air pressure, and other weather-related feature information associated with one or more geographic regions (e.g., cities, states, provinces, countries, continents, or other regional groupings (e.g., coastal and inland, hemispheres, etc.)) over one or more time periods (e.g., hours, days, weeks, months, seasons, decades, etc.). The weather forecast data 102 further includes uncertainty factors. For example, the forecast may include predictions within specific uncertainty ranges for items such as, but not necessarily limited to, temperature, humidity, or precipitation, or a combination thereof. The uncertainty may be associated with medium- to long-term climate variability. The extreme event data 104 includes forecasts of extreme events, including, but not limited to, hurricanes, severe weather, tornadoes, heat waves, cold waves, typhoons, or other extreme weather events, associated with one or more time periods and one or more geographic regions. The data input to the feature reconstruction engine 110 also includes historical weather data 106, including past weather observation data in the form of time series data, including collections of weather observations with repeated measurements of, for example, temperature, humidity, wind speed and direction, precipitation, barometric pressure, and other weather-related characteristics, for example, on an hourly, daily, weekly, monthly, seasonal, or decadal basis for different geographic regions. In some embodiments, future time series data may also be input to the feature reconstruction engine 110 along with past time series data.

[0023] The climate forecast data 102, 104 and historical weather data 106 (which may be collectively referred to herein as “climate data”) input into the feature reconstruction engine 110 include geospatial and spatiotemporal features. As used herein, the term “geospatial” is broadly interpreted to refer, for example, to geographic location. As used herein, the term “spatiotemporal” is broadly interpreted to refer, for example, to existing in both space (e.g., location) and time. For example, the present embodiment models climate data across geographies (e.g., the United States, India, Africa, etc.) and time periods (e.g., from 2000 to 2021). Several climate zones associated with the input climate data 102, 104, 106 may be defined using one or more indicators: tropical, arid, temperate, continental, polar, coastal, inland, urban, rural, altitude above sea level, agricultural, non-agricultural, forest, residential, and commercial.

[0024] In a non-limiting example, the input climate data 102, 104, or 106, or a combination thereof, may be represented as a geospatial climate data sequence S={S1, S2, ..., S n}. Each sequence S i captures climate time series data. For example, S i ={C t1 i ,C t2 i ,...,C tk i The set S includes spatiotemporal data across locations and time periods. Thus, the climate data 102, 104, or 106, or a combination thereof, includes multiple spatial components and multiple temporal components.

[0025] Referring to FIG. 1 , a masked climate forecast layer 112, a next-generation climate forecast prediction layer 114, and a spatiotemporal location embedding layer 115 are used by a transformer pre-training layer 111 to pre-train multiple transformers of a transformer-based neural network machine learning model. As used herein, “transformer” is broadly interpreted to refer to a deep learning model that differentially weights the importance of portions of input data. Similar to recurrent neural networks (RNNs), transformers manage input data sequentially. However, transformers do not necessarily process data sequentially, but rather utilize a mechanism to provide context for any position in the input sequence. By identifying context, transformers do not need to process the beginning of a data sequence before the end of the data sequence, allowing for more parallelization and shorter training times than RNNs. A non-limiting example of a Transformer-based neural network machine learning model that may be used by the present embodiment is the Bidirectional Encoder Representations from Transformers (BERT) model, which uses context from both directions and learns a representation for each token using the encoder portion of the Transformer.

[0026] The masking climate forecast layer 112 implements a masking strategy for the weather forecast data 102, extreme event data 104, or historical weather data 106 that uses climatological data to predetermine climate predictability for various regions for a certain time frame while masking the weather forecast data 102, extreme event data 104, or historical weather data 106. The masking climate forecast layer 112 determines how much (e.g., percentage) of the weather forecast data 102, extreme event data 104, or historical weather data 106 to mask, and which portions to mask. The masking strategy varies based on the granularity (e.g., daily, weekly, hourly, etc.) of the weather forecast data 102, extreme event data 104, or historical weather data 106. Referring to block diagram 211 of training a machine learning model for climate data in FIG. 2 , in one or more embodiments, the masked climate model masks a portion of the input weather forecast data 102, extreme event data 104, or historical weather data 106 for a first timestamp Ta (“climate data for timestamp Ta”) and a portion of the input weather forecast data 102 or extreme event data 104 for a second timestamp Tb (“climate data for timestamp Tb”) and attempts to predict the masked portion using their context (e.g., surrounding climate data). As noted in FIG. 2 , the input weather forecast data 102, extreme event data 104, or historical weather data 106 includes unlabeled climate data pairs. Also, while the input climate data is shown as a time series based on days, this embodiment is not limited thereto, and other granularities (e.g., hours, weeks, months, etc.) can be used.

[0027] In addition to masking, the Transformer pre-training layer 111 uses next-term climate forecast techniques on unlabeled climate data in conjunction with pre-training multiple Transformers. For example, various pairs of climate data points from the weather forecast data 102, extreme event data 104, or historical weather data 106 are generated based on climatology and historical weather data 106. For example, in a given pair, climate attributes for the first half of a given time period (e.g., one week) are followed by climate attributes for the second half of the given time period (e.g., one week). The next-term climate forecast prediction layer 114 predicts climate attributes for the second half of the time period. The next-term climate forecast prediction replaces the next-term climate forecast with a random climate forecast from a corpus to train a model that can understand the relationship between climate forecasts. For example, some of the time the next-term climate forecast is the original next-term climate forecast, and some of the time the original next-term climate forecast is replaced with a random climate forecast from the corpus. For a given climate data sequence, the transformer pre-training layer 111 attempts to predict the masked climate data, determine the climate data sequence order (e.g., the correct sequence or whether the sequence order needs to be changed), and predict the climate data for the next timestamp (e.g., given climate data for K timestamps, predict the climate data for the next timestamp).

[0028] Referring to FIG. 2, in connection with pre-training a transformer-based neural network, climate data for two different timestamps Ta and Tb are simultaneously managed. For example, a pair of climate data is input to a machine learning model with a separation [SEP] between each part of the pair. The first token of the input is represented as [CLS], which, after pre-training (“C”), can be used for aggregate sequence representation and employed for classification. “E” in FIG. 2 refers to the input embedding. The first sequence before the [SEP] token can be any contiguous span of climate data at different granularities. Furthermore, as described in more detail in connection with the spatiotemporal location embedding layer 115, tokens have learned embeddings that indicate whether a token belongs to the first or second part of the climate data pair.

[0029] As will be described in more detail in connection with FIG. 3, the input embedding is based on multiple embedding vectors, including, for example, a location embedding vector, a seasonality embedding vector, and a climate attribute embedding vector.

[0030] Pre-training of the transformer-based neural network is unsupervised, and the training data includes climate forecast data 102, 104, or historical weather data 106, or a combination thereof. Masked Climate Forecast (in FIG. 2, masked climate methods “Mask CM 209-1 and 209-2”) and Next Climate Forecast (in FIG. 2, Next Climate Forecast “NCP 208”) are unsupervised methods used for training. In one or more embodiments, given input climate data, the Masked Climate Forecast layer 112 randomly masks portions of the input forecast and uses that context to predict the masked climate tokens.

[0031] In a next-generation climate forecast task, given climate forecast data for two timestamps (e.g., timestamps Ta and Tb), the next-generation climate forecast layer 114 predicts whether the climate forecast data for the second timestamp (e.g., timestamp Tb) follows the climate forecast data for the first timestamp (e.g., timestamp Ta). During training, a certain percentage of the time, the climate forecast data for the second timestamp correctly follows the climate forecast data for the first timestamp, while the remaining percentage of the time, the climate forecast data for the second timestamp is a random forecast from the corpus. The masked climate forecast and next-generation climate forecast tasks are combined, and a transformer-based neural network model is trained with a combined loss function 130 (see FIG. 1 ). The loss function 130 may include, for example, a triplet loss function, a cross-entropy loss function, a reconstruction cost loss function, or a data loss function, or a combination thereof. This training relies on an autoregressive model, which is a time series model that uses observations of previous time steps as input to a regression algorithm to predict the value of the next time step.

[0032] 2 and 3, in one example of a spatiotemporal location embedding 315, the input embedding "E" 205 is based on multiple embedding vectors, including, for example, one or more location embedding vectors (POS1) 316, one or more seasonal embedding vectors (SEA1) 317, and one or more climate attribute (CA1) embedding vectors 318. "TC" in FIGS. 2 and 3 refers to climate tokens 203 and 303. Referring to FIG. 4, climate tokens 403 and a representation 400 of a particular one of the climate tokens 403 (TC1) are depicted. In accordance with this embodiment, time and spatial information is associated with each climate token. The climate tokens (TC1, TC2, TC3, ..., TC N) can represent climate data using different temporal and spatial granularities. Climate tokens TC can include historical observed climate data, forecasts, hindcasts, or a combination thereof. In one or more embodiments, a set of derived features (e.g., histograms, ranges, quantiles, etc.) is constructed from the probabilistic nature of seasonal forecasts. For example, FIGS. 6A and 6B illustrate temperature forecast representations using histograms 601 and 602, which include uncertainties associated with inputs. For example, as described above, weather forecast data 102 may include predictions within specific uncertainty ranges for items such as temperature, humidity, or precipitation, or combinations thereof, but are not necessarily limited to these items. As shown in histograms 601 and 602, temperatures (e.g., 10°C to 11°C, 11°C to 12°C, ..., 19°C to 20°C) are associated with different probabilities. The uncertainty representation layer 113 includes an uncertainty-aware model that generates uncertainty scores for climate change and catastrophic event parameters.

[0033] Histogram 601 in FIG. 6A shows an example of the distribution of temperature forecasts from multiple ensemble models. Each ensemble for each climate variable is created by running multiple simulations with varying initial conditions for the climate model, resulting in uncertain forecasts. For example, the European Centre for Medium-Range Weather Forecasts (ECMWF) seasonal-scale forecasts include 50 ensembles for each climate attribute for up to six months ahead, updated monthly. Histogram 602 in FIG. 6B shows an example of representing the uncertainty of temperature fluctuations while analyzing multiple ensembles using a histogram-based approach. The percentage values ​​shown represent the degree of agreement between the ensemble forecasts. The percentage values ​​indicate high uncertainty due to poor agreement across the ensembles. In this way, uncertainty in climate forecasts can be encoded while training the machine learning model in the feature reconstruction engine 110.

[0034] Referring back to FIG. 4 , relationship constraints, e.g., minimum, maximum, average, etc., and hierarchical constraints, e.g., hourly, daily, etc., may be incorporated into climate tokens (TCs). For example, in the representation 400 of climate token TC1, climate token TC1 includes minimum and maximum temperatures or humidity or a combination thereof at different times, daily precipitation values, average wind speed, and air pollution data, traffic data, and PDFs of minimum temperatures, precipitation, and extreme events. The masked climate forecast layer 112 utilizes surrounding left and right climate token information to learn a latent representation of the masked climate token. The feature reconstruction engine 110 attempts to predict the climate token by minimizing a surrogate loss function 130 that takes into account the predictability of climate attributes (e.g., low for precipitation compared to temperature) and the relationship and hierarchical constraints described above.

[0035] 2 and 3 , the spatiotemporal location embedding layer 115 captures different types of embeddings, including spatiotemporal features in the embedding space, while training a transformer-based neural network machine learning model. The location embeddings 316 may capture two different types of location characteristics: (i) location-specific, or (ii) data-specific, or a combination thereof. Location-specific embeddings capture location of climate attributes, such as, but not necessarily limited to, global positioning system (GPS) location, city, region, state, data resolution, etc. Data-specific embeddings capture data-specific features, such as, but not necessarily limited to, climate zone, agricultural vs. non-agricultural area, urban vs. rural area, residential vs. commercial area, etc. In one or more embodiments, the location embedding is a function of the coordinate system.

[0036] The seasonal embedding 317 corresponds to the inherent characteristics of time trends in climate data. The seasonal embedding facilitates learning the temporal trend changes of climate geospatial data while learning climate representations during the pre-training state. The seasonal embedding can be specified at multiple different granularities, such as, but not necessarily limited to, diurnal, weekly, seasonal, and yearly. The climate attribute embedding 318 captures the latent representation of geospatial climate attributes. In one or more embodiments, the timestamp embedding is the sum of an exogenous factor embedding, a location embedding, and a temporal embedding. The CLS embedding 319 refers to learning an overall vector representation across all climate tokens (TCs) to generate a climate attribute embedding for a specific time period (e.g., day, week, month, etc.). In FIG. 2, T i and T i '(207) refers to the final hidden representation of token i of the pair of climate data at timestamps Ta and Tb.

[0037] Referring back to FIG. 1 , the climate-aware forecasting engine 120 fine-tunes a trained transformer-based neural network machine learning model to perform specific climate-aware forecasts in connection with implementing various downstream applications, such as, but not necessarily limited to, demand forecasting and lead-time forecasting at retail nodes in a supply chain. In one or more embodiments, the same pre-trained parameters are used for multiple downstream tasks, with modifications corresponding to how the input and output layers are used. The transformer-based neural network machine learning model is initialized with the pre-trained parameters, and the climate-aware forecasting engine 120 fine-tunes the parameters for the desired downstream tasks. For example, at the input, the two parts of a climate data pair may differ depending on the task. For example, the granularity, location, type of weather data (e.g., temperature, precipitation, extreme events, etc.), temporal data, etc. may vary corresponding to a given downstream task. At the output, the token representation is used in the output layer for token-level tasks.

[0038] The feature reconstruction engine 110 uses trained machine learning models to encode complex spatiotemporal climate data into vector representations so that the climate-aware forecasting engine 120 can efficiently solve climate-aware forecasting use cases across different domains, regions, or time frames, or combinations thereof.

[0039] As described herein, the climate2vec model uses climate data (see S i ) time series, a pre-trained transformer-based climate model uses spatiotemporal location encoding and a climate masking strategy to estimate a d-dimensional vector representation of the climate data at multiple geographic locations. In the context of the climate-aware forecast engine 120, the climate2vec model generates pre-trained climate embeddings that can be used to learn other models to solve downstream tasks.

[0040] Advantageously, the climate2vec model can encode spatiotemporal relationships of climate data into a latent space without explicitly requiring a labeled dataset. This embodiment removes the dependency on downstream tasks, allowing the climate embeddings to be used to solve downstream tasks and fine-tuned without having to train latent representations from scratch using climate data.

[0041] Referring to operational flow 500 in FIG. 5 , climate forecast data 501 is input to a feature reconstruction engine 510, which may be the same as or similar to feature reconstruction engine 110. After analysis of the climate forecast data 501 by feature reconstruction engine 510, the pre-trained machine learning model and parameters output from feature reconstruction engine 510 are used by a climate-aware forecast engine 520, which may be the same as or similar to climate-aware forecast engine 120, for task-specific fine-tuning. The output from climate-aware forecast engine 520 is provided to a feed-forward network (FFN) 580 and inverse normalization and differencing layers 551 and 552. Time series windowed data 545 is provided to differencing layer 550, the output of which is provided to normalization layer 560 and inverse differencing layer 552. The output from normalization layer 560 is provided to FFN 580 and inverse normalization layer 551. The output from inverse differencing layer 552 comprises a prediction 590. As can be seen from the operational flow 500 in Figure 5, as part of the first stage, a compact representation of climate data is learned in a feature reconstruction engine 510, independent of the downstream forecasting task. In the second stage, a climate-aware forecasting engine 520 solves the downstream task by using climate embeddings (e.g., climate2vec) by using a pre-trained transformer-based machine learning model (learned as part of the first stage), resulting in forecasts 590 (e.g., retail demand, peak load, etc.) that are used to solve the downstream problem.

[0042] The differencing and normalization layers 550 and 560 are used to efficiently represent time series features, such as historical product demand (e.g., sales), and enable transfer between time series. The differencing layer 550 captures relative trends within a time series window, while the normalization layer serves to window-normalize each data point, transforming each input window to a comparable scale across multiple inputs.

[0043] With respect to the differencing layer 550, for a time window w=(x1,...,xn), the differentiated window w_diff is defined as w_diff=(x2-x2,...,xi-x(i-1),...,xn-x(n-1)). The procedure can be reversed by preserving x1.

[0044] With respect to the normalization layer 560, for a time window w=(x1,...,xn), μw refers to its empirical mean (σw refers to its empirical standard deviation without Bessel correction). The normalization window norm is defined as w_norm=((x1-μw) / σw,...,(xi-μw) / σw,...,(xn-μw) / σw). The normalization can be inverted by propagating μw and σw.

[0045] According to one embodiment, the feature reconstruction engine 510 is used to efficiently encode spatiotemporal climate features using a Transformer-based model that is pre-trained using climate masking techniques. In one or more embodiments, the feature reconstruction engine 510 is pre-trained using a generic corpus of global climate data and may require fine-tuning by updating the weights of the last few layers of the Transformer model for downstream tasks such as retail demand forecasting, renewable energy forecasting, etc.

[0046] FFN580 is trained by concatenating window-normalized time series features and compact climate features generated using a feature reconstruction engine to solve downstream tasks.

[0047] Considering these and other features described herein, FIG. 7 illustrates a climate data modeling methodology 700 that encodes spatiotemporal climate data into a vector representation to efficiently solve climate-aware forecasting use cases.

[0048] In step 702, climate data is received that includes multiple spatial components and multiple temporal components. The multiple spatial components include multiple geographic locations and the multiple temporal components include multiple time periods. The multiple spatial components and the multiple temporal components include different granularities. The climate data further includes one or more climate attributes.

[0049] In step 704, a portion of the climate data is masked. A latent representation of the masked portion of the climate data is learned by utilizing one or more adjacent unmasked portions of the climate data. In learning the latent representation of the masked portion of the climate data, a loss function that takes into account one or more constraints is minimized.

[0050] In step 706, a machine learning model is trained, the training based at least in part on the masked portion of the climate data. The machine learning model includes a transformer-based neural network.

[0051] At step 708, a vector representation of the climate data is generated via the machine learning model, the vector representation including one or more d-dimensional vector representations of the climate data at a plurality of geographic locations, where d is an integer.

[0052] In the method, location embedding is performed in conjunction with training the machine learning model to capture location characteristics of the climate data. The location embedding comprises a location-specific embedding procedure, where the location characteristics include location information for one or more locations associated with the climate data. The location embedding may also include a data-specific embedding, where the location characteristics include climate zone information for one or more climate zones associated with the climate data.

[0053] In the method, seasonal embedding may be performed in conjunction with training a machine learning model to capture time trend characteristics of the climate data, and climate attribute embedding may be performed in conjunction with training the machine learning model to capture one or more latent space representations of the climate data.

[0054] According to this embodiment, the machine learning model is fine-tuned to perform one or more enterprise-specific forecasting tasks, where the multiple time components have multiple timestamps and the machine learning model is used to predict the weather associated with a timestamp subsequent to the last timestamp of the multiple timestamps.

[0055] The techniques depicted in FIGS. 1-7 may also include providing a system, as described herein, including separate software modules, each embodied on a tangible computer-readable, recordable storage medium. For example, all of the modules (or any subset thereof) may be on the same medium, or each may be on a different medium. The modules may include any or all of the components shown in the figures or described herein, or a combination thereof. In one embodiment of the present invention, the modules may be executed, for example, on a hardware processor. Method steps may then be performed using the separate software modules of the system, as described above, and executed on the hardware processor. Additionally, a computer program product may include a tangible computer-readable, recordable storage medium having code adapted to be implemented for performing at least one method step described herein, including provisioning the system with the separate software modules.

[0056] 1-7 may be implemented via a computer program product that may include computer usable program code stored on a computer readable storage medium in a data processing system, the computer usable program code being downloaded over a network from a remote data processing system. Also, in one embodiment of the present invention, the computer program product may include computer usable program code stored on a computer readable storage medium at a server data processing system, the computer usable program code being downloaded over a network to the remote data processing system for use on the computer readable storage medium with the remote system.

[0057] An embodiment of the present invention or elements thereof may be implemented in the form of an apparatus including a memory and at least one processor, coupled to the memory, configured to perform the exemplary method steps.

[0058] Furthermore, an embodiment of the present invention may utilize software executed on a computer or workstation. Referring to FIG. 8 , such an implementation may employ, for example, a processor 802, memory 804, and an input / output interface formed, for example, by a display 806 and a keyboard 808. As used herein, the term “processor” is intended to include any processing device, such as one including other forms of processing circuitry, such as a multi-core CPU, a GPU, an FPGA, or one or more ASICs, or a combination thereof. Furthermore, the term “processor” may refer to two or more individual processors. The term “memory” is intended to include memory associated with a processor (e.g., a CPU, a GPU, an FPGA, an ASIC, etc.), such as, for example, RAM (random access memory), ROM (read-only memory), fixed storage (e.g., a hard disk), removable storage (e.g., a diskette), flash memory, etc. Also, as used herein, the phrase “input / output interface” is intended to include, for example, mechanisms for inputting data into a processing unit (e.g., a mouse) and mechanisms for providing results associated with a processing unit (e.g., a printer), etc. The processor 802, memory 804, display 806 and input / output interfaces such as keyboard 808 may be interconnected, for example, via a bus 810, as part of a data processing unit 812. For example, suitable interconnections via the bus 810 may also be provided to a network interface 814, such as a network card, which may be provided for interfacing with a computer network, and a media interface 816, such as a diskette or CD-ROM drive, which may be provided for interfacing with media 818.

[0059] Thus, computer software containing instructions or code for carrying out the methodologies of embodiments of the present invention as described herein may be stored in an associated memory device (e.g., ROM, fixed memory or removable memory) and loaded in part or in whole (e.g., into RAM) and executed by a CPU when ready for use. Such software may include, but is not limited to, firmware, resident software, microcode, etc.

[0060] A data processing system suitable for storing and / or executing program code includes at least one processor 802 coupled directly or indirectly to memory elements 804 via a system bus 810. The memory elements may include local memory employed during the actual implementation of the program code, bulk storage, and cache memory that provides temporary storage of at least some of the program code to reduce the number of times the code is retrieved from bulk storage during implementation.

[0061] Input / output or I / O devices (including but not limited to keyboard 808, display 806, pointing device, etc.) can be coupled to the system directly (e.g., via bus 810) or through intervening I / O controllers (omitted for clarity).

[0062] Network adapters, such as network interface 814, may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are just a few of the types of network adapters currently available.

[0063] As used herein, including in the claims, a "server" includes a physical data processing system (e.g., system 812 as shown in FIG. 8) running a server program. It should be understood that such a physical server may or may not include a display and keyboard.

[0064] The present invention may be a system, method, or computer program product, or combination thereof, integrated at any possible level of technical detail. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions that cause a processor to perform aspects of the present invention.

[0065] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or raised structures with instructions recorded in grooves, and any suitable combination of the foregoing. Computer-readable storage medium, as used herein, should not be construed as a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0066] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof). The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing / processing device.

[0067] The computer-readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk® or C++, and procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer readable program instructions to personalize the electronic circuitry by utilizing state information of the computer readable program instructions to perform aspects of the present invention.

[0068] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0069] These computer-readable program instructions can be provided to a computer processor or other programmable data processing apparatus to produce a machine, such that the instructions, executed by the computer processor or other programmable data processing apparatus, create means for implementing the functions / acts specified in the blocks of the flowcharts and / or block diagrams. These computer-readable program instructions can also be stored on a computer-readable storage medium, such that the instructions can instruct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium having the instructions stored thereon comprises an article of manufacture including instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0070] The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device and cause the computer, other programmable apparatus, or other device to perform a series of operational steps to produce a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram blocks.

[0071] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions shown in the blocks may occur out of the order shown in the figures. For example, two blocks shown in succession may actually be realized as a single step, or may be executed concurrently, substantially concurrently, in a partially or fully time-overlapping manner, or the blocks may possibly be executed in reverse order depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by special-purpose hardware-based systems that perform the specified functions or operations or execute a combination of special-purpose hardware and computer instructions.

[0072] It should be noted that any of the methods described herein may include the additional step of providing a system including separate software modules embodied on a computer-readable storage medium. The modules may include, for example, any or all of the components detailed herein. The method steps may then be performed using the separate software modules or sub-modules, or a combination thereof, of the system, as described above, and executed on the hardware processor 802. Additionally, a computer program product may include a computer-readable storage medium having code adapted to be implemented for the execution of at least one method step described herein, including provisioning the system with the separate software modules.

[0073] In any case, it should be understood that the components illustrated herein may be implemented in various forms of hardware, software, or combinations thereof, such as, for example, an application specific integrated circuit (ASIC), a functional circuit, a suitably programmed digital computer with associated memory, etc. Given the teachings of the invention provided herein, those of ordinary skill in the relevant art will be able to contemplate other implementations of the components of the invention.

[0074] Although this disclosure includes detailed descriptions of cloud computing, it should be understood that implementation of the teachings described herein is not limited to cloud computing environments. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0075] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with the service provider. The cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0076] The characteristics are as follows:

[0077] On-demand self-service: Cloud customers can unilaterally provide computing capacity, such as server time and network storage, automatically as needed, without the need for human interaction with the service provider.

[0078] Wide network access: Capabilities are available across the network and can be accessed through standard mechanisms, facilitating use by different types of thin or thick client platforms (eg, cell phones, laptops, and PDAs).

[0079] Resource Pooling: A provider's computing resources are pooled to serve multiple customers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. There is a sense of location independence in that customers generally have no control or knowledge over the exact location of the resources provided, although location may be identifiable at a higher level of abstraction (e.g., country, state, or data center).

[0080] Rapid Scalability: Capacity is provided quickly and elastically, sometimes automatically, with the ability to quickly scale out or quickly unwind and quickly scale in. To the consumer, the capacity available for provisioning often appears unlimited, and can be purchased at any time and in any amount.

[0081] Metered Services: Cloud systems automatically control and optimize resource usage using metering capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage is monitored, controlled, and reported, providing transparency to both providers and consumers of the services used.

[0082] The service model is as follows:

[0083] Software as a Service (SaaS): The consumer is offered the ability to use a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through a thin-client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application functions, with the possible exception of limited user-specific application configuration settings.

[0084] Platform as a Service (PaaS): The ability offered to consumers is to deploy consumer-created or acquired applications on a cloud infrastructure, written using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but do have control over the deployed applications and, in some cases, the configuration of the environment that hosts the applications.

[0085] Infrastructure as a Service (IaaS): The capability provided to a customer is to provide processing, storage, network, and other underlying computing resources on which the customer can deploy and run any software, which may include operating systems and applications. The customer does not manage or control the underlying cloud infrastructure, but does have control over the operating systems, storage, deployed applications, and, in some cases, limited control over selected networking components (e.g., host firewalls).

[0086] The deployment model is as follows:

[0087] Private Cloud: Cloud infrastructure is operated solely for one organization. A private cloud may be managed by that organization or a third party and may exist on-premise or off-premise.

[0088] Community Cloud: Cloud infrastructure is shared by several organizations to support a specific community of shared interests (e.g., mission, security requirements, policies, and compliance considerations). The community cloud may be managed by those organizations or a third party and may exist on-premises or off-premises.

[0089] Public Cloud: Cloud infrastructure is made available to the general public or large industry organizations and is owned by an organization that sells cloud services.

[0090] Hybrid Cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain their own entities but are tied together by standardized or proprietary technologies that allow for data and application portability (e.g., cloud bursting to balance load between clouds).

[0091] Cloud computing environments are service-oriented, emphasizing statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0092] Referring now to FIG. 9 , an exemplary cloud computing environment 950 is shown. As shown, the cloud computing environment 950 includes one or more cloud computing nodes 910 with which local computing devices used by cloud users, such as a personal digital assistant (PDA) or cellular phone 954A, a desktop computer 954B, a laptop computer 954C, or an automobile computer system 954N, or combinations thereof, may communicate. The nodes 910 may communicate with each other. They may be physically or virtually grouped in one or more networks (not shown), such as a private cloud, a community cloud, a public cloud, or a hybrid cloud, or combinations thereof, as described hereinabove. This enables the cloud computing environment 950 to provide infrastructure, platform, or software, or combinations thereof, as a service, without requiring the cloud user to maintain resources on the local computing device. It should be understood that the types of computing devices 954A-N illustrated in FIG. 9 are intended to be exemplary only, and that the computing node 910 and cloud computing environment 950 can communicate with any type of computerized device through any type of network or network-addressable connection (e.g., using a web browser) or both.

[0093] 10, a set of functional abstraction layers provided by cloud computing environment 950 (FIG. 9) is shown. It should be understood upfront that the components, layers, and functions shown in FIG. 10 are intended to be exemplary only, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0094] Hardware and software layer 1060 includes hardware and software components. Examples of hardware components include mainframe 1061, RISC (Reduced Instruction Set Computer) architecture-based servers 1062, servers 1063, blade servers 1064, storage devices 1065, and networks and networking components 1066. In some embodiments, software components include network application server software 1067 and database software 1068.

[0095] The virtualization layer 1070 provides an abstraction layer from which the following example virtual entities can be provided: virtual servers 1071, virtual storage 1072, virtual networks including virtual private networks 1073, virtual applications and operating systems 1074, and virtual clients 1075. In one example, management layer 1080 may provide the following functions: Resource provisioning 1081 provides dynamic procurement of computing resources and other resources utilized to perform tasks within the cloud computing environment. Metering and pricing 1082 provides cost tracking and charging or billing for the consumption of resources as they are used within the cloud computing environment.

[0096] In one example, these resources may include application software licenses. Security provides for the identification of cloud users and tasks, as well as the protection of data and other resources. A user portal 1083 provides users and system administrators with access to the cloud computing environment. Service level management 1084 provides cloud computing resource allocation and management to ensure required service levels are met. Service level agreement (SLA) planning and fulfillment 1085 provides for the advance arrangement and procurement of cloud computing resources in anticipation of future requirements according to SLAs.

[0097] The workload tier 1090 provides examples of functionality for which a cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this tier include mapping and navigation 1091, software development and lifecycle management 1092, virtual classroom instruction delivery 1093, data analytics processing 1094, transaction processing 1095, and climate data modeling and forecasting 1096, in accordance with one or more embodiments of the present invention.

[0098] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It is further understood that the terms "comprises" and / or "comprising," when used herein, specify the presence of stated features, steps, operations, elements, or components, or combinations thereof, but do not exclude the presence or addition of other features, steps, elements, components, or groups thereof, or combinations thereof.

[0099] At least one embodiment of the present invention may provide beneficial effects, such as, for example, a framework (e.g., a set of one or more framework configurations) for learning spatiotemporal uncertainty-aware climate vector representations for use in connection with climate-aware forecasting. Unlike the prior art, this embodiment provides for pre-training a deep bidirectional transformer model based on multi-faceted contextual climate data to capture spatiotemporal variability in input geospatial climate data.

[0100] This embodiment advantageously enables attention mechanisms for deep learning. For example, this embodiment enables pre-training of deep transformer-based bidirectional representations using a masked climate model. Furthermore, this embodiment utilizes spatiotemporal location embeddings to efficiently capture geospatial data characteristics (e.g., location and data-specific features) for pre-training of the climate2vec model.

[0101] As an additional advantage, the present embodiment fine-tunes the climate2vec model for application to climate-aware use cases of large-scale downstream tasks, such as, but not limited to, climate-aware demand forecasting, climate-aware energy forecasting, and other enterprise-related tasks. This fine-tuning is done by retraining the last few output layers of the climate2vec model for task-specific climate-aware forecasting use cases.

[0102] In one or more embodiments, the spatiotemporal climate forecast output is represented and transformed using a neural network by learning encoded representations of medium- to long-term seasonal climate forecasts and past observations. Climatology (e.g., the study of climate and its changes over time) is used in conjunction with model training to mask climate forecasts at certain timestamps (e.g., hours, days, weeks, etc.) to predict the original climate forecast. Spatiotemporal location encoding can easily capture complex spatiotemporal climate variability and the characteristics of different geospatial data (e.g., air pollution and traffic data).

[0103] Additionally, machine learning models would benefit from enforcing hierarchical constraints to efficiently capture the uncertainty information associated with seasonal forecasts (e.g., mean, variance, standard deviation, quartile distribution).

[0104] The description of various embodiments of the present invention is presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been selected to best explain the principles of the embodiments, practical applications of, or technical improvements to, the technology found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer program comprising program instructions, the program instructions being executed by one or more processors. receiving climate data comprising a plurality of spatial components and a plurality of temporal components; masking a portion of the climate data; training a machine learning model, the training based at least in part on the masked portion of the climate data; generating a vector representation of the climate data via the machine learning model; a computer program executable by said one or more processors to cause said one or more processors to perform

2. The computer program product of claim 1 , wherein the plurality of spatial components comprises a plurality of geographic locations and the plurality of temporal components comprises a plurality of time periods.

3. The computer program product of claim 2 , wherein the vector representation comprises one or more d-dimensional vector representations of the climate data at the plurality of geographic locations.

4. The computer program of claim 1 , wherein the machine learning model comprises a transformer-based neural network.

5. 2. The computer program product of claim 1, wherein the program instructions further cause the one or more processors to perform a location embedding procedure in connection with the training of the machine learning model to capture location characteristics of the climate data.

6. The computer program product of claim 5 , wherein the location embedding procedure comprises a location-specific embedding procedure, and the location characteristics include location information for one or more locations associated with the climate data.

7. The computer program product of claim 5 , wherein the location embedding procedure comprises a data-specific embedding procedure, and the location characteristics include climate zone information for one or more climate zones associated with the climate data.

8. 2. The computer program product of claim 1, wherein the program instructions further cause the one or more processors to perform a seasonality embedding procedure in connection with the training of the machine learning model to capture time trend characteristics of the climate data.

9. 2. The computer program product of claim 1 , wherein the program instructions further cause the one or more processors to perform a climate attribute embedding procedure in connection with the training of the machine learning model to capture one or more latent space representations of the climate data.

10. 10. The computer program product of claim 1, wherein the program instructions further cause the one or more processors to fine-tune the machine learning model to perform one or more enterprise-specific predictive tasks.

11. The computer program product of claim 1 , wherein the plurality of spatial components and the plurality of temporal components comprise different granularities.

12. The computer program of claim 1 , wherein the climate data further comprises one or more climate attributes.

13. 2. The computer program product of claim 1, wherein the program instructions further cause the one or more processors to perform a step of learning a latent representation of the masked portion of the climate data by utilizing one or more adjacent unmasked portions of the climate data.

14. 14. The computer program product of claim 13, wherein in learning the latent representation of the masked portion of the climate data, the program instructions cause the one or more processors to perform a procedure that minimizes a loss function that takes into account one or more constraints.

15. 15. The computer program product of claim 1, wherein the plurality of time components comprises a plurality of timestamps, and the program instructions further cause the one or more processors to perform a step of using the machine learning model to predict a climate associated with a timestamp subsequent to a final timestamp in the plurality of timestamps.

16. receiving climate data comprising a plurality of spatial components and a plurality of temporal components; masking a portion of the climate data; training a machine learning model, the training based at least in part on the masked portion of the climate data; generating a vector representation of the climate data via the machine learning model; Equipped with A computer-implemented method performed by at least one processing device having a processor coupled to a memory when executing program code.

17. 17. The computer-implemented method of claim 16, further comprising performing location embedding in connection with the training of the machine learning model to capture location characteristics of the climate data.

18. 18. The computer-implemented method of claim 16 or 17, further comprising learning a latent representation of the masked portion of the climate data by utilizing one or more adjacent unmasked portions of the climate data.

19. 1. An apparatus comprising at least one processing device having a processor coupled to a memory, wherein, when executing program code, the at least one processing device: receiving climate data having a plurality of spatial components and a plurality of temporal components; masking a portion of the climate data; training a machine learning model, said training based at least in part on the masked portion of the climate data; generating a vector representation of the climate data via the machine learning model; The apparatus is configured to:

20. 20. The apparatus of claim 19, wherein upon executing the program code, the at least one processing device is further configured to learn a latent representation of the masked portion of the climate data by utilizing one or more adjacent unmasked portions of the climate data.

Images