Space-time embedded NOx inversion method and device under Transform architecture

Abstract

The embodiment of the invention discloses a space-time embedded NOx inversion method and device under a Transform architecture, and the method comprises the steps: constructing a space-time large language model which comprises a space-time embedding layer, a fusion convolution layer, a large language model (LLM) layer and a regression convolution layer; wherein the spatio-temporal embedding layer responds to input screened channel data and outputs spatio-temporal representation, the fusion convolution layer is used for uniformly integrating output of the spatio-temporal embedding layer, obtaining spatio-temporal data and outputting the spatio-temporal data to the LLM layer, and the LLM layer is used for capturing a long-term dependency relationship in the spatio-temporal data by using a large language model and extracting deep features; outputting to a regression convolution layer; the regression convolution layer is used for converting the output of the LLM layer into NOx concentration; and training the space-time large language model according to a set training process, and realizing inversion of the NOx concentration by using the trained space-time large language model. According to the method, the defects of a traditional method in spatial dimension modeling and data quality processing are overcome.

Description

Technical Field

[0001] This application relates to a spatiotemporal embedding NO under a Transformer architecture x The invention discloses an inversion method and a device, belonging to the technical field of air pollution monitoring and remote sensing inversion. Background Art

[0002] AI models have been widely used in the field of meteorology. The world's mainstream AI meteorological models mainly focus on medium- and short-term weather forecasts to provide detailed short-term weather forecasts. Their models mainly include Huawei Cloud's Pangu-Weather, Nvidia's FourCastNet, Google DeepMind's GraphCast, Tsinghua University and China Meteorological Administration's NowcastNet, Fuxi model developed by Fudan University, and Fengwu model of Shanghai Artificial Intelligence Laboratory. Among them, Panu-Weather adopts a 3DEarth-specific Transformer structure, which replaces the dynamic core and physical parameterization scheme in the traditional numerical model based on a purely data-driven approach. This is a deep learning model using a standard encoder-decoder framework. The model is based on a hierarchical temporal aggregation method with a 3D Swin Transformer as the backbone network. It accepts the initial atmospheric state as input and outputs a future atmospheric state with the same structure as the initial state.

[0003] Large language models (LLMs) have shown excellent performance on time series analysis tasks, including prediction, classification, anomaly detection, interpolation, few-hotspot learning, and zero-shot learning. For example, TEMPO-GPT combines instant engineering and seasonal trend decomposition in generating pre-trained transformers (GPTs). TIME-LLM reprograms LLMs for time series prediction, maintaining the integrity of the backbone language model. OFA used the frozenGPT2 model in various key tasks of time series analysis and concluded that LLMs perform better in time series analysis tasks. TEST generates various embeddings of time series tags and performs time series prediction and classification tasks. The above models mainly model the time dimension. Another study proposed a new traffic prediction framework, which effectively captures the spatiotemporal correlations in traffic prediction by emphasizing the use of partially frozen attention LLMs, and its effectiveness has been verified on real traffic datasets.

[0004] In addition, the Transformer architecture has been widely used in meteorological research due to its powerful sequence modeling and long-distance dependency capture capabilities. In addition to the well-known large meteorological models, Transformers have also been used for prediction and analysis in the field of pollutants. For example, the Informer model, which is suitable for long-series time series prediction, is mainly aimed at general time series data. The Temporal Fusion Transformer (TFT) model is used for interpretable multi-step time series prediction. The TFT model has been applied to multiple fields, including meteorology, and can be applied to the prediction of air pollutant concentrations, capturing temporal patterns and covariate relationships. Research on the direct application of the Transformer architecture to pollutant prediction is still emerging. In summary, the model shows great potential in effectively modeling spatiotemporal dynamics, which helps to improve the accuracy and interpretability of air quality predictions.

[0005] Surface nitrogen dioxide (NO 2 ) is an important air pollutant that can have serious impacts on human health and the environment. 2 Monitoring methods mainly rely on ground observation stations, but these methods have the problems of limited coverage and high cost. With the development of remote sensing technology, the use of satellite remote sensing data for surface NO 2 The inversion of NO concentration has become an effective monitoring method. However, traditional remote sensing inversion methods are usually based on statistical models or empirical formulas, which are difficult to accurately capture NO 2 The complex nonlinear relationship between the inversion results and the remote sensing data leads to limited accuracy.

[0006] In recent years, artificial intelligence technology, especially deep learning, has made significant progress in image recognition, natural language processing and other fields. LLMs such as GPT and BERT learn rich semantic information and knowledge representation by pre-training a large amount of text data, showing strong feature extraction capabilities. The success of the above technologies has inspired the application of deep learning methods in remote sensing data analysis and surface NO 2 Inversion research.

[0007] Although existing AI models (such as Transformer and LLM architectures) have made progress in weather forecasting and time series analysis, most models focus mainly on modeling the temporal dimension and pay less attention to the spatial dimension. 2, whose distribution and concentration are affected by complex spatiotemporal dynamics, and simple time series models cannot fully capture spatial correlations. There are some challenges in directly applying deep learning models to remote sensing data inversion. First, remote sensing data are characterized by high dimensionality and strong spatiotemporal correlation, and traditional deep learning models are difficult to effectively process such spatiotemporal data. Secondly, the inversion process of atmospheric pollutants is affected by complex atmospheric physical and chemical processes, and the model needs to be able to understand and utilize these physical knowledge. In addition, the acquisition of remote sensing data is affected by factors such as cloud cover and instrument errors, resulting in uneven data quality and increasing the difficulty of model training and inversion. Although Transformer and LLM have been applied in some fields, for surface NO 2 Dedicated models for inversion are still scarce, and existing research is insufficient to address the problem of NO 2 Specific challenges in inversion. Summary of the invention

[0008] In order to solve the above technical problems, the embodiments of the present application respectively provide a spatiotemporal embedding NO under the Transformer architecture. x Inversion methods and devices, by developing spatiotemporal embedded large language model (ST-LLM) and maximum relevance minimum redundancy (mRMR) channel screening technology, improve the processing ability of remote sensing data and accurately capture surface NO 2 The complex temporal and spatial variation of NO concentration can be achieved 2 The high-precision inversion solves the shortcomings of traditional methods in spatial dimension modeling and data quality processing, and provides effective technical support for air pollution monitoring and environmental protection.

[0009] Specifically, this application preprocesses the remote sensing data through the spatiotemporal embedding layer to extract spatial and temporal features; uses the LLM layer and the Transformer module therein to learn the deep representation of the remote sensing data; and finally uses the fully connected regression layer to map the deep features to the surface NO x concentration, achieving high-precision NO x Inversion. In addition, this application also considers the constraints of atmospheric physical and chemical processes, enhancing the physical interpretability of the model. ST-LLM is more robust to data and can handle missing values ​​and noise in remote sensing data. Through spatiotemporal embedding and deep feature learning, the model can extract effective information from incomplete data. This method is effective for surface NO x The design is based on the inversion needs of NO, filling the gap in the existing models in this field and providing a more suitable x Monitoring solution. By combining spatiotemporal embedding and LLM, the model is effective in capturing NO x The results are excellent in complex nonlinear relationships of concentration. At the same time, the introduction of physical constraints makes the results more credible and helps in decision-making on environmental protection and air pollution control.

[0010] Therefore, the purpose of this application is to overcome the limitations of traditional methods and provide effective technical support for air pollution monitoring and environmental protection.

[0011] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0012] According to one aspect of the embodiments of the present application, a spatiotemporal embedding NO is provided under a Transformer architecture. x Inversion method and device, the method comprising:

[0013] Construct a spatiotemporal large language model, which includes a spatiotemporal embedding layer, a fused convolution layer, an LLM layer and a regression convolution layer; wherein the spatiotemporal embedding layer responds to the input filtered channel data and outputs a spatiotemporal representation, the fused convolution layer is used to integrate the output of the spatiotemporal embedding layer to obtain spatiotemporal data and output it to the LLM layer, the LLM layer is used to use a large language model to capture the long-term dependencies in the spatiotemporal data, and extract deep features, and output them to the regression convolution layer; the regression convolution layer is used to convert the output of the LLM layer into NO x concentration;

[0014] The spatiotemporal large language model is trained according to the set training process, and the trained spatiotemporal large language model is used to implement NO x Inversion of concentration; wherein, the spatiotemporal language model learns remote sensing data and NO during training. x The training process includes forward propagation, loss calculation, back propagation and iterative optimization. The forward propagation includes input data passing through the spatiotemporal embedding layer, the fusion convolution layer, the LLM layer and the regression convolution layer in sequence. The loss calculation includes calculating the loss between the model output and the true label according to the loss function. The back propagation includes updating the parameters according to the gradient of the loss function relative to the model parameters. The iterative optimization includes iterative optimization and repeating forward propagation, loss calculation and back propagation until the set number of training rounds is reached, wherein one round of training is to perform one forward propagation, loss calculation and back propagation.

[0015] According to one aspect of the embodiments of the present application, a spatiotemporal embedding NO is provided under a Transformer architecture. x Inversion device, comprising:

[0016] The model building module is configured to build a spatiotemporal large language model, which includes a spatiotemporal embedding layer, a fused convolution layer, an LLM layer and a regression convolution layer; wherein the spatiotemporal embedding layer outputs a spatiotemporal representation in response to the input filtered channel data, the fused convolution layer is used to integrate the output of the spatiotemporal embedding layer to obtain spatiotemporal data and output it to the LLM layer, the LLM layer is used to use the large language model to capture the long-term dependencies in the spatiotemporal data, and extract deep features, and output them to the regression convolution layer; the regression convolution layer is used to convert the output of the LLM layer into NO x concentration;

[0017] The model training module is configured to train the spatiotemporal large language model according to a set training process, and implement NO x Inversion of concentration; wherein, the spatiotemporal language model learns remote sensing data and NO during training. x The training process includes forward propagation, loss calculation, back propagation and iterative optimization. The forward propagation includes input data passing through the spatiotemporal embedding layer, the fusion convolution layer, the LLM layer and the regression convolution layer in sequence. The loss calculation includes calculating the loss between the model output and the true label according to the loss function. The back propagation includes updating the parameters according to the gradient of the loss function relative to the model parameters. The iterative optimization includes iterative optimization and repeating forward propagation, loss calculation and back propagation until the set number of training rounds is reached, wherein one round of training is to perform one forward propagation, loss calculation and back propagation.

[0018] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: a controller; a memory for storing one or more programs, and when the one or more programs are executed by the controller, the controller implements the spatiotemporal embedding NO under the Transformer architecture described above. x Inversion method and apparatus.

[0019] According to one aspect of the embodiment of the present application, a computer-readable storage medium is further provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a computer, the computer executes the spatiotemporal embedding NO under the Transformer architecture. x Inversion method and apparatus.

[0020] According to one aspect of the embodiments of the present application, a computer program product or a computer program is also provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned spatiotemporal embedding NO under the Transformer architecture. x Inversion method and apparatus.

[0021] The technical solution provided in the embodiments of the present application has at least the following advantages:

[0022] This application can not only effectively extract the spatiotemporal characteristics of remote sensing data, but also accurately capture the surface NO through deep learning models. x The complex nonlinear relationship between the remote sensing data and the spatial and temporal large language model is also very robust to the data and can handle missing values ​​and noise in the remote sensing data. Overall, this application improves the accuracy and interpretability of the spatial and temporal inversion of pollutants and provides strong technical support for air pollution monitoring and environmental protection.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0025] Figure 1 It is a spatiotemporal embedding NO under the Transformer architecture shown in an exemplary embodiment of this application x Overall flow chart of the inversion method and apparatus;

[0026] Figure 2 It is a structural block diagram of a spatiotemporal large language model shown in an exemplary embodiment of the present application;

[0027] Figure 3 It is a spatiotemporal embedding NO under the Transformer architecture shown in an exemplary embodiment of this application x Structural diagram of the inversion device. DETAILED DESCRIPTION

[0028] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0029] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0030] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0031] The term "multiple" as used in this application refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0032] Glossary:

[0033] Transformer architecture is a deep learning model architecture based on multi-head attention mechanism. Its design greatly improves the efficiency of parallel computing and can effectively handle long-distance dependencies. The whole model consists of two parts: encoder and decoder. The input sequence is first mapped into a high-dimensional vector space through the embedding layer, and then the position information is added through positional encoding, so that the model can understand the order of the sequence. The multi-head self-attention mechanism divides the attention mechanism into multiple subspaces (called "heads") and calculates the attention in parallel to capture the relationship between different positions in the sequence. Next, the feed-forward neural network (FFN) performs further nonlinear transformation on the output of the attention layer. In order to train deeper networks and stabilize model performance, residual connections and layer normalization are added after each sublayer. This architecture enables Transformer to efficiently process sequence data and has achieved remarkable results in fields such as natural language processing.

[0034] See also Figure 1 , Figure 1 It is a spatiotemporal embedding NO under a Transformer architecture shown in an exemplary embodiment of the present application. x The overall flow chart of the inversion method and device. One aspect of the present application provides a spatial-temporal embedding NO under the Transformer architecture. x Inversion method, it should be noted that NO x represents nitrogen oxides, which include various compounds such as nitrous oxide (N 2 O), nitric oxide (NO), nitrogen dioxide (NO 2 ), nitrogen trioxide (N 2 O 3 ), dinitrogen tetroxide (N 2 O 4 ) and nitrogen pentoxide (N 2 O 5 ) etc. Except for nitrogen dioxide, other nitrogen oxides are extremely unstable. When exposed to light, moisture or heat, they turn into nitrogen dioxide and nitric oxide, and nitric oxide turns into nitrogen dioxide. Therefore, what people are exposed to in the occupational environment is a mixture of several gases, often called gun smoke, which are mainly nitric oxide and nitrogen dioxide, with nitrogen dioxide being the main component. Nitrogen oxides all have varying degrees of toxicity. This example uses surface NO 2 Taking concentration inversion as an example, this paper introduces the spatiotemporal embedding of NO x The specific process of the inversion method and device. Figure 1As shown, the method includes steps S100 to S200. The detailed description is as follows:

[0035] S100: Building a spatiotemporal language model.

[0036] See also Figure 2 , Figure 2 FIG. 1 is a structural diagram of a spatiotemporal large language model shown in an exemplary embodiment of the present application. Figure 2 As shown in the figure, the spatiotemporal large language model includes a spatiotemporal embedding layer, a fused convolution layer, an LLM layer and a regression convolution layer; wherein the spatiotemporal embedding layer responds to the input filtered channel data and outputs a spatiotemporal representation; the filtered channel data is obtained by filtering the remote sensing data of the channel through the maximum correlation minimum redundancy method, the fused convolution layer is used to integrate the output of the spatiotemporal embedding layer, obtain the spatiotemporal data and output it to the LLM layer, the LLM layer is used to use the large language model to capture the long-term dependency relationship in the spatiotemporal data, and extract deep features, and output them to the regression convolution layer; the regression convolution layer is used to convert the output of the LLM layer into NO x concentration.

[0037] For the sake of ease of description, the spatiotemporal large language model will be abbreviated as ST-LLM model in the following text.

[0038] In this embodiment, the ST-LLM model integrates a spatiotemporal embedding layer, a fused convolutional layer, an LLM layer, and a regression convolutional layer. The spatiotemporal embedding layer is responsible for converting the filtered channel data into a spatiotemporal representation, emphasizing the correlation between space and time. The fused convolutional layer integrates the output of the spatiotemporal embedding layer to capture the complex interaction between spatial and temporal features. The LLM layer uses a large language model (such as GPT or BERT) to capture long-term dependencies in spatiotemporal data and extract deep features. The regression convolutional layer converts the output of the LLM layer into surface NO 2 Finally, the ST-LLM framework was used to process and analyze the screened Himawari-8TOA channel data to estimate the surface NO 2 The model learns remote sensing data and surface NO 2 The nonlinear mapping relationship between the concentrations is used to achieve the surface NO 2 High-precision inversion.

[0039] For example, this embodiment uses the maximum correlation minimum redundancy (mRMR) technology to screen the remote sensing data of the Himawari-8 TOA (Top of Atmosphere) channel. Through this technology, it is possible to effectively select the NOx related data. 2 The channels with the highest concentration correlation and the lowest redundancy are selected to reduce unnecessary computation and improve model performance.

[0040] S200: Train the spatiotemporal large language model according to the set training process, and use the trained spatiotemporal large language model to implement NO x Inversion of concentration.

[0041] In this embodiment, the spatiotemporal language model learns remote sensing data and NO x The nonlinear mapping relationship between concentrations, the training process includes forward propagation, loss calculation, back propagation and iterative optimization. Forward propagation includes the input data passing through the spatiotemporal embedding layer, the fusion convolution layer, the LLM layer and the regression convolution layer in sequence. The loss calculation includes calculating the loss between the model output and the true label according to the loss function. Back propagation includes updating the parameters according to the gradient of the loss function relative to the model parameters. Iterative optimization includes iterative optimization and repeated forward propagation, loss calculation and back propagation until the set number of training rounds is reached, where one round of training is to perform one forward propagation, loss calculation and back propagation.

[0042] Exemplarily, the entire ST-LLM model is implemented based on TensorFlow definition classes and instantiated objects. Its training process includes forward propagation, loss calculation, back propagation and iterative optimization. The forward propagation input data passes through the spatiotemporal embedding layer, the fused convolution layer, the LLM layer and the regression convolution layer in sequence; in particular, the LLM layer includes multiple Transformer blocks, in which the input data passes through all Transformer blocks in sequence. Then the loss between the model output and the true label is calculated, and the back propagation updates the parameters according to the gradient of the loss function relative to the model parameters, and the automatic differentiation function of TensorFlow is used to update the parameters. Iterative optimization repeats the above steps until the specified number of training rounds is reached. Among them, updating parameters refers to updating the learnable parameters of the Transformer blocks contained in the LLM layer. In some embodiments, some Transformer blocks are frozen. During the training process, the frozen Transformer blocks will not update the parameters to enhance the model in NO 2 Ability to handle long-range dependencies in inversion.

[0043] The specific data processing process of the spatiotemporal embedding layer, fusion convolution layer, LLM layer and regression convolution layer integrated in the ST-LLM model will be introduced in detail below.

[0044] The structure of the spatiotemporal embedding layer includes a convolutional layer Conv2D, whose 1*1 convolution kernel performs point convolution operation on the input data to extract the local features of the input data. The point convolution operation performs weighted combination of the map in the previous step in the depth direction to generate a new feature map. The dimension of the last dimension of the output value (i.e. the number of feature maps) is determined by the number of filters. In this study, the number of filters is the hyperparameter d that determines the width of the model network.model , for each pixel point X in the input data X lon,lat , based on formula 1, the convolution structure outputs the pixel value output1 at position (lon, lat) lon,lat :

[0045]

[0046] Where, output1 lon,lat is the output of the spatiotemporal embedding layer, which represents the pixel value output by the spatiotemporal embedding layer at position (lon, lat). ReLU represents the nonlinear activation function. lon,lat,d Represents each pixel in the input data, lon represents the specific longitude value of the grid, lat represents the specific latitude value of the grid, and w d represents the weight of the convolution kernel in the convolution layer contained in the spatiotemporal embedding layer at channel d, D represents the number of channels, and b is the bias term. This type of design allows PointwiseConv subclasses to perform efficient feature transformation to increase network depth without changing the spatial structure of the input feature map.

[0047] The function of the fused convolution layer is to integrate the spatiotemporal position information into the deep learning model in an embedded form. Its structure includes a positional encoding function positional_encoding and a one-dimensional convolution layer fconv for splicing and fusing the three parts of embedding, which organically integrates the output of the previous subclass PointwiseConv, the time series embedding, and the spatial series embedding. The output dimension and size remain unchanged from the input. Among them, the role of positional encoding is to provide the model with a concept of position. Obtaining positional encoding from the output of the upper layer allows the model to rely more on the specific features of the actual input (Formula 2):

[0048] pos=Dense(output 1 ) (Formula 2)

[0049] In the formula, pos represents the position encoding feature, Dense represents the fully connected layer, and output 1 represents the output of the spatiotemporal embedding layer;

[0050] Sine and cosine encoding can extract periodic information, which helps the model learn the characteristics of periodicity and short-term dependencies in time series (Equations 3 and 4). Sine is used for even positions and cosine is used for odd positions, allowing the model to use two orthogonal functions to capture unique information at each position in the sequence:

[0051] PE (pos,2i) = sin(pos×div term ) (Formula 3)

[0052] PE(pos,2i+1) =cos(pos×div term ) (Formula 4)

[0053] Where, PE (pos,2i) Indicates the position encoding wavelength of the position feature at the even position, PE (pos,2i+1) Indicates the position encoding wavelength of the position feature at the odd position, sin represents the sine function, cos represents the cosine function, i represents the index of the dimension, div term Indicates the calculation denominator, which is determined by the following formula:

[0054]

[0055] In the formula, e represents the natural logarithm, log(10000) represents the constant for determining the position encoding wavelength, range represents the value range, and d model Represents the dimension size of the feature vector that defines the embedding and attention mechanism. In summary, 2i and 2i+1 ensure that each dimension pair has a unique corresponding sine and cosine wavelength.

[0056] Spatial position encoding is usually different from temporal position encoding because spatial relationships may rely more on relative or linear relationships between positions rather than periodicity. Therefore, spatial position encoding and temporal position encoding are calculated using the following formula:

[0057]

[0058] In the formula, lin term represents linear position encoding, quad term represents nonlinear position encoding, and max represents the maximum value function.

[0059] In summary, the concepts of time series coding and space series coding are expressed as Equation 8 and Equation 9:

[0060] T seq =(PF (pos,2i) ||PE (pos,2i+1) ) (Formula 8)

[0061] S seq =(lin term ||quad term ) (Formula 9)

[0062] Where, T seq represents the time series characteristics, S seq represents spatial sequence features, and || represents the “splicing” operation.

[0063] This type of encoding allows the model to distinguish data at different positions in the sequence, improving learning efficiency and accuracy. Output of the SpatialTemporalEmbedding class output 2 It is expressed by formula 10:

[0064] output 2 =fconv(output1 lon,lat ||S seq ||T seq ) (Formula 10)

[0065] In the formula, output 2 Represents the output of the fused convolutional layer, fconv represents a one-dimensional convolution operation, output1 lon,lat represents the output of the spatiotemporal embedding layer.

[0066] The Transformer block contains a multi-head attention mechanism MultiHeadAttention, a feed-forward neural network FFN, and a layer normalization LayerNormalization. Its function is to capture the long-term dependencies between the fused embedding sequences of the previous layer. As a submodule in the PFA-LLM subclass, the output shape remains the same as the input.

[0067] The multi-head attention mechanism allows the model to learn different aspects of the input data in different representation subspaces. The mathematical expression is as shown in Equation 11:

[0068]

[0069] In the formula, Attention(Q,K,V) represents the multi-head attention mechanism, Q, K, V represent the query vector matrix, key vector matrix and value vector matrix respectively, and d k represents the dimension of the key vector matrix and T represents the matrix transpose.

[0070] In multi-head attention, the input is split into multiple heads, and the Q, K, and V corresponding to each head are calculated separately, and then the results are concatenated and projected again.

[0071] The feedforward network consists of two fully connected layers, as shown in Equation 12:

[0072] FFN(x)=max(0,W 1 x+b 1 )W 2 +b 2 (Formula 12)

[0073] In the formula, FFN(x) represents the feedforward layer, x represents the input of the feedforward layer, and W 1 and W 2Represent the weights of the two fully connected layers, b 1 and b 2 They represent the bias of the two fully connected layers respectively.

[0074] The layer normalization method used to stabilize the network training process is placed after the multi-head attention layer MultiHeadAttention and the feedforward neural network FFN, and its structure is as shown in Equation 13:

[0075]

[0076] Where LayerNorm(x) represents the layer normalization process, μ and σ represent the mean and standard deviation of the input x, respectively, and γ and β represent the learnable parameters, respectively.

[0077] In summary, the TransformerBlock structure is as follows:

[0078] x l = TransformerBlock(output 2 )

[0079] =LayerNorm(FFN(LayerNorm(output 2 +MultiHead(output 2 )))+LayerNorm(output 2 +MultiHead(output 2 ))) (Formula 14)

[0080] In the formula, x l Represents the level with index l, and TransformerBlock represents the transformer architecture module.

[0081] The structure of the LLM layer (PFA_LLM) consists of multiple Transformer blocks, where the first frozen_layers layer is frozen to keep the original information without training to retain the knowledge learned during pre-training, and the last unfrozen_layers layer is trained to capture the spatiotemporal dependencies between tokens. Therefore, the LLM layer is a hierarchical iterative structure, as shown in Equation 15:

[0082]

[0083] In the formula, L represents the number of layers in the Transformer architecture, that is, the number of TransformerBlocks. In other words, the model consists of L layers of Transformer blocks, and the output 3 represents the output of the LLM layer, x l-1Indicates the level with index l-1.

[0084] The main function of the LLM layer is to use the feedforward attention (PFA) mechanism to learn the relationship between sequences while retaining the knowledge learned during pre-training, and the output shape remains the same as the input. This structure allows the Transformer layer to capture complex input dependencies while maintaining the stability of the network in deep layers.

[0085] The structure of the regression convolution layer (RegressionConv) contains three fully connected layers Dense. The first and second layers are used to receive input features and perform nonlinear transformations, and the third layer is used for linear regression output. Its function is to perform regression inversion on the fused features. The last dimension in the output result is the regressor dimension, that is, one dimension. The corresponding formulas for the three fully connected layers are shown in Equations 16 to 18.

[0086] x 1 =ReLU(W 1 Output 3 +b 1 ) (Formula 16)

[0087] x 2 =ReLU(W 2 ·x 1 +b 2 ) (Formula 17)

[0088] output 4 =W 3 ·x 2 +b 3 (Formula 18)

[0089] Where W 1 , W 2 and W 3 denote the weights of the first fully connected layer, the second fully connected layer, and the third fully connected layer, respectively, and b 1 、b 2 and b 3 Respectively represent the bias of the first fully connected layer, the second fully connected layer, and the third fully connected layer, ReLU represents the nonlinear activation function, and output 3 represents the output of the LLM layer, x 1 and x 2 Represent the output of the first fully connected layer and the second fully connected layer, output 4 Represents the output of the regression convolution layer, which is the final output of the nitrogen oxide estimation value.

[0090] In the embodiment of the present application, based on the spatiotemporal embedded large language model driven inversion model (ST-LLM), NO 2Data is embedded in the spatiotemporal representation of a large language model, which is expected to effectively utilize the large language model for NO 2 The advantages of the method disclosed in the embodiments of the present application can be summarized as follows:

[0091] 1. In ST-LLM, a partial freezing strategy is adopted to enhance the model in NO 2 Ability to handle long-range dependencies in inversion.

[0092] 2. ST-LLM uses a specific spatiotemporal embedding layer to simultaneously model data features in spatial and temporal dimensions, capturing NO more accurately. 2 The variation pattern of concentration at different locations and times can improve the inversion accuracy.

[0093] 3. This method targets surface NO 2 It is specially designed for the inversion needs of NO, filling the gap in the existing models in this field and providing a more suitable 2 Monitoring solutions.

[0094] Therefore, the spatiotemporal embedding of the Transformer architecture is NO x The key technologies of the inversion method and device include: the design and implementation of the spatiotemporal embedding large language model in the field of atmospheric pollutant inversion. The model combines the spatiotemporal embedding layer, the fusion convolution layer, the LLM layer and the regression convolution layer to effectively process the spatiotemporal characteristics of remote sensing data and learn deep feature representation. In addition, the spatiotemporal feature extraction of remote sensing data is also one of the key technologies that this patent proposes to protect. The spatiotemporal embedding layer is used to pre-process the remote sensing data, extract spatial and temporal features, and provide input for the subsequent deep learning model. Then, the LLM layer is used to learn the deep representation of the remote sensing data and capture the complex nonlinear relationship in the data.

[0095] This application realizes the NO 2 The high-precision inversion of concentration provides effective technical support for air pollution monitoring and environmental protection.

[0096] See also Figure 3 , Figure 3 It is a spatiotemporal embedding NO under a Transformer architecture shown in an exemplary embodiment of the present application. x Another aspect of the present invention is to provide a spatial-temporal embedding NO in a Transformer architecture. x An inversion device, comprising:

[0097] The model building module 301 is configured to build a spatiotemporal large language model, which includes a spatiotemporal embedding layer, a fused convolution layer, an LLM layer and a regression convolution layer; wherein the spatiotemporal embedding layer outputs a spatiotemporal representation in response to the input filtered channel data, the fused convolution layer is used to integrate the output of the spatiotemporal embedding layer to obtain spatiotemporal data and output it to the LLM layer, the LLM layer is used to use a large language model to capture the long-term dependencies in the spatiotemporal data, and extract deep features, and output them to the regression convolution layer; the regression convolution layer is used to convert the output of the LLM layer into NO x concentration;

[0098] The model training module 302 is configured to train the spatiotemporal large language model according to a set training process, and use the trained spatiotemporal large language model to implement NO x Inversion of concentration; wherein, the spatiotemporal language model learns remote sensing data and NO during training. x The training process includes forward propagation, loss calculation, back propagation and iterative optimization. The forward propagation includes input data passing through the spatiotemporal embedding layer, the fusion convolution layer, the LLM layer and the regression convolution layer in sequence. The loss calculation includes calculating the loss between the model output and the true label according to the loss function. The back propagation includes updating the parameters according to the gradient of the loss function relative to the model parameters. The iterative optimization includes iterative optimization and repeating forward propagation, loss calculation and back propagation until the set number of training rounds is reached, wherein one round of training is to perform one forward propagation, loss calculation and back propagation.

[0099] In some embodiments, the spatiotemporal representation is a pixel value at the location of the data, and the spatiotemporal embedding layer outputs the spatiotemporal representation in response to the input filtered channel data through the following formula:

[0100]

[0101] Where, output1 lon,lat is the output of the spatiotemporal embedding layer, which represents the pixel value output by the spatiotemporal embedding layer at position (lon, lat). ReLU represents the nonlinear activation function. lon,lat,d Represents each pixel in the input data, lon represents the specific longitude value of the grid, lat represents the specific latitude value of the grid, and w d represents the weight of the convolution kernel in the convolution layer contained in the spatiotemporal embedding layer at channel d, D represents the number of channels, and b is the bias term.

[0102] In some embodiments, the fused convolution layer includes a position encoding function and a one-dimensional convolution layer; wherein the position encoding function is used to determine the time series embedding and the spatial sequence embedding according to the output of the spatiotemporal embedding layer, and the one-dimensional convolution layer is used to fuse the output of the spatiotemporal embedding layer, the time series embedding and the spatial sequence embedding to obtain the output of the fused convolution layer.

[0103] In some embodiments, determining the temporal sequence embedding and the spatial sequence embedding according to the output of the spatiotemporal embedding layer includes:

[0104] Based on the output of the spatiotemporal embedding layer, the position encoding feature is determined by the following formula:

[0105] pos=Dense(output 1 ) (Formula 2)

[0106] In the formula, pos represents the position encoding feature, Dense represents the fully connected layer, and output 1 represents the output of the spatiotemporal embedding layer;

[0107] Based on the position coding feature, the periodicity and short-term dependency features in the time series are obtained by the following formula:

[0108] PE (pos,2i) = sin(pos×div term ) (Formula 3)

[0109] PE (pos,2i+1) =cos(pos×div term ) (Formula 4)

[0110] Where, PE (pos,2i) Indicates the position encoding wavelength of the position feature at the even position, PE (pos,2i+1) Indicates the position encoding wavelength of the position feature at the odd position, sin represents the sine function, cos represents the cosine function, i represents the index of the dimension, div term Indicates the calculation denominator, which is determined by the following formula:

[0111]

[0112] In the formula, e represents the natural logarithm, log(10000) represents the constant for determining the position encoding wavelength, range represents the value range, and d model Represents the dimension size of the feature vector that defines the embedding and attention mechanism;

[0113] The spatial position coding and temporal position coding are calculated by the following formula:

[0114]

[0115]

[0116] In the formula, lin term represents linear position encoding, quad term represents nonlinear position coding, and max represents the maximum value function;

[0117] The time series embedding and spatial series embedding are calculated by the following formula:

[0118] T seq =(PE (pos,2i) ||PE (pos,2i+1) ) (Formula 8)

[0119] S seq =(lin term ||quad term ) (Formula 9)

[0120] Where, T seq represents the time series characteristics, S seq represents the spatial sequence feature, || represents the "splicing" operation, which is to merge two vectors (or matrices) along a certain dimension;

[0121] The output of the fused convolutional layer is expressed as:

[0122] output 2 =fconv(output1 lon,lat || seq ||T seq ) (Formula 10)

[0123] In the formula, output 2 Represents the output of the fused convolutional layer, fconv represents a one-dimensional convolution operation, output1 lon,lat represents the output of the spatiotemporal embedding layer.

[0124] In some embodiments, the LLM layer includes multiple Transformer blocks. When training the spatiotemporal large language model, some Transformer blocks are frozen, and the frozen Transformer blocks do not participate in the training process.

[0125] In some embodiments, the Transformer block is used to capture the long-term dependencies between the spatiotemporal data output by the fused convolutional layer, and the Transformer block includes a multi-head attention mechanism, a feedforward neural network, and layer normalization;

[0126] The calculation process of the multi-head attention mechanism is expressed as:

[0127]

[0128] In the formula, Attention(Q,K,V) represents the multi-head attention mechanism, Q, K, V represent the query vector matrix, key vector matrix and value vector matrix respectively, and d k represents the dimension of the key vector matrix, and T represents the matrix transpose;

[0129] The feedforward neural network consists of two fully connected layers, and the calculation process is expressed as:

[0130] FFN(x)=max(0,W 1 x+b 1 )W 2 +b 2 (Formula 12)

[0131] In the formula, FFN(x) represents the feedforward layer, x represents the input of the feedforward layer, and W 1 and W 2 Represent the weights of the two fully connected layers, b 1 and b 2 Represent the bias of two fully connected layers respectively;

[0132] The calculation process of the layer normalization is expressed as:

[0133]

[0134] Where LayerNorm(x) represents the layer normalization process, μ and σ represent the mean and standard deviation of the input x, respectively, and γ and β represent the learnable parameters, respectively;

[0135] The Transformer block is represented as:

[0136] x l = TransformerBlock(output 2 )

[0137] =LayerNorm(FFN(LayerNorm(output 2 +MultiHead(output 2 )))+LayerBorm(output 2 +MultiHead(output 2 ))) (Formula 14)

[0138] In the formula, x l Represents the level with index l, TransformerBlock represents the transformer module;

[0139] The LLM layer is represented as:

[0140]

[0141] In the formula, L represents the number of layers in the Transformer architecture, that is, the number of TransformerBlocks. In other words, the model consists of L layers of Transformer blocks, and the output 3 represents the output of the LLM layer, x l-1 In some embodiments, the regression convolution layer includes three fully connected layers connected in sequence, namely, a first fully connected layer, a second fully connected layer, and a third fully connected layer; wherein the calculation process of the first fully connected layer, the second fully connected layer, and the third fully connected layer is as follows:

[0142] x 1 =Relu(W 1 Output 3 +b 1 ) (Formula 16)

[0143] x 2 =Relu(W 2 ·x 1 +b 2 ) (Formula 17)

[0144] output 4 =W 3 ·x 2 +b 3 (Formula 18)

[0145] Where W 1 , W 2 and W 3 denote the weights of the first fully connected layer, the second fully connected layer, and the third fully connected layer, respectively, and b 1 、b 2 and b 3 Respectively represent the bias of the first fully connected layer, the second fully connected layer, and the third fully connected layer, ReLU represents the nonlinear activation function, and output 3 represents the output of the LLM layer, x 1 and x 2 Represent the output of the first fully connected layer and the second fully connected layer, output 4 Represents the output of the regression convolution layer, i.e. NO x concentration.

[0146] It should be noted that the spatiotemporal embedding NO under the Transformer architecture provided in the above embodiment x The inversion device and the time-space embedding NO under the Transformer architecture provided in the above embodiment xThe inversion method belongs to the same concept, and the specific manner of executing the steps has been described in detail in the method embodiment, which will not be repeated here.

[0147] Another aspect of the present application also provides an electronic device, including: a controller; a memory for storing one or more programs, when the one or more programs are executed by the controller, to perform the spatiotemporal embedding NO under the Transformer architecture in each of the above embodiments x Inversion method and apparatus.

[0148] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU) 701, various functions defined in the system of the present application are executed.

[0149] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, wherein a computer-readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0150] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0151] The modules / units involved in the embodiments described in this application may be implemented by software or hardware, and the described units may also be set in a processor. The names of these modules / units do not, in some cases, constitute limitations on the modules / units themselves.

[0152] Another aspect of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the spatiotemporal embedded NOx inversion method under the Transformer architecture as described above. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0153] Another aspect of the present application also provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the spatiotemporal embedding NO under the Transformer architecture provided in the above embodiments. x Inversion method.

[0154] According to one aspect of an embodiment of the present application, a computer system is also provided, including a central processing unit (CPU), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage portion into a random access memory (RAM), such as executing the method in the above embodiment. In RAM, various programs and data required for system operation are also stored. CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0155] The following components are connected to the I / O interface: an input part including a keyboard, a mouse, etc.; an output part including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage part including a hard disk, etc.; and a communication part including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication part performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface as needed. Removable media, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive as needed so that the computer program read therefrom is installed into the storage part as needed.

[0156] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A spatiotemporal embedding NO under the Transformer architecture x The inversion method is characterized in that The method comprises: A spatiotemporal large language model is constructed, the spatiotemporal large language model comprising a spatiotemporal embedding layer, a fused convolutional layer, a large language model (LLM) layer and a regression convolutional layer; wherein the spatiotemporal embedding layer outputs a spatiotemporal representation in response to the input filtered channel data; the filtered channel data is obtained by filtering the remote sensing data of the channel by the maximum correlation minimum redundancy method, the fused convolutional layer is used to integrate the output of the spatiotemporal embedding layer, obtain the spatiotemporal data and output it to the LLM layer, the LLM layer is used to use the large language model to capture the long-term dependency relationship in the spatiotemporal data, and extract deep features, and output them to the regression convolutional layer; the regression convolutional layer is used to convert the output of the LLM layer into NO x concentration; The spatiotemporal large language model is trained according to the set training process, and the trained spatiotemporal large language model is used to implement NO x Inversion of concentration; wherein, the spatiotemporal language model learns remote sensing data and NO during training. x The training process includes forward propagation, loss calculation, back propagation and iterative optimization. The forward propagation includes input data passing through the spatiotemporal embedding layer, the fusion convolution layer, the LLM layer and the regression convolution layer in sequence. The loss calculation includes calculating the loss between the model output and the true label according to the loss function. The back propagation includes updating the parameters according to the gradient of the loss function relative to the model parameters. The iterative optimization includes iterative optimization and repeating forward propagation, loss calculation and back propagation until the set number of training rounds is reached, wherein one round of training is to perform one forward propagation, loss calculation and back propagation.

2. Spatiotemporal embedding of the Transformer architecture according to claim 1 x The inversion method is characterized in that The spatiotemporal representation is the pixel value at the location of the data. The spatiotemporal embedding layer responds to the input filtered channel data and outputs the spatiotemporal representation through the following formula: Where, output1 lon,lat is the output of the spatiotemporal embedding layer, which represents the pixel value output by the spatiotemporal embedding layer at position (lon, lat). ReLU represents the nonlinear activation function. lon,lat,d Represents each pixel in the input data, lon represents the specific longitude value of the grid, lat represents the specific latitude value of the grid, and w d represents the weight of the convolution kernel in the convolution layer contained in the spatiotemporal embedding layer at channel d, D represents the number of channels, and b is the bias term.

3. Temporal and spatial embedding of NO under the Transformer architecture according to claim 1 x The inversion method is characterized in that The fused convolution layer includes a position encoding function and a one-dimensional convolution layer; wherein the position encoding function is used to determine the time series embedding and the space sequence embedding according to the output of the spatiotemporal embedding layer, and the one-dimensional convolution layer is used to fuse the output of the spatiotemporal embedding layer, the time series embedding and the space sequence embedding to obtain the output of the fused convolution layer.

4. Spatiotemporal embedding NO under the Transformer architecture according to claim 3 x The inversion method is characterized in that The time series embedding and spatial series embedding are determined based on the output of the spatiotemporal embedding layer, including: Based on the output of the spatiotemporal embedding layer, the position encoding feature is determined by the following formula: pos=Dense(output1) (Formula 2) In the formula, pos represents the position encoding feature, Dense represents the fully connected layer, and output1 represents the output of the spatiotemporal embedding layer; Based on the position coding feature, the periodicity and short-term dependency features in the time series are obtained by the following formula: PE (pos,2i) = sin(pos × div term ) (Equation 3) PE (pos,2i+1) =cos(pos×div term ) (Formula 4) Where, PE (pos,2i) Indicates the position encoding wavelength of the position feature at the even position, PE (pos,2i+1) Indicates the position encoding wavelength of the position feature at the odd position, sin represents the sine function, cos represents the cosine function, i represents the index of the dimension, div term Indicates the calculation denominator, which is determined by the following formula: In the formula, e represents the natural logarithm, log(10000) represents the constant for determining the position encoding wavelength, range represents the value range, and d model Represents the dimension size of the feature vector that defines the embedding and attention mechanism; The spatial position coding and temporal position coding are calculated by the following formula: In the formula, lin term represents linear position encoding, quad term represents nonlinear position encoding, and max represents the maximum value function; The time series embedding and spatial series embedding are calculated by the following formula: T seq = (PE (pos,2i) || PE (pos,2i+1) ) (Equation 8) S seq =(lin term ||quad term ) (Formula 9) Where, T seq represents the time series characteristics, S seq represents the spatial sequence feature, and || represents the "splicing" operation, which is to merge two vectors (or matrices) along a certain dimension; The output of the fused convolutional layer is expressed as: output2 = fconv(output1 lon,lat ||S seq ||T seq ) (Formula 10) In the formula, output2 represents the output of the fused convolutional layer, fconv represents the one-dimensional convolution operation, and output1 lon,lat represents the output of the spatiotemporal embedding layer.

5. Spatiotemporal embedding under the Transformer architecture according to claim 1 x The inversion method is characterized in that The LLM layer includes multiple Transformer blocks. When the spatiotemporal large language model is trained, some Transformer blocks are frozen, and the frozen Transformer blocks do not participate in the training process.

6. Spatiotemporal embedding NO under the Transformer architecture according to claim 5 x The inversion method is characterized in that The Transformer block is used to capture the long-term dependency between the spatiotemporal data output by the fused convolutional layer, and the Transformer block includes a multi-head attention mechanism, a feedforward neural network and layer normalization; The calculation process of the multi-head attention mechanism is expressed as: In the formula, Attention(Q,K,V) represents the multi-head attention mechanism, Q, K, V represent the query vector matrix, key vector matrix and value vector matrix respectively, and d k represents the dimension of the key vector matrix, and T represents the matrix transpose; The feedforward neural network consists of two fully connected layers, and the calculation process is expressed as: FFN(x)=max(0,W1x+b1)W2+b2 (Formula 12) Where FFN(x) represents the feedforward layer, x represents the input of the feedforward layer, W1 and W2 represent the weights of the two fully connected layers, b1 and b2 represent the biases of the two fully connected layers; The calculation process of the layer normalization is expressed as: Where LayerNorm(x) represents the layer normalization process, μ and σ represent the mean and standard deviation of the input x, respectively, and γ and β represent the learnable parameters, respectively; The Transformer block is represented as: xl=TransformerBlock(output2) =LayerNorm(FFN(LayerNorm(output2+MultiHead(output2)))+LayerNorm(output2+MultiHead(output2))) (Equation 14) In the formula, x l Represents the level with index l, TransformerBlock represents the transformer architecture module; The LLM layer is represented as: Where L represents the number of layers in the Transformer architecture, output3 represents the output of the LLM layer, and x l-1 Indicates the level with index l-1.

7. The spatiotemporal embedding of the Transformer architecture according to claim 1 x The inversion method is characterized in that The regression convolution layer includes three fully connected layers connected in sequence, namely the first fully connected layer, the second fully connected layer and the third fully connected layer; wherein the calculation process of the first fully connected layer, the second fully connected layer and the third fully connected layer is as follows: x1=ReLU(W1·output3+b1) (Equation 16) x2=ReLU(W2·x1+b2) (Formula 17) output4=W3·x2+b3 (Formula 18) Where W1, W2 and W3 represent the weights of the first fully connected layer, the second fully connected layer and the third fully connected layer, respectively; b1, b2 and b3 represent the biases of the first fully connected layer, the second fully connected layer and the third fully connected layer, respectively; ReLU represents the nonlinear activation function; output3 represents the output of the LLM layer; x1 and x2 represent the outputs of the first fully connected layer and the second fully connected layer, respectively; output4 represents the output of the regression convolution layer, i.e., NO x concentration.

8. A spatiotemporal embedding NO under a Transformer architecture x The inversion device is characterized in that The device comprises: The model building module is configured to build a spatiotemporal large language model, which includes a spatiotemporal embedding layer, a fused convolution layer, an LLM layer and a regression convolution layer; wherein the spatiotemporal embedding layer outputs a spatiotemporal representation in response to the input filtered channel data, the fused convolution layer is used to integrate the output of the spatiotemporal embedding layer to obtain spatiotemporal data and output it to the LLM layer, the LLM layer is used to use the large language model to capture the long-term dependencies in the spatiotemporal data, and extract deep features, and output them to the regression convolution layer; the regression convolution layer is used to convert the output of the LLM layer into NO x concentration; The model training module is configured to train the spatiotemporal large language model according to a set training process, and implement NO x Inversion of concentration; wherein, the spatiotemporal language model learns remote sensing data and NO during training. x The training process includes forward propagation, loss calculation, back propagation and iterative optimization. The forward propagation includes input data passing through the spatiotemporal embedding layer, the fusion convolution layer, the LLM layer and the regression convolution layer in sequence. The loss calculation includes calculating the loss between the model output and the true label according to the loss function. The back propagation includes updating the parameters according to the gradient of the loss function relative to the model parameters. The iterative optimization includes iterative optimization and repeating forward propagation, loss calculation and back propagation until the set number of training rounds is reached, wherein one round of training is to perform one forward propagation, loss calculation and back propagation.

9. An electronic device, characterized in that: include: Controller; A memory for storing one or more programs, when the one or more programs are executed by the controller, the controller implements the spatiotemporal embedding NO under the Transformer architecture according to any one of claims 1 to 7 x Inversion method and apparatus.

10. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the spatiotemporal embedding NO under the Transformer architecture according to any one of claims 1 to 7 x Inversion method and apparatus.

Images