A city environment performance prediction method and system based on multi-modal fusion and attention enhancement

Abstract

The application provides a kind of urban environment performance prediction method and system based on multi-modal fusion and attention enhancement, wherein the method comprises collecting image data and numerical data, generating a variety of urban environment performance distribution truth value map;Preprocess multi-modal data to get image map and numerical feature vector, pair the image map and numerical feature vector with the truth value map to form a multi-modal data set.Construct a multi-objective model based on attention-enhanced conditional generative adversarial network, the generator includes a double-path encoder, which can extract spatial features based on image maps and physical features based on numerical feature vectors;The image encoding path adopts the U-Net downsampling structure containing the convolution block attention module, and the numerical encoding path adopts the multilayer perceptron;The multi-head output layer outputs the predicted multi-urban environment performance distribution map.After training the model, input the target area data, and output three types of prediction distribution map.The application improves the urban environment performance prediction effect.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of urban environment performance evaluation, and particularly relates to a method and system for predicting urban environment performance based on multi-modal fusion and attention enhancement. BACKGROUND

[0002] In order to achieve the goal of urban environment monitoring, it is necessary to evaluate the urban environment performance (such as photovoltaic potential or building energy consumption). The existing technologies mainly include the following schemes:

[0003] (1) Physical-based simulation method: by establishing a detailed building physical model, inputting detailed building parameters (such as materials, structure, heating, ventilation and air conditioning system, etc.) and meteorological data, to simulate the building energy consumption and photovoltaic power generation, such as DOE-2 , EnergyPlus , TRNSYS . This method has high accuracy, but the modeling process is complex, the calculation cost is high, and the time is long.

[0004] (2) Traditional data-driven method based on statistics: using a large amount of historical data (such as building energy consumption bills, sensor readings, etc.) to predict energy consumption through machine learning or statistical regression model . This kind of method depends on large-scale and high-quality data set, but in many areas, especially for the built-up area, it is difficult to obtain such detailed data, and there is a problem of data scarcity.

[0005] (3) Single-dimensional image-driven method: some studies begin to use satellite images or aerial images to extract urban form, roof area and other features to predict photovoltaic potential or building energy consumption. However, this kind of method usually only focuses on a single target, and fails to comprehensively and simultaneously predict and evaluate multiple key environment performance indicators.

[0006] The above existing technical solutions have the following technical defects:

[0007] (1) Low prediction efficiency: physical simulation method has large calculation amount and long time consumption. Even for data-driven method, a lot of effort needs to be invested in large-scale data preprocessing and feature engineering, which is difficult to realize rapid energy audit of urban blocks.

[0008] (2) Difficulty in data acquisition: traditional physical simulation and data-driven method seriously depend on detailed building internal parameters and historical energy consumption data which are difficult to obtain, resulting in its difficulty in application and promotion in many data-scarce urban areas.

[0009] (3) The prediction index is single and independent: existing methods can usually only predict one isolated index (such as total energy consumption), or need to establish different models for different prediction targets (such as energy consumption, photovoltaic). This is not only inefficient, but also ignores the inherent physical correlation between different environmental performance (such as the common influence of building morphology on energy consumption, photovoltaic, and thermal environment).

[0010] (4) Insufficient fusion of multi-source heterogeneous data, limited prediction accuracy: existing data-driven methods usually use simple splicing or step-by-step processing when fusing image and numerical data. This shallow fusion method cannot effectively capture the synergistic effect between urban spatial form (from image) and dynamic physical parameters (such as meteorological numerical data).

[0011] (4) Lack of integrated analysis capability from multi-dimensional prediction to final carbon emission: existing technologies can predict energy consumption and photovoltaic, but cannot provide an integrated framework to automatically convert them into final carbon emission assessment, resulting in a disjointed analysis process.

[0012] Therefore, a new urban environmental performance evaluation method is provided. SUMMARY

[0013] The purpose of the present application is to provide a city environmental performance prediction method and system based on multi-modal fusion and attention enhancement, which improves the prediction effect of city environmental performance.

[0014] The technical scheme provided by the present application is:

[0015] In a first aspect, the present application provides a city environmental performance prediction method based on multi-modal fusion and attention enhancement, comprising:

[0016] Collecting multi-modal data and generating a true value map; the multi-modal data includes image data and numerical data, and the true value map includes a plurality of real city environmental performance distribution maps;

[0017] The collected multi-modal data is pre-processed respectively to obtain an image map and a numerical feature vector, and the image map and the numerical feature vector are paired with the true value map to form sample data matched with the true value map, and a multi-modal data set composed of a plurality of sample data;

[0018] A multi-objective model based on attention-enhanced conditional generative adversarial network AE-cGAN is constructed; a generator of the multi-objective model adopts a structure including a double-path encoder connected in sequence, a feature fusion layer, a decoder and a multi-head output layer; wherein the double-path encoder includes an image encoding path based on spatial features extracted from an image map and a numerical encoding path based on physical features extracted from a numerical feature vector; the image encoding path adopts a U-Net down-sampling structure including a convolution block attention module, and the numerical encoding path adopts a multi-layer perceptron; the multi-head output layer is used to output predicted multiple urban environmental performance distribution maps based on input data; and the multi-objective model is trained based on a multi-modal data set.

[0019] The image map and the numerical feature vector of the target urban area are input into the trained multi-objective model, and the predicted multiple urban environmental performance distribution maps are output by the multi-objective model.

[0020] In a possible implementation, the image data is collected, including: obtaining urban texture and building height from Google satellite images, GIS data, and building function, land surface temperature, solar radiation, vegetation index and wind speed from remote sensing data, and rasterizing the obtained data into an image format to obtain image data.

[0021] The numerical data is collected, including: annual / monthly / daily average dry-bulb temperature, relative humidity, total solar radiation and average wind speed obtained through a meteorological database and a government open data platform.

[0022] The multiple urban environmental performance distribution maps include: a block energy consumption distribution map, a photovoltaic power generation potential distribution map and an outdoor thermal comfort distribution map.

[0023] In a possible implementation, after obtaining the sample data, the sample data is subjected to data enhancement processing to expand the sample size, and finally a multi-modal data set is formed.

[0024] In a possible implementation, the feature fusion layer splices the flattened vector output by the image encoding path and the high-dimensional vector output by the numerical encoding path; the decoder restores the feature map resolution through symmetrical connection of the up-sampling module and the down-sampling module; and the multi-head output layer outputs the predicted multiple urban environmental performance distribution maps in parallel through three independent output heads.

[0025] In a possible implementation, the discriminator of the multi-objective model adopts a multi-scale double discriminator for distinguishing the predicted map output by the generator from the true value map from global and local scales.

[0026] In a possible implementation, the multi-objective model is trained by using a weighted combination loss function; wherein the weighted combination loss function is:

[0027] ;

[0028] wherein, is the total loss, is the adversarial loss; 、 and are the pixel-level L1 loss for the block energy consumption prediction task, the photovoltaic power generation potential prediction task, and the outdoor thermal comfort prediction task, respectively; 、 and are the weight hyperparameters of 、 and , respectively.

[0029] In a possible implementation, the method further comprises a carbon emission accounting step: based on the predicted block energy consumption distribution map and the photovoltaic power generation potential distribution map, combining the local power grid carbon emission factor and the clean energy emission reduction factor, calculating the net carbon emission pixel by pixel or building by building to generate a net carbon emission intensity map.

[0030] In a second aspect, the application provides a city environment performance prediction system based on multi-modal fusion and attention enhancement, comprising:

[0031] a data acquisition module for acquiring multi-modal data and generating a true value map; the multi-modal data includes image data and numerical data, and the true value map includes a plurality of real city environment performance distribution maps;

[0032] a data preprocessing module for preprocessing the acquired multi-modal data respectively to obtain an image map and a numerical feature vector, pairing the image map and the numerical feature vector with the true value map to form sample data matched with the true value map, and forming a multi-modal data set composed of a plurality of sample data;

[0033] a multi-objective model construction and training module for constructing a multi-objective model of an attention-enhanced conditional generative adversarial network AE-cGAN; the generator of the multi-objective model adopts a structure including a two-way encoder, a feature fusion layer, a decoder, and a multi-head output layer connected in sequence; wherein the two-way encoder includes an image encoding path based on spatial features extracted from the image map and a numerical encoding path based on physical features extracted from the numerical feature vector; the image encoding path adopts a U-Net downsampling structure containing a convolution block attention module, and the numerical encoding path adopts a multi-layer perceptron; the multi-head output layer is used to output a plurality of predicted city environment performance distribution maps based on the input data; the discriminator adopts a multi-scale double discriminator for distinguishing the predicted map output by the generator from the true value map from global and local scales; and the multi-modal data set is used to train the multi-objective model;

[0034] A multi-target prediction module is configured to input an image map and a numerical feature vector of a target urban area into a trained multi-target model, and output a predicted multi-urban environmental performance distribution map through the multi-target model.

[0035] In a third aspect, the present application provides an electronic device, comprising a memory and a processor;

[0036] The memory is configured to store a computer program.

[0037] The processor is configured to invoke the computer program to perform the method described above.

[0038] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program, when executed on an electronic device, causes the electronic device to implement the method described above.

[0039] In a fifth aspect, the present application provides a computer program product comprising a computer program, wherein the computer program, when executed on an electronic device, causes the electronic device to implement the method described above.

[0040] The specific implementation manners of the second to fourth aspects of the present application can refer to the implementation manners of the first aspect described above, which will not be described here.

[0041] Advantages:

[0042] The present application realizes multi-dimensional and high-correlation comprehensive evaluation: the present application outputs three core indexes of energy consumption, photovoltaic and thermal comfort through a unified model in parallel, which is more comprehensive in evaluation dimension compared with the prior art which can only predict one index at a time. At the same time, the model can capture the internal correlation between indexes (such as the common influence of building form on the three indexes) during the learning process, and the prediction result is more in line with physical logic.

[0043] The present application reduces the data threshold: the present application mainly relies on satellite images and remote sensing data which are easy to obtain on a large scale, overcoming the strong dependence of the prior art on detailed and private building energy consumption historical data or physical parameters. This makes the present application capable of being quickly deployed and applied in data scarce areas, and has strong practicality and generalizability.

[0044] The present application improves the prediction accuracy and robustness: since numerical data (such as meteorological parameters) directly related to the prediction target are introduced, the model can not only learn the spatial distribution rule from the image, but also correct according to the accurate physical quantity, greatly improving the accuracy and reliability of the prediction result. Compared with the inference relying only on the image, the prediction accuracy of the present application is higher, and due to the application of attention mechanism, the model prediction provides better explainability.

[0045] The model has the ability of scenario simulation and analysis: the scheme can evaluate the influence of different numerical conditions (such as different years of meteorological data, different electricity price strategies) on the performance of the city environment. By inputting a new numerical vector, the prediction result under the corresponding scenario can be quickly generated, so that the model has the ability of scenario simulation and sensitivity analysis, greatly enhancing the practicality and decision support value of the scheme.

[0046] The application provides a closed-loop analysis capability from prediction to accounting: the upstream multi-objective prediction model is innovatively combined with the downstream carbon emission accounting module to form a complete "prediction-evaluation-audit" closed loop. The user can not only see various performance indicators, but also directly obtain the final carbon emission conclusion, and the decision support capability is far superior to the prior art.

[0047] The prediction accuracy and interpretability of the application are higher: due to the introduction of the attention mechanism, the model can adaptively allocate feature weights for different tasks, which can theoretically achieve higher accuracy than multiple independent models, and provide visual explanation for "why the model predicts this way". BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 The overall block diagram of the method of one embodiment of the application is shown in the figure;

[0049] Figure 2 The flowchart of the method of one embodiment of the application is shown in the figure;

[0050] Figure 3 The overall block diagram of the system of one embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the scheme of the application, the technical scheme of the application will be further described in detail below in combination with the embodiments of the application and the drawings.

[0052] It should be noted that the terms "first", "second", etc. in the specification and claims of the application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0053] In addition, the terms "mounting", "arrangement", "provided with", "connected", "linked", "sleeved" should be broadly understood. For example, "connected" can be fixed connection, detachable connection, or integral structure; can be mechanical connection, or electrical connection; can be direct connection, or indirect connection through intermediate medium, or internal communication between two devices, elements or components. The specific meaning of the above terms in the present application can be understood by those skilled in the art according to the specific circumstances.

[0054] Existing methods have significant limitations in processing multi-modal data. They usually train and predict with independent models, for example, two independent models are established for energy consumption prediction and photovoltaic potential prediction, which leads to the inability to share features between models and ignores the inherent physical correlation between different prediction targets.

[0055] More importantly, at the data level, even if images and numerical data are used simultaneously, they are often combined in a simple and shallow way, for example: numerical data is used as a classification label (such as climate zoning), or only used for post-processing correction of the prediction results of pure image models. There is a lack of an end-to-end technical solution in the prior art that can dynamically and deeply fuse spatial features of images and physical features of numerical data within the model, resulting in the inability of the model to fully learn the complex nonlinear relationship between the two data sources, limiting the accuracy and generalization ability of the prediction.

[0056] The present application aims to solve the technical problems of insufficient data fusion, single prediction index, independent models, lack of comprehensiveness, and disjointed analysis process in the prior art, and provides an innovative technical solution that can effectively fuse easily accessible multi-source image and numerical data, predict multiple core environmental performance indicators in parallel, and calculate net carbon emissions based on this.

[0057] The present application provides a multi-modal urban environmental performance prediction method and system based on attention-enhanced generative network. The core is to represent the multi-dimensional influencing factors of urban blocks as multi-channel image data (such as urban morphology) and structured numerical data (such as meteorological parameters), and through an innovative generative adversarial network with a built-in multi-modal fusion mechanism, attention mechanism and multi-head output structure, to realize the parallel mapping from the fused multi-modal input to the prediction map of multiple core performance indicators (building energy consumption, photovoltaic power generation, outdoor thermal comfort), and integrate the downstream module to complete the accounting of net carbon emissions.

[0058] The attention mechanism refers to a technique in deep learning that simulates human visual attention, allowing the model to focus on the most important parts when processing large amounts of information.

[0059] Multi-head output / multi-task learning is a machine learning paradigm that uses the correlation between tasks to improve overall learning efficiency and performance by having a model learn multiple related tasks simultaneously.

[0060] A specific embodiment according to the present application will be described below with reference to the accompanying drawings.

[0061] Embodiment one:

[0062] The embodiment of the present application provides a city environment performance prediction method based on multi-modal fusion and attention enhancement, comprising:

[0063] Step one: collect multi-modal data and generate a true value graph; the multi-modal data includes image data and numerical data, and the true value graph includes a plurality of real city environment performance distribution graphs.

[0064] The multi-modal data includes: obtaining a plurality of variable data through a public channel and rasterizing the plurality of variable data into an image format. These data include:

[0065] City texture and building height: from Google satellite images and GIS data.

[0066] Building function, land surface temperature, solar radiation, vegetation index, wind speed, etc.: from remote sensing data.

[0067] Obtain numerical variables such as annual / monthly / daily average dry bulb temperature, relative humidity, total solar radiation, average wind speed, etc. through meteorological databases, government public data platforms, etc.

[0068] The true value graph includes three independent true value graphs, which are a block energy consumption distribution graph, a photovoltaic power generation potential distribution graph, and an outdoor thermal comfort (Universal Thermal Climate Index, referred to as UTCI, i.e. Universal Thermal Climate Index) distribution graph.

[0069] The block energy consumption distribution graph can be obtained through an official data set or physical simulation.

[0070] The photovoltaic power generation potential distribution graph can be obtained through professional tools (such as Ladybug).

[0071] The outdoor thermal comfort distribution graph can be calculated and generated based on meteorological data and city morphology data.

[0072] The above scheme of the present application realizes multi-source data fusion, that is, information from different sensors or sources is integrated and processed to produce more accurate and more comprehensive results than single source information. The present application is a specific application of this technology in the field of city energy evaluation.

[0073] Exemplarily, multi-source data (input data, including image data and numerical data) and prediction target data (target data, i.e., urban environmental performance distribution map) in the range of urban block slice are acquired.

[0074] Step two: the collected multi-modal data are respectively preprocessed to obtain an image map and a numerical feature vector, the image map and the numerical feature vector are paired with a true value map to form sample data matched with the image map, the numerical feature vector and the true value map, and a multi-modal data set composed of a plurality of sample data;

[0075] In this step, the acquired multi-channel image map, numerical feature vector and generated true value map are paired to form sample data of (image map, numerical feature vector, true value map). The sample data are subjected to data enhancement (such as rotation and flipping of the image) to expand the sample amount, and finally a multi-modal data set is formed.

[0076] Step three: a multi-target model based on attention-enhanced conditional generative adversarial network AE-cGAN is constructed; the generator of the multi-target model adopts a structure including a double-path encoder, a feature fusion layer, a decoder and a multi-head output layer connected in sequence; wherein the double-path encoder includes an image encoding path based on extraction of spatial features of the image map and a numerical encoding path based on extraction of physical features of the numerical feature vector; the image encoding path adopts a U-Net downsampling structure containing a convolution block attention module (CBAM), and the numerical encoding path adopts a multi-layer perceptron (MLP); the multi-head output layer is used for outputting a plurality of predicted urban environmental performance distribution maps based on the input data; the multi-target model is trained based on the multi-modal data set;

[0077] The model architecture of the application adopts an attention-enhanced conditional generative adversarial network (AE-cGAN). The AE-cGAN is a core model of the application for urban environmental performance prediction. Through the "generator-discriminator" double model architecture, deep fusion of multi-modal data (image+numerical value) is realized, and block energy consumption distribution map, photovoltaic power generation potential distribution map, outdoor thermal comfort (UTCI) distribution Figure Three prediction results are output in parallel, which provide basic data support for subsequent carbon emission accounting.

[0078] The generator (Generator) is a core innovative module of the AE-cGAN, adopts a customized structure of "double-path encoder→feature fusion layer→decoder→multi-head output", and is internally provided with an attention mechanism to adapt to multi-modal input and multi-task prediction requirements. The generator includes:

[0079] The double-path encoder is used for extracting different modal features; the double-path input encoder includes (1) an image encoding path for receiving image data input and extracting spatial features; and (2) a numerical encoding path for receiving numerical data input and extracting physical features.

[0080] The image data is multi-channel urban image data, including Google satellite image (including urban texture, building height), remote sensing data (including building function, ground temperature, solar radiation, vegetation index, wind speed) and other rasterized multi-channel image maps (the number of input channels is determined according to the actual number of image layers).

[0081] The image encoding path adopts a down-sampling structure based on U-Net, includes a plurality of sequentially connected down-sampling modules, and one convolutional block attention module (CBAM) is embedded after each down-sampling module.

[0082] The down-sampling module: through a convolutional layer (usually a 3x3 convolution), an activation function (such as ReLU), and a pooling layer (such as 2x2 max pooling), the feature dimension is reduced and spatial information is extracted, and macro-micro spatial features such as building layout and block texture are gradually captured.

[0083] CBAM: The spatial features extracted by the down-sampling module are filtered from two dimensions of channel attention and spatial attention - first, the channel attention weight distribution is allocated to highlight the key channels (such as the solar radiation channel) that affect energy consumption, photovoltaic, and thermal comfort; and then the spatial attention weight distribution is allocated to focus on key spatial positions such as building areas and vegetation areas, thereby improving feature utilization and model interpretability.

[0084] The image encoding path outputs multi-scale spatial feature maps through the plurality of down-sampling modules and CBAM, including feature maps of different resolutions generated during the down-sampling process, and a high-dimensional spatial feature map finally output.

[0085] The numerical data is structured numerical data, including meteorological station hourly data (annual / monthly / daily average dry bulb temperature, relative humidity, total solar radiation, average wind speed), social and economic statistical data, and fixed-dimension feature vectors (one-dimensional vectors, the dimension is determined according to the actual number of numerical variables) converted after cleaning and normalization.

[0086] The numerical encoding path adopts an independent multilayer perceptron (MLP) and includes multiple layers (such as 2-3 layers) of fully connected layers, each layer is matched with an activation function (such as a ReLU activation function), the low-dimensional numerical vector is mapped to a high-dimensional feature space matching the feature dimension output by the image encoding path, a high-dimensional numerical feature vector is output, the dimension is consistent with the flattened feature vector of the image encoding path, and subsequent splicing and fusion are ensured.

[0087] The feature fusion layer is used to realize deep integration of multi-modal features.

[0088] The input data of the feature fusion layer includes: a one-dimensional long vector converted from the high-dimensional spatial feature map output by the image encoding path through a flattening operation, and a high-dimensional numerical feature vector output by the numerical encoding path.

[0089] The feature fusion layer directly splices the above two vectors to form a fusion feature vector containing "spatial features + physical features" and outputs the same, solving the information fragmentation problem of traditional shallow fusion (such as post-processing correction).

[0090] The dimension of the fusion feature vector = the dimension of the one-dimensional long vector converted from the high-dimensional spatial feature map through the flattening operation + the dimension of the high-dimensional numerical feature vector.

[0091] The decoder is configured to generate a predicted feature map based on the fusion feature vector.

[0092] The input data of the decoder is the fusion feature vector output by the feature fusion layer (first converted into a feature map format through a dimension reshaping operation to match the input requirements of the decoder).

[0093] The decoder adopts a U-Net-based upsampling structure, including a plurality of sequentially connected upsampling modules, and is symmetrically connected (i.e., jump connection) with a downsampling module, gradually restores the resolution of the feature map through a plurality of upsampling modules (using transpose convolution to realize upsampling, such as 2x2 transpose convolution), while fusing the spatial features of the corresponding level of the downsampling module to retain detailed information.

[0094] The decoder finally outputs a high-dimensional predicted feature map consistent with the resolution of the input image, containing complete information after multi-modal fusion, providing a basis for subsequent multi-head output.

[0095] The multi-head output layer is configured to output multiple target prediction results in parallel.

[0096] The input data of the multi-head output layer is the high-dimensional predicted feature map output by the decoder.

[0097] The multi-head output layer adopts three independent output heads (such as 1x1 convolution layers), each output head corresponding to one prediction task, and is matched with a prediction task-specific activation function (to ensure that the output result meets the physical meaning, such as non-negative energy consumption / photovoltaic value, using Sigmoid or ReLU activation).

[0098] The multi-head output layer outputs three prediction maps with the same resolution as the input image through the three independent output heads. For example, the first output head outputs a street energy consumption distribution map; the second output head outputs a photovoltaic power generation potential distribution map; and the third output head outputs an outdoor thermal comfort (UTCI) distribution map.

[0099] The discriminator of the multi-target model is used to realize the adversarial training and precision constraint.

[0100] The discriminator adopts a multi-scale double discriminator to distinguish the predicted map output by the generator from the true value map, and to improve the fidelity of the predicted map by constraining the generator through an adversarial loss.

[0101] The input data of the discriminator includes conditional input and image to be discriminated. The conditional input refers to the multimodal input (image + numerical feature vector) consistent with the generator, which ensures that the discriminator judges the authenticity under the same conditions. That is, when the discriminator judges whether the image to be discriminated (predicted map output by the generator or true value map) is real, it needs to synchronously receive the multimodal basic data strongly associated with the image to be discriminated, provide the context conditions for the discriminator to judge, and ensure that the discrimination logic conforms to the physical scene of urban environment performance prediction.

[0102] The image to be discriminated refers to the predicted map (predicted block energy consumption distribution map, photovoltaic power generation potential distribution map and outdoor thermal comfort distribution map, single or multiple splicing) output by the generator, or the true value map (real block energy consumption distribution map, photovoltaic power generation potential distribution map and outdoor thermal comfort distribution map, obtained through official data set, Ladybug simulation and physical calculation respectively).

[0103] The discriminator includes a first sub-discriminator and a second sub-discriminator, which are respectively used as a global scale discriminator and a local scale discriminator. Through multi-scale design, different resolution image details can be focused on respectively.

[0104] The image to be discriminated input by the first sub-discriminator is the original resolution complete image (such as 256x256, 512x512 pixel image consistent with the input image) of the predicted map (or true value map) output by the generator. The first sub-discriminator focuses on capturing the global structural features of the image, and judges whether the overall spatial distribution of the predicted map conforms to the physical law (such as whether the high value area of energy consumption in a block matches the building dense area, and whether the high value area of photovoltaic potential matches the area with high roof area ratio), to avoid generating a predicted map with “global layout distortion” (such as energy consumption distribution completely disconnected from building morphology).

[0105] The second sub-discriminator input is the low-resolution image obtained by downsampling the predicted image (or the true value image) output by the generator (e.g., compressing the original resolution to 128x128, 64x64 pixels, and the resolution is usually 1 / 2 or 1 / 4 of the first sub-discriminator input). The second sub-discriminator focuses on capturing local detail features of the image, judging the pixel relevance and detail rationality of the local area in the predicted image (e.g., whether the thermal comfort gradient around a building is smooth, and whether the local influence of small green areas on energy consumption is logical), and avoiding generating a predicted image with "ambiguous or contradictory local details" (e.g., the energy consumption values of adjacent pixels suddenly change without physical basis).

[0106] Each sub-discriminator includes 3 layers of convolution layers connected in sequence, each layer of convolution layer includes 3x3 convolution, LeakyReLU activation function and batch normalization, and gradually reduces the dimension to extract the authenticity features of the image. The output of each sub-discriminator is one probability value (range 0-1), close to 1 indicates a true value image, and close to 0 indicates a generated predicted image.

[0107] Step four: input the image and numerical feature vector of the target city area into the trained multi-objective model, and output the predicted multi-city environment performance distribution map through the multi-objective model.

[0108] The generator is deeply customized in the application, and through the introduction of a multi-modal fusion encoder, an attention mechanism, and a multi-head output three key designs, it can efficiently and parallelly process multi-target city environment prediction tasks.

[0109] The encoder-decoder structure enhanced by multi-modal fusion attention: in order to enable the model to understand the spatial features of the image (such as building layout) and the physical meaning of the numerical value (such as the direct influence of temperature on energy consumption) at the same time, the application designs an encoder structure with double input. Among them, the image encoding path: a U-Net downsampling structure with a built-in attention module (CBAM) is used to deeply extract the spatial features of the input image. The numerical encoding path: a separate multi-layer perceptron (MLP) is designed to map the input one-dimensional numerical feature vector to a high-dimensional feature space. The feature fusion layer "flattens" the feature map output by the image encoding path before entering the decoder, becoming a long vector. Then, the image feature vector is spliced with the numerical feature vector output by the numerical encoding path to form a more comprehensive fusion feature vector containing two modal information. The fusion feature vector will be sent to the subsequent decoder for processing.

[0110] Attention enhanced U-Net structure: After each down-sampling module of the image encoding path, a convolution block attention module (CBAM) is embedded to enable the model to intelligently filter the spatial input features most important to the task. Parallel multi-head output layer and task-specific activation function: The feature map of the last layer of the decoder is sent into three independent 1x1 convolution output heads in parallel, each of which is responsible for generating a prediction map of a task. Each output head adopts a task-specific activation function to ensure the physical meaning of the output result.

[0111] In some embodiments, in order to balance and synergistically optimize the three parallel prediction tasks, the present application designs the following weighted combination loss function:

[0112] ;

[0113] In the formula, is the total loss, is the adversarial loss; 、 and are the pixel-level L1 losses for the block energy prediction task, the photovoltaic power generation potential prediction task, and the outdoor thermal comfort prediction task, respectively. 、 and are weight hyperparameters of 、 and , respectively.

[0114] wherein the pixel-level L1 loss refers to the sum of the absolute values of the numerical differences of all pixel points between the prediction map generated by the generator and the true value map of the corresponding task.

[0115] is the pixel-level L1 loss for the block energy prediction task, which measures the pixel numerical difference between the generated block energy distribution map and the true energy true value map (for example, the predicted energy of a certain pixel is 10 kWh / m 2 , the true energy is 12 kWh / m 2 , and the L1 loss of this pixel is 2).

[0116] is the pixel-level L1 loss for the photovoltaic power generation potential prediction task, which measures the pixel numerical difference between the generated photovoltaic power generation potential distribution map and the true photovoltaic true value map (for example, the predicted photovoltaic potential of a certain pixel is 500 kWh / year, and the true value is 480 kWh / year, and the L1 loss of this pixel is 20).

[0117] For the pixel-level L1 loss of the outdoor thermal comfort prediction task, the pixel value difference between the generated UTCI distribution map and the true UTCI ground truth map is measured (e.g., the predicted UTCI of a pixel is 28°C, and the true value is 29°C, the L1 loss of this pixel is 1).

[0118] In addition to the standard adversarial loss , the function also includes a weighted sum of the pixel-level L1 loss of each task.

[0119] By setting different weight hyperparameters (such as = 100, = 100, = 80), the convergence process of different tasks can be fine-tuned to ensure the optimal overall performance of the model.

[0120] Model training and hyperparameter setting:

[0121] Training process: The paired dataset is divided into a training set (90%) and a test set (10%). During training, by monitoring the change curve of the generator loss (LossG) and the discriminator loss (LossD), when the loss value tends to be stable and the fluctuation decreases in continuous periods, it is considered that the model has reached convergence, at which point training is stopped to prevent overfitting.

[0122] In some embodiments, to achieve optimal performance, in the simulation of the present application, after multiple optimizations, the key hyperparameter settings are as shown in Table 1.

[0123] ;

[0124] To quantitatively evaluate the prediction accuracy of the model of the present application, the recognized image quality evaluation indicators in the field are used to evaluate the prediction results on the test set, including:

[0125] Peak signal-to-noise ratio (PSNR): an indicator that measures the pixel-level difference between the predicted image and the true value image, the higher the value, the smaller the distortion.

[0126] Structural similarity (SSIM): an indicator that measures image similarity from brightness, contrast, and structure, with a value range of -1 to 1, the closer to 1, the higher the similarity.

[0127] Learned perceptual image patch similarity (LPIPS): a more human visual perception-based image similarity measure, the lower the value, the more similar the two images are in perception.

[0128] In a typical urban scene test, the evaluation results show that the model of the application achieves excellent performance, for example, the average SSIM value of the three prediction tasks is higher than 0.95, and the average LPIPS value is lower than 0.05, proving the high fidelity and high accuracy of the generated prediction map.

[0129] Step three: multi-target prediction;

[0130] Parallel prediction: input the multi-source image data of any target urban block into the trained AE-cGAN model.

[0131] Multi-dimensional output: the three output heads of the model will output three key performance prediction maps of the region at once: block energy consumption distribution map, photovoltaic power generation potential distribution map, and outdoor thermal comfort distribution map.

[0132] In some embodiments, the method of the application further comprises:

[0133] Carbon accounting: based on the predicted block energy consumption distribution map and photovoltaic power generation potential distribution map, combined with the local power grid carbon emission factor and clean energy emission reduction factor input by the user, perform the following calculation pixel by pixel (or building by building): net carbon emission = (predicted energy consumption x local power grid carbon emission factor) - (predicted photovoltaic power generation x clean energy emission reduction factor).

[0134] Thus, an intuitive net carbon emission intensity map can be generated.

[0135] In some embodiments, the net carbon emission intensity map is combined with the other three predicted Figure One For visual display, it provides closed-loop decision support for low-carbon planning.

[0136] Embodiment two:

[0137] The application provides a city environment performance prediction system based on multi-modal fusion and attention enhancement, comprising: a data acquisition module, a data preprocessing module, a multi-target model construction and training module, and a multi-target prediction module.

[0138] The data acquisition module is used to acquire multi-modal data and generate true value maps; the multi-modal data includes image data and numerical data, and the true value maps include various real city environment performance distribution maps.

[0139] The data acquisition module can obtain multi-modal data of urban blocks from different sources.

[0140] Among them, the image data acquisition includes: obtaining satellite images, aerial photographs, GIS data and other spatial geographic image information.

[0141] The numerical data collection includes: obtaining structured data related to the prediction target, such as hourly data of a weather station in the target area (temperature, humidity, wind speed, total solar radiation, etc.), social and economic statistical data, etc.

[0142] The data preprocessing module is configured to preprocess the collected multi-modal data respectively to obtain image graphs and numerical feature vectors, and pair the image graphs and the numerical feature vectors with true value graphs to form sample data matched with the image graphs, the numerical feature vectors and the true value graphs, and a multi-modal data set composed of a plurality of sample data.

[0143] The image data preprocessing includes: aligning, rasterizing and combining various image data into multi-channel image graphs.

[0144] The numerical data preprocessing includes: cleaning, normalizing and other operations on the collected numerical data, and converting the numerical data into numerical feature vectors with fixed dimensions.

[0145] Data pairing: pairing the preprocessed image graphs, corresponding numerical feature vectors and true value graphs to form complete sample data.

[0146] The multi-target model construction and training module is configured to construct a multi-target model of an attention-enhanced conditional generative adversarial network AE-cGAN; the generator of the multi-target model adopts a structure including a double-path encoder, a feature fusion layer, a decoder and a multi-head output layer connected in sequence; the double-path encoder includes an image encoding path for extracting spatial features based on the image graphs and a numerical encoding path for extracting physical features based on the numerical feature vectors; the image encoding path adopts a U-Net downsampling structure including a convolution block attention module, and the numerical encoding path adopts a multi-layer perceptron; the multi-head output layer is configured to output predicted multi-city environmental performance distribution graphs based on input data; the discriminator adopts a multi-scale double discriminator configured to distinguish the predicted graphs output by the generator from the true value graphs from global and local scales; and the multi-target model is trained based on the multi-modal data set.

[0147] The multi-target prediction module is configured to input the image graphs and the numerical feature vectors of a target city area into the trained multi-target model, and output predicted multi-city environmental performance distribution graphs through the multi-target model.

[0148] In some embodiments, the system further includes a carbon emission accounting module configured to calculate net carbon emissions pixel by pixel or building by building based on the predicted block energy consumption distribution graph and the photovoltaic power generation potential distribution graph, combined with local power grid carbon emission factors and clean energy emission reduction factors, to generate a net carbon emission intensity map.

[0149] The application provides a method and system for multi-modal urban environment performance prediction based on multi-modal data fusion and generative adversarial network. A unified deep learning model simultaneously receives multi-source urban image data and structured numerical data as input and outputs multiple different urban environment prediction indicators, including street energy consumption, photovoltaic power generation potential, and outdoor thermal comfort. The multi-objective model architecture innovation includes a dual-input encoder for fusion processing of image data and numerical data, an attention mechanism for intelligent selection of key input information, and a multi-head output architecture for parallel output of multiple different prediction results. The application provides an urban environment evaluation system integrating "multi-objective prediction" and "downstream accounting". The system first generates multiple basic performance prediction maps (energy consumption, photovoltaic, thermal comfort) through the aforementioned multi-modal fusion model, and then automatically transmits these prediction results to an independent carbon emission accounting module to calculate the final net carbon emission intensity map, realizing an end-to-end, automated process from multi-dimensional data input to final carbon audit.

[0150] Achieving comprehensive evaluation with multiple dimensions and high correlation: The application outputs three core indicators of energy consumption, photovoltaic, and thermal comfort in parallel through a unified model, which is more comprehensive in evaluation dimension than existing technologies that can only predict one indicator at a time. At the same time, the model can capture the internal correlation between indicators (such as the common influence of building form on the three indicators) during the learning process, and the prediction results are more consistent with physical logic.

[0151] Reducing data threshold: The application mainly relies on satellite images and remote sensing data that are easy to obtain on a large scale, overcoming the strong dependence of existing technologies on detailed and private building energy consumption historical data or physical parameters. This makes the solution capable of rapid deployment and application in data-scarce areas, with strong practicality and generalizability.

[0152] Higher prediction accuracy and robustness: By introducing numerical data directly related to the prediction target (such as weather parameters), the model can not only learn spatial distribution patterns from images, but also make corrections based on accurate physical quantities, greatly improving the accuracy and reliability of the prediction results. Compared to inference relying only on images, the solution has higher prediction accuracy and better interpretability due to the application of attention mechanisms.

[0153] The model has scenario simulation and analysis capabilities: The solution can evaluate the impact of different numerical conditions (such as different years of meteorological data and different electricity price strategies) on urban environment performance. By inputting new numerical vectors, the solution can quickly generate prediction results under corresponding scenarios, enabling scenario simulation and sensitivity analysis, greatly enhancing the practicality and decision support value of the solution.

[0154] Provide a closed-loop analysis capability from prediction to accounting: the application innovatively combines the upstream multi-objective prediction model with the downstream carbon emission accounting module to form a complete "prediction-evaluation-audit" closed loop. Users can not only see various performance indicators, but also directly obtain the final carbon emission conclusion, and the decision support capability is far superior to existing technologies.

[0155] Higher prediction accuracy and interpretability: due to the introduction of the attention mechanism, the model can adaptively assign feature weights for different tasks, which can theoretically achieve higher accuracy than multiple independent models, and provide visualized explanation for "why the model predicts like this".

[0156] Embodiment three:

[0157] The embodiment provides an electronic device, comprising a memory and a processor.

[0158] The memory is used for storing a computer program.

[0159] The processor is used for calling the computer program to execute the method in embodiment one.

[0160] Embodiment four:

[0161] The embodiment provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program runs on an electronic device, so that the electronic device implements the method in embodiment one.

[0162] Embodiment five:

[0163] The embodiment provides a computer program product, comprising a computer program, the computer program runs on an electronic device, so that the electronic device implements the method in embodiment one.

[0164] The specific implementation of the system, the electronic device, the computer readable storage medium and the computer program product provided by the embodiment of the application can refer to the specific embodiments of the above method, which will not be repeated here.

[0165] Obviously, those skilled in the art should understand that each unit or each step of the application described above can be realized by a general computing device, which can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and can be realized by program code executable by the computing device, so that it can be stored in the storage device and executed by the computing device, or it can be made into each integrated circuit module, or a plurality of modules or steps can be made into a single integrated circuit module. Thus, the application is not limited to any specific combination of hardware and software.

[0166] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Those skilled in the art can make various modifications and changes to the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0167] References:

[0168] [1] WINKELMANN F, BIRDSALL B, BUHL W, et al. DOE-2 supplement: version 2.1 E[R]. Lawrence Berkeley National Lab.(LBNL), Berkeley, CA (United States); Hirsch …, 1993.

[0169] [2] CRAWLEY DB, LAWRIE LK, WINKELMANN FC, et al. EnergyPlus: creating a new-generation building energy simulation program[J]. Energy and Buildings, 2001, 33(4): 319-331.

[0170] [3] SHRIVASTAVA R, KUMAR V, UNTAWALE S. Modeling and simulation of solar water heater: a TRNSYS perspective[J]. Renewable and Sustainable Energy Reviews, 2017, 67: 126-143.

[0171] [4] JIANG Q, HUANG C, WU Z, et al. Predicting building energy consumption in urban neighborhoods using machine learning algorithms[J]. Frontiers of Urban and Rural Planning, 2024, 2(1): 6.

[0172] [5] LI R, HUANG C, XIN W, et al. Data-driven optimization reveals the impact of Urban Heat Island effect on the retrofit potential of building envelopes[J]. Building and Environment, 2025, 269: 112367.

[0173] [6] NI H, WANG D, ZHAO W, et al. Enhancing rooftop solar energy potential evaluation in high-density cities: A Deep Learning and GIS based approach[J]. Energy and Buildings, 2024, 309: 113743.

Claims

1. A multi-modal fusion and attention enhancement-based urban environment performance prediction method, characterized in that, The method comprises the following steps: Collecting multi-modal data and generating a true value map; the multi-modal data comprises image data and numerical data, and the true value map comprises a plurality of real urban environment performance distribution maps; wherein the image data is collected by obtaining urban texture and building height, as well as building function, ground temperature, solar radiation, vegetation index and wind speed, and rasterizing the obtained data into image format to obtain the image data; the numerical data is collected by obtaining annual / monthly / daily average dry bulb temperature, relative humidity, total solar radiation and average wind speed; the plurality of urban environment performance distribution maps comprises a block energy consumption distribution map, a photovoltaic power generation potential distribution map and an outdoor thermal comfort distribution map; The collected multi-modal data is preprocessed to obtain an image map and a numerical feature vector, and the image map and the numerical feature vector are paired with the true value map to form sample data matched with the true value map, and a multi-modal data set composed of a plurality of sample data; A multi-objective model based on attention-enhanced conditional generative adversarial network (AE-cGAN) is constructed; the generator of the multi-objective model adopts a structure comprising a double-path encoder, a feature fusion layer, a decoder and a multi-head output layer connected in sequence; wherein the double-path encoder comprises an image encoding path for extracting spatial features based on the image map and a numerical encoding path for extracting physical features based on the numerical feature vector; the image encoding path adopts a U-Net downsampling structure comprising a convolution block attention module, and the numerical encoding path adopts a multi-layer perceptron; the multi-head output layer is used to output the predicted plurality of urban environment performance distribution maps based on the input data; and the multi-objective model is trained based on the multi-modal data set; The image map and the numerical feature vector of a target urban area are input into the trained multi-objective model, and the predicted plurality of urban environment performance distribution maps are output by the multi-objective model.

2. The method of claim 1, wherein, The urban texture and building height come from Google satellite images and GIS data; and the building function, ground temperature, solar radiation, vegetation index and wind speed come from remote sensing data. The annual / monthly / daily average dry bulb temperature, relative humidity, total solar radiation and average wind speed are obtained through a meteorological database and a government open data platform.

3. The method of claim 1, wherein, The feature fusion layer splices the flattened vector output by the image encoding path with the high-dimensional vector output by the numerical encoding path; the decoder restores the feature map resolution through symmetrical connection of an upsampling module and a downsampling module; and the multi-head output layer outputs the predicted plurality of urban environment performance distribution maps in parallel through three independent output heads.

4. The method of claim 1, wherein, The discriminator of the multi-objective model adopts a multi-scale double discriminator for distinguishing the predicted map output by the generator from the true value map from global and local scales.

5. The method of claim 1, wherein, The multi-objective model is trained by using a weighted combination loss function; wherein the weighted combination loss function is: ； wherein, is the total loss, is the adversarial loss; , and are the pixel-wise L1 loss for the block energy consumption prediction task, the photovoltaic power generation potential prediction task, and the outdoor thermal comfort prediction task, respectively; , and are the weight hyperparameters of , and , respectively.

6. The method of claim 1, wherein, The method further comprises a carbon emission accounting step: based on the predicted block energy consumption distribution map and photovoltaic power generation potential distribution map, combining local grid carbon emission factors and clean energy emission reduction factors, calculating net carbon emissions pixel by pixel or building by building to generate a net carbon emission intensity map.

7. A multi-modal fusion and attention enhanced based urban environment performance prediction system, characterized in that, The method comprises the following steps: The data acquisition module is configured to acquire multi-modal data and generate a true value map; the multi-modal data includes image data and numerical data, and the true value map includes a plurality of real urban environment performance distribution maps; wherein the image data is acquired by obtaining urban texture and building height, as well as building function, ground temperature, solar radiation, vegetation index and wind speed, and rasterizing the obtained data into an image format to obtain the image data; the numerical data is acquired by obtaining annual / monthly / daily average dry bulb temperature, relative humidity, total solar radiation and average wind speed; the plurality of urban environment performance distribution maps include a block energy consumption distribution map, a photovoltaic power generation potential distribution map and an outdoor thermal comfort distribution map; The data preprocessing module is configured to preprocess the acquired multi-modal data to obtain an image map and a numerical feature vector, and pair the image map and the numerical feature vector with the true value map to form sample data matched with the true value map, and a multi-modal data set composed of a plurality of sample data; The multi-target model construction and training module is configured to construct a multi-target model of an attention-enhanced conditional generative adversarial network (AE-cGAN); the generator of the multi-target model adopts a structure including a double-path encoder, a feature fusion layer, a decoder and a multi-head output layer connected in sequence; wherein the double-path encoder includes an image encoding path for extracting spatial features based on the image map and a numerical encoding path for extracting physical features based on the numerical feature vector; the image encoding path adopts a U-Net downsampling structure including a convolution block attention module, and the numerical encoding path adopts a multi-layer perceptron; the multi-head output layer is configured to output a plurality of predicted urban environment performance distribution maps based on input data; and the multi-target model is trained based on the multi-modal data set; The multi-target prediction module is configured to input the image map and the numerical feature vector of a target urban area into the trained multi-target model, and output a plurality of predicted urban environment performance distribution maps through the multi-target model. The system implements urban environment performance prediction by using the method of any one of claims 1-6.

8. An electronic device, comprising: comprises: a memory and a processor; the memory is configured to store a computer program; the processor is configured to invoke the computer program to execute the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer program is stored in the computer readable storage medium and is run on the electronic device, so that the electronic device implements the method of any one of claims 1-6.

10. A computer program product comprising a computer program, characterized in that, The computer program is run on the electronic device, so that the electronic device implements the method of any one of claims 1-6.

Images