Urban activity prediction method and system based on Transformer architecture and denoising diffusion probability model

Abstract

The present invention discloses a method and system for predicting urban activity based on a Transformer architecture and a denoised diffusion probability model. A trained model based on the Transformer architecture and denoised diffusion probability can be used to predict urban activity for any building over any future timeframe, given a known history of urban activity over any length of timeframe. The model construction specifically includes: fine-grained segmentation of spatiotemporal data by building and its internal room units, performing Z-Score normalization, inputting the spatiotemporal causal convolution module and adjusting the data structure to accommodate the Transformer structure, performing masking, and then enhancing the time series prediction capability using a noise diffusion probability model framework. The present invention can achieve high prediction accuracy with relatively short input times, and can achieve high-precision prediction of urban activity at both large scales and fine granularity. It is the first use of artificial intelligence to achieve arbitrary multi-step prediction of urban activity at any timeframe, filling a gap in artificial intelligence's capabilities in urban activity prediction.

Description

Technical Field

[0001] The present invention belongs to the field of urban activity prediction and relates to an innovative method for urban activity prediction using a Transformer architecture and a denoising diffusion probability model. Background Art

[0002] Urban activity, the sum total of all tangible and intangible activities within a city, encompasses a wide range of dimensions, including transportation, electricity supply, and economic activity. The core purpose of studying urban activity is to deeply explore the underlying patterns that govern the emergence, evolution, and demise of these activities. Notably, all urban activity stems from human behavior, and therefore, human activity plays a crucial role in the overall structure of urban activity. Constructing a model that can comprehensively measure and analyze at the macro level while maintaining high accuracy at the micro level has become a key challenge in urban research.

[0003] However, studying human activity is inherently complex, making it difficult to strike the perfect balance between model scale and accuracy. Past research has often been limited by challenges in data acquisition, computing power, and model design. This has resulted in either studying the changing trends of overall urban activity from a macro perspective or focusing on a relatively small sample of human behavior. In this process, micro-level studies of human behavior are difficult to effectively generalize to the macro-structural level, while macro-level population mobility data also struggle to accurately reflect the behavioral characteristics of micro-individuals. This trade-off between scale and accuracy makes related research fraught with challenges and uncertainties.

[0004] To address these challenges, researchers have attempted to approximate urban activity using a series of uncertain equations. For example, A.G. Wilson proposed using models and systems analysis to address urban issues in 1968. Since the 21st century, as urban issues have gained increasing attention, Hans van Ginkel and others have used models to describe the potential population explosion in future cities. Even before the advent of the artificial intelligence era, researchers such as Chaoming Song have conducted in-depth research on the patterns of human mobility. However, the article also emphasized that due to the limitations of the technology available at the time, this research still had many shortcomings and flaws.

[0005] In recent years, the continuous development of end-to-end training AI technology, particularly within the framework of black-box-versus-black-box models, has successfully solved macroscopic problems such as global weather and microscopic problems such as protein configuration. This has provided inspiration and hope for a new approach: one that measures the composition and effectiveness of models at a human scale and can predict, judge, and influence human choices and behavior.

[0006] Since 2017, the Transformer and its variants have been widely used in various fields, particularly in language modeling. The Transformer has demonstrated high efficiency and groundbreaking generative performance, and its variants, such as ViT and its variant DiT, have demonstrated capabilities that surpass traditional models. In the field of weather forecasting, systems based on the Transformer framework have also achieved breakthroughs. For example, the Pangu model, trained using ViT and a variant of the Swin Transformer, achieves global-scale weather forecasts. These successes suggest the potential for pre-training and generating predictive models based on Transformers for urban data. UniST further explores this approach and integrates multivariate data from cities. However, despite its training scale encompassing the entire size of a city, its prediction accuracy remains limited to two-dimensional space, lacking the precision of three-dimensional space. This poses certain challenges for its practical application. Summary of the Invention

[0007] The present invention aims to address a key issue currently existing in urban activity forecasting, namely how to effectively address the large-scale characteristics of urban space while maintaining prediction precision. Traditional forecasting methods often struggle to strike a balance between the macroscopic urban level and the microscopic individual activities, resulting in forecasts that are either too general and lack precision, or too complex and difficult to implement. Therefore, the present invention proposes an urban activity forecasting method based on a Transformer architecture and a denoising diffusion probability model, aiming to achieve comprehensive and accurate predictions of urban activities through refined data processing and an advanced model architecture.

[0008] The technical solution adopted in the present invention is as follows:

[0009] The urban activity prediction method based on the Transformer architecture and denoised diffusion probability model includes using a trained model based on the Transformer architecture and denoised diffusion probability to predict urban activities of any building for any length of future time given any length of historical urban activity. The model construction method includes the following:

[0010] (S1) The acquired spatiotemporal data are divided into fine-grained units according to the building and its internal room units. buildings, obtain their time series data, and divide them into time steps and vertical Layer and horizontal Room units, the personnel data of each place is expressed as: ;

[0011] (S2) Perform Z-Score normalization on the data after fine-grained division, and the obtained data is expressed as ;

[0012] (S3) Data after Z-Score standardization The input of the spatiotemporal causal convolution module ensures temporal causality, and the output of the spatiotemporal causal convolution module is ;

[0013] (S4) in The two-dimensional feature is expanded into a length of A one-dimensional vector to adapt to the Transformer structure;

[0014] The result after expansion is ;

[0015] (S5) Data Masking is performed to improve the robustness of the model in scenarios with small samples and incomplete data. The output after masking is ;

[0016] (S6) Use the noise diffusion probability model framework to enhance the time series prediction capability and conduct model training and prediction.

[0017] In the above technical solution, further, the step (S3) specifically includes:

[0018] (S3.1) Z-Score normalized data Split by building, for each building there are: ,in ;

[0019] (S3.2) Yes Perform horizontal convolution, that is Convolution on the dimension to ensure that the information of neighboring room units is obtained;

[0020] (S3.3) Perform depthwise convolution on the result of (S3.2), i.e. Convolution on the dimension to ensure the acquisition of temporal information;

[0021] (S3.4) The result of (S3.3) is input into the multilayer perceptron to compress the convolution kernel thickness dimension. The output of the multilayer perceptron is ;

[0022] (S3.5) For all buildings Merge, the result is .

[0023] Furthermore, the lateral convolution and the depth convolution are both based on the Causa Inception 3D structure, and include three groups of three-dimensional convolutions with different expansion rates, each group containing 64 convolution kernels.

[0024] Furthermore, the mask matrix in step (S5) is , the masking method used is one or more of the following:

[0025] (1) Random mask: randomly mask the features in a certain proportion, that is, Randomly select a certain proportion of elements and set them to 0. The elements at the rest of the positions are set to 1;

[0026] (2) Entire column masking: Masking the entire segment along the feature dimension to test the ability to recover from the complete loss of a specific feature, that is, in Dimension selects a dimension value, and all elements of this feature dimension are set to 0. The elements at the rest of the positions are set to 1;

[0027] (3) Masking the entire time step data: Masking the entire time step data to test the model's ability to recover from time series loss, that is, in Dimension selects a dimension value, and all elements of this feature dimension are set to 0. The elements at the rest of the positions are set to 1;

[0028] (4) Block masking: Masking continuous blocks in the feature dimension to simulate the scene of local information loss, that is, in Dimension selects several continuous dimension values ​​and in The dimension selects several continuous dimension values, and all elements of the feature dimension are set to 0. The elements at the remaining positions are set to 1.

[0029] Furthermore, the step (S6) specifically includes:

[0030] (S6.1) Data to be masked Split by building, for each building there is in ; For each building at each time step, ,in ;

[0031] (S6.2) For each building, Select a time subsequence from a sequence , and 1< ,have , , to The time range is the historical period. to The time range is in the future;

[0032] (S6.3) The diffusion process is defined as the process from pure data Gradually adding Gaussian noise yields: ; Specifically, the transition probability of each step follows a Gaussian distribution, that is: ;in , is a parameter that controls the degree of noise addition;

[0033] By accumulating these parameters, we get With initial data There is a direct relationship between the and random sampling time t, we get based on expression,

[0034] Thus As Input to the noise diffusion probability model; in the forward propagation After step 1 ;

[0035] (S6.4) Obtaining data from noisy Restore the original signal The transition probability of the inverse diffusion process follows a Gaussian distribution, which will , , as Input to the noise diffusion probability model; in the back propagation After step 1 ;

[0036] (S6.5) Perform position encoding: Data to be masked Split by building, for each building there is ,in ; For each building at each time step, ,in ;for Use sine and cosine functions for position encoding;

[0037] (S6.6) Use the Transformer model to learn the parameters of the transition probability of the inverse diffusion process. Input the Transformer model to get ;

[0038] (S6.7) Add a multilayer perceptron as a decoder at the output of the model. Mapping back , making prediction results understandable to humans and practical applications;

[0039] (S6.8) Compare the noise during forward propagation and the noise during back propagation , the training goal is to minimize the difference between the noise predicted by the model and the true noise.

[0040] Furthermore, the sine and cosine functions are position-encoded, specifically:

[0041]

[0042] in The result after encoding the data of each building and each time step is: The data at each point The position in is an even number, for .

[0043] Furthermore, for the structure The new data has the same time series partitioning method as in step (S6.2). to The time range is the historical period. to The time range is the future time, and the data of the future time part is set to 0 when entering the model. Input into the trained model to predict to Data whose time range is in the future.

[0044] The present invention also provides an urban activity prediction system based on a Transformer architecture and a denoising diffusion probability model, which is used to implement the method described in any of the above items.

[0045] The present invention further provides an electronic device, comprising:

[0046] one or more processors;

[0047] a memory for storing one or more programs;

[0048] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the above methods.

[0049] The present invention also provides a computer-readable storage medium storing computer-executable instructions, wherein the instructions are used to implement any of the above methods when executed.

[0050] Compared with the existing technology, the urban activity prediction method of the present invention shows significant advantages and positive effects:

[0051] (1) A method for predicting urban activities based on the Transformer architecture and denoising diffusion probability model was proposed. A model with a size of 187M was successfully constructed, and high prediction accuracy was achieved with a short input time. This method can achieve high-precision prediction of urban activities at large scales and fine granularity.

[0052] (2) For the first time, artificial intelligence was used to achieve arbitrary multi-step prediction of urban activities at arbitrary time steps, filling the gap in artificial intelligence in urban activity prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The specific embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0054] Figure 1 This is a flow chart of the urban activity prediction method based on the Transformer architecture and denoising diffusion probability model of the present invention;

[0055] Figure 2 yes Figure 1 Flowchart of step S3 in FIG.

[0056] Figure 3 yes Figure 1 Flowchart of step S6 in FIG.

[0057] Figure 4 This is the comparison result of the simulation prediction using the data of a certain day in Beijing. DETAILED DESCRIPTION

[0058] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] like Figure 1 As shown, according to a specific embodiment of the present invention, a new prediction system constructed by the present invention integrates the Building Transformer (BiT) model and the denoised diffusion probability model to achieve comprehensive and accurate prediction of urban activities. The specific implementation steps of the urban activity prediction method based on the Transformer architecture and the denoised diffusion probability model are as follows:

[0060] (S1) The acquired spatiotemporal data is divided into fine-grained units according to the building and its internal room units. buildings, obtain their time series data, and divide them into time steps and vertical Layer and horizontal Room units, the personnel data of each place is expressed as: The data structure in the form of a four-dimensional matrix will be used for spatiotemporal prediction in subsequent steps.

[0061] (S2) Perform Z-Score normalization on the data after fine-grained division. The data after Z-Score normalization is expressed as:

[0062] in and Respectively represent The overall mean and standard deviation of .

[0063] (S3) Z-Score normalizes the data Input the spatiotemporal causal convolution module to ensure temporal causality. The output of the spatiotemporal causal convolution module is .

[0064] (S4) in The two-dimensional feature is expanded into a length of A one-dimensional vector to adapt to the Transformer structure. That is:

[0065]

[0066] The result after expansion is .

[0067] (S5) Introduce a variety of masking strategies to improve the robustness of the model in scenarios with small samples and incomplete data. The mask matrix is , so the output after masking is:

[0068]

[0069] in is the Hadamard product operator.

[0070] The present invention employs four different types of masking methods:

[0071] (1) Random mask: randomly mask the features in a certain proportion. Randomly select a certain proportion of elements and set them to 0. The elements at the remaining positions are set to 1.

[0072] (2) Entire column masking: Masking the entire segment along the feature dimension to test the ability to recover from the complete loss of a specific feature. in Dimension selects a dimension value, and all elements of this feature dimension are set to 0. The elements at the remaining positions are set to 1.

[0073] (3) Masking the entire time step data: Masking the entire time step data to test the model’s ability to recover from time series loss. in Dimension selects a dimension value, and all elements of this feature dimension are set to 0. The elements at the remaining positions are set to 1.

[0074] (4) Block masking: Masking continuous blocks in the feature dimension to simulate the scene of local information loss. in Dimension selects several continuous dimension values ​​and in The dimension selects several continuous dimension values, and all elements of the feature dimension are set to 0. The elements at the remaining positions are set to 1.

[0075] (S6) A noise diffusion probability model framework is used to enhance the time series prediction capability and conduct model training and prediction.

[0076] (S7) The trained model is used to predict any building in the future at any length of time when any length of historical time is known. The new data has the same time series partitioning method as in step (S6.2). to The time range is the historical period. to The time range is the future time, and the data of the future time part is set to 0 when entering the model. Input into the trained model to predict to Data whose time range is in the future.

[0077] Among them, such as Figure 2 As shown, step (S3) is subdivided into the following five sub-steps:

[0078] (S3.1) Z-Score the normalized data Split by building, for each building there are:

[0079] in .

[0080] (S3.2) Yes Perform horizontal convolution, that is The horizontal convolution is based on the Causa Inception 3D structure and contains three groups of 3D convolutions with different expansion rates, each group contains 64 convolution kernels.

[0081] (S3.3) Perform depthwise convolution on the result of (S3.2), i.e. The depthwise convolution is based on the Causa Inception 3D architecture and consists of three groups of 3D convolutions with different dilation rates, each containing 64 convolution kernels.

[0082] (S3.4) Input the result of (S3.2) into the multilayer perceptron to compress the convolution kernel thickness dimension. The output of the multilayer perceptron is .

[0083] (S3.5) For all buildings Merge, the result is .

[0084] like Figure 3 As shown, step (S6) is subdivided into the following 6 sub-steps:

[0085] (S6.1) Data to be masked Split by building, for each building there is (in ); For each building at each time step, (in ).

[0086] (S6.2) For each building, Select a time subsequence from a sequence , and 1< ,have (in ), to The time range is the historical period. to The time range is in the future.

[0087] (S6.3) The diffusion process is defined as the process from pure data Gradually adding Gaussian noise yields: Specifically, the transition probability of each step follows a Gaussian distribution, namely:

[0088] in , is a parameter that controls the degree of noise addition.

[0089] By accumulating these parameters, we get With initial data The direct relationship between them is:

[0090]

[0091] in .

[0092] At a given noise and random sampling time t, it can be deduced that The expression:

[0093]

[0094] According to this formula, (in ) as Input to the noise diffusion probability model; in the forward propagation After step 1 .

[0095] (S6.4) Obtaining data from noisy Restore the original signal The transition probability of the inverse diffusion process is:

[0096]

[0097] in For Restore The mean of this conditional probability; For Restore The covariance of this conditional probability.

[0098] It also follows the Gaussian distribution. According to this formula, the future time ( ) as Input to the noise diffusion probability model; in the back propagation After step 1 .

[0099] (S6.5) Perform position encoding. The data to be masked Split by building, for each building there is (in ); For each building at each time step, (in ).for The sine and cosine functions are used for position encoding, and we have:

[0100]

[0101] in The result after encoding the data of each building and each time step is: The data at each point The position in An even number.

[0102]

[0103] (S6.6) Use the Transformer model to learn the parameters of the transition probability of the inverse diffusion process. Input the Transformer model to get The multi-head self-attention mechanism is the core component of the Transformer model:

[0104]

[0105] in:

[0106]

[0107] in 、 、 and is a learnable weight matrix, is the number of heads.

[0108] (S6.7) A simple multilayer perceptron is added as a decoder at the output of the model. Mapping back , making the prediction results understandable to humans and practical applications.

[0109] (S6.8) Compare the noise during forward propagation and the noise during back propagation , the training goal is to minimize the difference between the noise predicted by the model and the true noise, that is:

[0110]

[0111] The method of the present invention is used to analyze the data of Beijing on a certain day. The predicted results are similar to the actual results. Figure 4 As shown, the present invention can achieve accurate prediction of urban activities.

[0112] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0113] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0114] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0115] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0116] The embodiments described above are merely some preferred embodiments of the present invention and are not intended to limit the present invention. Persons skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent substitution or equivalent transformation falls within the scope of protection of the present invention.

Claims

1. Urban activity prediction method based on Transformer architecture and denoising diffusion probability model, characterized by: A trained model based on the Transformer architecture and denoised diffusion probability is used to predict urban activities of any building in the future for any length of time given the urban activities of any length of historical time. The model construction method includes the following: (S1) The acquired spatiotemporal data are divided into fine-grained units according to the building and its internal room units. buildings, obtain their time series data, and divide them into time steps and vertical Layer and horizontal Room units, the personnel data of each place is expressed as: ; (S2) Perform Z-Score normalization on the data after fine-grained division, and the obtained data is expressed as ; (S3) Data after Z-Score standardization The input of the spatiotemporal causal convolution module ensures temporal causality, and the output of the spatiotemporal causal convolution module is ; (S4) in The two-dimensional feature is expanded into a length of A one-dimensional vector to adapt to the Transformer structure; The result after expansion is ; (S5) Data Masking is performed to improve the robustness of the model in scenarios with small samples and incomplete data. The output after masking is ; (S6) Employ a noise diffusion probability model framework to enhance time series prediction capabilities and conduct model training and prediction, specifically including: (S6.1) Data to be masked Split by building, for each building there is in ; For each building at each time step, ,in ; (S6.2) For each building, Select a time subsequence from a sequence , and 1< ,have , , to The time range is the historical period. to The time range is in the future; (S6.3) The diffusion process is defined as the process from pure data Gradually adding Gaussian noise yields: ; Specifically, the transition probability of each step follows a Gaussian distribution, that is: ;in , is a parameter that controls the degree of noise addition; By accumulating these parameters, we get With initial data There is a direct relationship between the and random sampling time t, we get based on expression, Thus As Input to the noise diffusion probability model; in the forward propagation After step 1 ; (S6.4) Obtaining data from noisy Restore the original signal The transition probability of the inverse diffusion process follows a Gaussian distribution, which will , , as Input to the noise diffusion probability model; in the back propagation After step 1 ; (S6.5) Perform position encoding: Data to be masked Split by building, for each building there is ,in ; For each building at each time step, ,in ;for Use sine and cosine functions for position encoding; (S6.6) Use the Transformer model to learn the parameters of the transition probability of the inverse diffusion process, and combine the position encoding with Input the Transformer model to get ; (S6.7) Add a multilayer perceptron as a decoder at the output of the model. Mapping back , making prediction results understandable to humans and practical applications; (S6.8) Compare the noise during forward propagation and the noise during back propagation , the training goal is to minimize the difference between the noise predicted by the model and the true noise.

2. The urban activity prediction method based on the Transformer architecture and denoising diffusion probability model according to claim 1 is characterized in that: The step (S3) specifically includes: (S3.1) Z-Score normalized data Split by building, for each building there are: ,in ; (S3.2) Yes Perform horizontal convolution, that is Convolution on the dimension to ensure that the information of neighboring room units is obtained; (S3.3) Perform depthwise convolution on the result of (S3.2), i.e. Convolution on the dimension to ensure the acquisition of temporal information; (S3.4) The result of (S3.3) is input into the multilayer perceptron to compress the convolution kernel thickness dimension. The output of the multilayer perceptron is ; (S3.5) For all buildings Merge, the result is .

3. The urban activity prediction method based on the Transformer architecture and denoising diffusion probability model according to claim 2 is characterized in that: The lateral convolution and the depth convolution are both based on the Causa Inception 3D structure, and include three groups of 3D convolutions with different expansion rates, each group including 64 convolution kernels.

4. The urban activity prediction method based on the Transformer architecture and denoising diffusion probability model according to claim 1 is characterized in that: The mask matrix in step (S5) is , the masking method used is one or more of the following: (1) Random mask: randomly mask the features in a certain proportion, that is, Randomly select a certain proportion of elements and set them to 0. The elements at the rest of the positions are set to 1; (2) Entire column masking: Masking the entire segment along the feature dimension to test the ability to recover from the complete loss of a specific feature, that is, in Dimension selects a dimension value, and all elements of this feature dimension are set to 0. The elements at the rest of the positions are set to 1; (3) Masking the entire time step data: Masking the entire time step data to test the model's ability to recover from time series loss, that is, in Dimension selects a dimension value, and all elements of this feature dimension are set to 0. The elements at the rest of the positions are set to 1; (4) Block masking: Masking continuous blocks in the feature dimension to simulate the scene of local information loss, that is, in Dimension selects several continuous dimension values ​​and in The dimension selects several continuous dimension values, and all elements of the feature dimension are set to 0. The elements at the remaining positions are set to 1.

5. The urban activity prediction method based on the Transformer architecture and denoising diffusion probability model according to claim 1 is characterized in that: The sine and cosine functions are position-encoded, specifically: , in The result after encoding the data of each building and each time step is: The data at each point The position in is an even number, for .

6. The urban activity prediction method based on the Transformer architecture and denoising diffusion probability model according to claim 1 is characterized in that: For the structure The new data has the same time series partitioning method as in step (S6.2). to The time range is the historical period. to The time range is the future time, and the data of the future time part is set to 0 when entering the model. Input into the trained model to predict to Data whose time range is in the future.

7. An urban activity prediction system based on Transformer architecture and denoising diffusion probability model, characterized by: Used to implement the method according to any one of claims 1 to 6.

8. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing computer-executable instructions, wherein the instructions are used to implement the method according to any one of claims 1 to 6 when executed.

Images