Smart city environment health and safety risk prediction system and method based on multi-mode perception

Abstract

The invention discloses a smart city environment health and safety risk prediction system and method based on multi-modal perception, and relates to the field of smart city management, artificial intelligence and data fusion domains. The system comprises a heterogeneous data acquisition module used for acquiring and standardizing multi-modal heterogeneous data including time sequence environment data, space security and protection data and graph structure health association data in city management; the heterogeneous data field alignment module is used for carrying out cross-domain distribution alignment and modal-level feature calibration on the standardized multi-modal heterogeneous data; the joint representation learning module is used for coding and fusing the aligned and calibrated time sequence, space and graph structure features and outputting a fused feature vector; and the AI large-modulus prediction module is used for carrying out classification or regression prediction on environmental health and safety risks according to the fusion feature vector. Therefore, the problems of difficulty in cross-domain alignment, poor fusion effect, weak prediction model generalization ability and the like are solved, and the accuracy and robustness of the risk prediction task are improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the fields of smart city management, artificial intelligence and data fusion technology, and in particular to a smart city environmental health and safety risk prediction system and method based on multimodal perception. Background Technology

[0002] With the deepening of smart city construction, massive amounts of multi-source heterogeneous data have been accumulated in fields such as environmental monitoring, public safety and prevention, and public health management. These include time-series environmental monitoring data (such as PM2.5, water quality, and noise), spatial security monitoring data (such as video footage and personnel / vehicle trajectories), and graph-structured health-related data (such as resident health records and medical institution distribution networks). The effective integration of these data is the core foundation for achieving refined management of urban environmental health and safety.

[0003] However, existing technologies face significant bottlenecks when processing such multi-source heterogeneous data: First, the statistical distribution of different types of data exhibits significant heterogeneity. The dynamic fluctuations of time-series data, the geographical correlation of spatial data, and the node dependence of graph-structured data lead to significant differences in cross-domain distribution, making it difficult for traditional methods to achieve effective alignment. Second, the separation of data structure and modal characteristics results in poor feature fusion performance. Simple splicing or weighted fusion methods cannot fully explore cross-modal correlation knowledge, easily leading to a single modality dominating the fusion process. Third, existing domain adaptive methods are mostly designed for single-type heterogeneous problems, making it difficult to simultaneously address the three types of problems: heterogeneous distribution, structural differences, and modal separation. This results in poor generalization ability and insufficient prediction accuracy of downstream risk prediction models. Fourth, most fusion schemes do not consider the dynamic changes in the scale of data sources, resulting in insufficient adaptability in scenarios with a small number of regional data sources and massive sensor data sources. Furthermore, they lack constraints on the balance of modal contributions, further affecting model performance.

[0004] Furthermore, existing technologies often deeply couple data fusion with prediction models, resulting in poor reusability of the core fusion logic and high dependence on downstream prediction models, making it difficult to adapt to model replacement needs in different scenarios. Meanwhile, technologies such as digital twins and encrypted transmission often exist as independent auxiliary modules, failing to form efficient collaboration with the core data fusion logic and thus failing to fully realize the value of the data. These problems severely restrict the level of intelligence and precision in smart city environmental health and safety management. Summary of the Invention

[0005] To address the technical pain points of existing technologies for multi-source heterogeneous data (time-series environmental data, spatial security data, and graph-structured health-related data), such as heterogeneous statistical distribution, differences in data structure, and fragmented modal characteristics, as well as the resulting difficulties in cross-domain alignment, poor fusion effects, and weak generalization ability of prediction models, the core objective of this invention is to provide a smart city environmental health and safety risk prediction system and method based on multimodal perception.

[0006] In a first aspect, embodiments of the present invention provide a smart city environmental health and safety risk prediction system based on multimodal perception, comprising a heterogeneous data acquisition module, a heterogeneous data domain alignment module, a joint representation learning module, and an AI large-scale model risk prediction module connected in sequence. The heterogeneous data acquisition module is used to acquire and standardize multimodal heterogeneous data in urban management, including temporal environmental data, spatial security data, and graph structure health correlation data. The heterogeneous data domain alignment module is used to perform cross-domain distribution alignment and modal-level feature calibration on the standardized multimodal heterogeneous data. The heterogeneous data domain alignment module adopts a dual-adversarial domain alignment neural network architecture, including a domain-level adversarial alignment submodule and a modal-level feature calibration submodule. The joint representation learning module encodes and weights the temporal, spatial, and graph structure features obtained after alignment and calibration, outputting a fixed-dimensional fused feature vector. The AI ​​large-scale model risk prediction module is used to classify or regress environmental health and safety risks based on the fused feature vector, outputting risk prediction results including risk type and level.

[0007] Furthermore, the domain-level adversarial alignment submodule includes: a shared feature extraction network, employing a network structure comprising modality-specific branches and a cross-domain shared backbone, wherein the modality-specific branches include 1D convolutional layers for processing temporal environmental data, 2D convolutional layers for processing spatial security data, and graph convolutional layers for processing graph-structured health association data; the cross-domain shared backbone is composed of multi-layer Transformer encoders; and at least one domain discriminator, used to minimize the distribution differences between different data sources in the features extracted by the shared feature extraction network through adversarial training.

[0008] Furthermore, the domain discriminator is adapted to two implementation methods based on the size of the data source: Implementation method 1: When the number of data sources is less than or equal to a preset threshold, an independent discriminator is configured for each data source; Implementation method 2: When the number of data sources is greater than the preset threshold, a single condition discriminator is configured, the input of which is a concatenated vector of features and data source identifiers.

[0009] Furthermore, the modal-level feature calibration submodule is implemented through a weak parameter sharing mechanism, which is configured to: calculate the relationship between feature extraction parameters from different data sources under the same modality and the shared parameters of that modality. Norm deviation; regularization constraints are applied to the deviation based on a preset penalty coefficient to balance modal specificity and commonality; the calibrated modal features are then... Normalization is applied to eliminate scale differences between different modes.

[0010] Furthermore, the joint representation learning module is specifically a temporal-spatial-graph structure three-stream feature fusion network, including: a three-stream feature encoding submodule, which uses temporal encoding branch, spatial encoding branch and graph structure encoding branch respectively to perform deep encoding on the temporal, spatial and graph structure features processed by the heterogeneous data domain alignment module, and outputs feature vectors of equal dimensions; and a heterogeneous feature cross-fusion submodule, which is used to dynamically assign fusion weights to the modal feature vectors output by the three-stream feature encoding submodule, and perform weighted fusion to output the fixed-dimensional fused feature vector.

[0011] Furthermore, the temporal coding branch includes a bidirectional long short-term memory network and a multi-head attention mechanism; the spatial coding branch includes a convolutional neural network and a Transformer encoder; and the graph structure coding branch includes a multi-layer graph attention network.

[0012] Furthermore, the heterogeneous feature cross-fusion submodule is configured to perform the following operations: dynamically calculate the initial fusion weights of each modality based on the correlation between each modality feature vector and the downstream prediction task; normalize the initial fusion weights to obtain the final fusion weights of each modality, and introduce a modality contribution balance constraint during the normalization process, which is implemented by penalizing the degree to which the final fusion weights deviate from a preset balance value; use the final fusion weights to weight and concatenate the modality feature vectors to generate the fusion feature vector.

[0013] Furthermore, the system performs end-to-end training through multi-objective optimization, and the total loss function is composed of a weighted sum of downstream prediction loss, domain adversarial loss, weak parameter sharing loss, and modal contribution balancing loss.

[0014] Furthermore, the AI ​​large model risk prediction module is decoupled from the joint representation learning module, and is used to receive the fused feature vector and simultaneously output two types of prediction results: the first type of result is the risk type, which is output through the classification prediction head and is used to identify the specific category of environmental health or safety risk; the second type of result is the risk level, which is output through the regression prediction head and is used to quantify the severity or probability of occurrence of the risk type.

[0015] Secondly, embodiments of the present invention also provide a method for predicting environmental health and safety risks in smart cities based on multimodal perception, applied to the system described in the first aspect. The method includes: a data acquisition and standardization step: acquiring temporal environmental data, spatial security data, and graph structure health association data in urban management, and standardizing the multimodal heterogeneous data; a cross-domain alignment and feature calibration step: inputting the standardized data into a heterogeneous data domain alignment module; reducing the distribution differences between different data sources and different modalities through domain-level adversarial alignment and modality-level feature calibration, and calibrating the features; a joint representation learning step: inputting the aligned and calibrated temporal, spatial, and graph structure features into a joint representation learning module; encoding and dynamically weighting the features based on modality contribution balance constraints through a temporal-spatial-graph structure three-stream feature fusion network to generate a fixed-dimensional fusion feature vector; and a risk prediction step: inputting the fusion feature vector into an AI large-scale model risk prediction module, and outputting the prediction results of the type and level of environmental health and safety risks.

[0016] Thirdly, embodiments of the present invention also provide an electronic device, which includes: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the smart city environmental health and safety risk prediction method based on multimodal perception described in the second aspect of the present invention.

[0017] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for predicting the environmental health and safety risks of smart cities based on multimodal perception as described in the second aspect of the present invention.

[0018] Compared with existing technologies, the present invention achieves the following beneficial effects: (1) Significant cross-domain alignment effect and greatly improved generalization ability: The dual adversarial domain alignment network of the present invention effectively reduces the cross-domain distribution differences of temporal, spatial and graph structure data through the dual effects of domain-level adversarial and modal-level calibration. The cross-regional generalization error is reduced by 30.4% compared with the traditional domain adaptive method, so that the model can maintain stable performance in different regions and different data source scales, and solves the core pain point of weak generalization ability of existing methods.

[0019] (2) Excellent heterogeneous fusion quality and high feature discrimination: The three-stream feature fusion module designs a dedicated coding architecture for the structural characteristics of different types of data. Combined with modal contribution balance constraints, the fused features retain the core information of each modality and fully explore cross-modal correlation knowledge, effectively improving feature discrimination. In experimental verification, the risk prediction accuracy based on this fused feature reached 82.1%, and the weighted F1-Score reached 80.5%, which are 6.8% and 7.4% higher than traditional fusion methods, respectively. It has a significant advantage, especially in the prediction of cross-modal correlation events.

[0020] (3) Strong adaptability and flexible response to diverse scenarios: The domain-level adversarial alignment unit provides two optional implementation methods, which are adapted to regional scenarios with ≤50 data sources and massive sensor scenarios with >50 data sources respectively. Scenario switching can be achieved without reconstructing the network. At the same time, the front-end module is decoupled from the downstream AI large model, supporting flexible replacement of the prediction model and two training modes (fine-tuning the prediction head only / end-to-end training), greatly improving the reusability and adaptability of the technical solution.

[0021] (4) Stable and efficient training with controllable deployment costs: The multi-objective optimization training system balances prediction loss, domain adversarial loss, weak parameter sharing loss and modality balance loss. It adopts AdamW optimizer and alternating training strategy to ensure that the model converges stably within 200 iterations. Compared with similar domain adaptive methods, the training time per round only increases by 9.6%. It can be deployed efficiently with conventional hardware resources to meet the needs of real-time management of smart cities.

[0022] (5) Excellent robustness and strong anti-interference ability: The weak parameter sharing mechanism achieves this through a smooth... Regularization constraints enable "flexible adaptation to small deviations and gradual tightening of large deviations," avoiding performance fluctuations caused by extreme parameter deviations. Modal contribution balance constraints and prediction diversity guarantee mechanisms enable the model to have a strong tolerance for abnormal situations such as data noise and partial mode loss, further improving the engineering practicality of the technical solution. Attached Figure Description

[0023] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a schematic diagram of the modules of the smart city environmental health and safety risk prediction system based on multimodal perception provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the processing flow of the heterogeneous data domain alignment module and the joint representation learning module provided in the embodiments of the present invention; Figure 3 This is a schematic diagram of the system multi-objective optimization training process according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the intelligent management platform interface provided in an embodiment of the present invention; Figure 5 This is a flowchart illustrating the method for predicting environmental health and safety risks in smart cities based on multimodal perception, as provided in an embodiment of the present invention. Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0024] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0025] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as being processed sequentially, many of these operations (or steps) may be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations may be rearranged. The process may be terminated when its operation is completed, but may also have additional steps not included in the figures. The process may correspond to a method, function, procedure, subroutine, subroutine, etc.

[0026] Addressing the technical challenges in smart city environmental health and safety management, such as heterogeneous statistical distributions, differing data structures, and fragmented modal characteristics among temporal environmental data, spatial security data, and graph-structured health-related data, existing methods struggle to effectively align and fuse heterogeneous data. This results in poor generalization ability and insufficient prediction accuracy in downstream risk prediction models. This invention aims to design a neural network architecture to effectively align and fuse temporal, spatial, and graph-structured data with different statistical distributions, data structures, and modal characteristics. This will improve the accuracy and robustness of downstream risk prediction tasks, ultimately providing technical support for the accurate prediction and rapid response to urban environmental risks, public health hazards, and safety incidents.

[0027] Example 1 This invention constructs a complete technical system encompassing "perception-alignment-fusion-prediction-application," with a core focus on the design of a front-end neural network module for heterogeneous data alignment and fusion, and downstream AI large-scale models serving as replaceable predictors. For example... Figure 1The figure shown is a smart city environmental health and safety risk prediction system 100 based on multimodal perception provided in Embodiment 1 of the present invention. The overall architecture includes a heterogeneous data acquisition module 110, a heterogeneous data domain alignment module 120, a joint representation learning module 130, and an AI large model risk prediction module 140 connected in sequence. Specifically: The heterogeneous data acquisition module 110 is used to acquire and standardize multimodal heterogeneous data in urban management, including time-series environmental data, spatial security data, and graph structure health correlation data. S1, Heterogeneous data acquisition module 110, used to acquire heterogeneous data and standardize it.

[0028] (1) Time-series environmental data (such as PM2.5, water quality, noise): collected at a uniform time granularity (minutes / hours), and labeled with metadata such as monitoring point coordinates, equipment model, and sampling frequency to ensure consistency in the time dimension; (2) Spatial security data (such as video surveillance, personnel / vehicle trajectories, fire sensor signals): Unify the spatial coordinate system (WGS84), mark the equipment deployment location, detection range, signal threshold and other metadata, and realize unified calibration of spatial dimensions; (3) Graph-structured health-related data (such as resident health records, infectious disease transmission paths, and distribution of medical institutions): Standardize the definition of nodes (personnel / institution ID, attributes) and edges (contact frequency, service relevance), mark the data time window, and clarify the dynamic characteristics of the graph structure; The Message Queuing Telemetry Transport (MQTT) protocol combined with a blockchain-encrypted transmission scheme is adopted to ensure the integrity and immutability of data during the data acquisition and transmission process, providing high-quality input data for subsequent domain adaptive processing.

[0029] The above operations complete the standardization process for multi-source heterogeneous data. Specifically: (1) Time series environmental data standardization: aims to unify the time base and sampling frequency of different monitoring devices and solve the problem of asynchrony and inconsistency of raw data in the time dimension; (2) Spatial security data standardization: aims to map spatial information from different coordinate systems or acquisition devices to the same geographic reference frame to ensure the accuracy and comparability of spatial locations; (3) Standardization of graph structure health association data: The aim is to formally define entities (nodes) and relationships (edges) and normalize their attributes to build a graph data model with a clear structure and machine readability.

[0030] This standardization process provides high-quality input data with a unified format and clear semantics for subsequent modules, which is an important prerequisite for ensuring the effectiveness of cross-domain alignment and fusion algorithms.

[0031] The heterogeneous data domain alignment module 120 is used to perform cross-domain distribution alignment and modal-level feature calibration on standardized multimodal heterogeneous data. The heterogeneous data domain alignment module 120 adopts a dual adversarial domain alignment neural network architecture, which includes a domain-level adversarial alignment submodule 121 and a modal-level feature calibration submodule 122. S2. Construct heterogeneous data domain alignment module 120.

[0032] Figure 2 This is a schematic diagram of the processing flow of the heterogeneous data domain alignment module 120 and the joint representation learning module 130 provided in an embodiment of the present invention.

[0033] like Figure 2 As shown, a dual-adversarial domain alignment neural network architecture is designed to specifically address the distribution differences of three types of heterogeneous data. It includes two sub-modules: a domain-level adversarial alignment sub-module 121 and a modality-level feature calibration sub-module 122. It supports multiple implementation methods to adapt to different data source scales. S21. Construct a domain-level adversarial alignment submodule 121 (two optional implementation methods) 1. Implementation method 1 (few data source scenarios): When the number of data sources is less than or equal to a preset threshold, for example, applicable to scenarios where the number of data sources is ≤50 (such as regional monitoring networks), an independent discriminator is designed for each data source (domain); (1) Shared feature extraction network 1211: The network structure of modality-specific branch 12111 + cross-domain shared backbone 12112 is adopted.

[0034] The modality-specific branch 12111 includes a temporal data branch, a spatial data branch, and a graph structure data branch: the temporal data branch is configured with a 1D convolutional layer (kernel size 3, stride 1, output channels 64), the spatial data branch is configured with a 2D convolutional layer (kernel size 3×3, stride 1, output channels 64), and the graph structure data branch is configured with a 1-layer graph convolutional network (output dimension 64).

[0035] The cross-domain sharing backbone 1212 consists of a multi-layer (preferably 2-layer) Transformer encoder (each layer has 8 attention heads and 128 hidden layer dimensions) to extract cross-domain common knowledge; (2) Multi-source domain discriminator 1212: at least one, each discriminator contains 2 fully connected layers (hidden layer dimension is 64, output dimension is 1) + LeakyReLU activation function, which satisfies 1-Lipschitz continuity through spectral normalization constraint to avoid the computational overhead caused by gradient penalty; (3) Adversarial optimization mechanism: The discriminator maximizes the Wasserstein distance between the source and target domains, while the feature extraction network minimizes this distance, achieving cross-domain distribution alignment through alternating training. Through adversarial training, the features extracted by the shared feature extraction network minimize the distribution differences between different data sources.

[0036] 2. Implementation Method Two (Multi-Data Source Scenario): When the number of data sources exceeds a preset threshold, for example, applicable to scenarios where the number of data sources is >50 (such as massive sensor networks), a condition discriminator is used to reduce computational complexity, and a single condition discriminator is configured. The input to the conditional discriminator is a concatenated vector of features and data source identifiers. A single discriminator distinguishes the distribution differences between different source and target domains, while the rest of the network structure remains consistent with Implementation Method 1.

[0037] S22, Construct the modal-level feature calibration submodule 122 To balance modal specificity and cross-domain commonality, the modal-level feature calibration submodule 122 adopts a gradual... The regularization penalty term constructs a weak parameter sharing mechanism, including: calculating the relationship between feature extraction parameters from different data sources under the same modality and the shared parameters of that modality. Norm deviation; regularization constraints are applied to the deviation based on a preset penalty coefficient to balance modal specificity and commonality; the calibrated modal features are then... Normalization is applied to eliminate scale differences between different modes. The specific formula is as follows: ;in, The parameter weak sharing loss function is used to balance modality specificity and cross-domain commonality, and to constrain the deviation between feature extraction parameters and modality shared parameters from different data sources. : A set of feature extraction parameters for all modalities and all data sources, including specific parameters and shared parameters for each modality. Modal index, corresponding to three modalities: time-series environmental data, spatial security data, and graph structure health association data. . Total number of modes, in this patent (i.e., three heterogeneous modes: sequential, spatial, and graph structure). : Data source index under the same modality, used to distinguish different acquisition devices / data sources in the same modality. : No. The number of data sources included in the modality (such as the number of air quality monitoring stations for time-series modalities and the number of monitoring devices for spatial modalities). : Penalty coefficient, with a value range of 0.01-0.1, is used to adjust the constraint strength of parameter deviation, so as to achieve "flexible adaptation to small deviations and gradual tightening of large deviations". : No. Modal Feature extraction parameters from each data source (such as the 1D convolutional layer parameters of the 5th monitoring station in the first type of time-series modality). : No. Shared parameters for a class modality are feature extraction parameters common to all data sources of the same modality (such as Transformer encoder parameters shared by all monitoring stations in a time-series modality). The square of the norm is used for quantization. and The magnitude of parameter deviation is adjusted to ensure a smooth constraint.

[0038] This penalty term applies a gradual constraint on parameter deviations, achieving "flexible adaptation to small deviations and gradual tightening of large deviations," avoiding over-constraint issues; the calibrated modal characteristics are then... Normalization is a standardization process performed on the calibrated modal features, projecting the features onto a spherical feature space to eliminate the interference of scale differences between modalities on the fusion effect.

[0039] The joint representation learning module 130 encodes and weights the temporal, spatial, and graph structure features obtained after alignment and calibration, and outputs a fixed-dimensional fused feature vector. S3. Construct a joint representation learning module 130: heterogeneous feature fusion network.

[0040] The joint representation learning module 130 is specifically a temporal-spatial-graph structure three-stream feature fusion network, including: a three-stream feature encoding submodule 131, which uses a temporal encoding branch 1311, a spatial encoding branch 1312, and a graph structure encoding branch 1313 to perform deep encoding on the temporal, spatial, and graph structure features processed by the heterogeneous data domain alignment module, and outputs feature vectors of equal dimensions; and a heterogeneous feature cross-fusion submodule 132, which is used to dynamically assign fusion weights to the modal feature vectors output by the three-stream feature encoding submodule, and perform weighted fusion to output the fixed-dimensional fused feature vector.

[0041] like Figure 2 As shown, based on the temporal, spatial, and graph structure features obtained after alignment and calibration in step S2, a temporal-spatial-graph structure three-stream fusion network is designed to achieve deep coupling of heterogeneous features: S31, Three-stream Feature Coding Submodule 131 (1) Temporal coding branch 1311: It consists of a bidirectional long short-term memory network (128 hidden layer dimensions, 1 layer) + a multi-head attention mechanism (4 attention heads). The input is 64-dimensional aligned temporal features. The attention mechanism focuses on the abnormal fluctuation period and outputs a 128-dimensional temporal feature vector. ; (2) Spatial coding branch 1312: adopts a convolutional neural network (2 layers of convolution, convolution kernel) The architecture consists of a Transformer encoder (1 layer, 8 attention heads) with a step size of 1 and 128 output channels. The input is 64-dimensional aligned spatial features, which capture the spatial object correlations and output a 128-dimensional spatial feature vector. ; (3) Graph structure encoding branch 1313: A two-layer graph attention network is used (each layer outputs 64 dimensions and has 4 attention heads). The input is 64-dimensional aligned graph features. The attention coefficients are dynamically adjusted based on the node association strength, and the output is a 128-dimensional graph feature vector. .

[0042] S32, Heterogeneous Feature Cross-fusion Submodule 132 The heterogeneous feature cross-fusion submodule 132 is used to perform the following operations: dynamically calculate the initial fusion weights of each modality based on the correlation between the feature vectors of each modality and the downstream prediction task; normalize the initial fusion weights to obtain the final fusion weights of each modality, and introduce a modal contribution balance constraint during the normalization process, which is implemented by penalizing the degree to which the final fusion weights deviate from a preset balance value; use the final fusion weights to weight and concatenate the feature vectors of each modality to generate the fused feature vector. The specific implementation process is as follows: Design a heterogeneous feature cross-fusion submodule 132 with balanced modal contributions to avoid a single modality dominating the fusion process, while also adapting to the high confidence requirements of risk prediction: Calculate the correlation score between each modal feature and the downstream prediction task (quantified by Pearson correlation coefficient), and dynamically allocate the initial fusion weights. ; Introducing modal contribution equalization constraints, the formula is as follows: ;in : No. The final fusion weight of the modality is used to adjust the contribution ratio of the modality feature in heterogeneous fusion. : No. The initial weights of the modalities are dynamically allocated after quantifying the correlation between modal features and downstream risk prediction tasks using the Pearson correlation coefficient. : Modal identifier, with a value of These correspond to the time-series environmental data mode, the spatial security data mode, and the graph structure health association data mode, respectively. : A set of modalities, including three types of heterogeneous modalities: temporal, spatial, and graph-structured. The sum of the initial weights for the three modalities is used to normalize the final fusion weights. Minimum value (takes a value of) This avoids calculation anomalies where the denominator is zero, ensuring formula stability. The modal contribution balancing loss function constrains the final fusion weights of the three modalities to be not significantly different (avoiding the proportion of a single modality exceeding a certain threshold). ). : No. The squared deviation of the final weights of the modal class from the equilibrium benchmark is used to quantify the degree of weight imbalance.

[0043] Fusion feature calculation: Output a 256-dimensional fused feature vector.

[0044] The final feature vector after cross-fusion of heterogeneous features has a dimension of 256 and serves as the input to the downstream AI large model. : The final fusion weight of time-series environmental data modes. : The final fusion weight of spatial security data modalities. The final fusion weights of graph-structured health-related data modalities. : The 128-dimensional temporal feature vector output by the temporal feature branch. : The 128-dimensional spatial feature vector output by the spatial feature branch. : The 128-dimensional graph feature vector output by the graph structure feature branch. The operation combines element-wise multiplication with dimensional concatenation. First, various modal features are weighted, and then the weighted feature vectors are concatenated into a 256-dimensional fusion feature.

[0045] S33, Model Training System (Multi-Objective Optimization) Figure 3 This is a schematic diagram of the multi-objective optimization training process of the system according to an embodiment of the present invention. The system performs end-to-end training through multi-objective optimization, and the total loss function is composed of a weighted sum of downstream prediction loss, domain adversarial loss, weak parameter sharing loss, and modal contribution balancing loss. ;in The total loss function for model training balances various losses through multi-objective optimization to ensure that the model converges to the optimal state. Downstream risk prediction loss: cross-entropy loss is used for classification tasks (adapted to risk type classification), and mean squared error loss is used for regression tasks (adapted to risk level quantification). : The weighting coefficient of the domain-level adversarial loss, with a value ranging from 0.1 to 0.5, is used to adjust the importance of cross-domain distribution alignment. Domain-level adversarial loss, calculated based on Wasserstein distance, is used to minimize the distribution difference between the source and target domains. : The weighting coefficient of the weak parameter sharing loss, with a value range of 0.01-0.1, is used to balance modality specificity and cross-domain commonality. Weak parameter sharing loss, based on Regularization is used to constrain the deviation between modality-specific parameters and shared parameters. : The weighting coefficient for modal contribution balancing loss, with a value range of 0.05-0.2, is used to adjust the degree of balancing of modal fusion weights.

[0046] Training process: The domain discriminator is trained alternately (1 round), the feature extraction network and the fusion module are trained (2 rounds), and the AdamW optimizer is used (learning rate 1e-4). The number of iterations is 200 rounds to ensure model convergence.

[0047] The AI ​​large model risk prediction module 140 is used to classify or regress environmental health and safety risks based on the fused feature vector, and output risk prediction results including risk type and level.

[0048] S4, downstream AI large model adaptation.

[0049] The core innovation of this invention does not involve the large AI model itself; downstream applications can use Transformer-based classification / regression models. Input: The AI ​​large model risk prediction module 140 is decoupled from the joint representation learning module 130 and is used to receive the 256-dimensional fusion features output by the joint representation learning module 130; Model structure: It is recommended to use a 3-layer Transformer encoder + 2 prediction heads (the classification head outputs the risk type, and the regression head outputs the risk level). Two types of prediction results are output: (1) The first type of result is the risk type, which is output through the classification prediction head and is used to identify the specific category of environmental health or safety risk; (2) The second type of result is the risk level, which is output through the regression prediction head and is used to quantify the severity or probability of occurrence of the risk type.

[0050] Training methods: Supports two modes - when the fusion features are fixed, only the prediction head of the large model is fine-tuned; or end-to-end training is adopted to achieve joint optimization of the fusion module and the large model.

[0051] Example 2 (Specific Verification Scenario): (I) Experimental Scenario With the goal of "predicting the comprehensive public safety risk level of an industrial park in a certain city in the next 24 hours", the effectiveness of the technical solution of this invention is verified by using three types of heterogeneous data: risk-related environmental pollution (time series), personnel gathering (space), and occupational health (graph structure).

[0052] (II) Experimental Data Time-series environmental data: Hourly monitoring data of PM2.5 and SO2 from 10 air quality monitoring stations, with a collection period of 30 days and 720 records per station; Spatial security data: 15-minute-level personnel density statistics from 50 key area monitoring stations (collection period of 30 days), and real-time signal data from 10 fire sensors; Graph-structured health data: Health status node data of 500 workers in 5 communities, and service association edge data between community hospitals and workers (contact frequency ≥ 1 time / week).

[0053] (III) Comparative Experimental Design Baseline method: Baseline 1: Directly concatenate the features of the three types of data and input them into the multilayer perceptron model for prediction; Baseline 2: Prediction is made using only time-series environmental data and a Transformer model; Baseline 3: Prediction is performed using a traditional domain adaptive method (deep neural network) combined with simple feature fusion; The method of this invention consists of a heterogeneous data domain alignment module, a three-stream fusion module, and a Transformer prediction head.

[0054] (iv) Experimental setup All experiments were conducted on the same hardware platform (e.g., NVIDIA V100 GPU). The model training in this invention used the AdamW optimizer with an initial learning rate of 1e-4, a batch size of 32, and 200 iterations. The loss function weight coefficients were set as follows: =0.3, =0.05, =0.1. Parameter weak sharing penalty coefficient. =0.05. In the comparison baseline, baseline 3's traditional domain adaptation method uses classic structures such as DANN, while the remaining hyperparameters are kept as consistent as possible with the model of this invention to ensure a fair comparison. The experimental results are shown in Table 1 below: Table 1 Experimental Results

[0055] Experimental results show that the method of the present invention, through effective alignment and fusion of heterogeneous data, significantly outperforms traditional methods in prediction accuracy and generalization ability, while maintaining reasonable training efficiency, thus fully verifying the advanced nature and practicality of the technical solution.

[0056] Figure 4A schematic diagram of the intelligent management platform's interface is shown in the embodiment. This invention integrates digital twin technology with a heterogeneous data visualization engine to construct a comprehensive, visualized management system encompassing physical scenes, feature representations, and risk levels. The platform's core features include a heterogeneous data representation visualization layer, a multimodal risk monitoring dashboard, an intelligent decision-making and scheduling module, and a closed-loop response feedback module, enabling intelligent management throughout the entire process. The platform supports real-time monitoring of domain alignment effects, quantifies key model indicators such as modal contribution weights and feature fusion quality, and supports dynamic adaptive adjustment of model parameters. Based on risk prediction results from joint representation learning, the platform automatically completes graded alerts for risk events, intelligent dispatching of responsible units, emergency resource scheduling, progress tracking, and effectiveness evaluation, forming a comprehensive management mechanism from risk warning to intelligent response to closed-loop response and feedback. The digital twin platform deeply integrates urban geospatial information, sensor terminal layout, pipeline infrastructure, and transportation networks, enabling spatiotemporal mapping and dynamic updates of heterogeneous data in the environmental, health, and safety categories. All risk events, monitoring data, and response instructions can be accurately located and visualized on the digital twin map, providing intuitive, efficient, and precise decision support for comprehensive urban environmental health and safety management.

[0057] Example 3 Figure 5 This is a flowchart illustrating the smart city environmental health and safety risk prediction method based on multimodal perception provided in Embodiment 3 of the present invention. Embodiment 3 of the present invention provides a smart city environmental health and safety risk prediction method 200 based on multimodal perception, applied to the system 100 described in Embodiment 1. Method 200 includes: S210, Data Acquisition and Standardization Steps: Acquire time-series environmental data, spatial security data, and graph structure health correlation data in urban management, and standardize the multimodal heterogeneous data; S220, Cross-domain Alignment and Feature Calibration Steps: Input the standardized data into the heterogeneous data domain alignment module; reduce the distribution differences between different data sources and different modalities through domain-level adversarial alignment and modality-level feature calibration, and calibrate the features; S230, Joint Representation Learning Steps: Input the aligned and calibrated temporal, spatial, and graph structure features into the joint representation learning module; encode and dynamically weight fusion based on modal contribution balance constraints through a temporal-spatial-graph structure three-stream feature fusion network to generate a fixed-dimensional fusion feature vector; S240, Risk Prediction Step: Input the fused feature vector into the AI ​​large model risk prediction module, and output the prediction results of the type and level of environmental health and safety risks.

[0058] The smart city environmental health and safety risk prediction method based on multimodal perception provided in this embodiment of the invention can execute the operation steps of each module in the smart city environmental health and safety risk prediction system based on multimodal perception provided in any of the above embodiments of the invention, and has the corresponding functions and beneficial effects of the smart city environmental health and safety risk prediction system based on multimodal perception. For detailed process, please refer to the relevant operations of the smart city environmental health and safety risk prediction system based on multimodal perception in the foregoing embodiments.

[0059] Example 4 Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, and may also represent various forms of mobile devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the invention described and / or claimed herein.

[0060] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0061] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0062] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, central processing unit (CPU), graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the deep learning-based slow-wave EEG real-time feedback transcranial electrical stimulation method described above.

[0063] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0064] The above embodiments are merely illustrative examples and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A smart city environmental health and safety risk prediction system based on multimodal perception, characterized in that, It includes a heterogeneous data acquisition module, a heterogeneous data domain alignment module, a joint representation learning module, and an AI large model risk prediction module, which are connected in sequence. The heterogeneous data acquisition module is used to acquire and standardize multimodal heterogeneous data in urban management, including time-series environmental data, spatial security data, and graph structure health correlation data. The heterogeneous data domain alignment module is used to perform cross-domain distribution alignment and modal-level feature calibration on standardized multimodal heterogeneous data. The heterogeneous data domain alignment module adopts a dual adversarial domain alignment neural network architecture, which includes a domain-level adversarial alignment submodule and a modal-level feature calibration submodule. The joint representation learning module encodes and weights the temporal, spatial, and graph structure features obtained after alignment and calibration, and outputs a fixed-dimensional fused feature vector. The AI ​​large model risk prediction module is used to classify or regress environmental health and safety risks based on the fused feature vectors, and output risk prediction results including risk type and level.

2. The system according to claim 1, characterized in that, The domain-level adversarial alignment submodule includes: The shared feature extraction network adopts a network structure that includes modality-specific branches and a cross-domain shared backbone. The modality-specific branches include 1D convolutional layers for processing temporal environmental data, 2D convolutional layers for processing spatial security data, and graph convolutional layers for processing graph-structured health association data. The cross-domain shared backbone is composed of multi-layer Transformer encoders. At least one domain discriminator is used to minimize the distribution differences between different data sources in the features extracted by the shared feature extraction network through adversarial training.

3. The system according to claim 2, characterized in that, The domain discriminator is adapted to two implementation methods depending on the size of the data source: Implementation method 1: When the number of data sources is less than or equal to a preset threshold, configure an independent discriminator for each data source; Implementation Method 2: When the number of data sources exceeds a preset threshold, configure a single condition discriminator whose input is a concatenated vector of features and data source identifiers.

4. The system according to claim 1, characterized in that, The modal-level feature calibration submodule is implemented through a weak parameter sharing mechanism and is configured as follows: Calculate the L2 norm deviation between feature extraction parameters from different data sources and shared parameters of the same modality; According to a preset penalty coefficient, a regularization constraint is applied to the deviation to balance modal specificity and commonality; The calibrated modal features were subjected to L2 normalization to eliminate scale differences between different modes.

5. The system according to claim 1, characterized in that, The joint representation learning module is specifically a temporal-spatial-graph structured three-stream feature fusion network, including: The three-stream feature coding submodule uses temporal coding branch, spatial coding branch and graph structure coding branch respectively to perform deep coding on the temporal, spatial and graph structure features after being processed by the heterogeneous data domain alignment module, and outputs feature vectors of equal dimension. The heterogeneous feature cross-fusion submodule is used to dynamically assign fusion weights to the modal feature vectors output by the three-stream feature encoding submodule, perform weighted fusion, and output the fixed-dimensional fused feature vector.

6. The system according to claim 5, characterized in that: The temporal coding branch includes a bidirectional long short-term memory network and a multi-head attention mechanism; The spatial coding branch includes a convolutional neural network and a Transformer encoder; The graph structure encoding branch contains a multi-layer graph attention network.

7. The system according to claim 5 or 6, characterized in that, The heterogeneous feature cross-fusion submodule is configured to perform the following operations: Based on the correlation between the feature vectors of each modality and the downstream prediction task, the initial fusion weights of each modality are dynamically calculated. The initial fusion weights are normalized to obtain the final fusion weights of each mode. A mode contribution balance constraint is introduced during the normalization process. This constraint is achieved by penalizing the degree to which the final fusion weights deviate from the preset balance value. The final fusion weights are used to weight and concatenate the feature vectors of each modality to generate the fused feature vector.

8. The system according to claim 1, characterized in that, The system performs end-to-end training through multi-objective optimization, and the total loss function is composed of a weighted sum of downstream prediction loss, domain adversarial loss, weak parameter sharing loss, and modal contribution balancing loss.

9. The system according to claim 1, characterized in that, The AI ​​large-scale model risk prediction module is decoupled from the joint representation learning module, and is used to receive the fused feature vector and simultaneously output two types of prediction results: The first type of result is the risk type, which is output through a classification prediction head and used to identify the specific category of environmental health or safety risk; The second type of result is the risk level, which is output through a regression prediction head and used to quantify the severity or probability of occurrence of the risk type.

10. A method for predicting environmental health and safety risks in smart cities based on multimodal perception, applied to the system described in any one of claims 1 to 9, characterized in that, The method includes: Data acquisition and standardization steps: Acquire time-series environmental data, spatial security data, and graph structure health correlation data in urban management, and perform standardization processing on the multimodal heterogeneous data; Cross-domain alignment and feature calibration steps: Input the standardized data into the heterogeneous data domain alignment module; reduce the distribution differences between different data sources and different modalities through domain-level adversarial alignment and modality-level feature calibration, and calibrate the features; Joint representation learning steps: Aligned and calibrated temporal, spatial, and graph structure features are input into the joint representation learning module; encoding and dynamic weighted fusion based on modal contribution balance constraints are performed through a temporal-spatial-graph structure three-stream feature fusion network to generate a fixed-dimensional fusion feature vector; Risk prediction steps: Input the fused feature vector into the AI ​​large model risk prediction module, and output the prediction results of the type and level of environmental health and safety risks.

Images