Project risk dynamic prediction method and system based on multi-modal fusion deep learning

Abstract

The application relates to the technical field of artificial intelligence and project management, and discloses a project risk dynamic prediction method and system based on multi-modal fusion deep learning, wherein the project risk dynamic prediction method based on multi-modal fusion deep learning comprises the following steps: collecting multi-modal data of a target project, performing time alignment and quality inspection; performing multi-modal feature extraction and feature processing; performing multi-modal feature fusion; analyzing time sequence evolution rules and long-short term dependence relations; performing risk prediction and trend analysis; and performing risk attribution analysis and relief measure matching on multi-category risk prediction results; the application simultaneously captures short-term fluctuations and long-term trends through a multi-scale time window attention mechanism, thereby providing forward-looking decision support for project managers.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and project management technology, and more specifically, to a method and system for dynamic prediction of project risks based on multimodal fusion deep learning. Background Technology

[0002] With the continuous expansion and increasing complexity of modern projects, project risk management has become a key factor determining project success or failure. In fields such as engineering construction, software development, and manufacturing, project execution involves multiple dimensions including schedule, cost, quality, resources, personnel, and environment, generating a large amount of heterogeneous data. This includes structured numerical data in project management systems, unstructured text reports in document systems, image and video data from on-site monitoring, and real-time monitoring data collected by IoT sensors. However, traditional project risk management methods mainly rely on the project manager's experience and judgment, periodic manual reviews, and single-dimensional data analysis, which have technical limitations.

[0003] The main problems with existing technologies include: severe information silos, with different types of project data scattered and stored in their own independent systems, lacking a unified data integration and correlation analysis mechanism, resulting in the inability to form a comprehensive project status view during risk assessment; insufficient predictive timeliness, as traditional methods often adopt a periodic review model, which often only discovers risks after they have already manifested or caused impact, lacking early warning capabilities and missing the best opportunity to respond to risks; and the one-sidedness of analytical methods, as existing methods usually only rely on one type of data, such as financial data or schedule data, to make risk judgments, ignoring other important risk signals, especially early risk signs hidden in text reports, on-site images, and sensor data.

[0004] A more critical technical challenge lies in the fact that existing multimodal data processing technologies struggle to effectively address the specific needs of project management scenarios. Different modalities of data vary in format, collection frequency, and quality standards, making effective time alignment and semantic fusion a significant technical hurdle. Furthermore, project risks exhibit dynamic evolution, requiring the modeling of long-term dependencies and multi-scale temporal patterns, which traditional statistical methods and simple machine learning models struggle to capture. Moreover, different types of risks in projects often interconnect and influence each other, creating risk linkage effects and cascading propagation; existing technologies lack effective models for this risk propagation mechanism. Summary of the Invention

[0005] This invention provides a method and system for dynamic prediction of project risks based on multimodal fusion deep learning, which solves the technical problems of information silos, prediction lag, and insufficient accuracy in related technologies.

[0006] This invention provides a method for dynamic prediction of project risk based on multimodal fusion deep learning, comprising the following steps: Collect multimodal data of the target project, perform time alignment and quality checks, and obtain a time-synchronized multimodal dataset; Based on a time-synchronized multimodal dataset, multimodal feature extraction and feature processing are performed to obtain a set of multimodal feature vectors; Based on a set of multimodal feature vectors, an adaptive weight allocation strategy is used to fuse multimodal features to obtain a fused feature vector sequence. Based on the fused feature vector sequence, the temporal evolution pattern and long-short-term dependency relationship are analyzed to obtain the temporal feature representation sequence; Based on the temporal feature representation sequence, risk prediction and trend analysis are performed to obtain multi-category risk prediction results; Risk attribution analysis and mitigation measure matching are performed on the prediction results of multiple categories of risks to obtain a comprehensive risk analysis report.

[0007] In a preferred embodiment, the step of obtaining the time-synchronized multimodal dataset includes: Collect structured data and obtain task schedules, cost management data, resource management data, and quality management data in real time based on the standard interface of the project management information system; Collect text data, based on the document management system and email system, to obtain technical reports, security inspection logs, quality inspection reports, meeting minutes, and change request documents; Collect image and video data, and use keyframe extraction technology to obtain visual information of the construction site based on the on-site monitoring system; Collect sensor data and acquire real-time monitoring data from temperature and humidity sensors, geological subsidence monitoring instruments, and harmful gas detectors based on the Internet of Things sensor network; Based on the timestamp information of each modality data, a unified time base calibration and time window aggregation method is used to perform time alignment of multimodal data; Based on the time-aligned multimodal dataset, a data integrity check mechanism is used to identify and label missing modalities and missing data fields.

[0008] In a preferred embodiment, the step of obtaining the multimodal feature vector set includes: Time series feature engineering and deep time series networks are used to construct time series features and cross-index correlation features for structured data, and structured modal feature vectors are obtained by processing them through temporal convolutional networks. Natural language processing techniques and pre-trained language models are used to perform sentence segmentation, word segmentation, named entity recognition, language model encoding, sentiment analysis, and risk keyword extraction on text data to obtain text modal feature vectors. Computer vision technology and convolutional neural networks are used to perform image preprocessing, feature extraction, target detection, scene classification, and anomaly detection on image data to obtain image modal feature vectors; Time series analysis and recurrent neural networks are used to extract statistical features and time-series pattern features from sensor data, and sensor modal feature vectors are obtained by processing them through a long short-term memory network.

[0009] In a preferred embodiment, the step of obtaining the fused feature vector sequence includes: A self-attention mechanism is used to process the internal feature correlation of each modality feature vector; A cross-modal attention mechanism is used to calculate the correlation and interaction relationships between different modalities, resulting in a cross-modal attention representation; A gating mechanism is used to control the weights of cross-modal attention representations, adaptively controlling the fusion ratio of information from different modalities; The gated modal features are concatenated and processed by a deep fusion network to obtain a unified multimodal fusion representation; Orthogonal constraint loss function is used to reduce redundant information between modes.

[0010] In a preferred embodiment, the step of obtaining the temporal feature representation sequence includes: Learnable temporal location coding is used to embed location information for each time step in the sequence; A multi-head self-attention mechanism is employed to capture the dependencies between different time steps in the sequence; A multi-scale time window attention mechanism is adopted to divide the time series into time windows of different lengths to capture patterns at different time scales; A feedforward neural network is used for feature transformation, and information flow is maintained through residual connections. A deep temporal coding network is constructed by stacking multiple layers, and temporal features are extracted layer by layer.

[0011] In a preferred embodiment, the step of obtaining multi-category risk prediction results includes: A multi-branch network structure is adopted, with independent prediction branches designed for schedule risk, cost risk, quality risk, safety risk, and external risk; The risk level is divided into four levels: low risk, low-medium risk, medium-high risk, and high risk, using a softmax classifier, and the probability distribution of the risk level is obtained. A linear regression layer was used to obtain continuous numerical values ​​for the risk score; A sequence-to-sequence prediction model is used to predict risk trends over multiple future time steps; A joint loss function is used to collaboratively optimize multiple prediction tasks; We employ risk association networks to model the linkage effects between different risks.

[0012] In a preferred embodiment, the step of obtaining the comprehensive risk analysis report includes: Based on cross-modal attention weights, we use weight statistics and visualization methods to analyze the contribution of each modality to risk prediction. Based on the self-attention weights of the temporal encoder, we use weight aggregation and key moment identification methods to analyze the impact of historical time steps on current risks. Based on text modal feature representation and attention weight, keyword extraction and semantic analysis methods are used to identify key information related to risk; Based on image modal feature representation and activation mapping of convolutional networks, a class activation mapping method is used to generate image heatmaps and locate risk-related visual regions. Based on the results of risk prediction and risk attribution analysis, rule-based reasoning and case retrieval methods are used to generate risk mitigation recommendations. We use web technologies and data visualization technologies to build a user-friendly interactive interface.

[0013] In a preferred embodiment, the time window aggregation method includes: Divide the project execution cycle into fixed-length time windows; The high-frequency sensor data and image data are aggregated within each time window. Statistical features are calculated for the sensor data, and the image with the best quality is selected as the representative image for the image data. Assign the daily and weekly reports collected in low frequency to the corresponding time windows; Generate a data quality report for each time window, identifying available modalities, missing modalities, and data quality level.

[0014] In a preferred embodiment, the multi-scale temporal window attention mechanism includes: The time series is divided into time windows of different lengths, including short-term, medium-term, and long-term scale windows; The sequence is segmented using a sliding window approach for each scale, and local attention is calculated within each window. Representations at different scales are fused through a gating mechanism, and the weights of information at different scales are adaptively selected based on the current input.

[0015] In a preferred embodiment, the project risk dynamic prediction system based on multimodal fusion deep learning is used to execute the above-described project risk dynamic prediction method based on multimodal fusion deep learning, including: The data acquisition module collects multimodal data of the target project, performs time alignment and quality checks, and obtains a time-synchronized multimodal dataset. The feature extraction module performs multimodal feature extraction and processing based on a time-synchronized multimodal dataset to obtain a set of multimodal feature vectors. The feature fusion module, based on a set of multimodal feature vectors, uses an adaptive weight allocation strategy to fuse multimodal features and obtain a fused feature vector sequence. The time series modeling module analyzes the time series evolution patterns and long-short-term dependencies based on the fused feature vector sequence, and obtains the time series feature representation sequence. The risk prediction module performs risk prediction and trend analysis based on time-series feature representation sequences, and obtains multi-category risk prediction results; The interpretability analysis module performs risk attribution analysis and mitigation measure matching on the prediction results of multiple categories of risks, and obtains a comprehensive risk analysis report.

[0016] The beneficial effects of this invention are as follows: This invention effectively addresses the key issues in the aforementioned background technologies through an innovative multimodal fusion deep learning architecture. Employing a cross-modal attention mechanism and adaptive weight allocation strategy, it achieves deep fusion of structured data, text data, image data, and sensor data, overcoming the information silo limitations of traditional methods and improving the comprehensiveness and accuracy of risk identification. By capturing the evolution patterns and long-term / short-term dependencies of project risks through a time-series deep learning model, it can not only assess the current risk status but also predict future risk trends, realizing a shift from passive response to proactive prevention. The multi-scale time window attention mechanism can simultaneously capture short-term fluctuations and long-term trends, and the risk linkage effect modeling can predict the cascading propagation paths of risks, providing project managers with forward-looking decision support.

[0017] This invention transforms the black-box model of deep learning into an understandable decision support tool through interpretability analysis. By employing methods such as modality contribution assessment, temporal impact analysis, and risk attribution explanation, it clearly identifies the sources of risk and key driving factors, enhancing user trust in the system's output. Mechanisms such as missing modality handling, adversarial training, and incremental learning improve the system's robustness and adaptability, enabling stable operation in real-world projects with incomplete data and variable environments. The risk mitigation recommendation function, based on a historical case library and expert knowledge, provides targeted response suggestions for different types and levels of risks, forming a complete closed loop from risk identification to risk response, thus improving the intelligence and practical value of project risk management. Attached Figure Description

[0018] Figure 1 This is the main flowchart of the project risk dynamic prediction method based on multimodal fusion deep learning in this invention; Figure 2 This is a detailed flowchart of the project risk dynamic prediction method based on multimodal fusion deep learning in this invention; Figure 3 This is a module diagram of the project risk dynamic prediction system based on multimodal fusion deep learning in this invention. Detailed Implementation

[0019] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0020] At least one embodiment of the present invention discloses a method for dynamic prediction of project risk based on multimodal fusion deep learning, such as... Figures 1 to 2 As shown, it includes the following steps: Step 1: Collect multimodal data of the target project, perform time alignment and quality checks, and obtain a time-synchronized multimodal dataset; Step 1.1, Structured data acquisition; Based on the standard interface of the project management information system for the target project, project execution data is acquired in real time using RESTful API or direct database connection. The data types collected include: planned start time, planned end time, actual start time, actual end time, completion percentage, and delay days for each task in the task schedule table; budget amount, actual expenditure amount, cumulative expenditure percentage, cost deviation amount, and cost deviation percentage for each cost item in the cost management system; personnel configuration quantity, actual on-duty quantity, equipment quantity, equipment availability, material inventory, and material consumption rate in the resource management system; and the number of quality inspections, number of qualified items, number of unqualified items, number of reworks, and quality score in the quality management system. Incremental data collection is performed hourly on the data acquisition interface, extracting newly generated data records to the local cache to obtain structured time series data.

[0021] Step 1.2, Text Data Acquisition; Based on a document management system and an email system, document crawling and email parsing technologies were used to acquire unstructured text information related to the project. The collected text types included weekly technical reports describing the technical work completed that week, the technical challenges encountered, and the solutions implemented; daily safety inspection logs recording inspection personnel, inspection time, inspection area, discovered safety hazards, and handling measures; weekly quality inspection reports including inspection locations, inspection standards, inspection results, descriptions of non-conformities, and rectification requirements; meeting minutes recording meeting participants, meeting topics, discussion content, resolutions reached, responsible persons, and deadlines; and change request documents describing the change content, reasons for the change, impact analysis, and estimated change costs. The text data acquisition module performed format recognition and content extraction on the documents, converting documents in formats such as PDF, Word, and Excel into plain text to obtain a text data set.

[0022] Step 1.3, Image and video data acquisition; Based on the on-site monitoring system, video stream access and keyframe extraction technologies are used to acquire visual information about the construction site. High-definition network cameras deployed on-site cover the main construction areas, material storage areas, equipment storage areas, personnel passageways, and hazardous work areas. The video stream is continuously transmitted to the video server at a rate of 25 frames per second. A keyframe extraction algorithm extracts one frame at fixed time intervals for storage and analysis. The keyframe extraction strategy is based on scene change detection; keyframe extraction is triggered when the difference between consecutive frames exceeds a set threshold, ensuring that certain changes in the construction status are captured. The extracted images contain construction progress information such as the completion status of structures and equipment installation progress; construction quality information such as the surface quality of concrete pouring, the appearance quality of welds, and the standardization of equipment installation; and safety status information such as whether personnel are wearing safety helmets, the use of safety belts, and the integrity of protective facilities in hazardous areas. The keyframe images and their metadata are processed by the image acquisition module, saving the keyframe images and their metadata, including camera number, shooting time, and shooting location, to an image database, resulting in an image data sequence.

[0023] Step 1.4, Sensor data acquisition; Based on an IoT sensor network, real-time monitoring data on the site environment and equipment status is acquired using either Message Queuing Telemetry Transport (MQTT) or Constrained Application Protocol (CoAP). Sensor types include temperature and humidity sensors to monitor the temperature and humidity of the construction environment, ensuring that construction conditions meet specifications; geological settlement monitors to monitor the settlement and displacement of surface and underground structures, providing early warning of geological disaster risks; hazardous gas detectors to monitor oxygen, carbon monoxide, and methane concentrations in underground work spaces, ensuring worker safety; vibration sensors to monitor vibrations generated by blasting operations and the operation of large equipment, assessing the impact on surrounding buildings; and noise sensors to monitor construction noise levels, ensuring compliance with environmental protection requirements. Sensor data acquisition frequency is set according to the characteristics of the monitored objects: environmental parameter sensors acquire data once per minute, while safety-critical sensors acquire data once per second. The acquired raw sensor data includes sensor identification, measurement time, measured value, measurement unit, and data quality flags, resulting in a sensor data stream.

[0024] Step 1.5, Time Synchronization and Alignment; Based on the timestamp information of each modality's data, a unified time base calibration and time window aggregation method is adopted to achieve time alignment of multimodal data. Specifically, the timestamps of all data sources are calibrated, and the Network Time Protocol (NTP) is used to ensure that the clocks of each system are synchronized with the standard time server, eliminating time deviations between different systems. According to the time granularity requirements of project management, the project execution cycle is divided into fixed-length time windows; in this embodiment, the time window length is set to 1 day. For high-frequency acquired data, such as sensor data and image data, aggregation processing is performed within each time window. Statistical characteristics within the time window are calculated for sensor data, including mean, standard deviation, maximum value, minimum value, and trend. For image data, the image with the best quality within the time window is selected as the representative image, or the features of multiple images are fused. For low-frequency acquired data, such as daily and weekly reports, they are assigned to the corresponding time windows. By performing time synchronization processing on the timestamp information of each modality's data, a time-aligned multimodal dataset is obtained, where each time window contains data from all modalities within that time period.

[0025] Step 1.6, Data integrity check and missing data marker; Based on the time-aligned multimodal dataset, a data integrity check mechanism is employed to identify and label missing modalities and missing data fields. For each time window, the existence of each modality's data is checked. If a modality's data is completely missing within that time window, it is marked as a missing modality, and the reason for the missing data is recorded, such as equipment failure, network interruption, or system maintenance. For existing modal data, the completeness of key fields is checked. If certain key fields in structured data are missing, such as cost or schedule data; if certain required documents in text data, such as security logs, are not submitted; if cameras in certain key areas of image data fail to capture valid images; or if certain sensors in sensor data fail to report data for an extended period, the data is marked as incomplete. By performing integrity checks on the time-aligned multimodal dataset, a data quality report is generated for each time window, identifying available modalities, missing modalities, and data quality levels, resulting in a time-synchronized multimodal dataset.

[0026] This step, following the sub-steps described above, yields a time-synchronized multimodal dataset. This dataset is organized by time windows, with each window corresponding to one day of project execution. It includes structured schedule, cost, resource, and quality data for that day, text reports and records, on-site images, sensor monitoring data, and quality markers for each modality. This time-synchronized multimodal dataset serves as input for subsequent feature extraction.

[0027] Step 2: Based on the time-synchronized multimodal dataset, perform multimodal feature extraction and feature processing to obtain a set of multimodal feature vectors; Step 2.1, Structured Time Series Feature Extraction; Based on the structured schedule, cost, resource, and quality data in the time-synchronized multimodal dataset output in step 1, time series feature engineering and deep time series networks are used to process the task schedule, cost management data, resource management data, and quality management data within each time window to extract quantitative features reflecting the project execution status.

[0028] The raw numerical data is preprocessed, and the Z-score standardization method is used to transform data of different dimensions to a unified scale. The calculation formula is that the standardized value equals the original value minus the mean and then divided by the standard deviation. Time series features are constructed, and sliding window statistical features are calculated for each numerical indicator. Within a sliding window of 7 days, the mean reflects the recent average level, the standard deviation reflects the degree of fluctuation, the trend is represented by the slope of the linear fit, and the rate of change is calculated by the difference between the current value and the previous value. Cross-indicator correlation features are constructed, and the correlation between cost deviation rate and schedule deviation rate and the correlation between resource input intensity and schedule completion rate are calculated. These correlation features can reflect the mutual influence between different dimensions in project execution.

[0029] The constructed feature vectors are input into a temporal convolutional network, which employs a multi-layered, stacked one-dimensional convolutional structure. Expanding the convolutions achieves a large receptive field to capture long-term dependencies. The network input is a sequence of feature vectors from several past time windows, which are progressively abstracted through convolutional layers to obtain higher-level feature representations. By processing the constructed feature vectors through the temporal convolutional network, the output is a fixed-dimensional feature vector containing temporal patterns, trend information, and anomalous signals from the structured data, resulting in a structured modality feature vector.

[0030] Step 2.2, Extraction of semantic features from text; Based on the text reports and records in the time-synchronized multimodal dataset output in step 1, natural language processing technology and pre-trained language models are used to process the technical reports, safety inspection logs, quality inspection reports, meeting minutes, and change application documents within each time window, and extract the textual semantic features that reflect project risk information.

[0031] Specifically, the text is preprocessed by using sentence and word segmentation tools to divide the document into sentences and words, removing stop words, punctuation marks, and meaningless characters, while retaining content words and key terms. Part-of-speech tagging and named entity recognition are performed to identify entities such as names of people, places, organizations, time expressions, and quantity expressions in the text, with particular attention paid to risk-related entities such as delays, cost overruns, quality issues, safety hazards, changes, and disputes. A pre-trained Transformer language model is used to encode the text. This pre-trained model has been trained on a large-scale general corpus and has learned rich linguistic knowledge and semantic representation capabilities. The text is input into the pre-trained model, and through a multi-layered self-attention mechanism and a feedforward network, the context-related representation of each word in the text is obtained. For document-level representation, an attention pooling method is used to weight and aggregate the representations of all words. The attention weights are calculated using learnable parameters, enabling the model to focus on the parts of the document most important for risk prediction. Furthermore, the sentiment polarity features of the text are extracted, and a sentiment analysis model is used to determine the sentiment tendency of the text; negative sentiment is often associated with risk. The frequency and co-occurrence patterns of risk keywords are extracted to construct a risk dictionary containing terms such as delay, overdue, lag, overspending, budget overrun, quality problems, non-conformity, rework, safety hazards, accidents, changes, disputes, and claims. The frequency and positional distribution of these keywords in the text are statistically analyzed. Through preprocessing, entity recognition, language model encoding, sentiment analysis, and keyword extraction, the text modal feature vector is obtained, which includes the document's semantic representation, sentiment features, and risk keyword features.

[0032] Step 2.3, Image visual feature extraction; Based on the on-site image data from the time-synchronized multimodal dataset output in Step 1, computer vision techniques and convolutional neural networks are used to process the keyframe images and their metadata within each time window, extracting visual features reflecting construction status and risk signals. Specifically, image preprocessing is performed, employing image enhancement techniques to improve image quality, including brightness adjustment, contrast enhancement, and noise reduction. The image size is uniformly adjusted to the network input size using bilinear interpolation for scaling. A pre-trained convolutional neural network is used to extract image features. This pre-trained network has been trained on large-scale image datasets such as ImageNet and has learned general visual feature representation capabilities. The image is input into the convolutional network, and through multiple convolutional layers, pooling layers, and nonlinear activation functions, features are extracted layer by layer, from low-level edge texture features to high-level semantic features. Feature maps from the middle layers of the network are extracted as image representations. These middle layer features contain structural and semantic information of the image and have better generality than the classification features of the last layer. In addition, specific target detection and recognition are performed, using an object detection network to identify key objects in the images, such as personnel, equipment, materials, and safety protection facilities. The system analyzes the attributes of detected objects to determine whether personnel are wearing safety helmets and safety belts, whether equipment is placed correctly, whether material stacking meets requirements, and whether safety protection facilities are complete. Scene classification is performed, categorizing construction site images into different work scenarios such as earthwork excavation, concrete pouring, steel structure installation, and equipment debugging. Different scenarios correspond to different risk types and concerns. Anomaly detection identifies abnormal conditions in the images, such as chaotic construction sites, damaged equipment, personnel gatherings, open flames, and smoke. Through image preprocessing, feature extraction, object detection, scene classification, and anomaly detection, the system obtains an image modal feature vector, which includes the image's global visual features, object detection results, scene classification results, and anomaly detection markers.

[0033] Step 2.4, Sensor temporal pattern feature extraction; Based on the sensor monitoring data in the time-synchronized multimodal dataset output in step 1, time series analysis and recurrent neural networks are used to process the sensor data stream within each time window, extracting time-series features reflecting environmental and equipment conditions. Specifically, the raw sensor data is preprocessed by using sliding window mid-range filtering to remove sudden noise and abnormal spikes, and data normalization to convert the measured values ​​of different sensors to the same numerical range. Sensor fault data is detected and processed; if sensor data remains unchanged for a long time or exhibits abnormal values ​​exceeding the physical range, it is determined to be a sensor fault, and the data in the fault interval is interpolated or marked as missing. Statistical features of the sensor data are extracted: within each time window, the mean of the sensor measurements reflects the average level of that period, the standard deviation reflects the degree of fluctuation, the peak and trough values ​​reflect extreme cases, and the skewness and kurtosis reflect the distribution pattern. Time-series pattern features are extracted by detecting periodic patterns in the data through autocorrelation analysis and identifying the main frequency components through Fourier transform analysis. Trend changes are detected by using the moving average method to extract the long-term trend of the data, and the trend slope is used to determine the upward or downward trend of the measured values. Abrupt changes are detected by employing a change point detection algorithm to identify time points where the statistical characteristics of the data change. These abrupt changes often correspond to abnormal changes in the environment or equipment status. Sensor time-series data is input into a recurrent neural network (RNN) using a Long Short-Term Memory (LSTM) structure, capable of learning long-term dependencies in the time series. The LSTM network includes input gates, forget gates, output gates, and memory units. Through a gating mechanism, it selectively retains or forgets historical information, enabling the network to capture important patterns in the time series. After preprocessing the sensor time-series data, extracting statistical features, performing temporal pattern analysis, and processing with the LSTM network, the sensor data sequence from several past time steps is input into the LSTM network. The network output is a fixed-dimensional hidden state vector containing the temporal patterns and state characteristics of the sensor data, yielding the sensor modal feature vector.

[0034] Step 2.5, Feature dimension alignment and standardization; Based on the structured modality feature vectors output in step 2.1, the text modality feature vectors output in step 2.2, the image modality feature vectors output in step 2.3, and the sensor modality feature vectors output in step 2.4, dimensionality transformation and normalization methods are used to unify the different dimensions of each modality feature vector, achieving a unified feature space. The feature vectors output by different modality extraction networks may have different dimensions: structured features have a dimension of D1, text features have a dimension of D2, image features have a dimension of D3, and sensor features have a dimension of D4. For subsequent feature fusion, it is necessary to transform the features of each modality to a unified feature space. A fully connected mapping layer is used to transform the dimension of the feature vectors of each modality, mapping feature vectors of different dimensions to a unified target dimension D. The mapping layer contains a fully connected layer and a non-linear activation function, which can learn modality-specific feature transformations. The mapped feature vectors are then normalized using L2 normalization to normalize the magnitude of the feature vectors to 1, eliminating the differences in numerical scales between different modal features. By performing dimensional transformation, normalization, and other dimensional alignment and standardization processes on the feature vectors of each modality, feature vectors with uniform dimensions and consistent numerical ranges for each modality are obtained.

[0035] Step 2.6, missing mode processing; Based on the missing modality information centrally labeled in the time-synchronized multimodal dataset output in step 1, a missing modality completion or labeling strategy is adopted to process the missing feature vectors with uniform dimensions and consistent numerical ranges output in step 2.5, ensuring that the subsequent fusion process can handle incomplete modalities. For completely missing modalities, a zero-vector imputation method is used, setting the feature vector of the modality to an all-zero vector and setting the missing flag to 1 to indicate that the modality is missing. For partially missing modalities, if only some data fields are missing but the modality as a whole is available, the missing fields are imputed or default values ​​are used during feature extraction. A context-dependent missing modality inference method is adopted to infer the feature representation of the missing modality using information from other available modalities. Specifically, a modality inference network is trained, with the input being the feature vectors of other available modalities and the output being the estimated feature vector of the missing modality. The modality inference network is trained using complete data during the training phase to learn the correlation between modalities, and during the testing phase, when a modality is missing, the features of the missing modality are inferred using available modalities. By performing missing modality processing such as missing modality completion, labeling, and inference, we ensure that each time window has a complete set of modal features. Even if some modalities are actually missing, there are corresponding filling or inferred values, resulting in a complete set of multimodal features.

[0036] This step, following the sub-steps described above, yields a set of multimodal feature vectors, including structured modality feature vectors, text modality feature vectors, image modality feature vectors, sensor modality feature vectors, and modality missing markers. These feature vectors have uniform dimensions and standardized values, reflecting the semantic information and risk signals of each modality.

[0037] Step 3: Based on the multimodal feature vector set, an adaptive weight allocation strategy is used to fuse multimodal features to obtain a fused feature vector sequence; Step 3.1, Modal Internal Feature Enhancement; Based on the feature vectors of each modality in the multimodal feature vector set output in step 2, a self-attention mechanism is used to process the internal feature correlation of the structured modality feature vectors, text modality feature vectors, image modality feature vectors, and sensor modality feature vectors, enhancing the feature representation ability within each modality. Although features have been extracted for each modality through a dedicated network, correlations still exist between different feature dimensions within each modality. A self-attention layer is used to process the feature vectors of each modality. The self-attention mechanism calculates the correlation between different dimensions in the feature vectors and uses attention weights to weight and combine the feature dimensions, enabling the model to emphasize important feature dimensions. By processing the feature vectors of each modality using the self-attention mechanism, the self-attention layer contains three mapping matrices: Query, Key, and Value. The input feature vectors are mapped to query vectors, key vectors, and value vectors, respectively. The dot product of the query and key is calculated to obtain the attention score, which is then normalized using softmax to obtain the attention weight. The attention weights are multiplied by the value vectors and summed to obtain the output feature vector. Through self-attention enhancement, enhanced feature vectors for each modality are obtained.

[0038] Step 3.2, cross-modal correlation calculation; Based on the enhanced feature vectors of each modality output in step 3.1, a cross-modal attention mechanism is employed to calculate the correlations between structured modalities and text modalities, structured modalities and image modalities, structured modalities and sensor modalities, text modalities and image modalities, text modalities and sensor modalities, and image modalities and sensor modalities. This calculates the correlations and interactions between different modalities. Project risks are often the result of multiple factors working together, and there are correlations and complementarities between information from different modalities. For example, quality issues mentioned in a text report may be visually reflected in an image; cost overruns in structured data may be related to change requests in the text; and environmental anomalies detected by sensors may affect construction progress. The cross-modal attention mechanism can model these semantic associations between modalities. For each pair of modalities, the cross-modal attention between them is calculated. Taking structured modalities and text modalities as an example, the structured features are used as the query, and the text features as the key and value. The cross-modal attention is calculated to obtain the attention representation of the structured modality on the text modality, which reflects the supplementation and interpretation of structured data by textual information. Similarly, using textual features as queries and structured features as keys and values, we compute the attention representation of the text modality on the structured modality. Cross-modal attention is computed for all modality pairs to obtain the attention representation of each modality on other modalities; these representations capture the interaction information between modalities. Cross-modal attention representations are obtained through cross-modal correlation calculation.

[0039] Step 3.3, Gating fusion mechanism; Based on the cross-modal attention representation output in step 3.2, a gating mechanism is employed to control the weights of each modality's attention representation on other modalities, adaptively controlling the fusion ratio of information from different modalities. The importance of different modalities for risk prediction varies across different project stages and risk scenarios. For example, in the early stages of a project, text-based planning documents and contract terms are more important; during the construction phase, on-site images and sensor data are more critical; and in the later stages, cost and schedule data receive more attention. The gating mechanism can adaptively adjust the weights of each modality based on the current input. For each modality, a gating unit is designed. The input to the gating unit is the concatenation of the original features of that modality and all cross-modal interaction features. A gating value is obtained through a fully connected layer and a sigmoid activation function. The gating value, between 0 and 1, represents the proportion of information retained from that modality. The gating value is multiplied by the modal features to obtain the gated modal features. Gating mechanisms enable models to selectively utilize information from different modalities during the fusion process. When information from a particular modality is unreliable or irrelevant, its weight is reduced; conversely, when information from a particular modality is crucial and reliable, its weight is increased. Through this gating fusion mechanism, gated modal features are obtained.

[0040] Step 3.4, multimodal feature fusion; Based on the gated modal features output from step 3.3, a feature concatenation and deep fusion network are used to concatenate and nonlinearly transform the gated features of all modalities to obtain a unified multimodal fusion representation. Specifically, the gated features of all modalities are concatenated to form a high-dimensional joint feature vector, which contains information from all modalities. This joint feature vector is then input into the fusion network, which contains multiple fully connected layers and nonlinear activation functions. Through multiple nonlinear transformations, the joint features are mapped to the fusion feature space. The fusion network can learn the nonlinear interaction relationships between modalities and discover higher-order patterns inherent in modal combinations. Residual connections are added to the fusion network, directly adding the input to the output. Residual connections can alleviate the gradient vanishing problem in deep networks, making the network easier to train. Dropout regularization is added to randomly drop some neurons during training to prevent overfitting and improve generalization ability. Through the processing of the fusion network, a fusion feature vector is obtained, which is a comprehensive representation of information from all modalities.

[0041] Step 3.5, orthogonal constraints on fused features; Based on the fused feature vector output in step 3.4 and the original feature vectors of each modality output in step 2, an orthogonal constraint loss function is used to constrain the inner product between feature vectors of different modalities, reducing redundant information between modalities. A challenge of multimodal fusion is that different modalities may contain repetitive information. If multiple modalities repeatedly express the same risk factor, it will lead to information redundancy in the fused features, reducing the model's discriminative ability. Orthogonal constraints encourage the feature representations of different modalities to remain relatively independent in the feature space, allowing each modality to contribute unique information. Specifically, the inner product or cosine similarity between feature vectors of different modalities is calculated, and an orthogonal loss function is constructed. The orthogonal loss is defined as the sum of the squares of the pairwise inner products of modal feature vectors, and the goal is to minimize this loss to make the modal feature vectors as orthogonal as possible. During model training, the orthogonal loss is added to the total loss function and jointly optimized with the risk prediction loss. Through orthogonal constraints, the feature extraction networks of different modalities are prompted to learn complementary feature representations, reducing information redundancy and improving the effectiveness of fused features.

[0042] Step 3.6, enhance the robustness of missing fusion features; Based on the modality missing flags output in step 2, adversarial training and data augmentation methods are used to robustly process the orthogonally constrained fusion features output in step 3.5, enhancing the robustness of the fusion model to modality missing data. In practical applications, due to equipment failures, network interruptions, etc., data for certain modalities may be missing at certain times. The model needs to be able to make effective risk predictions even with incomplete modalities. An adversarial training strategy is adopted to randomly simulate modality missing situations during training, setting the feature vectors of some modalities to zero or replacing them with noise, training the model to still make accurate predictions under modality missing conditions. By training under different modality missing modes, the model learns the ability to use available modalities for reasoning, reducing dependence on any single modality. Modality Dropout regularization is adopted, randomly discarding one or more modalities in each batch of training, forcing the model not to over-rely on any one modality, but to comprehensively utilize information from multiple modalities. Through the enhancement of robustness to missing data, when the model encounters modality missing data during the testing phase, it can adaptively adjust the fusion strategy and make risk predictions based on available modalities, resulting in a fusion model that is robust to modality missing data.

[0043] This step, following the sub-steps described above, yields the fused feature vector. The fused feature vectors from each time window during project execution are then arranged chronologically to form a fused feature vector sequence. This sequence integrates information from all modalities, including structured data, text, images, and sensors. Adaptive information fusion is achieved through cross-modal interaction and gating mechanisms, redundancy is reduced through orthogonal constraints, and robustness enhancements improve tolerance to modality loss.

[0044] Step 4: Based on the fused feature vector sequence, analyze the temporal evolution pattern and long-term / short-term dependency relationship to obtain the temporal feature representation sequence; Step 4.1, Time and Location Encoding; Based on the fused feature vector sequence output from step 3, a learnable temporal positional encoding is used to embed positional information into the fused feature vector at each time step in the sequence. The Transformer architecture itself does not contain temporal order information; positional encoding is needed to explicitly tell the model the temporal order of the input sequence. For project risk prediction tasks, temporal order is crucial information because the evolution of risk follows temporal causality. A temporal positional encoding is designed to generate a positional encoding vector for each time step in the sequence. The positional encoding vector has the same dimension as the fused feature vector. The positional encoding vector is added to the fused feature vector to obtain the input representation with positional information. A learnable positional encoding method is used to map each time position to a learnable embedding vector, and the optimal positional representation is learned through training. Compared to fixed sinusoidal positional encoding, learnable positional encoding has greater flexibility and can learn task-related positional representations according to task requirements. Through positional encoding, the temporal encoder can distinguish the input at different time steps, retain temporal order information, and obtain a fused feature sequence with positional encoding.

[0045] Step 4.2, Multi-head self-attention mechanism; Based on the fused feature sequence with positional encoding output from step 4.1, a multi-head self-attention mechanism is employed to calculate attention for the positional input representation at each time step in the sequence, capturing the dependencies between different time steps. The self-attention mechanism can model the association between any two time steps in the sequence, regardless of distance, making it particularly suitable for capturing long-term dependencies. For project risk prediction, the current risk state is not only related to the most recent time steps but may also be affected by earlier events. For example, early contract terms may influence later cost risks, and early technical decisions may affect later schedule risks. The multi-head self-attention mechanism divides the feature vector into multiple heads, each calculating attention independently, and then concatenates the results from multiple heads. This multi-head mechanism enables the model to capture information from different representation subspaces, enhancing its representational capabilities. For each time step in the sequence, its attention weight with all other time steps is calculated. The attention score is obtained through the dot product of the query and key, and the attention weight distribution is obtained after softmax normalization. The attention weight reflects the importance of different time steps to the current time step; a high weight indicates that the historical time step has a certain influence on the current risk state. The attention weights are multiplied by the value vector and summed to obtain the self-attention output representation. This representation incorporates information from all time steps in the sequence and is weighted according to relevance. Through the multi-head self-attention mechanism, the representation of each time step in the sequence contains the contextual information of the entire historical sequence, resulting in a self-attention representation.

[0046] Step 4.3, Multi-scale temporal window attention; Based on the self-attention representation output in step 4.2, a multi-scale time window attention mechanism is adopted to divide the time step representation containing contextual information of the entire historical sequence into windows of different time scales and calculate attention to capture patterns at different time scales. The evolution of project risks has multi-scale characteristics; some risk factors change rapidly in the short term, while others accumulate slowly in the long term. A single-scale attention mechanism is difficult to capture these patterns at different time scales simultaneously. A multi-scale attention module is designed to divide the time series into time windows of different lengths and calculate attention independently at each scale. For example, three scales are defined: a short-term window of 3 days to capture recent rapid changes; a medium-term window of 7 days to capture weekly trends; and a long-term window of 30 days to capture monthly cumulative effects. For each scale, the sequence is segmented using a sliding window method, and local attention is calculated within each window to obtain the representation at that scale. The representations at different scales are fused through a gating mechanism. The gating unit adaptively selects the weights of information at different scales based on the current input, increasing the weight of the short-term scale when the risk changes rapidly and increasing the weight of the long-term scale when the risk changes slowly. By using multi-scale attention, the model can simultaneously focus on both the short-term fluctuations and long-term trends of project risk, thus obtaining a multi-scale attention representation.

[0047] Step 4.4: Connect the feedforward neural network to the residual; Based on the multi-scale attention representation output from step 4.3, a feedforward neural network is used for further feature transformation. Position-wise nonlinear transformations are performed on the outputs of multi-head self-attention and multi-scale attention, and information flow is maintained through residual connections. The feedforward neural network contains two fully connected layers and a nonlinear activation function, performing position-wise nonlinear transformations on the attention output to enhance the model's expressive power. The first fully connected layer expands the feature dimension to a higher dimension, then passes it through a ReLU or GELU activation function; the second fully connected layer compresses the dimension back to the original dimension. Residual connections are added between the input and output of the attention layer and the feedforward network, directly adding the input to the output to form a shortcut path. Residual connections serve two purposes: first, they alleviate the gradient vanishing problem in deep networks, allowing gradients to propagate directly to shallow layers, facilitating the training of deep networks; second, they preserve original feature information, preventing the loss of important information during multi-layer transformations. Layer normalization is added after the residual connections to standardize the features, stabilizing the feature distribution and accelerating training convergence. Through the feedforward network and residual connections, each layer of the temporal encoder can perform feature abstraction while preserving input information, resulting in the output of a single-layer Transformer encoder.

[0048] Step 4.5, Multi-layer stacking and deep timing representation; Based on the single-layer Transformer encoder output from step 4.4, a multi-layer stacking approach is adopted, using the output of the previous layer as the input of the next layer for layer-by-layer feature abstraction, thus constructing a deep temporal coding network. A single-layer Transformer can capture certain temporal dependencies, but for complex project risk evolution patterns, a multi-layer network is needed for hierarchical feature abstraction. Multiple Transformer encoder layers are stacked, using the output of the previous layer as the input of the next layer, extracting higher-level temporal features layer by layer. Shallow layers capture local temporal patterns and direct dependencies, while deep layers capture global temporal structures and abstract dependencies. The network depth is set to L layers, with the value of L determined based on the amount of data and task complexity. A network that is too shallow lacks expressive power, while a network that is too deep is prone to overfitting and difficult to train. In practice, for medium-sized project data, setting L to 4 to 6 layers yields good results. Through multi-layer stacking, a deep temporal feature representation is obtained. The representation of each time step in the sequence, after multi-layer transformation, contains rich temporal context information and long- and short-term dependencies.

[0049] Step 4.6, Causal masking and future information masking; Based on the deep temporal feature representation output in step 4.5, a causal masking mechanism is employed to mask future information during the attention calculation of the temporal encoder, ensuring that future information is not used in prediction. In risk prediction tasks, the model needs to predict future risks based on historical and current information, and cannot use information from future moments. During the training phase, to improve training efficiency, the entire sequence is usually input into the model at once. However, when calculating attention, it is necessary to ensure that each time step can only see information up to and including that time step, and not future information. A causal mask or autoregressive mask is used, setting the attention score for future time steps to negative infinity during attention calculation, so that its weight becomes zero after softmax, thus masking future information. The causal mask forms a lower triangular matrix, where elements on and below the diagonal are 0 to indicate no masking, and elements in the upper triangle are negative infinity to indicate masking. Through causal masking, it is ensured that the model learns genuine causal relationships rather than spurious correlations, avoiding information leakage and enabling the model to correctly predict based on known information during the testing phase, obtaining the temporal feature representation sequence.

[0050] This step, following the sub-steps described above, yields a sequence of temporal feature representations. These temporal feature representations, processed by a Transformer encoder, capture the temporal dependencies of project risk evolution, including long-term and short-term dependencies, multi-scale patterns, and global structure. Furthermore, causal masking ensures the effectiveness of the predictions. This sequence of temporal feature representations serves as input to the subsequent risk prediction module.

[0051] Step 5: Based on the time-series feature representation sequence, perform risk prediction and trend analysis to obtain multi-category risk prediction results; Step 5.1, Multi-branch Risk Prediction Network; Based on the temporal feature representation sequence output in step 4, a multi-branch network structure is adopted to branch the deep temporal feature representation, designing an independent prediction branch for each risk category. Project risks include multiple categories, each with different characteristics and influencing factors. This embodiment defines five main risk categories: schedule risk (project task delays, milestone delays, and project duration extensions); cost risk (budget overruns, cost overruns, and funding shortages); quality risk (substandard engineering quality, quality defects, and rework / repairs); safety risk (safety accidents, personnel injuries, and equipment damage); and external risk (risks caused by external factors such as policy changes, market fluctuations, and supply chain disruptions). A prediction branch is designed for each risk category, containing several fully connected layers that map the temporal feature representation to the risk prediction output. The network structure of different branches can be the same or different, customized according to the characteristics of the risk category. The last layer of each branch is the output layer, outputting the prediction result for that risk category. The output can be a probability distribution of the risk level or a numerical risk score. Through the multi-branch structure, the model can learn specific prediction patterns for different risk categories, obtaining the prediction branch results.

[0052] Step 5.2, risk level classification output; Based on the prediction branch results output in step 5.1, a softmax classifier is used to transform the probability distribution of the prediction branch output for each risk category, obtaining the probability distribution of the risk level. The project risk level is divided into several levels; this embodiment uses a four-level division: low risk indicates that the risk factors are within a controllable range and the project is proceeding normally; low-to-medium risk indicates that there are some risk signals but the impact is limited, requiring attention; medium-to-high risk indicates that the risk factors are significant and may affect the project, requiring countermeasures; high risk indicates that the risk is severe and may lead to the project's inability to achieve its goals, requiring urgent intervention. For each risk category, the prediction branch output is a four-dimensional vector, which is transformed into a probability distribution using the softmax function. Each dimension corresponds to the probability of a risk level, and the sum of the four probabilities is 1. The level with the highest probability is used as the predicted risk level, and the probability value reflects the confidence level of the prediction. Through risk level classification, managers can intuitively understand the current project's risk status in various dimensions and determine the priority of response strategies based on the severity of the risk level.

[0053] Step 5.3, Risk score regression output; Based on the prediction branch results output in step 5.1, a linear regression layer is used to perform numerical regression processing on the prediction branch output for each risk category, obtaining a continuous numerical risk score. Besides discrete risk levels, continuous risk scores provide finer-grained risk quantification. The risk score is defined as a real number between 0 and 1, where 0 represents no risk, 1 represents extremely high risk, and intermediate values ​​represent different levels of risk. The output layer of the prediction branch uses a sigmoid activation function to compress the output values ​​to the range of 0 to 1, obtaining the risk score. The risk score can correspond to a risk level; for example, a score of 0 to 0.25 corresponds to low risk, 0.25 to 0.5 corresponds to low-to-medium risk, 0.5 to 0.75 corresponds to medium-to-high risk, and 0.75 to 1 corresponds to high risk. The advantage of risk scores is that they can reflect subtle changes in risk; even within the same risk level, changes in the score can indicate shifts in the risk situation. Through risk score regression, the model outputs a numerical risk indicator, facilitating quantitative analysis and trend tracking.

[0054] Step 5.4, Risk Trend Prediction; Based on the time-series feature representation sequence output from step 4, a sequence-to-sequence prediction model is adopted to perform multi-step prediction processing on the current and historical time-series features, obtaining risk trends for multiple future time steps. Risk prediction not only assesses the current risk status but also predicts future risk evolution trends, providing project managers with forward-looking decision support. A risk trend prediction module is designed, taking current and historical time-series features as input to predict risk scores or risk levels for several future time steps, such as the next 7 days or the next 30 days. An autoregressive approach is used for multi-step prediction. The first step predicts the risk status of the next time step, and the prediction result is used as part of the input to continue predicting the risk status of the next time step, recursively obtaining the prediction sequence for multiple future steps. Alternatively, a multi-output approach can be used, with the prediction branch directly outputting risk prediction vectors for multiple future time steps. The trend prediction results are displayed in curve form, with time on the horizontal axis and risk score or risk level on the vertical axis. The trend of the curve shows whether the risk is rising, falling, or fluctuating. Trend prediction enables managers to identify the deterioration trend of risk in advance, take timely preventive measures, avoid risk outbreaks, and obtain a risk trend prediction sequence.

[0055] Step 5.5, Joint Loss Function Design; Based on the risk level probability distribution output in step 5.2, the continuous risk score values ​​output in step 5.3, the risk trend prediction sequence output in step 5.4, and the true labels, a joint loss function is used to calculate losses and assign weights to the outputs of multiple prediction tasks, achieving collaborative optimization across multiple tasks. The goal of multi-task learning is to simultaneously optimize multiple related tasks, enabling them to share knowledge and feature representations and improve overall performance. The joint loss function contains multiple sub-loss terms, with one loss term corresponding to each prediction task. For the risk level classification task, the cross-entropy loss function is used to measure the difference between the predicted probability distribution and the true level. For the risk score regression task, the mean squared error loss or mean absolute error loss is used to measure the difference between the predicted score and the true score. For the risk trend prediction task, a sequence loss function is used to accumulate the prediction error at each future time step. The total loss is obtained by weighted summation of the sub-losses, with weight coefficients reflecting the relative importance of different tasks and adjustable according to business needs. Regularization terms can also be added to the joint loss, such as L2 regularization to prevent excessively large model parameters and Dropout regularization to prevent overfitting. By optimizing the joint loss, the model learns multiple tasks simultaneously during training. The correlation and common information between tasks can be transmitted through shared feature representations, improving the model's generalization ability and prediction performance.

[0056] Step 5.6, Modeling the risk linkage effect; Based on the risk level probability distribution output in step 5.2 and the continuous risk score output in step 5.3, a risk association network is used to analyze the correlation of prediction results for multiple risk categories and model the linkage effect between different risks. Different types of risks in a project are often not independent but interconnected and mutually influential. For example, schedule delays may lead to expedited work, which may cause quality problems and safety hazards. Quality problems may lead to rework, further exacerbating schedule delays and cost overruns, and cost overruns may lead to tight cash flow and affect resource allocation. This cascading propagation and linkage effect of risks is a significant source of risk in project management. A risk association network is designed, with different risk categories as nodes and causal or correlational relationships between risks as edges. Based on historical project data, the association patterns between risks are learned, and the influence relationships between risks are identified through statistical analysis or causal inference methods. Risk association constraints are introduced into the risk prediction model; when the predicted value of a certain type of risk is high, the prediction of its associated risk categories should also be adjusted accordingly. A graph neural network is used to model the risk association network. Graph neural networks can propagate information on a graph structure, transmitting information from one node to its neighboring nodes. By modeling risk linkages, the model can predict the cascading effects of risks, identify possible risk propagation paths, provide more comprehensive information for risk response, and obtain multi-category risk prediction results.

[0057] This step, following the sub-steps described above, yields multi-category risk prediction results. For each time window, the output includes a classification of the five risk categories, risk scores, risk trend curves for multiple future time steps, and the interrelationships between risks. These risk prediction results provide project managers with a comprehensive risk view, encompassing both current status assessments and future trend predictions, focusing on both individual risks and their interconnections. These multi-category risk prediction results serve as the foundation for subsequent interpretability analysis and decision support.

[0058] Step 6: Perform risk attribution analysis and mitigation measure matching on the multi-category risk prediction results to obtain a comprehensive risk analysis report; Step 6.1, cross-modal attention weight analysis; Based on the cross-modal attention weights output in step 3, statistical analysis and visualization methods are used to perform statistical analysis on the attention weight matrix calculated by the cross-modal attention mechanism, analyzing the contribution of each modality to risk prediction. Cross-modal attention weights reflect the importance of different modal information during the fusion process; high weights indicate that the modality makes a significant contribution to the current fusion result. For each time window, the weight matrix of the cross-modal attention layer is extracted, with rows corresponding to different modalities and columns corresponding to attention distribution. The average attention weight of each modality is calculated as its importance score. The importance scores are normalized so that the sum of the importance scores of all modalities is 1, yielding the percentage of modal contribution. Visual charts of modal contribution are generated, using pie charts or bar charts to display the contribution ratio of each modality, allowing managers to intuitively understand which information sources the current risk prediction mainly relies on. By comparing the modal contribution of different time windows, the dynamic changes in modal importance can be observed; for example, text modality has a high proportion in the early stages of a project, image modality has a high proportion during the construction phase, and cost modality has a high proportion in the final stage. Modal contribution analysis helps managers understand the sources of information for risk prediction, enhances their trust in model output, and yields modal contribution analysis results.

[0059] Step 6.2, Temporal attention weight analysis; Based on the self-attention weights of the time-series encoder output in step 4, this study analyzes the attention weight vectors calculated by the self-attention mechanism of the time-series encoder using weight aggregation and key moment identification methods. This analysis examines the impact of historical time steps on current risks. The time-series attention weights reflect the importance of different time steps in the time series for predicting the current moment; high weights indicate that the state of that historical moment has a certain impact on the current risk. For the current time step, the attention weight vectors for all historical time steps in the self-attention layer are extracted. Each element of the weight vector corresponds to the importance of a historical time step. The time steps with the highest weight values ​​are identified; these time steps correspond to key moments or important events in project execution. The project data at these key moments is reviewed to see what important events occurred at these moments, such as major decisions, completion of key tasks, unexpected events, and policy changes. A time-axis attention heatmap is generated, with the horizontal axis representing historical time steps and the vertical axis representing attention weights. Color intensity indicates the magnitude of the weights, enabling managers to quickly locate historical key nodes affecting the current risk. Time-series attention analysis reveals the historical roots of risks, helping managers trace the formation process of risks and providing a basis for subsequent risk responses, resulting in a time-series impact analysis.

[0060] Step 6.3, Text Keyword Extraction and Risk Attribution; Based on the text modal feature representation output from step 2 and the attention weights output from step 3, keyword extraction and semantic analysis methods are used to analyze the activation values ​​in the text feature vector and the weights of the attention pooling layer to identify key information related to risk. The text data contains a large amount of project descriptions, problem records, and communication information, which contain explicit or implicit expressions of risk. By analyzing the activation values ​​in the text feature vector, words or phrases that contribute significantly to risk prediction are identified. The weights of the attention pooling layer are used to trace back to words in the original text; words with high weights are identified as keywords. A risk keyword dictionary is constructed, including words related to different risk categories. For example, words related to schedule risk include delay, lag, postponement, and postponement; words related to cost risk include overspending, exceeding budget, price increases, and funding shortages. The frequency and location distribution of keywords in the text are statistically analyzed; keywords that appear frequently and are widely distributed indicate that the risk factor is relatively common and persistent. Sentiment analysis is performed to identify the sentiment tendency of the text; negative sentiment is often related to risk. Entity relationship extraction is performed to identify risk entities mentioned in the text, such as task names, responsible persons, and equipment names, as well as the relationships between them, such as causal relationships and temporal relationships. A risk attribution report is generated, listing the main risk keywords, related text fragments, involved entities, and relationships, providing managers with textual evidence of risk. Text keyword extraction and attribution make the risk prediction results traceable and interpretable, yielding text keywords and risk attribution results.

[0061] Step 6.4, Image heatmap generation and anomaly localization; Based on the image modal feature representation and convolutional network activation mapping output from step 2, a class activation mapping method is used to generate heatmaps from the image feature representation and convolutional feature maps, thus locating risk-related visual regions. Image data contains a wealth of visual information about the construction site, and risk factors may be reflected in certain areas or targets within the image. The class activation mapping method generates a heatmap by calculating the contribution of each position in the convolutional feature map to the final prediction result. Darker areas on the heatmap represent visual regions that contribute significantly to risk prediction. For high-risk predictions, a corresponding image heatmap is generated and overlaid on the original image to highlight risk-related areas. Through the heatmap, managers can intuitively see which parts of the image the model focuses on; for example, the model may focus on chaotic areas, improperly placed equipment, areas with concentrated personnel, or areas with obvious quality defects. Combined with target detection results, specific targets in the heatmap are labeled, such as personnel not wearing safety helmets, damaged equipment, and substandard construction parts. An image risk location report is generated, including images labeled with heatmaps, detected risk targets, and risk level assessments of the targets. Image heatmaps and anomaly localization make the visual basis for risk prediction clearly visible, enhancing the credibility of the model and yielding image heatmap and anomaly localization results.

[0062] Step 6.5, Recommendations for risk mitigation measures; Based on the multi-category risk prediction results output in step 5 and the risk attribution analysis results output in steps 6.1 to 6.4, rule-based reasoning and case library retrieval methods are used to match the risk prediction results and risk attribution analysis with appropriate measures, generating risk mitigation recommendations. The ultimate goal of risk prediction is to support risk response decisions. Besides informing managers what risks exist, it should also provide recommendations on how to address them. A risk response measure library is established, containing standard response measures for different risk categories and risk levels. The library's content is derived from project management best practices, historical project experience, and expert knowledge. For currently predicted risks, matching response measures are retrieved from the library based on the risk category and risk level. Combined with the results of the risk attribution analysis, the response measures are customized, providing targeted recommendations for specific risk sources. For example, if schedule risk is mainly caused by resource shortages, it is recommended to increase resource input or adjust resource allocation; if cost risk is mainly caused by rising material prices, it is recommended to lock in material prices or find alternative materials; if quality risk is mainly caused by a specific construction process, it is recommended to strengthen quality inspection and technical guidance for that process. Employing a case-study reasoning approach, similar risk scenarios are retrieved from historical projects. By referencing the countermeasures and their effects in historical cases, lessons can be learned for current risks. A risk response recommendation report is generated, including a risk description, risk level, risk source, recommended countermeasures, implementation steps, expected results, and reference cases. The risk mitigation measure recommendations transform the risk prediction system from a diagnostic tool into a decision support tool, helping managers translate risk identification into risk response action.

[0063] Step 6.6, Visual interactive interface; Based on the interpretability analysis results output from steps 6.1 to 6.5, Web technology and data visualization technology are used to display the modal contribution, time series impact analysis, text keywords, image heatmap, risk attribution explanation, and mitigation measure suggestions in a user-friendly interface. The visualization interface serves as the window through which users interact with the risk prediction system; its design should be simple, intuitive, information-rich, and provide smooth interaction. The interface includes the following main modules: The Risk Overview module displays the level and score of various risks in the current project in a dashboard format, using color coding to indicate risk severity: red for high risk, orange for medium-high risk, yellow for medium-low risk, and green for low risk; the Risk Trend module displays the risk score trend over multiple future time steps in a line graph format, allowing users to view trend curves for different risk categories, with key time nodes and expected risk peaks marked on the trend graph; the Modal Contribution module displays the contribution of each modality to risk prediction in a pie chart or stacked bar chart format, allowing users to click to view detailed information for a specific modality; the Time-Series Impact module displays the impact of historical time steps on the current risk in a timeline heatmap format, allowing users to click on key moments to view project data and event records for that moment; the Text Analysis module displays risk keyword cloud maps, key text fragments, and sentiment analysis results, allowing users to perform keyword searches and text filtering; the Image Analysis module displays on-site images with heatmap annotations, detected risk targets, and abnormal area locations, allowing users to browse images from different times and regions; and the Response Suggestions module displays recommended risk mitigation measures, implementation steps, and reference cases, allowing users to evaluate and provide feedback on the suggestions. The interface supports various interactive operations, such as time range selection, risk category filtering, modal data drill-down, and report export. Through the visual interactive interface, project managers can gain a comprehensive understanding of the project's risk status, deeply analyze the sources of risk, make timely response decisions, and obtain a comprehensive risk analysis report and a visual interactive interface.

[0064] The above sub-steps in this step produce a comprehensive risk analysis report and a visual interactive interface. The comprehensive risk analysis report includes modal contribution, time-series impact analysis, text keywords, image heatmaps, risk attribution explanations, and mitigation recommendations. This interpretable output transforms the risk prediction system from a black-box model into a white-box system, enhancing user trust and practical value.

[0065] A project risk dynamic prediction system based on multimodal fusion deep learning, such as Figure 3 As shown, the method for performing the above-described dynamic prediction of project risk based on multimodal fusion deep learning includes: The data acquisition module collects multimodal data of the target project, performs time alignment and quality checks, and obtains a time-synchronized multimodal dataset. The feature extraction module performs multimodal feature extraction and processing based on a time-synchronized multimodal dataset to obtain a set of multimodal feature vectors. The feature fusion module, based on a set of multimodal feature vectors, uses an adaptive weight allocation strategy to fuse multimodal features and obtain a fused feature vector sequence. The time series modeling module analyzes the time series evolution patterns and long-short-term dependencies based on the fused feature vector sequence, and obtains the time series feature representation sequence. The risk prediction module performs risk prediction and trend analysis based on time-series feature representation sequences, and obtains multi-category risk prediction results; The interpretability analysis module performs risk attribution analysis and mitigation measure matching on the prediction results of multiple categories of risks, and obtains a comprehensive risk analysis report.

[0066] In one embodiment of the present invention, a specific example is provided: When a smart city infrastructure project reached its 18th month of implementation, the risk prediction system provided the following analysis results: Table 1 shows examples of multimodal data acquisition: Table 1: Examples of multimodal data acquisition;

[0067] Table 2 shows an example of risk prediction results: Table 2: Examples of Risk Prediction Results;

[0068] Modal contribution analysis: Structured data contributes 38%, text data contributes 27%, image data contributes 18%, and sensor data contributes 17%.

[0069] Critical Moments Identification: Day 520: Major changes initiated; Day 528: Delays begin on the critical path; Day 532: Material supplier notifies of price adjustments.

[0070] Text keywords: In recent reports, "delay" appeared 15 times, "cost" appeared 23 times, "budget" appeared 18 times, and "change" appeared 12 times.

[0071] Image anomaly: On-site images show that the material storage area is cramped and construction progress in some areas is significantly behind schedule.

[0072] Mitigation recommendations: For schedule risks, it is recommended to allocate additional human resources to critical path tasks and optimize task scheduling to reduce waiting time; for cost risks, it is recommended to negotiate and lock in prices with material suppliers and review non-critical expenditures to find ways to save.

[0073] As can be seen from this application example, the risk prediction system of the present invention can comprehensively utilize multimodal data to accurately identify project risks, provide interpretable analysis results and practical response suggestions, and provide effective decision support for project managers.

[0074] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A dynamic prediction method for project risk based on multimodal fusion deep learning, characterized in that, Includes the following steps: Collect multimodal data of the target project, perform time alignment and quality checks, and obtain a time-synchronized multimodal dataset; Based on a time-synchronized multimodal dataset, multimodal feature extraction and feature processing are performed to obtain a set of multimodal feature vectors; Based on a set of multimodal feature vectors, an adaptive weight allocation strategy is used to fuse multimodal features to obtain a fused feature vector sequence. Based on the fused feature vector sequence, the temporal evolution pattern and long-short-term dependency relationship are analyzed to obtain the temporal feature representation sequence; Based on the temporal feature representation sequence, risk prediction and trend analysis are performed to obtain multi-category risk prediction results; Risk attribution analysis and mitigation measure matching are performed on the prediction results of multiple categories of risks to obtain a comprehensive risk analysis report.

2. The project risk dynamic prediction method based on multimodal fusion deep learning according to claim 1, characterized in that, The steps for obtaining the time-synchronized multimodal dataset include: Collect structured data and obtain task schedules, cost management data, resource management data, and quality management data in real time based on the standard interface of the project management information system; Collect text data, based on the document management system and email system, to obtain technical reports, security inspection logs, quality inspection reports, meeting minutes, and change request documents; Collect image and video data, and use keyframe extraction technology to obtain visual information of the construction site based on the on-site monitoring system; Collect sensor data and acquire real-time monitoring data from temperature and humidity sensors, geological subsidence monitoring instruments, and harmful gas detectors based on the Internet of Things sensor network; Based on the timestamp information of each modality data, a unified time base calibration and time window aggregation method is used to perform time alignment of multimodal data; Based on the time-aligned multimodal dataset, a data integrity check mechanism is used to identify and label missing modalities and missing data fields.

3. The project risk dynamic prediction method based on multimodal fusion deep learning according to claim 1, characterized in that, The steps for obtaining the multimodal feature vector set include: Time series feature engineering and deep time series networks are used to construct time series features and cross-index correlation features for structured data, and structured modal feature vectors are obtained by processing them through temporal convolutional networks. Natural language processing techniques and pre-trained language models are used to perform sentence segmentation, word segmentation, named entity recognition, language model encoding, sentiment analysis, and risk keyword extraction on text data to obtain text modal feature vectors. Computer vision technology and convolutional neural networks are used to perform image preprocessing, feature extraction, target detection, scene classification, and anomaly detection on image data to obtain image modal feature vectors; Time series analysis and recurrent neural networks are used to extract statistical features and time-series pattern features from sensor data, and sensor modal feature vectors are obtained by processing them through long short-term memory networks.

4. The project risk dynamic prediction method based on multimodal fusion deep learning according to claim 1, characterized in that, The step of obtaining the fused feature vector sequence includes: A self-attention mechanism is used to process the internal feature correlation of each modality feature vector; A cross-modal attention mechanism is used to calculate the correlation and interaction relationships between different modalities, resulting in a cross-modal attention representation; A gating mechanism is used to control the weights of cross-modal attention representations, adaptively controlling the fusion ratio of information from different modalities; The gated modal features are concatenated and processed by a deep fusion network to obtain a unified multimodal fusion representation; Orthogonal constraint loss function is used to reduce redundant information between modes.

5. The project risk dynamic prediction method based on multimodal fusion deep learning according to claim 1, characterized in that, The step of obtaining the temporal feature representation sequence includes: Learnable temporal location coding is used to embed location information for each time step in the sequence; A multi-head self-attention mechanism is employed to capture the dependencies between different time steps in the sequence; A multi-scale time window attention mechanism is adopted to divide the time series into time windows of different lengths to capture patterns at different time scales; A feedforward neural network is used for feature transformation, and information flow is maintained through residual connections. A deep temporal coding network is constructed by stacking multiple layers, and temporal features are extracted layer by layer.

6. The project risk dynamic prediction method based on multimodal fusion deep learning according to claim 1, characterized in that, The steps for obtaining multi-category risk prediction results include: A multi-branch network structure is adopted, with independent prediction branches designed for schedule risk, cost risk, quality risk, safety risk, and external risk; The risk level is divided into four levels: low risk, low-medium risk, medium-high risk, and high risk, using a softmax classifier, and the probability distribution of the risk level is obtained. A linear regression layer was used to obtain continuous numerical values ​​for the risk score; A sequence-to-sequence prediction model is used to predict risk trends over multiple future time steps; A joint loss function is used to collaboratively optimize multiple prediction tasks; We employ risk association networks to model the linkage effects between different risks.

7. The project risk dynamic prediction method based on multimodal fusion deep learning according to claim 1, characterized in that, The steps to obtain a comprehensive risk analysis report include: Based on cross-modal attention weights, we use weight statistics and visualization methods to analyze the contribution of each modality to risk prediction. Based on the self-attention weights of the temporal encoder, we use weight aggregation and key moment identification methods to analyze the impact of historical time steps on current risks. Based on text modal feature representation and attention weight, keyword extraction and semantic analysis methods are used to identify key information related to risk; Based on image modal feature representation and activation mapping of convolutional networks, a class activation mapping method is used to generate image heatmaps and locate risk-related visual regions. Based on the results of risk prediction and risk attribution analysis, rule-based reasoning and case retrieval methods are used to generate risk mitigation recommendations. We use web technologies and data visualization technologies to build a user-friendly interactive interface.

8. The project risk dynamic prediction method based on multimodal fusion deep learning according to claim 2, characterized in that, The time window aggregation method includes: Divide the project execution cycle into fixed-length time windows; The high-frequency sensor data and image data are aggregated within each time window. Statistical features are calculated for the sensor data, and the image with the best quality is selected as the representative image for the image data. Assign the daily and weekly reports collected in low frequency to the corresponding time windows; Generate a data quality report for each time window, identifying available modalities, missing modalities, and data quality level.

9. The project risk dynamic prediction method based on multimodal fusion deep learning according to claim 5, characterized in that, The multi-scale temporal window attention mechanism includes: The time series is divided into time windows of different lengths, including short-term, medium-term, and long-term scale windows; The sequence is segmented using a sliding window approach for each scale, and local attention is calculated within each window. Representations at different scales are fused through a gating mechanism, and the weights of information at different scales are adaptively selected based on the current input.

10. A project risk dynamic prediction system based on multimodal fusion deep learning, characterized in that, The method for dynamically predicting project risks based on multimodal fusion deep learning as described in any one of claims 1-9 includes: The data acquisition module collects multimodal data of the target project, performs time alignment and quality checks, and obtains a time-synchronized multimodal dataset. The feature extraction module performs multimodal feature extraction and processing based on a time-synchronized multimodal dataset to obtain a set of multimodal feature vectors. The feature fusion module, based on a set of multimodal feature vectors, uses an adaptive weight allocation strategy to fuse multimodal features and obtain a fused feature vector sequence. The time series modeling module analyzes the time series evolution patterns and long-short-term dependencies based on the fused feature vector sequence, and obtains the time series feature representation sequence. The risk prediction module performs risk prediction and trend analysis based on time-series feature representation sequences, and obtains multi-category risk prediction results; The interpretability analysis module performs risk attribution analysis and mitigation measure matching on the prediction results of multiple categories of risks, and obtains a comprehensive risk analysis report.

Images