Intelligent tunnel multi-modal data collaborative management method and device based on deep learning

Abstract

This application discloses a method and apparatus for collaborative management of multimodal data in smart tunnels based on deep learning, relating to the field of artificial intelligence technology. The method includes: determining the spatial topological relationships of multi-source heterogeneous sensors within the tunnel, thereby mapping multimodal data to a unified coordinate system of a digital twin, and performing timestamp compensation to form multimodal aligned data; extracting its spatiotemporal and topological features, calculating feature correlation, and embedding the data into the tunnel semantic space accordingly; assessing environmental complexity and generating modal weight coefficients; and performing a security assessment on the weighted joint feature vector to determine the tunnel's safety status, thus overcoming the limitations of single-modal monitoring in complex environments.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and apparatus for collaborative management of multimodal data in smart tunnels based on deep learning. Background Technology

[0002] Tunnels are critical infrastructure in transportation networks, used to overcome geographical barriers and improve traffic efficiency; therefore, the safety of the tunnel environment is paramount. Currently, monitoring tunnel operation typically involves deploying a large number of personnel for inspections, but this method cannot promptly identify potential safety hazards within the tunnel. Some monitoring systems collect data using sensors or cameras, but these methods are mostly single-modal data acquisition approaches, which cannot comprehensively reflect the actual operating status of the tunnel in complex environments.

[0003] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a method and device for collaborative management of multimodal data in smart tunnels based on deep learning, which aims to solve the technical problem that single-modal monitoring methods are difficult to fully reflect the tunnel operation status in complex environments.

[0005] To achieve the above objectives, this application provides a deep learning-based intelligent tunnel multimodal data collaborative management method, the method comprising:

[0006] The spatial topology of the multi-source heterogeneous sensor cluster in the tunnel is determined. Based on the spatial topology, the multimodal data is mapped to the unified coordinate system of the tunnel digital twin to obtain multimodal mapping data. The multimodal mapping data is then timestamped to obtain multimodal aligned data.

[0007] Extract the spatiotemporal and topological features of the multimodal alignment data, calculate the feature correlation degree between the spatiotemporal features and the topological features, and embed the multimodal alignment data into the tunnel semantic space based on the feature correlation degree;

[0008] The environmental complexity of the tunnel semantic space is determined, modal weight coefficients are generated based on the environmental complexity, and the multimodal alignment data is weighted based on the modal weight coefficients to obtain a joint feature vector;

[0009] A safety assessment is performed on the joint feature vector to determine the tunnel safety assessment status.

[0010] In one embodiment, the step of determining the spatial topological relationship of a multi-source heterogeneous sensor cluster within the tunnel, and mapping multimodal data to a unified coordinate system of the tunnel digital twin based on the spatial topological relationship to obtain multimodal mapping data includes:

[0011] The installation location coordinates, spatial orientation angle, and sensing range parameters of each sensor in the multi-source heterogeneous sensor cluster in the tunnel are obtained, and a spatial topology matrix is ​​constructed based on the installation location coordinates, spatial orientation angle, and sensing range parameters.

[0012] A unified coordinate system for the tunnel digital twin is determined, and a transformation matrix from the local coordinate system of each sensor to the unified coordinate system is generated based on the spatial topology relation matrix.

[0013] Based on the transformation matrix, the multimodal data is mapped to the unified coordinate system of the tunnel digital twin, generating multimodal mapping data with spatial location labels.

[0014] In one embodiment, the step of performing timestamp compensation on the multimodal mapping data to obtain multimodal aligned data includes:

[0015] The acquisition time reference of each sensor in the multi-source heterogeneous sensor cluster is unified based on hardware clock synchronization.

[0016] For wired transmission sensors, the wired transmission delay is determined based on the communication distance and signal propagation speed;

[0017] For wireless transmission sensors, the wireless transmission delay is determined based on the timestamp handshake protocol.

[0018] The original timestamp of the multimodal mapping data is determined based on the acquisition time reference, and time compensation is performed on the original timestamp according to the wired transmission delay or the wireless transmission delay to obtain time-aligned data.

[0019] The time-aligned data is reordered in chronological order to obtain multimodal aligned data.

[0020] In one embodiment, the step of time-compensating the original timestamp based on the wired transmission delay or the wireless transmission delay to obtain time-aligned data further includes:

[0021] Extract the overlapping sensing areas of adjacent sensors in the spatial topology. When an event occurs within the overlapping sensing area, compare the event trigger time with the recorded time to determine the time deviation.

[0022] When the time deviation is greater than the time deviation threshold, dynamic compensation parameters are generated based on the time deviation.

[0023] The target transmission delay is determined from the wired transmission delay or the wireless transmission delay based on the data transmission type of the sensor.

[0024] Time-aligned data is obtained by performing time compensation on the original timestamp based on the dynamic compensation parameters and the target transmission delay.

[0025] In one embodiment, the multimodal alignment data includes visual data, point cloud data, and temporal data; the spatiotemporal features include spatiotemporal motion features, spatial geometric features, and dynamic change features; and the step of extracting the spatiotemporal features and topological features of the multimodal alignment data and calculating the feature correlation degree between the spatiotemporal features and the topological features includes:

[0026] Three-dimensional spatiotemporal feature extraction is performed on the visual data to generate the spatiotemporal motion features;

[0027] Spatial structure analysis is performed on the point cloud data to generate the spatial geometric features;

[0028] The time-series data is dynamically evolved to generate the dynamic change features;

[0029] Analyze the adjacency relationships of each sensor in the spatial topology;

[0030] A sensor topology graph is constructed using sensor nodes as vertices and the adjacency relationships as edges, and topological features are generated based on the sensor topology graph.

[0031] A feature interaction matrix is ​​established based on the spatiotemporal motion features, spatial geometric features, dynamic change features, and topological features;

[0032] Singular value decomposition is performed on the feature interaction matrix, and the maximum singular value is taken as the feature correlation degree.

[0033] In one embodiment, the step of embedding the multimodal alignment data into the tunnel semantic space based on the feature correlation degree includes:

[0034] Based on the aforementioned feature correlation, each modal feature of the multimodal aligned data is weighted by correlation to obtain the correlation embedding feature;

[0035] Semantic projection is performed on the associated embedding features to obtain semantic projection features;

[0036] The semantic similarity matrix is ​​determined based on the semantic basis vectors and the semantic projection features;

[0037] When the maximum similarity of the semantic similarity matrix is ​​greater than the similarity threshold, the semantic projection features are embedded into the tunnel semantic space.

[0038] In one embodiment, the steps of determining the environmental complexity of the tunnel semantic space, generating modal weight coefficients based on the environmental complexity, and weighting the multimodal alignment data based on the modal weight coefficients to obtain a joint feature vector include:

[0039] The information entropy of the embedding features in the tunnel semantic space is determined, and the environmental complexity is determined based on the environmental interference factor and the information entropy.

[0040] Based on the complexity weight mapping relationship, determine the modal weight coefficients corresponding to the environmental complexity;

[0041] The modal weight coefficients are optimized based on the environment adaptation loss function to obtain the environment adaptation weight coefficients.

[0042] The multimodal alignment data is weighted based on the environment adaptation weight coefficient to obtain a joint feature vector.

[0043] In one embodiment, the step of performing a security assessment on the joint feature vector to determine the tunnel security assessment status includes:

[0044] The joint feature vector is input into the safety assessment model to obtain the quantitative value of the tunnel safety level;

[0045] Based on the quantified value of the tunnel safety level, spatiotemporal anomaly detection is performed to determine the type of abnormal event;

[0046] The abnormal event types are associated with the tunnel safety level to determine the tunnel safety assessment status.

[0047] In one embodiment, before the step of performing a security assessment on the joint feature vector to determine the tunnel security assessment status, the method further includes:

[0048] When a conflict is detected in the multimodal data in the joint feature vector, conflict features are generated based on the conflicting multimodal data.

[0049] The conflict features are reconstructed based on a generative adversarial network to obtain reconstructed features;

[0050] Determine the spatial topological consistency between the reconstructed features and the conflicting features;

[0051] When the spatial topology consistency verification is successful, the joint feature vector is updated based on the reconstructed features;

[0052] When the spatial topology consistency verification fails, the redundant data of the redundant nodes is obtained, and the joint feature vector is updated based on the redundant data.

[0053] Furthermore, to achieve the above objectives, this application also proposes a deep learning-based smart tunnel multimodal data collaborative management device, which includes:

[0054] The data processing module is used to determine the spatial topological relationship of the multi-source heterogeneous sensor cluster in the tunnel, map the multimodal data to the unified coordinate system of the tunnel digital twin based on the spatial topological relationship to obtain multimodal mapping data, and perform timestamp compensation on the multimodal mapping data to obtain multimodal aligned data.

[0055] The feature extraction module is used to extract the spatiotemporal features and topological features of the multimodal alignment data, calculate the feature correlation degree between the spatiotemporal features and the topological features, and embed the multimodal alignment data into the tunnel semantic space based on the feature correlation degree.

[0056] The data association module is used to determine the environmental complexity of the tunnel semantic space, generate modal weight coefficients based on the environmental complexity, and weight the multimodal aligned data based on the modal weight coefficients to obtain a joint feature vector;

[0057] The safety assessment module is used to perform a safety assessment on the joint feature vector and determine the tunnel safety assessment status.

[0058] Furthermore, to achieve the above objectives, this application also proposes a deep learning-based smart tunnel multimodal data collaborative management device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is configured to implement the steps of the deep learning-based smart tunnel multimodal data collaborative management method described above.

[0059] In addition, to achieve the above objectives, the present invention also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the deep learning-based smart tunnel multimodal data collaborative management method described above.

[0060] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the deep learning-based smart tunnel multimodal data collaborative management method described above.

[0061] This application provides a deep learning-based method for collaborative management of multimodal data in smart tunnels. By determining the spatial topological relationships of multiple heterogeneous sensors within the tunnel, multimodal data is mapped to a unified coordinate system of a digital twin, and timestamp compensation is applied to form aligned multimodal data. Spatiotemporal and topological features are extracted, and feature correlation is calculated. Based on this, the data is embedded into the tunnel's semantic space to assess environmental complexity and generate modal weight coefficients. A security assessment is performed on the weighted joint feature vector to determine the tunnel's safety status. Coordinate unification is applied to the multimodal data, aligning it spatially, followed by timestamp compensation to align it temporally, resulting in aligned multimodal data that ensures accuracy. Then, spatiotemporal and topological features of the aligned multimodal data are extracted, and weight coefficients are determined based on their correlation. Finally, a security assessment is performed using these weight coefficients and the joint feature vector to determine the tunnel's safety status. This overcomes the limitations of single-modal monitoring in complex environments. Attached Figure Description

[0062] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0063] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a flowchart illustrating an embodiment of the deep learning-based intelligent tunnel multimodal data collaborative management method of this application;

[0065] Figure 2 This is an abstract diagram of the topological relationships in an embodiment of the deep learning-based intelligent tunnel multimodal data collaborative management method of this application;

[0066] Figure 3 This is a technical flowchart of an embodiment of the deep learning-based intelligent tunnel multimodal data collaborative management method of this application;

[0067] Figure 4 This is a flowchart illustrating the feature embedding process of an embodiment of the deep learning-based intelligent tunnel multimodal data collaborative management method of this application.

[0068] Figure 5 This is a schematic diagram of the module structure of the deep learning-based smart tunnel multimodal data collaborative management device according to an embodiment of this application;

[0069] Figure 6This is a schematic diagram of the hardware operating environment involved in the deep learning-based smart tunnel multimodal data collaborative management method in the embodiments of this application.

[0070] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0071] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0072] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0073] The main solution of this application embodiment is: to determine the spatial topology relationship of the multi-source heterogeneous sensor cluster in the tunnel, to map the multimodal data to the unified coordinate system of the tunnel digital twin based on the spatial topology relationship, to obtain multimodal mapping data, and to compensate the multimodal mapping data with timestamps to obtain multimodal aligned data;

[0074] Extract the spatiotemporal and topological features of the multimodal alignment data, calculate the feature correlation degree between the spatiotemporal features and the topological features, and embed the multimodal alignment data into the tunnel semantic space based on the feature correlation degree;

[0075] The environmental complexity of the tunnel semantic space is determined, modal weight coefficients are generated based on the environmental complexity, and the multimodal alignment data is weighted based on the modal weight coefficients to obtain a joint feature vector;

[0076] A safety assessment is performed on the joint feature vector to determine the tunnel safety assessment status.

[0077] Currently, tunnels are critical infrastructure in transportation networks, used to overcome geographical barriers and improve traffic efficiency; therefore, the safety of the tunnel environment is of paramount importance. Current methods for monitoring tunnel operation typically involve deploying a large number of personnel for inspections, but this approach cannot promptly identify potential safety hazards within the tunnel. Some monitoring systems collect data using sensors or cameras, but these methods are mostly single-modal data acquisition methods, which cannot comprehensively reflect the actual operating status of the tunnel in complex environments.

[0078] This application provides a solution that maps multimodal data to a unified coordinate system of a digital twin by determining the spatial topological relationships of multiple heterogeneous sensors within a tunnel, and performs timestamp compensation to form multimodal aligned data. Spatiotemporal and topological features are extracted, feature correlation is calculated, and the data is embedded into the tunnel semantic space to assess environmental complexity and generate modal weight coefficients. A safety assessment is performed on the weighted joint feature vector to determine the tunnel's safety status. The multimodal data undergoes coordinate unification, spatial alignment, and timestamp compensation for temporal alignment, resulting in multimodal aligned data that ensures accuracy. The spatiotemporal and topological features of the multimodal aligned data are then extracted, and weight coefficients are determined based on their correlation. Finally, a safety assessment is performed using these weight coefficients and the joint feature vector to determine the tunnel's safety status. This overcomes the limitations of single-modal monitoring in complex environments.

[0079] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, such as a deep learning-based smart tunnel multimodal data collaborative management device. This embodiment does not specifically limit it in this regard. The following uses a deep learning-based smart tunnel multimodal data collaborative management device as an example to describe this embodiment and the following embodiments.

[0080] All actions involving the acquisition of signals, information, or data in this application are carried out in accordance with the relevant data protection laws and policies of the country where the application is located, and with the authorization of the owner of the relevant device.

[0081] This application provides a method for collaborative management of multimodal data in smart tunnels based on deep learning, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the deep learning-based intelligent tunnel multimodal data collaborative management method of this application.

[0082] In this embodiment, the intelligent tunnel multimodal data collaborative management method based on deep learning includes steps S10~S40:

[0083] Step S10: Determine the spatial topology of the multi-source heterogeneous sensor cluster in the tunnel, map the multimodal data to the unified coordinate system of the tunnel digital twin based on the spatial topology, obtain multimodal mapping data, and perform timestamp compensation on the multimodal mapping data to obtain multimodal aligned data;

[0084] It should be noted that the multi-source heterogeneous sensor cluster in the tunnel refers to a collection of sensors deployed in the tunnel environment, including various types (such as cameras, lidar, temperature and humidity sensors, stress and strain gauges, etc.), with different working principles and data formats. The spatial topology describes the relative spatial positions and orientation relationships between the various sensors in the cluster, which is the basis for data fusion and coordinate unification. Multimodal data refers to different forms of data collected by heterogeneous sensors, such as image point clouds, video sequences, scalar measurements (temperature, pressure, etc.). The tunnel digital twin is a high-fidelity virtual model of the tunnel physical entity, which can map and reflect the tunnel status in real time. The unified coordinate system is a unique and global coordinate system defined for the digital twin. All sensor data must be transformed into this coordinate system for consistent analysis and presentation. Multimodal mapped data refers to the data after the original multimodal data has been transformed into the unified coordinate system according to the spatial topology of its sensors. Multimodal aligned data refers to data that, on the basis of spatial alignment, further eliminates the time delay between different sensors through timestamp compensation, achieving complete synchronization in the spatiotemporal dimensions.

[0085] It is understandable that after acquiring multimodal mapping data, timestamp compensation is required to eliminate time synchronization issues caused by factors such as sensor sampling frequency and transmission delay. Therefore, this embodiment uses linear interpolation, assuming the time series of a certain sensor data stream is... Its corresponding collected value is The target unified time series is Calculated by linear interpolation Time-based compensation value The processing formula is:

[0086]

[0087] in, , This represents the data values ​​at the same timestamp after compensation. The process processes all sensor data streams sequentially to obtain time-consistent multimodal aligned data.

[0088] In one feasible implementation, the step of determining the spatial topological relationship of the multi-source heterogeneous sensor cluster within the tunnel, and mapping the multimodal data to the unified coordinate system of the tunnel digital twin based on the spatial topological relationship to obtain the multimodal mapping data includes:

[0089] The installation location coordinates, spatial orientation angle, and sensing range parameters of each sensor in the multi-source heterogeneous sensor cluster in the tunnel are obtained, and a spatial topology matrix is ​​constructed based on the installation location coordinates, spatial orientation angle, and sensing range parameters.

[0090] A unified coordinate system for the tunnel digital twin is determined, and a transformation matrix from the local coordinate system of each sensor to the unified coordinate system is generated based on the spatial topology relation matrix.

[0091] Based on the transformation matrix, the multimodal data is mapped to the unified coordinate system of the tunnel digital twin, generating multimodal mapping data with spatial location labels.

[0092] It should be noted that the installation location coordinates refer to the actual physical location of the sensor in the tunnel (such as latitude, longitude, or local coordinates), the spatial orientation angle describes its yaw, pitch, and roll directions, and the sensing range parameters include its effective monitoring distance and field of view. The spatial topology matrix is ​​a matrix that integrates all the spatial attributes of the sensors to express the relationship between their relative positions and sensing ranges. The transformation matrix is ​​a mathematical matrix that realizes the rotation and translation transformation from the sensor's local coordinate system to the unified coordinate system of the digital twin.

[0093] In practical implementation, by collecting the installation location, orientation, and sensing parameters of each sensor, a topological relationship matrix can be constructed with sensors as nodes and spatial relationships as edges. An abstract graph of the topological relationship is shown below. Figure 2 As shown. Taking three nodes as an example, the three nodes are sensor A, sensor B, and sensor C, forming three nodes Node1, Node2, and Node3. Their topological relationships are connected by three edges: Edge1→2, Edge2→3, and Edge1→3. For example, Edge1→2 represents the relative topological relationship between Node1 and Node2. Node1's position Pos is (x1, y1, z1), and its orientation angle Angle is θ1; Node2's position Pos is (x2, y2, z2), and its orientation angle Angle is θ2; Node3's position Pos is (x3, y3, z3), and its orientation angle Angle is θ3. Edge1→2, Edge2→3, and Edge1→3 represent the following: Edge1→2 corresponds to the distance between Node1 and Node2, and the relative orientation angle Δθ is the relative orientation of θ1 and θ2. Edge1→3 corresponds to the distance between Node1 and Node3, and the relative orientation angle Δθ is the relative orientation of θ1 and θ3. The distance between Edge2 and Edge3 is the distance between Node2 and Node3, and the relative orientation angle Δθ is the relative orientation of θ2 and θ3. Based on this matrix and the definition of the unified coordinate system, the transformation matrix (usually including rotation and translation components) from the local coordinate system of each sensor to the unified coordinate system is calculated using the rigid body transformation principle. This matrix is ​​then used to transform the original multimodal data to a unified spatial reference, and a position label is added to each data frame, ultimately forming multimodal mapping data with spatial identification. The mapping of sensor data from the local coordinate system to the unified coordinate system can be expressed as:

[0094]

[0095] Among them, P local This represents the coordinates of the data point in the sensor's local coordinate system; The rotation matrix is ​​calculated from the sensor space orientation angle. P is the translation vector formed by the coordinates of the sensor's installation position; global This represents the mapped coordinates of the data point in a unified coordinate system.

[0096] In one feasible implementation, the step of performing timestamp compensation on the multimodal mapping data to obtain multimodal aligned data includes:

[0097] The acquisition time reference of each sensor in the multi-source heterogeneous sensor cluster is unified based on hardware clock synchronization.

[0098] For wired transmission sensors, the wired transmission delay is determined based on the communication distance and signal propagation speed;

[0099] For wireless transmission sensors, the wireless transmission delay is determined based on the timestamp handshake protocol.

[0100] The original timestamp of the multimodal mapping data is determined based on the acquisition time reference, and time compensation is performed on the original timestamp according to the wired transmission delay or the wireless transmission delay to obtain time-aligned data.

[0101] The time-aligned data is reordered in chronological order to obtain multimodal aligned data.

[0102] It should be noted that the acquisition time reference refers to a unified time reference established for all sensors through a hardware clock synchronization protocol (such as PTP or GPS), ensuring that all devices have a consistent time starting point and timing accuracy. Wired transmission sensors refer to sensors that transmit data via wired media such as cables or optical fibers; their transmission delay is relatively fixed. Wired transmission delay refers to the time lag caused by data transmission in the wired medium, mainly depending on the communication distance and the signal propagation speed in the medium. Wireless transmission sensors refer to sensors that transmit data wirelessly via radio waves, etc., and are susceptible to environmental interference, resulting in larger delay fluctuations. Wireless transmission delay refers to the time delay introduced by data transmission, processing, and queuing in the wireless channel, usually dynamically estimated through a timestamp handshake protocol. The raw timestamp refers to the time stamp applied to the data by the local clock at the moment of sensor acquisition, before any transmission or synchronization compensation.

[0103] In practical implementation, the first step is to unify the acquisition time base of all sensors through a hardware clock synchronization protocol (such as PTP or GPS) to establish a globally consistent time reference. For wired transmission sensors, the transmission delay... Based on communication distance and the speed of signal propagation in the medium (For example, the speed of light in an optical fiber) can be calculated directly. For wireless transmission sensors, messages are exchanged back and forth using a timestamp handshake protocol (such as IEEE 802.11 TSFs or NTP-like mechanisms), and the transmission delay is dynamically estimated based on the message sending and receiving timestamps. After obtaining the original timestamps of each data packet, the appropriate delay is compensated based on its transmission method. Preliminary time-aligned data was obtained. This is the original timestamp. The target transmission delay is determined. Finally, all data is processed using the compensated timestamps. A global reordering process is performed to generate a multimodal aligned data stream strictly arranged in chronological order, providing a time-consistent data foundation for subsequent fusion and analysis. The wired transmission delay calculation formula is as follows:

[0104]

[0105] in, d is the wired transmission delay, d is the communication distance, i.e., the physical length of the data cable, and v is the speed at which the signal propagates in a specific medium.

[0106] The formula for estimating wireless transmission delay is:

[0107]

[0108] in, For the estimated one-way wireless transmission delay, The timestamp of the host sending the request message. The timestamp of the request message received by the slave device The timestamp of the response message sent by the slave device. The timestamp of the host receiving the response message.

[0109] In one feasible implementation, the step of time-compensating the original timestamp based on the wired transmission delay or the wireless transmission delay to obtain time-aligned data further includes:

[0110] Extract the overlapping sensing areas of adjacent sensors in the spatial topology. When an event occurs within the overlapping sensing area, compare the event trigger time with the recorded time to determine the time deviation.

[0111] When the time deviation is greater than the time deviation threshold, dynamic compensation parameters are generated based on the time deviation.

[0112] The target transmission delay is determined from the wired transmission delay or the wireless transmission delay based on the data transmission type of the sensor.

[0113] Time-aligned data is obtained by performing time compensation on the original timestamp based on the dynamic compensation parameters and the target transmission delay.

[0114] It should be noted that the overlapping sensing area of ​​adjacent sensors refers to the physical area in the tunnel space jointly covered by the monitoring ranges of two or more adjacent sensors (such as the field of view of a camera, the point cloud coverage area of ​​LiDAR), where events occurring can be captured simultaneously by multiple sensors. An event refers to a physical change or abnormal state that occurs in the tunnel environment and can be sensed by sensors, such as a vehicle passing by, the appearance of smoke, changes in lighting, or structural vibration. The event trigger time refers to the original timestamp carried in each data frame when the same event is detected and recorded by different sensors. Due to differences in sensor clocks and transmission paths, the same event may correspond to multiple trigger times. The deviation threshold is a threshold value used to determine whether the time deviation is significant. When the deviation is below this threshold, it is considered normal fluctuation and no additional compensation is required; when the deviation exceeds this threshold, an anomaly is determined, and dynamic compensation needs to be initiated. The dynamic compensation parameter is an adjustment amount calculated in real time to correct abnormal time deviations (such as clock drift). The target transmission delay is based on the data transmission type of the sensor.

[0115] In its implementation, the system first calculates the overlapping sensing area between adjacent sensors based on known spatial topology. When a critical event (such as a vehicle passing through) occurs within this area, the system extracts the event trigger time from the data streams of multiple sensors that detected the event. By comparing these timestamps, the time deviation Δt between them can be calculated. The time deviation is used to determine whether there is an unexpected synchronization error and to establish a preset time deviation threshold. When the time deviation exceeds the threshold, it indicates that fixed transmission delay compensation alone is insufficient to eliminate the error. In this case, dynamic compensation parameters are generated based on the time deviation. Then, the target transmission delay τ is selected based on the sensor's data transmission type. The final time compensation amount is determined by both the fixed target transmission delay and the dynamic compensation parameter. The compensation formula is:

[0116]

[0117] Based on this, not only is the fixed transmission delay compensated, but random errors and minute clock drifts that are difficult to model accurately in advance are also dynamically corrected, resulting in more accurate time-aligned data.

[0118] Step S20: Extract the spatiotemporal features and topological features of the multimodal alignment data, calculate the feature correlation degree between the spatiotemporal features and the topological features, and embed the multimodal alignment data into the tunnel semantic space based on the feature correlation degree;

[0119] It's important to note that spatiotemporal features refer to characteristics extracted from time-series data that simultaneously reflect the changes in patterns over time and space. Examples include the motion trajectory of a vehicle in a video and the propagation patterns of vibration sensor signals at different locations within a tunnel. Topological features, on the other hand, are features extracted based on the spatial topological relationships of a sensor network, describing the correlation and propagation characteristics of data within the sensor network structure. Feature correlation is an indicator that quantifies the strength of the correlation between spatiotemporal features and topological features, measuring whether the spatiotemporal features are consistent with their topological features on the sensor network. The tunnel semantic space is a high-dimensional vector space where each vector represents the abstract semantics of a specific state or event in the tunnel. Embedding data into this space means transforming it from raw data features into a more semantically meaningful abstract representation.

[0120] Understandably, the approach involves first extracting two types of features in parallel from multimodal aligned data. For spatiotemporal feature extraction, a network designed for sequence data is used to capture local spatiotemporal patterns and long-term temporal dependencies within the data. For topological feature extraction, a graph neural network is used, treating the sensor network as a graph with sensors as nodes and spatial topological relationships as edges. The graph neural network learns a deep representation of each node's features by aggregating information from neighboring nodes, thereby capturing the correlation characteristics of the data within the network structure. Then, the feature correlation between these two types of features is calculated, and weights are generated through a learnable attention mechanism. Specifically, this can be tailored based on spatiotemporal features... Topological features Attention weights are calculated as keys and values. This can be considered as a feature correlation:

[0121]

[0122] in, The extracted spatiotemporal feature vector, The extracted topological feature vectors, These are trainable linear transformation matrices, used to generate queries and keys from input features, respectively. The dimension of the key vector is used to scale the dot product result and stabilize the training process. The calculated attention weights represent the degree of contribution of each part of the topological features to the current spatiotemporal features when generating the final fused features.

[0123] The weights reflect which parts of the spatiotemporal features are highly correlated with the topological features. Finally, based on the calculated feature correlation, the spatiotemporal features and topological features are fused to form a unified joint feature representation. This joint feature is then mapped into the tunnel semantic space through an encoder. In this space, data points with similar tunnel states cluster together, while data points with different states are separated, thus providing semantically rich input for subsequent complexity and security assessments.

[0124] In one feasible implementation, the multimodal alignment data includes visual data, point cloud data, and temporal data; the spatiotemporal features include spatiotemporal motion features, spatial geometric features, and dynamic change features; and the step of extracting the spatiotemporal features and topological features of the multimodal alignment data and calculating the feature correlation degree between the spatiotemporal features and the topological features includes:

[0125] Three-dimensional spatiotemporal feature extraction is performed on the visual data to generate the spatiotemporal motion features;

[0126] Spatial structure analysis is performed on the point cloud data to generate the spatial geometric features;

[0127] The time-series data is dynamically evolved to generate the dynamic change features;

[0128] Analyze the adjacency relationships of each sensor in the spatial topology;

[0129] A sensor topology graph is constructed using sensor nodes as vertices and the adjacency relationships as edges, and topological features are generated based on the sensor topology graph.

[0130] A feature interaction matrix is ​​established based on the spatiotemporal motion features, spatial geometric features, dynamic change features, and topological features;

[0131] Singular value decomposition is performed on the feature interaction matrix, and the maximum singular value is taken as the feature correlation degree.

[0132] It should be noted that visual data refers to two-dimensional images or video sequences captured by cameras inside the tunnel, containing rich texture, color, and brightness information. Point cloud data, typically collected by sensors such as LiDAR, is a collection of massive three-dimensional coordinate points (X, Y, Z), and may also contain information such as intensity, accurately describing the geometry of the tunnel's internal surface. Temporal data refers to a sequence of observations from environmental sensors arranged chronologically, reflecting the dynamic changes in tunnel environmental parameters. Spatiotemporal motion features are extracted from visual data and provide quantitative information that simultaneously characterizes the trajectory of a target in space and its temporal motion patterns, such as features extracted using 3D CNNs. Spatial geometric features are extracted from point cloud data and are used to describe the static three-dimensional spatial attributes such as tunnel structure, object shape, and surface curvature. Dynamic change features are extracted from temporal data and are used to describe the patterns and laws governing the evolution of physical parameters over time.

[0133] In practical implementation, extracting heterogeneous features from multimodal data and calculating their correlation involves processing data from different modalities separately to obtain their unique features, and then using graph models and linear algebra methods to explore the deep correlations between these features. (See reference...) Figure 3 , Figure 3 This is a technical flowchart. Visual data is processed using a 3D convolutional neural network. 3D convolutional kernels can perform convolution operations simultaneously across the spatiotemporal dimensions (length, width, and time) of the video, effectively capturing moving targets and their motion patterns, and outputting spatiotemporal motion features. The point cloud data is processed using unordered point set processing to learn the spatial distribution of the point cloud and extract spatial geometric features that characterize the tunnel structure and equipment shape. For time-series data, the LSTM model captures long-term dependencies within the temporal context, thereby extracting dynamic features representing trends, periodicity, and outliers. Simultaneously, the spatial topological relationships are analyzed to clarify the adjacency relationships between sensors. Then, a sensor topology graph is constructed, using sensors as vertices and adjacency relationships as edges. This graph and sensor data are input into a graph neural network, and through message passing and aggregation between neighboring nodes, topological features embedding network structure information are generated. Then, all features are combined to construct a feature interaction matrix M, where each row of the matrix represents an eigenvector. Finally, singular value decomposition is performed on matrix M, decomposing any matrix into the product of three matrices. The singular values ​​obtained after decomposition... Arranged from largest to smallest, the largest singular value This represents the strength of the principal components of matrix M, that is, the strength of the most dominant association pattern among all features. When performing singular value decomposition on the feature interaction matrix M, its expression is:

[0134]

[0135] Among them, the feature interaction matrix It is The matrix is ​​denoted by n, where n is the number of feature types and m is the dimension of the feature vector; It is The unitary matrix, whose column vectors are called left singular vectors, forms an orthogonal basis for the eigenspace; It is A diagonal matrix, elements on the diagonal It is a singular value, and satisfies ; It is unitary matrix The transpose of , whose row vectors are called right singular vectors, forms an orthogonal basis for the sample space.

[0136] In one feasible implementation, the step of embedding the multimodal aligned data into the tunnel semantic space based on the feature correlation degree includes:

[0137] Based on the aforementioned feature correlation, each modal feature of the multimodal aligned data is weighted by correlation to obtain the correlation embedding feature;

[0138] Semantic projection is performed on the associated embedding features to obtain semantic projection features;

[0139] The semantic similarity matrix is ​​determined based on the semantic basis vectors and the semantic projection features;

[0140] When the maximum similarity of the semantic similarity matrix is ​​greater than the similarity threshold, the semantic projection features are embedded into the tunnel semantic space.

[0141] It should be noted that associative embedding features refer to the new feature vector obtained by weighting and fusing multiple original modal features using feature association as weights. Semantic projection features refer to the feature representation obtained by mapping associative embedding features from a high-dimensional feature space to a low-dimensional or structured space through a transformation. The semantic similarity matrix is ​​used to quantify the degree of similarity between semantic projection features and predefined semantic basis vectors. Each element of this matrix represents the similarity score between the current data state and a certain baseline state.

[0142] In the specific implementation, the key process of transforming the weighted fusion-based deep features into semantic identifiers of tunnel states with clear physical meaning is illustrated in the feature embedding flowchart below. Figure 4As shown. First, the association embedding features are received. These features are a unified representation that integrates multimodal information and is weighted by association degree. Then, semantic projection is achieved through linear transformation, transforming the fused features from a high-dimensional feature space to a lower-dimensional, more interpretable tunnel semantic space. The projected result is called the semantic projection feature, which represents the coordinates of the original data in the semantic space. The semantic projection formula can be:

[0143]

[0144] in, The associated embedded feature vector is the input to the projection layer. The trainable projection weight matrix defines the transformation direction from the feature space to the semantic space. is a trainable bias vector. This is the semantic projection feature vector obtained after projection.

[0145] Then, the semantic similarity matrix is ​​calculated. Prior to this, a semantic basis vector library is pre-constructed, where each basis vector represents an idealized feature prototype of a typical tunnel operating state. By calculating the similarity between the semantic projection features and each semantic basis vector in the library, a list of similarity scores is generated, which can then be used to construct the semantic similarity matrix. (Calculation of semantic projection features) With the semantic basis vectors The formula for calculating semantic similarity is:

[0146]

[0147] in, The semantic basis vector library There are basis vectors. This is the dot product operation for vectors. Each corresponds to a module. The calculated cosine similarity value ranges from [-1, 1], with larger values ​​indicating greater similarity.

[0148] The system then decides whether to embed the current data point into the semantic space. The decision is based on whether the maximum similarity value in the semantic similarity matrix exceeds a preset similarity threshold. If it exceeds the threshold, it indicates that the current state highly matches a known typical state. In this case, the semantic projection feature is formally embedded into the tunnel semantic space and classified into the state category corresponding to the maximum similarity. If it is less than the threshold, it indicates that the current state is not similar enough to any known typical states, potentially representing an unknown or extreme anomaly. In this case, the system may mark it as "unknown" or trigger a special alarm, rather than including it in the existing semantic space classification.

[0149] Step S30: Determine the environmental complexity of the tunnel semantic space, generate modal weight coefficients based on the environmental complexity, and weight the multimodal alignment data based on the modal weight coefficients to obtain a joint feature vector;

[0150] It should be noted that environmental complexity refers to the degree of chaos, uncertainty, or abnormality in the tunnel's operational state as represented in the tunnel semantic space. Modal weight coefficients are weight vectors dynamically generated based on the current environmental complexity, with each weight value corresponding to a modal of data. The weight coefficients determine the reliability and contribution of each modal of data in the final decision. The joint feature vector refers to a unified and comprehensive feature vector formed by weighting and fusing the multimodal aligned data or its high-level features from various modalities using the modal weight coefficients.

[0151] Understandably, the first step is to determine the environmental complexity of the tunnel semantic space. This can be achieved in three ways. First, based on uncertainty metrics: this is measured by calculating the entropy of the predicted probability distribution. Higher entropy indicates greater uncertainty about the current state and a more complex environment. Second, based on clustering distance: this involves calculating the distance between the current semantic projection feature and the semantic basis vectors. If the distance to all basis vectors is large, or the distance to multiple basis vectors is small, it indicates that the current state is atypical or mixed, with high complexity. Third, based on feature dispersion: this involves analyzing the distribution of new data points in the semantic space. If the points are scattered and lack clear clustering, it indicates a complex environment. Then, the system generates modality weight coefficients based on the environmental complexity. For low-complexity environments, average weights or static weights pre-set based on the inherent reliability of the modalities can be used. For high-complexity environments, weights need to be adjusted, reducing the weights of potentially affected modalities and increasing the weights of more reliable modalities. For example, a fire inside a tunnel producing a large amount of smoke can be considered high complexity, leading to a severe deterioration in the image quality of the visual modality. At this point, the weighting coefficients of visual modalities are automatically reduced, while the weighting coefficients of temporal modalities, such as temperature and smoke concentration sensors, and point cloud modalities are increased. Finally, the multimodal aligned data is weighted based on the modal weighting coefficients to obtain a joint feature vector. This weighting process is typically performed at the feature level. Specifically, the feature vector extracted from each modality is multiplied by its corresponding modal weighting coefficient, and then all weighted feature vectors are concatenated or summed to form a unified joint feature vector.

[0152] In one feasible implementation, the steps of determining the environmental complexity of the tunnel semantic space, generating modal weight coefficients based on the environmental complexity, and weighting the multimodal alignment data based on the modal weight coefficients to obtain a joint feature vector include:

[0153] The information entropy of the embedding features in the tunnel semantic space is determined, and the environmental complexity is determined based on the environmental interference factor and the information entropy.

[0154] Based on the complexity weight mapping relationship, determine the modal weight coefficients corresponding to the environmental complexity;

[0155] The modal weight coefficients are optimized based on the environment adaptation loss function to obtain the environment adaptation weight coefficients.

[0156] The multimodal alignment data is weighted based on the environment adaptation weight coefficient to obtain a joint feature vector.

[0157] It should be noted that information entropy is used to quantify the degree of uncertainty or disorder. It is the entropy value calculated after probabilistically estimating the feature distribution of all data points in the semantic space. The higher the entropy value, the more dispersed and uncertain the state distribution of the data points, and the more complex the semantic space; the lower the entropy value, the more concentrated and certain the states, and the simpler the semantic space. The environmental interference factor represents the intensity of interference from the external environment to a specific sensor mode. The complexity weight mapping relationship is a predefined function or mapping table, whose input is the environmental complexity and output is a set of initial mode weight coefficients.

[0158] In practical implementation, the information entropy H of the embedded features in the tunnel semantic space is calculated. This is typically done by discretizing the semantic space or using kernel density estimation to estimate the probability density function of the feature distribution, and then substituting this into the entropy calculation formula. The formula for calculating information entropy is:

[0159]

[0160] in, Let be a random variable representing all embedded features in the tunnel semantic space. This represents the total number of intervals (or categories) that divide the feature space. For the eigenvalue to fall within the th The estimated probability within each interval. The higher the value of the calculated information entropy, the greater the complexity.

[0161] Simultaneously, a predefined environmental interference factor I is retrieved from the database or configuration. This environmental interference factor can be a vector, representing the degree of interference to different modalities. The environmental complexity C is determined jointly by the information entropy and the environmental interference factor, for example, through the formula: To determine the environmental complexity. Hyperparameters are used to reconcile the importance of the two factors.

[0162] Then, based on the calculated environment complexity, a preset complexity weight mapping relationship is queried to obtain a set of initial modality weight coefficients. These initial modality weight coefficients are first estimates based on prior knowledge. Then, an environment adaptation loss function is introduced to optimize the initial weights, ensuring that the joint feature vector obtained after weighted fusion performs optimally on subsequent tasks. The environment adaptation loss function is:

[0163]

[0164] in, This is the total loss function; For mission losses; These are the modal weight coefficients to be optimized; These are the initial weights; is the strength coefficient of the regularization term.

[0165] Then, the optimized weight coefficients are used to perform weighted fusion on the multimodal aligned data to generate the final joint feature vector.

[0166] Step S40: Perform a security assessment on the joint feature vector to determine the tunnel security assessment status.

[0167] It should be noted that the tunnel safety assessment status refers to a comprehensive and conclusive level or category of overall tunnel operational safety derived from the analysis and judgment of the fused joint feature vector. It is usually not a single numerical value, but a semantically clear status identifier, such as different levels like "normal," "risk warning," "local danger," and "overall danger," used to directly guide operation and maintenance decisions.

[0168] Understandably, a pre-trained classifier can be used to classify the joint feature vector, determining the complex nonlinear mapping between the joint feature vector and different safety states. A deep learning model can then output a probability distribution to represent the likelihood that the current state belongs to each safety level. Alternatively, threshold rules can be determined for each dimension of the joint feature vector, such as "IF temperature feature > θ1 AND smoke concentration feature > θ2 THEN state = fire hazard." A conclusion can then be drawn. A confidence threshold is set for this conclusion; when the highest probability exceeds this threshold, the result is adopted, and the tunnel safety assessment state is output.

[0169] In one feasible implementation, the step of performing a security assessment on the joint feature vector to determine the tunnel security assessment status includes:

[0170] The joint feature vector is input into the safety assessment model to obtain the quantitative value of the tunnel safety level;

[0171] Based on the quantified value of the tunnel safety level, spatiotemporal anomaly detection is performed to determine the type of abnormal event;

[0172] The abnormal event types are associated with the tunnel safety level to determine the tunnel safety assessment status.

[0173] It should be noted that the tunnel safety level quantification value is a numerical value or numerical vector output by the safety assessment model to represent the degree of tunnel safety. For example, it can be a score between 0 and 1 (the closer to 1, the more dangerous), or a vector representing the probability of different safety levels (normal probability, warning probability, danger probability). Anomaly event type refers to the specific event category identified through anomaly detection technology, such as "fire," "flooding," "structural cracks," and "traffic congestion." The type identifies the nature of the anomaly.

[0174] In the specific implementation, the joint feature vector representing the overall operational status of the tunnel is first input into a pre-trained safety assessment model. This model is typically a deep learning network or an ensemble learning model. The model's output is a quantified value of the tunnel safety level, which is either a regression score or a vector containing probabilities for each safety level. Then, spatiotemporal anomaly detection is triggered in parallel, analyzing the spatiotemporal dimensions of each feature in the joint feature vector to determine the presence of abnormal local events and identify the specific types of abnormal events. These abnormal event types are then correlated with the tunnel safety level to determine the tunnel safety assessment status.

[0175] In one feasible implementation, prior to the step of performing a security assessment on the joint feature vector to determine the tunnel security assessment status, the method further includes:

[0176] When a conflict is detected in the multimodal data in the joint feature vector, conflict features are generated based on the conflicting multimodal data.

[0177] The conflict features are reconstructed based on a generative adversarial network to obtain reconstructed features;

[0178] Determine the spatial topological consistency between the reconstructed features and the conflicting features;

[0179] When the spatial topology consistency verification is successful, the joint feature vector is updated based on the reconstructed features;

[0180] When the spatial topology consistency verification fails, the redundant data of the redundant nodes is obtained, and the joint feature vector is updated based on the redundant data.

[0181] In the implementation, the joint feature vector is continuously monitored. When there are significant contradictions in the information expressed by different modalities within the joint feature vector, a conflict is identified, and a conflict feature vector highlighting this contradiction is generated. Then, a generative adversarial network (GAN) is used to resolve the conflict. The generator uses the conflict feature as input to generate reconstructed features, and the discriminator judges the input features. Through this adversarial training, more realistic reconstructed features are obtained. Next, the spatial topological consistency between the reconstructed features and the conflict features is determined. If the spatial topological consistency verification is successful, it indicates that the generated reconstructed features are not only reasonable at the data level but also reliable at the physical space level. The reconstructed features are then used to replace the conflicting parts of the original joint feature vector, completing the update. If the spatial topological consistency verification fails, a redundancy backup mechanism is activated to obtain redundant data from redundant nodes, and the joint feature vector is updated based on the redundant data.

[0182] This embodiment provides a deep learning-based method for collaborative management of multimodal data in smart tunnels. By determining the spatial topological relationships of multiple heterogeneous sensors within the tunnel, multimodal data is mapped to a unified coordinate system of a digital twin and timestamped to form aligned multimodal data. Spatiotemporal and topological features are extracted, and feature correlation is calculated. Based on this, the data is embedded into the tunnel's semantic space to assess environmental complexity and generate modal weight coefficients. A security assessment is performed on the weighted joint feature vector to determine the tunnel's safety status. Coordinate unification is applied to the multimodal data, aligning it spatially, followed by timestamping to align it temporally, resulting in aligned multimodal data that ensures accuracy. The spatiotemporal and topological features of the aligned multimodal data are then extracted, and weight coefficients are determined based on their correlation. Finally, a security assessment is performed using these weight coefficients and the joint feature vector to determine the tunnel's safety status. This overcomes the limitations of single-modal monitoring in complex environments.

[0183] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the deep learning-based smart tunnel multimodal data collaborative management method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0184] This application also provides a deep learning-based intelligent tunnel multimodal data collaborative management device, please refer to... Figure 5 The deep learning-based smart tunnel multimodal data collaborative management device includes:

[0185] Data processing module 10 is used to determine the spatial topological relationship of the multi-source heterogeneous sensor cluster in the tunnel, map the multimodal data to the unified coordinate system of the tunnel digital twin based on the spatial topological relationship to obtain multimodal mapping data, and perform timestamp compensation on the multimodal mapping data to obtain multimodal aligned data.

[0186] Feature extraction module 20 is used to extract the spatiotemporal features and topological features of the multimodal alignment data, calculate the feature correlation degree between the spatiotemporal features and the topological features, and embed the multimodal alignment data into the tunnel semantic space based on the feature correlation degree;

[0187] Data association module 30 is used to determine the environmental complexity of the tunnel semantic space, generate modal weight coefficients based on the environmental complexity, and weight the multimodal alignment data based on the modal weight coefficients to obtain a joint feature vector;

[0188] The safety assessment module 40 is used to perform a safety assessment on the joint feature vector and determine the tunnel safety assessment status.

[0189] In one feasible implementation, the data processing module 10 is further configured to acquire the installation location coordinates, spatial orientation angle and sensing range parameters of each sensor in the multi-source heterogeneous sensor cluster in the tunnel, and construct a spatial topology matrix based on the installation location coordinates, the spatial orientation angle and the sensing range parameters;

[0190] A unified coordinate system for the tunnel digital twin is determined, and a transformation matrix from the local coordinate system of each sensor to the unified coordinate system is generated based on the spatial topology relation matrix.

[0191] Based on the transformation matrix, the multimodal data is mapped to the unified coordinate system of the tunnel digital twin, generating multimodal mapping data with spatial location labels.

[0192] In one feasible implementation, the data processing module 10 is further used to synchronize and unify the acquisition time reference of each sensor in the multi-source heterogeneous sensor cluster based on a hardware clock.

[0193] For wired transmission sensors, the wired transmission delay is determined based on the communication distance and signal propagation speed;

[0194] For wireless transmission sensors, the wireless transmission delay is determined based on the timestamp handshake protocol.

[0195] The original timestamp of the multimodal mapping data is determined based on the acquisition time reference, and time compensation is performed on the original timestamp according to the wired transmission delay or the wireless transmission delay to obtain time-aligned data.

[0196] The time-aligned data is reordered in chronological order to obtain multimodal aligned data.

[0197] In one feasible implementation, the data processing module 10 is further configured to extract the overlapping sensing area of ​​adjacent sensors in the spatial topology, and when an event occurs in the overlapping sensing area, compare the event trigger time recorded with the event trigger time to determine the time deviation.

[0198] When the time deviation is greater than the time deviation threshold, dynamic compensation parameters are generated based on the time deviation.

[0199] The target transmission delay is determined from the wired transmission delay or the wireless transmission delay based on the data transmission type of the sensor.

[0200] Time-aligned data is obtained by performing time compensation on the original timestamp based on the dynamic compensation parameters and the target transmission delay.

[0201] In one feasible implementation, the feature extraction module 20 is further configured to perform three-dimensional spatiotemporal feature extraction on the visual data to generate the spatiotemporal motion features;

[0202] Spatial structure analysis is performed on the point cloud data to generate the spatial geometric features;

[0203] The time-series data is dynamically evolved to generate the dynamic change features;

[0204] Analyze the adjacency relationships of each sensor in the spatial topology;

[0205] A sensor topology graph is constructed using sensor nodes as vertices and the adjacency relationships as edges, and topological features are generated based on the sensor topology graph.

[0206] A feature interaction matrix is ​​established based on the spatiotemporal motion features, spatial geometric features, dynamic change features, and topological features;

[0207] Singular value decomposition is performed on the feature interaction matrix, and the maximum singular value is taken as the feature correlation degree.

[0208] In one feasible implementation, the feature extraction module 20 is further configured to perform correlation weighting on each modal feature of the multimodal alignment data based on the feature correlation degree to obtain the correlation embedding feature;

[0209] Semantic projection is performed on the associated embedding features to obtain semantic projection features;

[0210] The semantic similarity matrix is ​​determined based on the semantic basis vectors and the semantic projection features;

[0211] When the maximum similarity of the semantic similarity matrix is ​​greater than the similarity threshold, the semantic projection features are embedded into the tunnel semantic space.

[0212] In one feasible implementation, the data association module 30 is further configured to determine the information entropy of the embedding features of the tunnel semantic space, and determine the environmental complexity based on the environmental interference factor and the information entropy.

[0213] Based on the complexity weight mapping relationship, determine the modal weight coefficients corresponding to the environmental complexity;

[0214] The modal weight coefficients are optimized based on the environment adaptation loss function to obtain the environment adaptation weight coefficients.

[0215] The multimodal alignment data is weighted based on the environment adaptation weight coefficient to obtain a joint feature vector.

[0216] In one feasible implementation, the safety assessment module 40 is further configured to input the joint feature vector into the safety assessment model to obtain a quantitative value of the tunnel safety level;

[0217] Based on the quantified value of the tunnel safety level, spatiotemporal anomaly detection is performed to determine the type of abnormal event;

[0218] The abnormal event types are associated with the tunnel safety level to determine the tunnel safety assessment status.

[0219] In one feasible implementation, the security assessment module 40 is further configured to generate conflict features based on the conflicting multimodal data when a conflict is detected in the joint feature vector.

[0220] The conflict features are reconstructed based on a generative adversarial network to obtain reconstructed features;

[0221] Determine the spatial topological consistency between the reconstructed features and the conflicting features;

[0222] When the spatial topology consistency verification is successful, the joint feature vector is updated based on the reconstructed features;

[0223] When the spatial topology consistency verification fails, the redundant data of the redundant nodes is obtained, and the joint feature vector is updated based on the redundant data.

[0224] The deep learning-based smart tunnel multimodal data collaborative management device provided in this application, employing the deep learning-based smart tunnel multimodal data collaborative management method described in the above embodiments, can solve the technical problem that single-modal monitoring methods are unable to comprehensively reflect the tunnel's operating status in complex environments. Compared with the prior art, the beneficial effects of the deep learning-based smart tunnel multimodal data collaborative management device provided in this application are the same as those of the deep learning-based smart tunnel multimodal data collaborative management method provided in the above embodiments, and other technical features in the deep learning-based smart tunnel multimodal data collaborative management device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0225] This application provides a deep learning-based smart tunnel multimodal data collaborative management device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the deep learning-based smart tunnel multimodal data collaborative management method in Embodiment 1 above.

[0226] The following is for reference. Figure 6 This document illustrates a structural schematic diagram of a deep learning-based smart tunnel multimodal data collaborative management device suitable for implementing embodiments of this application. The deep learning-based smart tunnel multimodal data collaborative management device in this application embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 6 The deep learning-based smart tunnel multimodal data collaborative management device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0227] like Figure 6As shown, the deep learning-based smart tunnel multimodal data collaborative management device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to programs stored in ROM (Read Only Memory) 1002 or programs loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the deep learning-based smart tunnel multimodal data collaborative management device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, LCDs (Liquid Crystal Displays), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the deep learning-based smart tunnel multimodal data collaborative management device to wirelessly or wiredly communicate with other devices to exchange data. Although the figure shows a deep learning-based smart tunnel multimodal data collaborative management device with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0228] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0229] The deep learning-based smart tunnel multimodal data collaborative management device provided in this application employs the deep learning-based smart tunnel multimodal data collaborative management method described in the above embodiments, and can solve the technical problems of deep learning-based smart tunnel multimodal data collaborative management. Compared with the prior art, the beneficial effects of the deep learning-based smart tunnel multimodal data collaborative management device provided in this application are the same as those of the deep learning-based smart tunnel multimodal data collaborative management method provided in the above embodiments, and other technical features in this deep learning-based smart tunnel multimodal data collaborative management device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0230] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0231] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0232] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the deep learning-based smart tunnel multimodal data collaborative management method in the above embodiments.

[0233] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory or Flash Memory), optical fibers, CD-ROM (CD-Read Only Memory), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0234] The aforementioned computer-readable storage medium may be included in the deep learning-based smart tunnel multimodal data collaborative management device; or it may exist independently and not assembled into the deep learning-based smart tunnel multimodal data collaborative management device.

[0235] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by the deep learning-based smart tunnel multimodal data collaborative management device, the deep learning-based smart tunnel multimodal data collaborative management device: determines the spatial topological relationship of the multi-source heterogeneous sensor cluster in the tunnel, maps the multimodal data to the unified coordinate system of the tunnel digital twin based on the spatial topological relationship to obtain multimodal mapping data, and performs timestamp compensation on the multimodal mapping data to obtain multimodal aligned data;

[0236] Extract the spatiotemporal and topological features of the multimodal alignment data, calculate the feature correlation degree between the spatiotemporal features and the topological features, and embed the multimodal alignment data into the tunnel semantic space based on the feature correlation degree;

[0237] The environmental complexity of the tunnel semantic space is determined, modal weight coefficients are generated based on the environmental complexity, and the multimodal alignment data is weighted based on the modal weight coefficients to obtain a joint feature vector;

[0238] A safety assessment is performed on the joint feature vector to determine the tunnel safety assessment status.

[0239] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including LAN (Local Area Network) or WAN (Wide Area Network)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0240] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0241] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0242] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described deep learning-based smart tunnel multimodal data collaborative management method, and can solve the technical problem of deep learning-based smart tunnel multimodal data collaborative management. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the deep learning-based smart tunnel multimodal data collaborative management method provided in the above embodiments, and will not be repeated here.

[0243] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the deep learning-based smart tunnel multimodal data collaborative management method described above.

[0244] The computer program product provided in this application can solve the technical problem of multimodal data collaborative management in smart tunnels based on deep learning. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the deep learning-based multimodal data collaborative management method for smart tunnels provided in the above embodiments, and will not be repeated here.

[0245] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for collaborative management of multimodal data in smart tunnels based on deep learning, characterized in that, The deep learning-based smart tunnel multimodal data collaborative management method includes: The spatial topology of the multi-source heterogeneous sensor cluster in the tunnel is determined. Based on the spatial topology, the multimodal data is mapped to the unified coordinate system of the tunnel digital twin to obtain multimodal mapping data. The multimodal mapping data is then timestamped to obtain multimodal aligned data. Extract the spatiotemporal and topological features of the multimodal alignment data, calculate the feature correlation degree between the spatiotemporal features and the topological features, and embed the multimodal alignment data into the tunnel semantic space based on the feature correlation degree; The environmental complexity of the tunnel semantic space is determined, modal weight coefficients are generated based on the environmental complexity, and the multimodal alignment data is weighted based on the modal weight coefficients to obtain a joint feature vector; A security assessment is performed on the joint feature vector to determine the tunnel security assessment status; The multimodal alignment data includes visual data, point cloud data, and temporal data; the spatiotemporal features include spatiotemporal motion features, spatial geometric features, and dynamic change features; the step of extracting the spatiotemporal features and topological features of the multimodal alignment data, and calculating the feature correlation degree between the spatiotemporal features and the topological features includes: Three-dimensional spatiotemporal feature extraction is performed on the visual data to generate the spatiotemporal motion features; Spatial structure analysis is performed on the point cloud data to generate the spatial geometric features; The time-series data is dynamically evolved to generate the dynamic change features; Analyze the adjacency relationships of each sensor in the spatial topology; A sensor topology graph is constructed using sensor nodes as vertices and the adjacency relationships as edges, and topological features are generated based on the sensor topology graph. A feature interaction matrix is ​​established based on the spatiotemporal motion features, spatial geometric features, dynamic change features, and topological features; Singular value decomposition is performed on the feature interaction matrix, and the maximum singular value is taken as the feature correlation degree.

2. The method as described in claim 1, characterized in that, The steps of determining the spatial topology of the multi-source heterogeneous sensor cluster within the tunnel, and mapping the multimodal data to the unified coordinate system of the tunnel digital twin based on the spatial topology to obtain the multimodal mapping data include: The installation location coordinates, spatial orientation angle, and sensing range parameters of each sensor in the multi-source heterogeneous sensor cluster in the tunnel are obtained, and a spatial topology matrix is ​​constructed based on the installation location coordinates, spatial orientation angle, and sensing range parameters. A unified coordinate system for the tunnel digital twin is determined, and a transformation matrix from the local coordinate system of each sensor to the unified coordinate system is generated based on the spatial topology relation matrix. Based on the transformation matrix, the multimodal data is mapped to the unified coordinate system of the tunnel digital twin, generating multimodal mapping data with spatial location labels.

3. The method as described in claim 1, characterized in that, The step of performing timestamp compensation on the multimodal mapping data to obtain multimodal aligned data includes: The acquisition time reference of each sensor in the multi-source heterogeneous sensor cluster is unified based on hardware clock synchronization. For wired transmission sensors, the wired transmission delay is determined based on the communication distance and signal propagation speed; For wireless transmission sensors, the wireless transmission delay is determined based on the timestamp handshake protocol. The original timestamp of the multimodal mapping data is determined based on the acquisition time reference, and time compensation is performed on the original timestamp according to the wired transmission delay or the wireless transmission delay to obtain time-aligned data. The time-aligned data is reordered in chronological order to obtain multimodal aligned data.

4. The method as described in claim 3, characterized in that, The step of time-compensating the original timestamp based on the wired transmission delay or the wireless transmission delay to obtain time-aligned data further includes: Extract the overlapping sensing areas of adjacent sensors in the spatial topology. When an event occurs within the overlapping sensing area, compare the event trigger time with the recorded time to determine the time deviation. When the time deviation is greater than the time deviation threshold, dynamic compensation parameters are generated based on the time deviation. The target transmission delay is determined from the wired transmission delay or the wireless transmission delay based on the data transmission type of the sensor. Time-aligned data is obtained by performing time compensation on the original timestamp based on the dynamic compensation parameters and the target transmission delay.

5. The method as described in claim 1, characterized in that, The step of embedding the multimodal aligned data into the tunnel semantic space based on the feature correlation includes: Based on the aforementioned feature correlation, each modal feature of the multimodal aligned data is weighted by correlation to obtain the correlation embedding feature; Semantic projection is performed on the associated embedding features to obtain semantic projection features; The semantic similarity matrix is ​​determined based on the semantic basis vectors and the semantic projection features; When the maximum similarity of the semantic similarity matrix is ​​greater than the similarity threshold, the semantic projection features are embedded into the tunnel semantic space.

6. The method as described in claim 1, characterized in that, The steps of determining the environmental complexity of the tunnel semantic space, generating modal weight coefficients based on the environmental complexity, and weighting the multimodal alignment data based on the modal weight coefficients to obtain a joint feature vector include: The information entropy of the embedding features in the tunnel semantic space is determined, and the environmental complexity is determined based on the environmental interference factor and the information entropy. Based on the complexity weight mapping relationship, determine the modal weight coefficients corresponding to the environmental complexity; The modal weight coefficients are optimized based on the environment adaptation loss function to obtain the environment adaptation weight coefficients. The multimodal alignment data is weighted based on the environment adaptation weight coefficient to obtain a joint feature vector.

7. The method as described in claim 1, characterized in that, The step of performing a security assessment on the joint feature vector to determine the tunnel security assessment status includes: The joint feature vector is input into the safety assessment model to obtain the quantitative value of the tunnel safety level; Based on the quantified value of the tunnel safety level, spatiotemporal anomaly detection is performed to determine the type of abnormal event; The abnormal event types are associated with the tunnel safety level to determine the tunnel safety assessment status.

8. The method as described in claim 7, characterized in that, Before the step of performing a security assessment on the joint feature vector to determine the tunnel security assessment status, the method further includes: When a conflict is detected in the multimodal data in the joint feature vector, conflict features are generated based on the conflicting multimodal data. The conflict features are reconstructed based on a generative adversarial network to obtain reconstructed features; Determine the spatial topological consistency between the reconstructed features and the conflicting features; When the spatial topology consistency verification is successful, the joint feature vector is updated based on the reconstructed features; When the spatial topology consistency verification fails, the redundant data of the redundant nodes is obtained, and the joint feature vector is updated based on the redundant data.

9. A smart tunnel multimodal data collaborative management device based on deep learning, characterized in that, The deep learning-based smart tunnel multimodal data collaborative management device includes: The data processing module is used to determine the spatial topological relationship of the multi-source heterogeneous sensor cluster in the tunnel, map the multimodal data to the unified coordinate system of the tunnel digital twin based on the spatial topological relationship to obtain multimodal mapping data, and perform timestamp compensation on the multimodal mapping data to obtain multimodal aligned data. The feature extraction module is used to extract the spatiotemporal features and topological features of the multimodal alignment data, calculate the feature correlation degree between the spatiotemporal features and the topological features, and embed the multimodal alignment data into the tunnel semantic space based on the feature correlation degree. The data association module is used to determine the environmental complexity of the tunnel semantic space, generate modal weight coefficients based on the environmental complexity, and weight the multimodal aligned data based on the modal weight coefficients to obtain a joint feature vector; The safety assessment module is used to perform a safety assessment on the joint feature vector and determine the tunnel safety assessment status. The multimodal alignment data includes visual data, point cloud data, and temporal data; the spatiotemporal features include spatiotemporal motion features, spatial geometric features, and dynamic change features; the step of extracting the spatiotemporal features and topological features of the multimodal alignment data, and calculating the feature correlation degree between the spatiotemporal features and the topological features includes: Three-dimensional spatiotemporal feature extraction is performed on the visual data to generate the spatiotemporal motion features; Spatial structure analysis is performed on the point cloud data to generate the spatial geometric features; The time-series data is dynamically evolved to generate the dynamic change features; Analyze the adjacency relationships of each sensor in the spatial topology; A sensor topology graph is constructed using sensor nodes as vertices and the adjacency relationships as edges, and topological features are generated based on the sensor topology graph. A feature interaction matrix is ​​established based on the spatiotemporal motion features, spatial geometric features, dynamic change features, and topological features; Singular value decomposition is performed on the feature interaction matrix, and the maximum singular value is taken as the feature correlation degree.

Images