Deep learning-based smart tunnel multi-modal data collaborative management method and device

Abstract

The invention discloses an intelligent tunnel multi-modal data collaborative management method and device based on deep learning, and relates to the technical field of artificial intelligence, and the method comprises the steps: determining the spatial topological relation of a multi-source heterogeneous sensor in a tunnel, mapping the multi-modal data to a digital twin unified coordinate system, and carrying out the timestamp compensation, thereby obtaining a digital twin unified coordinate system; and multi-modal alignment data is formed. And extracting space-time and topological features of the data, calculating a feature correlation degree, embedding the data into a tunnel semantic space according to the feature correlation degree, evaluating environmental complexity and generating a modal weight coefficient. And carrying out safety evaluation on the joint feature vector obtained by weighting, and judging the safety state of the tunnel, thereby overcoming the limitation of single-mode monitoring in a complex environment.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method and device for collaborative management of multimodal data in smart tunnels based on deep learning. Background Art

[0002] Tunnels are critical infrastructure in transportation networks, used to overcome geographical barriers and improve travel efficiency. Therefore, the safety of tunnel environments is crucial. Currently, monitoring tunnel operating conditions typically relies on a large number of inspectors, but this approach fails to promptly identify safety hazards within the tunnel. Some monitoring systems collect data through sensors or cameras, but these methods are often single-modal and struggle to fully reflect the tunnel's actual operating status in complex environments.

[0003] The above content is only used to assist in understanding the technical solution of the present invention and does not constitute an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a method and device for collaborative management of multimodal data in smart tunnels based on deep learning, aiming to solve the technical problem that a single modality monitoring method is difficult to fully reflect the operation status of the tunnel in a complex environment.

[0005] To achieve the above objectives, this application provides a method for collaborative management of multimodal data in smart tunnels based on deep learning, which includes: Determine the spatial topological relationship of the multi-source heterogeneous sensor cluster in the tunnel, map the multimodal data to the unified coordinate system of the tunnel digital twin based on the spatial topological relationship to obtain multimodal mapping data, and perform timestamp compensation on the multimodal mapping data to obtain multimodal alignment data; extracting spatiotemporal features and topological features of the multimodal alignment data, calculating feature correlations between the spatiotemporal features and the topological features, and embedding the multimodal alignment data into a tunnel semantic space based on the feature correlations; determining an environmental complexity of the tunnel semantic space, generating a modal weight coefficient according to the environmental complexity, and weighting the multimodal alignment data based on the modal weight coefficient to obtain a joint feature vector; A security assessment is performed on the joint feature vector to determine a tunnel security assessment state.

[0006] In one embodiment, the steps of determining the spatial topological relationship of the multi-source heterogeneous sensor cluster in the tunnel, mapping the multimodal data to a unified coordinate system of the tunnel digital twin based on the spatial topological relationship, and obtaining the multimodal mapping data include: Obtaining the installation location coordinates, spatial orientation angle, and sensing range parameters of each sensor in the multi-source heterogeneous sensor cluster in the tunnel, and constructing a spatial topological relationship matrix based on the installation location coordinates, the spatial orientation angle, and the sensing range parameters; Determine a unified coordinate system for the tunnel digital twin, and generate a transformation matrix from the local coordinate system of each sensor to the unified coordinate system based on the spatial topological relationship matrix; Based on the transformation matrix, the multimodal data is mapped to the unified coordinate system of the tunnel digital twin to generate multimodal mapping data with spatial position labels.

[0007] In one embodiment, the step of performing timestamp compensation on the multimodal mapping data to obtain multimodal alignment data includes: Unifying the acquisition time base of each sensor in the multi-source heterogeneous sensor cluster based on hardware clock synchronization; For wired transmission sensors, the wired transmission delay is determined based on the communication distance and signal propagation speed; For wireless transmission sensors, the wireless transmission delay is determined based on the timestamp handshake protocol; Determining an original timestamp of the multimodal mapping data based on the acquisition time reference, and performing time compensation on the original timestamp according to the wired transmission delay or the wireless transmission delay to obtain time-aligned data; The time-aligned data are reordered in time sequence to obtain multimodal aligned data.

[0008] In one embodiment, the step of performing time compensation on the original timestamp according to the wired transmission delay or the wireless transmission delay to obtain time-aligned data further includes: Extracting overlapping sensing areas of adjacent sensors in the spatial topological relationship, and when an event occurs in the overlapping sensing area, comparing the event triggering time recorded at the time to determine a time deviation; When the time deviation is greater than a time deviation threshold, generating a dynamic compensation parameter according to the time deviation; determining a target transmission delay from the wired transmission delay or the wireless transmission delay according to a data transmission type of the sensor; Time compensation is performed on the original timestamp based on the dynamic compensation parameter and the target transmission delay to obtain time-aligned data.

[0009] In one embodiment, the multimodal alignment data includes visual data, point cloud data, and time series data, the spatiotemporal features include spatiotemporal motion features, spatial geometric features, and dynamic change features, and the steps of extracting the spatiotemporal features and topological features of the multimodal alignment data and calculating the feature correlation between the spatiotemporal features and the topological features include: Performing three-dimensional spatiotemporal feature extraction on the visual data to generate the spatiotemporal motion feature; Performing spatial structure analysis on the point cloud data to generate the spatial geometric features; Dynamically evolving the time series data to generate the dynamic change feature; Analyzing the adjacency relationship of each sensor in the spatial topological relationship; Constructing a sensor topology graph with sensor nodes as vertices and the adjacency relationships as edges, and generating topological features based on the sensor topology graph; Establishing a feature interaction matrix based on the spatiotemporal motion feature, the spatial geometric feature, the dynamic change feature and the topological feature; Perform singular value decomposition on the feature interaction matrix and take the maximum singular value as the feature correlation degree.

[0010] In one embodiment, the step of embedding the multimodal alignment data into the tunnel semantic space based on the feature correlation includes: Performing correlation weighting on each modal feature of the multimodal alignment data based on the feature correlation to obtain a correlation embedding feature; Performing semantic projection on the associated embedded features to obtain semantic projection features; Determine a semantic similarity matrix based on the semantic basis vector and the semantic projection feature; When the maximum similarity of the semantic similarity matrix is ​​greater than a similarity threshold, the semantic projection feature is embedded in the tunnel semantic space.

[0011] In one embodiment, the steps of determining the environmental complexity of the tunnel semantic space, generating a modal weight coefficient according to the environmental complexity, and weighting the multimodal alignment data based on the modal weight coefficient to obtain a joint feature vector include: Determining information entropy of embedded features of the tunnel semantic space, and determining environmental complexity based on environmental interference factors and the information entropy; Determining a modal weight coefficient corresponding to the complexity of the environment based on the complexity-weight mapping relationship; Optimizing the modal weight coefficient based on the environment adaptation loss function to obtain the environment adaptation weight coefficient; The multimodal alignment data is weighted based on the environment adaptation weight coefficient to obtain a joint feature vector.

[0012] In one embodiment, the step of performing security assessment on the joint feature vector to determine the tunnel security assessment status includes: Inputting the joint feature vector into a safety assessment model to obtain a quantitative value of the tunnel safety level; Performing spatiotemporal anomaly detection based on the tunnel safety level quantified value to determine the type of abnormal event; The abnormal event type is associated with the tunnel safety level to determine the tunnel safety assessment status.

[0013] In one embodiment, before the step of performing security assessment on the joint feature vector and determining the tunnel security assessment status, the step further includes: When a conflict is detected in the multimodal data in the joint feature vector, a conflict feature is generated based on the conflicting multimodal data; Reconstructing the conflicting features based on a generative adversarial network to obtain reconstructed features; Determining the spatial topological consistency between the reconstructed feature and the conflicting feature; When the spatial topology consistency verification is successful, updating the joint feature vector based on the reconstructed feature; When the spatial topology consistency verification fails, redundant data of redundant nodes is obtained, and the joint feature vector is updated based on the redundant data.

[0014] In addition, to achieve the above objectives, this application also proposes a smart tunnel multimodal data collaborative management device based on deep learning, which includes: A data processing module is used to determine the spatial topological relationship of the multi-source heterogeneous sensor cluster in the tunnel, map the multimodal data to the unified coordinate system of the tunnel digital twin based on the spatial topological relationship to obtain multimodal mapping data, and perform timestamp compensation on the multimodal mapping data to obtain multimodal alignment data; a feature extraction module, configured to extract spatiotemporal features and topological features of the multimodal alignment data, calculate a feature correlation between the spatiotemporal features and the topological features, and embed the multimodal alignment data into a tunnel semantic space based on the feature correlation; a data association module, configured to determine the environmental complexity of the tunnel semantic space, generate a modal weight coefficient according to the environmental complexity, and weight the multimodal alignment data based on the modal weight coefficient to obtain a joint feature vector; The security assessment module is used to perform a security assessment on the joint feature vector and determine the tunnel security assessment status.

[0015] In addition, to achieve the above-mentioned purpose, the present application also proposes a smart tunnel multimodal data collaborative management device based on deep learning, and the smart tunnel multimodal data collaborative management device based on deep learning includes: a memory, a processor, and a computer program stored on the memory and runnable on the processor. The computer program is configured to implement the steps of the smart tunnel multimodal data collaborative management method based on deep learning as described above.

[0016] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the deep learning-based smart tunnel multimodal data collaborative management method as described above are implemented.

[0017] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the deep learning-based smart tunnel multimodal data collaborative management method as described above.

[0018] The present application provides a method for collaborative management of multimodal data in a smart tunnel based on deep learning. By determining the spatial topological relationship of multi-source heterogeneous sensors in the tunnel, the multimodal data is mapped to the unified coordinate system of the digital twin, and timestamp compensation is performed to form multimodal aligned data. The spatiotemporal and topological features are extracted, the feature correlation is calculated, and the data is embedded in the tunnel semantic space accordingly. The environmental complexity is evaluated and the modal weight coefficient is generated. A security assessment is performed on the weighted joint feature vector to determine the safety status of the tunnel. The coordinates of the multimodal data are unified, the data is aligned in space, and then timestamp compensation is performed to align the data in time. Based on this, multimodal aligned data is obtained to ensure the accuracy of the data. The spatiotemporal and topological features of the multimodal aligned data are then extracted, and the weight coefficient is determined according to its correlation. The weight coefficient and the joint feature vector are then used to perform a security assessment to determine the safety status of the tunnel. This overcomes the limitations of single-modality monitoring in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 This is a flowchart of Example 1 of the method for collaborative management of multimodal data in smart tunnels based on deep learning in this application; Figure 2 This is an abstract diagram of the topological relationship of an embodiment of the method for collaborative management of multimodal data in a smart tunnel based on deep learning in this application; Figure 3This is a technical flow chart of an embodiment of a method for collaborative management of multimodal data in a smart tunnel based on deep learning in this application; Figure 4 This is a feature embedding flowchart of an embodiment of the deep learning-based smart tunnel multimodal data collaborative management method; Figure 5 This is a schematic diagram of the module structure of the deep learning-based intelligent tunnel multimodal data collaborative management device according to an embodiment of the present application; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the deep learning-based smart tunnel multimodal data collaborative management method in the embodiment of the present application.

[0022] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0023] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0024] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0025] The main solution of the embodiment of the present application is to: determine the spatial topological relationship of the multi-source heterogeneous sensor cluster in the tunnel, map the multimodal data to the unified coordinate system of the tunnel digital twin based on the spatial topological relationship to obtain multimodal mapping data, and perform timestamp compensation on the multimodal mapping data to obtain multimodal alignment data; extracting spatiotemporal features and topological features of the multimodal alignment data, calculating feature correlations between the spatiotemporal features and the topological features, and embedding the multimodal alignment data into a tunnel semantic space based on the feature correlations; determining an environmental complexity of the tunnel semantic space, generating a modal weight coefficient according to the environmental complexity, and weighting the multimodal alignment data based on the modal weight coefficient to obtain a joint feature vector; A security assessment is performed on the joint feature vector to determine a tunnel security assessment state.

[0026] Tunnels are critical infrastructure in transportation networks, bridging geographical barriers and improving travel efficiency. Therefore, the safety of tunnel environments is crucial. Currently, monitoring tunnel operating conditions typically relies on a large number of inspectors, but this approach fails to promptly identify safety hazards within the tunnel. Some monitoring systems collect data through sensors or cameras, but these methods are often single-modal and struggle to fully reflect the tunnel's actual operating status in complex environments.

[0027] The present application provides a solution, which determines the spatial topological relationship of multi-source heterogeneous sensors in the tunnel, thereby mapping the multimodal data to the unified coordinate system of the digital twin, and performing timestamp compensation to form multimodal aligned data. The spatiotemporal and topological features are extracted, the feature correlation is calculated, and the data is embedded in the tunnel semantic space accordingly, the environmental complexity is evaluated and the modal weight coefficient is generated. A security assessment is performed on the weighted joint feature vector to determine the safety status of the tunnel, the coordinates of the multimodal data are unified, the data are aligned in space, and then timestamp compensation is performed to align the data in time. Based on this, multimodal aligned data is obtained to ensure the accuracy of the data, and then the spatiotemporal and topological features of the multimodal aligned data are extracted, and the weight coefficient is determined according to its correlation. The weight coefficient and the joint feature vector are then used to perform a security assessment to determine the safety status of the tunnel. This overcomes the limitations of single modality monitoring in complex environments.

[0028] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program execution capabilities, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device capable of implementing the above functions, such as a deep learning-based smart tunnel multimodal data collaborative management device, etc. This embodiment does not specifically limit this. The following uses the deep learning-based smart tunnel multimodal data collaborative management device as an example to illustrate this embodiment and the following embodiments.

[0029] All actions of acquiring signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0030] The present application embodiment provides a method for collaborative management of multimodal data in a smart tunnel based on deep learning, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the deep learning-based smart tunnel multimodal data collaborative management method of this application.

[0031] In this embodiment, the deep learning-based smart tunnel multimodal data collaborative management method includes steps S10 to S40: Step S10: determining the spatial topological relationship of the multi-source heterogeneous sensor cluster in the tunnel, mapping the multimodal data to the unified coordinate system of the tunnel digital twin based on the spatial topological relationship to obtain multimodal mapping data, and performing timestamp compensation on the multimodal mapping data to obtain multimodal alignment data; It should be noted that the multi-source heterogeneous sensor cluster in a tunnel refers to a collection of sensors deployed in the tunnel environment, including various types (such as cameras, lidar, temperature and humidity sensors, stress and strain gauges, etc.), with different operating principles and data formats. The spatial topological relationship describes the relative spatial position and orientation relationship between the sensors in the cluster and is the basis for data fusion and coordinate unification. Multimodal data refers to data in different forms collected by heterogeneous sensors, such as image point clouds, video sequences, and scalar measurements (temperature, pressure). The tunnel digital twin is a high-fidelity virtual model of the tunnel's physical entity that can map and reflect the tunnel's status in real time. The unified coordinate system is a unique, global coordinate system defined for the digital twin. All sensor data must be converted to this coordinate system for consistent analysis and presentation. Multimodal mapping data refers to data that has been converted from raw multimodal data to a unified coordinate system based on the spatial topological relationship of its sensors. Multimodal alignment data refers to data that has been spatially aligned and further timestamped to eliminate time delays between different sensors, achieving complete synchronization in both temporal and spatial dimensions.

[0032] It is understandable that after acquiring the multimodal mapping data, it is necessary to perform timestamp compensation on it to eliminate the time synchronization problem caused by factors such as sensor sampling frequency and transmission delay. To this end, this embodiment uses linear interpolation processing and assumes that the time series of a certain sensor data stream is , and its corresponding collection value is , the target unified time series is , calculated by linear interpolation in Compensation value at time , the processing formula is:

[0033] in, , Represents the data value at the same timestamp after compensation. This process processes all sensor data streams one by one to obtain time-consistent multimodal alignment data.

[0034] In a feasible embodiment, the steps of determining the spatial topological relationship of the multi-source heterogeneous sensor cluster in the tunnel, mapping the multimodal data to a unified coordinate system of the tunnel digital twin based on the spatial topological relationship, and obtaining the multimodal mapping data include: Obtaining the installation location coordinates, spatial orientation angle, and sensing range parameters of each sensor in the multi-source heterogeneous sensor cluster in the tunnel, and constructing a spatial topological relationship matrix based on the installation location coordinates, the spatial orientation angle, and the sensing range parameters; Determine a unified coordinate system for the tunnel digital twin, and generate a transformation matrix from the local coordinate system of each sensor to the unified coordinate system based on the spatial topological relationship matrix; Based on the transformation matrix, the multimodal data is mapped to the unified coordinate system of the tunnel digital twin to generate multimodal mapping data with spatial position labels.

[0035] It should be noted that the installation position coordinates refer to the actual physical position of the sensor in the tunnel (such as latitude and longitude or local coordinates), the spatial orientation angle describes the yaw, pitch and roll directions of its installation, and the perception range parameters include its effective monitoring distance and field of view angle; the spatial topological relationship matrix is ​​a matrix that integrates all sensor spatial attributes and is used to express the relationship between their relative positions and perception ranges; the transformation matrix is ​​a mathematical matrix that realizes the rotation and translation transformation from the sensor's local coordinate system to the unified coordinate system of the digital twin.

[0036] In the specific implementation, by collecting the installation position, orientation and perception parameters of each sensor, a topological relationship matrix can be constructed with sensors as nodes and spatial relationships as edges. The abstract diagram of the topological relationship is as follows: Figure 2 As shown. For example, consider three nodes: Sensor A, Sensor B, and Sensor C, forming three nodes: Node1, Node2, and Node3. The topological relationship between them is connected by three edges: Edge1→2, Edge2→3, and Edge1→3. For example, Edge1→2 represents the relative topological relationship between Node1 and Node2. Node1's position (Pos) is (x1, y1, z1), with an orientation angle (Angle) of θ1. Node2's position (Pos) is (x2, y2, z2), with an orientation angle (Angle) of θ2. Node3's position (Pos) is (x3, y3, z3), with an orientation angle (Angle) of θ3. For Edge1→2, Edge2→3, and Edge1→3, respectively, Edge1→2 represents the distance between Node1 and Node2, with the relative orientation angle (Δθ) representing the relative orientation between θ1 and θ2. Edge1→3 represents the distance between Node1 and Node3, with the relative orientation angle (Δθ) representing the relative orientation between θ1 and θ3. The distance corresponding to Edge2→3 is the distance between Node2 and Node3, and the relative orientation angle Δθ is the relative orientation of θ2 and θ3. Based on this matrix and the definition of the unified coordinate system, the rigid body transformation principle is used to calculate the transformation matrix from each sensor's local system to the unified coordinate system (usually including rotation and translation components). This matrix is ​​then used to transform the original multimodal data to a unified spatial reference, while a location label is added to each data frame, ultimately forming multimodal mapping data with spatial identification. The mapping of sensor data from the local coordinate system to the unified coordinate system can be expressed as:

[0037] Among them, P local Represents the coordinates of the data point in the sensor's local coordinate system; is the rotation matrix calculated from the sensor space orientation angle, is the translation vector composed of the sensor installation position coordinates; P global is the mapping coordinate of the data point in the unified coordinate system.

[0038] In a feasible implementation manner, the step of performing timestamp compensation on the multimodal mapping data to obtain multimodal alignment data includes: Unifying the acquisition time base of each sensor in the multi-source heterogeneous sensor cluster based on hardware clock synchronization; For wired transmission sensors, the wired transmission delay is determined based on the communication distance and signal propagation speed; For wireless transmission sensors, the wireless transmission delay is determined based on the timestamp handshake protocol; Determining an original timestamp of the multimodal mapping data based on the acquisition time reference, and performing time compensation on the original timestamp according to the wired transmission delay or the wireless transmission delay to obtain time-aligned data; The time-aligned data are reordered in time sequence to obtain multimodal aligned data.

[0039] It should be noted that the acquisition time base refers to a unified time reference established for all sensors through a hardware clock synchronization protocol (such as PTP or GPS), ensuring consistent time starting points and timing accuracy across all devices. Wired transmission sensors refer to sensors that transmit data via wired media such as cables or optical fibers, and their transmission latency is relatively fixed. Wired transmission latency refers to the time lag incurred during data transmission over wired media, which primarily depends on the communication distance and the signal propagation speed in the medium. Wireless transmission sensors refer to sensors that transmit data wirelessly, such as through radio waves, and are susceptible to environmental interference and experience significant latency fluctuations. Wireless transmission latency refers to the time delay introduced by data transmission, processing, and queuing in wireless channels. It is typically dynamically estimated using a timestamp handshake protocol. The original timestamp refers to the time stamp applied to the data at the time of acquisition, based on the sensor's local clock, before any transmission or synchronization compensation.

[0040] In the specific implementation, firstly, the acquisition time base of all sensors is unified through the hardware clock synchronization protocol (such as PTP or GPS) to establish a globally consistent time reference. For wired transmission sensors, the transmission delay According to the communication distance and the speed of signal propagation in the medium For wireless sensors, the transmission delay is dynamically estimated based on the timestamps of the messages sent and received, using a timestamp handshake protocol (such as IEEE 802.11 TSFs or NTP-like mechanisms) to exchange messages. After obtaining the original timestamp of each data packet, the corresponding delay amount is selected for compensation according to its transmission mode, that is, Get preliminary time-aligned data, is the original timestamp, is the target transmission delay. Finally, all data are timestamped after compensation. Perform global reordering to generate a multi-modal aligned data stream that is strictly arranged in chronological order, providing a consistent data foundation for subsequent fusion and analysis. The wired transmission delay calculation formula is:

[0041] in, is the wired transmission delay, d is the communication distance, that is, the physical length of the data cable, and v is the signal propagation speed in a specific medium.

[0042] The wireless transmission delay estimation formula is:

[0043] in, is the estimated one-way wireless transmission delay, The timestamp of the host sending the request message, The timestamp of the slave receiving the request message, The timestamp of the response message sent by the slave, Timestamp when the host receives the response message.

[0044] In a feasible implementation manner, the step of performing time compensation on the original timestamp according to the wired transmission delay or the wireless transmission delay to obtain time-aligned data further includes: Extracting overlapping sensing areas of adjacent sensors in the spatial topological relationship, and when an event occurs in the overlapping sensing area, comparing the event triggering time recorded at the time to determine a time deviation; When the time deviation is greater than a time deviation threshold, generating a dynamic compensation parameter according to the time deviation; determining a target transmission delay from the wired transmission delay or the wireless transmission delay according to a data transmission type of the sensor; Time compensation is performed on the original timestamp based on the dynamic compensation parameter and the target transmission delay to obtain time-aligned data.

[0045] It should be noted that the overlapping sensing area of ​​adjacent sensors refers to the physical area within the tunnel space covered by the monitoring ranges of two or more adjacent sensors (such as the field of view of a camera or the point cloud coverage of a LiDAR sensor). Events occurring within this area can be captured simultaneously by multiple sensors. An event refers to a physical change or abnormal state in the tunnel environment that can be sensed by sensors, such as the passage of a vehicle, the presence of smoke, changes in lighting, or structural vibration. The event trigger time refers to the original timestamp carried in each data frame when the same event is detected and recorded by different sensors. Due to differences in sensor clocks and transmission paths, the same event may have multiple trigger times. The deviation threshold is used to determine whether time deviation is significant. When the deviation is below this threshold, it is considered normal fluctuation and no additional compensation is required. When the deviation exceeds this threshold, it is considered an anomaly and dynamic compensation is required. The dynamic compensation parameter is an adjustment calculated in real time to correct for abnormal time deviations (such as clock drift). The target transmission delay is based on the sensor's data transmission type.

[0046] In the specific implementation, the overlapping sensing areas between adjacent sensors are first calculated based on the known spatial topological relationship. When a key event (such as a vehicle passing) occurs in the area, the system will extract the event trigger time recorded by each sensor from the data stream of the multiple sensors that monitor the event. By comparing these timestamps, the time deviation Δt between them can be calculated. The time deviation is used to determine whether there is an unexpected synchronization error and to determine the preset time deviation threshold. When the time deviation is greater than the time deviation threshold, it indicates that fixed transmission delay compensation alone is not enough to eliminate the error. At this time, dynamic compensation parameters are generated based on the time deviation. Then, the corresponding target transmission delay τ is selected according to the data transmission type of the sensor. The final time compensation amount is determined by the fixed target transmission delay and the dynamic compensation parameter. The compensation formula is:

[0047] Based on this, not only the fixed transmission delay is compensated, but also the random errors and tiny clock drifts that are difficult to accurately model in advance are dynamically corrected, thereby obtaining time-aligned data with higher accuracy.

[0048] Step S20: extracting spatiotemporal features and topological features of the multimodal alignment data, calculating a feature correlation between the spatiotemporal features and the topological features, and embedding the multimodal alignment data into a tunnel semantic space based on the feature correlation. It should be noted that spatiotemporal features refer to features extracted from time series data that can simultaneously reflect how patterns change over time and space. For example, the motion trajectory of a vehicle in a video or the propagation pattern of vibration sensor signals at different locations in a tunnel. Topological features refer to features extracted based on the spatial topological relationship of a sensor network, describing the association and propagation characteristics of data in the sensor network structure. Feature correlation is an indicator that quantifies the strength of the correlation between spatiotemporal features and topological features, and measures whether spatiotemporal features are consistent with their topological features on the sensor network. The tunnel semantic space is a high-dimensional vector space in which each vector represents the abstract semantics of a specific state or event in the tunnel. Embedding data in this space means converting it from raw data features into an abstract representation with more semantic meaning.

[0049] It can be understood that two types of features are first extracted in parallel from the multimodal aligned data. When extracting spatiotemporal features, a network designed for sequence data is used for processing to capture local spatiotemporal patterns and long-term temporal dependencies in the data. When extracting topological features, a graph neural network is used to regard the sensor network as a graph, with sensors as nodes and spatial topological relationships as edges. The graph neural network learns the deep representation of each node feature by aggregating information from adjacent nodes, thereby capturing the correlation characteristics of the data in the network structure. The feature correlation between the two types of features is then calculated, and the weights are generated through a learnable attention mechanism. Specifically, the spatiotemporal features can be used to determine the feature association between the two types of features. , through query (Query), topological features As the key and value, the calculated attention weight It can be regarded as a feature correlation:

[0050] in, is the extracted spatiotemporal feature vector, is the extracted topological feature vector, is a trainable linear transformation matrix, which is used to generate query and key from input features. is the dimension of the key vector, used to scale the dot product results and stabilize the training process. The calculated attention weight indicates the contribution of each part of the topological feature to the current spatiotemporal feature when generating the final fusion feature.

[0051] The weights reflect which parts of the spatiotemporal features are highly correlated with the topological features. Finally, based on the calculated feature correlation, the spatiotemporal and topological features are fused to form a unified joint feature representation. This joint feature is then mapped into the tunnel semantic space through an encoder. In this space, data points with similar tunnel states are clustered together, while data points with different states are separated, providing semantically rich input for subsequent complexity and security assessments.

[0052] In a feasible implementation, the multimodal alignment data includes visual data, point cloud data, and time series data, the spatiotemporal features include spatiotemporal motion features, spatial geometric features, and dynamic change features, and the steps of extracting the spatiotemporal features and topological features of the multimodal alignment data and calculating the feature correlation between the spatiotemporal features and the topological features include: Performing three-dimensional spatiotemporal feature extraction on the visual data to generate the spatiotemporal motion feature; Performing spatial structure analysis on the point cloud data to generate the spatial geometric features; Dynamically evolving the time series data to generate the dynamic change feature; Analyzing the adjacency relationship of each sensor in the spatial topological relationship; Constructing a sensor topology graph with sensor nodes as vertices and the adjacency relationships as edges, and generating topological features based on the sensor topology graph; Establishing a feature interaction matrix based on the spatiotemporal motion feature, the spatial geometric feature, the dynamic change feature and the topological feature; Perform singular value decomposition on the feature interaction matrix and take the maximum singular value as the feature correlation degree.

[0053] It should be noted that visual data refers to two-dimensional images or video sequences captured by cameras inside the tunnel, containing rich texture, color, and brightness information. Point cloud data, typically collected by sensors such as lidar, consists of a large number of three-dimensional coordinate points (X, Y, Z) and may also contain information such as intensity, accurately describing the geometry of the tunnel's internal surface. Time series data refers to a chronological sequence of observations from environmental sensors, reflecting the dynamic evolution of tunnel environmental parameters. Spatiotemporal motion features, such as those extracted by 3D CNN, are extracted from visual data and can simultaneously characterize the target's trajectory in space and its temporal motion pattern. Spatial geometric features, extracted from point cloud data, are used to quantitatively describe static three-dimensional spatial properties such as tunnel structure, object shape, and surface curvature. Dynamic change features, extracted from time series data, are used to quantitatively describe the patterns and regularities in the temporal evolution of physical parameters.

[0054] In the specific implementation, extracting heterogeneous features from multimodal data and calculating their correlation is to process data of different modalities separately to obtain their unique features, and then use graphical models and linear algebra methods to explore the deep correlation between these features. Figure 3 , Figure 3 The following is a technical flow chart. Visual data is processed using a 3D convolutional neural network. The 3D convolution kernel can simultaneously perform convolution operations on the spatiotemporal dimensions (length, width, and time) of the video, effectively capturing moving targets and their motion patterns, and outputting spatiotemporal motion features. Process the point cloud data into disordered point sets, learn the spatial distribution of the point cloud, and extract the spatial geometric features that characterize the tunnel structure and equipment shape. For time series data, the LSTM model is used to capture long-term dependencies in the time context, thereby extracting dynamic change features that represent trends, periodicity, and anomalies. At the same time, the spatial topological relationship is analyzed to clarify the adjacency relationship between each sensor. Subsequently, a sensor topology graph is constructed with the sensor as the vertex and the adjacency relationship as the edge. This graph and sensor data are input into the graph neural network, and through message passing and aggregation between neighboring nodes, topological features embedded in the network structure information are generated. Then all the features are combined to build the feature interaction matrix M, where each row of the matrix represents an eigenvector. Finally, the matrix M is decomposed into the product of three matrices by singular value decomposition. Arrange from large to small, the largest singular value It represents the strength of the principal component of the matrix M, that is, the strength of the most dominant correlation pattern among all features. When performing singular value decomposition on the feature interaction matrix M, its expression is:

[0055] Among them, the feature interaction matrix is a , n is the number of feature types, and m is the dimension of the feature vector; is a The unitary matrix of , whose column vectors are called left singular vectors, constitutes a set of orthogonal bases of the feature space; is a A diagonal matrix with elements on the diagonal is a singular value and satisfies ; is a The unitary matrix The transpose of , whose row vectors are called right singular vectors, constitutes a set of orthogonal bases of the sample space.

[0056] In a feasible implementation manner, the step of embedding the multimodal alignment data into the tunnel semantic space based on the feature correlation includes: Performing correlation weighting on each modal feature of the multimodal alignment data based on the feature correlation to obtain a correlation embedding feature; Performing semantic projection on the associated embedded features to obtain semantic projection features; Determine a semantic similarity matrix based on the semantic basis vector and the semantic projection feature; When the maximum similarity of the semantic similarity matrix is ​​greater than a similarity threshold, the semantic projection feature is embedded in the tunnel semantic space.

[0057] It should be noted that the associative embedding feature refers to a new feature vector obtained by weighted fusion of the original multiple modal features using feature correlation as weight. The semantic projection feature refers to the feature representation obtained by mapping the associative embedding feature from a high-dimensional feature space to a low-dimensional or structured space through a transformation. The semantic similarity matrix is ​​used to quantify the degree of similarity between the semantic projection feature and the predefined semantic basis vectors. Each element of this matrix represents the similarity score between the current data state and a certain baseline state.

[0058] In the specific implementation, the feature embedding flow chart of the key process of converting the weighted fusion deep features into the tunnel state semantic identification with clear physical meaning is as follows: Figure 4 As shown in Figure 1. First, the associated embedding feature is received. The associated embedding feature is a unified representation that integrates multimodal information and is weighted by the degree of association. Then, semantic projection is achieved through linear transformation, transforming the fused feature from the high-dimensional feature space to a tunnel semantic space with lower dimensions and greater interpretability. The result of the projection is called the semantic projection feature, which is the coordinate of the original data in the semantic space. The semantic projection formula can be:

[0059] in, is the associated embedding feature vector, which is the input of the projection layer. It is a trainable projection weight matrix that defines the transformation direction from feature space to semantic space. is a trainable bias vector. is the semantic projection feature vector obtained after projection.

[0060] Then, the semantic similarity matrix is ​​calculated. Before this, a semantic base vector library is pre-built. Each base vector in the library represents an idealized feature prototype of a typical tunnel operation state. By calculating the similarity between the semantic projection feature and each semantic base vector in the library, a similarity score list is generated, which can be constructed into a semantic similarity matrix. With the Semantic basis vectors The formula for calculating the semantic similarity is:

[0061] in, Semantic basis vector library basis vectors. The dot product operation of the vectors. are the corresponding modules, The calculated cosine similarity value is between -1 and 1. The larger the value, the more similar it is.

[0062] A decision is then made as to whether to embed the current data point into the semantic space. This decision is based on checking whether the maximum similarity value in the semantic similarity matrix exceeds a preset similarity threshold. If it exceeds the threshold, the current state closely matches a known typical state. The semantic projection feature is then formally embedded into the tunnel semantic space and classified into the state category corresponding to the maximum similarity. If it falls below the threshold, the current state is insufficiently similar to any known typical state and may represent an unknown or extremely abnormal situation. In this case, the system may label it as "unknown" or trigger a special alert, rather than including it in the existing semantic space classification.

[0063] Step S30: determining the environmental complexity of the tunnel semantic space, generating a modal weight coefficient according to the environmental complexity, and weighting the multimodal alignment data based on the modal weight coefficient to obtain a joint feature vector; It should be noted that environmental complexity refers to the degree of chaos, uncertainty, or abnormality in the tunnel's operating state as represented in the tunnel's semantic space. The modal weight coefficient is a weight vector dynamically generated based on the current environmental complexity. Each weight value corresponds to a modal data type, and the weight coefficient determines the reliability and contribution of each modal data in the final decision. The joint feature vector is a unified, comprehensive feature vector formed by weighted fusion of the multimodal alignment data or its high-level features from each modality using the modal weight coefficient.

[0064] As you can understand, the complexity of the tunnel semantic space environment must first be determined. This can be achieved in three ways. The first is based on uncertainty measurement: this is measured by calculating the entropy of the predicted probability distribution. A higher entropy value indicates greater uncertainty about the system's current state and a more complex environment. The second is based on cluster distance: this involves calculating the distance between the current semantic projection feature and the semantic basis vectors. If the distance from all basis vectors is very large, or if the distance from multiple basis vectors is close, this indicates that the current state is atypical or mixed, indicating high complexity. The third is based on feature dispersion: this involves analyzing the distribution of new data points in the semantic space. If the points are dispersed and lack clear clusters, this indicates a complex environment. The system then generates modal weight coefficients based on the environmental complexity. For low-complexity environments, average weights can be used, or static weights pre-set based on the inherent reliability of the modalities can be used. For high-complexity environments, weights need to be adjusted, reducing the weight of potentially disruptive modalities and increasing the weight of more reliable modalities. For example, a fire in a tunnel, generating large amounts of smoke, can be considered high complexity, resulting in a significant degradation of image quality in the visual modality. At this point, the weight coefficient of the visual modality is automatically reduced, while the weight coefficients of the temporal modalities, such as temperature and smoke concentration sensors, and the point cloud modality are increased. Finally, the multimodal aligned data is weighted based on the modal weight coefficients to obtain a joint feature vector. This weighting process is usually performed at the feature level. Specifically, the feature vector extracted from each modality is multiplied by the corresponding modal weight coefficient, and then all weighted feature vectors are concatenated or summed to form a unified joint feature vector.

[0065] In a feasible implementation, the steps of determining the environmental complexity of the tunnel semantic space, generating a modal weight coefficient according to the environmental complexity, and weighting the multimodal alignment data based on the modal weight coefficient to obtain a joint feature vector include: Determining information entropy of embedded features of the tunnel semantic space, and determining environmental complexity based on environmental interference factors and the information entropy; Determining a modal weight coefficient corresponding to the complexity of the environment based on the complexity-weight mapping relationship; Optimizing the modal weight coefficient based on the environment adaptation loss function to obtain the environment adaptation weight coefficient; The multimodal alignment data is weighted based on the environment adaptation weight coefficient to obtain a joint feature vector.

[0066] It should be noted that information entropy is used to quantify the degree of uncertainty or confusion. It is calculated by probabilistically estimating the characteristic distribution of all data points in the semantic space. Higher entropy values ​​indicate more dispersed and uncertain data point states, and a more complex semantic space. Lower entropy values ​​indicate more concentrated and certain states, and a simpler semantic space. The environmental interference factor represents the intensity of external environmental interference on a specific sensor modality. The complexity-weight mapping relationship is a predefined function or mapping table whose input is environmental complexity and whose output is a set of initial modal weight coefficients.

[0067] In the specific implementation, the information entropy H of the embedded features in the tunnel semantic space is calculated. This is usually done by discretizing the semantic space or using kernel density estimation to estimate the probability density function of the feature distribution, and then substituting it into the entropy calculation formula. The information entropy calculation formula is:

[0068] in, are random variables of all embedded features in the tunnel semantic space. is the total number of intervals (or categories) into which the feature space is divided. The eigenvalue falls in The estimated probability within the interval. The information entropy is calculated. The larger the value, the higher the complexity.

[0069] At the same time, a predefined environmental interference factor I is obtained from the database or configuration. The environmental interference factor can be a vector that represents the degree of interference to different modes. The environmental complexity C is determined by the information entropy and the environmental interference factor, for example, by the formula: To determine the complexity of the environment. is a hyperparameter that reconciles the importance of the two factors.

[0070] Then, based on the calculated environment complexity, the preset complexity weight mapping relationship is queried to obtain a set of initial modal weight coefficients. The initial modal weight coefficients are the first estimate based on prior knowledge. Then, an environment adaptation loss function is introduced to optimize the initial weights so that the joint feature vector obtained after weighted fusion performs best on subsequent tasks. The environment adaptation loss function is:

[0071] in, is the total loss function; Loss of mission; is the modal weight coefficient to be optimized; is the initial weight; is the strength coefficient of the regularization term.

[0072] The optimized weight coefficients are then used to perform weight fusion on the multimodal alignment data to generate the final joint feature vector.

[0073] Step S40: Perform a security assessment on the joint feature vector to determine the tunnel security assessment status.

[0074] It's important to note that the tunnel safety assessment status refers to a comprehensive, conclusive level or category of overall tunnel operational safety, derived through analysis and judgment of the fused joint feature vector. It's typically not a single numerical value, but rather a semantically clear status indicator, such as "normal," "risk warning," "locally dangerous," or "overall dangerous," which directly guides operational decisions.

[0075] It can be understood that a pre-trained classifier classifies the joint feature vector, determining the complex nonlinear mapping relationship between the joint feature vector and different safety states. A deep learning model then outputs a probability distribution representing the probability of the current state belonging to each safety level. Alternatively, a threshold rule for each dimension of the joint feature vector can be determined based on a rule, such as "IF temperature feature > θ1 AND smoke density feature > θ2 THEN state = fire hazard." This is used to draw a conclusion. A confidence threshold is then set for this conclusion. When the highest probability exceeds this threshold, the result is adopted and the tunnel safety assessment status is output.

[0076] In a feasible implementation manner, the step of performing security assessment on the joint feature vector and determining the tunnel security assessment status includes: Inputting the joint feature vector into a safety assessment model to obtain a quantitative value of the tunnel safety level; Performing spatiotemporal anomaly detection based on the tunnel safety level quantified value to determine the type of abnormal event; The abnormal event type is associated with the tunnel safety level to determine the tunnel safety assessment status.

[0077] It should be noted that the tunnel safety level quantification value is a numerical value or numerical vector output by the safety assessment model that represents the tunnel's safety level. For example, it can be a score between 0 and 1 (the closer to 1, the more dangerous) or a vector representing the probabilities of different safety levels (normal probability, warning probability, and critical probability). The abnormal event type refers to the specific event category identified by anomaly detection technology, such as "fire," "flooding," "structural cracks," and "traffic congestion." The type identifies the nature of the anomaly.

[0078] In a specific implementation, a joint feature vector representing the overall tunnel operating status is first input into a pre-trained safety assessment model. This model is typically a deep learning network or an ensemble learning model. The model outputs a quantified tunnel safety rating, which is a regression score or a vector containing the probabilities of each safety level. Spatiotemporal anomaly detection is then triggered in parallel, analyzing the spatiotemporal dimensions of each feature in the joint feature vector to determine whether there are any abnormal local events and identify the specific type of abnormal event. This abnormal event type is then associated with the tunnel safety level to determine the tunnel safety assessment status.

[0079] In a feasible implementation manner, before the step of performing security assessment on the joint feature vector and determining the tunnel security assessment status, the step further includes: When a conflict is detected in the multimodal data in the joint feature vector, a conflict feature is generated based on the conflicting multimodal data; Reconstructing the conflicting features based on a generative adversarial network to obtain reconstructed features; Determining the spatial topological consistency between the reconstructed feature and the conflicting feature; When the spatial topology consistency verification is successful, updating the joint feature vector based on the reconstructed feature; When the spatial topology consistency verification fails, redundant data of redundant nodes is obtained, and the joint feature vector is updated based on the redundant data.

[0080] In the specific implementation, the joint feature vector is continuously monitored. When there is a significant contradiction between the information expressed by the data of different modalities within the joint feature vector, a conflict is determined to exist, and a conflict feature vector is generated to highlight this contradiction. The conflict is then resolved using a generative adversarial network. The generator uses the conflicting features as input to generate reconstructed features, and the discriminator is used to judge the input features. Through this adversarial training, more realistic reconstructed features are obtained. The spatial topological consistency of the reconstructed features and the conflicting features is then determined. If the spatial topological consistency verification is successful, it indicates that the generated reconstructed features are not only reasonable at the data level but also credible at the physical space level. The reconstructed features are then used to replace the conflicting parts of the original joint feature vector, completing the update. If the spatial topological consistency verification fails, the redundant backup mechanism is activated to obtain redundant data from redundant nodes and update the joint feature vector based on this redundant data.

[0081] This embodiment provides a method for collaborative management of multimodal data in a smart tunnel based on deep learning. By determining the spatial topological relationship of multi-source heterogeneous sensors in the tunnel, the multimodal data is mapped to a unified coordinate system of a digital twin and timestamp compensation is performed to form multimodal aligned data. Its spatiotemporal and topological features are extracted, and the feature correlation is calculated. Based on this, the data is embedded in the tunnel semantic space, the environmental complexity is evaluated, and a modal weight coefficient is generated. A security assessment is performed on the weighted joint feature vector to determine the tunnel's safety status. The multimodal data is coordinate-unified, spatially aligned, and then timestamp compensation is performed to align the data in time. Based on this, multimodal aligned data is obtained to ensure data accuracy. The spatiotemporal and topological features of the multimodal aligned data are then extracted, and the weight coefficient is determined based on its correlation. The weight coefficient and the joint feature vector are then used to perform a security assessment and determine the tunnel's safety status. This overcomes the limitations of single-modality monitoring in complex environments.

[0082] It should be noted that the above examples are only used to understand this application and do not constitute a limitation on the deep learning-based smart tunnel multimodal data collaborative management method of this application. More simple transformations based on this technical concept are all within the scope of protection of this application.

[0083] This application also provides a smart tunnel multimodal data collaborative management device based on deep learning, please refer to Figure 5 The deep learning-based intelligent tunnel multimodal data collaborative management device includes: A data processing module 10 is configured to determine the spatial topological relationship of a multi-source heterogeneous sensor cluster within a tunnel, map the multimodal data to a unified coordinate system of the tunnel digital twin based on the spatial topological relationship, obtain multimodal mapping data, and timestamp-compensate the multimodal mapping data to obtain multimodal alignment data. a feature extraction module 20 for extracting spatiotemporal features and topological features of the multimodal alignment data, calculating a feature correlation between the spatiotemporal features and the topological features, and embedding the multimodal alignment data into a tunnel semantic space based on the feature correlation; a data association module 30 for determining the environmental complexity of the tunnel semantic space, generating a modal weight coefficient according to the environmental complexity, and weighting the multimodal alignment data based on the modal weight coefficient to obtain a joint feature vector; The security assessment module 40 is configured to perform a security assessment on the joint feature vector to determine a tunnel security assessment status.

[0084] In a feasible embodiment, the data processing module 10 is further configured to obtain the installation position coordinates, spatial orientation angle, and sensing range parameters of each sensor in the multi-source heterogeneous sensor cluster in the tunnel, and construct a spatial topological relationship matrix based on the installation position coordinates, the spatial orientation angle, and the sensing range parameters; Determine a unified coordinate system for the tunnel digital twin, and generate a transformation matrix from the local coordinate system of each sensor to the unified coordinate system based on the spatial topological relationship matrix; Based on the transformation matrix, the multimodal data is mapped to the unified coordinate system of the tunnel digital twin to generate multimodal mapping data with spatial position labels.

[0085] In a feasible implementation manner, the data processing module 10 is further configured to unify the acquisition time reference of each sensor in the multi-source heterogeneous sensor cluster based on hardware clock synchronization; For wired transmission sensors, the wired transmission delay is determined based on the communication distance and signal propagation speed; For wireless transmission sensors, the wireless transmission delay is determined based on the timestamp handshake protocol; Determining an original timestamp of the multimodal mapping data based on the acquisition time reference, and performing time compensation on the original timestamp according to the wired transmission delay or the wireless transmission delay to obtain time-aligned data; The time-aligned data are reordered in time sequence to obtain multimodal aligned data.

[0086] In a feasible embodiment, the data processing module 10 is further configured to extract overlapping sensing areas of adjacent sensors in the spatial topological relationship, and when an event occurs in the overlapping sensing area, compare the event triggering time recorded at the time to determine a time deviation; When the time deviation is greater than a time deviation threshold, generating a dynamic compensation parameter according to the time deviation; determining a target transmission delay from the wired transmission delay or the wireless transmission delay according to a data transmission type of the sensor; Time compensation is performed on the original timestamp based on the dynamic compensation parameter and the target transmission delay to obtain time-aligned data.

[0087] In a feasible implementation manner, the feature extraction module 20 is further configured to perform three-dimensional spatiotemporal feature extraction on the visual data to generate the spatiotemporal motion feature; Performing spatial structure analysis on the point cloud data to generate the spatial geometric features; Dynamically evolving the time series data to generate the dynamic change feature; Analyzing the adjacency relationship of each sensor in the spatial topological relationship; Constructing a sensor topology graph with sensor nodes as vertices and the adjacency relationships as edges, and generating topological features based on the sensor topology graph; Establishing a feature interaction matrix based on the spatiotemporal motion feature, the spatial geometric feature, the dynamic change feature and the topological feature; Perform singular value decomposition on the feature interaction matrix and take the maximum singular value as the feature correlation degree.

[0088] In a feasible implementation manner, the feature extraction module 20 is further configured to perform correlation weighting on each modal feature of the multimodal alignment data based on the feature correlation to obtain a correlation embedding feature; Performing semantic projection on the associated embedded features to obtain semantic projection features; Determine a semantic similarity matrix based on the semantic basis vector and the semantic projection feature; When the maximum similarity of the semantic similarity matrix is ​​greater than a similarity threshold, the semantic projection feature is embedded in the tunnel semantic space.

[0089] In a feasible implementation manner, the data association module 30 is further configured to determine the information entropy of the embedded features of the tunnel semantic space, and determine the environmental complexity according to the environmental interference factor and the information entropy; Determining a modal weight coefficient corresponding to the complexity of the environment based on the complexity-weight mapping relationship; Optimizing the modal weight coefficient based on the environment adaptation loss function to obtain the environment adaptation weight coefficient; The multimodal alignment data is weighted based on the environment adaptation weight coefficient to obtain a joint feature vector.

[0090] In a feasible implementation manner, the security assessment module 40 is further configured to input the joint feature vector into a security assessment model to obtain a quantified value of the tunnel security level; Performing spatiotemporal anomaly detection based on the tunnel safety level quantified value to determine the type of abnormal event; The abnormal event type is associated with the tunnel safety level to determine the tunnel safety assessment status.

[0091] In a feasible implementation, the security assessment module 40 is further configured to generate a conflict feature based on the conflicting multimodal data when a conflict is detected in the multimodal data in the joint feature vector; Reconstructing the conflicting features based on a generative adversarial network to obtain reconstructed features; Determining the spatial topological consistency between the reconstructed feature and the conflicting feature; When the spatial topology consistency verification is successful, updating the joint feature vector based on the reconstructed feature; When the spatial topology consistency verification fails, redundant data of redundant nodes is obtained, and the joint feature vector is updated based on the redundant data.

[0092] The deep learning-based intelligent tunnel multimodal data collaborative management device provided in this application adopts the deep learning-based intelligent tunnel multimodal data collaborative management method in the above-mentioned embodiment, which can solve the technical problem that a single modality monitoring method is difficult to fully reflect the tunnel operation status in a complex environment. Compared with the existing technology, the beneficial effects of the deep learning-based intelligent tunnel multimodal data collaborative management device provided in this application are the same as the beneficial effects of the deep learning-based intelligent tunnel multimodal data collaborative management method provided in the above-mentioned embodiment, and the other technical features of the deep learning-based intelligent tunnel multimodal data collaborative management device are the same as the features disclosed in the above-mentioned embodiment method, and will not be repeated here.

[0093] The present application provides a smart tunnel multimodal data collaborative management device based on deep learning. The smart tunnel multimodal data collaborative management device based on deep learning includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the smart tunnel multimodal data collaborative management method based on deep learning in the above-mentioned embodiment one.

[0094] Reference below Figure 6 , which shows a schematic diagram of the structure of a deep learning-based intelligent tunnel multimodal data collaborative management device suitable for implementing the embodiments of the present application. The deep learning-based intelligent tunnel multimodal data collaborative management device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The deep learning-based smart tunnel multimodal data collaborative management device shown is only an example and should not bring any limitations to the functions and scope of use of the embodiments of this application.

[0095] like Figure 6As shown, the deep learning-based intelligent tunnel multimodal data collaborative management device may include a processing device 1001 (e.g., a central processing unit, graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in ROM (Read Only Memory) 1002 or programs loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the deep learning-based intelligent tunnel multimodal data collaborative management device. Processing device 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input device 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output device 1008 including, for example, an LCD (Liquid Crystal Display), speaker, vibrator, etc.; storage device 1003 including, for example, a magnetic tape, hard disk, etc.; and communication device 1009. Communication device 1009 can allow the deep learning-based intelligent tunnel multimodal data collaborative management device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a deep learning-based intelligent tunnel multimodal data collaborative management device with various systems, it should be understood that implementation or presence of all the illustrated systems is not required. More or fewer systems may alternatively be implemented or present.

[0096] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0097] The deep learning-based intelligent tunnel multimodal data collaborative management device provided in this application adopts the deep learning-based intelligent tunnel multimodal data collaborative management method in the above-mentioned embodiment, which can solve the technical problems of deep learning-based intelligent tunnel multimodal data collaborative management. Compared with the prior art, the beneficial effects of the deep learning-based intelligent tunnel multimodal data collaborative management device provided in this application are the same as the beneficial effects of the deep learning-based intelligent tunnel multimodal data collaborative management method provided in the above-mentioned embodiment, and the other technical features of the deep learning-based intelligent tunnel multimodal data collaborative management device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0098] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0099] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0100] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, and the computer-readable program instructions are used to execute the deep learning-based smart tunnel multimodal data collaborative management method in the above-mentioned embodiment.

[0101] The computer-readable storage medium provided herein may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory or Flash memory), optical fiber, CD-ROM (CD-Read Only Memory), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including, but not limited to, wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0102] The above-mentioned computer-readable storage medium may be included in the smart tunnel multimodal data collaborative management device based on deep learning; or it may exist independently without being assembled into the smart tunnel multimodal data collaborative management device based on deep learning.

[0103] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the deep learning-based intelligent tunnel multimodal data collaborative management device, the deep learning-based intelligent tunnel multimodal data collaborative management device: determines the spatial topological relationship of the multi-source heterogeneous sensor cluster in the tunnel, maps the multimodal data to the unified coordinate system of the tunnel digital twin based on the spatial topological relationship to obtain multimodal mapping data, and performs timestamp compensation on the multimodal mapping data to obtain multimodal alignment data; extracting spatiotemporal features and topological features of the multimodal alignment data, calculating feature correlations between the spatiotemporal features and the topological features, and embedding the multimodal alignment data into a tunnel semantic space based on the feature correlations; determining an environmental complexity of the tunnel semantic space, generating a modal weight coefficient according to the environmental complexity, and weighting the multimodal alignment data based on the modal weight coefficient to obtain a joint feature vector; A security assessment is performed on the joint feature vector to determine a tunnel security assessment state.

[0104] The computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a LAN (Local Area Network) or a WAN (Wide Area Network), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0105] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0106] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0107] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned method for collaborative management of multimodal data in smart tunnels based on deep learning. This computer-readable storage medium can address the technical issues surrounding collaborative management of multimodal data in smart tunnels based on deep learning. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are similar to those of the method for collaborative management of multimodal data in smart tunnels based on deep learning provided in the aforementioned embodiments, and are not further elaborated here.

[0108] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned deep learning-based smart tunnel multimodal data collaborative management method.

[0109] The computer program product provided in this application can solve the technical problem of collaborative management of multimodal data in smart tunnels based on deep learning. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the collaborative management method of multimodal data in smart tunnels based on deep learning provided in the above-mentioned embodiment, and will not be repeated here.

[0110] The above are only some embodiments of the present application and are not intended to limit the patent scope of the present application. All equivalent structural transformations made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for collaborative management of multimodal data in smart tunnels based on deep learning, characterized in that: The deep learning-based smart tunnel multimodal data collaborative management method includes: Determine the spatial topological relationship of the multi-source heterogeneous sensor cluster in the tunnel, map the multimodal data to the unified coordinate system of the tunnel digital twin based on the spatial topological relationship to obtain multimodal mapping data, and perform timestamp compensation on the multimodal mapping data to obtain multimodal alignment data; extracting spatiotemporal features and topological features of the multimodal alignment data, calculating feature correlations between the spatiotemporal features and the topological features, and embedding the multimodal alignment data into a tunnel semantic space based on the feature correlations; determining an environmental complexity of the tunnel semantic space, generating a modal weight coefficient according to the environmental complexity, and weighting the multimodal alignment data based on the modal weight coefficient to obtain a joint feature vector; A security assessment is performed on the joint feature vector to determine a tunnel security assessment state.

2. The method according to claim 1, wherein The steps of determining the spatial topological relationship of the multi-source heterogeneous sensor cluster in the tunnel, and mapping the multimodal data to a unified coordinate system of the tunnel digital twin based on the spatial topological relationship to obtain the multimodal mapping data include: Obtaining the installation location coordinates, spatial orientation angle, and sensing range parameters of each sensor in the multi-source heterogeneous sensor cluster in the tunnel, and constructing a spatial topological relationship matrix based on the installation location coordinates, the spatial orientation angle, and the sensing range parameters; Determine a unified coordinate system for the tunnel digital twin, and generate a transformation matrix from the local coordinate system of each sensor to the unified coordinate system based on the spatial topological relationship matrix; Based on the transformation matrix, the multimodal data is mapped to the unified coordinate system of the tunnel digital twin to generate multimodal mapping data with spatial position labels.

3. The method according to claim 1, wherein The step of performing timestamp compensation on the multimodal mapping data to obtain multimodal alignment data includes: Unifying the acquisition time base of each sensor in the multi-source heterogeneous sensor cluster based on hardware clock synchronization; For wired transmission sensors, the wired transmission delay is determined based on the communication distance and signal propagation speed; For wireless transmission sensors, the wireless transmission delay is determined based on the timestamp handshake protocol; Determining an original timestamp of the multimodal mapping data based on the acquisition time reference, and performing time compensation on the original timestamp according to the wired transmission delay or the wireless transmission delay to obtain time-aligned data; The time-aligned data are reordered in time sequence to obtain multimodal aligned data.

4. The method according to claim 3, wherein The step of performing time compensation on the original timestamp according to the wired transmission delay or the wireless transmission delay to obtain time-aligned data further includes: Extracting overlapping sensing areas of adjacent sensors in the spatial topological relationship, and when an event occurs in the overlapping sensing area, comparing the event triggering time recorded at the time to determine a time deviation; When the time deviation is greater than a time deviation threshold, generating a dynamic compensation parameter according to the time deviation; determining a target transmission delay from the wired transmission delay or the wireless transmission delay according to a data transmission type of the sensor; Time compensation is performed on the original timestamp based on the dynamic compensation parameter and the target transmission delay to obtain time-aligned data.

5. The method according to claim 1, wherein The multimodal alignment data includes visual data, point cloud data, and time series data; the spatiotemporal features include spatiotemporal motion features, spatial geometric features, and dynamic change features; and the steps of extracting the spatiotemporal features and topological features of the multimodal alignment data and calculating the feature correlation between the spatiotemporal features and the topological features include: Performing three-dimensional spatiotemporal feature extraction on the visual data to generate the spatiotemporal motion feature; Performing spatial structure analysis on the point cloud data to generate the spatial geometric features; Dynamically evolving the time series data to generate the dynamic change feature; Analyzing the adjacency relationship of each sensor in the spatial topological relationship; Constructing a sensor topology graph with sensor nodes as vertices and the adjacency relationships as edges, and generating topological features based on the sensor topology graph; Establishing a feature interaction matrix based on the spatiotemporal motion feature, the spatial geometric feature, the dynamic change feature and the topological feature; Perform singular value decomposition on the feature interaction matrix and take the maximum singular value as the feature correlation degree.

6. The method according to claim 1, wherein The step of embedding the multimodal alignment data into the tunnel semantic space based on the feature correlation comprises: Performing correlation weighting on each modal feature of the multimodal alignment data based on the feature correlation to obtain a correlation embedding feature; Performing semantic projection on the associated embedded features to obtain semantic projection features; Determine a semantic similarity matrix based on the semantic basis vector and the semantic projection feature; When the maximum similarity of the semantic similarity matrix is ​​greater than a similarity threshold, the semantic projection feature is embedded in the tunnel semantic space.

7. The method according to claim 1, wherein The steps of determining the environmental complexity of the tunnel semantic space, generating a modal weight coefficient according to the environmental complexity, and weighting the multimodal alignment data based on the modal weight coefficient to obtain a joint feature vector include: Determining information entropy of embedded features of the tunnel semantic space, and determining environmental complexity based on environmental interference factors and the information entropy; Determining a modal weight coefficient corresponding to the complexity of the environment based on the complexity-weight mapping relationship; Optimizing the modal weight coefficient based on the environment adaptation loss function to obtain the environment adaptation weight coefficient; The multimodal alignment data is weighted based on the environment adaptation weight coefficient to obtain a joint feature vector.

8. The method according to claim 1, wherein The step of performing security assessment on the joint feature vector to determine the tunnel security assessment status includes: Inputting the joint feature vector into a safety assessment model to obtain a quantitative value of the tunnel safety level; Performing spatiotemporal anomaly detection based on the tunnel safety level quantified value to determine the type of abnormal event; The abnormal event type is associated with the tunnel safety level to determine the tunnel safety assessment status.

9. The method according to claim 8, wherein Before the step of performing security assessment on the joint feature vector and determining the tunnel security assessment status, the method further includes: When a conflict is detected in the multimodal data in the joint feature vector, a conflict feature is generated based on the conflicting multimodal data; Reconstructing the conflicting features based on a generative adversarial network to obtain reconstructed features; Determining the spatial topological consistency between the reconstructed feature and the conflicting feature; When the spatial topology consistency verification is successful, updating the joint feature vector based on the reconstructed feature; When the spatial topology consistency verification fails, redundant data of redundant nodes is obtained, and the joint feature vector is updated based on the redundant data.

10. A smart tunnel multimodal data collaborative management device based on deep learning, characterized in that: The deep learning-based intelligent tunnel multimodal data collaborative management device includes: A data processing module is used to determine the spatial topological relationship of the multi-source heterogeneous sensor cluster in the tunnel, map the multimodal data to the unified coordinate system of the tunnel digital twin based on the spatial topological relationship to obtain multimodal mapping data, and perform timestamp compensation on the multimodal mapping data to obtain multimodal alignment data; a feature extraction module, configured to extract spatiotemporal features and topological features of the multimodal alignment data, calculate a feature correlation between the spatiotemporal features and the topological features, and embed the multimodal alignment data into a tunnel semantic space based on the feature correlation; a data association module, configured to determine the environmental complexity of the tunnel semantic space, generate a modal weight coefficient according to the environmental complexity, and weight the multimodal alignment data based on the modal weight coefficient to obtain a joint feature vector; The security assessment module is used to perform a security assessment on the joint feature vector and determine the tunnel security assessment status.

Images