A tunnel apparent disease detection method and system based on event camera multi-modal fusion and pulse neural network

Abstract

This application discloses a method and system for detecting tunnel surface defects based on multimodal fusion of event cameras and spiking neural networks, relating to the field of tunnel monitoring. The method includes acquiring sensor data within the tunnel; constructing a multi-channel spatiotemporal event tensor based on the asynchronous event stream; performing motion compensation processing on the multi-channel spatiotemporal event tensor using the inertial data to obtain a compensated event tensor; fusing the compensated event tensor with the grayscale image using a dual-stream fusion and super-resolution reconstruction network to obtain a fused feature map; and determining the detection result of tunnel surface defects using a spiking neural network based on the fused feature map. This application can improve the anti-vibration interference capability, the accuracy of identifying minor defects, environmental adaptability, and real-time performance of tunnel surface defect detection.

Description

Technical Field

[0001] This application relates to the field of tunnel monitoring, and in particular to a method and system for detecting apparent defects in tunnels based on event camera multimodal fusion and spiking neural networks. Background Technology

[0002] With the expansion of rail transit networks, routine monitoring of tunnel surface defects (such as cracks and leaks) is crucial. Currently, the mainstream detection methods mainly rely on manual inspections or inspection vehicles based on industrial line scan cameras. However, manual methods are inefficient and pose safety hazards. Machine vision solutions based on traditional frame cameras face severe challenges under high-speed train operation (>80km / h): First, due to limitations in exposure principles, motion blur is easily generated in the dim environment of tunnels, and insufficient dynamic range leads to overexposure of highlights and loss of details in shadows; second, severe train vibrations can cause false image displacement, resulting in a large number of false alarms; third, limited by sensor resolution, it is difficult to simultaneously cover a large field of view and identify small defects (<0.3mm); fourth, full-frame imaging generates massive amounts of redundant background data, placing huge bandwidth pressure on onboard edge computing and making real-time alarms impossible.

[0003] Based on the above problems, there is an urgent need to provide a method for detecting tunnel surface defects that is vibration-resistant, has a high dynamic range, and possesses ultra-high-speed real-time processing capabilities. Summary of the Invention

[0004] The purpose of this application is to provide a method and system for detecting tunnel surface defects based on multimodal fusion of event cameras and spiking neural networks, which can improve the anti-vibration interference capability, the accuracy of identifying minor defects, the environmental adaptability, and the real-time performance of tunnel surface defect detection.

[0005] To achieve the above objectives, this application provides the following solution: In a first aspect, this application provides a method for detecting tunnel surface defects based on event camera multimodal fusion and spiking neural networks, the method comprising: Acquire sensor data inside the tunnel; the sensor data includes asynchronous event streams and grayscale images of the tunnel interior wall acquired using an event camera, as well as inertial data acquired using an auxiliary inertial measurement unit; Construct a multi-channel spatiotemporal event tensor based on the asynchronous event stream; The inertial data is used to perform motion compensation processing on the multi-channel spatiotemporal event tensor to obtain the compensated event tensor; The compensated event tensor and the grayscale image are then fused using a multimodal fusion method based on a two-stream fusion and super-resolution reconstruction network to obtain a fused feature map. Based on the fused feature map, a spiking neural network is used to determine the detection results of tunnel surface defects.

[0006] Secondly, this application provides a tunnel surface defect detection system based on event camera multimodal fusion and spiking neural network. This system is deployed on a rail transit train and includes: A data acquisition subsystem is used to acquire sensor data inside the tunnel; the sensor data includes asynchronous event streams and grayscale images of the tunnel wall acquired using an event camera, as well as inertial data acquired using an auxiliary inertial measurement unit. Memory, used to store computer programs; The processor, which is communicatively connected to the data acquisition subsystem and the memory, is used to execute the tunnel appearance defect detection method based on event camera multimodal fusion and spiking neural network.

[0007] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a method and system for detecting tunnel surface defects based on multimodal fusion of event cameras and spiking neural networks. The method utilizes the inertial data to perform motion compensation processing on the multi-channel spatiotemporal event tensor to obtain the compensated event tensor. Specifically, for the severe vibrations during high-speed operation of the subway, motion compensation based on the manifold of the auxiliary inertial measurement unit (IMU) is introduced. Through physical-level manifold reprojection, motion blur and false texture noise caused by vibration are effectively eliminated, so that the edge features of the defects remain sharp and focused even under high-speed shaking. The compensated event tensor and the grayscale image are fused using a multimodal method based on dual-stream fusion and a super-resolution reconstruction network to obtain a fused feature map. This overcomes the limitation of insufficient spatial resolution in bionic sensors. Based on the fused feature map, a spiking neural network is used to determine the detection results of tunnel surface defects. Specifically, by utilizing the asynchronous sparsity characteristics of the event camera and the "event-driven" computing mechanism of the spiking neural network (SNN), pulses are emitted and inferred only in sparse regions (<5%) where defects exist. This significantly reduces computing power and bandwidth consumption, making it possible to achieve millisecond-level (<20ms) real-time alarms at ultra-high speeds above 80km / h on vehicle-mounted embedded devices. This application can improve the anti-vibration interference capability, the accuracy of identifying minor defects, environmental adaptability, and real-time performance of tunnel surface defect detection. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a schematic diagram of the tunnel appearance defect detection method based on event camera multimodal fusion and spiking neural network in one embodiment of this application. Detailed Implementation

[0010] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0012] In one exemplary embodiment, such as Figure 1 As shown, a method for detecting tunnel surface defects based on multimodal fusion of event cameras and spiking neural networks is provided. This method includes steps S101 to S105. Wherein: S101, acquire sensor data inside the tunnel; the sensor data includes asynchronous event streams and grayscale images of the tunnel wall acquired using an event camera, as well as inertial data acquired using an auxiliary inertial measurement unit; S101 specifically includes: S11, The event camera and the auxiliary inertial measurement unit are mounted on the bracket on the top of the rail transit train car; the lens of the event camera faces the inner wall of the tunnel; the event camera and the auxiliary inertial measurement unit are rigidly connected by a mechanical bracket, thereby ensuring that their coordinate systems are relatively fixed. S12, use the event camera to acquire the asynchronous event stream and grayscale image of the tunnel wall; Specifically, when apparent defects (such as cracks, leaks, and spalling) appear on the tunnel wall, causing pixel-level light intensity changes to exceed a threshold, the asynchronous event stream output in real time by the event camera is acquired. and low-resolution grayscale images; wherein, the event camera is a stacked event camera; S13, using an auxiliary inertial measurement unit to acquire inertial data (i.e. IMU data, including triaxial angular velocity and triaxial acceleration) synchronized with the asynchronous event stream. S14, perform histogram equalization and normalization on the grayscale image to distribute its pixel values ​​within a certain range. This section aims to eliminate the impact of uneven tunnel lighting.

[0013] S102, Construct a multi-channel spatiotemporal event tensor based on the asynchronous event stream; S102 specifically includes: S21, Set the basic processing time window (e.g., 20ms), and the base processing time window Divided into Sub-time steps (e.g.) ); S22, within each sub-time step, statistically analyze the spatial distribution of positive polarity (ON) events and the spatial distribution of negative polarity (OFF) events, and generate two polarity feature maps; Specifically, the classification of polar events is based on the pixel-level logarithmic light intensity change principle of the visual sensor (EVS) in the event camera. Definition pixel coordinates At any moment The light intensity value is calculated in real time by the internal circuitry of the vision sensor. Compared to the last time the event was triggered logarithmic change in light intensity The determination formula is as follows: Based on a preset contrast threshold ( ), perform the following polarity division: when When this occurs, it is determined to be a significant increase in light intensity, generating a positive polarity event, and the polarity is marked as... (i.e., ON event); when When this occurs, it is determined that the light intensity has significantly decreased, generating a negative polarity event, and the polarity is marked as follows: (i.e., the OFF event).

[0014] S23, stack the polarity feature maps of all sub-time steps in the channel dimension to construct the multi-channel spatiotemporal event tensor.

[0015] Specifically, the stacking process of multi-channel spatiotemporal event tensors is as follows: traversing the... Each time step ( All event points within ) . Initialize two spatial dimensions as The zero matrix is ​​used as the ON and OFF feature maps for that time step, respectively. Next, a two-dimensional histogram statistical algorithm or a bilinear interpolation accumulation strategy is employed to determine the polarity of the feature maps. Events based on their pixel coordinates Accumulate to the corresponding position in the ON feature map; similarly, add to the polarity... The events are accumulated into the OFF feature map. At this point, a pair (two in total) of two-dimensional polar feature maps are generated for each sub-time step. Finally, the data concatenation instructions from the memory are used to... A total of [number] consecutive sub-time steps generated [number] The two-dimensional feature maps are merged strictly according to time sequence along the channel dimension of the tensor. Finally, a feature map of size [dimensionality missing] is constructed in memory. The three-dimensional dense spatiotemporal tensor is used for efficient reading and feature extraction by subsequent three-dimensional or two-dimensional convolutional neural networks. The multi-channel spatiotemporal event tensor is formed by stacking sparse event streams, which not only preserves the spatial morphology of the disease, but also implies its dynamic trajectory over time through the channel order; S103, use the inertial data to perform motion compensation processing on the multi-channel spatiotemporal event tensor to obtain the compensated event tensor; In order to eliminate imaging blur caused by high-speed vibration, preprocessing of inertial data is performed to clean and correct it; S103 specifically includes: S31, establish a manifold model based on the inertial data, and calculate the pose transformation matrix of the event camera within the basic processing time window; In response to the high-frequency mechanical vibrations accompanying high-speed train operation, within the basic treatment time window Inside, the camera's motion trajectory conforms to the manifold properties of a Lie group. A manifold model is built using inertial data, and the inertial data (i.e., IMU data, including three-axis angular velocity and three-axis acceleration) output by the synchronously acquired auxiliary positioning unit is analyzed. The camera motion within is integrated to calculate the result. At any time within Relative to reference time (usually taken) pose transformation matrix at the center time .

[0016] exist The triaxial angular velocities output by the IMU (Integrated Measurement Unit) are denoted as... The three-axis linear acceleration is denoted as The instantaneous pose of the event camera in 3D space is determined by the rotation matrix. Translation vector Composition. (Based on reference time) Given the initial state, based on the principles of rigid body kinematics, the kinematic differential equations of the event camera are as follows: ; The above formula describes the change in the rotational attitude of the event camera over time. It is the known quantity of the triaxial angular velocity measured by the IMU; It is transformed into an antisymmetric matrix; This is the first derivative of the rotation matrix. By integrating this formula, the rotation matrix of the event camera at any given time can be obtained. : ; The above formula describes the change of the camera's instantaneous velocity over time (i.e., acceleration). It is a known quantity of triaxial linear acceleration measured by the IMU; It is the gravitational acceleration constant. That is, it represents the local acceleration expressed through a rotation matrix. After transforming to the global coordinate system and subtracting the effect of gravity, the true acceleration is obtained. Integrating this velocity yields the instantaneous speed of the camera. .

[0017] ; The above formula describes the change of the spatial position of the event camera over time. That is, the derivative of the position vector. The instantaneous velocity obtained in the previous step... By performing time integration again, the translation vector of the event camera in three-dimensional space can be obtained. .

[0018] By analyzing the above differential equation over the time interval... By performing Runge-Kutta numerical integration within the inner circle, the exact solution can be obtained. Time relative to Relative rotation matrix at time step With relative translation vector .

[0019] Finally, a description of this relative motion is constructed. Homogeneous pose transformation matrix : ; in, for Time relative to The relative rotation matrix at time t, for Time relative to The relative translation vector at time T, where the superscript T is the transpose.

[0020] It precisely describes the instantaneous rotation and translation of the event camera in 6-DoF (Six Degrees of Freedom) space, specifically including three rotational degrees of freedom (roll, pitch, and yaw) and three translational degrees of freedom (linear displacement along the X, Y, and Z spatial coordinate axes). Under conditions of high-speed train travel and severe vibration, 6-DoF space can capture any minute spatial attitude deviation of the camera without any blind spots.

[0021] S32, a depth projection model of the event coordinates is established using the pose transformation matrix; the depth projection model uses the original pixel coordinates of each non-zero event point in the spatiotemporal event tensor. Reverse mapping back to the reference time On the virtual imaging plane; Using formula Depth projection model for determining event coordinates; in, The target coordinates after compensation (unknown, to be determined). The coordinates of the original event (known quantities). This is an estimate of the tunnel wall depth. and These are the projection and back projection functions of the event camera (known quantities, which can be set to constants). This is the pose transformation matrix.

[0022] The physical meaning of the depth projection model is a three-stage coordinate system transformation process: the first stage is "back projection": using the back projection function. And combined with the estimated depth of the tunnel wall The original event pixels on the two-dimensional imaging plane that have shifted due to vibration This restores the coordinates to their true physical coordinates in three-dimensional space. The second stage, "spatial manifold alignment," involves expanding these three-dimensional coordinates to homogeneous coordinates and then left-multiplying them by the pose transformation matrix. In three-dimensional physical space, the 6-DoF displacement error introduced by train vibration is forcibly canceled and aligned to the reference time. Spatial location. The third stage, "forward reprojection": utilizing the camera's projection function. The aligned 3D spatial points are then remapped back to the 2D imaging plane, ultimately yielding compensated coordinates completely free of vibration noise. .

[0023] S33, using the depth projection model of event coordinates, the event point coordinates in the multi-channel spatiotemporal event tensor are inversely mapped back to the virtual imaging plane at the reference time to obtain the compensated event tensor.

[0024] The reprojection transformation of the depth projection model of the event coordinates causes the background clutter caused by random vibrations to be filtered out due to non-compliance with geometric constraints; while the real disease edge events are accurately superimposed on the reference plane, providing a clear input for subsequent feature extraction. S104, the compensated event tensor and the grayscale image are fused using a multimodal fusion based on a dual-stream fusion and super-resolution reconstruction network to obtain a fused feature map; The dual-stream fusion and super-resolution reconstruction network includes a multi-layered cascaded two-dimensional convolutional layer (Conv2D), an Inception multi-scale convolutional module, a fusion module, and a super-resolution reconstruction module. The multi-layered cascaded two-dimensional convolutional layers are used to encode the compensated event tensor to obtain event dynamic features; the encoding of the multi-layered cascaded two-dimensional convolutional layers focuses on extracting high-frequency dynamic features such as edge flow direction and brightness abrupt changes of apparent diseases. The Inception multi-scale convolution module is used to extract features from grayscale images to obtain static image features; the Inception multi-scale convolution module extracts static features such as different background textures and water stain color differences in parallel to make up for the lack of grayscale information in the event stream. As can be seen from the above, multi-layered cascaded two-dimensional convolutional layers constitute dynamic flow branches, while Inception multi-scale convolutional modules constitute static flow branches. Under high-speed conditions, low-resolution grayscale images may inherently possess some degree of motion blur, but their primary purpose is to provide macroscopic low-frequency background textures (such as large areas of water stains or wall material colors). In the subsequent fusion module, the network utilizes the high-frequency event dynamic features with extremely high sharpness after motion compensation to perform spatial edge guidance and texture alignment on the blurred static features of the image, thereby achieving complementary advantages. The fusion module is used to concatenate and fuse dynamic event features with static image features to obtain a spatiotemporal consistency feature map; specifically, it concatenates dynamic branch features and static branch features along the channel dimension, and then... The convolutional layer performs dimensionality reduction and fusion, extracting and fusing dynamic features of events and static features of images to generate a spatiotemporal consistency feature map with rich semantic information. The super-resolution reconstruction module is used to obtain a fused feature map by using transposed convolution super-resolution based on the spatiotemporal consistency feature map and by employing two cascaded transposed convolution operations.

[0025] Specifically, using transposed convolution super-resolution, two cascaded transposed convolution operations are employed (each layer has a specific kernel size). Step length ,filling By combining the high-frequency temporal information of the event stream, interpolation and detail completion are performed on the low-resolution feature map, and the spatial resolution is magnified by 4 times to generate a high-resolution, high-definition contrast fused feature map. S105. Based on the fused feature map, a spiking neural network is used to determine the detection results of tunnel surface defects.

[0026] The spiking neural network comprises multiple spiking convolutional layers (SCL) and employs leaky integrator neurons (LIF) as computational units. The specific computation process is as follows: Using the fused feature map as the input current Input the first layer of the spiking neural network. Utilize the LIF neuron dynamics equations. Perform sparse reasoning, where The input current is a known quantity. This refers to membrane potential (a state variable). Neuronal membrane potential only accumulates above a threshold in lesion regions with high eigenvalues ​​in the fusion feature map. The neurons in the SNN emit pulses; however, in the healthy wall background area, the neurons remain silent, neither generating pulses nor consuming dynamic computing power. The output pulses of the SNN are decoded (e.g., using membrane potential readout mechanisms or pulse firing rate statistics) and converted into continuous feature vectors before entering the detection head. The head outputs the confidence level and type of the defect (cracks, leaks, spalling, etc.), and, combined with the odometry data provided by the auxiliary positioning unit, maps the image coordinates to the absolute physical location within the tunnel. It then regresses and calculates the bounding box coordinates of the defect, completing the detection report generation. Existing technologies widely using convolutional neural networks (CNNs) in decision recognition employ a "full-image intensive computation" mode, resulting in significant computational waste and high inference latency, making it difficult to adapt to the power consumption limitations of in-vehicle embedded devices. This mechanism fundamentally solves the computational bottleneck in high-speed detection.

[0027] This application has the following advantages over the prior art: (1) Strong anti-vibration interference capability (corresponding to IMU manifold compensation): In response to the severe vibration during high-speed operation of the subway, this invention innovatively introduces a motion compensation algorithm based on IMU manifold. Through physical-level manifold reprojection, motion blur and false texture noise caused by vibration are effectively eliminated, so that the edge features of the defects remain sharp and focused even under high-speed shaking.

[0028] (2) High accuracy in identifying minute defects (corresponding to super-resolution reconstruction): Overcomes the shortcomings of insufficient spatial resolution of bionic sensors. By using the "dual-stream feature fusion" and "transposed convolution super-resolution" techniques, the high-frequency temporal information of the event stream is used to fill in the spatial details, and the clear reconstruction and accurate identification of minute apparent defects with a width of less than 1 mm are successfully achieved.

[0029] (3) Ultra-high speed real-time performance and low power consumption (corresponding to SNN sparse computing): Utilizing the asynchronous sparse characteristics of the event camera and the "event-driven" computing mechanism of the SNN spiking neural network, the system only performs pulse emission and inference for the sparse areas (<5%) where the disease exists. This significantly reduces computing power and bandwidth consumption, making it possible to achieve millisecond-level (<20ms) real-time alarms at ultra-high speeds above 80km / h on vehicle-mounted embedded devices.

[0030] (4) Excellent environmental adaptability (corresponding to high dynamic range perception): Relying on the high dynamic range (HDR) of the sensor, the system can adapt to the extreme environment of dim lighting, high reflectivity and drastic changes in light at the entrance and exit of the tunnel without complicated supplementary lighting.

[0031] Based on the same inventive concept, this application also provides a tunnel appearance defect detection system based on event camera multimodal fusion and spiking neural networks for implementing the tunnel appearance defect detection method based on event camera multimodal fusion and spiking neural networks described above. The solution provided by this system is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more tunnel appearance defect detection system embodiments based on event camera multimodal fusion and spiking neural networks provided below can be found in the limitations of the tunnel appearance defect detection method based on event camera multimodal fusion and spiking neural networks described above, and will not be repeated here.

[0032] In one exemplary embodiment, a tunnel surface defect detection system based on event camera multimodal fusion and spiking neural network is provided. This system is deployed on a rail transit train and includes: A data acquisition subsystem is used to acquire sensor data inside the tunnel; the sensor data includes asynchronous event streams and grayscale images of the tunnel wall acquired using an event camera, as well as inertial data acquired using an auxiliary inertial measurement unit. Memory, used to store computer programs; The processor, which is communicatively connected to the data acquisition subsystem and the memory, is used to execute the tunnel appearance defect detection method based on event camera multimodal fusion and spiking neural network.

[0033] The system provided in this application relies on an onboard terminal system deployed on a rail transit train. The core sensors of the system include an event camera and an auxiliary inertial measurement unit (IMU) rigidly connected by a mechanical bracket. These are mounted on a support on the top of the rail transit train car, with the lens facing the tunnel wall to ensure that their coordinate systems are relatively fixed. Subsequent processing by multiple system units, including a processor (which may specifically be an onboard edge intelligent computing unit), enables ultra-high-speed, low-latency, and high-precision real-time identification of tunnel surface defects under high-speed train operation and strong vibration environments.

[0034] In an exemplary embodiment, a computer device is provided, which may be a server or a terminal. The computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The I / O interfaces of the computer device are used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a tunnel appearance defect detection method based on event camera multimodal fusion and spiking neural networks.

[0035] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0036] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0037] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0038] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0039] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0040] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0041] In this application, all actions to acquire signals, information, or data are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with the authorization granted by the owner of the relevant device.

[0042] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0043] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for detecting apparent defects in tunnels based on event camera multimodal fusion and spiking neural networks, characterized in that, The tunnel appearance defect detection method based on event camera multimodal fusion and spiking neural network includes: Acquire sensor data inside the tunnel; the sensor data includes asynchronous event streams and grayscale images of the tunnel interior wall acquired using an event camera, as well as inertial data acquired using an auxiliary inertial measurement unit; Construct a multi-channel spatiotemporal event tensor based on the asynchronous event stream; The inertial data is used to perform motion compensation processing on the multi-channel spatiotemporal event tensor to obtain the compensated event tensor; The compensated event tensor and the grayscale image are then fused using a multimodal fusion method based on a two-stream fusion and super-resolution reconstruction network to obtain a fused feature map. Based on the fused feature map, a spiking neural network is used to determine the detection results of tunnel surface defects.

2. The tunnel appearance defect detection method based on event camera multimodal fusion and spiking neural network according to claim 1, characterized in that, The acquisition of sensor data within the tunnel specifically includes: An event camera and an auxiliary inertial measurement unit are mounted on a support on the top of a rail transit train car; the lens of the event camera faces the inner wall of the tunnel; the event camera and the auxiliary inertial measurement unit are rigidly connected by a mechanical support. The event camera is used to acquire asynchronous event streams and grayscale images of the tunnel interior walls; Inertial data synchronized with the asynchronous event stream is acquired using an auxiliary inertial measurement unit; Histogram equalization and normalization are performed on grayscale images.

3. The method for detecting tunnel surface defects based on event camera multimodal fusion and spiking neural network according to claim 1, characterized in that, The construction of a multi-channel spatiotemporal event tensor based on the asynchronous event stream specifically includes: Set a basic processing time window and divide the basic processing time window into multiple sub-time steps; Within each sub-time step, the spatial distribution of positive polarity events and the spatial distribution of negative polarity events are statistically analyzed, and two polarity feature maps are generated. The polarity feature maps of all sub-time steps are stacked along the channel dimension to construct the multi-channel spatiotemporal event tensor.

4. The tunnel appearance defect detection method based on event camera multimodal fusion and spiking neural network according to claim 3, characterized in that, The step of using the inertial data to perform motion compensation processing on the multi-channel spatiotemporal event tensor to obtain the compensated event tensor specifically includes: A manifold model is established based on the inertial data, and the pose transformation matrix of the event camera within the basic processing time window is calculated. A depth projection model of the event coordinates is established using the pose transformation matrix; The event point coordinates in the multi-channel spatiotemporal event tensor are inversely mapped back to the virtual imaging plane at the reference time using the depth projection model of event coordinates, thus obtaining the compensated event tensor.

5. The tunnel appearance defect detection method based on event camera multimodal fusion and spiking neural network according to claim 4, characterized in that, The process of establishing a manifold model based on the inertial data and calculating the pose transformation matrix of the event camera within the basic processing time window specifically includes: Using formula Determine the pose transformation matrix ; in, for Time relative to The relative rotation matrix at time t, for Time relative to The relative translation vector at time T, where the superscript T is the transpose.

6. The tunnel appearance defect detection method based on event camera multimodal fusion and spiking neural network according to claim 4, characterized in that, The process of establishing a depth projection model of event coordinates using the pose transformation matrix specifically includes: Using formula Depth projection model for determining event coordinates; in, The target coordinates after compensation. The original event coordinates, This is an estimate of the tunnel wall depth. and These are the projection and back-projection functions of the event camera, respectively. This is the pose transformation matrix.

7. The method for detecting tunnel surface defects based on event camera multimodal fusion and spiking neural network according to claim 1, characterized in that, The dual-stream fusion and super-resolution reconstruction network includes a multi-layered cascaded two-dimensional convolutional layer, an Inception multi-scale convolutional module, a fusion module, and a super-resolution reconstruction module. The multi-layered cascaded two-dimensional convolutional layers are used to encode the compensated event tensor to obtain the event dynamic features; The Inception multi-scale convolution module is used to extract features from grayscale images to obtain static image features; The fusion module is used to stitch and fuse the dynamic features of the event with the static features of the image to obtain a spatiotemporal consistency feature map; The super-resolution reconstruction module is used to obtain a fused feature map by using transposed convolution super-resolution based on the spatiotemporal consistency feature map and by employing two cascaded transposed convolution operations.

8. The method for detecting tunnel surface defects based on event camera multimodal fusion and spiking neural network according to claim 7, characterized in that, The fusion module utilizes Dimensionality reduction and fusion are performed on convolutional layers.

9. The method for detecting tunnel surface defects based on event camera multimodal fusion and spiking neural network according to claim 1, characterized in that, The spiking neural network includes multiple spiking convolutional layers and uses leaky firing neurons as computational units.

10. A tunnel appearance defect detection system based on event camera multimodal fusion and spiking neural network, characterized in that, The tunnel surface defect detection system based on event camera multimodal fusion and spiking neural network is deployed on rail transit trains and includes: A data acquisition subsystem is used to acquire sensor data inside the tunnel; the sensor data includes asynchronous event streams and grayscale images of the tunnel wall acquired using an event camera, as well as inertial data acquired using an auxiliary inertial measurement unit. Memory, used to store computer programs; The processor, communicatively connected to the data acquisition subsystem and the memory, is used to execute the tunnel appearance defect detection method based on event camera multimodal fusion and spiking neural network as described in any one of claims 1-9.

Images