High space-time sampling fusion method and system, terminal and storage medium

Abstract

The invention relates to the technical field of structural health monitoring, and discloses a high space-time sampling fusion method and system, a terminal and a storage medium, and the high space-time sampling fusion method comprises the steps: obtaining an event stream of a monitoring target, a grayscale image, and a visible light image in the same time dimension, the event stream and the grayscale image are homologous data, and the resolution of the visible light image is higher than that of the event stream; performing feature matching and affine transformation according to the gray level image and the visible light image to obtain a registration gray level image; event up-sampling is carried out according to the event stream and the visible light image, a matching event stream is obtained, and the resolution ratio of the matching event stream is consistent with that of the registration grayscale image; and generating monitoring data of the monitoring target according to the registration grayscale image and the matching event stream. According to the invention, high-frequency vibration capture and millimeter-level deformation analysis of the monitored target can be realized synchronously, and the monitoring effect is improved.

Description

Technical Field

[0001] This application relates to the technical field of structural health monitoring, and particularly to a high spatio-temporal sampling fusion method, system, terminal and storage medium. Background Art

[0002] With the rapid development of structural health monitoring (SHM), industrial vibration analysis and intelligent operation and maintenance technologies, the demand for the collaborative perception of high time resolution and high spatial resolution in engineering structures (such as bridges, wind turbine blades, precision instruments) is becoming increasingly urgent. Under complex working conditions, high-frequency vibrations (such as the initiation of fatigue cracks in composite materials, micro-amplitude resonance of bearings) require microsecond-level time resolution (≥1 MHz sampling rate) to capture transient events, while micron-level deformations (such as bolt loosening displacement <0.1 mm) require sub-pixel-level spatial resolution (≥4K pixel density) for accurate measurement. However, there are fundamental contradictions in the collaborative spatio-temporal resolution and dynamic range adaptation of existing technologies.

[0003] Although traditional contact monitoring devices (such as accelerometers, laser rangefinders) can provide high-precision data, they need to be in contact with or installed close to the object to be measured, resulting in complex deployment, high maintenance costs, and possible interference with the structural dynamic response. Especially in long-term unattended monitoring, contact sensors are prone to failure due to environmental corrosion or mechanical vibration, and the data reliability is significantly reduced. Video monitoring technology has gradually become a research hotspot due to its non-contact and full-field coverage advantages, but existing technologies still face core challenges: high-resolution cameras are limited by the frame rate and cannot resolve high-frequency vibrations; event cameras have microsecond-level time resolution, but their spatial resolution is low and the output is a sparse event stream, which requires complex reconstruction algorithms and it is difficult to directly obtain complete information of static scenes.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main purpose of this application is to provide a high spatio-temporal sampling fusion method, system, terminal and storage medium, aiming to solve the problem that high-resolution cameras used in structural monitoring in existing technologies cannot resolve high-frequency vibrations or event cameras are difficult to directly obtain complete information of static scenes, resulting in poor monitoring effects.

[0006] In the first aspect of the embodiments of the present application, a high spatio-temporal sampling fusion method is provided. The high spatio-temporal sampling fusion method includes the following steps: obtaining an event stream, a grayscale image, and a visible light image in the same time dimension of a monitoring target, where the event stream and the grayscale image are homologous data, and the resolution of the visible light image is higher than that of the event stream; performing feature matching and affine transformation according to the grayscale image and the visible light image to obtain a registered grayscale image; performing event upsampling according to the event stream and the visible light image to obtain a matching event stream, where the resolution of the matching event stream is the same as that of the registered grayscale image; and generating monitoring data of the monitoring target according to the registered grayscale image and the matching event stream.

[0007] Optionally, in an embodiment of the present application, before obtaining the event stream, the grayscale image, and the visible light image in the same time dimension of the monitoring target, it further includes: after aligning the global shutter exposure period of the visible light camera with the event sampling time window of the event camera, obtaining the event timestamp of the event camera; compensating for the row exposure transmission delay of the event timestamp to synchronize the visible light camera and the event camera in the data acquisition time dimension.

[0008] Optionally, in an embodiment of the present application, the performing feature matching and affine transformation according to the grayscale image and the visible light image to obtain a registered grayscale image specifically includes: performing feature extraction and matching on the grayscale image and the visible light image to obtain feature matching point pairs; solving for affine transformation according to the feature matching point pairs to obtain the mapping relationship from the coordinate system of the visible light image to the coordinate system of the grayscale image; and mapping the coordinates of the visible light image to the coordinate system of the grayscale image according to the mapping relationship to obtain a registered grayscale image.

[0009] Optionally, in an embodiment of the present application, the solving for affine transformation according to the feature matching point pairs to obtain the mapping relationship from the coordinate system of the visible light image to the coordinate system of the grayscale image specifically includes: Calculating a homography matrix according to the feature matching points; The homography matrix is expressed as: ; Where is the homography matrix, is the rotation parameter of the first row elements of the homography matrix, is the scaling parameter of the first row elements of the homography matrix, is the translation parameter of the first row elements of the homography matrix, is the rotation parameter of the second row elements of the homography matrix, is the scaling parameter of the second row elements of the homography matrix, is the translation parameter of the second row elements of the homography matrix; Minimize the reprojection error according to the homography matrix to obtain the mapping relationship from the coordinate system of the visible light image to the coordinate system of the grayscale image; The mapping relationship is expressed as: ; where, is the abscissa of the pixel point of the registered grayscale image, is the ordinate of the pixel point of the registered grayscale image, is the abscissa of the pixel point of the visible light image, is the ordinate of the pixel point of the visible light image.

[0010] Optionally, in an embodiment of the present application, the event upsampling according to the event stream and the visible light image to obtain a matching event stream specifically includes: performing time window segmentation on the event stream according to the period of the visible light image to obtain event frames; calculating the weight distribution of each event in the spatio-temporal domain according to the event frames to obtain the three-dimensional Gaussian weights corresponding to multiple events in the event frames; reconstructing the event frames according to the three-dimensional Gaussian weights corresponding to multiple events in the event frames to obtain a matching event stream, where the resolution of the matching event stream is consistent with that of the visible light image.

[0011] Optionally, in an embodiment of the present application, the formula for the time window segmentation is: ; where, is the set of events within the th time window, is a single event in the time window, , are the horizontal and vertical position coordinates in the space of the th event, is the time in the th event, is the polarity in the th event, is the trigger period; The formula for the weight distribution is: ; where, is the three-dimensional Gaussian weight, , are the horizontal and vertical coordinates in the spatial domain, is the coordinate in the time domain, is the standard deviation of the spatial Gaussian kernel, is the standard deviation of the time Gaussian kernel.

[0012] Optionally, in an embodiment of the present application, after generating a matching event stream corresponding to the resolution of the registered grayscale image by performing event upsampling on the basis of the event stream and the visible light image pair, the method further includes: generating monitoring data of the monitoring target according to the registered grayscale image and the matching event stream.

[0013] A second aspect of the embodiments of the present application further provides a high spatio-temporal sampling fusion system, where the high spatio-temporal sampling fusion system includes: An image acquisition module, configured to acquire an event stream, a grayscale image, and a visible light image in the same time dimension of a monitoring target, where the event stream and the grayscale image are homologous data, and the resolution of the visible light image is higher than that of the event stream; A spatial alignment module, configured to perform feature matching and affine transformation according to the grayscale image and the visible light image to obtain a registered grayscale image; A resolution synchronization module, configured to perform event upsampling according to the event stream and the visible light image to obtain a matching event stream, where the resolution of the matching event stream is consistent with that of the registered grayscale image; A data fusion module, configured to generate monitoring data of the monitoring target according to the registered grayscale image and the matching event stream.

[0014] A third aspect of the embodiments of the present application further provides a terminal, where the terminal includes: a memory, a processor, and a high spatio-temporal sampling fusion program stored on the memory and executable on the processor, and when the high spatio-temporal sampling fusion program is executed by the processor, the steps of the high spatio-temporal sampling fusion method as described above are implemented.

[0015] A fourth aspect of the embodiments of the present application further provides a computer-readable storage medium, where the computer-readable storage medium stores a high spatio-temporal sampling fusion program, and when the high spatio-temporal sampling fusion program is executed by a processor, the steps of the high spatio-temporal sampling fusion method as described above are implemented.

[0016] Beneficial effects: The present application provides a high spatio-temporal sampling fusion method, system, terminal, and storage medium. By acquiring an event stream and a grayscale image collected by an event camera and a visible light image collected by a visible light camera in the same time dimension, the present application replaces the grayscale of the visible light image into the grayscale image to complete spatial alignment, and then improves the resolution of the event stream to be consistent with that of the visible light image, so as to obtain a registered grayscale image and a matching event stream with consistent resolutions, that is, through complementary fusion of sub-millimeter-level static deformation data of the visible light image and microsecond-level dynamic vibration signals of the event stream, high-frequency vibration capture and sub-millimeter-level deformation analysis of the monitoring target are synchronously realized, thereby improving the monitoring effect. Brief Description of the Drawings

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments recorded in the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0018] Figure 1 It is a schematic structural diagram of a preferred embodiment of the high spatio-temporal sampling fusion device of the present application; Figure 2 It is a schematic structural diagram of a preferred embodiment of the high spatio-temporal sampling fusion device of the present application; Figure 3 It is a flowchart of a preferred embodiment of the high spatio-temporal sampling fusion method of the present application; Figure 4 It is a schematic flowchart of event stream upsampling in a preferred embodiment of the high spatio-temporal sampling fusion method of the present application; Figure 5 It is a structural diagram of a preferred embodiment of the high spatio-temporal sampling fusion system of the present application; Figure 6 It is a structural diagram of a preferred embodiment of the terminal of the present application.

[0019] Description of the Reference Numerals in the Drawings: 11. Translation stage; 12. Visible light camera; 13. Beam splitter prism; 14. Event camera; 100. Image acquisition module; 200. Spatial alignment module; 300. Resolution synchronization module; 400. Data fusion module. Detailed Embodiments

[0020] To make the objectives, technical solutions and effects of the present application clearer and more definite, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. The described embodiments are only possible technical implementations of the present application, not all possible implementations. Based on the embodiments in the present application, those skilled in the art can fully combine the embodiments of the present application to obtain other embodiments without creative efforts, and these embodiments are also within the protection scope of the present application.

[0021] First, introduce the terms involved in the embodiments of the present application: CMOS (Complementary Metal-Oxide-Semiconductor), complementary metal oxide semiconductor, referring to the type of sensor technology; DR (Dynamic Range), the dynamic range, refers to the range of signal intensities that a sensor can handle; DAVIS (Dynamic and Active-pixel Vision Sensor), a type of event camera, refers to a dynamic vision sensor.

[0022] RGB (Red, Green, Blue), the three primary colors of red, green, and blue, refers to the three-channel color model of a color image; 4K: Refers to the resolution of an image or display, usually 3840×2160 pixels; 1000fps@4K refers to the 4K resolution video capture speed of 1000 frames per second.

[0023] In the related art, a single camera is used to monitor the structure, or multiple different visible light cameras are used to fuse images for structure monitoring. The following introduces the visible light cameras, event cameras, and high-speed cameras in the related art: Working principle of a visible light camera: A visible light camera captures the light intensity information of the three channels of red, green, and blue (or a single channel of grayscale information) at a fixed frame rate through a global shutter or rolling shutter CMOS sensor, generating high-resolution static images. Its core advantage lies in the spatial detail analysis ability and is suitable for texture feature extraction (such as morphological analysis of cracks on the structure surface). Technical advantages: High spatial resolution, supporting sub-pixel level detail analysis (such as pixel size ≤ 1.4μm at 4K resolution); Cost and ease of use, the cost of industrial visible light cameras is controllable, and no complex calibration process is required, suitable for large-scale deployment.

[0024] However, the visible light camera has low temporal resolution. Limited by the sensor readout speed and shutter mechanism, it cannot capture high-frequency dynamic events (such as transient crack propagation under impact loads); the dynamic range of the visible light camera is limited. In the global shutter mode, the dynamic range (DR) is usually ≤ 70dB, and overexposure or underexposure is likely to occur in areas of strong light reflection (such as the surface of a metal structure) and weak light shadows.

[0025] Working principle of an event camera: An event camera is based on an asynchronous pixel-level light intensity change detection mechanism, and only records the pixel points with brightness changes and their timestamps. Its temporal resolution can reach the microsecond level, but it sacrifices spatial resolution. Technical advantages: Ultra-high temporal resolution, capable of capturing high-speed dynamic events (such as a bullet penetrating, structural impact); Low power consumption and high dynamic range: Single-pixel independent response, and the dynamic range (DR) can reach more than 120dB.

[0026] However, in event cameras, sparse data output, that is, only discrete event streams of brightness changes are output, lacks complete information of static scenes; motion blur and artifacts, that is, fast-moving objects are prone to event accumulation (such as deformed areas under structural vibration), resulting in distorted reconstructed images.

[0027] Therefore, although visible light cameras support sub-pixel level spatial resolution analysis, they cannot capture high-frequency vibrations. Although event cameras have microsecond-level time resolution, their spatial resolution is low. As a result, using the same type of camera for structural monitoring has a poor effect, and related technologies cannot be combined due to the contradiction between the spatio-temporal resolution coordination and dynamic range adaptation of visible light cameras and event cameras.

[0028] Working principle of high-speed cameras: High-speed cameras achieve ultra-high frame rates by reducing resolution or compressing pixel depth and are commonly used to capture transient phenomena. Technical advantages: Ultra-high frame rates can record millisecond-level dynamic processes; Trigger synchronization ability: Supports external signal triggering for precise alignment of physical events.

[0029] However, the continuous shooting time of high-speed cameras is limited. Even with TB (Terabyte)-level storage, the continuous recording time in the 1000fps@4K mode is only about 1 - 2 minutes, which is difficult to meet the long-term monitoring requirements; The hardware cost is high, requiring dedicated storage and processing units, and the system integration complexity is high; The resource consumption for post-processing is large: One hour of high-speed video may contain millions of frames, and the processing efficiency of manual annotation or traditional algorithms is extremely low, requiring AI acceleration. That is to say, although high-speed cameras can capture transient events and have sufficient resolution, they are expensive and require powerful acquisition devices and are not portable enough.

[0030] Aiming at the problem that high-resolution cameras used for structural monitoring cannot resolve high-frequency vibrations or event cameras are difficult to directly obtain complete information of static scenes, resulting in poor monitoring effects, this application obtains the event stream and grayscale image collected by the event camera, as well as the visible light image collected by the visible light camera in the same time dimension, replaces the grayscale of the visible light image into the grayscale image to complete spatial alignment, and then improves the resolution of the event stream to be consistent with the resolution of the visible light image, so as to obtain a registered grayscale image and a matching event stream with consistent resolutions, that is, through complementary fusion of sub-millimeter-level static deformation data of the visible light image and microsecond-level dynamic vibration signals of the event stream, thereby synchronously achieving high-frequency vibration capture and sub-millimeter-level deformation analysis of the monitoring target, and further improving the monitoring effect.

[0031] The technical solution of this application will be described in detail with specific embodiments below. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0032] Such as Figure 1 andFigure 2 As shown, an embodiment of the present application provides a high spatiotemporal sampling fusion device, which is applied to a high spatiotemporal sampling fusion method. The high spatiotemporal sampling fusion device includes a visible light camera 12, a spectroscopic prism 13, an event camera 14 and a controller. The visible light camera 12 and the event camera 14 are coaxially arranged, and the visible light camera 12 and the event camera 14 are respectively arranged opposite to the spectroscopic prism 13; the controller includes a time alignment module and a space alignment module, and the time alignment module and the space alignment module are respectively connected to the visible light camera 12 and the event camera 14.

[0033] Specifically, the visible light camera 12, the beam splitter prism 13, and the event camera 14 are installed on the translation stage 11 to form an optical coupling module. In the optical coupling module, a 5:5 beam splitter prism 13 is used to achieve optical path coupling. The visible light camera 12 and the DAVIS event camera 14 are coaxially installed on the precision translation stage 11. The beam splitter prism 13 is used to achieve optical path coupling between the visible light camera 12 and the DAVIS event camera 14. The beam splitter prism 13 evenly distributes the incident light to the two cameras to ensure the consistency of the observation field of the two. Due to the differences in physical size and imaging characteristics between the high-resolution photosensitive element of the visible light camera 12 and the asynchronous event sensor of the DAVIS event camera 14, the camera position is fine-tuned by the precision translation stage 11 to compensate for the field of view offset. The coaxial installation design combined with the mechanical adjustment mechanism allows the visible light image and the event stream to be initially aligned in space, providing a basis for subsequent processing.

[0034] In view of the core defects of existing technologies in structural health monitoring and vibration analysis, such as the contradiction between time and space resolution (low frame rate of high-resolution sensors, insufficient spatial resolution of high-speed sensors), limited dynamic range (poor signal-to-noise ratio in strong light / weak light environment), and high system complexity (time-division multiplexing delay, low reliability of mechanical turntable), the high-time and space sampling fusion method of this application is based on a non-contact monitoring strategy of heterogeneous visual fusion and phase adaptive algorithm. Its core idea is to achieve high-frequency vibration capture and sub-millimeter deformation analysis simultaneously through multi-modal sensing collaboration and dynamic signal enhancement technology, breaking through the technical barriers of traditional single-modal sensors in the time-space-dynamic range dimension. This application is centered on heterogeneous visual fusion, using a 5k visible light camera to capture high-resolution static details, DAVIS to capture high-dynamic micro-vibration signals, and using a dynamic range extension module to optimize the imaging quality of the entire scene. This application does not need to rely on contact sensors or complex mechanical structures, and directly realizes non-contact, all-weather dynamic perception of structural health through visual data streams and embedded edge computing. The core value of this application lies in solving the industry problem of collaborative perception of "high-frequency vibration-submillimeter deformation", providing a non-contact, high-precision, low-cost real-time health monitoring solution for large-scale infrastructure such as bridges and wind turbine blades. It can also be expanded to precision manufacturing, aerospace and other fields, combining technological advancement with potential for large-scale application.

[0035] Compared with the related technologies, the specific advantages of this application are as follows: Breakthrough heterogeneous fusion architecture: Hardware-level optical path coupling is performed between a visible light camera with high spatial resolution and low temporal resolution (such as a 25-megapixel RGB or grayscale camera, 10 Hz) and an event camera with high temporal resolution and low spatial resolution (such as a DAVIS camera with less than 1 megapixel, 10 MHz). Through a coaxial design of a beam splitting prism and by integrating a cross-scale spatio-temporal registration method, data homologous acquisition at the physical level is achieved, realizing synchronous perception of sub-pixel displacement measurement at the 10-μm level and vibration capture at the 1-ms level, and solving the industry problem of "high frame rate must reduce resolution, and high resolution must reduce frame rate" in traditional methods.

[0036] Wide range coverage and strong adaptability: In the spatial dimension, it supports cross-scale perception from centimeter-level overall deformation (such as bridge deflection) to micron-level local damage (such as structural microcracks); in the time dimension, it is compatible with quasi-static deformation (below 0.1 Hz) to high-frequency vibration (above 5 kHz); the signal-to-noise ratio can still be maintained > 35 dB in strong light reflection areas (such as metal surfaces) and weak light shadow areas.

[0037] Controllable cost: Compared with traditional high-speed cameras, the present invention uses standard industrial camera equipment and DAVIS, reducing the cost of the system and simplifying the information density of the acquisition process and processing.

[0038] The high spatio-temporal sampling fusion method described in the preferred embodiment of this application, as Figure 3 shown, the high spatio-temporal sampling fusion method includes the following steps: In step S101, an event stream, a grayscale image, and a visible light image in the same time dimension of a monitoring target are acquired, where the event stream and the grayscale image are homologous data, and the resolution of the visible light image is higher than that of the event stream.

[0039] It should be noted that the event stream and the grayscale image acquired by the event camera are homologous. The event stream provides dynamic change information in the scene, while the grayscale image (generated by the dynamic vision sensor (DAVIS) in the DAVIS event camera) provides static background information of the scene. It can be understood that the event stream is a time series composed of a series of discrete and non-uniformly distributed dynamic events, and the event stream is represented as a set of a series of events; each dynamic event refers to a pixel-level brightness change captured by the DAVIS event camera. When the logarithmic brightness change at a certain point in the scene exceeds a set threshold, the DAVIS will record the event, and each dynamic event includes spatial coordinates, a timestamp, and a polarity.

[0040] In a possible implementation, before the visible light camera and the event camera acquire images, after aligning the global shutter exposure period of the visible light camera with the time window of event sampling of the event camera, the event timestamp of the event camera is acquired; the row exposure transmission delay of the event timestamp is compensated to synchronize the visible light camera and the event camera in the data acquisition time dimension.

[0041] It should be noted that the purpose of time calibration is to eliminate the cross-sensor time deviation and ensure the strict synchronization of the event stream and the visible light image.

[0042] Specifically, the time registration module ensures the time consistency of the data through the cooperation of hardware signals and software. At the hardware level, the external trigger box generates a synchronization pulse signal to strictly align the global shutter exposure of the visible light camera with the time window of DAVIS event sampling, eliminating the timing deviation of traditional time-sharing acquisition. At the software level, the dynamic calibration algorithm based on timestamps further compensates for the inherent delay of the sensor (such as exposure transmission time) to achieve a time synchronization accuracy of microseconds and ensure the unity of the time reference for cross-modal data.

[0043] Furthermore, in the hardware trigger synchronization model, the trigger signal period is T , the exposure time of the visible light camera is T , the event sampling window of the DAVIS event camera is , is the start time of the th sampling of the DAVIS event camera. Hardware synchronization ensures that: , ; where is the clock jitter, is the standard deviation of the clock jitter, <1 μs .

[0044] When compensating the software timestamp, for the DAVIS event timestamp , the row exposure transmission delay is compensated as follows: ; where is the original event timestamp, is the compensated event timestamp, is the row exposure transmission delay, is the total number of rows of the DAVIS sensor, is the row coordinate where the event occurs.

[0045] In step S102, feature matching and affine transformation are performed based on the grayscale image and the visible light image to obtain a registered grayscale image.

[0046] In a possible implementation, feature extraction and matching are performed on the grayscale image and the visible light image to obtain feature matching point pairs; an affine transformation solution is performed based on the feature matching point pairs to obtain the mapping relationship from the coordinate system of the visible light image to the coordinate system of the grayscale image; according to the mapping relationship, the coordinates of the visible light image are mapped onto the coordinate system of the grayscale image to obtain a registered grayscale image.

[0047] It should be noted that the spatial registration module solves the problems of spatial alignment and time synchronization of cross-modal data. The purpose of spatial registration is to establish the spatial mapping relationship between the visible light image and the DAVIS event stream (including the grayscale image and the event stream), and to compensate for the sensor resolution and field of view differences.

[0048] In a possible implementation, a homography matrix is calculated based on the feature matching points; the mapping relationship from the coordinate system of the visible light image to the coordinate system of the grayscale image is obtained by minimizing the reprojection error according to the homography matrix.

[0049] Specifically, during the spatial registration process, first, feature extraction and matching are performed. Local feature points (such as corner points and edges) of the visible light image and the DAVIS grayscale image are extracted by the feature matching method. The matching strategy of the feature scanning sub is used to calculate the similarity between the feature points. The nearest neighbor matching search is used to screen out the inliers and remove the outliers to eliminate the false matches and ensure the matching accuracy. It should be noted that the grayscale of the visible light image is registered with the grayscale of the grayscale image. Since the field of view of the visible light image does not match the field of view of the grayscale image (the viewing fields are different), image cropping is performed, so that the grayscale of the visible light image can be registered with the grayscale of the grayscale image. Then, a 3×3 homography matrix (affine transformation matrix) is calculated based on the matching point pairs, and the mapping relationship is obtained by optimizing the target by minimizing the reprojection error. The matrix parameters include rotation, scaling, translation, and shear transformation to compensate for the sensor installation error. Then, the coordinates of the high-resolution visible light image are mapped to the low-resolution DAVIS coordinate system through the homography matrix to achieve spatial alignment. That is, the grayscale of the visible light image is replaced into the DAVIS grayscale image to complete the grayscale registration.

[0050] Furthermore, in the establishment of the physical model, let the imaging model of the visible light camera be: ; where, is the pixel coordinate of the visible light camera imaging, representing the projection of the three-dimensional space point in the visible light image; is the internal parameter matrix of the visible light camera, is the external parameter matrix of the visible light camera, representing the rotation ( ) and translation ( ) relationship; is the three - dimensional space point coordinates.

[0051] The DAVIS event camera model is: ; ; Among them, is the pixel coordinates of the DAVIS event camera, representing the projection of the three - dimensional space point in the DAVIS event stream; is the internal parameter matrix of the DAVIS event camera, is the internal parameter matrix; is the external parameter matrix of the DAVIS event camera, representing the rotation ( ) and translation ( ) relationship between the event camera coordinate system and the world coordinate system; is the event noise term, representing sensor noise or model error; , are the horizontal and vertical focal lengths, representing the distance from the camera lens to the imaging plane; , are the horizontal and vertical principal point coordinates, representing the projection position of the optical axis on the imaging plane.

[0052] Solve the affine transformation, and obtain the homography matrix between the two sensors through feature matching for image registration. The homography matrix is expressed as: ; Among them, is the homography matrix, is the rotation parameter of the first - row elements of the homography matrix, is the scaling parameter of the first - row elements of the homography matrix, is the translation parameter of the first - row elements of the homography matrix, is the rotation parameter of the second - row elements of the homography matrix, is the scaling parameter of the second - row elements of the homography matrix, is the translation parameter of the second - row elements of the homography matrix.

[0053] Then the mapping relationship can be obtained. The mapping relationship is expressed as: ; Among them, is the abscissa of the pixel point of the registered grayscale image, is the ordinate of the pixel point of the registered grayscale image, is the abscissa of the pixel point of the visible - light image, is the ordinate of the pixel point of the visible - light image.

[0054] In the prior art, sensors with high spatial resolution (visible light cameras) and high temporal resolution (DAVIS) usually work independently, limited by the spatio-temporal resolution contradiction: the high frame rate sensor (DAVIS) has insufficient spatial resolution due to pixel scale limitations, while the high-resolution sensor (5K camera) is difficult to increase the frame rate (30fps) due to transmission bandwidth limitations; and using a high-speed camera is too costly and not very portable. In this application, heterogeneous data cross-scale fusion is adopted. Through the heterogeneous data fusion architecture, the sub-millimeter static deformation data of the visible light camera and the microsecond-level dynamic vibration signals of the DAVIS event camera are complementarily fused, breaking through the performance upper limit of traditional single-modal sensors in the spatio-temporal dimension. It can achieve cross-scale perception: global high-resolution deformation monitoring and local high-frequency vibration capture are realized synchronously; and the dynamic range is improved: by fusing the wide dynamic range of visible light and the high-frequency response of DAVIS, the full-scene monitoring requirements from static deformation to transient impact are covered.

[0055] In step S103, event upsampling is performed according to the event stream and the visible light image to obtain a matching event stream, where the resolution of the matching event stream is the same as that of the registered grayscale image.

[0056] In a possible implementation, the event stream is time-window sliced according to the period of the visible light image to obtain event frames; the weight distribution of each event in the spatio-temporal domain is calculated according to the event frames to obtain the three-dimensional Gaussian weights corresponding to multiple events in the event frames; according to the three-dimensional Gaussian weights corresponding to multiple events in the event frames, the event frames are reconstructed to obtain a matching event stream, where the resolution of the matching event stream is the same as that of the visible light image.

[0057] It should be noted that the purpose of event upsampling is to improve the spatial resolution of the DAVIS event stream and match the detail level of the visible light image.

[0058] Specifically, as Figure 4 shown, event upsampling utilizes the homology between the DAVIS event stream and the grayscale image. First, the event stream is time-window sliced to obtain event frames, that is, the continuous event stream is divided into discrete time windows at a fixed period, and each window contains multiple events to form event frames. Then, upsampling is performed, and a three-dimensional Gaussian kernel is used for modeling to achieve high-precision registration. Finally, for each pixel position, the Gaussian weights of all events within the window are accumulated to output a high-resolution event frame, with the spatial resolution being the same as that of the visible light image. That is, the event stream is upsampled to raise the resolution of the event stream itself to be the same as that of the visible light camera.

[0059] Further, let the trigger period of DAVIS be , the DAVIS event stream is segmented according to the same time window, and the formula for the time window segmentation is: ; Among them, is the set of events within the th time window. Each time window contains events, is a single event in the time window, , are the horizontal and vertical position coordinates of the th event, is the time in the th event, is the polarity in the th event, is the trigger period; Three-dimensional Gaussian kernel modeling is to calculate the weight distribution (without considering polarity) of each event ∈ in the spatio-temporal domain. The formula for the weight distribution is: ; Among them, is the three-dimensional Gaussian weight; , are the horizontal and vertical coordinates in the spatial domain; is the coordinate in the time domain; is the standard deviation of the spatial Gaussian kernel, which controls the diffusion range of events in space; is the standard deviation of the temporal Gaussian kernel, which controls the diffusion range of events in time.

[0060] Specifically, during the time calibration process, a time offset compensation model is constructed by combining the hardware trigger signal and the event timestamp to ensure strict alignment of the data stream in the time dimension.

[0061] In the prior art, multi-sensor registration usually relies on single-modal feature matching (such as registration based on grayscale or edges), and it is difficult to solve the spatio-temporal alignment problem of cross-modal (such as visible light images and event streams) with significant differences in resolution. In this application, a spatio-temporal synchronization registration scheme is adopted, that is, a cross-modal three-level registration scheme: First, preliminary spatial coarse registration: Based on the feature matching between the visible light image and the grayscale image of the DAVIS camera, an initial spatial mapping relationship is established. Since the grayscale pixels and event pixels of the DAVIS camera are homologous, mapping the visible light image space to the grayscale image space of the DAVIS camera can ensure the rough registration of the event stream and the visible light image; Second, dynamic spatial fine registration: Generate a dense 3D point cloud through spatio-temporal upsampling of the event stream, and use spatio-temporal Gaussian kernel interpolation to optimize the sub-pixel alignment between the visible light image and the DAVIS; Third, time series synchronization calibration: Adopt a hardware-level trigger synchronization architecture, and achieve microsecond-level time reference alignment between the visible light camera and the DAVIS event camera through the precise trigger signal generated by the trigger circuit. Combine the timestamp calibration circuit to eliminate the exposure delay and build a unified time reference for the entire system. This application can achieve high-precision alignment by generating a dense motion field that matches the resolution of the visible light camera through the spatio-temporal upsampling technology of the event stream for the resolution difference of up to 15 times between the visible light image (5496×3672) and the DAVIS event stream (346×240), breaking through the technical bottleneck that traditional methods are difficult to register when the resolution difference > 3 times.

[0062] In step S104, after obtaining the registered grayscale image and the matching event stream, data fusion is performed. According to the registered grayscale image and the matching event stream, monitoring data of the monitoring target is generated.

[0063] Specifically, the controller further includes a data fusion module. The data fusion module integrates the static texture details of the visible light and the dynamic response characteristics of the event stream, and realizes cross-modal feature fusion through a dynamic weight distribution algorithm.

[0064] In a possible implementation manner, a static fusion weight and a dynamic fusion weight are obtained; according to the static fusion weight, the dynamic fusion weight, the registered grayscale image, and the matching event stream, the fusion value at each pixel position is calculated; according to the fusion values at all pixel positions, the monitoring data of the monitoring target is obtained.

[0065] It can be understood that the fusion weight is generated based on feature importance (such as deformation significance, vibration energy); pixel-level fusion is performed to calculate the fusion value at each pixel position; spatio-temporal consistency optimization is performed, and combined with the time offset compensation model output by the time calibration module, to ensure the continuity of the fusion data stream in the spatio-temporal domain.

[0066] Furthermore, the purpose of static feature enhancement is to utilize the high-resolution background information of visible light images to locate and analyze structural deformations. During the process of static feature enhancement, the visible light image is preprocessed to identify the structural deformation area, and static feature mapping is performed to encode the structural deformation information into feature vectors. The purpose of dynamic feature extraction is to capture high-frequency vibration signals through the event stream and enhance the micron-level displacement details. During the process of dynamic feature extraction, the event stream captures high-frequency vibration signals, and the micro-displacement details are enhanced through time-domain amplification technology. During the fusion output process, combined with the attention mechanism, the contribution weights of different modalities are dynamically adjusted to generate a composite data stream with high spatio-temporal resolution (i.e., monitoring data), supporting precise analysis from the overall vibration mode to local micron-level defects.

[0067] In another embodiment of the present application, optical component adjustment is performed. The beam splitter prism can be replaced with a dichroic mirror or a polarization beam splitter device to adapt to specific wavelength bands (such as near-infrared) or polarization-sensitive scenarios; in the low-cost version, a fixed bracket can be used to replace the precision translation stage, and the field-of-view offset can be compensated by software algorithms.

[0068] In another embodiment of the present application, synchronization mechanism optimization is performed. The hardware trigger signal can be extended to multi-node cascading, supporting nanosecond-level synchronization of distributed monitoring networks; at the software level, the NTP / PTP protocol (NTP is the Network Time Protocol, and PTP is the Precision Time Protocol) is supported for supplementary calibration to improve the robustness in complex environments.

[0069] In another embodiment of the present application, algorithm variation is performed. Dense optical flow calculation is used to replace sparse feature matching and linear interpolation to increase the computational load to improve partial accuracy; an end-to-end neural network is constructed to directly input the original data and output the fusion result to improve the automation level.

[0070] In another embodiment of the present application, cross-modal expansion is performed. An infrared camera or lidar is integrated to achieve multi-modal fusion monitoring of thermal-mechanical coupling or three-dimensional deformation; combined with a wireless transmission module, it supports drone carrying and mobile inspection scenarios.

[0071] The technical advantages of the present application are reflected in the following aspects: High spatio-temporal resolution collaboration: Through the sub-millimeter spatial resolution of the visible light camera (static deformation analysis) and the kilohertz-level time resolution of the DAVIS (dynamic vibration capture), high-precision monitoring of the entire region is achieved; Non-contact measurement: There is no need to contact or mark the object to be measured, avoiding interference to the structure caused by the installation of traditional sensors, especially suitable for sensitive scenarios such as bridges, aircraft, and spacecraft; Dynamic adaptability: Under complex lighting (strong light / weak light) and fast movement conditions, it can still maintain stable data acquisition and processing capabilities; Real-time performance: The system can collect and process data in real time, quickly feedback vibration and displacement information, and is suitable for scenarios that require real-time monitoring, such as vibration monitoring of bridges, buildings, and equipment. Multi-scenario expansion: It supports full-scale monitoring from large infrastructure (such as wind turbine blades) to precision equipment (such as chip packaging machines), covering various working conditions such as static deformation, high-frequency vibration, and transient shock.

[0072] Next, the high spatio-temporal sampling fusion system according to the embodiments of the present application is described with reference to the accompanying drawings.

[0073] Figure 5 It is a structural diagram of the high spatio-temporal sampling fusion system of the embodiments of the present application.

[0074] As Figure 5 shown, the high spatio-temporal sampling fusion system includes: an image acquisition module 100, a spatial alignment module 200, a resolution synchronization module 300, and a data fusion module 400.

[0075] Specifically, the image acquisition module 100 is used to acquire an event stream, a grayscale image, and a visible light image in the same time dimension of the monitoring target. Among them, the event stream and the grayscale image are homologous data, and the resolution of the visible light image is higher than that of the event stream. The spatial alignment module 200 is used to perform feature matching and affine transformation according to the grayscale image and the visible light image to obtain a registered grayscale image. The resolution synchronization module 300 is used to perform event upsampling according to the event stream and the visible light image to obtain a matching event stream, where the resolution of the matching event stream is the same as that of the registered grayscale image. The data fusion module 400 is used to generate monitoring data of the monitoring target according to the registered grayscale image and the matching event stream.

[0076] Figure 6 It is a structural diagram of the terminal provided by the embodiments of the present application. The terminal may include: A memory 501, a processor 502, and a computer program stored on the memory 501 and executable on the processor 502.

[0077] When the processor 502 executes the program, it implements the high spatio-temporal sampling fusion method provided in the above embodiments.

[0078] Furthermore, the terminal further includes: A communication interface 503 for communication between the memory 501 and the processor 502.

[0079] The memory 501 is used to store a computer program executable on the processor 502.

[0080] The memory 501 may include high-speed RAM memory and may also include non-volatile memory, such as at least one disk memory.

[0081] If the memory 501, the processor 502, and the communication interface 503 are implemented independently, the communication interface 503, the memory 501, and the processor 502 can be interconnected through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, an Extended Industry Standard Architecture (EIS) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity of representation, Figure 6 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.

[0082] Optionally, in a specific implementation, if the memory 501, the processor 502, and the communication interface 503 are integrated on a single chip, the memory 501, the processor 502, and the communication interface 503 can communicate with each other through an internal interface.

[0083] The processor 502 may be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0084] This embodiment also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the above high spatio-temporal sampling fusion method is implemented.

[0085] An embodiment of the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the high spatio-temporal sampling fusion method provided in any of the embodiments corresponding to the present application Figure 3 as described in the embodiments.

[0086] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of this application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0087] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of this application, the meaning of "N" is at least two, such as two, three, etc., unless otherwise specifically and clearly defined.

[0088] Any process or method description depicted in a flowchart or otherwise described herein can be understood to represent a module, segment, or portion of code including one or N executable instructions for implementing a customized logic function or process, and the scope of the preferred embodiments of this application includes additional implementations, where the functions can be executed in a manner that may not be in the order shown or discussed, including in a substantially simultaneous manner according to the functions involved or in the reverse order, which should be understood by those skilled in the art to which the embodiments of this application pertain.

[0089] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable storage medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in conjunction with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable storage medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection part (electronic device) having one or N wirings, a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable storage medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically by optically scanning the paper or other media, followed by editing, interpretation, or otherwise processing as appropriate, and then stored in a computer memory.

[0090] It should be understood that various parts of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0091] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the method of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0092] In addition, in each embodiment of the present application, each functional unit can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0093] The above-mentioned storage medium can be a read-only memory, a magnetic disk, an optical disc, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

[0094] It should be understood that the application of the present application is not limited to the above examples. For those of ordinary skill in the art, improvements or transformations can be made according to the above description, and all such improvements and transformations should fall within the protection scope of the appended claims of the present application.

[0095] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent substitutions for some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A high spatio-temporal sampling fusion method, characterized in that The high spatio-temporal sampling fusion method includes: Obtain the event stream, grayscale image, and visible light image of the monitoring target in the same time dimension, where the event stream and the grayscale image are homologous data, and the resolution of the visible light image is higher than that of the event stream; Perform feature matching and affine transformation based on the grayscale image and the visible light image to obtain a registered grayscale image; Perform event upsampling based on the event stream and the visible light image to obtain a matching event stream, where the resolution of the matching event stream is the same as that of the registered grayscale image; Generate monitoring data of the monitoring target based on the registered grayscale image and the matching event stream.

2. The high spatio-temporal sampling fusion method according to claim 1, wherein Before obtaining the event stream, grayscale image, and visible light image of the monitoring target, it further includes: After aligning the global shutter exposure period of the visible light camera with the event sampling time window of the event camera, obtain the event timestamp of the event camera; Compensate for the row exposure transmission delay of the event timestamp to synchronize the visible light camera and the event camera in the data acquisition time dimension.

3. The high spatio-temporal sampling fusion method according to claim 1, characterized in that The specific steps of performing feature matching and affine transformation based on the grayscale image and the visible light image to obtain a registered grayscale image include: Perform feature extraction and matching on the grayscale image and the visible light image to obtain feature matching point pairs; Solve the affine transformation based on the feature matching point pairs to obtain the mapping relationship from the coordinate system of the visible light image to the coordinate system of the grayscale image; According to the mapping relationship, map the coordinates of the visible light image to the coordinate system of the grayscale image to obtain a registered grayscale image.

4. The high spatio-temporal sampling fusion method according to claim 3, wherein The specific steps of solving the affine transformation based on the feature matching point pairs to obtain the mapping relationship from the coordinate system of the visible light image to the coordinate system of the grayscale image include: Calculate the homography matrix based on the feature matching points; The homography matrix is expressed as: ； wherein, is a homography matrix, is the rotation parameter of the first row elements of the homography matrix, is the scaling parameter of the first row elements of the homography matrix, is the translation parameter of the first row elements of the homography matrix, is the rotation parameter of the second row elements of the homography matrix, is the scaling parameter of the second row elements of the homography matrix, is the translation parameter of the second row elements of the homography matrix; Minimize the reprojection error based on the homography matrix to obtain the mapping relationship from the coordinate system of the visible light image to the coordinate system of the grayscale image; The mapping relationship is expressed as: ； Among them, is the abscissa of the pixel point of the registered grayscale image, is the ordinate of the pixel point of the registered grayscale image, is the abscissa of the pixel point of the visible light image, is the ordinate of the pixel point of the visible light image.

5. The high spatio-temporal sampling fusion method according to claim 1, wherein The specific steps of performing event upsampling based on the event stream and the visible light image to obtain a matching event stream include: Perform time window slicing on the event stream according to the period of the visible light image to obtain event frames; Calculate the weight distribution of each event in the spatio-temporal domain based on the event frames to obtain the three-dimensional Gaussian weights corresponding to multiple events in the event frames; Reconstruct the event frames based on the three-dimensional Gaussian weights corresponding to multiple events in the event frames to obtain a matching event stream, where the resolution of the matching event stream is the same as that of the visible light image.

6. The high spatio-temporal sampling fusion method according to claim 5, wherein The formula for the time window slicing is: ； Among them, is the set of events within the th time window, is a single event in the time window, , are the horizontal and vertical position coordinates of the th event, is the time in the th event, is the polarity in the th event, is the trigger period; The formula for the weight distribution is: ； Among them, is a three-dimensional Gaussian weight, , are the horizontal and vertical coordinates in the spatial domain, is the coordinate in the time domain, is the standard deviation of the spatial Gaussian kernel, is the standard deviation of the temporal Gaussian kernel.

7. The high spatio-temporal sampling fusion method according to any one of claims 1-6, characterized in that The specific steps of generating the monitoring data of the monitoring target based on the registered grayscale image and the matching event stream include: Obtain static fusion weights and dynamic fusion weights; Calculate the fusion value at each pixel position based on the static fusion weights, the dynamic fusion weights, the registered grayscale image, and the matching event stream. Based on the fusion values at all pixel positions, monitoring data of the monitored target is obtained.

8. A high spatio-temporal sampling fusion system, characterized in that, The high spatio-temporal sampling fusion system includes: An image acquisition module, configured to acquire an event stream, a grayscale image, and a visible light image in the same time dimension of a monitored target, wherein the event stream and the grayscale image are homologous data, and the resolution of the visible light image is higher than that of the event stream; A spatial alignment module, configured to perform feature matching and affine transformation based on the grayscale image and the visible light image to obtain a registered grayscale image; A resolution synchronization module, configured to perform event upsampling based on the event stream and the visible light image to obtain a matching event stream, wherein the resolution of the matching event stream is consistent with that of the registered grayscale image; A data fusion module, configured to generate monitoring data of the monitored target based on the registered grayscale image and the matching event stream.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and a high spatio-temporal sampling fusion program stored on the memory and executable on the processor. When the high spatio-temporal sampling fusion program is executed by the processor, the steps of the high spatio-temporal sampling fusion method according to any one of claims 1-7 are implemented.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a high spatio-temporal sampling fusion program. When the high spatio-temporal sampling fusion program is executed by a processor, the steps of the high spatio-temporal sampling fusion method according to any one of claims 1-7 are implemented.

Images