Image anti-shake method and device, computer device and storage medium

Abstract

The present disclosure relates to an image anti-shake method, device, system and computer device. Relates to image processing technology, solves the problem that vibration compensation has no pertinence and leads to poor image anti-shake effect. The method comprises: acquiring vibration data in the track running process of the inspection robot in real time, wherein the vibration data comprises three-axis acceleration data and / or angular velocity signal; performing frequency domain analysis on the vibration data to identify vibration events; and generating a corresponding image jitter compensation strategy according to the vibration events. The technical scheme provided by the present disclosure is suitable for a coal mine intelligent inspection system, and realizes an accurate image vibration compensation processing mechanism.

Description

Technical Field

[0001] This disclosure relates to image processing technology, and more particularly to an image stabilization method, apparatus, system, and computer device. Background Technology

[0002] The coal mine environment is uniquely complex, and to improve safety, safety inspection work is gradually developing towards intelligence and automation. Rail-mounted inspection robots are an important component of intelligent coal mine inspection systems. Operating on mine tracks and equipped with image acquisition devices, they provide all-weather environmental monitoring image data. However, due to the special design of mine tracks, inspection robots are often affected by vibrations during operation, leading to a significant decrease in video image quality.

[0003] Specifically, coal mine rail surfaces are typically designed with raised textures to increase the friction of the inspection robot's wheels and prevent slippage. In addition, the rail surface is often covered with dust, coal dust, and other impurities, which can cause vibrations during the inspection robot's movement.

[0004] Image stabilization methods mainly rely on feature point registration or motion compensation based on inertial measurement units (IMUs). However, in complex and multi-source vibration environments such as coal mines, it is difficult to effectively identify the causes of vibration and perform targeted compensation, resulting in noticeable shaking in the video and poor stabilization performance. Summary of the Invention

[0005] To overcome the problems existing in related technologies, this disclosure provides an image stabilization method, apparatus, system, and computer device. By performing frequency domain analysis on vibration data to identify vibration events, and applying corresponding image jitter compensation strategies for different vibration events, an accurate image vibration compensation processing mechanism is achieved, solving the problem of poor image stabilization effect caused by non-targeted vibration compensation.

[0006] According to a first aspect of the embodiments of this disclosure, an image stabilization method is provided, comprising: The vibration data of the inspection robot during its track operation is acquired in real time, and the vibration data includes triaxial acceleration data and / or angular velocity signals; Frequency domain analysis is performed on the vibration data to identify vibration events; Based on the vibration event, a corresponding image jitter compensation strategy is generated.

[0007] Furthermore, the step of performing frequency domain analysis on the vibration data to identify vibration events includes: The preprocessed vibration data is decomposed in the frequency domain to obtain the vibration data of the first frequency band, the vibration data of the second frequency band, and the vibration data of the third frequency band in the third frequency band. The upper limit of the first frequency band is not greater than the lower limit of the second frequency band, and the upper limit of the second frequency band is not greater than the lower limit of the third frequency band. When the vibration data in the first frequency band matches the preset smooth physical motion model, a first type of vibration event indicating normal physical displacement is identified. When periodic peaks are present in the vibration data of the second frequency band, a second type of vibration event indicating low-frequency periodic vibration is identified; In the case of time-domain impact in the vibration data of the third frequency band, a third type of vibration event indicating high-frequency random vibration is identified.

[0008] Furthermore, the step of identifying a second type of vibration event indicating low-frequency periodic vibration when periodic peaks exist in the vibration data of the second frequency band includes: Perform a fast Fourier transform on the vibration data of the second frequency band to extract the fundamental frequency and significant peaks in the spectrum; When the fundamental frequency is stable and lasts for multiple cycles, a periodic peak is determined to exist, and the second type of vibration event is identified.

[0009] Furthermore, the step of identifying a third type of vibration event indicating high-frequency random vibration when a time-domain impact exists in the third frequency band vibration data includes: When the signal amplitude of the vibration data in the third frequency band continuously exceeds a preset amplitude threshold, and the duration of each exceedance of the amplitude threshold is less than a preset duration threshold, it is identified as a time-domain impact event. The time interval between the occurrence of multiple consecutive time-domain shock events is statistically analyzed. The third type of vibration event is identified when the occurrence time intervals are randomly distributed.

[0010] Furthermore, the image jitter compensation strategy includes: For the first type of vibration event, no compensation processing is performed on the corresponding image frame; For the second type of vibration event, the position offset and angle offset at each moment are calculated based on the vibration data of the second frequency band, and a reverse compensation amount is generated. The image frame is then spatially transformed based on the reverse compensation amount. For the third type of vibration event, stable feature points of adjacent image frames are extracted, the displacement vector of the stable feature points is calculated, an affine transformation matrix is ​​obtained by fitting all the stable feature points, and the image frames are compensated by the affine transformation matrix.

[0011] Furthermore, the method also includes: The image jitter compensation strategy is used to perform vibration compensation on the corresponding image frames in the acquired image data. The image data is time-aligned with the vibration data. The image data is an image sequence containing multiple consecutive image frames. Pixel remapping is performed on the vibration-compensated image sequence to repair blank and / or distorted areas caused by vibration compensation.

[0012] Furthermore, the method also includes: The image sequence after vibration compensation or pixel remapping is smoothed by multi-frame fusion.

[0013] According to a second aspect of the embodiments of this disclosure, an image stabilization device is also provided, the device comprising: The vibration data acquisition module is used to acquire vibration data in real time during the operation of the inspection robot's track. The vibration data includes triaxial acceleration data and / or angular velocity signals. The vibration event identification module is used to perform frequency domain analysis on the vibration data to identify vibration events; The strategy generation module is used to generate a corresponding image jitter compensation strategy based on the vibration event.

[0014] According to a third aspect of the embodiments of this disclosure, a computer apparatus is also provided, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to perform the image stabilization method provided in the embodiments of this disclosure.

[0015] According to a fourth aspect of the embodiments of the present disclosure, a non-transitory computer-readable storage medium is also provided, wherein when instructions in the storage medium are executed by a computer processor, the computer is able to perform the image stabilization method provided by the embodiments of the present disclosure.

[0016] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: real-time acquisition of vibration data during the operation of the inspection robot's track, the vibration data including triaxial acceleration data and / or angular velocity signals; then, frequency domain analysis of the vibration data to identify vibration events; and finally, generation of corresponding image jitter compensation strategies based on the vibration events. By subdividing and identifying vibration events based on the scenarios that generate vibration, and proposing targeted image jitter compensation strategies, an accurate image vibration compensation processing mechanism is achieved, solving the problem of poor image stabilization caused by non-targeted vibration compensation.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0019] Figure 1 This is a schematic diagram of the architecture of an image stabilization system according to an exemplary embodiment.

[0020] Figure 2 This is a flowchart illustrating an image stabilization method according to an exemplary embodiment.

[0021] Figure 3 This is a flowchart illustrating yet another image stabilization method according to an exemplary embodiment.

[0022] Figure 4 This is a flowchart illustrating yet another image stabilization method according to an exemplary embodiment.

[0023] Figure 5 This is a flowchart illustrating yet another image stabilization method according to an exemplary embodiment.

[0024] Figure 6 This is a flowchart illustrating yet another image stabilization method according to an exemplary embodiment.

[0025] Figure 7 This is a flowchart illustrating yet another image stabilization method according to an exemplary embodiment.

[0026] Figure 8 This is a block diagram illustrating an image stabilization device according to an exemplary embodiment.

[0027] Figure 9 This is a schematic diagram of the architecture of another image stabilization system according to an exemplary embodiment.

[0028] Figure 10 This is a schematic diagram of the structure of a vibration mode analysis module according to an exemplary embodiment.

[0029] Figure 11 This is a schematic diagram of the structure of a vibration compensation module according to an exemplary embodiment.

[0030] Figure 12 This is a flowchart illustrating yet another image stabilization method according to an exemplary embodiment. Detailed Implementation

[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0032] Image stabilization in coal mine environments faces several challenges, primarily in vibration pattern recognition, algorithm robustness, and real-time processing. These issues limit the quality and stability of image acquisition by rail-mounted inspection robots in practical applications, thus affecting the effectiveness of coal mine safety monitoring.

[0033] 1. High Challenges in Vibration Pattern Recognition: Coal mine tracks have a unique design, with regularly distributed raised textures on the surface. These textures increase the friction of the inspection robot's wheels, preventing slippage and skidding. However, this structure inevitably causes low-frequency periodic vibrations, and these vibration signals exhibit strong regularity. Furthermore, the surface of coal mine tracks is often covered with dust, coal dust, and other impurities. When the robot runs, the wheels generate high-frequency random vibrations as they pass over these impurities. The mixture of low-frequency periodic vibrations and high-frequency random vibrations poses significant challenges to traditional vibration compensation methods. Existing anti-shake algorithms struggle to effectively separate and identify both types of vibrations simultaneously, often failing to achieve accurate compensation, resulting in noticeable image jitter.

[0034] 2. Poor Algorithm Robustness: The coal mine environment itself is extremely complex, with dim or fluctuating lighting conditions, and external interference factors such as dust and suspended coal dust in the air. These pose additional challenges to image stabilization algorithms. Existing image stabilization methods typically rely on feature point detection and motion vector calculation. However, in a coal mine environment, image feature points may be obscured by dust or difficult to detect due to changes in lighting, leading to unstable feature point matching. Furthermore, dynamic changes within the coal mine scene can also disrupt the stability of the stabilization algorithm, resulting in a significant reduction in compensation effectiveness. For example, the accumulation or sudden falling of dust and coal dust may temporarily obstruct the robot's camera field of view, preventing the image shake compensation algorithm from accurately correcting the shake. Therefore, the problem of poor robustness is particularly prominent in complex environments.

[0035] 3. Poor Real-Time Performance: Coal mine inspection tasks typically require highly real-time image acquisition and processing to promptly detect potential safety hazards. However, existing image stabilization technologies, especially those based on deep learning or complex computations, usually require substantial computing resources, making real-time processing difficult under limited hardware conditions. For example, while deep learning models can achieve good stabilization results, their computational complexity is extremely high, especially when processing high-resolution images or long-term sequences, where the algorithm's response speed significantly decreases, failing to meet the demands of real-time monitoring. Furthermore, coal mine rail-mounted inspection robots are typically equipped with low-power embedded processing units, further limiting the application of complex algorithms. Therefore, poor real-time performance has become a major bottleneck for the application of current image stabilization technologies in coal mine scenarios.

[0036] To address the aforementioned problems, embodiments of this disclosure provide an image stabilization method, apparatus, computer device, and storage medium. It utilizes vibration patterns unique to the orbital environment for precise separation and identification, and performs targeted compensation for each vibration pattern. Through frequency domain analysis, low-frequency periodic vibrations and high-frequency random vibrations are separated, and corresponding image jitter compensation strategies are designed based on different vibration characteristics, thereby achieving efficient and accurate image stabilization.

[0037] An exemplary embodiment of this disclosure provides an image stabilization system, such as Figure 1 As shown, it includes a vibration sensing unit 101 installed inside the inspection robot, an image acquisition unit 102 installed on the inspection robot body, and a main control processing unit 103.

[0038] The vibration sensing unit 101 is used to synchronously acquire the vibration signal during the operation of the inspection robot with the image acquisition unit 102, and transmit the vibration signal to the main control processing unit 103.

[0039] According to one exemplary embodiment, the vibration sensing unit 101 is positioned close to the track walking mechanism of the inspection robot, enabling it to collect the real vibration signals generated during the robot's movement to the greatest extent possible, thus avoiding data distortion caused by installation position deviations. In the complex working conditions of underground coal mines, the vibration sensing unit 101 continuously collects dynamic vibration data, and simultaneously marks each set of vibration data with a high-precision timestamp based on the same time system, thereby achieving precise time alignment with image data.

[0040] The image acquisition unit 102 is used to synchronously acquire image signals during the operation of the inspection robot with the vibration sensing unit 101, and transmit the image signals to the main control processing unit 103.

[0041] According to one exemplary embodiment, the image acquisition unit 102 is positioned in an unobstructed area above the inspection robot body, specifically adapted to the unique imaging environment of uneven lighting and pervasive dust in coal mines. Following a time alignment mechanism, a corresponding timestamp is synchronously recorded for each acquired image frame, ensuring that each image frame matches the vibration data at the same moment, thus preventing the stabilization algorithm from failing due to data misalignment at the source.

[0042] The main control processing unit 103 is used to receive the vibration signal transmitted by the vibration sensing unit 101 and the image signal transmitted by the image acquisition unit 102 in real time.

[0043] According to one exemplary implementation, the main control processing unit 103 integrates a complete set of algorithm logic, including data preprocessing, frequency domain decomposition, vibration event recognition, compensation strategy generation, image correction, pixel remapping, and multi-frame fusion, and can process vibration data and image data in parallel.

[0044] Furthermore, the vibration sensing unit 101 includes at least an accelerometer and / or a gyroscope, and the image acquisition unit 102 includes a dual-spectrum camera that supports both visible and infrared spectra.

[0045] According to one exemplary implementation, an accelerometer and a gyroscope work together to form an IMU. The accelerometer is responsible for collecting three-axis linear acceleration data of the inspection robot and quantifying the displacement vibration amplitude of the inspection robot in three dimensions: forward / backward, left / right. The gyroscope is responsible for collecting three-axis angular velocity data of the inspection robot and capturing state changes such as robot posture deflection and angular jitter.

[0046] Dual-spectrum cameras acquire real-world images under normal lighting conditions; unaffected by light intensity, they can perform normal imaging in dark areas, fog, and dusty environments underground using infrared spectral imaging, while simultaneously capturing areas of abnormal equipment temperature for fault thermal hazard detection. Simultaneous imaging by dual-spectrum cameras constructs an all-weather, all-scenario visual monitoring system.

[0047] An exemplary embodiment of this disclosure also provides an image stabilization method applicable to the image stabilization system provided in the embodiments of this disclosure. The process of using this method to correct image jitter during the operation of an inspection robot is as follows: Figure 2 As shown, it includes: Step 201: Acquire vibration data in real time during the operation of the inspection robot track. The vibration data includes triaxial acceleration data and / or angular velocity signals.

[0048] In this step, the raw vibration data is acquired in real time, time-series marked, and transmitted, providing raw data input for subsequent frequency domain analysis and vibration event identification.

[0049] During normal operation of the inspection robot along the coal mine track, the accelerometer and gyroscope in the vibration sensing unit continuously operate at a preset sampling frequency, uninterruptedly collecting three-axis acceleration and three-axis angular velocity data. Affected by the periodic raised texture of the track, coal dust on the track surface, and the robot's own movement, the vibration data exhibits multiple superimposed characteristics. Each set of collected vibration data is bound with a high-precision timestamp and then transmitted in real time to the main control processing unit, where the vibration data is sorted and cached according to time sequence.

[0050] Step 202: Perform frequency domain analysis on the vibration data to identify vibration events.

[0051] In this step, vibration data is preprocessed, converted from time domain to frequency domain, frequency band splitting and feature detection are performed to distinguish between three types of events: normal physical displacement, low-frequency periodic vibration and high-frequency random vibration, and to locate the cause of image jitter.

[0052] The main control processing unit first performs preprocessing on the raw vibration data, using a bandpass filter algorithm to remove invalid frequency components such as underground electromagnetic interference, sensor background noise, and environmental clutter, obtaining a clean and effective vibration signal. Subsequently, the time-domain vibration signal is converted into a frequency-domain spectrum signal using a Fast Fourier Transform (FFT), and multiple frequency intervals are divided based on the vibration characteristics of the coal mine track, completing frequency domain decomposition. For vibration data in different frequency bands, a comprehensive judgment is made by matching motion models, spectral characteristics, and time-domain impact characteristics, accurately distinguishing between normal motion generated by the robot's active movement and two types of harmful vibrations caused by track structure and track impurities.

[0053] Step 203: Generate a corresponding image jitter compensation strategy based on the vibration event.

[0054] In this step, based on the identification results of vibration events, a dedicated image jitter compensation scheme is matched to achieve differentiated processing with a one-to-one correspondence between vibration and image jitter compensation strategies. This avoids problems such as over-compensation, under-compensation, and image distortion caused by uniform compensation, while taking into account both the anti-shake effect and the algorithm's operating efficiency.

[0055] The main control processing unit pre-stores compensation rule libraries and algorithm call interfaces corresponding to three types of vibration events. After completing vibration event identification, the system automatically retrieves the corresponding rules to generate standardized compensation strategies. If the vibration event is identified as the first type (i.e., normal physical displacement event), it is determined that this type of motion will not produce harmful image jitter, and a no-compensation strategy is generated. If the vibration event is identified as the second type (low-frequency periodic vibration event), an inverse displacement compensation strategy is generated. If the vibration event is identified as the third type (i.e., high-frequency random vibration event), an affine transformation compensation strategy based on stable feature points is generated. All image jitter compensation strategies are bound to corresponding timestamps and image frame numbers, achieving a one-to-one matching between image jitter compensation strategies and image frames, ensuring that the optimal processing scheme can be adopted for each frame.

[0056] An exemplary embodiment of this disclosure also provides an image stabilization method, which uses this method to perform frequency domain analysis on vibration data to identify vibration events, as follows: Figure 3 As shown, it includes: Step 301: Perform frequency domain decomposition on the preprocessed vibration data to obtain the first frequency band vibration data in the first frequency band, the second frequency band vibration data in the second frequency band, and the third frequency band vibration data in the third frequency band.

[0057] The upper limit of the first frequency band is not greater than the lower limit of the second frequency band, and the upper limit of the second frequency band is not greater than the lower limit of the third frequency band.

[0058] According to an exemplary implementation, in this step, a multi-level filtering algorithm is used to split the pure vibration signal into frequency bands, and the mixed and superimposed vibration data is split into three groups of independent frequency band vibration data, thus physically isolating the signals from different vibration sources.

[0059] After filtering and denoising, the vibration data retains only the effective signals related to the robot's movement and track. The main control processing unit sequentially activates low-pass, band-pass, and high-pass filters according to preset frequency thresholds to complete three-level frequency domain decomposition. Based on the actual operating conditions of the coal mine track-mounted inspection robot, the following frequency bands are defined: the first band is 0~0.5Hz, where the low-pass filter intercepts the signal in this range, filtering out all frequency components higher than 0.5Hz; the second band is 0.5Hz~5Hz, where the band-pass filter intercepts the signal in this mid-frequency range, isolating signals below 0.5Hz and above 5Hz; the third band is greater than 5Hz, where the high-pass filter intercepts the high-frequency signal, filtering out all frequency components below 5Hz.

[0060] Preferably, the three frequency bands strictly follow a frequency increment rule, with no overlap or gaps between them, ensuring that all valid vibration signals are completely divided into their corresponding intervals. The three sets of frequency band data, after decomposition, are stored independently and then proceed to the next stage of the feature recognition process.

[0061] Step 302: When the vibration data of the first frequency band matches the preset smooth physical motion model, the first type of vibration event indicating normal physical displacement is identified.

[0062] In this step, motion pattern verification is performed on ultra-low frequency band data to distinguish between the active driving action of the inspection robot and weak low-frequency vibrations, thereby identifying the first type of vibration event that does not require compensation (i.e., normal physical displacement event) and reducing invalid calculations.

[0063] According to one exemplary implementation, the main control processing unit has a pre-built smooth physical motion model, which includes the acceleration and angular velocity variation patterns corresponding to all conventional driving postures such as straight-line driving, climbing, descending, turning, changing tracks, and U-turns. Simultaneously, it combines on-site track layout information to complete parameter calibration. The system performs integral calculations on the vibration data in the first frequency band, deducing the real-time velocity and displacement parameters of the inspection robot from the acceleration, and the attitude deflection angle parameters from the angular velocity. Then, the measured parameters are compared with the preset smooth physical motion model, and the matching error between the two is calculated. When the matching error is less than the system's preset error threshold, the current signal is determined to be motion generated by the robot's normal driving, identified as a first-type vibration event, which does not require image jitter compensation processing.

[0064] Step 303: In the case that there are periodic peaks in the vibration data of the second frequency band, identify a second type of vibration event that indicates low-frequency periodic vibration.

[0065] In this step, frequency domain feature detection is performed on the mid-frequency band data to identify persistent low-frequency vibrations caused by periodic ridges in the track, and harmful vibration events requiring specific compensation are marked. The specific process is as follows: Figure 4 As shown, it includes: Step 401: Perform a fast Fourier transform on the vibration data of the second frequency band to extract the fundamental frequency and significant peaks in the spectrum.

[0066] In this step, the mid-frequency vibration data is subjected to refined frequency domain analysis to extract the core spectral feature parameters, providing data basis for judging periodic vibration.

[0067] The second frequency band (0.5Hz~5Hz) is the main frequency band of vibration generated by the robot wheel rolling over the periodic raised texture of the track. The system performs FFT again on the vibration data of this second frequency band to generate a refined spectrum. Through spectrum analysis algorithms, the fundamental frequency component and significant peaks with prominent amplitudes in the spectrum are accurately extracted, and key parameters such as fundamental value, peak position, and peak amplitude are recorded. This can amplify the frequency domain characteristics of mid-frequency vibrations, transforming periodic fluctuations that are difficult to distinguish in the time domain into intuitive spectrum features, thus improving recognition accuracy.

[0068] Step 402: When the fundamental frequency is stable and lasts for multiple cycles, it is determined that there is a periodic peak, and the second type of vibration event is identified.

[0069] In this step, low-frequency periodic vibrations are detected and determined based on the stability and persistence characteristics of the fundamental frequency and peak value of the spectrum, thus completing the identification and labeling of the second type of vibration event.

[0070] With a fixed spacing between the raised textures on the track and a relatively stable robot speed, the fundamental frequency of this type of vibration remains constant over a long period, and the position and amplitude of the spectral peaks do not change significantly. The system continuously monitors the fundamental frequency and peak states. If the fundamental frequency remains stable and the periodic peak can be maintained for multiple vibration cycles, it is determined that a stable periodic peak exists in this frequency band, confirming that the current vibration is a low-frequency periodic vibration caused by the track textures, and is identified as a type II vibration event. This type of vibration persists throughout the robot's entire inspection process, causing continuous translation and rotation of the image, and is the most significant source of image jitter in coal mine scenarios, necessitating targeted compensation.

[0071] Step 304: In the case of time-domain impact in the vibration data of the third frequency band, identify a third type of vibration event indicating high-frequency random vibration.

[0072] In this step, time-domain feature analysis is performed on high-frequency band data to identify instantaneous impact vibrations caused by track dust, coal dust, and small debris. The specific process is as follows: Figure 5 As shown, it includes: Step 501: When the signal amplitude of the vibration data in the third frequency band continuously exceeds a preset amplitude threshold, and the duration of each exceedance of the amplitude threshold is less than a preset duration threshold, it is identified as a time-domain impact event.

[0073] In this step, high-frequency signals are initially screened, invalid high-frequency noise is eliminated, and the occurrence point of a single instantaneous impact jitter is located.

[0074] The third frequency band (greater than 5Hz) exhibits rapid signal changes, encompassing both effective impact vibrations caused by track impurities and ineffective high-frequency interference such as sensor self-noise and equipment electromagnetic clutter. The system pre-sets fixed amplitude and duration thresholds. The amplitude threshold distinguishes between effective vibrations and weak noise, while the duration threshold distinguishes between short-duration impacts and continuous high-frequency vibrations. The system iterates through the third frequency band vibration data point by point, starting a timer when the signal amplitude continuously exceeds the amplitude threshold. If the duration of the high-amplitude state is less than the preset duration threshold, it is identified as a time-domain impact event, and the start and end times, peak amplitude, and other information of the event are recorded. If the duration exceeds the threshold, it is marked as abnormal continuous vibration and triggers a separate alarm.

[0075] Step 502: Calculate the time interval between the occurrence of multiple consecutive time-domain impact events.

[0076] In this step, time-series data of multiple time-domain impact events are collected to form a statistical sequence of time intervals, providing statistical samples for judging the vibration distribution pattern.

[0077] During the inspection robot's operation, the coal dust and particles on the track surface are scattered and unevenly distributed, generating multiple time-domain impact events. The system continuously records the start time of each time-domain impact event, calculates the time interval between two adjacent time-domain impact events, and generates a statistical sequence of time intervals in real time. The statistical process is performed synchronously with image acquisition and data processing, without interrupting normal business processes.

[0078] Step 503: When the occurrence time intervals are randomly distributed, the third type of vibration event is identified.

[0079] In this step, based on the distribution characteristics of time intervals, the type of high-frequency vibration is finally determined, thus completing the identification of high-frequency random vibration events.

[0080] The system analyzes the distribution patterns of the time interval statistical sequence. If the time intervals between adjacent time-domain impact events are of varying lengths, have no fixed period, and exhibit no repetitive patterns, displaying typical random distribution characteristics, it proves that this type of vibration is caused by the crushing of scattered random obstacles such as dust and coal dust, rather than periodic vibrations caused by fixed structures. At this point, the system formally identifies it as a third type of vibration event, namely high-frequency random vibration. This type of vibration has a short duration and unpredictable timing, causing instantaneous and violent shaking and shifting of the image. It belongs to sudden and harmful jitter and can be compensated using strategies adapted to instantaneous jitter.

[0081] This disclosure also provides an image stabilization method, the process of which uses this method to perform an image shake compensation strategy to compensate for image vibration is as follows: Figure 6 As shown, it includes: Step 601: Acquire vibration data during the real-time operation of the inspection robot's track. The vibration data includes triaxial acceleration data and / or angular velocity signals.

[0082] Step 602: Perform frequency domain analysis on the vibration data to identify vibration events.

[0083] Step 603: Generate a corresponding image jitter compensation strategy based on the vibration event.

[0084] It should be noted that the implementation principles of steps 601 to 603 are the same as those of steps 201 to 203, and will not be repeated here.

[0085] Step 604: Use the vibration compensation strategy to perform vibration compensation on the corresponding image frames in the acquired image data.

[0086] The image data is time-aligned with the vibration data, and the image data is an image sequence containing multiple consecutive image frames.

[0087] In this step, the corresponding algorithm is invoked according to the preset compensation strategy to perform pixel-level correction on the time-aligned image frames, thereby offsetting the image shift, rotation, and deformation caused by different types of vibration and eliminating image jitter from the root.

[0088] The image vibration compensation strategy includes: 1. For the first type of vibration event, no compensation processing is performed on the corresponding image frame.

[0089] Normal physical displacements of the robot, such as straight-line movement, climbing slopes, and turning, constitute stable changes in the overall viewing angle and are not harmful jitters. Forcibly performing compensation operations would disrupt the normal field of view, causing image stuttering and viewing angle shifts. Therefore, when a vibration event is identified as a Type I vibration event, the system skips all image compensation calculations, and the original image frame is directly forwarded. This rule significantly reduces the computational load of the algorithm and improves the overall operating efficiency of the system.

[0090] 2. For the second type of vibration event, calculate the position offset and angle offset at each moment based on the vibration data of the second frequency band, generate the reverse compensation amount, and perform spatial transformation on the image frame based on the reverse compensation amount.

[0091] Low-frequency periodic vibrations cause continuous and regular translational and rotational shifts in image frames. Therefore, a reverse displacement compensation algorithm is used for correction. The system extracts real-time vibration data in the second frequency band and, combined with vibration amplitude and period, calculates the horizontal and vertical positional offsets and angular offsets of the image frame by frame. Based on the offsets, it calculates an equal-sized but opposite-direction reverse compensation amount. This reverse compensation amount is then used to perform an overall spatial transformation on the original image frame, completely offsetting the image shifts and rotations caused by periodic vibrations. Leveraging the periodic characteristics of the vibrations, the algorithm can predict the compensation parameters for the next frame, further improving computational speed and meeting real-time requirements.

[0092] 3. For the third type of vibration event, extract stable feature points from adjacent image frames, calculate the displacement vector of the stable feature points, fit an affine transformation matrix based on all the stable feature points, and compensate the image frames using the affine transformation matrix.

[0093] High-frequency random vibration is a type of instantaneous impact jitter, with no fixed pattern in direction or amplitude. A feature-point-based affine transformation algorithm is used for correction. The system extracts stable, interference-resistant feature points such as SIFT and ORB from the current and previous frames. These feature points remain recognizable even under dust obstruction and varying lighting conditions. A matching operation is performed on the two sets of feature points, calculating the displacement vector for each set to quantify the inter-frame jitter state. Then, all displacement vectors are integrated, and a global affine transformation matrix is ​​obtained through least-squares fitting, uniformly representing the jitter pattern of the entire image. Finally, the affine transformation matrix is ​​used to perform global correction on the image, eliminating instantaneous random jitter.

[0094] Step 605: Perform pixel remapping on the vibration-compensated image sequence to repair the blank and / or distorted areas caused by vibration compensation.

[0095] In this step, issues such as edge blanking, pixel misalignment, and image distortion caused by spatial transformation and affine transformation of the image are repaired to ensure the integrity and image quality consistency of a single frame.

[0096] During translation, rotation, and affine transformations, blank areas without pixel coverage will appear at the edges of the original image, and some pixels will experience coordinate misalignment and edge stretching distortion. All image frames that have undergone low-frequency and high-frequency vibration compensation need to perform pixel remapping operations; image frames corresponding to the first type of normal physical displacement do not need to perform this step.

[0097] According to one exemplary implementation, the system generates a global remapping table based on the theoretical pixel coordinates of the transformed image, and performs pixel-by-pixel relocation based on the remapping table to correct misaligned pixels. For blank areas, a bilinear interpolation algorithm is used to fill pixels, ensuring natural pixel transitions in the filled areas without jagged edges or color blocks. Finally, distorted edges are smoothed. This step simultaneously utilizes optimization techniques such as image block processing and GPU parallel computing to control computation time while ensuring image quality.

[0098] An exemplary embodiment of this disclosure also provides an image stabilization method, which, after vibration compensation, further smooths the image to obtain a better display effect, as follows: Figure 7 As shown, it includes: Step 701: Acquire vibration data during the real-time operation of the inspection robot's track. The vibration data includes triaxial acceleration data and / or angular velocity signals.

[0099] Step 702: Perform frequency domain analysis on the vibration data to identify vibration events.

[0100] Step 703: Generate a corresponding image jitter compensation strategy based on the vibration event.

[0101] Step 704: Use the vibration compensation strategy to perform vibration compensation on the corresponding image frames in the acquired image data.

[0102] Step 705: Perform pixel remapping on the vibration-compensated image sequence to repair the blank and / or distorted areas caused by vibration compensation.

[0103] Steps 701 to 705 are implemented in the same way as steps 601 to 605, and will not be described again here.

[0104] Step 706: Smooth the image sequence after vibration compensation or pixel remapping by multi-frame fusion.

[0105] In this step, multi-frame fusion is used to eliminate minor jumps, residual micro-jitter, brightness fluctuations, and other imperfections between consecutive image frames, thereby improving the overall smoothness and visual comfort of the video sequence, completing the final optimization of image stabilization, and outputting the finished image sequence.

[0106] After compensation and pixel remapping, the single-frame image has eliminated significant jitter. However, due to factors such as algorithm calculation errors, instantaneous changes in vibration, and slight fluctuations in downhole lighting, subtle jumps and residual jitter may still exist between consecutive frames. Therefore, this step introduces a multi-frame weighted fusion algorithm to perform global smoothing optimization.

[0107] According to one exemplary implementation, the system can dynamically set the number of fusion frames N according to the hardware load. N is a positive integer, typically ranging from 3 to 5 frames. The current frame and the N-1 adjacent frames are selected to form a fusion image group. Then, a weighted average algorithm is used to perform fusion calculations on the images within the group, assigning the highest weight to the central current frame, with the weights of adjacent image frames decreasing sequentially. This preserves the core image information of the current frame while utilizing the data from adjacent image frames to neutralize inter-frame jumps and residual jitter.

[0108] After multi-frame fusion is completed, the final image sequence can be output in two ways: one is transmitted in real time to the ground remote monitoring terminal for inspection personnel to view the underground working conditions in real time; the other is stored locally in the robot's built-in memory as historical inspection data archive.

[0109] Simultaneously, the system records information such as vibration event type, compensation strategy, algorithm operating parameters, and image quality status, generating a complete operation log. Under extreme conditions of high dust and strong vibration, the multi-frame fusion step can significantly improve video continuity and further enhance the robustness of the algorithm in complex environments.

[0110] An exemplary embodiment of this disclosure also provides an image stabilization device, applicable to the image stabilization system provided in the embodiments of this disclosure, the structure of which is as follows: Figure 8 As shown, it includes: The vibration data acquisition module 801 is used to acquire vibration data in real time during the operation of the inspection robot track. The vibration data includes triaxial acceleration data and / or angular velocity signals. The vibration event identification module 802 is used to perform frequency domain analysis on the vibration data to identify vibration events; The strategy generation module 803 is used to generate a corresponding image jitter compensation strategy based on the vibration event.

[0111] The image stabilization device consists of three modules operating in series in a closed-loop logic. It also integrates a global real-time performance optimization subroutine, employing GPU parallel computing, image block processing, and dynamic threshold adjustment to significantly reduce algorithm computational complexity. The device is compatible with low-power embedded hardware for coal mine rail-mounted inspection robots, enabling stable, long-term operation in complex underground environments characterized by darkness, high dust levels, and strong vibrations, fully meeting the functional, real-time, and robust requirements of image stabilization.

[0112] The aforementioned device can be integrated into the main control processing unit, remote monitoring system, and on-site monitoring equipment of the inspection robot, with the corresponding functions implemented by the entity. Regarding the device in the above embodiments, the specific methods by which each module performs its operations have been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0113] An exemplary embodiment of this disclosure also provides an image stabilization system, the architecture of which is as follows: Figure 9 As shown, it includes: Vibration sensing and data acquisition module: The core is an IMU composed of an accelerometer and a gyroscope, which is used to collect three-axis acceleration and angular velocity information in real time during the operation of the inspection robot. At the same time, it marks each set of data with a high-precision timestamp to achieve time alignment with the image data.

[0114] Image acquisition module: Equipped with a dual-spectrum camera, it acquires image data including visible light and infrared spectra for subsequent image stability analysis and vibration compensation processing, and can be combined with IMU data for collaborative analysis.

[0115] Data preprocessing module: Used to preprocess vibration data and image data, including data filtering, signal denoising and image quality enhancement; at the same time, it aligns IMU data and image data in time so that the two data sources are aligned during analysis.

[0116] Vibration mode analysis module: such as Figure 10As shown, the vibration mode analysis module includes a normal physical displacement recognition module, a low-frequency periodic vibration recognition module, a high-frequency random vibration recognition module, and a vibration classification and marking module. It is used to analyze the motion mode and image vibration mode of the inspection robot, distinguish between normal physical displacement, low-frequency periodic vibration and high-frequency random vibration, and identify the first type of vibration event, the second type of vibration event and the third type of vibration event.

[0117] Vibration compensation module: such as Figure 11 As shown, the system includes a compensation strategy selection unit, an image correction and transformation unit, an image remapping unit, and a real-time performance optimization unit. It is the execution unit of the image stabilization system and is responsible for performing targeted compensation on the acquired video images based on the vibration mode analysis results.

[0118] Image post-processing module: Smooths the image sequence after jitter compensation to further improve image quality and eliminate residual compensation errors and inter-frame jumps.

[0119] The vibration sensing unit is installed near the robot's track running mechanism to collect vibration signals caused by environmental factors such as track texture, dust, or coal dust in real time; the image acquisition module is installed above the imaging unit on the robot body to collect video data generated during the inspection process; the main control processing unit is installed in the robot's host to execute vibration mode analysis and compensation algorithms; the execution unit is responsible for controlling image remapping and post-processing operations; and the remote monitoring terminal is located at the ground control center to receive and display the final output results after image processing.

[0120] Based on the above image stabilization system, this disclosure also provides an image stabilization method. The process of using this method to correct image jitter during the operation of an inspection robot is as follows: Figure 12 As shown, it includes: 1. Vibration data acquisition and preprocessing Accelerometers and gyroscopes are used to collect three-axis acceleration and angular velocity data in real time during the operation of the inspection robot, and timestamps are recorded to synchronize with image frames.

[0121] The collected vibration data is bandpass filtered to remove environmental noise and unnecessary frequency components, providing clean data input for subsequent analysis.

[0122] Let the acceleration signal be Using a bandpass filter The filtered signal is Calculate the filtered signal according to the following expression. :

[0123] in, The meaning is the original acceleration signal; The meaning is Fourier transform operator; The meaning is the frequency response function of the bandpass filter; The meaning is the inverse Fourier transform operator; The meaning is the acceleration signal after bandpass filtering.

[0124] 2. Image Acquisition and Preprocessing A dual-spectrum camera is used to acquire continuous image frames, and the acquired image signals are preliminarily processed, such as image denoising, contrast enhancement, and gamma correction, to improve image quality and the stability of subsequent processing, and to obtain the image data to be processed.

[0125] Let the image signal be Denoising image Through the filter The result is obtained through processing. Specifically, it can be calculated using the following expression:

[0126] in, The meaning is the original image signal; The meaning is image denoising filter; The meaning is convolution operation; The meaning is the image signal after denoising.

[0127] It should be noted that image acquisition and vibration data acquisition can be performed simultaneously, with each set its own parameters such as acquisition period and data structure. Image data (such as image signals) and vibration data are synchronized in time, and time correlation is achieved through timestamps.

[0128] 3. Vibration event identification.

[0129] In the vibration mode analysis module, the filtered acceleration is first analyzed. and angular velocity Frequency domain analysis, such as performing a Fast Fourier Transform (FFT), can yield the spectral distribution of the vibration signal. and Based on the spectral characteristics, a threshold cutoff point is set. To separate low-frequency and high-frequency components, in practice, a low-pass filter can be used to extract the low-frequency signal, and a high-pass filter can be used to separate the high-frequency signal. After separation, specific event identification can be performed.

[0130] Normal physical displacement (such as straight-line travel, climbing, descending, turning, changing tracks, and U-turns) typically manifests as smooth, continuous changes over a relatively long period, specifically including: Low frequency and large amplitude changes: Acceleration and angular velocity change slowly and smoothly, with low frequency and long duration (such as the acceleration and deceleration process when climbing a hill or turning).

[0131] Consistent attitude changes: The angular velocity signal measured by the gyroscope shows a continuous and consistent angle change (such as when turning or turning around).

[0132] Steady-state trend: By integrating the acceleration data, the velocity and displacement trends can be obtained. If these trends conform to the expected smooth physical motion model (such as slow linear acceleration or uniform motion), they can be identified as normal physical displacements, thus identifying a Type I vibration event.

[0133] Based on the above characteristics, signals with frequencies below a certain threshold (e.g., 0.5 Hz) are retained. Then, based on the track information and the robot's motion model, the detected low-frequency changes are matched with preset physical motion patterns (such as climbing or turning around) to verify whether they belong to normal motion. If they belong to normal motion, they are marked as normal images and no jitter compensation is applied. The specific matching method is as follows: First, based on the trajectory information and the robot's kinematic model, the translational displacement and rotational angle of normal motion are predicted, denoted as follows: and Specifically, it can be expressed as:

[0134] in, The meaning is The normal translational displacement predicted at any given time; The meaning is a time variable; The meaning is the acceleration signal after low-pass filtering; , The meaning is the integral variable; The meaning is The predicted normal motion rotation angle at any given time; The meaning is the angular velocity signal after low-pass filtering.

[0135] Then record the actual displacement value measured by the sensor. and rotation angle Then, it is matched with the preset normal motion mode of the track, and the matching error is defined as:

[0136] in, The meaning is Matching error of translational displacement at any given time; The meaning is The actual translational displacement measured by the time sensor; The meaning is The normal translational displacement predicted at any given time; The meaning is Matching error of rotation angle at any given time; The meaning is The actual rotation angle measured by the time sensor; The meaning is The normal rotation angle predicted at any given time.

[0137] Then the matching error is compared with the defined error threshold. and Normal movement is determined when the following conditions are met:

[0138] in, The meaning is the norm of the vector (used to quantize the magnitude of the error); The meaning is the preset threshold for translational displacement matching error; The meaning is the preset threshold for rotation angle matching error.

[0139] Low-frequency periodic vibrations are caused by the patterns on the track and have certain frequency characteristics. They are typically concentrated in the frequency range of 0.5 Hz to 5 Hz and have significant peaks in the spectrum. After identifying these periodic peaks, it can be determined that the vibration is caused by track patterns rather than normal physical displacement. In other words, by detecting periodic patterns in the signal, if continuous periodic fluctuations are found, it is judged as low-frequency periodic vibration.

[0140] High-frequency random vibration is a transient vibration caused by dust, coal dust, etc., characterized by high frequency and short duration. It can be identified by analyzing the duration and amplitude changes of the high-frequency signal. Random vibration signals have large amplitudes but short durations. The occurrence intervals of high-frequency signals can be statistically analyzed. Does the time difference between the start of two high-frequency events conform to a random distribution? What is the duration of the high-frequency jitter when the amplitude exceeds a threshold? To further determine whether it is high-frequency random vibration.

[0141] 4. Low-frequency vibration compensation.

[0142] Vibration events (normal physical displacement, low-frequency periodic vibration, and high-frequency random vibration) received from the vibration mode analysis module are analyzed, and appropriate image jitter compensation strategies are selected based on the category. For the first type of vibration event (normal physical displacement), no image jitter compensation is performed; for the second type of vibration event (low-frequency periodic vibration) and the third type of vibration event (high-frequency random vibration), different image jitter compensation strategies are selected.

[0143] For the second type of vibration event, a reverse displacement compensation method is used to compensate for low-frequency periodic vibration. The calculation is performed at each moment under low-frequency vibration. Position offset and angular offset And generate a reverse compensation amount:

[0144] in, The meaning is The amount of reverse compensation for the translational displacement at any given time; The meaning is The translational displacement caused by low-frequency vibration at any given moment; The meaning is The amount of reverse compensation for the rotation angle at any given time; The meaning is The rotational angle offset caused by low-frequency vibration at any given time.

[0145] When updating an image frame, the image position and angle are adjusted by a compensation amount according to the following expression:

[0146] in, The meaning is the image signal after low-frequency vibration compensation; The meaning is the original image signal after denoising; , The meaning is the horizontal and vertical coordinates of the image pixels; The meaning is the reverse compensation amount in the horizontal direction; The meaning is the reverse compensation amount in the vertical direction.

[0147] Image stabilization is achieved by adjusting the position of the image frame to cancel out the displacement error caused by low-frequency vibration.

[0148] 5. High-frequency vibration compensation.

[0149] For the third type of vibration event, a feature-point-based image matching method is used to select stable feature points (such as SIFT or ORB features) for motion estimation and jitter correction.

[0150] Let the feature point set of two frames be... and Calculate the displacement vector of the feature point according to the following expression. :

[0151] in, The meaning is the first Inter-frame displacement vectors of group-matched feature points; The meaning is the first The pixel coordinate vector of each feature point in the current frame to be compensated; The meaning is the first The pixel coordinate vector of each feature point in the previous reference frame; The meaning is the index number of the feature point.

[0152] The global jitter displacement is obtained by combining the displacement vectors of all feature points, and then the corresponding affine transformation matrix is ​​applied. Image compensation:

[0153] in, The meaning is the image signal after high-frequency vibration compensation; The meaning is the global affine transformation matrix; The meaning is matrix multiplication and image multiplication. The meaning is the original image signal after denoising.

[0154] The displacement vector of the feature point is obtained by calculating the least squares method, and the calculation expression is as follows:

[0155] in, The meaning is the global affine transformation matrix (here it is in the form of translation transformation, corresponding to the main jitter form of high-frequency random vibration); The meaning is the horizontal global jitter displacement caused by high-frequency vibration; The meaning is the vertical global jitter displacement caused by high-frequency vibration.

[0156] Finally, the image remapping unit remaps the compensated image, calculates the remapping table based on the corrected pixel coordinates, maps the corrected pixels to the target position, and then uses bilinear interpolation to fill the blank areas of the image to generate the final remapped image.

[0157] To reduce algorithm complexity and minimize computational resource usage, a real-time performance optimization unit is introduced throughout the entire process. Its purpose is to optimize the operational efficiency of each submodule. This includes parallel computing (using GPUs to parallelize FFT analysis, feature point extraction, and image transformation operations), block processing (dividing the image into multiple sub-regions, calculating compensation amounts separately, and then stitching the results together), and threshold optimization (dynamically adjusting the threshold of the compensation strategy to adapt to different vibration intensities). This significantly improves the algorithm's response speed, meeting the requirements of real-time monitoring and making it suitable for low-power embedded processing units. This further promotes the industrial application of rail-mounted robots.

[0158] 6. Image post-processing.

[0159] To further improve image smoothness, a multi-frame fusion algorithm is used to smooth adjacent image frames in the image sequence, reducing jumps during the compensation process. A weighted average method is used to fuse adjacent frames, and the smoothed image sequence is obtained according to the following expression. :

[0160] in, The meaning is the coordinates of the image pixels. The meaning is the total number of adjacent image frames participating in the fusion; The meaning is the index number of the fused frame; The meaning is the first Image signal after frame vibration compensation.

[0161] An exemplary embodiment of this disclosure also provides a computer apparatus, including: processor; Memory used to store processor-executable instructions; The processor is configured to perform the image stabilization method provided in the embodiments of this disclosure.

[0162] An exemplary embodiment of this disclosure also provides a non-transitory computer-readable storage medium, wherein instructions in the storage medium, when executed by a computer's processor, enable the computer to perform the image stabilization method provided by embodiments of this disclosure.

[0163] This disclosure provides an image stabilization method, apparatus, system, and computer device. It acquires vibration data in real time during the operation of an inspection robot on its track. The vibration data includes triaxial acceleration data and / or angular velocity signals. Then, it performs frequency domain analysis on the vibration data to identify vibration events. Based on the vibration events, it generates corresponding image jitter compensation strategies. By subdividing and identifying vibration events based on the vibration scenario and proposing targeted image jitter compensation strategies, it achieves an accurate image vibration compensation processing mechanism, solving the problem of poor image stabilization effect caused by non-targeted vibration compensation.

[0164] The vibration identification method based on frequency domain analysis and a track vibration mode library accurately distinguishes between normal physical displacement, low-frequency periodic vibration, and high-frequency random vibration. Through a specially designed inverse displacement compensation algorithm and local feature point correction method in the vibration compensation module, the method effectively avoids miscompensation or insufficient compensation, ensuring accurate processing of vibration signals and achieving high-precision vibration identification and separation.

[0165] The system employs optimized bandpass filters, frequency domain analysis, and multi-source data fusion strategies to improve processing efficiency and meet the high real-time requirements of coal mine inspection. Simultaneously, through anti-interference image compensation strategies and adaptive feature point selection, the system can operate stably even in complex mine environments with high dust levels, demonstrating excellent robustness.

[0166] Through the coordinated operation of vibration compensation and image remapping units, the system significantly eliminates image jitter and distortion caused by vibration, improving the clarity and consistency of inspection videos. Combined with image post-processing using multi-frame fusion technology, it provides high-quality visual data support for coal mine safety inspections, further enhancing the reliability of fault identification and early warning.

[0167] Those skilled in the art will also understand that the various illustrative logical blocks and steps listed in the embodiments of this disclosure can be implemented by electronic hardware, computer software, or a combination of both. Whether such functionality is implemented in hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functionality using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of this disclosure.

[0168] Furthermore, the term “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as advantageous compared to other aspects or designs. Rather, the use of the term “exemplary” is intended to present the concept in a concrete manner. As used herein, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from the context, “X applies A or B” is intended to mean any of the natural inclusive arrangements. That is, “X applies A or B” satisfies any of the foregoing instances if X applies A; X applies B; or both X applies A and B. Additionally, unless otherwise specified or clear from the context to refer to the singular form, the articles “a” and “an” as used in this application and the appended claims are generally understood to mean “one or more.”

[0169] Similarly, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding this specification and the accompanying drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terminology used to describe such components is intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if structurally not equivalent to the disclosed structure. Furthermore, although specific features of this disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations, as may be desired and advantageous to any given or particular application. Moreover, with regard to the terms “comprising,” “owning,” “having,” “having,” or variations thereof as used in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term “including.”

[0170] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0171] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image stabilization method, characterized in that, include: The vibration data of the inspection robot during its track operation is acquired in real time, and the vibration data includes triaxial acceleration data and / or angular velocity signals; Frequency domain analysis is performed on the vibration data to identify vibration events; Based on the vibration event, a corresponding image jitter compensation strategy is generated.

2. The image stabilization method according to claim 1, characterized in that, The step of performing frequency domain analysis on the vibration data to identify vibration events includes: The preprocessed vibration data is decomposed in the frequency domain to obtain the vibration data of the first frequency band, the vibration data of the second frequency band, and the vibration data of the third frequency band in the third frequency band. The upper limit of the first frequency band is not greater than the lower limit of the second frequency band, and the upper limit of the second frequency band is not greater than the lower limit of the third frequency band. When the vibration data in the first frequency band matches the preset smooth physical motion model, a first type of vibration event indicating normal physical displacement is identified. When periodic peaks are present in the vibration data of the second frequency band, a second type of vibration event indicating low-frequency periodic vibration is identified; In the case of time-domain impact in the vibration data of the third frequency band, a third type of vibration event indicating high-frequency random vibration is identified.

3. The image stabilization method according to claim 2, characterized in that, The step of identifying a second type of vibration event indicating low-frequency periodic vibration when there is a periodic peak in the vibration data of the second frequency band includes: Perform a fast Fourier transform on the vibration data of the second frequency band to extract the fundamental frequency and significant peaks in the spectrum; When the fundamental frequency is stable and lasts for multiple cycles, a periodic peak is determined to exist, and the second type of vibration event is identified.

4. The image stabilization method according to claim 2, characterized in that, The step of identifying a third type of vibration event indicating high-frequency random vibration when there is a time-domain impact in the vibration data of the third frequency band includes: When the signal amplitude of the vibration data in the third frequency band continuously exceeds a preset amplitude threshold, and the duration of each exceedance of the amplitude threshold is less than a preset duration threshold, it is identified as a time-domain impact event. The time interval between the occurrence of multiple consecutive time-domain shock events is statistically analyzed. The third type of vibration event is identified when the occurrence time intervals are randomly distributed.

5. The image stabilization method according to claim 2, characterized in that, The image jitter compensation strategy includes: For the first type of vibration event, no compensation processing is performed on the corresponding image frame; For the second type of vibration event, the position offset and angle offset at each moment are calculated based on the vibration data of the second frequency band, and a reverse compensation amount is generated. The image frame is then spatially transformed based on the reverse compensation amount. For the third type of vibration event, stable feature points of adjacent image frames are extracted, the displacement vector of the stable feature points is calculated, an affine transformation matrix is ​​obtained by fitting all the stable feature points, and the image frames are compensated by the affine transformation matrix.

6. The image stabilization method according to claim 2, characterized in that, The method further includes: The image jitter compensation strategy is used to perform vibration compensation on the corresponding image frames in the acquired image data. The image data is time-aligned with the vibration data. The image data is an image sequence containing multiple consecutive image frames. Pixel remapping is performed on the vibration-compensated image sequence to repair blank and / or distorted areas caused by vibration compensation.

7. The image stabilization method according to claim 6, characterized in that, The method further includes: The image sequence after vibration compensation or pixel remapping is smoothed by multi-frame fusion.

8. An image stabilization device, characterized in that, The device includes: The vibration data acquisition module is used to acquire vibration data in real time during the operation of the inspection robot's track. The vibration data includes triaxial acceleration data and / or angular velocity signals. The vibration event identification module is used to perform frequency domain analysis on the vibration data to identify vibration events; The strategy generation module is used to generate a corresponding image jitter compensation strategy based on the vibration event.

9. A computer device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to perform the image stabilization method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of a computer, the computer is able to perform the image stabilization method as described in any one of claims 1 to 7.

Images