A visual data preprocessing method for double-ball nested rolling robot

Abstract

The present application relates to the technical field of robot vision data processing, and discloses a kind of for double ball nesting type rolling robot visual data preprocessing method, comprising: collecting robot vision image data and attitude sensing data and pre-processing;Based on attitude data, identify the rolling motion state of sphere, output motion blur kernel parameter and attitude angle sequence;Rolling geometry compensation and adaptive deblurring processing are executed, and the image frame after correction deblurring is output;After correction image is carried out regional exposure equalization and direction region of interest enhancement, and output enhanced image sequence;After interframe registration and time sequence smoothing processing, output standardization vision data, and registration residual error is fed back to optimize motion blur estimation.The present application is aimed at the spherical surface rolling characteristics of double ball nesting type rolling robot, and realizes hierarchical preprocessing by fusing attitude and vision data, effectively solves the problems of picture rotation, motion blur and field fluctuation caused by rolling, and significantly improves the quality and stability of vision data.

Description

Technical Field

[0001] This invention relates to the field of robot vision data processing technology, and in particular to a vision data preprocessing method for a nested double-ball rolling robot. Background Technology

[0002] The nested double-sphere rolling robot employs a single-body structure with two spheres nested together, relying on an internal drive mechanism to achieve overall rolling movement. It features flexible steering and strong terrain adaptability. These robots typically incorporate visual sensors for environmental perception, autonomous navigation, and target recognition. However, due to the robot's spherical rolling shape, the camera rotates and pitches synchronously with the spheres during movement, resulting in severe issues with periodic image rotation, motion blur, and field-of-view fluctuations in the acquired visual data.

[0003] Existing visual data preprocessing methods are mostly designed for smooth motion scenarios of wheeled and legged robots, primarily employing general denoising and equalization processing. They fail to incorporate the spherical motion characteristics and IMU attitude information of rolling robots, thus failing to effectively compensate for the geometric deformation and motion blur caused by the rolling of the sphere. Furthermore, they lack mechanisms for addressing spherical shell reflection and differential enhancement of the travel area, resulting in preprocessed images still exhibiting blurriness, distortion, and uneven brightness, directly impacting the accuracy and stability of subsequent visual navigation and target detection. Therefore, a hierarchical visual data preprocessing method that can integrate IMU attitude information and adapt to the characteristics of dual-sphere rolling motion is needed. Summary of the Invention

[0004] To address the aforementioned technical shortcomings, the present invention aims to propose a visual data preprocessing method for a nested double-ball rolling robot. This method addresses the technical problems in existing technologies, such as insufficient image rotation compensation, poor motion blur removal, low inter-frame stability, and difficulty in adapting to the navigation requirements of rolling robots, due to the lack of integration with spherical rolling characteristics and the absence of a posture and vision fusion mechanism.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: The present invention provides a visual data preprocessing method for a double-ball nested rolling robot.

[0006] The visual data preprocessing method for a nested double-ball rolling robot includes: Step S10: Synchronously acquire visual image data and IMU attitude sensing data of the double-ball nested rolling robot. After fisheye distortion correction, spatiotemporal alignment and frame group division preprocessing, output the aligned digital image sequence, filtered angular velocity and attitude quaternion. Step S20: Based on the filtered angular velocity and attitude quaternions obtained in step S10, the rolling motion state detection is performed using the Madgwick attitude calculation and motion fuzzy kernel estimation mechanism, and the motion fuzzy kernel parameters, attitude angle sequence and motion intensity index are output. Step S30: Based on the digital image sequence from step S10 and the motion blur kernel parameters and attitude angle sequence output from step S20, image correction and deblurring are performed using a rolling geometric compensation and adaptive Wiener filtering mechanism, and the corrected and deblurred standard image frame is output. Step S40: Based on the corrected and deblurred image frame output in step S30, image enhancement processing is performed using a regional exposure equalization and region of interest enhancement mechanism, and the enhanced image sequence is output. Step S50: Based on the enhanced image sequence output in step S40 and the motion intensity index output in step S20, perform inter-frame stabilization processing using sparse optical flow registration and temporal smoothing mechanisms, output standardized visual preprocessing results, and provide feedback to optimize motion blur estimation parameters.

[0007] Preferably, step S10, which involves simultaneously acquiring visual image data and IMU attitude sensing data of a nested double-ball rolling robot, and performing fisheye distortion correction, spatiotemporal alignment, and frame group division preprocessing to obtain an aligned digital image sequence, filtered angular velocity, and attitude quaternions, specifically includes: Step S101: Acquire the visual image output by the monocular fisheye camera installed at the front end of the upper sphere at a frame rate of 30fps, and obtain a digital image sequence I(k) with a resolution of 640×480 through 8-bit analog-to-digital conversion. At the same time, acquire the triaxial acceleration, triaxial angular velocity and attitude quaternion q(k) output by the IMU installed at the center of gravity of the sphere at a sampling rate of 200Hz. Step S102: Perform fisheye lens distortion correction on the digital image sequence, remove radial and tangential distortion based on pre-calibrated intrinsic distortion coefficients to obtain a distortion-free image; perform fourth-order Butterworth low-pass filtering on the triaxial acceleration and triaxial angular velocity output by the IMU with a cutoff frequency of 30Hz to obtain filtered acceleration and angular velocity data, and retain the original attitude quaternion q(k); Step S103: Based on the image frame timestamp, perform linear interpolation alignment on the IMU data to achieve millisecond-level spatiotemporal synchronization of visual and attitude data, and divide the image frame group into groups of 8 consecutive frames, outputting the aligned digital image sequence, filtered angular velocity and attitude quaternions.

[0008] Preferably, step S20, based on the filtered angular velocity and attitude quaternions obtained in step S10, employs the Madgwick attitude calculation and motion fuzzy kernel estimation mechanism to perform rolling motion state detection, and outputs motion fuzzy kernel parameters, attitude angle sequence, and motion intensity index, specifically includes: Step S201: Using the filtered acceleration, angular velocity data and attitude quaternion q(k), the pitch angle θ, roll angle φ and yaw angle ψ are calculated by fusing them using the Madgwick algorithm to form an attitude angle sequence; Step S202: Based on the filtered triaxial angular velocity data and the exposure time of a single frame image, calculate the equivalent motion amplitude within the image exposure period, generate motion blur kernel parameters, and simultaneously calculate the motion intensity index. The calculation formula is as follows:

[0009] in , and The filtered triaxial angular velocities, , and These are the pre-calibrated axis weighting coefficients; Step S203: When the motion intensity index exceeds the preset threshold, it is determined to be a strong rolling motion state and marked as high fuzziness level; otherwise, it is determined to be a weak motion state and marked as low fuzziness level. At the same time, the motion fuzzy kernel parameters, attitude angle sequence and motion intensity index are output.

[0010] Preferably, the axis weighting coefficient in step S202 is taken as: =0.5、 =0.3、 =0.2, the motion intensity threshold is set to 1.2 rad / s, and when the threshold is exceeded, the length of the blur kernel increases exponentially linearly.

[0011] Preferably, step S30, based on the digital image sequence from step S10 and the motion blur kernel parameters and attitude angle sequence output from step S20, employs a rolling geometric compensation and adaptive Wiener filtering mechanism to perform image correction and deblurring, and outputs a corrected and deblurred standard image frame, specifically includes: Step S301: Based on the attitude angle sequence output in step S20, extract the roll angle and pitch angle at the exposure time of a single frame image, perform reverse rotation and translation transformation on the image, compensate for the screen rotation and field of view shift caused by the rolling of the two balls, and obtain the geometrically corrected image. Step S302: Match the blur kernel parameters corresponding to the current motion intensity, perform adaptive Wiener filtering to deblur the geometrically corrected image, dynamically adjust the filtering intensity according to the local image signal-to-noise ratio, and remove motion blur while preserving edge details; Step S303: Perform edge sharpening compensation on the deblurred image, output the corrected and deblurred standard image frame, and synchronously record the compensation residual value of the current frame.

[0012] Preferably, step S40, which involves performing image enhancement processing based on the corrected and deblurred image frame output in step S30, using a region-specific exposure equalization and region of interest enhancement mechanism, and outputting the enhanced image sequence, specifically includes: Step S401: Divide the corrected and deblurred image frame into three regions: the central traveling region, the peripheral transition region, and the edge spherical shell region. Use adaptive histogram equalization of different intensities for different regions to suppress local overexposure and underexposure in shadow areas caused by reflection from the spherical shell. Step S402: Extract the region of interest (ROI) in the lower half of the image for the direction of travel, and perform contrast-limited adaptive histogram equalization and detail enhancement on the region to improve the clarity of key navigation information such as ground texture and obstacles; Step S403: Perform uniform grayscale normalization on the entire image to eliminate overall brightness fluctuations between frames and output the enhanced image sequence.

[0013] Preferably, in step S50, based on the enhanced image sequence output in step S40 and the motion intensity index output in step S20, inter-frame stabilization processing is performed using a sparse optical flow registration and temporal smoothing mechanism to output standardized visual preprocessing results and to provide feedback for optimizing motion blur estimation parameters. Specifically, this includes: Step S501: Extract corner features from continuously enhanced image frames, calculate the inter-frame feature point displacement using the sparse optical flow method, and calculate the inter-frame global motion vector by weighting with the motion intensity index. Perform electronic image stabilization compensation on the image to eliminate residual jitter. Step S502: Perform temporal exponential smoothing filtering on the stabilized image sequence. The smoothing intensity is dynamically adjusted according to the intensity of motion. The smoothing intensity is increased in the weak motion state and decreased in the strong rolling state to avoid motion blur. Step S503: Output the final standardized visual preprocessing result, and feed back the inter-frame registration residual to the motion blur estimation module in step S20 to iteratively correct the blur kernel weight coefficients and optimize the deblurring effect of subsequent frames.

[0014] The present invention also provides a visual data preprocessing system for a nested double-ball rolling robot, comprising: Vision and IMU synchronous acquisition and preprocessing module: used to synchronously acquire visual image data and IMU attitude sensing data of the dual-ball nested rolling robot. After fisheye distortion correction, spatiotemporal alignment and frame group division preprocessing, the module outputs aligned digital image sequence, filtered angular velocity and attitude quaternion. Attitude calculation and fuzzy estimation module: Based on the filtered angular velocity and attitude quaternions obtained by the vision and IMU synchronous acquisition preprocessing module, it uses the Madgwick attitude calculation and motion fuzzy kernel estimation mechanism to perform rolling motion state detection and outputs motion fuzzy kernel parameters, attitude angle sequence and motion intensity index. Rolling compensation and deblurring module: Based on the digital image sequence acquired by the vision and IMU synchronous acquisition preprocessing module and the motion blur kernel parameters and attitude angle sequence output by the attitude calculation and fuzzy estimation module, the module uses rolling geometric compensation and adaptive Wiener filtering mechanism to perform image correction and deblurring, and outputs the corrected and deblurred standard image frame. The regional image enhancement module is used to perform image enhancement processing based on the corrected and deblurred image frames output by the rolling compensation and deblurring processing module. It adopts regional exposure equalization and region of interest enhancement mechanisms to output the enhanced image sequence. Inter-frame stabilization and loop closure optimization module: Based on the enhanced image sequence output by the regional image enhancement module and the motion intensity index output by the pose calculation and blur estimation module, it performs inter-frame stabilization processing using sparse optical flow registration and temporal smoothing mechanisms, outputs standardized visual preprocessing results, and feeds back optimized motion blur estimation parameters.

[0015] The present invention also provides a visual data preprocessing device for a nested double-ball rolling robot. The visual data preprocessing device for the nested double-ball rolling robot includes: a memory, a processor, and a visual data preprocessing program for the nested double-ball rolling robot stored in the memory and executable on the processor. When the visual data preprocessing program for the nested double-ball rolling robot is executed by the processor, it implements the above-described method.

[0016] The present invention also provides a computer program product comprising a visual data preprocessing program for a double-ball nested rolling robot, wherein the visual data preprocessing program for the double-ball nested rolling robot implements the above-described method when executed by a processor.

[0017] The beneficial effects of this invention are as follows: To address the spherical rolling characteristics of nested double-ball rolling robots, visual and IMU attitude data are collected simultaneously and precisely aligned in time and space. This introduces attitude motion information into the entire visual preprocessing process, overcoming the limitations of traditional pure image preprocessing methods that cannot adapt to rolling motion. It fundamentally compensates for the geometric deformation and blurring caused by spherical rolling.

[0018] By dynamically estimating the motion blur kernel through posture calculation and combining it with rolling geometry compensation and adaptive deblurring, the joint removal of image rotation correction and motion blur is achieved. Compared with general deblurring algorithms, it is more in line with the rolling motion mode of the dual-ball robot, with higher deblurring accuracy and better detail preservation.

[0019] By employing a regional exposure equalization and region of interest enhancement strategy along the direction of travel, the system effectively suppresses interference from reflections on the sphere's outer shell and focuses on enhancing details in navigation-related ground areas. This not only improves the overall image quality but also increases the information density of key perception areas, thus meeting the actual needs of autonomous robot navigation.

[0020] Electronic image stabilization is achieved through inter-frame optical flow registration and temporal adaptive smoothing. The registration residual is then fed back to optimize the fuzzy estimation parameters, forming a closed-loop processing mechanism of estimation, processing, feedback, and optimization. This mechanism can dynamically adjust the preprocessing intensity according to the robot's motion state, continuously improving the stability and quality of the output image. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a first embodiment of a visual data preprocessing method for a nested double-ball rolling robot according to the present invention. Detailed Implementation

[0022] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0023] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0024] Example 1: As Figure 1 The diagram shown is a flowchart illustrating a first embodiment of a visual data preprocessing method for a nested double-ball rolling robot according to the present invention.

[0025] In a first embodiment, the visual data preprocessing method for a nested double-ball rolling robot includes: Step S10: Synchronously acquire visual image data and IMU attitude sensing data of the double-ball nested rolling robot. After fisheye distortion correction, spatiotemporal alignment and frame group division preprocessing, output the aligned digital image sequence, filtered angular velocity and attitude quaternion.

[0026] It should be noted that in this step, the visual images are acquired by a fisheye camera mounted on the front end of the upper sphere to obtain information about the environment in front of the robot; the IMU is mounted at the center of gravity of the nested double-sphere structure, which can accurately acquire the three-axis acceleration and angular velocity during the sphere's rolling process. Fisheye distortion correction removes lens distortion through pre-calibrated camera intrinsic parameters to restore the geometric relationship of the real scene; IMU low-pass filtering is used to filter out high-frequency noise caused by mechanical vibration; spatiotemporal alignment interpolates and matches the IMU data based on image frames to ensure that the attitude data and image frames are strictly corresponding in time. The output digital image sequence is used for correction and enhancement in subsequent steps, and the filtered angular velocity and attitude quaternions are used as the raw input for attitude calculation and fuzzy estimation in step S20.

[0027] Understandably, through this preprocessing step, the original heterogeneous visual and pose data are transformed into standardized inputs that are spatiotemporally synchronized, distortion-free, and low-noise, providing a unified data benchmark for subsequent fusion processing. The high-sampling-rate IMU data is precisely aligned with the image frames, ensuring the temporal accuracy of subsequent roll compensation and blur estimation, which is a prerequisite for achieving preprocessing adapted to the rolling characteristics of dual balls.

[0028] It should be understood that traditional preprocessing methods typically process image data separately without incorporating synchronous pose information. This results in an inability to ascertain the physical causes of image blurring and deformation, forcing post-processing to rely solely on the image's own information, thus limiting its effectiveness. This step, through multi-sensor synchronous acquisition and alignment, introduces prior motion information from the IMU into the visual processing chain, providing a physical basis for subsequent accurate correction.

[0029] For example, when a nested double-sphere robot rolls rapidly forward on a flat surface, the upper sphere drives the camera to roll and swing synchronously. The camera captures images at 30fps, and the IMU outputs angular velocity data at 200Hz. After correcting fisheye distortion through this step, the arc-shaped distortion at the image edges is corrected; after the IMU data is low-pass filtered at 30Hz, the high-frequency spikes generated by the vibration of the motors inside the spheres are filtered out; finally, each frame of the image is matched with the attitude quaternion and angular velocity value at the corresponding moment, providing accurate time synchronization data for subsequent rolling compensation.

[0030] Step S20: Based on the filtered angular velocity and attitude quaternions obtained in step S10, the Madgwick attitude calculation and motion fuzzy kernel estimation mechanism is used to perform rolling motion state detection, and output motion fuzzy kernel parameters, attitude angle sequence and motion intensity index.

[0031] It should be noted that this step utilizes the Madgwick algorithm to fuse acceleration and angular velocity data, calculating the robot's real-time pitch, roll, and yaw angles to accurately reflect the attitude changes of the rolling dual balls. The motion intensity index is obtained through weighted calculation of three-axis angular velocities, quantifying the intensity of the current rolling motion; combined with the exposure time of a single frame image, the equivalent displacement of pixels within the exposure period can be calculated, thereby generating a motion blur kernel with corresponding direction and length. The motion state classification is based on the intensity threshold, providing a basis for subsequent deblurring and smoothing intensity adjustment. The output attitude angle sequence is used for geometric compensation in step S30, the blur kernel parameters are used for deblurring processing, and the motion intensity index is simultaneously input into step S50 for image stabilization intensity adjustment.

[0032] Understandably, this step transforms IMU pose data into quantized parameters for image degradation, establishing a correspondence between robot rolling motion and image blurring and rotation. Compared to methods that estimate the blur kernel from the image itself, IMU-based estimation is faster, more accurate, and can predict the degree of blur in advance, enabling feedforward control during preprocessing.

[0033] It should be understood that the direction and intensity of motion blur during the rolling of the dual-ball robot change in real time with its posture, making deblurring algorithms with fixed blur kernels unsuitable. This step dynamically generates blur kernels through real-time posture calculation, enabling the deblurring process to match the current real motion state, significantly improving the targeting and accuracy of deblurring.

[0034] For example, when the robot encounters an obstacle and suddenly turns, the roll velocity increases sharply, and the motion intensity index rises to 1.8 rad / s, exceeding the threshold of 1.2 rad / s. The system determines this as a strong rolling motion state and generates a longer lateral motion fuzzy kernel accordingly. When the robot moves slowly in a straight line, the angular velocity is smaller, and the intensity index is 0.6 rad / s. This is determined to be a weak motion state, and a short fuzzy kernel is generated. This achieves dynamic adaptation of fuzzy parameters to the motion state.

[0035] Step S30: Based on the digital image sequence from step S10 and the motion blur kernel parameters and attitude angle sequence output from step S20, image correction and deblurring are performed using a rolling geometric compensation and adaptive Wiener filtering mechanism, and the corrected and deblurred standard image frame is output.

[0036] It should be noted that roll geometry compensation performs an inverse affine transformation on the image based on the roll and pitch angles of the current frame, correcting the tilted and rotating image to a horizontal, frontal viewpoint, thus eliminating the image rotation and field of view shift caused by the sphere's roll. Adaptive Wiener filtering, based on the blur kernel output in step S20, dynamically adjusts the filtering coefficients according to the local signal-to-noise ratio of the image, smoothing blurred areas while preserving edge details and avoiding detail loss due to excessive blurring. Edge sharpening compensation is used to counteract the edge softening caused by deblurring, improving image contour clarity. The output standard image frame is input into step S40 for enhancement processing.

[0037] Understandably, this step of correcting geometric deformation before removing motion blur aligns with the physical process of image degradation caused by the rolling of two balls, making the processing order more scientific. Geometric correction eliminates image rotation, allowing subsequent deblurring to be performed in a standard coordinate system; adaptive filtering balances deblurring effectiveness with detail preservation, avoiding the drawbacks of traditional global filtering.

[0038] It should be understood that existing image correction methods for rolling robots mostly perform simple rotation processing without simultaneously removing motion blur, resulting in persistent image ghosting. Furthermore, deblurring algorithms, if not preceded by geometric correction, will suffer from a mismatch between the blur direction and the image, significantly reducing their effectiveness. This step combines geometric compensation and deblurring to collaboratively address the two major image degradation problems caused by the rolling of two balls.

[0039] For example, when a robot rolls to the right, the image rolls 15° horizontally, resulting in lateral motion blur. The system first rotates the image 15° in the opposite direction based on the roll angle to correct it to a horizontal image; then it performs adaptive Wiener filtering by matching the current lateral blur kernel to remove the motion blur caused by the roll; finally, it performs edge sharpening to output a standard image frame with neither rotation nor significant blur, where the ground texture and obstacle outlines are clearly visible.

[0040] Step S40: Based on the corrected and deblurred image frame output in step S30, image enhancement processing is performed using a regional exposure equalization and region of interest enhancement mechanism, and the enhanced image sequence is output.

[0041] It should be noted that the regional exposure equalization divides the image into a central traveling area, a peripheral transition area, and an edge spherical shell area. A lower equalization intensity is used for the edge areas with severe spherical shell reflections to prevent further magnification of overexposed areas; a moderate intensity is used for the central traveling area to improve the scene's dynamic range. Region of Interest (ROI) enhancement focuses on the lower half of the image, specifically the traveling ground area, enhancing texture and contrast to highlight key navigation information such as obstacles and path markers. Full-image grayscale normalization eliminates overall brightness fluctuations between frames, ensuring brightness consistency in the output image sequence. The enhanced image sequence is then input into step S50 for image stabilization.

[0042] Understandably, this step involves differentiated enhancement based on the imaging characteristics of the dual-sphere robot. It not only suppresses image quality interference caused by reflections from the sphere's outer shell, but also focuses on enhancing the travel area information required for navigation, making image enhancement more targeted rather than blindly increasing the overall image contrast.

[0043] It should be understood that general image enhancement algorithms apply the same processing to the entire image, which can easily lead to overexposure and whitening of reflective areas on the spherical shell, loss of detail in shadow areas, and failure to highlight key navigation areas. The region-specific enhancement strategy in this step perfectly suits the visual imaging characteristics of the dual-sphere robot, effectively improving the usable information density of the image.

[0044] For example, under indoor lighting, the transparent upper shell of the dual-sphere robot creates a noticeable overexposed and reflective area at the edge of the image. The system sets the equalization intensity of the edge shell area to a low level to prevent further aggravation of reflections; it performs detail enhancement on the ground travel area in the lower half of the image, making the textures of ground gaps and small obstacles clearer; finally, after normalizing the overall image brightness, there are no obvious jumps in brightness between frames, and the overall image quality and navigation information usability are significantly improved.

[0045] Step S50: Based on the enhanced image sequence output in step S40 and the motion intensity index output in step S20, perform inter-frame stabilization processing using sparse optical flow registration and temporal smoothing mechanisms, output standardized visual preprocessing results, and provide feedback to optimize motion blur estimation parameters.

[0046] It should be noted that sparse optical flow registration extracts corner features from consecutive frames, calculates inter-frame motion vectors, and performs sub-pixel-level displacement compensation on the image, eliminating residual micro-jitter after geometric correction. The temporal smoothing intensity is dynamically adjusted according to the intensity of motion: the smoothing coefficient is increased during weak motion to further reduce inter-frame flicker; the smoothing coefficient is decreased during strong scrolling to avoid motion blur caused by rapid movement. A closed-loop feedback mechanism feeds the inter-frame registration residual back to the blur estimation module in step S20, iteratively correcting the weight coefficients to make the blur kernel estimation increasingly closer to the actual imaging effect. The final output standardized visual data can be directly used in subsequent visual navigation and object detection algorithms.

[0047] Understandably, this step achieves temporal stabilization of the image sequence, solving the problem of inter-frame jumps caused by screen jitter in the rolling robot; at the same time, it introduces closed-loop feedback, enabling the preprocessing system to have self-optimization capabilities, and the processing effect continues to improve as the running time increases.

[0048] It should be understood that traditional preprocessing methods are mostly open-loop processes, with parameters fixed and not adjusted, making them unable to adapt to differences in rolling characteristics under different ground surfaces and speeds. The closed-loop feedback mechanism in this step can optimize the front-end parameters in reverse based on the actual processing effect, enabling the system to have environmental adaptability and maintain optimal preprocessing results in different scenarios.

[0049] For example, when the robot rolls slowly on a carpet, the motion intensity is low, so the system increases the temporal smoothing intensity, resulting in a very smooth image sequence with almost no inter-frame jitter. When switching to a smooth surface and rolling quickly, the motion intensity increases, and the system automatically reduces the smoothing intensity to prevent motion blur from fast-moving obstacles. After running for a period of time, the registration residual feedback corrects the axis weight coefficients of the blur kernel, improving the deblurring effect by about 15% compared to the initial state, and the system achieves self-optimization.

[0050] Example 2: Furthermore, the present invention provides a visual data preprocessing system for a nested double-ball rolling robot, employing a visual data preprocessing method for a nested double-ball rolling robot as described in the above embodiments. This system addresses the technical problems in the prior art, such as insufficient image rotation compensation, poor motion blur removal, low inter-frame stability, and difficulty in adapting to the navigation requirements of rolling robots, due to the lack of integration with spherical rolling characteristics and the absence of a posture and visual fusion mechanism. The beneficial effects of the visual data preprocessing system for a nested double-ball rolling robot provided by the present invention are the same as those of the visual data preprocessing method for a nested double-ball rolling robot provided in the above embodiments. Other technical features of the visual data preprocessing system for a nested double-ball rolling robot are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0051] Example 3: This invention provides a visual data preprocessing device for a nested double-ball rolling robot. The device includes: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which, when executed, enable the at least one processor to perform the visual data preprocessing method for a nested double-ball rolling robot described in Example 1. The visual data preprocessing device for a nested double-ball rolling robot in this invention can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. This visual data preprocessing device for a nested double-ball rolling robot is merely an example and should not limit the functionality or scope of the invention. A visual data preprocessing device for a nested double-ball rolling robot may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory or a program loaded from a storage device into a random access memory. The random access memory also stores various programs and data required for the operation of the visual data preprocessing device for the nested double-ball rolling robot. The processing unit, the read-only memory, and the random access memory are interconnected via a bus. An I / O interface is also connected to the bus. Typically, the following systems can be connected to the I / O interface: input devices including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices including, for example, magnetic tapes, hard disks, etc.; and communication devices. The communication device allows the visual data preprocessing device for the nested double-ball rolling robot to communicate wirelessly or wiredly with other devices to exchange data. While a visual data preprocessing apparatus for a nested double-ball rolling robot with various systems has been described, it should be understood that implementation of all the described systems is not required. Alternatively, more or fewer systems may be implemented.

[0052] Example 4: This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the visual data preprocessing method for a nested double-ball rolling robot described above. The computer program product provided by this invention can solve the technical problems in the prior art, such as insufficient image rotation compensation, poor motion blur removal, low inter-frame stability, and difficulty in adapting to the navigation requirements of rolling robots due to the lack of integration with spherical rolling characteristics and the absence of a posture and vision fusion mechanism. Compared with the prior art, the beneficial effects of the computer program product provided by this invention are the same as those of the visual data preprocessing method for a nested double-ball rolling robot provided in the above embodiments, and will not be repeated here.

[0053] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a read-only memory. When the computer program is executed by a processing device, it performs the functions defined in the methods of the embodiments disclosed in this invention.

[0054] It should be understood that the various parts disclosed in this invention can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0055] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the present invention and its equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A visual data preprocessing method for a nested double-ball rolling robot, characterized in that, The methods include: Step S10: Synchronously acquire visual image data and IMU attitude sensing data of the double-ball nested rolling robot. After fisheye distortion correction, spatiotemporal alignment and frame group division preprocessing, output the aligned digital image sequence, filtered angular velocity and attitude quaternion. Step S20: Based on the filtered angular velocity and attitude quaternions obtained in step S10, the rolling motion state detection is performed using the Madgwick attitude calculation and motion fuzzy kernel estimation mechanism, and the motion fuzzy kernel parameters, attitude angle sequence and motion intensity index are output. Step S30: Based on the digital image sequence from step S10 and the motion blur kernel parameters and attitude angle sequence output from step S20, image correction and deblurring are performed using a rolling geometric compensation and adaptive Wiener filtering mechanism, and the corrected and deblurred standard image frame is output. Step S40: Based on the corrected and deblurred image frame output in step S30, image enhancement processing is performed using a regional exposure equalization and region of interest enhancement mechanism, and the enhanced image sequence is output. Step S50: Based on the enhanced image sequence output in step S40 and the motion intensity index output in step S20, inter-frame stabilization processing is performed using sparse optical flow registration and temporal smoothing mechanism, outputting standardized visual preprocessing results, and feeding back optimized motion blur estimation parameters. In step S20, based on the filtered angular velocity and attitude quaternions obtained in step S10, the rolling motion state detection is performed using the Madgwick attitude calculation and motion fuzzy kernel estimation mechanism, and the steps of outputting motion fuzzy kernel parameters, attitude angle sequence, and motion intensity index specifically include: Step S201: Using the filtered acceleration, angular velocity data and attitude quaternion q(k), the pitch angle θ, roll angle φ and yaw angle ψ are calculated by fusing them using the Madgwick algorithm to form an attitude angle sequence; Step S202: Based on the filtered triaxial angular velocity data and the exposure time of a single frame image, calculate the equivalent motion amplitude within the image exposure period, generate motion blur kernel parameters, and simultaneously calculate the motion intensity index. The calculation formula is as follows: in , and The filtered triaxial angular velocities, , and These are the pre-calibrated axis weighting coefficients; Step S203: When the motion intensity index exceeds the preset threshold, it is determined to be a strong rolling motion state and marked as a high fuzziness level; otherwise, it is determined to be a weak motion state and marked as a low fuzziness level. At the same time, the motion fuzzy kernel parameters, attitude angle sequence and motion intensity index are output. In step S50, based on the enhanced image sequence output in step S40 and the motion intensity index output in step S20, inter-frame stabilization processing is performed using a sparse optical flow registration and temporal smoothing mechanism. The standardized visual preprocessing result is output, and feedback is provided to optimize the motion blur estimation parameters. Specifically, this includes: Step S501: Extract corner features from continuously enhanced image frames, calculate the inter-frame feature point displacement using the sparse optical flow method, and calculate the inter-frame global motion vector by weighting with the motion intensity index. Perform electronic image stabilization compensation on the image to eliminate residual jitter. Step S502: Perform temporal exponential smoothing filtering on the stabilized image sequence. The smoothing intensity is dynamically adjusted according to the intensity of motion. The smoothing intensity is increased in the weak motion state and decreased in the strong rolling state to avoid motion blur. Step S503: Output the final standardized visual preprocessing result, and feed back the inter-frame registration residual to the motion blur estimation module in step S20 to iteratively correct the blur kernel weight coefficients and optimize the deblurring effect of subsequent frames.

2. The visual data preprocessing method for a nested double-ball rolling robot according to claim 1, characterized in that, In step S10, the steps of simultaneously acquiring visual image data and IMU attitude sensing data of the nested double-ball rolling robot, and preprocessing them by fisheye distortion correction, spatiotemporal alignment, and frame group division to obtain the aligned digital image sequence, filtered angular velocity, and attitude quaternions specifically include: Step S101: Acquire the visual image output by the monocular fisheye camera installed at the front end of the upper sphere at a frame rate of 30fps, and obtain a digital image sequence I(k) with a resolution of 640×480 through 8-bit analog-to-digital conversion. At the same time, acquire the triaxial acceleration, triaxial angular velocity and attitude quaternion q(k) output by the IMU installed at the center of gravity of the sphere at a sampling rate of 200Hz. Step S102: Perform fisheye lens distortion correction on the digital image sequence, remove radial and tangential distortion based on the pre-calibrated intrinsic distortion coefficients to obtain a distortion-free image, perform fourth-order Butterworth low-pass filtering on the triaxial acceleration and triaxial angular velocity output by the IMU, set the cutoff frequency to 30Hz, obtain the filtered acceleration and angular velocity data, and retain the original attitude quaternion q(k). Step S103: Based on the image frame timestamp, perform linear interpolation alignment on the IMU data to achieve millisecond-level spatiotemporal synchronization of visual and attitude data, and divide the image frame group into groups of 8 consecutive frames, outputting the aligned digital image sequence, filtered angular velocity and attitude quaternions.

3. The visual data preprocessing method for a nested double-ball rolling robot according to claim 1, characterized in that, In step S202, the axis weight coefficient is taken as: =0.5、 =0.3、 =0.2, the motion intensity threshold is set to 1.2 rad / s, and when the threshold is exceeded, the length of the blur kernel increases exponentially linearly.

4. The visual data preprocessing method for a nested double-ball rolling robot according to claim 1, characterized in that, In step S30, based on the digital image sequence from step S10 and the motion blur kernel parameters and attitude angle sequence output from step S20, image correction and deblurring are performed using a rolling geometric compensation and adaptive Wiener filtering mechanism, and the corrected and deblurred standard image frame is output. Specifically, this includes: Step S301: Based on the digital image sequence of step S10 and the attitude angle sequence output in step S20, extract the roll angle and pitch angle at the exposure time of a single frame image, perform reverse rotation and translation transformation on the image, compensate for the screen rotation and field of view shift caused by the rolling of the two balls, and obtain the geometrically corrected image. Step S302: Match the blur kernel parameters corresponding to the current motion intensity, perform adaptive Wiener filtering to deblur the geometrically corrected image, dynamically adjust the filtering intensity according to the local image signal-to-noise ratio, and remove motion blur while preserving edge details; Step S303: Perform edge sharpening compensation on the deblurred image, output the corrected and deblurred standard image frame, and synchronously record the compensation residual value of the current frame.

5. The visual data preprocessing method for a nested double-ball rolling robot according to claim 1, characterized in that, In step S40, based on the corrected and deblurred image frame output in step S30, image enhancement processing is performed using a region-based exposure equalization and region of interest enhancement mechanism to output the enhanced image sequence. Specifically, this includes: Step S401: Divide the corrected and deblurred image frame into three regions: the central traveling region, the peripheral transition region, and the edge spherical shell region. Use adaptive histogram equalization of different intensities for different regions to suppress local overexposure and underexposure in shadow areas caused by reflection from the spherical shell. Step S402: Extract the region of interest in the direction of travel in the lower half of the image, and perform contrast-limited adaptive histogram equalization and detail enhancement on the region; Step S403: Perform uniform grayscale normalization on the entire image to eliminate overall brightness fluctuations between frames and output the enhanced image sequence.

6. A visual data preprocessing system for a nested double-ball rolling robot, applied to the visual data preprocessing method for a nested double-ball rolling robot according to any one of claims 1 to 5, characterized in that, The visual data preprocessing system for the nested double-ball rolling robot includes: Vision and IMU synchronous acquisition and preprocessing module: used to synchronously acquire visual image data and IMU attitude sensing data of the dual-ball nested rolling robot. After fisheye distortion correction, spatiotemporal alignment and frame group division preprocessing, the module outputs aligned digital image sequence, filtered angular velocity and attitude quaternion. Attitude calculation and fuzzy estimation module: Based on the filtered angular velocity and attitude quaternions obtained by the vision and IMU synchronous acquisition preprocessing module, it uses the Madgwick attitude calculation and motion fuzzy kernel estimation mechanism to perform rolling motion state detection and outputs motion fuzzy kernel parameters, attitude angle sequence and motion intensity index. Rolling compensation and deblurring module: Based on the digital image sequence acquired by the vision and IMU synchronous acquisition preprocessing module and the motion blur kernel parameters and attitude angle sequence output by the attitude calculation and fuzzy estimation module, the module uses rolling geometric compensation and adaptive Wiener filtering mechanism to perform image correction and deblurring, and outputs the corrected and deblurred standard image frame. The regional image enhancement module is used to perform image enhancement processing based on the corrected and deblurred image frames output by the rolling compensation and deblurring processing module. It adopts regional exposure equalization and region of interest enhancement mechanisms to output the enhanced image sequence. Inter-frame stabilization and loop closure optimization module: Based on the enhanced image sequence output by the regional image enhancement module and the motion intensity index output by the pose calculation and blur estimation module, it performs inter-frame stabilization processing using sparse optical flow registration and temporal smoothing mechanisms, outputs standardized visual preprocessing results, and feeds back optimized motion blur estimation parameters.

7. A visual data preprocessing device for a double-ball nested rolling robot, characterized in that, The visual data preprocessing device for a nested double-ball rolling robot includes: a memory, a processor, and a visual data preprocessing program for a nested double-ball rolling robot stored in the memory and executable on the processor. When the visual data preprocessing program for a nested double-ball rolling robot is executed by the processor, it implements a visual data preprocessing method for a nested double-ball rolling robot according to any one of claims 1 to 5.

8. A computer program product, characterized in that, The computer program product includes a visual data preprocessing program for a nested double-ball rolling robot, which, when executed by a processor, implements a visual data preprocessing method for a nested double-ball rolling robot according to any one of claims 1 to 5.

Images