Omnidirectional fisheye image feature tracking and extracting method based on non-parallel virtual binocular

Abstract

The invention discloses an omnidirectional fisheye image feature tracking and extracting method based on a non-parallel virtual binocular, and the method comprises the steps: constructing a four-eye fisheye camera system, obtaining the relative pose transformation based on a pre-calibration external parameter, predicting the pose based on a uniform speed model or IMU pre-integration, and constructing a re-projection error function through common-view feature points. According to the invention, by constructing four groups of virtual binocular combinations, on the premise of not increasing the number of physical cameras, eight-view-angle equivalent coverage is realized, so that a high-overlapping common-view area is formed between adjacent cameras; on the basis of a pre-calibrated external parameter and an MEI fisheye model, a spherical epipolar line is directly generated for a distorted image to carry out matching search, and meanwhile, a time sequence-space two-dimensional fault-tolerant mechanism is put forward for challenges of shielding and environmental interference: according to a dynamic direction switching design, when a tracking feature number is lower than a threshold value due to shielding in a camera direction, the tracking feature number is smaller than the threshold value; the camera direction is actively switched to serve as a main tracking source, new feature points are supplemented at the same time, and system collapse caused by local view failure is avoided.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing technology, specifically to a method for tracking and extracting omnidirectional fisheye image features based on non-parallel virtual binoculars. Background Technology

[0002] Large field-of-view fisheye cameras offer significant advantages in autonomous navigation and 3D reconstruction, but their imaging characteristics introduce three major technical bottlenecks: First, the barrel distortion of fisheye lenses leads to severe distortion of the geometric structure of image edge regions, rendering traditional epipolar constraints based on perspective models ineffective, and causing a significant increase in the mismatch rate of conventional feature matching algorithms in distorted regions. Second, fixed-view multi-camera systems lack adaptability in dynamically occluded scenes; when the main viewpoint is obscured by obstacles, the lack of an active viewpoint switching mechanism easily leads to a break in tracking continuity. Third, the physical baseline length of multi-camera systems is easily affected by mechanical vibration or temperature deformation, and small drifts in offline calibration extrinsic parameters can amplify depth calculation errors in long-distance observations. While existing solutions attempt to improve matching performance through spherical projection or complex distortion correction, real-time performance and robustness are difficult to balance, especially in omnidirectional, highly dynamic, and strongly disturbed environments where stable feature tracking accuracy cannot be maintained.

[0003] Chinese patent literature discloses a visual-inertial tightly coupled SLAM method based on four-eye panoramic cameras [Application No.: 202411440551.X, Publication No.: CN118967795A], which includes: extracting image features from images of four independent and synchronized fisheye cameras; correcting the distortion of the fisheye cameras to obtain the corrected feature points corresponding to each image feature; and performing panoramic projection on the corrected feature points to obtain the corresponding panoramic spherical coordinates. Although this invention can also achieve the purpose of panoramic view, this invention generates four sets of virtual binoculars (such as 0a-3b) through sub-viewpoint segmentation, achieving eight-view equivalent while maintaining the physical four-camera layout, significantly improving the overlap rate of the common viewing area. Combined with feature tracking robustness and distortion robust feature training, the achieved effect is better than that of the comparative patent. Summary of the Invention

[0004] A method for feature tracking and extraction of omnidirectional fisheye images based on non-parallel virtual binoculars, characterized by comprising: Step 1: Construct a four-eye fisheye camera system with a square layout of four fisheye lenses with a field of view greater than 220°. The cameras are numbered 0, 1, 2, and 3 clockwise. The field of view of each camera is divided into a left sub-viewpoint a and a right sub-viewpoint b along the central vertical line. Based on the pre-calibrated extrinsic parameters, construct four sets of virtual non-parallel binocular combinations: 0a-3b, 0b-1a, 2a-1b, and 2b-3a. Step 2: Obtain the relative pose transformation T based on the pre-calibrated extrinsic parameters 12 The MEI model is used to analyze image I. rpixels u i Within the depth range of the target's viewpoint d min -d max The projection generates spherical epipolar lines, and the matching points are searched by combining sparse optical flow and affine constraints. Step 3: Based on the uniform velocity model or IMU pre-integration, predict the pose of the current frame. For the three-dimensional spatial points that have been triangulated in the historical frames, use the MEI model adopted in Step 2 to project the three-dimensional points onto the current frame image to obtain their predicted positions, thereby achieving tracking. When the number of tracking features in the directions of camera 0 and camera 2 is lower than the set threshold, new feature points are added. When occlusion is detected, the main tracking direction is switched to the directions of camera 1 and camera 3. Step 4: Use a superpoint network to extract features. During training, apply contrast adjustment, brightness adjustment, motion blur, fogging, shadows, speckle noise, and Gaussian noise enhancement. Apply MEI distortion to the original image to generate a distorted image. Supervise training using the position consistency loss function. Step 5: Construct a reprojection error function using common-view feature points and nonlinearly optimize the extrinsic parameters.

[0005] Preferably, the four sets of virtual non-parallel binocular combinations in step one are defined as follows: Binocular view 0a-3b: The combination of the left sub-view a of camera 0 and the right sub-view b of camera 3; Binocular view 0b-1a: The combination of the right sub-viewpoint b of camera 0 and the left sub-viewpoint a of camera 1; Binocular pair 2a-1b: The combination of the left sub-view a of camera 2 and the right sub-view b of camera 1; Binocular pair 2b-3a: The right sub-viewpoint b of camera 2 and the left sub-viewpoint a of camera 3 are combined.

[0006] Through the above technical solution, this invention can achieve high-precision feature tracking and extraction in omnidirectional environments. This design of four virtual binocular sets enables the system to achieve seamless coverage of omnidirectional space, making it particularly suitable for feature tracking tasks in omnidirectional environments. When target feature points are occluded in the fields of view of camera 0 and camera 2, or when the number of features falls below a threshold, the system can quickly switch to camera 1 and camera 3 to continue tracking, ensuring the continuity of feature points.

[0007] Specifically, this non-parallel binocular combination design forms four independent yet complementary virtual binocular systems by cross-pairing the sub-viewpoints of adjacent cameras. This design not only avoids the blind spot problem of traditional parallel binocular systems but also ensures that a suitable matching point can be found from any viewpoint through the cooperation of the four directions. At the same time, this design also takes into account the distortion characteristics of fisheye cameras, effectively reducing the impact of distortion on feature matching by dividing the camera's field of view into sub-viewpoints along the central vertical line.

[0008] In practical applications, this strategy of combining four virtual binocular systems can better adapt to dynamic environmental changes, especially in fast-moving or complex scenes, enabling rapid and stable feature tracking and ensuring the system always operates at its optimal state. This innovative design of the combination method is of great significance for improving the overall performance and reliability of computer vision systems.

[0009] Preferably, the spherical epipolar projection in step two is implemented using the function fgt, where fgt is the projection function from a spatial point to the image plane based on the MEI fisheye model. The specific process of generating the spherical epipolar includes: (a) Project the pixel coordinates backwards onto the unit sphere; (b) Based on the depth range d min -d max Generate spatial point rays; (c) Call the projection function fgt to generate the spherical epipolar trajectory of the target viewpoint.

[0010] Through the above technical solution, this invention can achieve high-precision feature matching in omnidirectional environments. This spherical epipolar projection technique based on the MEI fisheye model can effectively handle the distortion problem of fisheye images and ensure accurate matching of feature points under different viewpoints.

[0011] Specifically, the fgt function uses the parameters of the MEI model to map spatial points along the depth range d from the target's viewpoint. min -d max A spherical epipolar line is generated by projection, and then a matching point is searched by combining sparse optical flow and affine constraints. This projection method not only considers the distortion characteristics of fisheye lenses, but also ensures the continuity and consistency of matching points at different depths through the construction of the spherical epipolar line. Furthermore, this MEI-based projection method effectively avoids the matching errors that may occur in fisheye images using traditional projection methods.

[0012] In practical applications, this spherical epipolar projection technique based on the MEI model can better adapt to dynamic environmental changes, especially in fast-moving or complex scenes. It can quickly and stably match features, ensuring that the system always operates at its optimal state. This innovative design of the projection method is of great significance for improving the overall performance and reliability of computer vision systems.

[0013] Preferably, in step two, the reprojection error of the successfully triangulated 3D points is verified. The error is the spherical angle error between the observation vector and the projection vector, and the calculation includes the following formula: .

[0014] Through the above technical solution, this invention can achieve high-precision verification of triangulated points. This verification method based on reprojection error can effectively eliminate erroneous 3D points, ensuring the accuracy of feature point reconstruction. Simultaneously, by setting a threshold, computational complexity can be controlled while maintaining accuracy, improving the system's real-time performance.

[0015] In practical applications, this reprojection error verification mechanism can better adapt to dynamic environmental changes, especially in fast-moving or complex scenes. It can quickly eliminate erroneous 3D points, ensuring that the system always operates at its optimal state. This innovative design of the verification method is of great significance for improving the overall performance and reliability of computer vision systems.

[0016] Preferably, the position consistency loss function in step four is: ; Where: P warped To extract corner points from the distorted image, corner points are extracted from the original image and used as pseudo-true values ​​P. original , will P original Projecting onto the distortion map coordinate system, we obtain the desired position P. warped_gt .

[0017] Through the above technical solution, this invention can achieve precise supervision of feature point positions. The design of this position consistency loss function, by considering the positional differences of feature points in the original and distorted images, ensures the positional consistency of the feature extraction network under different viewpoints.

[0018] In practical applications, this positional consistency loss function can better adapt to dynamic environmental changes, especially in fast-moving or complex scenarios. It can ensure the continuity and consistency of feature points from different perspectives and improve the overall performance of the system.

[0019] Preferably, the superpoint network in step four includes: Shared encoder: Extracts intermediate features from the input grayscale image; Detection head: Outputs a corner probability map; Description Header: Outputs corner descriptors.

[0020] Through the above technical solution, this invention can achieve efficient and accurate feature point detection and description. This super-point network design extracts intermediate features from the image through a shared encoder, and then uses two dedicated head networks for corner detection and descriptor generation respectively, ensuring both efficiency and accuracy in feature extraction.

[0021] Specifically, the shared encoder design avoids redundant calculations and improves computational efficiency; the detection head can accurately detect the location of feature points in the image by outputting a corner probability map; and the descriptor head generates a unique feature vector for each feature point by outputting corner descriptors for subsequent feature matching.

[0022] In practical applications, this network structure can better adapt to dynamic environmental changes, especially in fast-moving or complex scenes, enabling rapid and accurate extraction and matching of feature points, thus improving the overall performance of the system. The innovation of this network design is of great significance for improving the overall performance and reliability of computer vision systems.

[0023] Preferably, step four, the photometric enhancement operation, specifically includes: contrast adjustment, brightness adjustment, motion blur simulation, fog simulation, shadow simulation, speckle noise addition, and Gaussian noise addition, etc., specifically: Contrast and brightness adjustment: Simulates overall changes in ambient light intensity, such as from indoors to outdoors, or flickering lights. The formula is: ; in: The input image is in coordinates The pixel grayscale value at that location, It outputs the pixel grayscale value at the corresponding position in the image. It is the contrast gain factor. It is a brightness bias; Gaussian noise addition: This simulates the electronic noise generated by an image sensor under low light or high temperature conditions. The formula is as follows: ; in: It is a random number sampled from a Gaussian distribution with a mean of 0 and a variance of . ; Motion blur simulation: Simulates the motion blur effect produced when objects in a camera or scene move rapidly relative to each other. The formula is: ; in: Represents a two-dimensional convolution operation; Fog simulation: This simulates the imaging effect in environments with low visibility, such as fog, haze, or underwater. The formula is: ; in: This is a transmittance map, representing the data at scene points. The proportion of light that can penetrate the fog and reach the camera. Transmittance. It can be calculated using a formula, which is: ; in, It is the scattering coefficient, which controls the concentration of fog. It is a depth map of the scene, representing the distance of each pixel to the real-world object; Shadow simulation: Simulates the irregularly shaped shadows cast by objects (such as shelves, other AGVs) on an image. The formula is: ; in: It's the blurred shadow mask at the point. The value at that location (between 0 and 1). It is the shadow intensity factor, a number between 0 and 1, which controls the degree to which shadow areas become darker; Speckle noise addition: A common type of noise in simulated coherent imaging systems, characterized by its dependence on signal strength. ; in: It is a value with a mean of 0 and a variance of 0. Random numbers sampled from a uniform distribution.

[0024] Through the above technical solution, this invention can achieve comprehensive image enhancement and improve the robustness of feature extraction. The design of this photometric enhancement operation, by simulating changes in lighting and imaging conditions in various real-world scenarios, ensures the adaptability of the feature extraction network under different environments.

[0025] In practical applications, this comprehensive photometric enhancement strategy can improve the generalization ability of feature extraction, enhance the stability of feature points in different environments, and improve the accuracy of feature matching.

[0026] Preferably, step four also includes defining a fisheye distortion mask for each image to shield severely distorted areas at the edge of the field of view, and triggering a collaborative supplementation mechanism of the viewpoints of camera 1 and camera 3 when the number of features is insufficient.

[0027] Through the above technical solution, this invention can effectively process fisheye distortion and optimize the distribution of feature points. This mask-based and multi-view collaborative processing method can effectively avoid feature point extraction in edge distortion areas, while ensuring the distribution density and quality of feature points through the collaborative supplementation mechanism of camera views.

[0028] Specifically, the fisheye distortion mask defined for each image can accurately shield severely distorted areas at the edge of the field of view, avoiding the extraction of poor-quality feature points in these areas. When the number of features is insufficient, the system automatically triggers a collaborative supplementation mechanism between the viewpoints of camera 1 and camera 3, using the complementary characteristics of multiple viewpoints to acquire more effective feature points. This design not only solves the feature extraction problem caused by fisheye camera distortion but also improves the distribution density and quality of feature points through a multi-view collaborative mechanism.

[0029] In practical applications, this processing mechanism is better suited to feature extraction needs in complex scenarios, especially in fisheye camera applications, ensuring the extraction of high-quality feature points and improving the accuracy and reliability of feature matching. This innovative design is of great significance for improving the performance and reliability of computer vision systems in fisheye camera applications.

[0030] Preferably, step two further includes: using a binarized search mask generated based on image gradient magnitude to eliminate inefficient matching regions.

[0031] Through the above technical solution, the present invention can significantly improve the efficiency and accuracy of feature point matching. Specifically, step two employs a binarized search mask generated based on image gradient magnitude to effectively eliminate inefficient matching regions. This method analyzes the gradient information of pixels in the image to identify regions that contribute significantly to feature matching, thereby prioritizing these regions for feature point detection and matching.

[0032] Specifically, the generated binarized search mask can clearly distinguish between key feature regions and inefficient matching regions, ensuring that the system concentrates resources on high-quality feature points during feature point extraction and matching. This approach not only optimizes the use of computational resources but also improves matching accuracy and reduces the risk of mismatches.

[0033] In practical applications, this gradient magnitude-based masking mechanism can quickly adapt to different scenes and lighting conditions, improving the real-time performance and robustness of feature point matching. This innovative design not only enhances the algorithm's performance in complex visual environments but also provides strong support for the overall efficiency of computer vision systems.

[0034] Preferably, step five, extrinsic parameter optimization, includes: (a) Construction of the reprojection error function, using the same spherical angle error calculation method as in claim 4: ; in: For the observed normalized spherical coordinates, For three-dimensional points After transformation Normalized spherical coordinates afterward; (b) Prioritize optimizing the baseline length by scaling the translation vector: ; in: To use the scaled baseline Calculate the spherical coordinates; (c) When the reprojection error is higher than the threshold At the same time, jointly optimize the rotation matrix and translation vector: ; (d) Solve iteratively using the Levenberg-Marquardt algorithm: ; in: Let be the Jacobian matrix of the spherical error function. Let be the transpose of the Jacobian matrix of the spherical error function. It is the identity matrix. Let be the spherical angle error vector. is the damping factor.

[0035] Through the above technical solutions, this invention achieves online optimization and self-calibration capabilities for extrinsic parameters. Prioritizing baseline length optimization can quickly correct scale drift caused by vibration, while subsequent joint optimization can more accurately compensate for minute pose changes in the camera. The Levenberg-Marquardt algorithm ensures rapid convergence and stability of the optimization process, thereby significantly improving the accuracy and robustness of the system's 3D perception during long-term operation.

[0036] Compared with the prior art, the present invention has the following advantages: 1. This invention constructs four virtual binocular combinations by dividing each field of view of four fisheye lenses into left and right sub-views. This design achieves equivalent coverage of eight views without increasing the number of physical cameras, creating highly overlapping common viewing areas between adjacent cameras. Based on pre-calibrated extrinsic parameters and the MEI fisheye model, it directly generates spherical epipolar lines from distorted images for matching search, avoiding edge stretching and resampling errors caused by traditional panoramic stitching. Combining a joint optimization mechanism of sparse optical flow and affine constraints, it effectively overcomes the geometric distortion problem caused by barrel distortion in large field-of-view fisheye images, ensuring stable feature association across viewpoints even at a 220° ultra-wide field of view, and significantly reducing the mismatch rate.

[0037] 2. To address the challenges of occlusion and environmental interference, this invention proposes a temporal-spatial dual-dimensional fault-tolerance mechanism: A dynamic direction-switching design actively switches to camera direction 1 or 3 as the primary tracking source when the number of tracking features falls below a threshold due to occlusion in camera direction 0 or 2, while simultaneously supplementing new feature points to avoid system crashes caused by local view failures. Distortion-invariant feature training employs a super-point network combined with a fisheye-specific data augmentation strategy, and innovatively introduces a position consistency loss function. This loss is supervised by projecting corner points of the original image onto the coordinate system of the distorted image, forcing the network to learn distortion-invariant feature representations. The synergistic effect of these two mechanisms enables the system to maintain high feature repetition rates and tracking continuity even in areas with frequent occlusion, abrupt changes in illumination, and strong distortion at edges. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the configuration scheme of the four-eye omnidirectional fisheye camera of the present invention; Figure 2 This is a schematic diagram of the spherical polar line search of the present invention; Figure 3 This is a schematic diagram illustrating the reprojection error verification of 3D points that have been successfully triangulated according to the present invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] A method for feature tracking and extraction of omnidirectional fisheye images based on non-parallel virtual binoculars includes: Step 1: Construct a four-eye fisheye camera system with a square layout of four fisheye lenses with a field of view greater than 220°. The cameras are numbered clockwise as 0, 1, 2, and 3. The field of view of each camera is divided into a left sub-viewpoint a and a right sub-viewpoint b along the central vertical line. Based on pre-calibrated extrinsic parameters, construct four sets of virtual non-parallel binocular combinations: 0a-3b, 0b-1a, 2a-1b, and 2b-3a. (See schematic diagram below.) Figure 1 As shown; Step 2: Obtain the relative pose transformation T based on the pre-calibrated extrinsic parameters 12 The MEI model is used to analyze image I. r pixels u i Within the depth range of the target's viewpoint d min -d max The projection generates spherical epipolar lines, and the matching points are searched by combining sparse optical flow and affine constraints. Step 3: Based on the uniform velocity model or IMU pre-integration, predict the pose and project the 3D points of the historical frame onto the current frame for tracking; when the number of tracking features in the directions of camera 0 and camera 2 is lower than the set threshold, add new feature points; when occlusion is detected, switch the main tracking direction to the directions of camera 1 and camera 3. Step 4: Use a superpoint network to extract features. During training, apply contrast adjustment, brightness adjustment, motion blur, fogging, shadows, speckle noise, and Gaussian noise enhancement. Apply MEI distortion to the original image to generate a distorted image. Supervise training using the position consistency loss function. Step 5: Construct a reprojection error function using common-view feature points and nonlinearly optimize the extrinsic parameters.

[0041] The method for projecting spatial points to the image plane based on the MEI fisheye model is as follows: Based on known multi-camera extrinsic parameters and the MEI fisheye model, virtual binocular pairs are constructed. Cameras 0a-3b, 0b-1a, 2a-1b, and 2b-3a form four binocular sets, and epipolar geometric search is performed in the image plane. The matching process combines sparse optical flow and affine constraints, and inefficient regions are eliminated through a custom search mask, effectively improving the robustness and accuracy of cross-view feature matching. It also adapts to epipolar search methods for large FOV cameras, such as... Figure 2 As shown.

[0042] The specific implementation steps of spherical epipolar search are as follows: (a) Pixel backprojection onto the unit sphere Given a reference image pixels on Points on the unit sphere are obtained using the MEI model back projection function: ; The MEI back projection process is as follows: Calculate normalized coordinates: , ; Calculate radial distance: ; Solution depth: ; Obtain the spherical point: .

[0043] (b) Generate depth sampling rays Along the depth range Uniformly sample M depth values: ; For each depth value, calculate the corresponding 3D spatial point: .

[0044] (c) Projection generates spherical polar lines Projecting a spatial point sequence onto a target image : ; The MEI projection function is: Normalization: , ; Calculate the distortion radius: ; Projected onto pixels: , .

[0045] (d) Polar matching search Normalized cross-correlation (NCC) is used as the matching measure: ; Optimal depth: .

[0046] Given two poses C1 and C2, and a known relative pose transformation T 12 In the case of image I r There is a pixel u on it i And the corresponding patch block, then there should be a corresponding image patch on image C2 that can be accessed by f. gt This is obtained by projecting it onto the image plane. Assume u i The corresponding depth range of the 3D points in space is d min to d max Between, d min to d max The straight line between them will form an epipolar line on the image frame corresponding to C2, that is... Figure 2 The UnitSphere in this invention continuously minimizes the breadth error through epipolar search on the sphere to obtain corresponding matches and simultaneously generates triangulated 3D points. Error verification is then performed on successfully triangulated points. The error calculation method of this invention is as follows: Figure 3 As shown.

[0047] Example 1: Navigation of AGVs in Large-Scale Warehousing and Logistics Scene description: A large automated warehouse has the following internal environmental characteristics: Structured environment: The aisle is long and narrow (approximately 1.8 meters wide), with high-rise shelves on both sides (approximately 8 meters high), and the shelf structure has a lot of repetitive textures.

[0048] Dynamic interference: The presence of other AGVs, staff, or forklifts passing through the passageway can frequently cause localized obstruction of the field of vision.

[0049] The lighting is complex: there are fluorescent lights at the top, which may flicker; some areas are close to windows, so the lighting changes over time; and the shelves cast large shadows.

[0050] Operating conditions: AGVs need to run continuously for long periods of time, and vehicle vibration and ambient temperature differences may cause slight drift in the camera's external parameters.

[0051] Technical process: 1. Virtual binocular activation and task allocation When the AGV starts, the system activates all four sets of virtual non-parallel binocular cameras according to the pre-calibrated extrinsic parameters and assigns them different main monitoring tasks: Forward main binocular pair (0b-1a): mainly responsible for monitoring the travel path directly in front of the AGV, extracting landmarks and ground textures at the far end of the channel for heading control and speed planning.

[0052] The rearward main binocular pair (2b-3a) is mainly responsible for monitoring the rear path of the AGV. It forms a long baseline with the forward binocular pair, providing the system with stronger rotation and translation observation and realizing closed-loop detection of motion.

[0053] Left-side binocular pair (0a-3b): mainly responsible for accurately sensing the left-side shelves and identifying the storage location labels, QR codes or structural corners on the shelves, for the precise positioning and docking of the AGV between shelves.

[0054] Right-side binocular pair (2a-1b): Similar in function to the left-side binocular pair, it is responsible for accurately sensing the right-side shelf to ensure that the AGV always stays in the center of the aisle and travels safely.

[0055] 2. Dynamic polar tracking and fault-tolerant switching Regular tracking: When the AGV travels along the channel, the system uses the corner points of the shelf uprights detected by the right-side working binocular pair (2a-1b) to generate spherical epipolar lines in the field of view of camera 1b based on the MEI model along the depth range of 0.5m to 8m, and combines sparse optical flow for stable tracking.

[0056] Occlusion Response and Switching: When a forklift enters from the side aisle, temporarily obstructing the field of view of camera 2, the number of tracking features of the binocular pairs (2a-1b) and (2b-3a) drops sharply to below the preset threshold of 30. The system immediately determines that the main tracking direction is blocked and automatically shifts the weight of the localization calculation mainly to the unobstructed forward (0b-1a) and left (0a-3b) binocular pairs.

[0057] Feature Supplementation: During the switching process, the system calls the superpoint network to actively detect and supplement new stable feature points (such as bolts or labels on the other side of the shelf) in the sub-fields of view of the unobstructed cameras 0a and 1a. This ensures that the total number of feature points is always maintained at a healthy level (such as more than 100), thereby guaranteeing the continuity and stability of the positioning attitude calculation and avoiding system drift or crash due to the failure of a single-sided field of view.

[0058] Eliminating inefficient areas: By using a binarized search mask generated based on image gradient magnitude, the highly reflective areas of the stainless steel protective plate on the shelf and the overexposed areas caused by direct light are actively ignored, and the computational resources are concentrated on the effective areas with rich texture.

[0059] 3. Distortion Robust Feature Extraction and Training During the model training phase, the dataset is specifically augmented for the warehouse environment: Light intensity enhancement: Apply ±30% brightness adjustment to simulate lighting differences in different areas; ±20% contrast adjustment to simulate shadow changes; simulate motion blur when AGVs start or turn; simulate shadows cast by shelves and other AGVs.

[0060] Noise enhancement: Gaussian noise and speckle noise are added to simulate the electronic noise generated by the sensor in low-light environments such as at the bottom of the shelf.

[0061] Position consistency loss supervision: Feature points such as shelf label corners and bolts extracted from the original image are used as pseudo-ground values. These are projected onto the enhanced distortion image using the MEI model to obtain the desired positions. The position consistency loss function is used to supervise network training, which ensures that the feature points learned by the network maintain high position consistency and robustness even in strongly distorted regions at the edges of fisheye images.

[0062] 4. Online self-calibration of external parameters Triggering conditions: After the AGV has been running for a cumulative 2 hours, or when the system background detects that the average reprojection error of the common-view feature points is continuously higher than 1.5 pixels, the system determines that the external parameters may drift due to accumulated vibration and automatically triggers online optimization.

[0063] Optimization process: The system uses long-term stable common viewpoints in the scene (such as fixed shelf column corners) to construct a reprojection error function.

[0064] First, prioritize optimizing the baseline length (i.e., the scale of the translation vector t) to quickly correct major scale drift.

[0065] If the step error after optimization is still higher than the threshold (e.g., 1.0 pixels), then further optimize the rotation matrix R and translation vector t for more refined adjustments.

[0066] The entire nonlinear optimization process is solved iteratively using the Levenberg-Marquardt algorithm until the error converges.

[0067] Effect: By implementing this invention, the AGV navigation system achieves the following performance improvements: Accuracy: In a typical narrow channel with a width of 1.8 meters, after continuous round-trip operation for more than 500 meters, the root mean square error (RMSE) of the end-to-end positioning trajectory is less than 2.0 cm compared with the ground lidar trajectory.

[0068] Robustness: In stress tests simulating frequent forklift and personnel traffic (with 15% of the time spent in the field of view being obscured), the feature tracking success rate remained above 98%, and no positioning system resets or tracking failures occurred due to partial occlusion throughout the test.

[0069] Continuity: Thanks to online self-calibration of external parameters, the positioning accuracy of the equipment did not decrease significantly after 8 hours of continuous operation, proving that the system has excellent long-term operational stability.

[0070] Example 2: 3D Modeling of a Mine Inspection Robot Scene description: Unstructured environment: The tunnel shape is irregular, the rock wall texture is complex and varied, and there are no obvious artificial geometric features.

[0071] Extreme lighting: No natural light, relying entirely on the robot's own high-powered searchlight. This results in a field of view containing simultaneously bright areas directly illuminated by strong light, completely dark areas, and areas of light and shadow with obvious ghosting.

[0072] Harsh Atmosphere: The air is filled with dust and water vapor of varying concentrations, which severely affects image clarity and visibility.

[0073] Severe vibration: When the robot travels on rough mine tunnels or tracks, it generates severe jolts, posing a serious challenge to the stability of the camera's external parameters.

[0074] Technical process: 1. Omnidirectional Virtual Binocular Configuration Omnidirectional perception by robots is crucial for mine safety. Four sets of virtual binocular pairs are configured for full-space monitoring: Forward / backward binocular pair (0b-1a, 2b-3a): Used for the robot's primary navigation in alleyways, perceiving the path ahead and the trajectory behind.

[0075] Left / right binocular pair (0a-3b, 2a-1b): Used for detailed scanning of the rock walls on both sides of the tunnel to detect cracks, water seepage, and the condition of the support frame.

[0076] Implicit top / bottom viewpoint: Utilizing the ultra-wide field of view of the fisheye lens, the above binocular pair simultaneously covers the top plate (detecting dangerous rocks) and the bottom plate (track or road surface conditions), achieving three-dimensional environmental perception without blind spots.

[0077] 2. Feature tracking in high dynamic environments Feature selection: Prioritize relatively stable structural features in the mine, such as bolts on the support frame, track joints, and unique ore nodes on the rock wall, as landmarks for long-term tracking.

[0078] Fault-tolerant switching: When the robot passes through an area with high dust concentration, causing the forward camera's (0,1) field of view to become blurred and the number of tracked features to fall below the safe threshold of 40, the system automatically switches the localization weights to a virtual binocular pair for the less affected lateral (0a-3b, 2a-1b) and rearward (2b-3a) directions. This relies on feature tracking of the sidewalls and the already traversed path to maintain pose estimation, ensuring the robot does not lose its way in a "blind" state. After leaving the dusty area, the system replenishes feature points in the forward field of view.

[0079] Dynamic masking: Gradient amplitude masking is used to actively remove textureless, high-brightness light spots caused by direct illumination of the rock wall by searchlights.

[0080] 3. Training on the Special Characteristics of the Mine Environment To cope with the extreme environment of the mine, targeted data augmentation was applied when training the feature extraction network: Extreme lighting simulation: Apply non-linear brightness transformation to the image, increasing the brightness of local areas by 80% while decreasing it by 50% in other areas to simulate the effect of a searchlight; significantly reduce contrast to simulate low-light areas.

[0081] Dust atomization simulation: Add atomization effects of different concentration levels to simulate different levels of dust environment in the tunnel, and combine it with spot noise to simulate the scattering of light on dust particles.

[0082] Distortion invariance enhancement: The same location consistency loss function is used to focus on supervising the network's recognition consistency of key features such as rock strata cracks and bolts in areas of dramatic lighting changes and strong distortion, ensuring that high-quality features can be extracted for 3D modeling even under harsh conditions.

[0083] 4. Online external parameter vibration optimization Triggering conditions: The system triggers online optimization when it detects through IMU data that the robot is experiencing severe vibration (such as angular velocity or acceleration exceeding a threshold), or when the sliding average of the reprojection error exceeds 2.0 pixels.

[0084] Optimization Goal: This function is crucial for the accuracy of 3D modeling. It can compensate for camera physical position shifts caused by severe bumps in real time, avoiding "ghosting" or "layering" phenomena in the map.

[0085] Optimization process: Using common track bolt points or support frame connection points as stable anchor points, the Levenberg-Marquardt algorithm is used to rapidly iterate and optimize the extrinsic parameters R and t, ensuring that each frame of image can be accurately aligned to the global map.

[0086] Effect: Mapping quality: Even in harsh environments where visibility drops to 5 meters due to dust, the robot is still able to complete continuous mapping tasks. The final generated 3D point cloud model has a structural integrity of over 96%, and the dimensional measurements of key tunnel structures (such as support frames) show an error of less than 1% compared to laser scanning results.

[0087] Positioning robustness: Throughout the entire inspection mission, even with frequent entry and exit from high-dust areas, the system did not experience a single tracking loss. Compared to the offline optimized true trajectory, the relative attitude error of the real-time positioning trajectory is less than 0.5%.

[0088] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for feature tracking and extraction of omnidirectional fisheye images based on non-parallel virtual binoculars, characterized in that, include: Step 1: Construct a four-eye fisheye camera system with a square layout of four fisheye lenses with a field of view greater than 220°. The cameras are numbered 0, 1, 2, and 3 clockwise. The field of view of each camera is divided into a left sub-viewpoint a and a right sub-viewpoint b along the central vertical line. Based on the pre-calibrated extrinsic parameters, construct four sets of virtual non-parallel binocular combinations: 0a-3b, 0b-1a, 2a-1b, and 2b-3a. Step 2: Obtain the relative pose transformation T based on the pre-calibrated extrinsic parameters 12 The MEI model is used to analyze image I. r pixels u i Within the depth range of the target's viewpoint d min -d max The projection generates spherical epipolar lines, and the matching points are searched by combining sparse optical flow and affine constraints. Step 3: Predict the pose of the current frame based on a uniform velocity model or IMU pre-integration. For three-dimensional spatial points that have already been triangulated in historical frames Using the MEI model employed in step two, the 3D point is projected onto the current frame image to obtain its predicted position. This enables tracking; when the number of tracking features in the directions of camera 0 and camera 2 is lower than the set threshold, new feature points are added; when occlusion is detected, the main tracking direction is switched to the directions of camera 1 and camera 3. Step 4: Use a superpoint network to extract features. During training, contrast adjustment, brightness adjustment, motion blur, fogging, shadows, speckle noise, and Gaussian noise enhancement can be applied. Apply MEI distortion to the original image to generate a distorted image, and supervise training through the position consistency loss function. Step 5: Construct a reprojection error function using common-view feature points and nonlinearly optimize the extrinsic parameters.

2. The method according to claim 1, characterized in that, The four sets of virtual non-parallel binocular combinations in step one are defined as follows: Binocular view 0a-3b: The combination of the left sub-view a of camera 0 and the right sub-view b of camera 3; Binocular view 0b-1a: The combination of the right sub-viewpoint b of camera 0 and the left sub-viewpoint a of camera 1; Binocular pair 2a-1b: The combination of the left sub-view a of camera 2 and the right sub-view b of camera 1; Binocular pair 2b-3a: The right sub-viewpoint b of camera 2 and the left sub-viewpoint a of camera 3 are combined.

3. The method according to claim 1, characterized in that, The spherical epipolar projection in step two is implemented using the function fgt, where fgt is the projection function from a spatial point to the image plane based on the MEI fisheye model. The specific process of generating the spherical epipolar lines includes: (a) Project the pixel coordinates backwards onto the unit sphere; (b) Based on the depth range d min -d max Generate spatial point rays; (c) Call the projection function fgt to generate the spherical epipolar trajectory of the target viewpoint.

4. The method according to claim 1, characterized in that, In step two, the reprojection error of the successfully triangulated 3D points is verified. The error is the spherical angle error between the observation vector and the projection vector, and its calculation includes the following formula: 。 5. The method according to claim 1, characterized in that, The position consistency loss function in step four is: ； Where: P warped To extract corner points from the distorted image, corner points are extracted from the original image and used as pseudo-true values ​​P. original , will P original Projecting onto the distortion map coordinate system, we obtain the desired position P. warped_gt .

6. The method according to claim 1, characterized in that, The superpoint network in step four includes: Shared encoder: Extracts intermediate features from the input grayscale image; Detection head: Outputs a corner probability map; Description Header: Outputs corner descriptors.

7. The method according to claim 1, characterized in that, The photometric enhancement operation in step four specifically includes: contrast adjustment, brightness adjustment, motion blur simulation, fog simulation, shadow simulation, speckle noise addition, and Gaussian noise addition, etc., in detail: Contrast and brightness adjustment: Simulates the overall change in ambient light intensity; the formula is: ； in: The input image is in coordinates The pixel grayscale value at that location, It outputs the pixel grayscale value at the corresponding position in the image. It is the contrast gain factor. It is a brightness bias; Gaussian noise addition: This simulates the electronic noise generated by the image sensor, and its formula is: ； in: It is a random number sampled from a Gaussian distribution with a mean of 0 and a variance of . ; Motion blur simulation: Simulates the motion blur effect produced when objects in a camera or scene move rapidly relative to each other. The formula is: ； in: Represents a two-dimensional convolution operation; Fog simulation: Simulates the imaging effect in a foggy environment; its formula is: ； in: This is a transmittance map, representing the data at scene points. The proportion of light that can penetrate the fog and reach the camera, and the transmittance. It can be calculated using a formula, which is: ； in: It is the scattering coefficient, which controls the concentration of fog. It is a depth map of the scene, representing the distance of each pixel to the real-world object; Shadow simulation: Simulates the irregularly shaped shadows cast by objects on an image; the formula is: ； in: It's the blurred shadow mask at the point. The value at that location, It is the shadow intensity factor, which controls the degree to which shadow areas become darker; Speckle noise addition: A common type of noise in simulated coherent imaging systems, characterized by its dependence on signal strength. ； in: It is a value with a mean of 0 and a variance of 0. Random numbers sampled from a uniform distribution.

8. The method according to claim 1, characterized in that, Step four also includes defining a fisheye distortion mask for each image to shield severely distorted areas at the edge of the field of view, and triggering a collaborative supplementation mechanism of the viewpoints of camera 1 and camera 3 when the number of features is insufficient.

9. The method according to claim 1, characterized in that, Step two further includes: using a binarized search mask generated based on image gradient magnitude to eliminate inefficient matching regions.

10. The method according to claim 1, characterized in that, Step five, extrinsic parameter optimization, includes: (a) Construction of the reprojection error function, using the same spherical angle error calculation method as in claim 4: ； in: For the observed normalized spherical coordinates, For three-dimensional points After transformation Normalized spherical coordinates afterward; (b) Prioritize optimizing the baseline length by scaling the translation vector: ； in: To use the scaled baseline Calculate the spherical coordinates; (c) When the reprojection error is higher than the threshold At the same time, jointly optimize the rotation matrix and translation vector: ； (d) Solve iteratively using the Levenberg-Marquardt algorithm: ； in: Let be the Jacobian matrix of the spherical error function. Let be the transpose of the Jacobian matrix of the spherical error function. It is the identity matrix. Let be the spherical angle error vector. is the damping factor.

Images