An omnidirectional fisheye image feature tracking and extraction method based on non-parallel virtual binoculars

Abstract

The application discloses a kind of omnidirectional fish-eye image feature tracking and extraction method based on non-parallel virtual binoculars, including constructing four fish-eye camera systems, obtaining relative pose transformation based on pre-calibration external parameter, predicting pose based on uniform speed model or IMU pre-integration, and using co-observation feature points to construct re-projection error function etc.Steps, the present application realizes eight view angle equivalent coverage by constructing four virtual binoculars combinations, without increasing the number of physical cameras, so that high-overlapping co-observation area is formed between adjacent cameras;Based on the external parameter of pre-calibration and MEI fish-eye model, spherical polar line is directly generated for matching search on distorted image, and for the challenge of occlusion and environmental interference, a time-space two-dimensional fault-tolerant mechanism is proposed: dynamic direction switching design, when the number of tracked features is lower than the threshold due to occlusion, the camera direction is actively switched as the main tracking source, and new feature points are supplemented, to avoid system crash caused by local visual field failure.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and image processing, and particularly relates to a kind of omnidirectional fish-eye image feature tracking and extraction method based on non-parallel virtual binoculars. BACKGROUND

[0002] Ultra-large field angle fisheye camera has significant advantages in autonomous navigation and three-dimensional reconstruction, but its imaging characteristics introduce three technical bottlenecks: first, the barrel distortion of fisheye lens leads to severe distortion of the geometric structure of the image edge region, the traditional polar constraint based on the perspective model is invalid, and the false matching rate of conventional feature matching algorithms in the distortion region increases significantly; second, the multi-view system with fixed viewing angle is not adaptive enough in dynamic occlusion scenes, and when the main viewing angle is blocked by obstacles, the lack of active viewing angle switching mechanism easily leads to tracking continuity break; third, the physical baseline length of the multi-camera system is easily affected by mechanical vibration or temperature deformation, and the small drift of the offline calibrated external parameters will amplify the depth calculation error in long-distance observation. Although existing solutions attempt to improve matching performance through spherical projection or complex distortion correction, real-time performance and robustness are difficult to balance, especially in omnidirectional, high dynamic and strong interference environments, which cannot maintain stable feature tracking accuracy.

[0003] Chinese patent document discloses a visual inertial tightly coupled SLAM method based on four-view panoramic camera

application number: 202411440551.X, publication number: CN118967795A

[0004] A kind of omnidirectional fish-eye image feature tracking and extraction method based on non-parallel virtual binoculars, characterized in that, it includes:

[0005] Step one: construct four fisheye camera systems, four fisheye lenses with a field of view angle greater than 220° are arranged in a square, and the cameras are numbered clockwise as 0, 1, 2 and 3; the field of view of each camera is divided into a left sub-view angle a and a right sub-view angle b along the central vertical line; based on the pre-calibrated external parameters, four groups of virtual non-parallel binoculars are constructed: 0a-3b, 0b-1a, 2a-1b and 2b-3a;

[0006] Step two: obtain the relative pose transformation T based on the pre-calibrated external parameters 12, the MEI model is used to project the pixel u r of the image I i along the depth range d min -d max to generate a spherical epipolar line, and the matching points are searched by combining the sparse optical flow and the affine constraint;

[0007] Step three: based on the uniform speed model or the IMU pre-integration prediction of the current frame pose, for the three-dimensional space points which have completed triangulation in the history frame, the MEI model adopted in the step two is used to project the three-dimensional points to the current frame image to obtain the predicted positions of the three-dimensional points, so that the tracking is realized; when the number of tracking features in the directions of the camera 0 and the camera 2 is lower than a set threshold, new feature points are supplemented, and when the occlusion is detected, the main tracking direction is switched to the directions of the camera 1 and the camera 3;

[0008] Step four: the feature is extracted by using the super point network, and the contrast adjustment, the brightness adjustment, the motion blur, the fogging, the shadow, the speckle noise and the Gaussian noise enhancement are applied during the training; the MEI distortion is applied to the original image to generate a distorted image, and the training is supervised by using the position consistency loss function;

[0009] Step five: the reprojection error function is constructed by using the co-view feature points, and the nonlinear optimization is performed on the external parameters.

[0010] Preferably, the four groups of virtual non-parallel binocular combinations in the step one are defined as:

[0011] The binocular pair 0a-3b is composed of the left sub-view angle a of the camera 0 and the right sub-view angle b of the camera 3;

[0012] The binocular pair 0b-1a is composed of the right sub-view angle b of the camera 0 and the left sub-view angle a of the camera 1;

[0013] The binocular pair 2a-1b is composed of the left sub-view angle a of the camera 2 and the right sub-view angle b of the camera 1;

[0014] The binocular pair 2b-3a is composed of the right sub-view angle b of the camera 2 and the left sub-view angle a of the camera 3.

[0015] By the above technical solution, the high-precision feature tracking and extraction in the omnidirectional environment can be realized. The design of the four groups of virtual binocular combinations enables the system to realize the seamless coverage of the omnidirectional space, and is particularly suitable for the feature tracking task in the omnidirectional environment. When the target feature points appear in the field of view in the directions of the camera 0 and the camera 2, or the number of features is lower than the threshold, the system can quickly switch to the directions of the camera 1 and the camera 3 to continue tracking, so that the continuity of the feature points is ensured.

[0016] Specifically, this non-parallel binocular combination design forms four independent and complementary virtual binocular systems by cross-pairing the sub-views of adjacent cameras. This design not only avoids the blind area problem of traditional parallel binoculars, but also ensures that suitable matching points can be found at any viewing angle through the cooperation of the four directions. At the same time, this design also takes into account the distortion characteristics of fisheye cameras, effectively reducing the influence of distortion on feature matching by dividing the camera field of view along the central vertical line into sub-views.

[0017] In practical applications, this four-group virtual binocular combination strategy can better adapt to dynamic environmental changes, especially in fast motion or complex scenes, and can quickly stabilize feature tracking to ensure that the system always works in the best state. The innovative design of this combination method is of great significance for improving the overall performance and reliability of computer vision systems.

[0018] Preferably, the spherical epipolar projection in step two is implemented by the function fgt, which is a projection function from a spatial point to an image plane based on the MEI fisheye model. The specific process of spherical epipolar generation includes:

[0019] (a) Reverse project the pixel coordinates to the unit sphere;

[0020] (b) According to the depth range d min -d max Generate a spatial point ray;

[0021] (c) Call the projection function fgt to generate the spherical epipolar trajectory of the target view.

[0022] Through the above technical solution, the application can realize high-precision feature matching in an omnidirectional environment. This spherical epipolar projection technology based on the MEI fisheye model can effectively handle the distortion problem of fisheye images, ensuring accurate matching of feature points at different viewing angles.

[0023] Specifically, the fgt function uses the parameters of the MEI model to project the spatial point along the depth range d min -d max to generate a spherical epipolar line, and then searches for matching points in combination with sparse optical flow and affine constraints. This projection method not only takes into account the distortion characteristics of fisheye lenses, but also ensures the continuity and consistency of matching points at different depths through the construction of spherical epipolar lines. At the same time, this projection method based on the MEI model can effectively avoid the matching errors that may occur in traditional projection methods in fisheye images.

[0024] In practical applications, the spherical polar projection technology based on the MEI model can better adapt to dynamic environmental changes, especially in fast motion or complex scenes, can quickly stabilize feature matching, and ensure that the system always works in the best state. The innovative design of this projection method is of great significance to improve the overall performance and reliability of the computer vision system.

[0025] Preferably, the 3D points successfully triangulated in step two are verified by re-projection error, and the error is the spherical angle error between the observation vector and the projection vector, and the calculation includes the following formula:

[0026] .

[0027] Through the above technical scheme, the high-precision verification of the triangulation point can be realized. This verification method based on re-projection error can effectively eliminate the wrong 3D points and ensure the reconstruction accuracy of the feature points. At the same time, by setting the threshold, the calculation complexity can be controlled while ensuring the accuracy, and the real-time performance of the system is improved.

[0028] In practical applications, the re-projection error verification mechanism can better adapt to dynamic environmental changes, especially in fast motion or complex scenes, can quickly eliminate the wrong 3D points, and ensure that the system always works in the best state. The innovative design of this verification method is of great significance to improve the overall performance and reliability of the computer vision system.

[0029] Preferably, the position consistency loss function of step four is:

[0030] ;

[0031] Wherein: P warped is the extracted corner point on the distorted image, and the corner point on the original image is taken as the pseudo-true value P original , P original is projected to the distorted image coordinate system to obtain the expected position P warped_gt .

[0032] Through the above technical scheme, the accurate supervision of the feature point position can be realized. The design of this position consistency loss function considers the position difference of the feature points on the original image and the distorted image, and ensures the position consistency of the feature extraction network under different viewing angles.

[0033] In practical applications, the position consistency loss function can better adapt to dynamic environmental changes, especially in fast motion or complex scenes, can ensure the continuity and consistency of the feature points under different viewing angles, and improve the overall performance of the system.

[0034] Preferably, the super point network of step four includes:

[0035] shared encoder: extracts intermediate features of the input grayscale image;

[0036] detection head: outputs an angle point probability map;

[0037] description head: outputs an angle point descriptor.

[0038] Through the above technical solutions, the application can realize efficient and accurate feature point detection and description. The design of this super point network extracts the intermediate features of the image through a shared encoder, and then generates angle point detection and descriptor through two special head networks respectively, ensuring the efficiency and accuracy of feature extraction.

[0039] Specifically, the design of the shared encoder can avoid repeated calculation and improve computational efficiency; the detection head can accurately detect the position of the feature points in the image by outputting an angle point probability map; and the description head generates a unique feature vector for each feature point by outputting an angle point descriptor, which is used for subsequent feature matching.

[0040] In practical applications, this network structure can better adapt to dynamic environmental changes, especially in fast motion or complex scenes, and can quickly and accurately extract and match feature points to improve the overall performance of the system. The innovation of this network design is of great significance to improving the overall performance and reliability of computer vision systems.

[0041] Preferably, the photometric enhancement operation of step four specifically includes contrast adjustment, brightness adjustment, motion blur simulation, fog simulation, shadow simulation, spot noise addition, and Gaussian noise addition, etc. Specifically:

[0042] Contrast and brightness adjustment: simulate the overall change of environmental light intensity, such as from indoor to outdoor, or light flickering, and its formula is:

[0043] ;

[0044] Wherein: is the pixel gray value of the input image at coordinates , is the pixel gray value of the output image at the corresponding position, is the contrast gain factor, is the brightness bias;

[0045] Gaussian noise addition: simulates the electronic noise generated by the image sensor under insufficient light or high temperature conditions, and its formula is:

[0046] ;

[0047] Wherein: ​is a random number sampled from a Gaussian distribution with a mean of 0 and a variance of ;

[0048] Motion blur simulation: simulates the trailing effect when there is fast relative motion between the camera and the object in the scene. The formula is:

[0049] ;

[0050] where: represents a two-dimensional convolution operation;

[0051] Fog simulation: simulates the imaging effect in low-visibility environments such as fog, haze, or underwater. The formula is:

[0052] ;

[0053] where: is the transmittance map, which represents the proportion of light that can pass through the fog to reach the camera at the scene point . The transmittance can be calculated by the formula:

[0054] ;

[0055] where, is the scattering coefficient, which controls the density of the fog. is the depth map of the scene, which represents the distance of the real-world object corresponding to each pixel point;

[0056] Shadow simulation: simulates the irregularly shaped shadow cast by objects (such as shelves, other AGVs) on the image. The formula is:

[0057] ;

[0058] where: is the value of the blurred shadow mask at point ( between 0 and 1), is the shadow intensity factor, which is a number between 0 and 1, controlling the degree of darkening in the shadow area;

[0059] Speckle noise addition: simulates the common noise in coherent imaging systems, which is related to signal intensity:

[0060] ;

[0061] where: is a random number sampled from a uniform distribution with a mean of 0 and a variance .

[0062] By the technical solution, the image can be comprehensively enhanced, and the robustness of feature extraction is improved. The design of the luminosity enhancement operation ensures the adaptability of the feature extraction network in different environments by simulating the changes of illumination and imaging conditions in various practical scenarios.

[0063] In practical applications, the comprehensive luminosity enhancement strategy can improve the generalization ability of feature extraction, enhance the stability of feature points in different environments, and improve the accuracy of feature matching.

[0064] Preferably, step four further comprises defining a fisheye distortion mask for each image to mask the severely distorted area at the edge of the field of view, and triggering a cooperative supplement mechanism of the camera 1 and camera 3 view angles when the number of features is insufficient.

[0065] By the technical solution, the fisheye distortion can be effectively processed and the distribution of feature points can be optimized. The processing method based on the mask and multi-view cooperation can effectively avoid the extraction of feature points in the edge distortion area, and through the cooperative supplement mechanism of the camera view angles, the distribution density and quality of the feature points are ensured.

[0066] Specifically, the fisheye distortion mask defined for each image can accurately mask the severely distorted area at the edge of the field of view, avoiding the extraction of poor quality feature points in these areas. When the number of features is insufficient, the system will automatically trigger the cooperative supplement mechanism of the camera 1 and camera 3 view angles to obtain more effective feature points through the complementary characteristics of multi-view. This design not only solves the feature extraction problem caused by fisheye camera distortion, but also improves the distribution density and quality of feature points through the multi-view cooperative mechanism.

[0067] In practical applications, this processing mechanism can better adapt to the feature extraction requirements in complex scenes, especially in the application scenarios of fisheye cameras, which can ensure the extraction of high-quality feature points and improve the accuracy and reliability of feature matching. The innovative design of this processing method is of great significance to improving the performance and reliability of computer vision systems in fisheye camera applications.

[0068] Preferably, step two further comprises: using a binary search mask generated based on image gradient amplitude to eliminate low-efficiency matching areas.

[0069] By the technical solution, the efficiency and accuracy of feature point matching can be significantly improved. Specifically, in step two, a binary search mask generated based on image gradient amplitude is used to effectively eliminate low-efficiency matching areas. This method analyzes the gradient information of pixels in the image to identify areas that contribute more to feature matching, so as to preferentially focus on these areas for feature point detection and matching.

[0070] Specifically, the generated binary search mask can clearly distinguish between key feature regions and low-efficiency matching regions, ensuring that the system concentrates resources on high-quality feature points during feature point extraction and matching. This approach not only optimizes the use of computing resources, but also improves the accuracy of matching and reduces the risk of false matching.

[0071] In practical applications, this gradient amplitude-based mask mechanism can quickly adapt to different scenes and lighting conditions, improving the real-time performance and robustness of feature point matching. The innovative design of this method not only enhances the performance of the algorithm in complex visual environments, but also provides strong support for the overall efficiency of computer vision systems.

[0072] Preferably, the extrinsic parameter optimization of step five includes:

[0073] (a) Reprojection error function construction, using the spherical angle error calculation consistent with claim 4:

[0074] ;

[0075] wherein: is the observed normalized spherical coordinate, is the three-dimensional point after transformation of the normalized spherical coordinate;

[0076] (b) Prioritize baseline length optimization by scaling the translation vector:

[0077] ;

[0078] wherein: is the spherical coordinate calculated using the scaled baseline ;

[0079] (c) When the reprojection error is higher than the threshold , jointly optimize the rotation matrix and translation vector:

[0080] ;

[0081] (d) Use the Levenberg-Marquardt algorithm to iteratively solve:

[0082] ;

[0083] wherein: is the Jacobian matrix of the spherical error function, is the transpose of the Jacobian matrix of the spherical error function, is the identity matrix, is the spherical angle error vector, is the damping factor.

[0084] By the above technical solution, the online optimization and self-calibration capability of the external parameter are realized. The baseline length is preferentially optimized to quickly correct the scale drift caused by vibration and the like, and the subsequent joint optimization can more accurately compensate for the small pose changes of the camera. The Levenberg-Marquardt algorithm is adopted to ensure the rapid convergence and stability of the optimization process, thereby significantly improving the accuracy and robustness of the three-dimensional perception of the system in long-term operation.

[0085] Compared with the prior art, the present application has the following advantages:

[0086] 1. The present application divides the field of view of each channel of the four fisheye lenses into left and right sub-angles, and constructs four virtual binocular combinations. This design realizes eight-view equivalent coverage without increasing the number of physical cameras, and forms a high-overlapping co-view area between adjacent cameras. Based on the pre-calibrated external parameters and the MEI fisheye model, the spherical epipolar lines are directly generated for matching search on the distorted images, avoiding the edge stretching and resampling errors caused by traditional panoramic stitching. Combined with the joint optimization mechanism of sparse optical flow and affine constraint, the geometric distortion problem caused by barrel distortion in the fisheye image with a large field of view is effectively overcome, ensuring stable feature correlation across the view angle under a 220° ultra-wide view angle, and greatly reducing the mismatch rate.

[0087] 2. The present application proposes a time-space two-dimensional fault-tolerant mechanism to address the challenges of occlusion and environmental interference: a dynamic direction switching design actively switches to camera 1 or 3 as the main tracking source when the tracking feature number in camera 0 or 2 direction is below the threshold due to occlusion, while supplementing new feature points, avoiding system collapse caused by local field of view failure; the distortion-invariant feature training uses a superpoint network combined with fisheye-specific data augmentation strategy, and innovatively introduces a position consistency loss function; this loss supervises the projection of the original image corner points to the distorted image coordinate system, forcing the network to learn the distortion-invariant feature expression; the cooperation of the two enables the system to maintain high feature repetition rate and tracking continuity in frequent occlusion, sudden changes in illumination, and strong distortion edge regions. BRIEF DESCRIPTION OF DRAWINGS

[0088] Figure 1 is a schematic diagram of the four-eye omnidirectional fisheye camera configuration scheme of the present application;

[0089] Figure 2 is a schematic diagram of the spherical epipolar search of the present application;

[0090] Figure 3 is a schematic diagram of the re-projection error verification of the triangulated 3D points. DETAILED DESCRIPTION

[0091] With reference to the accompanying drawings: the technical solutions in the embodiments of the present application will be described clearly and completely, obviously, the described embodiments are only part of the embodiments of the present application, and not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor belong to the scope of protection of the present application.

[0092] A omnidirectional fisheye image feature tracking and extraction method based on non-parallel virtual binoculars, comprising:

[0093] Step one: construct a four-fisheye camera system, four fisheye lenses with a field of view angle greater than 220° are arranged in a square, and the cameras are numbered clockwise as 0, 1, 2 and 3; the field of view of each camera is divided into a left sub-angle a and a right sub-angle b along the central vertical line; based on the pre-calibration external parameters, four groups of virtual non-parallel binoculars are constructed: 0a-3b, 0b-1a, 2a-1b and 2b-3a; the schematic diagram is shown in Figure 1 ;

[0094] Step two: based on the pre-calibration external parameters, the relative pose transformation T 12 is obtained, and the MEI model is used to project the pixels u r of the image I i in the target view along the depth range d min -d max to generate a spherical polar line, and the matching points are searched combining sparse optical flow and affine constraint;

[0095] Step three: based on the uniform speed model or IMU pre-integration prediction pose, the three-dimensional points of the historical frame are projected to the current frame tracking; when the number of tracking features in the directions of camera 0 and camera 2 is lower than the set threshold, new feature points are supplemented, and when occlusion is detected, the main tracking direction is switched to the directions of camera 1 and camera 3;

[0096] Step four: super point network is used to extract features, and contrast adjustment, brightness adjustment, motion blur, fogging, shadow, speckle noise and Gaussian noise enhancement are applied during training; MEI distortion is applied to the original image to generate a distorted image, and the training is supervised by a position consistency loss function;

[0097] Step five: use the co-view feature points to construct a re-projection error function, and optimize the external parameters nonlinearly.

[0098] The projection function of a spatial point to an image plane based on the MEI fisheye model is as follows:

[0099] Based on the known multi-view extrinsic parameters and MEI fisheye model, the cameras 0a-3b, 0b-1a, 2a-1b, 2b-3a are configured into four groups of binoculars, and the epipolar geometry search is performed in the image plane. The matching process combines sparse optical flow and affine constraints, and removes inefficient areas through a custom search mask, effectively improving the robustness and accuracy of cross-view feature matching. At the same time, it adapts to the epipolar search method of large FOV cameras, such as Figure 2 .

[0100] The specific implementation steps of spherical epipolar search are as follows:

[0101] (a) Pixel back-projection to unit sphere

[0102] Given a pixel point on the reference image , the point on the unit sphere is obtained through the MEI model back-projection function:

[0103] ;

[0104] The MEI back-projection process is:

[0105] Calculate the normalized coordinates: , ;

[0106] Calculate the radial distance: ;

[0107] Solve the depth: ;

[0108] Get the spherical point: .

[0109] (b) Generate depth sampling rays

[0110] Uniformly sample M depth values along the depth range :

[0111] ;

[0112] For each depth value, calculate the corresponding 3D space point: .

[0113] (c) Project to generate spherical epipolar lines

[0114] Project the space point sequence to the target image : ;

[0115] The MEI projection function is:

[0116] Normalization: , ;

[0117] Compute the distortion radius: ;

[0118] Project to pixel: , .

[0119] (d) Polar line matching search

[0120] Normalized Cross Correlation (NCC) is used as the matching measure:

[0121] ;

[0122] Optimal depth: .

[0123] For two poses C1 and C2, given the relative pose transformation T 12 , there is a pixel u r and its corresponding patch in image I i , then there should be a corresponding image patch in image C2 which can be projected to the image plane by f gt . Assume the depth range of the corresponding 3D point of u i is between d min and d max , the line between d min and d max will form a polar line in the corresponding image frame of C2, i.e. Figure 2 The polar line search on the UnitSphere of Figure 3 continuously minimizes the depth error to get the corresponding match, and the triangulated 3D point is obtained at the same time. At the same time, the error verification is carried out for the triangulation success point, and the error calculation method of the application is shown in

[0124] Example one: large-scale warehouse logistics AGV navigation

[0125] Scene description:

[0126] A large-scale automated warehouse, the internal environment characteristics include:

[0127] Structured environment: the channel is narrow (about 1.8 meters wide), and both sides are high shelves (about 8 meters high), and the shelf structure has a large number of repetitive textures.

[0128] Dynamic interference: other AGVs, workers or forklifts pass through the channel, which will frequently cause partial visual occlusion.

[0129] Complex lighting: there are fluorescent lamps on the top, which may have stroboscopic; some areas are close to the window, and the lighting will change with time; the shelves will cast a large area of shadow.

[0130] Operating condition: AGV needs to run continuously for a long time, and the vibration of the vehicle body and the temperature difference of the environment may cause the camera external parameter to drift slightly.

[0131] Technical process:

[0132] 1. Virtual binocular activation and task allocation

[0133] When the AGV starts, the system activates all four groups of virtual non-parallel binoculars according to the pre-calibrated external parameters, and allocates different main monitoring tasks to them:

[0134] Forward main binocular pair (0b-1a): mainly responsible for monitoring the driving path in front of the AGV, extracting the landmark and ground texture at the far end of the channel, for heading control and speed planning.

[0135] Rear main binocular pair (2b-3a): mainly responsible for monitoring the path behind the AGV, forming a long baseline with the forward binocular pair, providing stronger rotation and translation observation for the system, and realizing closed-loop detection of motion.

[0136] Left task binocular pair (0a-3b): mainly responsible for accurately perceiving the left shelf, identifying the shelf position label, two-dimensional code or structure corner point, for accurate positioning and parking work of AGV between shelves.

[0137] Right task binocular pair (2a-1b): similar to the left binocular pair, responsible for accurately perceiving the right shelf, ensuring that the AGV always drives safely in the center of the channel.

[0138] 2. Dynamic epipolar tracking and fault-tolerant switching

[0139] Normal tracking: when the AGV drives along the channel, the system generates a spherical epipolar line based on the MEI model along the depth range of 0.5m to 8m in the field of view of camera 1b, and combines sparse optical flow for stable tracking, using the shelf column corner points detected by the right task binocular pair (2a-1b).

[0140] Shielding response and switching: when a forklift enters from the side channel, temporarily blocking the view of camera 2, causing the tracking feature number of binocular pairs (2a-1b) and (2b-3a) to drop sharply below the preset threshold of 30. The system immediately determines that the main tracking direction is blocked, and automatically transfers the weight of positioning calculation mainly to the unblocked forward (0b-1a) and left (0a-3b) binocular pairs.

[0141] Feature complementation: At the same time of switching, the system calls the super point network to actively detect and complement new stable feature points (such as the bolts or labels of the shelves on the other side) in the sub-visual fields of the un-occluded cameras 0a and 1a, ensuring that the total number of feature points is always maintained at a healthy level (such as more than 100), thereby ensuring the continuity and stability of the pose solution, and avoiding system drift or collapse caused by the failure of single-side vision.

[0142] Pruning of inefficient areas: using a binary search mask generated based on image gradient amplitude, the high-reflective area of the stainless steel guard plate on the shelf and the overexposed area formed by direct light are actively ignored, and the computing resources are concentrated in the effective area with rich texture.

[0143] 3. Distortion-robust feature extraction and training

[0144] During the model training phase, the data set is specially enhanced for the warehouse environment:

[0145] Light intensity enhancement: apply ±30% brightness adjustment to simulate the light difference in different areas; ±20% contrast adjustment to simulate shadow changes; simulate motion blur caused by AGV starting or turning; simulate the shadows cast by shelves and other AGVs.

[0146] Noise enhancement: add Gaussian noise and speckle noise to simulate electronic noise generated by sensors in weak light environments such as the bottom of the shelf.

[0147] Position consistency loss supervision: the feature points such as shelf label corners and bolts extracted on the original image are used as pseudo-true values, which are projected into the enhanced distortion image through the MEI model to obtain the expected position. The network is supervised by using the position consistency loss function, which makes the feature points learned by the network maintain high position consistency and robustness even in the strong distortion area at the edge of the fisheye image.

[0148] 4. Online self-calibration of external parameters

[0149] Triggering condition: after AGV accumulates 2 hours of running, or when the system background monitors that the average re-projection error of the common-view feature points continuously exceeds 1.5 pixels, the system determines that the external parameters may have drifted due to accumulated vibration, and automatically triggers online optimization.

[0150] Optimization process: the system uses the long-term stable common-view points (such as the fixed shelf column corners) in the scene to construct a re-projection error function.

[0151] First, the baseline length (i.e. the scale of the translation vector t) is optimized first to quickly correct the main scale drift.

[0152] If the optimized step error is still higher than the threshold (e.g. 1.0 pixel), further jointly optimize the rotation matrix R and translation vector t for more fine-tuning.

[0153] The entire nonlinear optimization process uses the Levenberg-Marquardt algorithm to iteratively solve until the error converges.

[0154] Effects:

[0155] By implementing the present application, the AGV navigation system achieves the following performance improvements:

[0156] Accuracy: In a typical narrow corridor with a width of 1.8 meters, continuous back-and-forth operation exceeds 500 meters. Compared with the ground laser radar trajectory, the root mean square error (RMSE) of the end-to-end positioning trajectory is less than 2.0 cm.

[0157] Robustness: In a simulated forklift and personnel frequently passing through (visual occlusion time ratio up to 15%) stress test, the feature tracking success rate is maintained at more than 98%, and there is no positioning system reset or tracking failure event due to local occlusion throughout the journey.

[0158] Continuity: Thanks to online self-calibration of external parameters, the device's positioning accuracy does not decrease significantly after 8 hours of continuous operation, demonstrating the system's excellent long-term operation stability.

[0159] Example Two: Three-dimensional modeling of mine tunnel inspection robots

[0160] Scene Description:

[0161] Unstructured environment: The shape of the tunnel is irregular, the texture of the rock wall is complex and variable, and there are no obvious artificial geometric features.

[0162] Extreme lighting: No natural light, completely relying on the robot's own high-power searchlight. This results in a high-light area directly under the strong light, a completely dark area with no light, and a bright-dark transition area with obvious trailing.

[0163] Bad atmosphere: Different concentrations of dust and water vapor are diffused in the air, severely affecting image clarity and visibility.

[0164] Severe vibration: The robot travels on rough and uneven mine paths or tracks, causing severe jolts, posing a serious challenge to the stability of the camera's external parameters.

[0165] Technical Process:

[0166] 1. Omnidirectional virtual binocular configuration

[0167] The robot's omnidirectional perception is crucial for mine safety. Four virtual binocular pairs are configured for full-space monitoring:

[0168] Front / back facing binocular pair (0b-1a, 2b-3a): used for main navigation of the robot in the tunnel, perceiving the path ahead and the trajectory behind.

[0169] Left / right facing binocular pair (0a-3b, 2a-1b): used for detailed scanning of the rock walls on both sides of the tunnel, detecting cracks, water seepage, and the state of the support frame.

[0170] Implicit up / down perspective: with the ultra-wide field of view of the fisheye lens, the above binocular pairs simultaneously cover the roof (detecting dangerous rocks) and the floor (track or road conditions), achieving 3D environment perception without dead angles.

[0171] 2. Feature tracking in high dynamic environments

[0172] Feature selection: preferentially select relatively stable structural features in the mine, such as bolts on the support frame, track joints, and unique ore nodes on the rock wall as long-term tracking landmarks.

[0173] Fault-tolerant switching: when the robot passes through an area with high dust concentration, causing the front-facing camera (0, 1) to have a blurred view, and the number of tracking features is below the safety threshold of 40, the system automatically switches the positioning weight to the less affected side-facing (0a-3b, 2a-1b) and rear-facing (2b-3a) virtual binocular pairs, relying on feature tracking of the side walls and the path already passed to maintain pose estimation, ensuring that the robot does not lose its way in a "blind" state. When driving out of the dust area, the system replenishes feature points in the front view.

[0174] Dynamic mask: use a gradient amplitude mask to actively exclude the textureless highlight spot area formed by the direct reflection of the searchlight on the rock wall.

[0175] 3. Training of specialized features in mine environments

[0176] To cope with the extreme environment of the mine, specific data augmentation is applied when training the feature extraction network:

[0177] Extreme lighting simulation: apply a non-linear brightness transformation to the image, increasing the local area brightness by 80% and reducing the other area by 50%, simulating the effect of the searchlight; significantly reduce the contrast to simulate the weak light area.

[0178] Dust atomization simulation: add different concentration levels of atomization effects to simulate different levels of dust environment in the tunnel, and combine with speckle noise to simulate the scattering of light on dust particles.

[0179] Distortion invariance enhancement: also use a position consistency loss function to focus on supervising the network's recognition consistency of key features such as rock cracks and bolts in areas with dramatic changes in lighting and strong distortion, ensuring that high-quality features can be extracted for 3D modeling even in harsh conditions.

[0180] 4. Online anti-shake optimization

[0181] Trigger condition: The system detects that the robot is experiencing severe vibration (such as angular velocity or acceleration exceeding threshold) through IMU data, or when the sliding average of re-projection error exceeds 2.0 pixels, triggering online optimization.

[0182] Optimization goal: This function is crucial for the accuracy of three-dimensional modeling. It can compensate for the physical position shift of the camera caused by severe shaking in real time, avoiding "ghosting" or "layering" in the map.

[0183] Optimization process: Using co-visible track bolt points or support frame connection points as stable anchor points, the Levenberg-Marquardt algorithm is used to quickly iterate and optimize the extrinsic parameters R and t, ensuring that each frame of image can be accurately aligned to the global map.

[0184] Effect:

[0185] Mapping quality: In a harsh environment where visibility is reduced to 5 meters due to dust, the robot can still complete a continuous mapping task. The final generated three-dimensional point cloud model has a structural integrity of more than 96%, and the error in measuring the size of key structures such as support frames compared to laser scanning results is less than 1%.

[0186] Positioning robustness: Throughout the inspection task, even when frequently entering and exiting high-dust areas, the system has not lost tracking once. Compared to the true value of the trajectory optimized offline, the relative attitude error of the real-time positioning trajectory is less than 0.5%.

[0187] The above is only the preferred specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art, within the technical scope disclosed by the present application, according to the technical solution and inventive concept of the present application, makes equivalent replacement or change, should be covered within the protection scope of the present application.

Claims

1. A method for feature tracking and extraction of omnidirectional fisheye images based on non-parallel virtual binoculars, characterized in that, include: Step 1: Construct a four-eye fisheye camera system with a square layout of four fisheye lenses with a field of view greater than 220°. The cameras are numbered 0, 1, 2, and 3 clockwise. The field of view of each camera is divided into a left sub-viewpoint a and a right sub-viewpoint b along the central vertical line. Based on the pre-calibrated extrinsic parameters, construct four sets of virtual non-parallel binocular combinations: 0a-3b, 0b-1a, 2a-1b, and 2b-3a. Step 2: Obtain the relative pose transformation T based on the pre-calibrated extrinsic parameters 12 The MEI model is used to analyze image I. r pixels u i Within the depth range of the target's viewpoint d min -d max The projection generates spherical epipolar lines, and the matching points are searched by combining sparse optical flow and affine constraints. Step 3: Predict the pose of the current frame based on a uniform velocity model or IMU pre-integration. For three-dimensional spatial points that have already been triangulated in historical frames Using the MEI model employed in step two, the 3D point is projected onto the current frame image to obtain its predicted position. This enables tracking; when the number of tracking features in the directions of camera 0 and camera 2 is lower than the set threshold, new feature points are added; when occlusion is detected, the main tracking direction is switched to the directions of camera 1 and camera 3. Step 4: Use a superpoint network to extract features. During training, contrast adjustment, brightness adjustment, motion blur, fogging, shadows, speckle noise, and Gaussian noise enhancement can be applied. Apply MEI distortion to the original image to generate a distorted image, and supervise training through the position consistency loss function. Step 5: Construct a reprojection error function using common-view feature points and nonlinearly optimize the extrinsic parameters.

2. The method according to claim 1, characterized in that, The four sets of virtual non-parallel binocular combinations in step one are defined as follows: Binocular view 0a-3b: The combination of the left sub-view a of camera 0 and the right sub-view b of camera 3; Binocular view 0b-1a: The combination of the right sub-viewpoint b of camera 0 and the left sub-viewpoint a of camera 1; Binocular pair 2a-1b: The combination of the left sub-view a of camera 2 and the right sub-view b of camera 1; Binocular pair 2b-3a: The right sub-viewpoint b of camera 2 and the left sub-viewpoint a of camera 3 are combined.

3. The method according to claim 1, characterized in that, The spherical epipolar projection in step two is implemented using the function fgt, where fgt is the projection function from a spatial point to the image plane based on the MEI fisheye model. The specific process of generating the spherical epipolar lines includes: (a) Project the pixel coordinates backwards onto the unit sphere; (b) Based on the depth range d min -d max Generate spatial point rays; (c) Call the projection function fgt to generate the spherical epipolar trajectory of the target viewpoint.

4. The method according to claim 1, characterized in that, In step two, the reprojection error of the successfully triangulated 3D points is verified. The error is the spherical angle error between the observation vector and the projection vector, and its calculation includes the following formula: ； in: For the first The normalized representation of an observation vector on a unit sphere. The normalized representation of the projection vector obtained by reprojecting the same 3D point according to the current camera pose on the unit sphere.

5. The method according to claim 1, characterized in that, The position consistency loss function in step four is: ； Where: P warped To extract corner points from the distorted image, corner points are extracted from the original image and used as pseudo-true values ​​P. original , will P original Projecting onto the distortion map coordinate system, we obtain the desired position P. warped_gt .

6. The method according to claim 1, characterized in that, The superpoint network in step four includes: Shared encoder: Extracts intermediate features from the input grayscale image; Detection head: Outputs a corner probability map; Description Header: Outputs corner descriptors.

7. The method according to claim 1, characterized in that, The photometric enhancement operation in step four specifically includes: contrast adjustment, brightness adjustment, motion blur simulation, fog simulation, shadow simulation, speckle noise addition, and Gaussian noise addition, etc., in detail: Contrast and brightness adjustment: Simulates the overall change in ambient light intensity; the formula is: ； in: The input image is in coordinates The pixel grayscale value at that location, It outputs the pixel grayscale value at the corresponding position in the image. It is the contrast gain factor. It is a brightness bias; Gaussian noise addition: This simulates the electronic noise generated by the image sensor, and its formula is: ； in: It is a random number sampled from a Gaussian distribution with a mean of 0 and a variance of . ; Motion blur simulation: Simulates the motion blur effect produced when objects in a camera or scene move rapidly relative to each other. The formula is: ； in: Represents a two-dimensional convolution operation; Fog simulation: Simulates the imaging effect in a foggy environment; its formula is: ； in: This is a transmittance map, representing the data at scene points. The proportion of light that can penetrate the fog and reach the camera, and the transmittance. It can be calculated using a formula, which is: ； in: It is the scattering coefficient, which controls the concentration of fog. It is a depth map of the scene, representing the distance of each pixel to the real-world object; Shadow simulation: Simulates the irregularly shaped shadows cast by objects on an image; the formula is: ； in: It's the blurred shadow mask at the point. The value at that location, It is the shadow intensity factor, which controls the degree to which shadow areas become darker; Speckle noise addition: A common type of noise in simulated coherent imaging systems, characterized by its dependence on signal strength. ； in: It is a value with a mean of 0 and a variance of 0. Random numbers sampled from a uniform distribution.

8. The method according to claim 1, characterized in that, Step four also includes defining a fisheye distortion mask for each image to shield severely distorted areas at the edge of the field of view, and triggering a collaborative supplementation mechanism of the viewpoints of camera 1 and camera 3 when the number of features is insufficient.

9. The method according to claim 1, characterized in that, Step two further includes: using a binarized search mask generated based on image gradient magnitude to eliminate inefficient matching regions.

10. The method according to claim 1, characterized in that, Step five, extrinsic parameter optimization, includes: (a) Construction of reprojection error function and calculation of spherical angle error: ； in: For the observed normalized spherical coordinates, For three-dimensional points After transformation Normalized spherical coordinates afterward; Where R is the rotation matrix in the camera's extrinsic parameters, and t is the translation vector corresponding to the rotation matrix. The three-dimensional coordinates of the three-dimensional point in the target camera coordinate system after transformation by R and t; (b) Prioritize optimizing the baseline length by scaling the translation vector: ； in: To use the scaled baseline Calculate the spherical coordinates; (c) When the reprojection error is higher than the threshold At the same time, jointly optimize the rotation matrix and translation vector: ； (d) Solve iteratively using the Levenberg-Marquardt algorithm: ； in: Let be the Jacobian matrix of the spherical error function. Let be the transpose of the Jacobian matrix of the spherical error function. It is the identity matrix. Let be the spherical angle error vector. is the damping factor.

Images