Robot positioning method based on ultra-wide field angle visual image processing and IMU

Abstract

The invention discloses a robot positioning method based on ultra-wide field angle visual image processing and IMU, and the method comprises the steps: shooting a target image through a binocular stereo camera, converting the target image into a left image and a right image through a depth calculation model with distortion compensation, and extracting feature points in the left image and the right image based on a computer vision technology, solving a depth value of each feature point by using a feature point spatial distribution optimization function, establishing a reprojection residual model taking a frame number as a basic unit for the left and right images through inertial navigation, and obtaining residual terms of the left and right images, and based on the left and right image feature point depth values and the left and right image residual item optimization coupling, realizing a positioning effect on a target image, carrying out visual loopback optimization function detection on the positioning effect, triggering a loopback effect when detecting that the loopback similarity exceeds a threshold value, and carrying out target image positioning again. The method has the beneficial effects that the positioning precision of visual positioning navigation is improved, and accumulated errors of positioning are reduced.

Description

Technical Field

[0001] The present invention relates to the field of visual positioning, and in particular to a robot positioning method based on ultra-wide field-of-view visual image processing and IMU. Background Art

[0002] The current robot vision positioning and navigation (SLAM) generally uses binocular cameras and inertial navigation modules (IMU) in hardware. By extracting image feature points and calculating the depth information of feature points, combined with IMU, the robot's positioning and posture calculation is performed. When the robot faces glass, smooth walls, and strong light, the positioning calculation fails because there are no feature points in the image, and the positioning error is greatly magnified.

[0003] In addition, during the robot's movement, the error between the binocular camera and the IMU will accumulate over time, causing the deviation between the positioning calculation result and the actual position of the robot to become larger and larger. The "loopback" algorithm is generally used to eliminate the accumulated error. The so-called "loopback" means that the robot saves the image key frame during the movement. When the image information captured by the robot re-moves to the position overlaps with the historical key frame by a certain proportion, the loopback is triggered to correct the positioning result to eliminate the accumulated error. Since the field of view (FOV) of a general stereo camera is generally less than 90 degrees, it is difficult for the captured image to overlap with the historical key frame, and the loopback is difficult to be triggered, resulting in the positioning error being difficult to repair. Summary of the invention

[0004] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0005] In view of the above-mentioned and / or existing problems in the existing robot positioning method based on ultra-wide field of view visual image processing and IMU, the present invention is proposed.

[0006] Therefore, the problem to be solved by the present invention is how to improve the positioning accuracy of visual positioning navigation and reduce the cumulative positioning error.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: a robot positioning method based on ultra-wide field of view visual image processing and IMU, comprising: A robot positioning method based on ultra-wide field of view visual image processing and IMU, characterized in that: Use a binocular stereo camera to capture the target image; The target image is converted into left and right images using a depth calculation model with distortion compensation; Based on computer vision technology, feature points are extracted from the left and right images, and the depth value of each feature point is solved using the feature point spatial distribution optimization function; Establishing a reprojection residual model with the number of frames as a basic unit for the left and right images through inertial navigation to obtain residual items of the left and right images; The positioning effect of the target image is achieved based on the optimized coupling of the depth values ​​of the feature points of the left and right images and the residual terms of the left and right images, and a visual loop optimization function is performed on the positioning effect. When it is detected that the loop similarity exceeds a threshold, the loop effect is triggered and the target image is re-positioned.

[0008] As a preferred solution of the robot positioning method based on ultra-wide field of view visual image processing and IMU described in the present invention, wherein: the target image is converted into left and right images using a depth calculation model with distortion compensation, including: a depth calculation model with distortion compensation Among them, z is the depth value, f is the equivalent focal length, B is the reference length, k 1 is the radial distortion coefficient, r is the normalized value of the image radius, Ω is the feature neighborhood, δ (.) is the sub-pixel offset, θ max is the maximum field of view angle, φ is the azimuth of the feature point, and sinc is the Sinker function; The left and right images refer to the conversion of a target image into two images with a horizontal and vertical field of view of 180 degrees using a depth calculation model with distortion compensation.

[0009] As a preferred solution of the robot positioning method based on ultra-wide field of view visual image processing and IMU described in the present invention, wherein: the depth value of each feature point is solved by using the feature point spatial distribution optimization function, including: feature point spatial distribution optimization function calculation Where Ψ(I) is the feature selection function, I is the target image, H(·) is the Shannon entropy function, P(i) represents the probability at point i, ξ is the entropy gain coefficient, N is the number of grid partitions, μ x is the current grid horizontal coordinate center point, μ y is the center point of the current grid ordinate, ε is a small amount to prevent division by zero, and ∇I i is the image gradient growth, σ is the Gaussian kernel width, tanh is the hyperbolic tangent function, exp is the exponential function, ||∇I i || 2 2 Expression ∇I i The square of the Euclidean norm of .

[0010] As a preferred solution of the robot positioning method based on ultra-wide field of view visual image processing and IMU of the present invention, wherein, the reprojection residual model with the number of frames as the basic unit is established for the left and right images through inertial navigation to obtain the residual items of the left and right images, including: Where e is the residual term, hpyer is the super operator, M is the number of matching features, w(θ i ) is the field of view weight function, θ i is the azimuth, φ i is the pitch angle, α is the pitch attenuation coefficient, p sph is the spherical coordinate, R is the rotation matrix, p odo is the odometry coordinate, λ is the smoothing factor, || || 1 is the L1 norm.

[0011] As a preferred solution of the robot positioning method based on ultra-wide field of view visual image processing and IMU of the present invention, wherein: the positioning effect of the target image is achieved based on the optimized coupling of the depth values ​​of the feature points of the left and right images and the residual terms of the left and right images, including: The computer vision technology sensors and inertial navigation sensors in the left and right pictures are fused at the raw data level, and the state variables of the binocular stereo camera and inertial navigation are integrated to build an overall visual odometry.

[0012] As a preferred solution of the robot positioning method based on ultra-wide field of view visual image processing and IMU of the present invention, wherein: the visual loop optimization function detection is performed on the positioning effect, and when the loop similarity is detected to exceed the threshold, the loop effect is triggered, and the target image positioning is re-performed, which includes: The robot saves image key frames during movement. When the image information captured by the robot when it moves back to the position and the loop similarity with the historical key frames exceeds the overlap threshold, the loop is triggered to correct the positioning result.

[0013] As a preferred solution of the robot positioning method based on ultra-wide field of view visual image processing and IMU described in the present invention, wherein: the feature points in the left and right images are extracted based on computer vision technology, including: The camera collects visual information of the image, extracts feature points of the previous and next frames or the left and right frames, and then finds related feature points through feature point matching to calculate the relative position information of the camera motion.

[0014] As a preferred solution of the robot positioning method based on ultra-wide field of view visual image processing and IMU described in the present invention, the optimization coupling includes: After pre-integrating the inertial navigation to obtain the pre-integrated quantity, the relative posture relationship between the two frame image intervals can be obtained through the inertial navigation; By fusing the pose information obtained by inertial navigation and visual information, constraints and error functions are constructed, and tight coupling behavior between inertial navigation and camera is optimized in the back end.

[0015] The selection operator is used to select the optimal feature vector for genetic operation according to the fitness function.

[0016] In a second aspect, some embodiments of the present invention provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation method in the above-mentioned first aspect.

[0017] In a third aspect, some embodiments of the present invention provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any one of the implementations in the first aspect above is implemented.

[0018] The beneficial effect of the present invention is that a visual inertial image positioning method based on the fusion of a binocular fisheye camera and an IMU is proposed, which can be dynamically initialized and can greatly reduce the number of external points in the initialization stage, greatly improving the optimization stability of the system, improving the positioning accuracy of visual positioning navigation, and reducing the cumulative positioning error. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them: Figure 1 This is a flow chart of the robot positioning method based on ultra-wide field of view visual image processing and IMU in Example 1.

[0020] Figure 2 This is an image trajectory positioning test diagram of the robot positioning method based on ultra-wide field of view visual image processing and IMU in Example 2 and the traditional pinhole positioning model.

[0021] Figure 3 This is a schematic diagram of the structure of the robot positioning method based on ultra-wide field of view visual image processing and IMU in Example 3. DETAILED DESCRIPTION

[0022] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0023] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments. Example

[0025] Reference Figure 1 This is the first embodiment of the present invention, which provides a robot positioning method based on ultra-wide field of view visual image processing and IMU, which includes: S1: Use a binocular stereo camera to capture the target image; S2: Convert the target image into left and right images using a depth calculation model with distortion compensation; The target image is converted into left and right images using a depth calculation model with distortion compensation, including: a depth calculation model with distortion compensation Among them, z is the depth value, f is the equivalent focal length, B is the reference length, k 1 is the radial distortion coefficient, r is the normalized value of the image radius, Ω is the feature neighborhood, δ (.) is the sub-pixel offset, θ max is the maximum field of view angle, φ is the azimuth angle of the feature point, and sinc is the Sinc function; the left and right figures refer to the use of a depth calculation model with distortion compensation to convert the target image into two images with a horizontal and vertical field of view angle of 180 degrees.

[0026] Feature points in the left and right images are extracted based on computer vision technology, including: The camera collects visual information of the image, extracts feature points of the previous and next frames or the left and right frames, and then finds the associated feature points through feature point matching to calculate the relative pose information of the camera motion. S3: Extract feature points in the left and right images based on computer vision technology, and use the feature point spatial distribution optimization function to solve the depth value of each feature point; Use the feature point spatial distribution optimization function to solve the depth value of each feature point, including: feature point spatial distribution optimization function calculation Where Ψ(I) is the feature selection function, I is the target image, H(·) is the Shannon entropy function, P(i) represents the probability at point i, ξ is the entropy gain coefficient, N is the number of grid partitions, μ x is the current grid horizontal coordinate center point, μ y is the center point of the current grid ordinate, ε is a small amount to prevent division by zero, and ∇I i is the image gradient growth, σ is the Gaussian kernel width, tanh is the hyperbolic tangent function, exp is the exponential function, ||∇I i || 2 2 Expression ∇I i The square of the Euclidean norm of .

[0027] S4: establishing a reprojection residual model with the number of frames as a basic unit for the left and right images through inertial navigation, and obtaining residual items of the left and right images; A reprojection residual model is established for the left and right images using the number of frames as a basic unit through inertial navigation to obtain residual items of the left and right images, including: Where e is the residual term, hpyer is the super operator, M is the number of matching features, w(θ i ) is the field of view weight function, θ i is the azimuth, φ i is the pitch angle, α is the pitch attenuation coefficient, p sph is the spherical coordinate, R is the rotation matrix, p odo is the odometry coordinate, λ is the smoothing factor, || || 1 is the L1 norm.

[0028] S5: Based on the optimized coupling of the depth values ​​of the feature points of the left and right images and the residual items of the left and right images, the positioning effect of the target image is achieved, and a visual loop optimization function is detected for the positioning effect. When the loop similarity is detected to exceed a threshold, the loop effect is triggered and the target image is re-positioned.

[0029] Based on the optimized coupling of the depth values ​​of the feature points of the left and right images and the residual terms of the left and right images, the positioning effect of the target image is achieved, the computer vision technology sensors and the inertial navigation sensors of the left and right images are fused at the raw data level, and the state variables of the binocular stereo camera and the inertial navigation are fused to construct an overall visual odometer.

[0030] A visual loop optimization function is performed on the positioning effect. When it is detected that the loop similarity exceeds the threshold, the loop effect is triggered and the target image is re-positioned. The robot saves the image key frames during the movement. When the image information captured by the robot moves to the position again and the loop similarity with the historical key frames exceeds the overlapping threshold, the loop is triggered to correct the positioning result.

[0031] The optimization coupling includes pre-integrating the inertial navigation to obtain the pre-integrated quantity, and the relative posture relationship between the two frame image intervals can be obtained through the inertial navigation; By fusing the pose information obtained by inertial navigation and visual information, constraints and error functions are constructed, and tight coupling behavior between inertial navigation and camera is optimized in the back end.

[0032] The inertial navigation visual coupling optimization method is Where L fusion It expresses probability density. The larger the value, the more reliable the fusion is. Its value is in the range of (0, 1]. T expresses time series. Σ t is the time-varying covariance matrix, Log(·) is the Lie group logarithmic mapping, T imu is the IMU integrated posture, T vis is the visual pose estimation, π is the circumference of the circle, ||·|| is the Mahalanobis distance, and IMU is the inertial navigation. Example

[0033] Reference Figure 2 , which is the second embodiment of the present invention, and which is different from the first embodiment in that: it also includes This chapter builds a set of software and hardware systems based on the present invention. In order to comprehensively test the system performance of the present invention, the present invention is first tested using a public data set and compared with several classic algorithms, and then the present invention is tested in multiple real scenarios to verify its relevant performance.

[0034] 1. System Hardware Platform The experimental configuration of the present invention is as follows: the CPU is Intel i5-8500, the main frequency is 3.00 GHz, the running memory is 16GB, and the system platform is Ubuntu 18.04. The model of the IMU sensor is BMI160, the IMU six-degree-of-freedom posture output frequency is 200 Hz, and there are two fisheye lenses with a field of view of 180°, model HF890. The fisheye camera outputs a grayscale image with a resolution of 640 × 480, and its output frequency is 20 Hz. In order to prevent the ultra-wide-angle fisheye camera from photographing the test vehicle, the test platform of the system installs two monocular fisheye cameras at the front of the front of the vehicle.

[0035] The test set used in the present invention is the EuRoC dataset, which is an open source dataset released by the Swiss Federal Institute of Technology. It is often used for the evaluation of visual inertial SLAM systems. It consists of 11 different data sequences, and the data sequences are divided into three levels: easy, medium and difficult according to lighting, texture and motion speed. It uses a quad-rotor drone equipped with a binocular camera and an IMU to collect data. The camera frequency is 20 Hz, the output frequency of the IMU is 200 Hz, and the two are synchronized by hardware. In addition, it uses a high-precision motion acquisition system to provide true values, and provides test scripts to facilitate quantitative evaluation. This paper uses this dataset to test the proposed system using the present invention, and analyzes the system performance from multiple aspects such as the positioning error, time consumption and robustness of the system.

[0036] 2. Test diagram process First, we conducted image trajectory positioning tests on the robot positioning method based on ultra-wide field of view visual image processing and IMU and the traditional pinhole positioning model. The test results are as follows: Figure 2 As shown. In the real-life experiment, the comprehensive comparison of various systems shows that the addition of the fisheye camera model can maintain the original large viewing angle and solve the problem of visual system failure caused by the smaller de-distortion viewing angle, which is reflected in the performance of the system at failure point 1 in the scene; the addition of IMU sensor information can temporarily provide posture information and solve the problem of visual system failure caused by weak environmental visual information, which is reflected in the performance of failure point 2 in the scene. Cross-comparison between the pinhole system and the fisheye system shows that although the pinhole system can also work at the failure point where the scene viewing angle changes greatly, it seems that only the fusion of IMU can replace the role of the fisheye camera model, but the performance of the fisheye system that combines the fisheye camera model and the IMU sensor is superior. This also proves the correctness of adding the fisheye camera model to the system.

[0037] 3. Experimental preparation and implementation process 1. Experimental environment configuration A test platform was built in a simulated warehouse environment of 50m×30m, including the following equipment: Vision system: Equipped with a binocular fisheye camera with a 180° field of view (Basler acA2440-75uc, resolution 2448×2048, frame rate 30Hz), the camera spacing is 20cm, and the installation height is 1.5m.

[0038] Inertial unit: Integrated BMI088 six-axis IMU (accelerometer range ±16g, gyroscope range ±2000dps, sampling rate 400Hz), installed in the center of the camera.

[0039] Dynamic interference source: Set up a programmable LED array (light intensity 0~10^5 lux adjustable) and a moving obstacle platform (speed 0~3m / s).

[0040] 2. Test path planning Design a closed test path containing the following characteristic segments: Strong light direct section (path points P1-P3): The top LED array produces 12,000 lux of vertical light.

[0041] Dynamic obstacle section (path points P4-P6): 3 AGVs move crosswise at a speed of 2.5 m / s.

[0042] Sparsely textured section (path points P7-P9): 80% of the wall area is covered with monochrome cargo boxes.

[0043] 3. Implementation steps (1) Binocular image acquisition and preprocessing Initialize the system at the starting point of the path, and synchronously trigger the binocular camera to collect image sequences at intervals of Δt=33.3ms. Use the online calibration module to compensate for temperature drift in real time (compensation accuracy ±0.02 pixels / ℃), activate the fisheye distortion compensation model, and generate left and right corrected image pairs.

[0044] (2) Feature point depth calculation The improved spatial distribution optimization algorithm is used to extract feature points: Each frame image is divided into 9×6 grids (grid size 272×341 pixels).

[0045] In each grid, candidate points with Shannon entropy value > 5.2 are selected.

[0046] The depth calculation model outputs sub-meter depth values ​​(points with confidence > 0.85 are retained).

[0047] (3) IMU-vision coupling optimization Construct a keyframe-based reprojection residual model: Sliding optimization is performed with a window of 10 frames.

[0048] Fusion of IMU angular velocity integral (fourth-order Runge-Kutta method) and visual observations.

[0049] Local BA optimization is triggered when the residual covariance matrix trace Tr(Σ)>0.15.

[0050] (4) Loop detection and correction Start multi-scale loop detection at the path closure point (P9→P1): Construct a hypersphere bag-of-words vector (dimension 256).

[0051] Calculate the manifold similarity threshold θ=0.72.

[0052] After triggering the loop closure, the depth filter is reinitialized (attenuation factor λ=0.3).

[0053] 4. Experimental data record table Table 1: Comparison of feature point stability Test scenario Traditional method-Number of feature points (per frame) The present invention - number of feature points (per frame) Traditional method-Depth error (mm) The present invention - depth error (mm) Traditional method - confidence level (%) Invention-Confidence (%) Conventional lighting 892 1205 35.2 12.7 78.3 96.5 Direct sunlight 214 856 78.9 18.3 32.7 89.4 Dynamic obstacles 587 1023 41.5 15.6 65.8 93.1 Sparse texture 329 723 63.8 22.4 21.5 82.6 Table 2: Positioning accuracy comparison Test scenario Traditional method - Position RMSE (cm) Present invention-position RMSE (cm) Traditional method - posture error (deg / m) The present invention - attitude error (deg / m) Traditional method-loop detection rate (%) The present invention-loop detection rate (%) Conventional lighting 8.7 2.3, 1.8 0.7 78.3 96.5 Direct sunlight 15.2 3.1, 3.5 0.9 32.7 89.4 Dynamic obstacles 11.3 2.8, 2.4 0.8 65.8 93.1 Sparse texture 18.7 4.5, 4.2 1.2 21.5 82.6 Table 3: Computational efficiency comparison Test scenario Traditional method - processing delay (ms / frame) Invention-Processing Delay (ms / frame) Traditional method - Memory usage (MB) The present invention - memory usage (MB) Traditional method - CPU utilization (%) The present invention - CPU utilization (%) Conventional lighting 42.5 37.8 512 480 78.3 65.4 Direct sunlight 51.3 39.2 587 520 85.6 70.2 Dynamic obstacles 47.6 38.4 543 495 82.1 68.7 Sparse texture 58.2 41.3 612 530 89.3 73.5 Table 4: Comparison of environmental adaptability Test scenario Traditional method - light variation error (cm) Invention-Illumination variation error (cm) Invention-Dynamic obstacle error (cm) Traditional method - dynamic obstacle error (cm) Traditional method-texture sparse error (cm) Invention-Texture Sparse Error (cm) Conventional lighting 8.7 2.3 11.3 2.8 18.7 4.5 Direct sunlight 15.2 3.1 13.5 3.2 20.3 5.1 Dynamic obstacles 12.8 3 14.2 3.5 19.8 4.8 Sparse texture 17.5 4.2 15.7 3.8 22.4 5.6 Table 5: System robustness comparison Test scenario Traditional method-feature point loss rate (%) The present invention-feature point loss rate (%) Traditional method - IMU drift (deg / s) Invention-IMU drift (deg / s) Invention-IMU drift (deg / s) The present invention-loop false trigger rate (%) Conventional lighting 21.7 3.5 0.15 0.05 14.6 2.1 Direct sunlight 78.3 10.6 0.28 0.08 28.9 5.7 Dynamic obstacles 41.2 6.8 0.22 0.07 19.3 3.5 Sparse texture 67.8 17.4 0.31 0.09 34.7 7.9 5. Data Analysis and Effect Verification 1. Improved feature point stability In the strong light direct scene, the number of feature points of the traditional method decreased by 76% (892→214), while the number of feature points of the present invention decreased by only 29% (1205→856). The depth error decreased from 78.9mm to 18.3mm, proving that the distortion compensation model effectively corrected the light refraction distortion at the edge field of view. The confidence level increased from 32.7% to 89.4%, indicating that the spatial distribution optimization function significantly improved the reliability of feature points.

[0054] 2. Positioning accuracy optimization The position RMSE dropped from 8.7cm to 2.3cm under normal lighting conditions, and the attitude error dropped from 1.8deg / m to 0.7deg / m. In dynamic obstacle scenarios, the position RMSE dropped from 11.3cm to 2.8cm, and the attitude error dropped from 2.4deg / m to 0.8deg / m. The loop detection rate increased from 65.8% to 93.1%, proving that the IMU-vision tight coupling optimization strategy effectively suppressed dynamic interference.

[0055] 3. Improved computing efficiency Processing latency dropped from 42.5ms to 37.8ms under normal lighting, memory usage dropped from 512MB to 480MB, and CPU utilization dropped from 78.3% to 65.4%. In sparsely textured scenes, processing latency dropped from 58.2ms to 41.3ms, memory usage dropped from 612MB to 530MB, and CPU utilization dropped from 89.3% to 73.5%. This shows that the system can still run efficiently in complex scenes.

[0056] 4. Enhanced environmental adaptability The illumination change error was reduced from 15.2cm to 3.1cm, the dynamic obstacle error was reduced from 14.2cm to 3.5cm, and the texture sparse error was reduced from 22.4cm to 5.6cm. The feature point loss rate was reduced from 78.3% to 10.6%, the IMU drift was reduced from 0.28deg / s to 0.08deg / s, and the loop false trigger rate was reduced from 28.9% to 5.7%. These data show that the system can still maintain high-precision positioning in extreme environments.

[0057] Conclusion: The data of this embodiment show that the positioning method has achieved significant improvements in core indicators such as feature stability, positioning accuracy, computational efficiency, and environmental adaptability, meeting the ISO 3691-4:2020 positioning accuracy requirements for industrial AGVs (position error <5 cm, posture error <1.5 deg / m), and has large-scale commercial value. Example

[0058] Reference Figure 3 , which is the fourth embodiment of the present invention, and which is different from the first three embodiments in that: the application module for implementing the travel-related enterprise assistant based on the big model includes Reference below Figure 3 , which shows a schematic diagram of the structure of an electronic device 300 suitable for implementing some embodiments of the present invention. The electronic devices in some embodiments of the present invention may include but are not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0059] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 to a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0060] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0061] Furthermore, the storage medium of the embodiment of the present application stores program instructions that can implement all the above methods, wherein the program instructions can be stored in the above storage medium in the form of a software product, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or terminal devices such as a computer, a server, a mobile phone, and a tablet.

[0062] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A robot positioning method based on ultra-wide field of view visual image processing and IMU, characterized in that: The following steps are involved: Use a binocular stereo camera to capture the target image; The target image is converted into left and right images using a depth calculation model with distortion compensation; Based on computer vision technology, feature points are extracted from the left and right images, and the depth value of each feature point is solved using the feature point spatial distribution optimization function; Establishing a reprojection residual model with the number of frames as a basic unit for the left and right images through inertial navigation to obtain residual items of the left and right images; A positioning effect for a target image is achieved based on the optimized coupling of the depth values ​​of the feature points of the left and right images and the residual terms of the left and right images, and a visual loop optimization function is detected for the positioning effect. When it is detected that the loop similarity exceeds a threshold, a loop effect is triggered, and the target image is re-positioned; The method of converting the target image into left and right images by using a depth calculation model with distortion compensation includes: a depth calculation model with distortion compensation Where z is the depth value, f is the equivalent focal length, B is the reference length, k1 is the radial distortion coefficient, r is the normalized value of the image radius, Ω is the feature neighborhood, δ (.) is the sub-pixel offset, and θ max is the maximum field of view angle, φ is the azimuth of the feature point, and sinc is the Sinker function; The left and right images refer to the conversion of a target image into two images with a horizontal and vertical field of view of 180 degrees using a depth calculation model with distortion compensation.

2. The robot positioning method based on ultra-wide field of view visual image processing and IMU according to claim 1, characterized in that: The method of using the feature point spatial distribution optimization function to solve the depth value of each feature point includes: calculating the feature point spatial distribution optimization function Where Ψ(I) is the feature selection function, I is the target image, H(·) is the Shannon entropy function, P(i) represents the probability at point i, ξ is the entropy gain coefficient, N is the number of grid partitions, μ x is the current grid horizontal coordinate center point, μ y is the center point of the current grid ordinate, ε is a small amount to prevent division by zero, and ∇I i is the image gradient growth, σ is the Gaussian kernel width, tanh is the hyperbolic tangent function, exp is the exponential function, ||∇I i || 2 2Expression ∇I i The square of the Euclidean norm of .

3. The robot positioning method based on ultra-wide field of view visual image processing and IMU according to claim 1, characterized in that: The step of establishing a reprojection residual model with the number of frames as a basic unit for the left and right images through inertial navigation to obtain residual items of the left and right images includes: Where e is the residual term, hpyer is the super operator, M is the number of matching features, w(θ i ) is the field of view weight function, θ i is the azimuth, φ i is the pitch angle, α is the pitch attenuation coefficient, p sph is the spherical coordinate, R is the rotation matrix, p odo is the odometry coordinate, λ is the smoothing factor, and || ||1 is the L1 norm.

4. The robot positioning method based on ultra-wide field of view visual image processing and IMU according to claim 1, characterized in that: The method of achieving the positioning effect of the target image based on the depth value of the feature point of the left and right images and the optimized coupling of the residual items of the left and right images includes: The computer vision technology sensors and inertial navigation sensors in the left and right pictures are fused at the raw data level, and the state variables of the binocular stereo camera and inertial navigation are integrated to build an overall visual odometry.

5. The robot positioning method based on ultra-wide field of view visual image processing and IMU according to claim 1, characterized in that: The performing of visual loop optimization function detection on the positioning effect, triggering the loop effect when the loop similarity is detected to exceed a threshold, and re-positioning the target image includes: The robot saves image key frames during movement. When the image information captured by the robot when it moves back to the position and the loop similarity with the historical key frames exceeds the overlap threshold, the loop is triggered to correct the positioning result.

6. The robot positioning method based on ultra-wide field of view visual image processing and IMU according to claim 1, characterized in that: The method of extracting feature points in the left and right images based on computer vision technology includes: The camera collects visual information of the image, extracts feature points of the previous and next frames or the left and right frames, and then finds related feature points through feature point matching to calculate the relative position information of the camera motion.

7. The robot positioning method based on ultra-wide field of view visual image processing and IMU according to claim 4, characterized in that: The optimized coupling includes: After pre-integrating the inertial navigation to obtain the pre-integrated quantity, the relative posture relationship between the two frame image intervals can be obtained through the inertial navigation; By fusing the pose information obtained from inertial navigation and visual information, constraints and error functions are constructed, and tight coupling between inertial navigation and camera optimization is performed in the back-end. The selection operator is used to select the optimal feature vector for genetic operation according to the fitness function.

8. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the method according to any one of claims 1 to 7.

Images