A multi-eye vision system and method for AGV navigation

Through a multi-eye vision system, a three-dimensional scene is constructed using multiple binocular cameras and processors, which solves the problem of small observation range of binocular vision sensors, improves the accuracy and reliability of navigation and positioning, and is suitable for application scenarios of environmental changes.

CN113095107BActive Publication Date: 2024-11-26SHENYANG SIASUN ROBOT & AUTOMATION
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN201911337406.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-12-23
Publication Date
2024-11-26
Estimated Expiration
2039-12-23

AI Technical Summary

Technical Problem

Existing binocular vision sensors have a small observation range, resulting in failure of navigation positioning when the field of view image is obscured or the feature graphics are lacking.

Method used

Using a multi-eye vision system, including the main processor and multiple binocular cameras, image acquisition, preprocessing, geometric correction, feature extraction and local screening are performed through the FPGA hardware processing part, and the ARM processing part of the FPGA core performs gradient direction, descriptor, Feature point matching, depth calculation and data output to construct three-dimensional scenes and obstacle positioning.

Benefits of technology

It improves the field of view of the binocular vision sensor, obtains more positioning information, improves the accuracy and reliability of navigation positioning, and is suitable for application scenarios of environmental changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113095107B_ABST
    Figure CN113095107B_ABST
Patent Text Reader

Abstract

The present invention relates to a positioning device for automatic guided vehicle navigation, and more particularly to a multi-eye vision system and method for AGV navigation. A multi-eye vision system for AGV navigation, characterized in that it includes a main processor and a plurality of binocular cameras connected thereto; the binocular camera is used to extract feature points and send feature point depth information to the main processor; the main processor is used to receive the feature point depth information sent by the binocular camera, construct a three-dimensional scene and locate obstacles. The present invention can improve the field of view of the binocular vision sensor, obtain more positioning information, and improve the accuracy and reliability of navigation positioning. There are often environmental changes in industrial sites. Compared with the positioning of a group of binocular cameras, this device is more suitable for application scenarios with open environments and changing environments.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a positioning device for automatic guided vehicle navigation, and in particular to a multi-eye vision system and method for AGV navigation. Background Art

[0002] At present, natural navigation is divided into laser contour navigation and visual navigation. Single-line laser contour navigation has limited measurement information, and multi-line laser contour navigation is relatively expensive. Compared with laser contour navigation, visual navigation does not require the installation of positioning laser radar and has low subsequent maintenance costs. In terms of ease of use, visual navigation can plan paths on the map arbitrarily, which is convenient for path switching; in terms of safety and stability, visual navigation uses 3D vision, which is rich in information and stable navigation. When avoiding obstacles, it can not only stop and bypass obstacles, but also avoid obstacles at negative scales (detect potholes), which is something that laser radar cannot do.

[0003] However, existing machine vision technologies have the following disadvantages: The visual sensor has a small viewing angle, and a set of binocular lenses is required to measure the depth of feature points. However, a set of binocular lenses can generally only observe a 90-degree field of view. When the field of view image is blocked or there is no feature pattern in the field of view, it often leads to navigation and positioning failure. Summary of the invention

[0004] In order to solve the shortcoming of a small binocular observation range, the present invention proposes a multi-eye vision system and method for AGV navigation.

[0005] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: a multi-eye vision system for AGV navigation, including a main processor and a plurality of binocular cameras connected thereto;

[0006] The binocular camera is used to extract feature points and send feature point depth information to the main processor;

[0007] The main processor is used to receive the feature point depth information sent by the binocular camera, construct a three-dimensional scene and locate obstacles.

[0008] The acquisition frame synchronization signal of the binocular camera triggers other binocular cameras.

[0009] The binocular camera comprises:

[0010] FPGA hardware processing part, used for image acquisition, image preprocessing, geometric correction, multi-size reconstruction, feature extraction and local screening;

[0011] The FPGA core ARM processing part is used for gradient direction, descriptor, feature point matching, depth calculation, data output and image storage.

[0012] The FPGA hardware processing part includes: a CCD image acquisition module, an image preprocessing module, a geometric correction module, a multi-size reorganization module, a feature extraction module and a local screening module;

[0013] The CCD image acquisition module is used to collect images through a binocular camera;

[0014] The image preprocessing module is used to perform image preprocessing on the collected images;

[0015] The geometric correction module is used to perform lens distortion correction on the preprocessed image;

[0016] The multi-size reorganization module is used to perform multi-size transformation on the corrected image;

[0017] The feature extraction module is used for feature point extraction;

[0018] The local screening module is used to screen out feature points.

[0019] The FPGA core ARM processing part includes a gradient direction module, a descriptor module, a feature point matching module, a depth calculation module, a data output module and a DDR3 cache module;

[0020] The gradient direction module is used to encode the selected feature points;

[0021] The descriptor module is used to encode the selected feature points;

[0022] The feature point matching module is used to match the encoding information of the image to obtain a disparity map;

[0023] The depth calculation module is used to synthesize the disparity map into an image;

[0024] The data output module is used to output the depth information containing feature points in the image;

[0025] The DDR3 cache module is used to store images.

[0026] The multi-eye vision method for AGV navigation is characterized by comprising the following steps:

[0027] The binocular camera collects images as the output source of three-dimensional scene information;

[0028] 1) The left image and the right image are processed as follows:

[0029] Perform image preprocessing on the acquired images;

[0030] The processed image is passed to the geometric correction module for lens distortion correction;

[0031] Perform multi-size transformation on the rectified image to obtain images of different sizes;

[0032] All images of different sizes are extracted using the FASTX method, and adjacent points are removed by non-maximum suppression, that is, redundant cross-repeated feature points are removed;

[0033] Calculate the gradient direction and descriptor of the feature points selected after non-maximum suppression;

[0034] 2) Feature matching is performed on the coded information of the left image and the coded information of the right image. By matching the position coordinates of the pixels, the horizontal pixel position between the two points is the disparity, thereby obtaining a disparity map of all feature points;

[0035] According to the principle of triangulation, the depth of the feature point is obtained, and the coordinate information, gradient direction information, descriptor information and depth information of the feature point in an image are output;

[0036] 3) The main processor receives the depth information of feature points uploaded by multiple binocular cameras to construct a three-dimensional scene and locate obstacles.

[0037] The image preprocessing includes processing the collected image using 5*5 Gaussian blur to filter out image noise.

[0038] The collected image is a checkerboard image; and the feature points are corner points.

[0039] The gradient direction is used to sample in a neighborhood window centered on the feature point, and a histogram is used to count the gradient directions of the neighborhood pixels. The peak value of the histogram represents the information of the main direction of the neighborhood gradient of the feature point.

[0040] The descriptor is used to calculate the brightness relationship between the feature point and the surrounding neighborhood pixels, and encode the information of this relationship in a certain order.

[0041] The present invention has the following beneficial effects and advantages:

[0042] The present invention can improve the field of view of binocular vision sensors, obtain more positioning information, and improve the accuracy and reliability of navigation positioning. In industrial sites, there are often environmental changes. Compared with the positioning of a set of binocular cameras, this device is more suitable for application scenarios in open environments and changing environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of the structure of the present invention;

[0044] Figure 2 FPGA binocular camera structure and processing flow of the present invention;

[0045] Figure 3 The ARM main processor structure of the present invention;

[0046] Figure 4 A schematic diagram of the shell structure of the present invention. DETAILED DESCRIPTION

[0047] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0048] like Figure 1 to Figure 4 As shown, a multi-eye vision system for AGV navigation includes a main processor and multiple binocular cameras connected thereto;

[0049] A binocular camera is used to extract feature points and send feature point depth information to the main processor;

[0050] like Figure 3 As shown, the main processor is used to receive the feature point depth information sent by the binocular camera, build a three-dimensional scene and locate obstacles; it includes an internal communication bus, a processor, an interface circuit, and a power supply. The data is sent from the internal communication bus to the processor part, and the processor includes multi-device feature point correction, feature point cloud matching, and positioning processing functions. The processor sends the results to the interface circuit;

[0051] The main processor part is divided into bus communication part, processor part, interface and indication circuit part, and power supply part.

[0052] The communication bus is connected to the data output of multiple groups of FPGA binocular cameras through a network router to receive the depth information of feature points collected by binocular cameras in different directions. In order to avoid bus congestion, multiple groups of binocular cameras use the time-sharing upload function when uploading data.

[0053] The processor is divided into two parts: map reconstruction and navigation. Map reconstruction includes calling timestamp unification, unified coordinate system, front-end visual odometer, back-end nonlinear optimization, loop detection, and map building. Positioning includes loading the environment and calculating positioning data.

[0054] The data obtained from the communication bus enters the unified timestamp processing. This part processes the data received from different devices at different times. Since the data content contains the timestamp information of the data, this processing will integrate the data collected at the same time.

[0055] The unified data after time is processed into a unified coordinate system. The unified coordinate system is used to unify the measurement origins of different binocular lenses, and the measurement origin is set at the center of the sensor.

[0056] The data after the unified coordinate system enters the front-end visual odometry for processing. The task of the visual odometry is to estimate the movement of the camera between adjacent images and the appearance of the local map.

[0057] The data processed by the front-end visual odometry enters the back-end nonlinear optimization processing. The back-end receives the camera posture measured by the visual odometry at different times, as well as the information of loop detection, and optimizes them to obtain a globally consistent trajectory and map.

[0058] The data in the unified coordinate system is also processed by loop detection, and the detection data is sent to the back-end nonlinear optimization. Loop detection determines whether the robot has reached the previous position. If a loop is detected, it will provide the information to the back-end for processing.

[0059] The data processed by the backend is then used for mapping, which creates a map corresponding to the task requirements based on the estimated trajectory.

[0060] The data after the map is built is saved. The saved data will not be changed, and the map will not be changed with each run.

[0061] The loading environment is to load the stored map during use. The AGV has a fixed online point, and only the map information near the online point is loaded after going online.

[0062] Positioning data calculation, based on the detected feature point depth data and the loaded map information, the best matching position is obtained and the positioning data is transmitted through the interface circuit.

[0063] The interfaces in the interface and indication circuit are divided into power input interface, debugging interface and data output interface. The power interface is used to connect to the power supply part, and the data output interface is connected to the outside of the sensor to provide positioning information for the AGV. The indication circuit includes a power indicator light and a status indicator light.

[0064] The power supply part supplies power to the entire sensor and connects the interface to the main processor part.

[0065] The acquisition frame synchronization signal of the binocular camera triggers other binocular cameras.

[0066] The shell is used to fix multiple binocular cameras and the main processor. The shell is divided into two parts: 6 top plate and 7 cover plate. The five faces of the top plate are perpendicular to each other. Each face is equipped with a group of binocular cameras, namely: 1 front binocular camera, 2 right binocular camera, 3 bottom binocular camera, 4 left binocular camera and 5 top binocular camera. The top plate has 8 communication interfaces, and there are two 9 status indicator lights on the top surface for the operator to observe the working status of the sensor.

[0067] like Figure 2As shown in the figure, the FPGA part is a SoC binocular real-time stereo vision system designed based on the Xilinx platform, which mainly invented a high-precision, high-real-time stereo vision system. The system modules mainly include image acquisition module, binocular video image synchronization module, binocular epipolar geometry correction module, DDR3 cache module, and image post-processing module. The development platform of the Xilinx Zynq series is adopted, and the internal chip integrates a dual-core ARM core, which has greatly improved the system integration, cost, power consumption, and debugging convenience. In terms of stereo matching algorithm, through the comparison of matching algorithms, a semi-global stereo matching algorithm with high robustness and real-time performance in real scene applications is selected.

[0068] The FPGA binocular camera data processing flow is divided into the FPGA hardware parallel processing part and the FPGA core pipeline part. Image acquisition, image preprocessing, geometric correction, multi-size reconstruction, feature extraction, and local screening functions are implemented in the FPGA hardware parallel processing part, and the remaining gradient direction, descriptor, corner point matching, depth calculation, data output, DDR3 cache module, and binocular video image synchronization module are implemented in the FPGA core pipeline part.

[0069] The binocular camera includes:

[0070] FPGA hardware processing part, used for image acquisition, image preprocessing, geometric correction, multi-size reconstruction, feature extraction and local screening;

[0071] The FPGA core ARM processing part is used for gradient direction, descriptor, feature point matching, depth calculation, data output and image storage.

[0072] The FPGA hardware processing part includes: CCD image acquisition module, image preprocessing module, geometric correction module, multi-size reconstruction module, feature extraction module and local screening module;

[0073] CCD image acquisition module, used to collect images through a binocular camera;

[0074] Image acquisition: The camera uses a global shutter CMOS sensor and has multi-device synchronization function.

[0075] Image acquisition is an important part of the binocular vision system, as the input source of three-dimensional scene information. Usually, in order to avoid problems such as accuracy in the later matching process, it is necessary to perform simple pre-processing such as denoising, filtering, and conversion on the initial image to enhance the effective information of the image and minimize irrelevant information. FPGA is connected to the image sensor, and the data transmission uses high-speed parallel signals. It is equipped with the same reset and clock chips, which can achieve left and right binocular frame synchronization of the same binocular camera.

[0076] An image preprocessing module, used for performing image preprocessing on the collected images;

[0077] The geometric correction module is used to correct the lens distortion of the preprocessed image. The camera calibration uses an effective image acquisition system to calibrate the relevant parameters inside and outside the camera to determine the correspondence between the pixel points of the image and the points in the spatial coordinate system. In this design, a calibrated dual camera is used. By acquiring a chessboard image, the image is used to calculate the internal and external parameters of the lens outside the device. Then it is configured to the hardware circuit through software.

[0078] A multi-size reorganization module is used to perform multi-size transformation on the rectified image;

[0079] Feature extraction module, used for feature point extraction; common image features include color, texture, shape and spatial relationship, etc. Selecting appropriate image feature extraction plays a vital role in image stereo matching. Common image feature types include edges, regions, corners, etc., which will follow relevant constraint criteria and matching strategies during stereo matching. In the present invention, FASTX corner point extraction is adopted, and non-maximum suppression is used to remove adjacent points.

[0080] Local filtering module, used to filter out feature points.

[0081] The FPGA core ARM processing part includes a gradient direction module, a descriptor module, a feature point matching module, a depth calculation module, a data output module and a DDR3 cache module;

[0082] Gradient direction module, used to encode the selected feature points;

[0083] The description submodule is used to encode the selected feature points;

[0084] The feature point matching module is used to match the encoding information of the image to obtain a disparity map;

[0085] A depth calculation module, used to synthesize the disparity map into an image;

[0086] A data output module is used to output the depth information of feature points in the image;

[0087] DDR3 cache module for storing images.

[0088] like Figure 1-2 As shown, the multi-eye vision method for AGV navigation is characterized by comprising the following steps:

[0089] The binocular camera collects images as the output source of three-dimensional scene information;

[0090] 1) The left image and the right image are processed as follows:

[0091] Perform image preprocessing on the acquired images;

[0092] The processed image is passed to the geometric correction module for lens distortion correction;

[0093] Perform multi-size transformation on the rectified image to obtain images of different sizes;

[0094] All images of different sizes are extracted using the FASTX method, and adjacent points are removed by non-maximum suppression, that is, redundant cross-repeated feature points are removed;

[0095] Calculate the gradient direction and descriptor of the feature points selected after non-maximum suppression;

[0096] 2) Feature matching is performed on the coded information of the left image and the coded information of the right image. By matching the position coordinates of the pixels, the horizontal pixel position between the two points is the disparity, thereby obtaining a disparity map of all feature points;

[0097] According to the principle of triangulation, the depth of the feature point is obtained, and the coordinate information, gradient direction information, descriptor information and depth information of the feature point in an image are output;

[0098] The purpose of matching is to obtain the depth of feature points. The descriptors are basically the same, and the depth values ​​are the same for the left and right eyes. Therefore, the output depth, descriptor and gradient direction information are all output based on one image data.

[0099] 3) The main processor receives the depth information of feature points uploaded by multiple binocular cameras to construct a three-dimensional scene and locate obstacles.

[0100] Image preprocessing includes using 5*5 Gaussian blur to process the collected images and filter out image noise.

[0101] The collected image is a checkerboard image; the feature points are corner points.

[0102] Gradient direction is used to sample in the neighborhood window centered on the feature point, and use a histogram to count the gradient directions of the neighborhood pixels. The peak of the histogram represents the main direction of the gradient of the neighborhood of the feature point.

[0103] The descriptor is used to calculate the brightness relationship between the feature point and the surrounding neighborhood pixels, and encode this relationship in a certain order.

Claims

1. A multi-eye vision method for AGV navigation, based on a multi-eye vision system for AGV navigation, including a main processor and multiple binocular cameras connected thereto; The binocular camera is used to extract feature points and send feature point depth information to the main processor; The binocular camera comprises: FPGA hardware processing part, used for image acquisition, image preprocessing, geometric correction, multi-size reconstruction, feature extraction and local screening; FPGA core ARM processing part, used for gradient direction, descriptor, feature point matching, depth calculation, data output and image storage; The FPGA core ARM processing part includes a gradient direction module, a descriptor module, a feature point matching module, a depth calculation module, a data output module and a DDR3 cache module; The gradient direction module is used to encode the selected feature points; The descriptor module is used to encode the selected feature points; The feature point matching module is used to match the encoding information of the image to obtain a disparity map; The depth calculation module is used to synthesize the disparity map into an image; The data output module is used to output the depth information containing feature points in the image; The DDR3 cache module is used to store images; The main processor is used to receive the feature point depth information sent by the binocular camera, construct a three-dimensional scene and locate obstacles; The main processor part is divided into bus communication part, processor part, interface and indication circuit part, and power supply part; The communication bus is connected to the data output of multiple groups of FPGA binocular cameras through a network router, and is used to receive the depth information of feature points collected by binocular cameras in different directions. In order to avoid bus congestion, multiple groups of binocular cameras use a time-sharing upload function when uploading data. It is characterized by comprising the following steps: the binocular camera collects images as an output source of three-dimensional scene information; 1) The left image and the right image are processed as follows: Perform image preprocessing on the acquired images; The processed image is passed to the geometric correction module for lens distortion correction; Perform multi-size transformation on the rectified image to obtain images of different sizes; All images of different sizes are extracted using the FASTX method, and adjacent points are removed by non-maximum suppression, that is, redundant cross-repeated feature points are removed; Calculate the gradient direction and descriptor of the feature points selected after non-maximum suppression; 2) Feature matching is performed on the coded information of the left image and the coded information of the right image. By matching the position coordinates of the pixels, the horizontal pixel position between the two points is the disparity, thereby obtaining a disparity map of all feature points; According to the principle of triangulation, the depth of the feature point is obtained, and the coordinate information, gradient direction information, descriptor information and depth information of the feature point in an image are output; The gradient direction is used to sample in a neighborhood window centered on the feature point, and a histogram is used to count the gradient directions of the neighborhood pixels. The peak value of the histogram represents the information of the main direction of the neighborhood gradient of the feature point. The descriptor is used to calculate the brightness relationship between the feature point and the surrounding neighborhood pixels, and encode the information of this relationship in a certain order; 3) The main processor receives the depth information of feature points uploaded by multiple binocular cameras to construct a three-dimensional scene and locate obstacles; The processor is divided into two parts: map reconstruction and navigation. Map reconstruction includes the simultaneous call of timestamp unification, unified coordinate system, front-end visual odometer, back-end nonlinear optimization, loop detection, mapping, and positioning, including loading environment and positioning data calculation; The data obtained from the communication bus enters the unified timestamp processing, and the data received from different devices at different times are uniformly processed through the timestamp. Since the data content contains the timestamp information of the data, the unified timestamp processing method integrates the data collected at the same time; The time-unified data enters the unified coordinate system for processing. The unified coordinate system is used to unify the measurement origins of different binocular lenses. The measurement origin is set at the center of the sensor. The data after the unified coordinate system is processed by the front-end visual odometry. The task of the visual odometry is to estimate the movement of the camera between adjacent images and the appearance of the local map. The data processed by the front-end visual odometer enters the back-end nonlinear optimization process. The back-end receives the camera posture measured by the visual odometer at different times and the information of loop detection, optimizes them, and obtains a globally consistent trajectory and map. The data in the unified coordinate system is also processed by loop detection, and the detection data is transmitted to the back-end nonlinear optimization; loop detection determines whether the robot has reached the previous position. If a loop is detected, it will provide the information to the back-end for processing; The data processed by the backend is processed for mapping, which builds a map corresponding to the task requirements based on the estimated trajectory; The data after the map is built is saved, and the saved data will not be changed, and the map will not be changed due to each run; Loading environment means loading the stored map during use. AGV has a fixed online point, and only the map information near the online point is loaded after going online. Positioning data calculation, based on the detected feature point depth data and the loaded map information, the best matching position is obtained and the positioning data is transmitted through the interface circuit.

2. A multi-eye vision method for AGV navigation according to claim 1, characterized in that: The image preprocessing includes processing the collected image using 5*5 Gaussian blur to filter out image noise.

3. The multi-eye vision method for AGV navigation according to claim 1, characterized in that: The collected image is a checkerboard image.

4. The multi-eye vision method for AGV navigation according to claim 1, characterized in that: The feature points are corner points.

Citation Information

Patent Citations

  • Binocular vision navigation system and method based on power robot

    CN103413313A

  • Camera tracking method and device

    CN104915965A

  • Binocular camera-based high-precision visual sense positioning map generation system and method

    CN105674993A

  • Assistant robot based on binocular vision, and control method of assistant robot

    CN106610666A

  • Binocular-stereoscopic-vision-based workpiece detection and three-dimensional measurement system and detection method

    CN106969706A