Vision-based path center line detection method
Through the vision-based path midline detection method, combined with threshold segmentation, morphological filtering and depth map information, the misjudgment problem of road center line detection in complex environments is solved, and the stable driving of the robot on the road center line is achieved.
Patent Information
- Application Number
- CN202411749335.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-02
- Publication Date
- 2025-03-21
AI Technical Summary
The prior art is difficult to accurately detect the road center line in complex environments, especially in mixed environments of trees and grasslands. Traditional methods are prone to misjudgment due to changes in light and complexity of tree morphology.
The vision-based path midline detection method is used to collect images through the camera, and the mask of the fruit tree is calculated using threshold segmentation and morphological filtering. The coordinate transformation and grid division are combined with the depth map. The three-dimensional coordinates of the center point are calculated by grid, the point cloud information of the obstacle is obtained, and the obstacle situation is judged based on the point cloud information, and the heading of the obstacles is calculated.
It realizes real-time detection of obstacles on both sides of the road in complex environments, and calculates the heading angle for the robot to avoid obstacles or achieves the minimum collision in extremely narrow areas, ensuring that the robot always drives on the center line of the road, overcoming the influence of irregular environmental characteristics.
Smart Images
Figure CN119672674A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of path detection, and in particular to a vision-based path centerline detection method. Background Art
[0002] At present, the sensors widely used in obstacle detection are mainly laser radar and camera, which can extract the shape features of obstacles. Among them, laser radar obtains stable signals and high depth value accuracy, but it is large in size and high in cost. Camera sensors can not only obtain geometric feature information of the target, but also judge semantic information based on color. They are low in cost, but cannot directly obtain depth information. The depth camera Realsense D455 can not only capture color information, but also directly obtain depth information of corresponding points. It is widely used in various industrial scenarios. In addition to sensors, environmental characteristics also affect the application of this technology. For example, in an agricultural production environment, there is a height difference between tree branches and grass, and the colors of trees and grass are similar, which can easily cause misjudgment due to changes in light. In addition, the shapes of trees are different, and the shape of tree branches changes irregularly.
[0003] At present, the detection of the centerline of the road is mainly divided into traditional image strategies and deep learning. The traditional image strategies include edge detection based on Hough line transform and contour detection, and vanishing point detection based on clustering. In this scenario, due to the complex structure of the environment and the uneven edge points, the edge detection class will produce large errors whether it is fitted or regressed. Therefore, it is impossible to filter the edges with the help of texture information. It is also difficult for the vanishing point detection class to fit parallel straight lines from the uneven tree tops on both sides. Deep learning is divided into three types: target recognition, target detection, and target segmentation. Target recognition is good at knowing what the image is, and does not meet the requirements of the scene area screening. Target detection can know what is in the image and where it is, but due to the limitation of the confidence box, the obtained target is easy to contain areas of other categories. Target segmentation can extract pixels belonging to the same category in the image, which is more suitable for the environment with irregular shapes and few target types, but it requires a huge amount of data support and is not suitable for ordinary single scenes.
[0004] When a robot using sensor technology moves, it needs to constantly calculate its posture and compare it with the established route to prevent deviation. This requires the use of sensors such as odometers and gyroscopes for calculation, which is not only computationally intensive but also too cumbersome. Therefore, a new and efficient method is urgently needed to enable robots to automatically detect obstacles in short distances and keep driving on the center line of the road. Summary of the invention
[0005] The purpose of the present invention is to provide a vision-based path centerline detection method, aiming to solve the problem of environmental characteristics affecting the detection of the road centerline when a robot is walking.
[0006] To achieve the above object, the present invention provides a path centerline detection method based on vision, comprising the following steps:
[0007] The camera collects images, uses threshold segmentation and morphological filtering to calculate the mask of the fruit tree, and uses the connected domain to calculate the fruit tree area;
[0008] Align the color image and the depth image, perform coordinate transformation, divide the grid, and then calculate the three-dimensional coordinates of the center point in the camera coordinate system grid by grid based on the fruit tree area and the aligned depth image to obtain obstacle point cloud information;
[0009] Based on the obstacle point cloud information layout and obstacle situation judgment, a heading to avoid the obstacle is calculated.
[0010] The camera collects images, calculates the grayscale value variation range of the fruit tree area, calculates the mask of the fruit tree using threshold segmentation and morphological filtering, and uses the connected domain to calculate the specific method of the fruit tree area:
[0011] Collect images through the camera and calculate the grayscale value variation range of the fruit tree area;
[0012] The mask of the fruit tree is calculated using threshold segmentation and morphological filtering, and the fruit tree area is calculated using the connected domain.
[0013] The grid size is 5x5.
[0014] The specific method of calculating the three-dimensional coordinates of the center point in the camera coordinate system grid by grid is as follows:
[0015] The area 2000 mm in front of the robot, 1200 mm to the left and right, 1200 mm above, and 200 mm below is set as the collision area, and the coordinate points belonging to the area in the frame are extracted.
[0016] Among them, the obstacle situations include obstacles on both sides, obstacles on one side and no obstacles.
[0017] Among them, the specific method of arranging and judging the obstacle situation based on the obstacle point cloud information is:
[0018] Search along the positive direction from the starting point of the camera coordinate Z axis, with a search gradient of 20mm;
[0019] Calculate the closest points on both sides of the 20mm interval to the central axis and determine the obstacle situation.
[0020] The calculation of the heading for avoiding obstacles is performed based on the principle of minimum collision cost.
[0021] The invention discloses a path centerline detection method based on vision. The camera collects images, and the mask of the fruit tree is calculated by using threshold segmentation and morphological filtering, and the fruit tree area is settled by using the connected domain; the color image and the depth image are aligned, coordinate transformation is performed, and grids are divided; then, based on the fruit tree area and the aligned depth image, the three-dimensional coordinates of the center point in the camera coordinate system are calculated grid by grid to obtain obstacle point cloud information; based on the obstacle point cloud information, the obstacle situation is arranged and judged, and the heading for avoiding the obstacle is calculated. The method integrates traditional image processing strategies and point cloud information processing technologies, and reasonably uses the color information and depth information in the scene, and separates the fruit trees on both sides of the road, so that the robot can detect the obstacles on both sides of the road in real time during driving, and calculates the heading angle for the robot to avoid obstacles or achieve minimum collision in an extremely narrow area, and can return to the center line of the main road after leaving the obstacle area, so as to ensure that the robot can avoid obstacles and always drive on the relative center line of the road, realize the detection of the center line of the road, overcome the influence of irregular environmental characteristics, and solve the problem of environmental characteristics on the detection of the center line of the road when the robot walks. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0023] Figure 1 It is a flow chart of a vision-based path centerline detection method provided by the present invention.
[0024] Figure 2 This is a schematic diagram of the layout of the four types of obstacles.
[0025] Figure 3 It is a flowchart diagram of a vision-based path centerline detection method provided by the present invention.
[0026] Figure 4 It is a flowchart of the specific method of collecting images with a camera, calculating the mask of the fruit tree using threshold segmentation and morphological filtering, and calculating the fruit tree area using the connected domain.
[0027] Figure 5 It is a connection diagram of a vision-based path centerline detection system provided by the present invention.
[0028] In the figure: 1-image color information processing module, 2-point cloud processing module, 3-classification calculation module. DETAILED DESCRIPTION
[0029] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.
[0030] See also Figures 1 to 4 In a first aspect, the present invention provides a path centerline detection method based on vision, comprising the following steps:
[0031] The S1 camera collects images, calculates the mask of the fruit tree using threshold segmentation and morphological filtering, and calculates the fruit tree area using the connected domain;
[0032] Specific method:
[0033] S11 collects images through a camera and calculates the grayscale value variation range of the fruit tree area;
[0034] S12 uses threshold segmentation and morphological filtering to calculate the mask of the fruit tree, and uses the connected domain to calculate the fruit tree area.
[0035] In the embodiment of the present invention, grayscale image threshold segmentation: according to the pixel difference between the branch and the environment after grayscale, a rough screening range is set to obtain a preliminary branch segmentation effect, and connected domain segmentation: because the image obtained by grayscale image threshold segmentation contains many pixels of other categories, morphological filtering is performed, and according to the branch distribution characteristics, the connected domain on the way is calculated, and the area with too small area is eliminated to obtain a more accurate branch area.
[0036] S2 aligns the color image and the depth image, performs coordinate transformation, divides the grid, and then calculates the three-dimensional coordinates of the center point in the camera coordinate system grid by grid based on the fruit tree area and the aligned depth image to obtain obstacle point cloud information;
[0037] In an embodiment of the present invention, the color image and the depth image obtained by Realsene are aligned, transformed into the same coordinate system to remove distortion, and divided into 5x5 grids. Then, according to the segmented fruit tree area and the aligned depth map, the three-dimensional coordinates of the center point in the camera coordinate system are calculated grid by grid. According to the segmented branch image and the aligned depth map, the three-dimensional coordinates of the center point in the camera coordinate system are calculated grid by grid. The area 2000mm in front of the robot, 1200mm to the left and right, 1200mm above, and 200mm below is set as the collision area, and the coordinate points belonging to the area in the frame are extracted for subsequent classification calculations.
[0038] S3 arranges and determines the obstacle situation based on the obstacle point cloud information, and calculates a heading to avoid the obstacle.
[0039] In an embodiment of the present invention, a search is performed along the positive direction from the starting point of the camera coordinate Z axis, with 20 mm as the search gradient, and the nearest points on both sides of the 20 mm interval from the central axis are calculated. Then, the judgment logic involved is divided into three situations: obstacles on both sides, obstacles on one side, and no obstacles. After the nearest obstacles on both sides are obtained, the direction of travel is calculated according to the principle of minimum collision cost, so as to obtain the three-dimensional coordinates of the driving point and the heading angle that should be corrected from the current position to the coordinates.
[0040] This method combines traditional image processing strategies and point cloud information processing technology, and makes reasonable use of the color information and depth information in the scene, separating the fruit trees on both sides of the road. This allows the robot to detect obstacles on both sides of the road in real time during driving, and calculates the heading angle for the robot to avoid obstacles or achieve minimum collision in extremely narrow areas. After leaving the obstacle area, it can return to the center line of the main road, thereby ensuring that the robot can avoid obstacles and always drive on the relative center line of the road, realizing the detection of the center line of the road, overcoming the influence of irregular environmental characteristics, and solving the problem of environmental characteristics on the detection of the center line of the road when the robot is walking.
[0041] See also Figure 5 In a second aspect, the present invention further provides a vision-based path centerline detection system, which is applied to the vision-based path centerline detection method as described in the first aspect, and includes an image color information processing module 1, a point cloud processing module 2 and a classification calculation module 3, wherein the image color information processing module 1, the point cloud processing module 2 and the classification calculation module 3 are connected in sequence;
[0042] The image color information processing module 1 is used to calculate the grayscale value variation range of the fruit tree area after the camera acquires the image, and then calculate the mask of the fruit tree by using threshold segmentation and morphological filtering, and finally calculate the complete fruit tree area by using the connected domain;
[0043] The point cloud processing module 2 is used to align the color image and the depth image obtained by Realsene, transform them into the same coordinate system to remove distortion, and divide them into 5x5 grids. Then, according to the segmented fruit tree area and the aligned depth map, the three-dimensional coordinates of the center point in the camera coordinate system are calculated grid by grid.
[0044] The classification calculation module 3 is used to classify the layout of obstacles into four types, and can use geometric methods to calculate the heading angle for avoiding obstacles according to different layouts, so that the robot always drives on the relative center line of the road.
[0045] In an embodiment of the present invention, the image color information processing module 1 is used to calculate the grayscale value variation range of the fruit tree area after the camera acquires the image, and then use threshold segmentation and morphological filtering to calculate the mask of the fruit tree, and finally use the connected domain to settle the complete fruit tree area; the point cloud processing module 2 is used to align the color image and depth image obtained by Realsene, remove the distortion after transforming them to the same coordinate system, and divide them into 5x5 grids, and then calculate the three-dimensional coordinates of the center point in the camera coordinate system grid by grid according to the segmented fruit tree area and the aligned depth map; the classification calculation module 3 is used to divide the layout of the obstacle into 4 types, where the blue is the position of the robot, the vertical axis is the center line of the field of view, and the green square is the relative position of the obstacle, and the heading angle for avoiding obstacles can be calculated according to different layouts using geometric methods, so that the robot always drives on the relative center line of the road.
[0046] What is disclosed above is only a preferred embodiment of a vision-based path centerline detection method of the present invention. Of course, this cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiments and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.
Claims
1. A vision-based path centerline detection method, characterized in that: The following steps are involved: The camera collects images, uses threshold segmentation and morphological filtering to calculate the mask of the fruit tree, and uses the connected domain to calculate the fruit tree area; Align the color image and the depth image, perform coordinate transformation, divide the grid, and then calculate the three-dimensional coordinates of the center point in the camera coordinate system grid by grid based on the fruit tree area and the aligned depth image to obtain obstacle point cloud information; Based on the obstacle point cloud information layout and obstacle situation judgment, a heading to avoid the obstacle is calculated.
2. The vision-based path centerline detection method according to claim 1, characterized in that: The camera collects images, calculates the grayscale value variation range of the fruit tree area, calculates the mask of the fruit tree using threshold segmentation and morphological filtering, and calculates the fruit tree area using the connected domain: Collect images through the camera and calculate the grayscale value variation range of the fruit tree area; The mask of the fruit tree is calculated using threshold segmentation and morphological filtering, and the fruit tree area is calculated using the connected domain.
3. The vision-based path centerline detection method according to claim 1, characterized in that: The grid size is 5x5.
4. The vision-based path centerline detection method according to claim 1, characterized in that: The specific method of calculating the three-dimensional coordinates of the center point in the camera coordinate system grid by grid is: The area 2000 mm in front of the robot, 1200 mm to the left and right, 1200 mm above, and 200 mm below is set as the collision area, and the coordinate points belonging to the area in the frame are extracted.
5. The method for detecting a path centerline based on vision according to claim 1, characterized in that: The obstacle conditions include obstacles on both sides, obstacles on one side, and no obstacles.
6. The method for detecting a path centerline based on vision according to claim 1, characterized in that: The specific method of arranging and judging the obstacle situation based on the obstacle point cloud information is as follows: Search along the positive direction from the starting point of the camera coordinate Z axis, with a search gradient of 20mm; Calculate the closest points on both sides of the 20mm interval to the central axis and determine the obstacle situation.
7. The vision-based path centerline detection method according to claim 1, characterized in that: The calculation of the obstacle avoidance heading is performed based on the minimum collision cost principle.