A deepstream-based intelligent forklift obstacle recognition and avoidance method

Abstract

The present invention discloses a method for intelligent forklift recognition and obstacle avoidance based on deepstream, which belongs to the field of warehouse sorting technology. The method of the present invention uses machine vision technology to achieve obstacle avoidance, considers using binocular vision technology, performs dense point cloud three-dimensional modeling of the scene on the forklift's forward path, and obtains the depth information of the scene in three-dimensional space. According to the scene depth information, the distance between objects such as obstacles in the scene and the vehicle that needs to avoid obstacles can be obtained. Thus, obstacle avoidance is achieved.

Description

Technical Field

[0001] The invention belongs to the technical field of warehousing and sorting, and in particular relates to an intelligent forklift obstacle recognition and avoidance method based on deepstream. Background Art

[0002] At present, the application of intelligent forklifts in the automated warehousing industry has been very extensive, but for intelligent forklifts, the most common method is to use lasers to identify obstacles to avoid obstacles. Because lasers are single-point, the recognition range is very narrow during recognition, and it cannot accurately identify all obstacles in front of or behind the intelligent forklift. And the recognition of obstacles by laser is through the reflection of laser to the receiver, but obstacles are varied and some have uneven surfaces, so this provides a lot of uncertainty in the recognition of obstacles and increases the error rate of recognition. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a deepstream-based intelligent forklift recognition and obstacle avoidance method, so that the intelligent forklift is no longer limited by the traditional laser obstacle avoidance, and can more comprehensively identify all obstacles in front of or behind the forklift and avoid them, thereby reducing the error rate of recognition.

[0004] In order to achieve the above object, the present invention provides the following technical solutions:

[0005] The present invention provides a method for intelligent forklift obstacle recognition and avoidance based on deepstream, comprising the following steps:

[0006] S1, uses mixed Gaussian model background modeling combined with texture detection or deep learning to detect pallets and goods;

[0007] S2, set the forklift, shelf and pallet as the background, calculate the centroid of the detected foreground goods according to their shape, and further calculate their center of mass;

[0008] S3, considers the forklift and the shelf as the background, and determines whether the detected foreground pallet is placed properly based on the relative position of the pallet and the center of mass of the object, and the relative position of the pallet and the markers on the shelf;

[0009] S4, calibrates and dedistorts the camera, detects the defect location using the target recognition neural network, searches for specific markers, and projects the shelf onto the same vertical plane using projection transformation according to the location of the markers, obtains the area of ​​interest, grayscales it, linearly interpolates and scales it to a uniform size, and comprehensively calculates the similarity between the structure comparison part of the SSIM and the NCC matching between the current shelf image and the standard shelf image;

[0010] S5 uses machine vision technology to achieve obstacle avoidance, performs dense point cloud 3D modeling of the scene on the forklift's path, and obtains the depth information of the scene in 3D space. Based on the scene depth information, the distance between objects such as obstacles in the scene and the vehicle that needs to be avoided can be obtained, thereby achieving obstacle avoidance.

[0011] Furthermore, two cameras are selected to use binocular vision technology to perform three-dimensional modeling of the goods and pallets; first, the image is subjected to monocular correction, including correction of intrinsic parameters, extrinsic parameters and distortion, and then binocular correction is performed to make the optical axes between the left and right cameras parallel; feature points of the scene image are matched to obtain a sparse point cloud map; the scene disparity map is obtained through the sparse point cloud map, and thus a dense point cloud map is obtained, and the depth information of the scene is obtained through the dense point cloud map, and the position of the goods and pallets can be located according to the depth information.

[0012] Further, the target positioning algorithm is as follows:

[0013] After obtaining the coordinates of a large number of feature points, the position and shape of the target can be determined. According to the similar triangle formula, the depth of the target can be simply deduced to be calculated by the following formula:

[0014]

[0015] Where b is the baseline length, f is the distance from the optical center of the camera to the imaging plane, b and f are both fixed values, d w After obtaining the depth of the object from the parallax, according to the imaging principle of the camera, the target can be located as follows:

[0016]

[0017] Among them, S is the scale factor, F is the camera intrinsic matrix, [RT] is the camera extrinsic matrix, R and T are the rotation and translation components of the camera coordinate system relative to the world coordinate system, (u, v) is the pixel coordinate of the object, (x w ,y w ,z w ) is the world coordinate of the object.

[0018] Furthermore, SSIM is the structural similarity between two images. In actual engineering, the calculation formula selected is as follows:

[0019]

[0020] Among them, μ x and μ y are the average values ​​of the first and second images, σ x and σ y are the standard deviations of the first and second images, respectively, c 1and c 2 is a constant to avoid the denominator being 0.

[0021] SSIM measures the brightness, contrast and structure comparison between two images. The brightness and contrast have little effect on the deformation of the shelf, while the structure directly measures the degree of deformation of the goods. The structure comparison part is used for calculation. The calculation formula is as follows:

[0022]

[0023] Among them, σ x and σ y are the standard deviations of the first and second images respectively, c is a constant to avoid the denominator being 0. NCC matching can effectively reduce the impact of illumination on image comparison results. It is suitable for industrial detection and is generally used to search for targets on images. For two images of the same size, the following formula can be used directly to calculate the similarity:

[0024]

[0025] Among them, μ f and μ r are the pixel means of the two images, f and r are the two images to be compared, m and n are the length and width of the images, respectively.

[0026] The beneficial effects of the present invention are:

[0027] The invention discloses a method for identifying and avoiding obstacles for intelligent forklifts based on deepstream, which solves the technical problem that the single-point detection brought by the traditional use of laser cannot identify all objects in front, so that the intelligent forklift is no longer limited by the traditional laser obstacle avoidance, and can more comprehensively identify and avoid all obstacles in front of or behind the forklift.

[0028] Other advantages, objectives and features of the present invention will be described in the following description and will be apparent to those skilled in the art to some extent, or those skilled in the art may be taught from the practice of the present invention. The objectives and other advantages of the present invention may be realized and obtained through the following description. DETAILED DESCRIPTION

[0029] The present invention provides a method for intelligent forklift obstacle recognition and avoidance based on deepstream, comprising the following steps:

[0030] First, check whether the goods are placed correctly.

[0031] S1, uses mixed Gaussian model background modeling combined with texture detection or deep learning to detect pallets and goods;

[0032] S2, set the forklift, shelf and pallet as the background, calculate the centroid of the detected foreground goods according to their shape, and further calculate their center of mass;

[0033] S3, considers the forklift and the shelf as the background, and determines whether the detected foreground pallet is placed properly based on the relative position of the pallet and the center of mass of the object, and the relative position of the pallet and the markers on the shelf;

[0034] Then use shelf detection,

[0035] S4, calibrates and dedistorts the camera, detects the defect location using the target recognition neural network, searches for specific markers, and projects the shelf onto the same vertical plane using projection transformation according to the location of the markers, obtains the area of ​​interest, grayscales it, linearly interpolates and scales it to a uniform size, and comprehensively calculates the similarity between the structure comparison part of the SSIM and the NCC matching between the current shelf image and the standard shelf image;

[0036] Forklift moves to avoid obstacles,

[0037] S5 uses machine vision technology to achieve obstacle avoidance, performs dense point cloud 3D modeling of the scene on the forklift's path, and obtains the depth information of the scene in 3D space. Based on the scene depth information, the distance between objects such as obstacles in the scene and the vehicle that needs to be avoided can be obtained, thereby achieving obstacle avoidance.

[0038] In this embodiment, two cameras are selected to perform three-dimensional modeling of goods and pallets using binocular vision technology. First, a monocular correction is performed on the image, including correction of intrinsic parameters, extrinsic parameters and distortion, and then a binocular correction is performed to make the optical axes of the left and right cameras parallel. Feature points of the scene image are matched to obtain a sparse point cloud map. A scene disparity map is obtained through the sparse point cloud map, thereby obtaining a dense point cloud map. Through the dense point cloud map, the depth information of the scene is obtained, and the position of the goods and pallets can be located according to the depth information.

[0039] In this embodiment, the target positioning algorithm is as follows:

[0040] After obtaining the coordinates of a large number of feature points, the position and shape of the target can be determined. According to the similar triangle formula, the depth of the target can be simply deduced to be calculated by the following formula:

[0041]

[0042] Where b is the baseline length, f is the distance from the optical center of the camera to the imaging plane, b and f are both fixed values, d w After obtaining the depth of the object from the parallax, according to the imaging principle of the camera, the target can be located as follows:

[0043]

[0044] Among them, S is the scale factor, F is the camera intrinsic matrix, [RT] is the camera extrinsic matrix, R and T are the rotation and translation components of the camera coordinate system relative to the world coordinate system, (u, v) is the pixel coordinate of the object, (x w ,y w ,z w ) is the world coordinate of the object.

[0045] In this embodiment, SSIM is the structural similarity between two images. In actual engineering, the calculation formula selected is as follows:

[0046]

[0047] Among them, μ x and μ y are the average values ​​of the first and second images, σ x and σ y are the standard deviations of the first and second images, respectively, c 1 and c 2 is a constant to avoid the denominator being 0.

[0048] SSIM measures the brightness, contrast and structure comparison between two images. The brightness and contrast have little effect on the deformation of the shelf, while the structure directly measures the degree of deformation of the goods. The structure comparison part is used for calculation. The calculation formula is as follows:

[0049]

[0050] Among them, σ x and σ y are the standard deviations of the first and second images respectively, c is a constant to avoid the denominator being 0. NCC matching can effectively reduce the impact of illumination on image comparison results. It is suitable for industrial detection and is generally used to search for targets on images. For two images of the same size, the following formula can be used directly to calculate the similarity:

[0051]

[0052] Among them, μ f and μ r are the pixel means of the two images, f and r are the two images to be compared, m and n are the length and width of the images, respectively.

[0053] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. A method for intelligent forklift obstacle recognition and avoidance based on deepstream, characterized in that: The following steps are involved: S1, uses mixed Gaussian model background modeling combined with texture detection or deep learning to detect pallets and goods; S2, set the forklift, shelf and pallet as the background, calculate the centroid of the detected foreground goods according to their shape, and further calculate their center of mass; S3, considers the forklift and the shelf as the background, and determines whether the detected foreground pallet is placed properly based on the relative position of the pallet and the center of mass of the object, and the relative position of the pallet and the markers on the shelf; S4, calibrates and dedistorts the camera, detects the defect location using the target recognition neural network, searches for specific markers, and projects the shelf onto the same vertical plane using projection transformation according to the location of the markers, obtains the area of ​​interest, grayscales it, linearly interpolates and scales it to a uniform size, and comprehensively calculates the similarity between the structure comparison part of the SSIM and the NCC matching between the current shelf image and the standard shelf image; S5 uses machine vision technology to achieve obstacle avoidance, performs dense point cloud 3D modeling of the scene on the forklift's path, and obtains the depth information of the scene in 3D space. Based on the scene depth information, the distance between the obstacle in the scene and the vehicle that needs to be avoided can be obtained, thereby achieving obstacle avoidance.

2. According to the deepstream-based intelligent forklift obstacle avoidance method of claim 1, it is characterized in that: Two cameras are selected to use binocular vision technology to build three-dimensional models of goods and pallets. First, the image is subjected to monocular correction, including correction of intrinsic parameters, extrinsic parameters and distortion, and then binocular correction is performed to make the optical axes of the left and right cameras parallel. Feature points of the scene image are matched to obtain a sparse point cloud map. The scene disparity map is obtained from the sparse point cloud map, and then a dense point cloud map is obtained. The depth information of the scene is obtained from the dense point cloud map, and the position of the goods and pallets can be located based on the depth information.

3. The deepstream-based intelligent forklift obstacle avoidance method according to claim 2 is characterized in that: The target positioning algorithm is as follows: After obtaining the coordinates of a large number of feature points, the position and shape of the target can be determined. According to the similar triangle formula, the depth of the target can be simply deduced to be calculated by the following formula: Where b is the baseline length, f is the distance from the optical center of the camera to the imaging plane, b and f are both fixed values, d w After obtaining the depth of the object from the parallax, according to the imaging principle of the camera, the target can be located as follows: Among them, S is the scale factor, F is the camera intrinsic matrix, [RT] is the camera extrinsic matrix, R and T are the rotation and translation components of the camera coordinate system relative to the world coordinate system, (u, v) is the pixel coordinate of the object, (x w ,y w ,z w ) is the world coordinate of the object.

4. According to claim 3, the intelligent forklift obstacle recognition and avoidance method based on deepstream is characterized in that: SSIM is the structural similarity between two images. In actual engineering, the calculation formula selected is as follows: Among them, μ x and μ y are the average values ​​of the first and second images, σ x and σ y are the standard deviations of the first and second images respectively. c1 and c2 are constants to avoid the denominator being 0. SSIM measures the brightness, contrast and structure comparison between two images. The brightness and contrast have little effect on the deformation of the shelf, while the structure directly measures the degree of deformation of the goods. The structure comparison part is used for calculation. The calculation formula is as follows: Among them, σ x and σ y are the standard deviations of the first and second images respectively, c is a constant to avoid the denominator being 0. NCC matching can effectively reduce the impact of illumination on image comparison results. It is suitable for industrial inspection and is used to search for targets on images. For two images of the same size, the following formula can be used directly to calculate the similarity: Among them, μ f and μ r are the pixel means of the two images, f and r are the two images to be compared, m and n are the length and width of the images, respectively.

Images