Fruit localization method and device based on binocular vision system

Abstract

The present invention provides a method and device for locating fruits based on a binocular vision system, the method comprising: taking a pre-acquired initial visible light image as a reference, based on a local peak pre-selection strategy and a pulse coupled neural network, registering a pre-acquired initial ToF image and an initial visible light image to obtain a registered ToF image; performing fruit recognition on the fruits in the initial visible light image according to a preset YOLO v3 convolutional neural network model to obtain a fruit recognition result; if the fruit recognition result indicates that the fruit exists in the initial visible light image, determining the position information of the fruit according to the registered ToF image. This scheme can improve the accuracy of fruit recognition and positioning.

Description

Technical Field

[0001] The invention relates to the technical field of agricultural automation, and in particular to a fruit positioning method and device based on a binocular vision system. Background Art

[0002] Automatic fruit picking is an essential function of modern agricultural automation equipment. In orchard operations, the accurate recognition of fruits by the visual system of agricultural harvesting robots is the key to the successful implementation of automatic picking. In the complex orchard environment, the existing technology is not accurate enough in the recognition and positioning of fruits, which will lead to the inability of equipment to accurately pick fruits. Summary of the invention

[0003] Based on this, the object of the present invention is to provide a fruit positioning method and device based on a binocular vision system to improve the accuracy of fruit recognition and positioning.

[0004] In a first aspect, the present invention provides a fruit positioning method based on a binocular vision system, the method comprising: taking a pre-acquired initial visible light image as a reference, based on a local peak preselection strategy and a pulse coupled neural network, registering the pre-acquired initial ToF image and the initial visible light image to obtain a registered ToF image; performing fruit recognition on the fruit in the initial visible light image according to a preset YOLO v3 convolutional neural network model to obtain a fruit recognition result; if the fruit recognition result indicates that there is fruit in the initial visible light image, determining the position information of the fruit according to the registered ToF image.

[0005] In an optional embodiment, the initial visible light image and the initial ToF image are both referred to as initial images; taking the pre-acquired initial visible light image as a reference, based on the local peak preselection strategy and the pulse coupled neural network, the pre-acquired initial ToF image and the initial visible light image are aligned to obtain the aligned ToF image, comprising: determining the target salient area of ​​the target in the initial image according to the local peak preselection strategy, wherein the target includes fruit and background, and the background includes simple background and complex background; determining the target contour of the target salient area according to the maximum expectation algorithm and the pulse coupled neural network; determining first information according to the target salient area in the initial visible light image and the target salient area in the initial ToF image; determining second information according to the target contour in the initial visible light image and the target contour in the initial ToF image; determining the aligned ToF image according to the first information and the second information.

[0006] In an optional embodiment, the step of determining a target salient area of ​​a target in an initial image according to a local peak preselection strategy comprises: dividing the initial image into a preset number of image blocks; extracting the grayscale value of each pixel of the image blocks; distinguishing and determining a fruit salient area and a background salient area according to the grayscale value and the local peak preselection strategy, wherein the target salient area comprises a fruit salient area and a background salient area.

[0007] In an optional embodiment, the steps of preliminarily distinguishing and determining the fruit-significant area and the background-significant area based on the grayscale value and the local peak preselection strategy include: determining the two-dimensional distribution probability of the fruit based on the two-dimensional normal distribution; performing peak characteristic discrimination calculation on the two-dimensional distribution probability of the fruit according to a preset fixed difference threshold, and distinguishing and determining the fruit-significant area and the background-significant area.

[0008] In an optional embodiment, the step of determining the target contour of the target salient area according to the maximum expectation algorithm and the pulse coupled neural network includes: determining the dynamic threshold and connection strength of the target salient area based on the maximum expectation algorithm and the pulse coupled neural network; converging the fruit salient area based on the dynamic threshold and the connection strength to obtain the fruit contour; converging the background salient area based on the dynamic threshold and the connection strength to obtain the background contour, wherein the target contour includes the fruit contour and the background contour.

[0009] In an optional embodiment, the step of determining the dynamic threshold of the target salient area includes: using a preset maximum expectation algorithm to predict the maximum likelihood value of the target salient area, and determining the mean and covariance matrix in the horizontal and vertical directions; iteratively calculating the dynamic threshold of the target salient area on the premise that the grayscale value of the target salient area is consistent; stopping the step of iteratively calculating the dynamic threshold of the target salient area according to a preset iteration termination condition to obtain the final dynamic threshold of the target salient area.

[0010] In an optional embodiment, first information is determined based on the target salient area in the initial visible light image and the target salient area in the initial ToF image; second information is determined based on the target contour in the initial visible light image and the target contour in the initial ToF image; and steps for registering the ToF image are determined based on the first information and the second information, including: registering the fruit salient area of ​​the initial visible light image with the fruit salient area of ​​the initial ToF image based on the Hu invariant moment target area contour matching processing method and the random sampling consensus algorithm to obtain a first registration result; registering the background salient area of ​​the initial visible light image with the background salient area of ​​the initial ToF image to obtain a second registration result; registering the fruit contour of the initial visible light image with the fruit contour of the initial ToF image to obtain a third registration result; registering the background contour of the initial visible light image with the background contour of the initial ToF image to obtain a fourth registration result; and changing the angle and position of the initial ToF image based on the first registration result, the second registration result, the third registration result and the fourth registration result to obtain a registered ToF image.

[0011] In an optional embodiment, the step of performing fruit recognition on the fruits in the initial visible light image according to a preset YOLO v3 convolutional neural network model to obtain a fruit recognition result includes: training the preset convolutional neural network model according to a pre-acquired data set of labeled images to obtain a fruit recognition weight model; and determining whether there is fruit in the initial visible light image based on a preset fruit prior frame and the fruit recognition weight model.

[0012] In an optional embodiment, the initial ToF image includes an initial depth image, an initial confidence image, an initial point cloud image and an initial intensity image; the registered ToF image includes a registered depth image, a registered confidence image, a registered point cloud image and a registered intensity image; the step of determining the position information of the fruit based on the registered ToF image includes: determining the horizontal coordinate, vertical coordinate, and vertical coordinate of the fruit and the rotation angle of the registered ToF image relative to the initial ToF image through the registered depth image.

[0013] In a second aspect, the present invention provides a fruit positioning device based on a binocular vision system, the device comprising: a registration module, which is used to register the pre-acquired initial ToF image and the initial visible light image based on the pre-acquired initial visible light image, based on the local peak preselection strategy and the pulse coupled neural network, to obtain a registered ToF image; an identification module, which is used to identify the fruit in the initial visible light image according to a preset YOLO v3 convolutional neural network model to obtain a fruit identification result; a positioning module, which is used to determine the position information of the fruit according to the registered ToF image if the fruit identification result indicates that there is fruit in the initial visible light image.

[0014] The beneficial effects of the embodiments of the present invention are as follows:

[0015] The present invention provides a method and device for locating fruits based on a binocular vision system, the method comprising: taking a pre-acquired initial visible light image as a reference, based on a local peak pre-selection strategy and a pulse coupled neural network, registering a pre-acquired initial ToF image and an initial visible light image to obtain a registered ToF image; performing fruit recognition on the fruits in the initial visible light image according to a preset YOLO v3 convolutional neural network model to obtain a fruit recognition result; if the fruit recognition result indicates that the fruit exists in the initial visible light image, determining the position information of the fruit according to the registered ToF image. This scheme can improve the accuracy of fruit recognition and positioning.

[0016] Other features and advantages of the present invention will be set forth in the following description, or some features and advantages may be inferred or unambiguously determined from the description, or may be learned by implementing the above-mentioned technology of the present invention.

[0017] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are specifically listed below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A flow chart of a fruit positioning method based on a binocular vision system provided by an embodiment of the present invention;

[0020] Figure 2 A flowchart of another fruit positioning method based on a binocular vision system provided by an embodiment of the present invention;

[0021] Figure 3 A local peak pre-selection map of a ToF image provided by an embodiment of the present invention;

[0022] Figure 4 A local peak pre-selection map of a visible light image provided by an embodiment of the present invention;

[0023] Figure 5 A registration flow chart provided for an embodiment of the present invention;

[0024] Figure 6A schematic diagram of a fruit positioning device based on a binocular vision system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0026] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] In orchard operations, the accurate recognition of fruits by the vision system of agricultural harvesting robots is the key to the successful implementation of automated harvesting. The most promising technology in the visual research of harvesting robots is the technology of heterogeneous image information fusion between Time of Flight (ToF) images and color images (i.e., visible light images). ToF images have many different properties, such as illumination invariance, spatial hierarchy, near-infrared perception, and reliability of analytical data. In heterogeneous images, image registration is an important preprocessing technology in image analysis.

[0028] At present, image registration methods are divided into two categories: registration methods based on point features and registration methods based on regional features. Among them, the scale-invariant feature transform (SIFT) algorithm and the speeded up robust features (SURF) algorithm are two typical point feature-based registration algorithms. These two methods have certain stability to rotation, scaling, and brightness changes. Paul proposed an improved SIFT algorithm in anisotropic scale space to solve the problem of detail loss in the operation. Feng Juan et al. used the SURF algorithm to extract the scale-invariant features of the ToF image to be registered, and the Euclidean distance was used as a measure to judge the similarity of the features to complete the matching. Zhou Wei used Harris to extract the corner points in the ToF and visible light images as feature points, and used the mutual correlation information and the support strength of the neighborhood to achieve feature point matching. Zeng used the morphological gradient method to extract the rough edges of the image and align its visual features into a single similarity type; secondly, the C_SIFT feature detection operator was used to detect and extract feature points on the extracted edges to complete the registration of visible light and infrared images, and the registration accuracy was good.

[0029] The applicability of the SIFT operator has been analyzed in the existing technologies. The results show that the SIFT operator is stable for the registration of homologous images, but not ideal for heterologous images. In addition, most of the current technologies are for infrared and visible light images, and are not applicable to ToF images.

[0030] The technology and prior art of PCNN (Pulse Coupled Neural Network) are as follows:

[0031] PCNN uses the principle of similarity clustering and capture characteristics to classify similar pattern features into one category and dissimilar features into other categories. This is completely consistent with the basic idea of ​​cluster analysis and can be well applied to data clustering. The output forms of all neurons form a two-dimensional ignition matrix, and the output binary image is used as the segmentation result of this time; the ignition map will be used as a link input feedback at the beginning of the next iteration to directly affect the internal behavior of each neuron. When the termination condition is reached, the iteration is terminated, the segmentation result is retained and the result image is output.

[0032] Traditional PCNN only uses grayscale similarity clusters for segmentation, which makes it impossible to completely separate targets and backgrounds with similar grayscales. Because in most cases, objects in the data set cannot be divided into clearly separated clusters, assigning an object to a specific cluster is somewhat blunt and may be wrong. PCNN captures too monotonously with grayscale similarity, which will affect segmentation when the grayscale difference between the target and the background is not large.

[0033] Moreover, in the complex environment of orchard operations, the imaging mechanisms between visible light images and ToF images are completely different, the imaging angles have a certain offset, and the grayscale values ​​of the same-name points on the image generally do not have a linear transformation relationship, making the registration method that relies on the feature information of the image points invalid.

[0034] Based on this, the present invention provides a method and device for fruit positioning based on a binocular vision system to register heterogeneous images (visible light images and ToF images) and then identify and locate the fruit. This technology can be applied to the application scenario of automated picking in orchards.

[0035] Embodiment 1

[0036] The embodiment of the present invention provides a fruit positioning method based on a binocular vision system. Figure 1 As shown, the method includes:

[0037] Step S102 , taking the pre-acquired initial visible light image as a reference, based on a local peak pre-selection strategy and a pulse coupled neural network, the pre-acquired initial ToF image and the initial visible light image are registered to obtain a registered ToF image.

[0038] Specifically, the above-mentioned initial visible light image and the above-mentioned ToF image are called a set of heterogeneous images. Here, the ToF image is divided into an intensity map, a depth map, a confidence map and a 3D point cloud map. At the beginning, at the same time and the same position, a ToF image is taken with a ToF camera, and a visible light image is taken with a visible light camera. Then, based on the visible light image, the confidence map is rotated and translated to align the confidence map with the visible light image. In this way, the intensity map, the depth map and the 3D point cloud map can also be automatically aligned with the visible light image.

[0039] Specifically, this application uses a ToF camera and a visible light camera to establish a binocular acquisition system for an orchard environment. A ToF camera from Basler, Germany, is used, with a built-in light source with an emission wavelength of 850nm and ±30nm, and a field of view (H×V) of 57°×43°. This article uses a confidence map as an experimental image. The visible light camera uses a Canon color digital camera, the sensor type is CCD, and the effective pixels are 1.41 million. The ToF camera and the visible light camera are fixed in parallel on a camera bracket to form a binocular acquisition system (i.e., a binocular vision system).

[0040] Step S104, performing fruit recognition on the fruits in the initial visible light image according to a preset YOLO v3 convolutional neural network model to obtain a fruit recognition result.

[0041] Specifically, the above visible light image is also called RGB image. Here, the YOLO model is used to perform fruit recognition on the RGB image to determine whether there is fruit in the image. The above fruit includes objects that need to be recognized in agriculture, such as apples and peaches.

[0042] Step S106: If the fruit recognition result indicates that there is fruit in the initial visible light image, the position information of the fruit is determined based on the registered ToF image.

[0043] Specifically, the registered depth image is used to calculate the spatial coordinates X, Y, Z and the rotation angle R of the fruit, thereby completing the spatial positioning of the fruit.

[0044] This application transforms the two image data into the same coordinate system through registration, combines spatial information and semantic information to give full play to the advantages of ToF images and visible light images, and makes up for the limitations of a single data source. The images targeted all contain apples, branches, leaves, and complex orchard environments. There is an obvious spatial rotation and offset between the ToF image and the visible light image, so the registration process of this application solves the difference in this rotation and offset.

[0045] The present invention provides a method for locating fruits based on a binocular vision system, the method comprising: taking a pre-acquired initial visible light image as a reference, based on a local peak pre-selection strategy and a pulse coupled neural network, registering a pre-acquired initial ToF image and an initial visible light image to obtain a registered ToF image; performing fruit recognition on the fruits in the initial visible light image according to a preset YOLO v3 convolutional neural network model to obtain a fruit recognition result; if the fruit recognition result indicates that the fruit exists in the initial visible light image, determining the position information of the fruit according to the registered ToF image. This scheme can improve the accuracy of fruit recognition and positioning.

[0046] This application goes through the first step: First, use the visible light image and Tof confidence image to perform a local peak preselection strategy and an improved PCNN segmentation method to accurately align the two images of the binocular vision system. The second step: Secondly, use the YOLOv3 convolutional neural network model in deep learning to recognize the visible light image and mark the identified fruits respectively. The third step: Finally, use the aligned depth image to calculate the marked fruits one by one to determine the spatial coordinates XYZ and the rotation angle R of the fruit, and complete the spatial positioning of the fruit. This application realizes the recognition and positioning of the fruit through the above three steps.

[0047] Embodiment 2

[0048] The embodiment of the present invention provides another fruit positioning method based on a binocular vision system. Figure 2 As shown, the method includes:

[0049] Step S202: determining a salient target region of the target in the initial image according to a local peak preselection strategy.

[0050] In a specific implementation, the initial visible light image and the initial ToF image are both referred to as initial images. The initial ToF image includes an initial depth image, an initial confidence image, an initial point cloud image, and an initial intensity image.

[0051] In the specific implementation, the goals include fruits and background, and the background includes simple background and complex background.

[0052] In the specific implementation, for step S202, first, the initial image is divided into a preset number of image blocks; the gray value of each pixel of the image block is extracted; according to the gray value and the local peak pre-selection strategy, the fruit salient area and the background salient area are distinguished and determined, wherein the target salient area includes the fruit salient area and the background salient area. Then, based on the two-dimensional normal distribution, the two-dimensional distribution probability of the fruit is determined; according to the preset fixed difference threshold, the peak characteristic discrimination calculation is performed on the two-dimensional distribution probability of the fruit, and the fruit salient area and the background salient area are distinguished and determined.

[0053] Specifically, both the fruit salient region and the background salient region are target salient regions.

[0054] More specifically, the steps for determining the target salient area are:

[0055] 1. Collect the initial visible light image and the initial confidence image at the same time and location.

[0056] Specifically, the advantages of the confidence image are: illumination invariance, spatial hierarchy, near-infrared perception, and reliability of data analysis. The following operation is to calculate the initial visible light image and the initial confidence image at the same time, and obtain the target salient area of ​​the initial visible light image and the target salient area of ​​the initial confidence image respectively.

[0057] 2. Divide the initial image into a preset number of image blocks.

[0058] 3. Extract the gray value of each pixel in the small image block.

[0059] 4. Assume that the target block (i.e., the image block) can be divided into three parts according to the gray value: fruit, ordinary background, and complex background. The specific expressions are as follows:

[0060] F(x, y) = f t (x,y)+b c (x, y) + b(x, y) (1);

[0061] Among them, F(x, y) is the gray value of the target small block image at (x, y), f t (x, y) is the fruit component, b(x, y) is the normal background component (such as the sky), and b c (x, y) is the complex background component (such as leaves, branches and other debris). Formula (1) only reflects the idea of ​​dividing the image into three parts, and there is no specific calculation process.

[0062] 5. The local gray value of the fruit component has a peak-like characteristic, which is approximately a two-dimensional normal distribution, that is, the fruit component f t (x, y) can be approximately described as:

[0063]

[0064]

[0065] Where I(x, y) is the intensity of the gray value of the i-th target at (x, y), exp is the power of e, x oi is the horizontal coordinate of the center point of the image, y oi is the ordinate of the center point of the image, is the i-th calculation area, n is the total number of calculation areas, δ 1i and δ 2i are all preset two-dimensional normal distribution parameters. The target area for the fruit.

[0066] For the target (fruit), any Δx 2 +Δy 2 =ω 2 ,satisfy Among them, ω is the diameter of the fruit, which is a preset value.

[0067] Based on the above formula, the two-dimensional distribution probability of the fruit ft(x, y) can be calculated.

[0068] Substituting (2) into (1), we obtain F(x, y) with the two-dimensional distribution probability of fruit.

[0069] 6. Target salient region extraction strategy based on local peak (TSE-LP) is used to extract the salient regions of the image. t (x, y) is used to perform peak characteristic discrimination calculation to obtain the target salient area.

[0070]

[0071] Among them, max(|Δx|, |Δy|)=ω.

[0072] Among them, Otsu is an automatic threshold segmentation algorithm, U is a preset fixed difference threshold, and F(x oi ,y oi ) is the grayscale value of the target small block image at (x, y).

[0073] By calculating the above formula (3), the background or target in the target small block image can be distinguished. The distinction method is that if ≤U, the position is the target; if >U, the position is a complex background or a common background, that is, the target salient area is obtained. Figure 3 and Figure 4As shown in the figure, the target salient area is the area enclosed by the box. Figure 3 It is the local peak pre-selection map of the ToF image. Figure 4 It is the local peak pre-selection map of the visible light image.

[0074] Specifically, after the salient regions are extracted, the overall image can be segmented into local areas, simplifying the next segmentation problem. It also preliminarily screens the fruit targets and speeds up the operation.

[0075] Specifically, at this point, we have only preliminarily obtained the approximate area where the fruit or the background is located. The result is not accurate, and it is impossible to determine whether a certain position is the background or the fruit.

[0076] Step S204, determining the target contour of the target salient area according to the maximum expectation algorithm and the pulse coupled neural network.

[0077] In the specific implementation, the process of the above step S204 is: based on the maximum expectation algorithm and the pulse coupled neural network, determine the dynamic threshold and connection strength of the target salient area; based on the dynamic threshold and connection strength, converge the fruit salient area to obtain the fruit contour; based on the dynamic threshold and connection strength, converge the background salient area to obtain the background contour, wherein the target contour includes the fruit contour and the background contour.

[0078] In the specific implementation, the step of determining the dynamic threshold of the target salient area includes: using a preset maximum expectation algorithm to predict the maximum likelihood value of the target salient area, and determining the mean and covariance matrix in the horizontal and vertical directions; iteratively calculating the dynamic threshold of the target salient area under the premise that the grayscale value of the target salient area is consistent; stopping the step of iteratively calculating the dynamic threshold of the target salient area through a preset iteration termination condition to obtain the final dynamic threshold of the target salient area.

[0079] Since the previous step only extracted the approximate area where the target is located, but did not extract the contour, the contour needs to be extracted next. The prior art also has a contour extraction step, and the prior art extracts contours based on a preset specific dynamic threshold. This extraction method is inaccurate, so this application will dynamically determine the dynamic threshold and connection strength based on the grayscale value in the target salient area, and then extract the contour, so that the extracted contour will be more accurate.

[0080] More specifically, the steps of determining the target contour of the target salient area are:

[0081] 1. The edges of the fruit or the background are smooth, so they satisfy the two-dimensional Gaussian normal distribution. Therefore, the target salient area (i.e. the box area) can be processed with two-dimensional Gaussian normal. The target area has a smooth arc, that is, the data satisfies the two-dimensional Gaussian normal distribution:

[0082]

[0083] Among them, μ1 is the expectation of the X-axis, μ2 is the expectation of the Y-axis, is the mean square error in the X direction, is the mean square error in the Y direction, and ρ is the correlation coefficient in the X and Y directions.

[0084] 2. Take the likelihood function as the sample joint probability density, recorded as:

[0085] L(μ,∑)=f(x1,x2,x3,…,x N ) (5);

[0086] Among them, L(μ, ∑) is the joint probability density of samples, and N represents the total number of samples.

[0087] 3. Use the preset Expectation-Maximization algorithm (EM algorithm) to predict the maximum likelihood value of the target salient area and determine the mean and covariance matrix in the horizontal and vertical directions.

[0088] Specifically, the maximum expectation algorithm (EM algorithm) is used to predict the maximum likelihood value (many values ​​can be predicted, but only two are predicted here). The mean and covariance matrix in the horizontal and vertical directions are determined based on the two predicted maximum likelihood values. Specifically, they are:

[0089]

[0090] in, is the mean of each variable (i.e. μ1 and μ2). is the covariance matrix. i is the gray value of the pixel at the i-th point. is the grayscale mean. The maximum likelihood value is to find the most likely result of a set of data. The purpose of formula (6) is to find the mean and covariance of the two-dimensional normal distribution.

[0091] Specifically, the Expectation Maximization Algorithm (EM) uses the maximum likelihood value to determine the classification of elements, which is very consistent with PCNN. The maximum likelihood estimate is recalculated according to the current ignition map, and the grayscale similarity clustering is completed during the ignition process, providing a more flexible clustering result. Based on this, an improved pulse coupled neural network model of the Expectation Maximization Algorithm is proposed.

[0092] 4. Under the premise that the grayscale value of the target salient area is consistent, iteratively calculate the dynamic threshold of the target salient area.

[0093] Specifically, the steps for calculating the above dynamic threshold are:

[0094]

[0095] Where i and j are the current image coordinates; f(μ1, μ2) is the center value of the two-dimensional Gaussian distribution, which is calculated by formula (6); is the sample joint probability density; v θ (i, j) is the dynamic threshold of the i, jth point.

[0096] 5. According to the preset iteration termination condition, the step of iteratively calculating the dynamic threshold of the target salient area is stopped to obtain the final dynamic threshold of the target salient area.

[0097] This application calculates the model iteration termination condition by the following formula:

[0098]

[0099] Among them, I refers to the target salient area (i.e., the gray value in the box), U I is the regional variance consistency, R k is the set of pixels in the target salient area. For a segmentation result image, U I The larger it is, the better the consistency within the region. max 、f min are the maximum and minimum grayscale values ​​of the image respectively; is the average gray value of the target salient area; f i is the gray value of point i; M and N are the length and width of the target salient area respectively.

[0100] When iterating until The iteration is terminated when , which indicates that the output area and the target area are mostly consistent and no longer updated. At this time, the iteration can be stopped and the result can be output.

[0101] Here, ε1 is the first stop iteration parameter, which is a preset value; ε2 is the second stop iteration parameter, which is also a preset value; and n is the number of iterations. After the iteration stops, multiple dynamic thresholds are obtained.

[0102] 6. Calculate the connection strength.

[0103] The connection strength adjusts the influence of the surrounding neurons on the central neuron. Therefore, the image gradient is used to adjust the connection strength. When the image gradient amplitude is larger, the grayscale of the central pixel changes faster and the discreteness is greater. Therefore, a larger capture ability can make similar neurons fire synchronously. Similarly, the smaller the image gradient amplitude, the smaller the capture ability can fire synchronously.

[0104] Based on the above analysis, the link strength is positively correlated with the amplitude of the image gradient. The calculation method of the link strength is:

[0105]

[0106] Where β is the connection strength. M and N are the length and width of the target salient area, respectively. Derivative x, y. Ω represents the image area (i.e., the target salient area).

[0107] 7. Based on the dynamic threshold and connection strength, the background salient area is converged to obtain the background contour, where the target contour includes the fruit contour and the background contour.

[0108] The above 1-7 are the improved EM-PCNN method, which is the important invention of this application. This method can be more accurate for segmentation, and has better segmentation robustness for background targets such as branches and leaves. The dynamic threshold is changed to an adaptive method, and the link strength (i.e., connection strength) is improved based on the regional gradient. In the process of continuous convergence, the response to the edge is more sensitive and the segmentation effect is better. This application proposes an adaptive termination condition, which simplifies the steps that require obtaining the number of cycles through artificial experience, and is more applicable.

[0109] Specifically, the image segmentation implemented by PCNN is an iterative output process, and the dynamic threshold is the key parameter to control this iterative output process. In ToF images and visible light images, fruits often appear as areas with relatively high grayscale values, and the grayscale distribution satisfies a two-dimensional normal distribution.

[0110] Step S206, determining first information based on the target salient area in the initial visible light image and the target salient area in the initial ToF image; determining second information based on the target contour in the initial visible light image and the target contour in the initial ToF image; and determining a registered ToF image based on the first information and the second information.

[0111] In specific implementation, the ToF image registration includes the depth image registration, the confidence image registration, the point cloud image registration and the intensity image registration. The target salient area is divided into the fruit salient area and the background salient area. The target contour is divided into the fruit contour and the background contour.

[0112] In the specific implementation, step S206 is specifically divided into: based on the target area contour matching processing method of Hu invariant moment and the random sampling consensus algorithm, the fruit salient area of ​​the initial visible light image is aligned with the fruit salient area of ​​the initial ToF image to obtain a first alignment result; the background salient area of ​​the initial visible light image is aligned with the background salient area of ​​the initial ToF image to obtain a second alignment result; the fruit contour of the initial visible light image is aligned with the fruit contour of the initial ToF image to obtain a third alignment result; the background contour of the initial visible light image is aligned with the background contour of the initial ToF image to obtain a fourth alignment result; based on the first alignment result, the second alignment result, the third alignment result and the fourth alignment result, the angle and position of the initial ToF image are changed to obtain a aligned ToF image.

[0113] Specifically, in layman's terms, the steps for determining the registration of the ToF image are:

[0114] Based on the target area contour matching processing of Hu invariant moment and the Random Sample Consensus (RANSAC) algorithm purification processing, the visible light target salient area and the confident target salient area are registered to obtain the registered target salient area; the visible light contour and the confident contour are registered to obtain the registered contour. In this way, the contours and positions that are consistent in both photos can be used as the registered target salient area and the registered contour. Using the normalized central moment, 7 moment invariants with rotation, scaling or translation are further calculated: k represents the kth order invariant moment of the target area I. In this paper, the following formula is used to match the closed target area of ​​the segmented ToF image (including the confidence map) and the visible light image:

[0115]

[0116] in represents the kth order invariant moment of the i-th closed target region in the visible light image, represents the kth order invariant moment of the jth closed target region in the visible light image. ij The smaller the value is, the more similar the features of the two regions are. 7 means that 7 calculations are performed. The above are all the processes of target region contour matching processing using Hu invariant moments.

[0117] For the ToF image and the visible light image, the number of contours and the same-name points retrieved (i.e., the contours obtained by matching the target area contours with the Hu invariant moment) are different. The same-name points of the two images are extracted and then purified by the Random Sample Consensus (RANSAC) algorithm to complete the registration. Specifically, the registration flow chart is as follows: Figure 5 shown.

[0118] Specifically, the result of the registration is a registration confidence image. Based on the registration confidence image and the same transformation method, a registration depth image, a registration point cloud image and a registration intensity image can be quickly obtained.

[0119] Step S208, training the preset convolutional neural network model according to the pre-acquired image labeled data set to obtain a fruit recognition weight model; based on the preset fruit prior frame and the fruit recognition weight model, determining whether there is fruit in the initial visible light image.

[0120] Specifically, this step is also called fruit recognition. Taking apples as an example, the YOLO recognition process is as follows:

[0121] The YOLOv3 convolutional neural network model in deep learning is used to identify visible light images. The Yolo network used in the Yolo recognition module is the Yolo v3 version (the third version). The original pre-trained model of Yolo v3 is used, and the prior box (i.e., anchor box) is defined as a square box, so that the model can have better sensitivity to apples. The Yolo v3 model is built using PyTorch and deployed on high-performance servers. The model is trained using CUDA (Compute Unified Device Architecture) technology for hardware accelerated training. The data used for training and testing is a data set after image labeling by image software. By training the pre-trained model, a weight model that is accurate in identifying apples can be obtained. At this point, the training part is completed, and the weight model will be placed in the front-end edge control chip to perform the recognition function.

[0122] The computing power of the front-end edge control chip is used to accurately identify apples, and multi-CPU parallel acceleration is enabled on the hardware to improve the recognition speed of the front-end edge control chip. The front-end edge control chip also uses PyTorch to build a neural network, then loads the pre-trained model weight file to complete a neural network dedicated to identifying apples. The front-end edge control chip then obtains a color image from the RGB camera, loads the image into the network, and detects the apple in the image through Yolo v3. Since the prior frame is predetermined in advance, it can be more sensitive to apples, so that the position of the apple and the size in the image can be accurately captured.

[0123] Through the Yolo v3 neural network in the front-end edge control chip, a list of recognition results will be returned. The list consists of five columns of attributes. For a single row of data, the first column is the recognized category. After training, Yolo v3 can recognize apples more accurately, but it can also recognize other target objects. The remaining four columns describe the upper left corner, upper right corner, lower left corner, and lower right corner of the target respectively. These four positions will form a square box, and the edge of the square box should fit the edge of the target as close as possible. Through calculation, the center position of the object can be roughly obtained. Therefore, the Yolo v3 recognition function module can return the location information of all recognizable objects in the image, as well as the specific category of the object.

[0124] The beneficial effects of using YOLO recognition are as follows:

[0125] In the current era, there are many neural network models that can perform target detection, such as multi-network detection models that use ResNet and other networks to cooperate, etc. However, only the Yolo series of network models can use a single network to simultaneously achieve object detection and classification. The Yolo series of network models are relatively lightweight network models that can be applied to some small lightweight computers and are suitable for mobile devices. Yolo v3 (that is, three versions of it), compared to the later larger network versions, can be deployed in some lightweight computers. At the same time, compared to previous versions, it can better detect small objects. Therefore, in the case of this project, considering the need to share computer resources with PCNN registration, using Yolo v3 for target detection is a very cost-effective decision.

[0126] Step S210, by registering the depth image, determining the horizontal coordinate, the vertical coordinate, and the rotation angle of the registered ToF image relative to the initial ToF image.

[0127] This application can quickly locate the position of fruits by combining visible light and ToF images. Compared with binocular color and other methods, it is not affected by the grayscale and features of the object surface, and can measure longer distances. The accuracy is independent of distance, the measurement accuracy is high, and the robustness is higher.

[0128] This application can improve the accuracy of fruit recognition and positioning through a local peak preselection strategy and an improved PCNN segmentation method.

[0129] Aiming at the problem of poor matching accuracy between ToF and visible light heterogeneous images collected by binocular acquisition system in orchard environment, this application proposes a ToF and visible light orchard apple image registration method based on pulse coupled neural network segmentation of local peak target salient region extraction strategy and maximum expectation algorithm. First, the Gaussian difference function is used to calculate the salient region in the visible light image, and the red and green components of the visible light image are preprocessed; then, the two-dimensional normal distribution of the local gray value of the image is used as the target component, and the maximum inter-class variance Otsu is used to extract the foreground with a fixed threshold as the local peak extraction strategy, and the feature regions of the ToF and visible light images are preliminarily screened, and the maximum expectation algorithm is used to improve the PCNN dynamic threshold, and the local image gradient is used to calculate the link strength. The link strength is calculated, and the image area variance is used to improve the termination condition, and an EM-PCNN algorithm is proposed to refine the segmentation of the pre-selected area; then the invariant moment of the connected area is calculated, and the invariant principle is used to find the target center homonymous points, and the feature areas are further screened; finally, the homonymous points are RANSAC purified, and the coordinates of the purified homonymous points are substituted into the transformation model to calculate the model parameters to complete the registration. Experimental results show that this method can achieve good registration effect for ToF and visible light heterogeneous orchard apple images collected by the binocular acquisition system, and is resistant to image translation, rotation and scaling.

[0130] Embodiment 3

[0131] The embodiment of the present invention provides a fruit positioning device based on a binocular vision system. Figure 6 As shown, the device comprises:

[0132] The registration module 51 is used to register the pre-acquired initial ToF image and the initial visible light image based on the pre-acquired initial visible light image as a reference, based on the local peak pre-selection strategy and the pulse coupled neural network, to obtain a registered ToF image.

[0133] The recognition module 52 is used to perform fruit recognition on the fruits in the initial visible light image according to a preset YOLOv3 convolutional neural network model to obtain a fruit recognition result.

[0134] The positioning module 53 is used to determine the position information of the fruit according to the registered ToF image if the fruit recognition result indicates that the fruit exists in the initial visible light image.

[0135] The fruit positioning device based on the binocular vision system provided in the embodiment of the present invention has the same implementation principle and technical effects as the aforementioned fruit positioning method embodiment based on the binocular vision system. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fruit positioning method based on a binocular vision system, characterized in that: The method comprises: According to the local peak preselection strategy, a target salient area of ​​the target in the initial image is determined, wherein the target includes a fruit and a background, and the background includes a simple background and a complex background; the initial visible light image and the initial ToF image are both referred to as the initial image; Determine the dynamic threshold and connection strength of the target salient area based on the maximum expectation algorithm and the pulse coupled neural network; Converging the fruit salient area based on the dynamic threshold and the connection strength to obtain a fruit outline; Based on the dynamic threshold and the connection strength, the background salient area is converged to obtain a background contour, wherein the target contour includes the fruit contour and the background contour; Determining first information according to the target salient area in the initial visible light image and the target salient area in the initial ToF image; Determining second information according to the target outline in the initial visible light image and the target outline in the initial ToF image; Determine a registered ToF image according to the first information and the second information; Performing fruit recognition on the fruits in the initial visible light image according to a preset YOLOv3 convolutional neural network model to obtain a fruit recognition result; If the fruit recognition result indicates that the fruit exists in the initial visible light image, the position information of the fruit is determined according to the registered ToF image.

2. The method according to claim 1, characterized in that The step of determining a target salient area of ​​a target in an initial image according to the local peak preselection strategy comprises: Dividing the initial image into a preset number of small image blocks; Extracting the grayscale value of each pixel of the image block; According to the gray value and the local peak pre-selection strategy, the fruit salient area and the background salient area are distinguished and determined, wherein the target salient area includes the fruit salient area and the background salient area.

3. The method according to claim 2, characterized in that The step of preliminarily distinguishing and determining the fruit salient area and the background salient area according to the grayscale value and the local peak preselection strategy comprises: Based on the two-dimensional normal distribution, determining the two-dimensional distribution probability of the fruit; According to a preset fixed difference threshold, the peak characteristic discrimination calculation is performed on the two-dimensional distribution probability of the fruit, and the fruit significant area and the background significant area are distinguished and determined.

4. The method according to claim 1, characterized in that The step of determining a dynamic threshold of the target salient area comprises: Using a preset maximum expectation algorithm, predicting the maximum likelihood value of the target salient area, and determining the mean and covariance matrix in the horizontal and vertical directions; On the premise that the grayscale values ​​of the target salient area are consistent, iteratively calculating the dynamic threshold of the target salient area; The step of iteratively calculating the dynamic threshold of the target salient area is stopped according to a preset iterative termination condition, so as to obtain a final dynamic threshold of the target salient area.

5. The method according to claim 1, characterized in that Determine first information according to the target salient area in the initial visible light image and the target salient area in the initial ToF image; determine second information according to the target outline in the initial visible light image and the target outline in the initial ToF image; Determining a step of registering a ToF image according to the first information and the second information includes: Based on the target area contour matching processing method of Hu invariant moment and the random sampling consensus algorithm, the fruit salient area of ​​the initial visible light image is registered with the fruit salient area of ​​the initial ToF image to obtain a first registration result; Registering the background salient area of ​​the initial visible light image with the background salient area of ​​the initial ToF image to obtain a second registration result; Registering the fruit contour in the initial visible light image with the fruit contour in the initial ToF image to obtain a third registration result; Registering the background contour of the initial visible light image with the background contour of the initial ToF image to obtain a fourth registration result; Based on the first registration result, the second registration result, the third registration result and the fourth registration result, the angle and position of the initial ToF image are changed to obtain the registered ToF image.

6. The method according to claim 1, characterized in that The step of performing fruit recognition on the fruits in the initial visible light image according to a preset YOLOv3 convolutional neural network model to obtain a fruit recognition result includes: According to the pre-acquired labeled image data set, the preset convolutional neural network model is trained to obtain a fruit recognition weight model; Based on a preset fruit priori frame and the fruit recognition weight model, it is determined whether the fruit exists in the initial visible light image.

7. The method according to claim 1, characterized in that The initial ToF image includes an initial depth image, an initial confidence image, an initial point cloud image and an initial intensity image; The registered ToF image includes a registered depth image, a registered confidence image, a registered point cloud image and a registered intensity image; The step of determining the position information of the fruit according to the registered ToF image comprises: The horizontal coordinate, the vertical coordinate, the ordinate and the perpendicular coordinate of the fruit and the rotation angle of the registered ToF image relative to the initial ToF image are determined by the registered depth image.

8. A fruit positioning device based on a binocular vision system, characterized in that: The device comprises: A registration module is used to determine a target salient area of ​​a target in an initial image according to a local peak preselection strategy, wherein the target includes a fruit and a background, and the background includes a simple background and a complex background; the initial visible light image and the initial ToF image are both referred to as the initial image; based on a maximum expectation algorithm and a pulse coupled neural network, a dynamic threshold and a connection strength of the target salient area are determined; based on the dynamic threshold and the connection strength, the fruit salient area is converged to obtain a fruit outline; based on the dynamic threshold and the connection strength, the background salient area is converged to obtain a background outline, wherein the target outline includes the fruit outline and the background outline; based on the target salient area in the initial visible light image and the target salient area in the initial ToF image, first information is determined; based on the target outline in the initial visible light image and the target outline in the initial ToF image, second information is determined; based on the first information and the second information, a registered ToF image is determined; A recognition module, used for performing fruit recognition on the fruits in the initial visible light image according to a preset YOLOv3 convolutional neural network model to obtain a fruit recognition result; A positioning module is used to determine the position information of the fruit according to the registered ToF image if the fruit recognition result indicates that the fruit exists in the initial visible light image.

Images