A method, system, device and medium for three-dimensional reconstruction of wine picking based on a binocular camera

Abstract

The present application relates to a kind of based on binocular camera's three-dimensional reconstruction's wine picking method, system, equipment and medium, its method includes: according to hop image pair camera calibration obtains internal and external parameters;Based on internal and external parameters, get the polar line on hop image, carry out polar line correction to hop image and obtain corrected image;Every two corrected images are stereomatching, and three-dimensional reconstruction original point cloud is carried out;All point clouds are registered with reference point cloud and nearest neighbor search for multiple times, until meeting the condition to stop iteration, while in the iteration process, the parameter of original point cloud is constantly optimized to obtain best point cloud;According to best point cloud and camera pose, obtain liquid surface gully information;According to liquid surface gully information, segment out bubble region on best point cloud and obtain bubble information, determine the quality grade of wine based on bubble information.The present application uses binocular machine vision technology, analyzes wine flower form, identifies picking stage and executes corresponding production strategy, replaces artificial and realizes intelligent production.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a liquor picking method, system, device and medium based on three-dimensional reconstruction of a binocular camera. BACKGROUND

[0002] Liquor picking is a traditional technique in the distillation process of Baijiu (Chinese liquor), which refers to the process of judging the alcohol degree of the distillate by observing the size and residence time of the wine flower, and separating the medium-high alcohol liquor and low-concentration liquor at the appropriate time. The wine flower is the bubbles of different shapes caused by the different surface tensions of the distillate at different stages due to the different alcohol degrees and temperatures. In the liquor section detection process of liquor picking, the shape of the wine flower is a crucial parameter.

[0003] The existing liquor section detection methods are divided into three categories according to different detection principles: automatic liquor picking based on process parameters, automatic liquor picking based on analytical instruments, and automatic liquor picking based on machine vision. The automatic liquor picking based on process parameters uses temperature, pressure and other process parameters as indicators for Baijiu segmentation. This method is simple and easy to implement, but it is easily disturbed and has low reliability. The automatic liquor picking based on analytical instruments uses spectrometers, chromatographs, mass spectrometers and other analytical instruments to analyze the chemical composition of the liquor as the basis for Baijiu segmentation. This method is expensive and has strict requirements for the working environment, and the detection speed is slow, which cannot be monitored online. The existing automatic liquor picking based on machine vision is basically based on two-dimensional image operation analysis, and does not use spatial information, which makes the expression of wine flower information not rich enough, thereby affecting the subsequent liquor section judgment. SUMMARY

[0004] (I) Technical problems to be solved

[0005] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present application provides a liquor picking method, system, device and medium based on three-dimensional reconstruction of a binocular camera, which solves the technical problem that the existing automatic liquor picking based on machine vision only analyzes two-dimensional images, and the expression of wine flower information is not rich enough.

[0006] (II) Technical solutions

[0007] In order to achieve the above-mentioned purposes, the main technical solutions adopted by the present application include:

[0008] In a first aspect, the present application provides a liquor picking method based on three-dimensional reconstruction of a binocular camera, comprising:

[0009] Calibrating the binocular camera according to the pixel points in the obtained wine flower image to obtain the internal and external parameters of the binocular camera;

[0010] The epipolar lines on the beer flower images are obtained based on the internal and external parameters of the binocular camera, the beer flower images are transformed with the epipolar line alignment as a target, interpolation or resampling is performed on the transformed images, and the corrected images are obtained;

[0011] Stereo matching is performed on each two corrected images to obtain depth or three-dimensional information, and three-dimensional reconstruction of the original point cloud is performed according to the depth or three-dimensional information;

[0012] The registration and nearest neighbor search of all point clouds in the original point cloud with the reference point cloud are performed multiple times until a stop criterion is met to stop iteration, and in the iteration process, the transformation parameters of the original point cloud are continuously optimized by minimizing the error between the matching points obtained in the nearest neighbor search to obtain the best point cloud;

[0013] The liquid surface gully information is obtained according to the best point cloud and the pose information of the binocular camera;

[0014] The bubble region is segmented on the best point cloud according to the liquid surface gully information, the bubble information is calculated on the bubble region, and then the quality grade of the current beer is determined based on the bubble information.

[0015] Optionally, obtaining the epipolar lines on the beer flower images based on the internal and external parameters of the binocular camera, transforming the beer flower images with the epipolar line alignment as a target, and then performing interpolation or resampling on the transformed images to obtain corrected images comprises:

[0016] The epipolar lines corresponding to each pixel point on the beer flower images are obtained based on the internal and external parameters of the binocular camera;

[0017] The image transformation of the beer flower image includes translation and rotation transformation, so that all epipolar lines are aligned on the same row or the same column;

[0018] Interpolation or resampling is performed on the transformed images to obtain corrected images.

[0019] Optionally, stereo matching is performed on each two corrected images to obtain depth or three-dimensional information, and three-dimensional reconstruction of the original point cloud is performed according to the depth or three-dimensional information, which comprises:

[0020] Corresponding pixel points are found from each two corrected images, and the depth or disparity between the corresponding pixel points is determined;

[0021] The disparity between the corresponding pixel points is optimized by a preset disparity optimization function, the original point cloud is three-dimensionally reconstructed according to the optimized depth, and noise points, mismatched points and NaN points generated in the point cloud three-dimensional reconstruction process are processed accordingly;

[0022] The disparity optimization function is:

[0023]

[0024] In the above formula, E(d) represents an energy function of the disparity map d, N p is a neighborhood of p, q is a pixel point in N p except p, C(p, d p ) is a matching cost function of the pixel point p in the disparity map d p within its disparity range, the second term on the right side of the equation is d q is a pixel point q within its disparity range, P1 is a penalty factor, and T is a logical judgment function, which takes a value of 1 if |d p -d q | = 1, and 0 otherwise; P2 is also a penalty factor, and the corresponding T function takes a value of 1 if |d p -d q | > 1, and 0 otherwise.

[0025] Optionally, the registration and the nearest neighbor search of all point clouds in the original point cloud with the reference point cloud are performed multiple times until a stop criterion is met to stop iteration, and in the iteration process, the transformation parameters of the original point cloud are continuously optimized by minimizing the error between the matching points obtained in the nearest neighbor search to obtain the best point cloud, including:

[0026] A point cloud is selected from the original point cloud as a reference point cloud, and other point clouds are initially aligned with the reference point cloud;

[0027] The nearest corresponding point of each point in the initially aligned point cloud in the reference point cloud is found, the error between each group of point pairs is calculated, and the transformation parameters of the original point cloud are continuously adjusted to minimize the error;

[0028] The nearest neighbor search and registration steps are repeated until a stop criterion is met, and in the iteration process, the transformation parameters are continuously optimized until the best point cloud is obtained.

[0029] Optionally, according to the best point cloud and the pose information of the binocular camera, the liquid surface groove information is obtained, including:

[0030] According to the best point cloud and the pose information of the binocular camera, a plurality of inliers are randomly selected from the best point cloud to form a candidate liquid surface based on camera coordinates, and whether other points are consistent with the candidate liquid surface is determined by combining a set threshold, and through an iterative process, a liquid surface with the most inliers is found;

[0031] Based on the liquid surface with the most inliers, other inliers are analyzed to obtain the liquid surface groove information.

[0032] Optionally, the bubble region is segmented on the optimal point cloud according to the liquid surface groove information, bubble information is obtained on the bubble region, and the quality grade of the current wine is determined based on the bubble information, including:

[0033] The normal of each liquid surface position point is obtained by using a normal estimation method combined with the liquid surface groove information, and the curvature information of the point cloud is obtained according to the normal of each liquid surface position point.

[0034] The bubble region and the non-bubble region are segmented on the optimal point cloud according to a pre-set threshold of the curvature.

[0035] The volume of each bubble in the segmented bubble region is obtained by integrating the liquid surface plane to the bubble surface.

[0036] The bubble information including the volume of each bubble, the total volume of the bubbles, the number of bubbles with different volumes, and the proportion of bubbles with different volumes is obtained by counting the bubbles.

[0037] The quality grade of the current wine is determined based on the bubble information.

[0038] Optionally, the quality grade of the current wine is determined based on the bubble information, including:

[0039] When the total volume of the bubbles is less than a threshold V0, the current wine segment is determined to be tail water.

[0040] When the total volume of the bubbles is greater than the threshold V0 and less than V1, the current wine segment is determined to be head wine.

[0041] When the timing is not earlier than the head wine and the total volume of the bubbles is greater than the threshold V1, the current wine segment is determined to be second segment wine.

[0042] When the timing is not earlier than the second segment wine and the total volume of the bubbles is greater than the threshold V0 and less than the threshold V1, the current wine segment is determined to be third segment wine.

[0043] Wherein, V1>V0 and the value range of V1 / V0 is 1.3-4.

[0044] In a second aspect, an embodiment of the present application provides a wine picking system based on three-dimensional reconstruction of a binocular camera, including:

[0045] A camera calibration module is configured to calibrate the binocular camera according to the pixel points in the acquired wine flower image, and obtain the internal and external parameters of the binocular camera.

[0046] A epipolar line correction module is configured to obtain the epipolar line on the wine flower image based on the internal and external parameters of the binocular camera, transform the wine flower image by taking the epipolar line alignment as the target, and perform interpolation or resampling on the transformed image to obtain the corrected image.

[0047] a three-dimensional reconstruction module configured to perform stereo matching on each two of the corrected images to obtain depth or three-dimensional information, and perform three-dimensional reconstruction of the original point cloud according to the depth or three-dimensional information;

[0048] a registration and nearest neighbor search module configured to perform registration and nearest neighbor search of all point clouds in the original point cloud with the reference point cloud for multiple times until a stop criterion is met to stop iteration, while in the iteration process, transformation parameters of the original point cloud are constantly optimized by minimizing errors between matching points obtained in the nearest neighbor search to obtain an optimal point cloud;

[0049] a liquid surface groove obtaining module configured to obtain liquid surface groove information according to the optimal point cloud and pose information of the binocular camera;

[0050] a point cloud segmentation module configured to segment a bubble region on the optimal point cloud according to the liquid surface groove information, and obtain bubble information on the bubble region;

[0051] a wine quality determination module configured to determine a quality grade of the current wine based on the bubble information.

[0052] In a third aspect, an embodiment of the present application provides a wine picking device based on three-dimensional reconstruction of a binocular camera, comprising: a wine picking container; a binocular camera; at least one database; and a memory in communication connection with the at least one database; wherein the memory stores instructions executable by the at least one database, and the instructions are executed by the at least one database to enable the at least one database to execute the wine picking method based on three-dimensional reconstruction of a binocular camera as described above.

[0053] In a fourth aspect, an embodiment of the present application provides a computer readable medium having computer executable instructions stored thereon, and the executable instructions are executed by a processor to implement the wine picking method based on three-dimensional reconstruction of a binocular camera as described above.

[0054] (III) Advantages

[0055] The present application discloses a wine picking method using a binocular camera to perform three-dimensional reconstruction of wine flowers and analyze bubble sizes of the wine flowers, which performs stereo matching on two pictures captured by the binocular camera to obtain three-dimensional reconstruction point clouds of a wine bowl liquid surface, the point cloud construction method is simple and flexible, the reconstruction speed is fast, the spatial information of the stereo images is fully considered, the expressed wine flower information is more abundant, high-quality data is provided for wine quality analysis, each bubble volume information is fully estimated, and segmented wine picking is realized according to different grades and a pre-set automatic wine picking control system, so that the intelligentization is realized by replacing manual work. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1A flowchart of a picking wine process based on three-dimensional reconstruction of a binocular camera is provided for an embodiment of the present application.

[0057] Figure 2 A flowchart of an overall picking wine process based on three-dimensional reconstruction of a binocular camera is provided for an embodiment of the present application.

[0058] Figure 3 A flowchart of a specific process of step S2 of the picking wine process based on three-dimensional reconstruction of a binocular camera is provided for an embodiment of the present application.

[0059] Figure 4 A flowchart of a specific process of step S3 of the picking wine process based on three-dimensional reconstruction of a binocular camera is provided for an embodiment of the present application.

[0060] Figure 5 A flowchart of a specific process of step S4 of the picking wine process based on three-dimensional reconstruction of a binocular camera is provided for an embodiment of the present application.

[0061] Figure 6 A flowchart of a specific process of step S6 of the picking wine process based on three-dimensional reconstruction of a binocular camera is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to better explain the present application so as to be understood, the present application is described in detail below through specific embodiments in combination with the accompanying drawings.

[0063] As shown in Figure 1 and Figure 2 , the picking wine method based on three-dimensional reconstruction of a binocular camera proposed by the embodiment of the present application comprises: calibrating a binocular camera according to pixel points in an acquired hop image to obtain internal and external parameters of the binocular camera; obtaining epipolar lines on the hop image based on the internal and external parameters of the binocular camera, transforming the hop image with the epipolar line alignment as a target, and then interpolating or resampling the transformed image to obtain a corrected image; performing stereo matching on every two corrected images to obtain depth or three-dimensional information, performing three-dimensional reconstruction on original point clouds according to the depth or three-dimensional information; performing registration and nearest neighbor searching on all point clouds in the original point clouds with reference point clouds for multiple times until a stop criterion is met to stop iteration, and in the iteration process, continuously optimizing transformation parameters of the original point clouds by minimizing errors between matching points obtained in the nearest neighbor searching to obtain optimal point clouds; obtaining liquid surface gully information according to the optimal point clouds and pose information of the binocular camera; segmenting a bubble region on the optimal point clouds according to the liquid surface gully information, calculating bubble information on the bubble region, and then determining a quality grade of the current wine based on the bubble information.

[0064] The application discloses a wine picking method using a binocular camera to perform three-dimensional reconstruction on wine flowers and analyze bubble sizes, which performs stereo matching on two pictures captured by the binocular camera, obtains a three-dimensional reconstruction point cloud of a wine bowl liquid surface, and is simple and flexible in point cloud construction method, fast in reconstruction speed, fully considers spatial information of the stereo pictures, makes wine flower information expressed more rich, provides high-quality data for wine quality analysis, fully estimates each bubble volume information, and then can realize segmented wine picking according to different grades and a pre-set wine picking automatic control system, and replaces manual work to realize intelligentization.

[0065] In order to better understand the above technical solutions, the exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided so that the present application can be more clearly, thoroughly understood, and the scope of the present application can be completely conveyed to those skilled in the art.

[0066] Specifically, the present application provides a wine picking method based on three-dimensional reconstruction of a binocular camera, which comprises:

[0067] S1, calibrating the binocular camera according to the pixel points in the obtained wine flower image, to obtain the internal and external parameters of the binocular camera.

[0068] In specific embodiments, two images I1 and I2 of the binocular camera are obtained. The pixel points in I1 and I2 are extracted, and the camera internal parameter M1 and external parameter M2 are calculated according to the following formula:

[0069]

[0070] In the above formula, [X c , Y c , Z c , 1] T is the camera coordinate system, [X W , Y W , Z W , 1] T is the world coordinate system, (u, v) T is the pixel coordinate system, and R and T are rotation and translation parameter matrices respectively.

[0071] S2, obtaining the epipolar line on the wine flower image based on the internal and external parameters of the binocular camera, transforming the wine flower image with the epipolar line alignment as the target, and then performing interpolation or resampling on the transformed image to obtain the corrected image.

[0072] Further, as shown in Figure 3 , step S2 comprises:

[0073] S21, obtain the epipolar line corresponding to each pixel point on the hop cone image based on the internal and external parameters of the binocular camera.

[0074] S22, transform the hop cone image to an image containing translation and rotation transformation, so that all epipolar lines are aligned on the same row or column.

[0075] S23, interpolate or resample the transformed image to obtain the corrected image.

[0076] Epipolar line correction is an important step in stereo vision, used for preprocessing stereo images to simplify the stereo matching process. Its main purpose is to align the epipolar lines of two stereo images to the same row or column, thereby converting the stereo matching problem into a one-dimensional search problem, reducing the amount of calculation and improving the accuracy of matching.

[0077] S3, stereo matching is performed on each two corrected images to obtain depth or three-dimensional information, and three-dimensional reconstruction of the original point cloud is performed according to the depth or three-dimensional information.

[0078] Further, as shown in Figure 4 , step S3 includes:

[0079] S31, find corresponding pixel points from each two corrected images and determine the depth or disparity between the corresponding pixel points.

[0080] S32, optimize the disparity between the corresponding pixel points by a pre-set disparity optimization function, perform three-dimensional reconstruction of the original point cloud according to the optimized depth, and perform corresponding processing on the noise points, mismatched points and NaN points generated in the three-dimensional reconstruction process of the point cloud. Among them, the noise points, mismatched points and NaN points generated in the three-dimensional reconstruction process of the point cloud are preprocessed, and the main methods include straight line filtering, voxel filtering, statistical filtering, etc.

[0081] Wherein, the disparity optimization function is:

[0082]

[0083] In the above formula, E(d) represents the energy function of the disparity map d, N p is the neighborhood of p, q is a pixel point other than p in N p , C(p, d p ) is the matching cost function of pixel point p in the disparity map d p within its disparity range, d q in the second term on the right side of the equality is the disparity map within its disparity range, P1 is a penalty factor, and T is a logical judgment function. p -d q| = 1, T = 1, otherwise T = 0; P2 is also a penalty factor, and the corresponding T function is T = 1 when |d p -d q | = 1, T = 1, otherwise T = 0; P2 is also a penalty factor, and the corresponding T function is T = 1 when |d

[0084] S4, repeatedly performing the registration and nearest neighbor search of all point clouds in the original point cloud with the reference point cloud until the stopping criterion is met to stop iteration, and in the iteration process, the transformation parameters of the original point cloud are continuously optimized by minimizing the error between the matching points obtained in the nearest neighbor search to obtain the best point cloud.

[0085] Further, as shown in Figure 5 , step S4 includes:

[0086] S41, selecting one point cloud from the original point cloud as a reference point cloud, and initially aligning other point clouds with the reference point cloud.

[0087] S42, finding the nearest corresponding point of each point in the initially aligned point cloud in the reference point cloud, calculating the error between each group of point pairs, and continuously adjusting the transformation parameters of the original point cloud to minimize the error.

[0088] S43, repeatedly performing the nearest neighbor search and registration steps until the stopping criterion is met, and in the iteration process, continuously optimizing the transformation parameters until the best point cloud is obtained.

[0089] In the above steps, fine registration is first performed; first, one point cloud is selected as a reference, and other point clouds are initially roughly aligned; then, nearest neighbor search is performed: for each point in the point cloud, the nearest corresponding point in the target point cloud is found. By calculating the error between the point pairs and using an optimization algorithm (such as least squares) to adjust the transformation matrix, the error is minimized. Then, the nearest neighbor search and registration steps are repeated until the stopping criterion is met (such as reaching the maximum number of iterations or the error being less than a threshold). And in the iteration process, the transformation matrix is continuously optimized until the best alignment result is obtained.

[0090] S5, obtaining the liquid surface groove information according to the best point cloud and the pose information of the binocular camera.

[0091] Further, step S5 includes: according to the best point cloud and the pose information of the binocular camera, randomly selecting a plurality of inliers from the best point cloud based on the camera coordinates to form a candidate liquid surface, and combining the set threshold to judge whether other points are consistent with the candidate liquid surface, and continuously finding the liquid surface with the most inliers through the iteration process; based on the liquid surface with the most inliers, analyzing other inliers to obtain liquid surface groove information containing the position information of a plurality of non-liquid surface points along the vertical direction.

[0092] Therefore, the liquid surface coordinates can be estimated by the point cloud in the camera coordinate system, and vice versa, the camera pose can be deduced in the world coordinate system of the liquid surface.

[0093] S6, segmenting the bubble region on the optimal point cloud according to the liquid surface groove information, calculating the bubble information on the bubble region, and determining the quality grade of the current wine based on the bubble information.

[0094] Further, as shown in Figure 6 S6 includes:

[0095] S61, calculating the normal of each liquid surface position point using the normal estimation method combined with the liquid surface groove information, and obtaining the curvature information of the point cloud according to the normal of each liquid surface position point.

[0096] S62, segmenting the bubble region and the non-bubble region on the optimal point cloud according to the pre-set threshold of the curvature.

[0097] According to the groove between the bubbles, the point cloud is segmented, which can be specifically segmented based on geometric features, segmented based on clustering, or segmented based on model fitting. A common method is to use normal vector estimation to judge the curvature of the point cloud, and then detect the spherical region according to the change of the curvature: using the normal estimation method (such as the least square fitting normal) to calculate the normal of each point, and then calculating the curvature. On the surface of the sphere, the curvature value is relatively high, and the spherical region can be separated from other regions according to the threshold of the curvature.

[0098] S63, calculating the volume of each bubble in the segmented bubble region by integrating the liquid surface plane to the bubble surface.

[0099] S64, obtaining the bubble information including the volume of each bubble, the total volume of the bubbles, the number of bubbles with different volumes, and the proportion of bubbles with different volumes by counting the bubbles.

[0100] S65, determining the quality grade of the current wine based on the bubble information.

[0101] Further, determining the quality grade of the current wine based on the bubble information includes:

[0102] When the total volume of the bubbles is less than the threshold V0, it is determined that the current wine segment is tail water.

[0103] When the total volume of the bubbles is greater than the threshold V0 and less than V1, it is determined that the current wine segment is head wine.

[0104] When the timing is not earlier than the head wine and the total volume of the bubbles is greater than the threshold V1, it is determined that the current wine segment is the second segment of wine.

[0105] When the timing is not earlier than the second wine segment, and the total volume of the bubbles is greater than a threshold V0 and less than a threshold V1, the current wine segment is determined as the third wine segment.

[0106] V1 / V0 is in a range of 1.3 to 4.

[0107] In addition, the embodiment of the present application also provides a wine picking system based on three-dimensional reconstruction of a binocular camera, which comprises:

[0108] A camera calibration module is configured to calibrate the binocular camera according to the pixel points in the acquired hop image, and obtain the internal and external parameters of the binocular camera.

[0109] A epipolar rectification module is configured to obtain the epipolar line on the hop image based on the internal and external parameters of the binocular camera, transform the hop image by taking the epipolar line alignment as a target, and then perform interpolation or resampling on the transformed image to obtain the rectified image.

[0110] A three-dimensional reconstruction module is configured to perform stereo matching on each two rectified images to obtain depth or three-dimensional information, and perform three-dimensional reconstruction on the original point cloud according to the depth or three-dimensional information.

[0111] A registration and nearest neighbor searching module is configured to perform registration and nearest neighbor searching on all point clouds in the original point cloud with reference point clouds for multiple times until a stop criterion is met to stop iteration, and in the iteration process, the transformation parameters of the original point cloud are constantly optimized by minimizing the error between the matching points obtained in the nearest neighbor searching to obtain the best point cloud.

[0112] A liquid surface gully obtaining module is configured to obtain the liquid surface gully information according to the best point cloud and the pose information of the binocular camera.

[0113] A point cloud segmentation module is configured to segment the bubble region on the best point cloud according to the liquid surface gully information, and obtain the bubble information on the bubble region.

[0114] A wine quality determination module is configured to determine the quality grade of the current wine based on the bubble information.

[0115] Meanwhile, the embodiment of the present application also provides a wine picking device based on three-dimensional reconstruction of a binocular camera, which comprises: a wine picking container; a binocular camera; at least one database; and a memory in communication connection with the at least one database; wherein the memory stores instructions executable by the at least one database, and the instructions are executed by the at least one database to enable the at least one database to execute the wine picking method based on three-dimensional reconstruction of a binocular camera as described above.

[0116] Also, the embodiment of the present application further provides a computer readable medium, which stores computer executable instructions, and the executable instructions are executed by a processor to realize the method for picking up wine based on three-dimensional reconstruction of binocular camera as described above.

[0117] To sum up, the present application provides a method, system, device and medium for picking up wine based on three-dimensional reconstruction of binocular camera, which realizes "looking at flowers to pick up wine" by using machine vision technology, integrates artificial experience into the classification model, analyzes the volume and shape of hops to determine different stages of picking up wine, replaces manual operation to achieve accurate, robust and real-time picking up of wine, and feeds back to the production equipment to execute the corresponding production strategy in time. It is worth mentioning that the scheme in the present application realizes rapid identification at a second level.

[0118] Since the system / device described in the above embodiments of the present application is the system / device used for implementing the method of the above embodiments of the present application, the specific structure and modification of the system / device can be understood by those skilled in the art based on the method described in the above embodiments of the present application, and thus will not be described here. Any system / device used by the method of the above embodiments of the present application belongs to the scope of the present application.

[0119] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0120] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system) and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions.

[0121] It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in the claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the claims, the word "comprising" does not exclude the presence of other elements or steps than those listed in the claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The word "first", "second", "third", etc. does not imply any order. The use of these words is to be construed as being the use of these words in their "ordinal" sense and not to be construed as an order.

[0122] Furthermore, it is to be understood that the use of "a" or "an", "the" or "said" employed throughout the present description denotes a non-limiting inclusion of one or more of the referenced features or steps. Any reference to prior art in this description is not to be understood as an admission that it was widely known or formed part of the common general knowledge in any jurisdiction as at the priority date of the application.

[0123] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the preferred embodiments without departing from the spirit and scope of the application. Therefore, it should be understood that the application is not limited by the preferred embodiments and many variations and modifications can be made. It is also possible to use some of the features of the present application in combinations with features of others without the loss of their distinct properties. In addition, many modifications can be made to adapt a particular situation or material to the teaching of the application without departing from its scope. Therefore, it is intended that the application not be limited to the preferred embodiments disclosed in the present description, but that the application can include all embodiments falling within the scope of the application.

[0124] Obviously, many modifications and variations of the present application are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A method for picking up wine based on three-dimensional reconstruction of binocular cameras, characterized in that, include: The binocular camera is calibrated based on the pixels in the acquired hop images to obtain the intrinsic and extrinsic parameters of the binocular camera. The epipolar lines on the hop image are obtained based on the intrinsic and extrinsic parameters of the binocular camera. The hop image is transformed with the goal of epipolar line alignment. Then, the transformed image is interpolated or resampled to obtain the corrected image. Stereo matching is performed on every two corrected images to obtain depth or 3D information, and the original point cloud is reconstructed in 3D based on the depth or 3D information. The process involves repeatedly registering all points in the original point cloud with the reference point cloud and performing nearest neighbor searches until the stopping criterion is met. During this iteration, the transformation parameters of the original point cloud are continuously optimized by minimizing the error between the matching points obtained in the nearest neighbor search, in order to obtain the optimal point cloud. Based on the optimal point cloud and the pose information of the binocular camera, the information of the grooves on the liquid surface is obtained; Based on the information about the grooves on the liquid surface, the bubble region is segmented on the optimal point cloud. Bubble information is then obtained from the bubble region, and the quality grade of the wine is determined based on the bubble information.

2. The binocular camera based three-dimensional reconstruction based wine picking method of claim 1, wherein, The epipolar lines on the hop image are obtained based on the intrinsic and extrinsic parameters of the stereo camera. The hop image is then transformed with the goal of epipolar line alignment. The transformed image is then interpolated or resampled to obtain the corrected image, which includes: The epipolar line corresponding to each pixel in the hop image is obtained based on the intrinsic and extrinsic parameters of the binocular camera. The image of the hops image is transformed by translation and rotation so that all epipolar lines are aligned in the same row or column. The transformed image is interpolated or resampled to obtain the corrected image. 3.The binocular camera based three-dimensional reconstruction method of wine picking according to claim 1, wherein, Stereo matching is performed on every two corrected images to obtain depth or 3D information. Based on the depth or 3D information, the original point cloud is reconstructed in 3D, including: Find the corresponding pixels in every two corrected images and determine the depth or parallax between the corresponding pixels; The parallax between corresponding pixels is optimized by a preset parallax optimization function. The original point cloud is reconstructed in 3D based on the optimized depth. Noise points, mismatched points and NaN points generated during the 3D reconstruction of the point cloud are processed accordingly. The disparity optimization function is: In the above formula, E(d) represents the energy function of the disparity map d, N p is the neighborhood of p, q is a pixel point other than p in N p , C(p, d p ) is the matching cost function of the pixel point p in the disparity map d p within its disparity range, d q in the second term on the right side of the equation is the disparity map within the disparity range of the point q, P1 is a penalty factor, and T is a logical judgment function, which is 1 if |d p -d q | = 1, and 0 otherwise; P2 is also a penalty factor, and the corresponding T function is 1 if |d p -d q | > 1, and 0 otherwise. 4.The binocular camera based three-dimensional reconstruction method of wine picking according to claim 1, wherein, The process involves repeatedly registering all points in the original point cloud with the reference point cloud and performing nearest neighbor searches until the stopping criterion is met. During this iteration, the transformation parameters of the original point cloud are continuously optimized by minimizing the error between matching points obtained in the nearest neighbor search, in order to obtain the optimal point cloud, including: Select a point cloud from the original point cloud as a reference point cloud, and perform initial alignment of the other point clouds with reference to the reference point cloud; Find the nearest corresponding point in the reference point cloud for each point in the initially aligned point cloud, calculate the error between each pair of points, and continuously adjust the transformation parameters of the original point cloud to minimize the error. Repeat the nearest neighbor search and registration steps until the stopping criterion is met. During this iteration, continuously optimize the transformation parameters until the optimal point cloud is obtained. 5.The binocular camera based three-dimensional reconstruction method of wine picking according to claim 1, wherein, Based on the optimal point cloud and the pose information of the stereo camera, the information on the grooves in the liquid surface is obtained, including: A candidate liquid surface is formed from a number of random inliers in the best point cloud based on the pose information of the best point cloud and the binocular camera, and a threshold is set to determine whether other points are consistent with the candidate liquid surface, and the liquid surface with the most inliers is found through an iterative process; Based on the liquid surface with the most inliers, the other inliers are analyzed to obtain the liquid surface gully information.

6. The binocular camera based three-dimensional reconstruction based wine picking method according to any one of claims 1-5, wherein, According to the liquid surface gully information, the bubble region is segmented on the best point cloud, the bubble information is obtained on the bubble region, and then the quality grade of the current wine is determined based on the bubble information, including: The normal estimation method is used to obtain the normal of each liquid surface position point combined with the liquid surface gully information, and the curvature information of the point cloud is obtained according to the normal of each liquid surface position point; According to the pre-set threshold of the curvature, the bubble region and the non-bubble region are segmented on the best point cloud; The volume of each bubble in the segmented bubble region is obtained by integrating the liquid surface to the bubble surface; Through the statistics of the bubbles, the bubble information including the volume of each bubble, the total volume of the bubbles, the number of bubbles with different volumes, and the proportion of bubbles with different volumes is obtained; The quality grade of the current wine is determined based on the bubble information.

7. The binocular camera based three-dimensional reconstruction based wine picking method of claim 6, wherein, The quality grade of the current wine is determined based on the bubble information, including: When the total volume of the bubbles is less than a threshold V0, the current wine segment is determined to be tail water; When the total volume of the bubbles is greater than the threshold V0 and less than V1, the current wine segment is determined to be head wine; When the timing is not earlier than the head wine and the total volume of the bubbles is greater than the threshold V1, the current wine segment is determined to be second segment wine; When the timing is not earlier than the second segment wine and the total volume of the bubbles is greater than the threshold V0 and less than the threshold V1, the current wine segment is determined to be third segment wine; Wherein, V1>V0 and V1 / V0 takes the value range of 1.3-4.

8. A bin picking system based on three-dimensional reconstruction of a stereo camera, characterized in that, Including: A camera calibration module is configured to calibrate the binocular camera based on the pixel points in the acquired wine image to obtain the internal and external parameters of the binocular camera; A epipolar rectification module is configured to obtain the epipolar line on the wine image based on the internal and external parameters of the binocular camera, to transform the wine image with the epipolar line alignment as the target, and to perform interpolation or resampling on the transformed image to obtain the rectified image; A three-dimensional reconstruction module is configured to perform stereo matching on each two rectified images to obtain depth or three-dimensional information, and to perform three-dimensional reconstruction on the original point cloud based on the depth or three-dimensional information; A registration and nearest neighbor search module is configured to perform registration and nearest neighbor search on all point clouds in the original point cloud with a reference point cloud multiple times until a stop criterion is met to stop iteration, and in the iteration process, the transformation parameters of the original point cloud are continuously optimized by minimizing the error between the matching points obtained in the nearest neighbor search to obtain the best point cloud; A liquid surface gully obtaining module is configured to obtain the liquid surface gully information based on the best point cloud and the pose information of the binocular camera; A point cloud segmentation module is configured to segment the bubble region on the best point cloud based on the liquid surface gully information, and to obtain the bubble information on the bubble region; A wine quality determination module is configured to determine the quality grade of the current wine based on the bubble information.

9. A grape picking apparatus based on three-dimensional reconstruction of binocular cameras, characterized in that, Including: A wine extraction container; A binocular camera; At least one database; And A memory in communication connection with the at least one database; The memory stores instructions executable by the at least one database, and the instructions are executed by the at least one database to enable the at least one database to perform the method for three-dimensional reconstruction of wine picking based on a binocular camera according to any one of claims 1-7.

10. A computer readable medium having stored thereon computer- executable instructions, characterized in that, The executable instructions are executed by the processor to implement the method for three-dimensional reconstruction of wine picking based on a binocular camera according to any one of claims 1-7.

Images