Method and system for measuring volume of dry bulk cargo pile using yard camera

Abstract

The present invention discloses a method and system for measuring the volume of a dry bulk cargo pile using a yard camera, the method comprising the steps of: installing a high-definition camera around the yard or on a mobile operating device to collect yard image data; data preprocessing; image matching based on an improved SIFT algorithm; reconstructing the three-dimensional structure of the yard; wherein the improved SIFT algorithm specifically connects each rough matching pair after performing rough matching of feature points according to the Euclidean distance during feature point matching, and removes the wrong matching pairs according to the comparison of the angle between the connecting lines with the preset angle threshold, obtains the feature point matching pairs after precise matching, and then performs image matching; segmenting and extracting the point cloud data of the pile, and converting it into a triangular network model, and using a definite integral with the yard ground as the bottom surface to calculate the volume of the pile. The present invention can be specifically applied to large-scale dry bulk cargo distribution ports or warehouses such as ports, power plants, and coal storage bases to achieve fast, accurate, and real-time acquisition of the volume of the pile of the yard.

Description

Technical Field

[0001] The present invention relates to the field of intelligent logistics, and in particular to a method and system for measuring the volume of a dry bulk cargo pile using a yard camera. Background Art

[0002] Modern dry bulk ports technically require that the measurement error of the volume of stockpile in the yard should not be higher than 0.5%, corresponding to the cargo damage rate acceptable to customers. It is also required that the volume measurement can be carried out in all weather conditions, including rain, fog, snow, and windy weather. The measurement frequency must be guaranteed to be no less than once a day, and the measurement time should be less than 30 minutes. When using drone solutions for measurement, due to the influence of flight altitude restrictions, errors, and image distortion caused by flight shaking, the measurement error is about 0.5%~5%, and the measurement time often exceeds 1 hour, which cannot well meet the technical needs of ports for the volume measurement of stockpile in the yard.

[0003] With the development of automation in dry bulk terminals, the traditional method of manually measuring the volume of stockpiles in the yard can no longer meet the actual needs of ports. The existing solution of using drones for measurement can achieve high-precision measurement of the volume of stockpiles in the yard of dry bulk terminals, but there are problems such as slow data acquisition, high technical requirements and difficulty in adapting to the changeable weather in ports. Summary of the invention

[0004] The main purpose of the present invention is to provide a method and system for measuring the volume of dry bulk cargo piles by using a yard camera, which can achieve high-precision measurement, and the data collection and calculation processes are highly efficient and are not affected by the changeable weather in the port.

[0005] The technical solution adopted by the present invention is:

[0006] A method for measuring the volume of a dry bulk cargo pile using a yard camera is provided, comprising the following steps:

[0007] S1. Install high-definition cameras on lighting poles or mobile operating equipment around the yard to collect yard image data;

[0008] S2, preprocessing the storage yard image data using histogram equalization and guided filtering;

[0009] S3. Based on the improved SIFT algorithm, feature point extraction, feature point matching and image pair geometric filtering are performed on the pre-processed image data to obtain image matching pairs; the external parameters of the camera are solved according to the image matching pairs, and the two-dimensional points of the yard image are converted into three-dimensional points of the yard in the world coordinate system in combination with the position and internal parameters of the camera in the yard, so as to generate yard point cloud data and reconstruct the three-dimensional structure of the yard; wherein, the improved SIFT algorithm is specifically that when matching feature points, after rough matching of feature points is performed according to the Euclidean distance, each rough matching pair is connected, and the wrong matching pairs are eliminated according to the comparison of the angle between the connecting lines with the preset angle threshold, so as to obtain the feature point matching pairs after precise matching, and then perform image pairing;

[0010] S4, using a plane segmentation algorithm to segment the point cloud of the reconstructed three-dimensional structure of the stockpile, and extracting the segmented point cloud data of the stockpile according to the stacking distribution characteristics of the stockpile;

[0011] S5. Convert the point cloud data of the stockpile into a triangulated network model, combine the positional relationship between the stockpile ground and the point cloud contour of the stockpile, and use a definite integral with the stockpile ground as the bottom surface to calculate the volume of the stockpile.

[0012] Following the above technical solution, in step S3, when the feature points are precisely matched, based on the fact that the connecting lines between the correct feature point matching pairs are approximately parallel, and based on the fact that the angle between the connecting lines of the incorrect feature point matching pairs and the correct feature point matching pairs is greater than a preset angle threshold, the feature point pairs with similar parallel relationships are clustered, and hierarchical clustering is performed by continuously increasing the connecting line angle condition to obtain the feature point matching pairs after precise matching.

[0013] Following the above technical solution, in step S3, when the image pairs are geometrically filtered, several groups of feature point pairs are randomly selected from each matching image pair, a basic matrix is ​​estimated using each group of feature point pairs, and the error values ​​of all feature point matching pairs in the matching image pair are calculated to obtain the error values ​​of the several groups, which are sorted, and the basic matrices corresponding to the 7 feature point matching pairs with the smallest error values ​​are selected as the final basic matrices. Finally, the reliability of the matching image pairs is judged to complete the geometric filtering of the image pairs.

[0014] Following the above technical solution, in step S3, the solution of the camera extrinsic parameters is based on the Euclidean structure recovery problem. Specifically, the normalized eight-point method and the PnP algorithm are used to estimate the basic matrix, and the basic matrix is ​​decomposed by SVD to obtain the rotation matrix and translation vector of the camera extrinsic parameters.

[0015] Following the above technical solution, when reconstructing the three-dimensional structure of the storage yard in step S3, the BA algorithm is specifically used to optimize the external and internal parameters of the camera. Every time the three-dimensional coordinates of a pair of feature points of a storage yard image are restored, a BA optimization is performed to incrementally realize the three-dimensional structure restoration.

[0016] Following the above technical solution, high-definition cameras of uniform models and fixed relative positions are installed on lighting poles around the storage yard.

[0017] Following the above technical solution, step S4 specifically includes the following steps:

[0018] S41. Assuming that the bottom surface of the stockpile is a plane, the plane cutting algorithm is used to fit the plane position of the stockpile, and the point cloud and three-dimensional plane parameters of the equipment, facilities, and stockpile are obtained;

[0019] S42. According to the distribution characteristics of the material piles in the stockpile yard, a density clustering algorithm is used to extract the point cloud data set of each material pile, and other point cloud set classes are discarded.

[0020] Following the above technical solution, the Statistical Outlier Removal, ACSegmentation, and Euclidean Cluster Extraction modules in the PCL library are used to implement stockpile point cloud filtering, plane segmentation, and stockpile clustering, respectively.

[0021] The present invention also provides a dry bulk cargo pile volume measurement system using a yard camera, comprising:

[0022] A data acquisition module is used to acquire yard image data collected by high-definition cameras installed on lighting poles around the yard or on mobile operating equipment;

[0023] A preprocessing module, used for preprocessing the storage yard image data by using histogram equalization and guided filtering;

[0024] The 3D reconstruction module of the storage yard is used to extract feature points, match feature points and perform geometric filtering on the pre-processed image data based on the improved SIFT algorithm to obtain image matching pairs; the external parameters of the camera are solved according to the image matching pairs, and the position and internal parameters of the camera in the storage yard are combined to convert the 2D points of the storage yard image into 3D points of the storage yard in the world coordinate system, generate storage yard point cloud data, and reconstruct the 3D structure of the storage yard; wherein, the improved SIFT algorithm is specifically used in matching feature points, after performing rough matching of feature points according to Euclidean distance, connecting each rough matching pair, and eliminating wrong matching pairs according to the comparison of the angle between the connecting lines with the preset angle threshold, to obtain the feature point matching pairs after precise matching, and then perform image matching;

[0025] The stockpile point cloud data extraction module is used to segment the point cloud of the reconstructed three-dimensional structure of the stockpile using a plane segmentation algorithm, and extract the segmented point cloud data of the stockpile according to the stacking distribution characteristics of the stockpile;

[0026] The stockpile volume calculation module is used to convert the point cloud data of the stockpile into a triangulated network model, combine the positional relationship between the stockpile ground and the point cloud contour of the stockpile, and use a definite integral with the stockpile ground as the bottom surface to calculate the stockpile volume.

[0027] The present invention also provides a computer storage medium, characterized in that a computer program executable by a processor is stored therein, and the computer program executes the method for measuring the volume of a dry bulk cargo pile using a yard camera as described in the above technical solution.

[0028] The beneficial effects of the present invention are as follows: the present invention combines the layout characteristics of the yard and the existing facilities and conditions, combines the existing monitoring camera equipment in the yard, uses the projection imaging principle and the three-dimensional reconstruction principle, establishes the corresponding relationship between the two-dimensional image and the three-dimensional object, uses the three-dimensional point cloud data of the stockpile to calculate the volume of the stockpile, and realizes the volume measurement of the stockpile of the dry bulk terminal yard based on the multi-view fusion of the cameras around the yard, so as to achieve the monitoring of the bulk volume at any time when the weather conditions permit, and provide basic data support for the intelligent dry bulk terminal; wherein, in particular, the specific position relationship between the yard cameras is translation or rotation on the horizontal plane, and the SIFT algorithm is optimized, and the wrong matching pairs can be quickly eliminated according to the angle relationship between the connecting lines of the wrong matching pairs and the correct matching pairs during the precise matching of the feature points, thereby improving the matching efficiency of the feature points; the present invention can quickly, accurately and real-time obtain the stockpile volume of the yard, so as to effectively grasp the storage status and utilization of the dry bulk yard, warehouse, etc., support the yard planning and stacking and reclaiming equipment regulation and control decision, improve the yard utilization rate, and reduce the operation preparation time of the stacking and reclaiming equipment such as the stacking and reclaiming machine. The present invention can be specifically applied to large-scale dry bulk cargo distribution ports or warehouses such as ports, power plants, and coal storage bases.

[0029] Furthermore, by replacing the image source with cameras of uniform model and fixed relative position on the yard facilities and operating equipment, the image acquisition source is more stable, which not only effectively reduces the distance between the camera and the pile and image distortion, and improves the image accuracy, but also can acquire images from all cameras at the same time, greatly improving the image acquisition speed.

[0030] Furthermore, when reconstructing the three-dimensional structure of the storage yard, the BA algorithm is used to optimize the camera's external and internal parameters. Every time the three-dimensional coordinates of a pair of feature points in a storage yard image are restored, a BA optimization is performed to incrementally restore the three-dimensional structure and improve the accuracy of three-dimensional reconstruction.

[0031] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 is a flow chart of a method for measuring the volume of a dry bulk cargo pile using a yard camera according to an embodiment of the present invention;

[0034] Figure 2 This is a schematic diagram of the camera layout of a dry bulk cargo terminal yard according to an embodiment of the present invention;

[0035] FIG3 (a) is a schematic diagram of a storage yard image collected according to an embodiment of the present invention;

[0036] FIG3( b ) is a schematic diagram of an embodiment of the present invention when performing precise matching of feature points after considering the positional relationship between cameras in the storage yard;

[0037] FIG4( a ) is a schematic diagram of an original image matching pair according to an embodiment of the present invention;

[0038] FIG4( b ) is a schematic diagram of the image matching pair after geometric filtering in FIG4( a );

[0039] Figure 5 3D reconstruction schematic diagram of a stockpile in a storage yard according to an embodiment of the present invention;

[0040] Figure 6 is a schematic diagram of point cloud plane segmentation according to an embodiment of the present invention;

[0041] Figure 7 Schematic diagram of hierarchical clustering of material pile point clouds according to stacking characteristics in an embodiment of the present invention;

[0042] Figure 8 is a schematic diagram of a target stockpile point cloud extracted by an embodiment of the present invention;

[0043] Fig. 9 is a schematic diagram of a triangular mesh model of a target point cloud according to an embodiment of the present invention;

[0044] Fig.10 It is a schematic diagram of a triangular prism formed by any triangular grid and the stockpile ground in the stockpile volume calculation principle of the embodiment of the present invention;

[0045] Fig.11 It is a schematic structural diagram of a dry bulk cargo pile volume measurement system using a yard camera according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0047] It should be noted that the illustrations provided in the embodiments of the present invention are only used to illustrate the basic concept of the present invention in a schematic manner. Therefore, the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout type may also be more complicated.

[0048] In the present invention, it is also necessary to explain that, if the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, if the terms "first" and "second" appear, they are only used for description and distinction purposes, and cannot be understood as indicating or implying relative importance.

[0049] In addition, it should be noted that the features of the various embodiments of the present invention may be combined or combined in part or in whole, and may interact and operate in different ways as will be appreciated by those skilled in the art. Each embodiment may be implemented independently of one another, or in an associated relationship.

[0050] like Figure 1 As shown, a method for measuring the volume of a dry bulk cargo pile using a yard camera according to an embodiment of the present invention comprises the following steps:

[0051] S1. Install high-definition cameras on lighting poles around the storage yard and collect image data of the storage yard through the high-definition cameras;

[0052] S2, preprocessing the storage yard image data using histogram equalization and guided filtering;

[0053] S3. Based on the improved SIFT algorithm, feature point extraction, feature point matching and image pair geometric filtering are performed on the pre-processed image data to obtain image matching pairs; the external parameters of the camera are solved according to the image matching pairs, and the two-dimensional points of the yard image are converted into three-dimensional points of the yard in the world coordinate system in combination with the position and internal parameters of the camera in the yard, so as to generate yard point cloud data and reconstruct the three-dimensional structure of the yard; wherein, the improved SIFT algorithm is specifically that when matching feature points, after rough matching of feature points is performed according to the Euclidean distance, each rough matching pair is connected, and the wrong matching pairs are eliminated according to the comparison of the angle between the connecting lines with the preset angle threshold, so as to obtain the feature point matching pairs after precise matching, and then perform image pairing;

[0054] S4, using a plane segmentation algorithm to segment the point cloud of the reconstructed three-dimensional structure of the stockpile, and extracting the segmented point cloud data of the stockpile according to the stacking distribution characteristics of the stockpile;

[0055] S5. Convert the point cloud data of the stockpile into a triangulated network model, combine the positional relationship between the stockpile ground and the point cloud contour of the stockpile, and use a definite integral with the stockpile ground as the bottom surface to calculate the volume of the stockpile.

[0056] The specific steps of step S1 are as follows: Figure 2 As shown, in the bulk cargo terminal yard, lighting poles are installed around the yard, and high-definition cameras are installed on the poles. Mobile operating equipment in the yard can also be equipped with cameras. All cameras are fixed on facilities and equipment, and the camera posture can be adjusted and the camera images of fixed facilities and mobile equipment in the yard can be retrieved at any time. For the cameras installed in the yard, the camera models and some relative position relationships are known. The precise relative position relationship can be determined using surveying and mapping equipment such as total stations, and the rough estimate of the position relationship can be converted based on the Miller projection using the latitude, longitude and altitude of the EXIF ​​information of the camera photos.

[0057] Step S2 may specifically include the following steps:

[0058] S21, determining an input image and a guide image, wherein the input image is the image to be processed P, the yard camera image after histogram equalization (i.e., the original image) is used, and the guide image I uses the original image;

[0059] The specific implementation process of discrete histogram equalization is as follows: Assume that the number of pixels of an image is , a total of Gray levels, Represents the gray level The number of pixels, then The probability of a gray level appearing can be expressed as follows:

[0060]

[0061] Substitute the above formula into the change formula of image histogram equalization , the change formula is as follows:

[0062]

[0063] S22, calculate , The mean value at the filter window position is calculated The autocorrelation coefficient, and The correlation coefficient of

[0064] S23. Utilization , The mean and correlation coefficient of The variance of and The covariance of

[0065] S24. Use The variance of and The covariance and penalty value of , calculate the filter parameters , ;

[0066] in, Pixels representing the guide map , Represents the pixel to be processed ， Represents the pixel of the output image corresponding to the guide image , , are the parameters of the linear model, Contains pixels , And in pixels The center radius is All square filter kernels of yes The number of pixels of the window, and It is a guide map exist The mean and variance corresponding to the region, Is the image to be processed P The mean value of the pixels at the corresponding position of the filter kernel, ,parameter , The solution formula is as follows:

[0067]

[0068]

[0069] S25, repeat the above steps to calculate the parameters of all filter windows containing a certain pixel , , and finally calculated , ;

[0070] in, Is the image to be processed The mean value of the pixels at the corresponding position in the filter window, ; but the same pixels Multiple overlapping filter windows will appear. So The values ​​in different windows will be different. The guided filtering algorithm uses the mean strategy to calculate the average value of all Average to get the true output pixel value. However, in the actual algorithm implementation, consider simplifying the algorithm process and The model parameters of all filters , Average, first calculate the pixels of the guide map and the output map The numerical relationship of , and finally calculate the pixel value of the output image, the formula is as follows:

[0071]

[0072]

[0073] S26, according to the filter model, calculate the pixel value after filtering , the formula is as follows:

[0074]

[0075] S27, repeat the process to filter all pixels, and finally obtain the filtered output image .

[0076] In one embodiment of the present invention, the number of image matching pairs using the guided filtering algorithm is 1738, the number of matching pairs after RANSANC processing is 1550, the effective matching pairs account for 89.18%, and the number of unmatched feature points is 6775. The effective matching pairs using the Gaussian filtering + histogram equalization theory account for 87.59%, and the number of unmatched feature points is 7308. It can be seen that the number of unmatched feature points is less than that of the image using Gaussian filtering + histogram equalization, and there is no incorrect matching.

[0077] In step S3, when the feature points are precisely matched, the connecting lines between the correct feature point matching pairs are approximately parallel, and the angle between the connecting lines of the incorrect feature point matching pairs and the correct feature point matching pairs is greater than the preset angle threshold. The feature point pairs with similar parallel relationships are clustered, and hierarchical clustering is performed by continuously increasing the connecting line angle condition to obtain the feature point matching pairs after precise matching. According to the distribution and movement rules of cameras in special yard scenes, the precise matching in the SIFT algorithm (Scale-Invariant Feature Transform) is improved and optimized to make the calculation faster and more efficient.

[0078] Furthermore, step S3 specifically includes the following steps:

[0079] S31. Extraction and matching of port yard feature points. Based on the improved SIFT algorithm, the corresponding relationship between the pixel points with significant features in each image and the three-dimensional yard is obtained. ; The specific SIFT algorithm is implemented using the functions provided by openMVG, including feature point extraction, feature point matching and image pair geometric filtering. When using the openMVG library, select the function interface and input parameters according to the specific algorithm flow requirements to complete the SIFT algorithm program writing. SIFT feature point extraction requires the generation of Gaussian pyramids, DOG pyramids, extreme point detection, key point determination and descriptor generation. The feature point matching algorithm first uses the hash table data structure to quickly obtain a set of feature points with similar descriptors to the feature points in the real-time image, and then calculates the Euclidean distance between the descriptors to achieve rough matching of feature points. Considering the positional relationship between the cameras in the yard, which is translation and rotation on the horizontal plane, the connecting lines between the correct matching pairs should be parallel, and the connecting lines of the wrong matching pairs intersect with the correct matching pairs and the angle (10-45°) is large. Based on the positional relationship between the matching pairs, hierarchical clustering is used to eliminate wrong matches and complete accurate feature point matching. As shown in Figure 3 (a), it is a collected yard image, and Figure 3 (b) is a schematic diagram of feature point precise matching after considering the positional relationship between the cameras in the yard.

[0080] The basic principle of image pair geometric filtering is: by randomly extracting several groups of feature point pairs from each yard image pair, each group has 7 feature point pairs, using each group of point pairs to estimate a basic matrix, and using the model to calculate the error values ​​of all feature point matching pairs in the image pair, the error values ​​of several groups can be obtained, sorted, and the basic matrices corresponding to the 7 feature point matching pairs with the smallest error values ​​are selected as the final basic matrix. Finally, the reliability of the image pair is judged to complete the geometric filtering of the image pair. As shown in Figure 4 (a), the original image matching pair and Figure 4 (b), the image matching pair after geometric filtering, can be seen that the number of image pairs is significantly reduced, which greatly improves the speed of sparse reconstruction without affecting the reconstruction accuracy.

[0081] S32, camera internal parameter solution, internal parameter By focal length , sensor element pixel size and the pixel coordinates of the center of the image Determine, the camera internal parameter formula is:

[0082] =

[0083] In the actual investigation of the port, it was found that the surveillance cameras in the yard are generally of the same model, and the camera internal parameters are the same. If the image is scaled, the internal parameter matrix In and will change in inverse proportion, and Direct proportional changes.

[0084] S33, camera extrinsic parameter solution. The solution of camera extrinsic parameters is based on the Euclidean structure recovery problem. The normalized eight-point method and PnP (Perspective-n-Point) algorithm are used to estimate the basic matrix. The basic matrix is ​​decomposed by SVD (singular value decomposition) to obtain the camera extrinsic parameters, that is, the rotation matrix. , the translation vector ;

[0085] Among them, the normalized eight-point method is used to estimate the camera pose of the image pair at the beginning of the Euclidean structure recovery problem. In the normalized eight-point method, the template image refers to the image used as the matching reference, and the real-time image refers to the image that needs to be matched with the template image. First, it is assumed that the camera pose of the template image is in the world coordinate system, and then 8 feature point matching pairs of the template image and the real-time image are randomly selected as known information. The corresponding eight points in the image pair are normalized, that is, translation and scaling transformations are performed so that the average distance from each point to the origin with the center of gravity is Finally, the eight-point method is used to calculate the basic matrix of the transformed image, and then the inverse transformation is performed to the basic matrix corresponding to the original image.

[0086] The calculation process of the template graph is the same as that of the real-time graph. The normalization formula of the real-time graph and The solution formula is as follows:

[0087]

[0088]

[0089] in, is the distance from the transformed coordinate to the origin. is the scaling factor, is the pixel translation amount, Indicates zooming first and then panning.

[0090] The basic matrix after eight-point scaling Solve using the least squares method to get the optimal solution , the formula is as follows:

[0091]

[0092] in, express The straightened one-dimensional vector is The right singular value vector of the minimum singular value and the unknown quantity to be sought; is the coefficient matrix of the numerical equation group of the relationship between the 8 feature point matching pairs and the triangulated basic matrix. However, due to image errors and calculation accuracy, the optimal solution Generally full rank, not the true basic matrix, needs to be decomposed using SVD , and then use the inverse normalization transformation to get the true basic matrix To improve the basic matrix The reliability of the normalized eight-point method is that it needs to be combined with RANSAC (Random Sample Consensus) when it is actually used. The calculation is complex and the speed is slow. The formula for SVD decomposition and inverse normalization transformation is as follows:

[0093]

[0094]

[0095] The specific calculation process is as follows: randomly select an image pair from the image pairs, take the camera coordinate system of one of the images as the world coordinate system, and use the normalized eight-point method and SVD decomposition to calculate the pose of the other image in the camera coordinate system; because the poses of the cameras corresponding to the two images in the world coordinate system, that is, the camera extrinsic parameters, have been calculated, combined with the camera intrinsic parameters, the three-dimensional coordinates of all feature point matching pairs in the world coordinate system can be calculated through matrix transformation; after restoring the three-dimensional coordinates of all feature point pairs in the image with known camera extrinsic parameters, use the PnP method to estimate the camera poses of other images in turn, and the image to be calculated must form an image pair with the image with known camera extrinsic parameters.

[0096] S34, coordinate system transformation, combining the internal and external parameters of the camera to calculate the coordinates of the three-dimensional point corresponding to the feature point in the camera coordinate system, and the camera coordinate system posture is rotated relative to the world coordinate system , translated , so the transformation formula of the three-dimensional point in the camera coordinate system and the world coordinate system is:

[0097]

[0098] Therefore, the homogeneous coordinate transformation relationship of the three-dimensional points of the yard image that are directly restored from the pixel coordinate system to the world coordinate system is as follows:

[0099]

[0100] S35, yard contour point cloud generation, match two images that meet the requirements for image content overlap, perform three-dimensional structure recovery calculation of yard feature points, and then calculate the calculated points Add the yard point cloud data set , update the point cloud data set, and then use the BA algorithm (Bat Algorithm) to minimize the reprojection error, complete the generation of the sparse point cloud of the yard, and finally use the openMVS library to increase the shape details of the yard scene, such as Figure 5 As shown, this is a three-dimensional reconstruction of the stockpile.

[0101] The specific calculation process is as follows: restore feature points without corresponding three-dimensional coordinates through the same method in S33, and do not recalculate the three-dimensional coordinates for feature points with corresponding three-dimensional coordinates; then, consider image distortion and calculation errors, and use the BA algorithm to optimize the camera pose and internal parameters. To ensure accuracy, perform a BA optimization once each time the three-dimensional coordinates of a feature point pair in an image are restored, that is, incremental three-dimensional structure recovery is achieved; repeat the above steps until all image pairs are traversed, the three-dimensional coordinates of all feature point matching pairs are restored, and the sparse point cloud corresponding to the image sequence is obtained; then use the seed point-based regional growing algorithm to obtain the dense point cloud of the yard.

[0102] Step S4 specifically includes the following steps:

[0103] S41, plane cutting of the yard point cloud, assuming that the bottom of the yard is flat and regarded as a plane, so the plane cutting algorithm is used to fit the plane position of the yard (such as Figure 6 As shown, the black part is the calculated segmentation plane), and the point cloud and three-dimensional plane parameters of the equipment and facility stockpile are obtained;

[0104] S42, material pile point cloud extraction, considering the distribution characteristics of the material pile in the yard, using the density clustering algorithm to extract the point cloud data set of each material pile After density clustering, the point cloud data set with a significantly larger number is the point cloud set corresponding to the stockpile. Other point cloud sets can be judged as equipment facilities or noise, and then other point cloud set classes are discarded. The density clustering expression is:

[0105]

[0106] Cloud data set for clustering algorithm The center of the interior point, is the external point to be judged, Belongs to point cloud collection , The formula for calculating the distance between two points is: The threshold for determining whether an outlier can be added to the inlier set, Greater than but Not part of the cloud data collection Among them, the point cloud contour extraction and segmentation algorithm is implemented based on the PCL library. Specifically, the Statistical Outlier Removal, AC Segmentation, and Euclidean Cluster Extraction modules in the PCL library are used to implement the functions of yard point cloud filtering, plane segmentation, and stockpile clustering. Figure 7 As shown in the figure, it is a schematic diagram of hierarchical clustering of the stockpile point cloud according to the stacking characteristics, where the white one is the target stockpile and the black one is the segmentation plane. Figure 8 As shown, the point cloud of the target stockpile is extracted.

[0107] Step S5 specifically includes the following steps:

[0108] S51. To triangulate the stockpile point cloud, you can use the GreedyProjectionTriangulation class in the PCL library to convert the point cloud data into a triangular mesh model, such as Fig. 9 As shown, it is the triangulated network model of the target point cloud.

[0109] S52. Calculate the scale factor. There is a scale factor between the scale of the real world and the world coordinate system. First, calculate the scale factor between the real world and the world coordinate system between two cameras, and then calculate the average of all scale factors as the scale factor between the real world and the world coordinate system of the yard point cloud data. The formulas for the scale factor between cameras and the scale factor of the yard point cloud data are as follows:

[0110]

[0111]

[0112] is the camera number, is the total number of cameras with known relative position relationships, For Camera The actual distance between ), For Camera The distance between the camera positions solved in 3D structure recovery, It is the scaling factor between the real world of the yard point cloud data and the calculated world coordinate system. For Camera The scale factor between the real world and the calculated world coordinate system. Based on the camera position relationship in the world coordinate system, the distance formula is used to calculate ,and It is a priori information and is calculated in advance based on the layout information of the storage yard cameras.

[0113] S53, volume calculation, combining the positional relationship between the yard ground and the stockpile point cloud contour, using the definite integral with the yard ground as the bottom surface to calculate the stockpile Point cloud volume, the formula is as follows:

[0114]

[0115] like Fig.10 As shown, according to the formula, first calculate the area of ​​the projected triangle and point cloud collection Triangle three vertices to plane Distance , , The mean ; For each projected triangle, the area of ​​the triangle can be calculated through its three vertices; and the point cloud set can be calculated at the same time Orthographic projection of the three vertices of a triangle onto a plane The volume of a prism whose bases are not parallel but whose edges are perpendicular to the lower base can be calculated by measuring the distance.

[0116] For each projected triangle, the area of ​​the triangle can be calculated through its three vertices; at the same time, the point cloud set can be calculated Orthographic projection of the three vertices of a triangle onto a plane The distance between the bottom and the prism can be calculated to calculate the volume of the prism whose bases are not parallel but whose edges are perpendicular to the bottom. The area of ​​the bottom triangle is calculated by the cross product of the vectors, and the formula is as follows:

[0117]

[0118] in, Based on the cross product formula of coordinate vectors, let vector , is the length vector of any two sides of the projected triangle, and the formula is as follows:

[0119]

[0120] Solving Point Cloud Collection Triangle three vertices to plane The distance from a point outside the plane to the plane is calculated using the distance formula from a point outside the plane to the plane. The formula is as follows:

[0121]

[0122] in, To determine the parameters of the three-dimensional yard plane, The plane is represented as: . is the mean of the three edges.

[0123] Solving Point Cloud Collection Orthographic projection of the three vertices of a triangle onto a plane The intersection point of the three-dimensional plane is obtained by using the formula for the intersection of a straight line passing through a certain point outside the plane and with a certain slope. The formula is as follows:

[0124]

[0125] The specific calculation process is as follows: calculate the height of the triangular prism, use the three-dimensional point-line-plane relationship to find the distance from the projected triangle vertices and the point cloud points to the bottom plane of the yard, the height of the triangular prism whose bottom surfaces are not parallel but whose edges are perpendicular to the bottom surfaces is the average of the distances from the three vertices to the plane; calculate the area of ​​the projected triangle, for each projected triangle face, calculate the area of ​​the triangle through the cross product of its three vertices; calculate the volume of the entire mesh model, traverse all the faces of the stockpile point cloud, calculate the volume of the triangular prism and accumulate it to get the volume of the stockpile.

[0126] It can be seen that the present invention realizes the image data collection of the stockpile in the yard through the monitoring cameras installed around the yard or on the mobile device; the image of the yard is preprocessed by using histogram equalization and guided filtering to strengthen the image texture and reduce noise; based on the known relative position and internal parameters of the camera, combined with the positional relationship between the camera and the stockpile in the yard, a three-dimensional reconstruction algorithm is designed to realize the three-dimensional structure recovery of the stockpile scene, and obtain the dense point cloud of the yard; considering the distribution characteristics of the stockpile point cloud, clustering and plane segmentation algorithms are used to segment and extract the stockpile point cloud in the yard, and the stockpile plane is determined; the stockpile point cloud is triangulated, and the stockpile point cloud is converted into a triangular mesh using greedy triangulation, and then considering the positional relationship between the stockpile ground and the stockpile point cloud contour, each triangular mesh is projected onto the stockpile plane to form a triangular prism with a ridge perpendicular to the bottom surface but two bottom surfaces are not parallel, and the volume of the stockpile in the yard is obtained by cumulative calculation. The method of the present invention makes full use of the existing equipment of the port, the image data acquisition speed is fast, and the measurement process is fully automated, so as to realize the monitoring of the stockpile bulk volume at any time when the weather conditions permit.

[0127] like Fig.11 As shown, the dry bulk cargo pile volume measurement system using a yard camera in the embodiment of the present invention is mainly used to implement the above method embodiment, and the system mainly includes:

[0128] A data acquisition module is used to acquire yard image data collected by high-definition cameras installed on lighting poles around the yard or on mobile operating equipment;

[0129] A preprocessing module, used for preprocessing the storage yard image data by using histogram equalization and guided filtering;

[0130] The 3D reconstruction module of the storage yard is used to extract feature points, match feature points and perform geometric filtering on the pre-processed image data based on the improved SIFT algorithm to obtain image matching pairs; the external parameters of the camera are solved according to the image matching pairs, and the position and internal parameters of the camera in the storage yard are combined to convert the 2D points of the storage yard image into 3D points of the storage yard in the world coordinate system, generate storage yard point cloud data, and reconstruct the 3D structure of the storage yard; wherein, the improved SIFT algorithm is specifically used in matching feature points, after performing rough matching of feature points according to Euclidean distance, connecting each rough matching pair, and eliminating wrong matching pairs according to the comparison of the angle between the connecting lines with the preset angle threshold, to obtain the feature point matching pairs after precise matching, and then perform image matching;

[0131] The stockpile point cloud data extraction module is used to segment the point cloud of the reconstructed three-dimensional structure of the stockpile using a plane segmentation algorithm, and extract the segmented point cloud data of the stockpile according to the stacking distribution characteristics of the stockpile;

[0132] The stockpile volume calculation module is used to convert the point cloud data of the stockpile into a triangulated network model, combine the positional relationship between the stockpile ground and the point cloud contour of the stockpile, and use a definite integral with the stockpile ground as the bottom surface to calculate the stockpile volume.

[0133] Each module is mainly used to implement each step of the above method embodiment, which will not be described one by one here.

[0134] The present invention also provides a computer storage medium, characterized in that a computer program executable by a processor is stored therein, and the computer program executes the method for measuring the volume of a dry bulk cargo pile using a yard camera as described in the above technical solution.

[0135] The present application also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, a server, an App application store, etc., on which a computer program is stored, and when the program is executed by a processor, a corresponding function is realized. When the computer-readable storage medium of this embodiment is executed by a processor, a dry bulk cargo pile volume measurement method using a yard camera of the method embodiment is realized.

[0136] It should be pointed out that, according to the needs of implementation, the various steps / components described in this application can be split into more steps / components, and two or more steps / components or partial operations of steps / components can be combined into new steps / components to achieve the purpose of the present invention.

[0137] The order of execution of each step in the above embodiment does not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0138] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A method for measuring the volume of a dry bulk cargo pile using a yard camera, characterized in that: The following steps are involved: S1. Install high-definition cameras on lighting poles or mobile operating equipment around the yard to collect yard image data; S2, preprocessing the storage yard image data using histogram equalization and guided filtering; S3. Based on the improved SIFT algorithm, feature point extraction, feature point matching and image pair geometric filtering are performed on the pre-processed image data to obtain image matching pairs; the external parameters of the camera are solved according to the image matching pairs, and the two-dimensional points of the yard image are converted into three-dimensional points of the yard in the world coordinate system in combination with the position and internal parameters of the camera in the yard, so as to generate yard point cloud data and reconstruct the three-dimensional structure of the yard; wherein, the improved SIFT algorithm is specifically that when matching feature points, after rough matching of feature points is performed according to the Euclidean distance, each rough matching pair is connected, and the wrong matching pairs are eliminated according to the comparison of the angle between the connecting lines with the preset angle threshold, so as to obtain the feature point matching pairs after precise matching, and then perform image pairing; S4, using a plane segmentation algorithm to segment the point cloud of the reconstructed three-dimensional structure of the stockpile, and extracting the segmented point cloud data of the stockpile according to the stacking distribution characteristics of the stockpile; S5. Convert the point cloud data of the stockpile into a triangulated network model, combine the positional relationship between the stockpile ground and the point cloud contour of the stockpile, and use a definite integral with the stockpile ground as the bottom surface to calculate the volume of the stockpile.

2. The method for measuring the volume of a dry bulk cargo pile using a yard camera according to claim 1, characterized in that: In step S3, when the feature points are precisely matched, based on the fact that the connecting lines between the correct feature point matching pairs are approximately parallel, and based on the fact that the angle between the connecting lines of the incorrect feature point matching pairs and the correct feature point matching pairs is greater than a preset angle threshold, the feature point pairs with similar parallel relationships are clustered, and hierarchical clustering is performed by continuously increasing the connecting line angle condition to obtain the feature point matching pairs after precise matching.

3. The method for measuring the volume of a dry bulk cargo pile using a yard camera according to claim 1, characterized in that: In step S3, when geometric filtering is performed on the image pairs, several groups of feature point pairs are randomly selected from each matching image pair, a basic matrix is ​​estimated using each group of feature point pairs, and the error values ​​of all feature point matching pairs in the matching image pair are calculated to obtain the error values ​​of the several groups, sort them, and select the basic matrices corresponding to the 7 feature point matching pairs with the smallest error values ​​as the final basic matrices. Finally, the reliability of the matching image pairs is determined to complete the geometric filtering of the image pairs.

4. The method for measuring the volume of a dry bulk cargo pile using a yard camera according to claim 1, characterized in that: In step S3, the solution of the camera extrinsic parameters is based on the Euclidean structure recovery problem. Specifically, the normalized eight-point method and the PnP algorithm are used to estimate the basic matrix, and the basic matrix is ​​decomposed by SVD to obtain the rotation matrix and translation vector of the camera extrinsic parameters.

5. The method for measuring the volume of a dry bulk cargo pile using a yard camera according to claim 1, characterized in that: When reconstructing the three-dimensional structure of the storage yard in step S3, the BA algorithm is specifically used to optimize the external and internal parameters of the camera. Each time the three-dimensional coordinates of a pair of feature points in a storage yard image are restored, a BA optimization is performed to incrementally restore the three-dimensional structure.

6. The method for measuring the volume of a dry bulk cargo pile using a yard camera according to claim 1, characterized in that: The high-definition cameras installed on the lighting poles around the yard are of uniform model and fixed relative position.

7. The method for measuring the volume of a dry bulk cargo pile using a yard camera according to claim 1, characterized in that: Step S4 specifically includes the following steps: S41. Assuming that the bottom surface of the stockpile is a plane, the plane cutting algorithm is used to fit the plane position of the stockpile, and the point cloud and three-dimensional plane parameters of the equipment, facilities, and stockpile are obtained; S42. According to the distribution characteristics of the material piles in the stockpile yard, a density clustering algorithm is used to extract the point cloud data set of each material pile, and other point cloud set classes are discarded.

8. The method for measuring the volume of a dry bulk cargo pile using a yard camera according to claim 7, characterized in that: Specifically, the StatisticalOutlierRemoval, ACSegmentation, and EuclideanClusterExtraction modules in the PCL library are used to implement yard point cloud filtering, plane segmentation, and stockpile clustering respectively.

9. A dry bulk cargo stockpile volume measurement system using a yard camera, characterized in that: include: A data acquisition module is used to acquire yard image data collected by high-definition cameras installed on lighting poles around the yard or on mobile operating equipment; A preprocessing module, used for preprocessing the storage yard image data by using histogram equalization and guided filtering; The 3D reconstruction module of the storage yard is used to extract feature points, match feature points and perform geometric filtering on the pre-processed image data based on the improved SIFT algorithm to obtain image matching pairs; the external parameters of the camera are solved according to the image matching pairs, and the position and internal parameters of the camera in the storage yard are combined to convert the 2D points of the storage yard image into 3D points of the storage yard in the world coordinate system, generate storage yard point cloud data, and reconstruct the 3D structure of the storage yard; wherein, the improved SIFT algorithm is specifically used in matching feature points, after performing rough matching of feature points according to Euclidean distance, connecting each rough matching pair, and eliminating wrong matching pairs according to the comparison of the angle between the connecting lines with the preset angle threshold, to obtain the feature point matching pairs after precise matching, and then perform image matching; The stockpile point cloud data extraction module is used to segment the point cloud of the reconstructed three-dimensional structure of the stockpile using a plane segmentation algorithm, and extract the segmented point cloud data of the stockpile according to the stacking distribution characteristics of the stockpile; The stockpile volume calculation module is used to convert the point cloud data of the stockpile into a triangulated network model, combine the positional relationship between the stockpile ground and the point cloud contour of the stockpile, and use a definite integral with the stockpile ground as the bottom surface to calculate the stockpile volume.

10. A computer storage medium, characterized in that: A computer program executable by a processor is stored therein, and the computer program executes the method for measuring the volume of a dry bulk cargo pile using a yard camera as described in any one of claims 1 to 8.

Images