Aggregate volume calculation method and device based on visual image detection

Abstract

The invention relates to the technical field of concrete production, in particular to an aggregate volume calculation method and device based on visual image detection. The invention discloses an aggregate volume calculation method based on visual image detection. The method comprises the following steps: S1, image acquisition; s2, image preprocessing; s3, acquiring three-dimensional data of the aggregate: aligning the depth image with the RGB image to acquire the three-dimensional data of the aggregate; and S4, volume calculation. According to the method, the aggregate image data acquired by the depth camera and the RGB camera are acquired, the volume of the aggregate is calculated based on the improved Maskrcnn network, the aggregate images can be quickly and continuously acquired and processed, and compared with a traditional manual measurement and physical screening method, the speed of aggregate volume calculation and the overall working efficiency are greatly improved, and the method is suitable for popularization and application. By utilizing a high-precision image processing technology and an advanced algorithm, the outline of the aggregate can be identified more accurately, the volume of the aggregate can be calculated more accurately, personal errors are reduced, and the accuracy of a measurement result is improved.

Description

Technical Field

[0001] The invention relates to the technical field of concrete, and in particular to a method and a device for calculating aggregate volume based on visual image detection. Background Art

[0002] In the production of concrete, asphalt and other building materials, the volume and gradation of aggregates are key factors affecting the performance of the final product. Traditional methods for measuring coarse aggregate volume rely on physical screening and manual measurement, which are not only time-consuming and labor-intensive, but also have limited accuracy. With the development of computer vision and image processing technology, the method of calculating coarse aggregate volume based on visual inspection images provides an efficient and accurate means of aggregate analysis.

[0003] In the related art, the statistical method of coarse aggregate grading data based on visual detection images uses two-dimensional visual detection images. However, the traditional method of obtaining volume from two-dimensional data has the defect of poor aggregate analysis accuracy due to the lack of aggregate height information. Summary of the invention

[0004] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or be understood by implementing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description and other drawings of the description.

[0005] The purpose of the present invention is to overcome the above-mentioned shortcomings and provide a method and device for calculating the volume of aggregate based on visual image detection. By acquiring aggregate image data collected by a depth camera and an RGB camera, the volume of the aggregate is calculated based on an improved Maskrcnn network. Aggregate images can be collected and processed quickly and continuously. Compared with traditional manual measurement and physical screening methods, the speed and overall work efficiency of aggregate volume calculation are greatly improved. By using high-precision image processing technology and advanced algorithms, the outline of the aggregate can be more accurately identified and its volume can be calculated, thereby reducing human errors and improving the accuracy of the measurement results.

[0006] The present invention provides a method for calculating aggregate volume based on visual image detection, comprising: S1. Image acquisition: Use the depth camera and RGB camera to detect the visual image data of aggregates on the conveyor belt in real time, obtain the depth image and RGB image respectively, and input them into the PC; S2, image preprocessing: PC preprocesses the visual image data and repairs the missing values ​​in the depth image; S3, aggregate three-dimensional data acquisition: align the depth image with the RGB image to obtain the three-dimensional data of the aggregate; S4. Volume calculation: The mask image of the aggregate is obtained using the Maskrcnn network. The contour information is obtained based on the mask image of the aggregate and the pixel points within the aggregate contour are obtained. The height of each pixel is highly integrated and accumulated to obtain the volume of the aggregate by fitting.

[0007] In some embodiments, since there are a large number of missing values ​​in the depth image, the depth image is repaired, and the specific repair steps are: S21, creating a mask image for the depth image, marking pixels with a value of 0 in the depth image as pixels that need to be repaired; S22, replacing the pixels that need to be repaired in the depth image with NaN; S23, using the nearest neighbor interpolation method to fill the NaN value, by finding the nearest pixel value of the target pixel in the original image to assign it to the target pixel; S24. Call the function in opencv to perform patching.

[0008] In some embodiments, in step S3, the specific steps of aligning the depth image with the RGB image are: S31. Construct a three-dimensional vector in the coordinate system of the depth camera , where (x, y) is the horizontal and vertical coordinates of the pixel point, and z is the depth value of the pixel point; S32, converting the coordinates of the pixel point into the coordinates of the space point, the spatial point coordinate calculation formula is: in, is the intrinsic parameter matrix of the depth camera, is the three-dimensional coordinate of the pixel point in the depth camera coordinate system; S33. Due to is the coordinate of the pixel in the depth camera coordinate system, so it needs to be converted to the RGB camera coordinate system. The conversion formula is: in, is the pixel coordinate in the RGB camera coordinate system, R is the rotation matrix, and T is the translation vector; S34, will Multiply by the intrinsic parameter matrix of the RGB camera Get the point coordinates of the depth image corresponding to the RGB image , For three-dimensional coordinates, the conversion formula is: (x, y) is the pixel coordinate of the pixel in the RGB image, and the color of the pixel is the color of the corresponding pixel in the depth image; S35 . Assign the color of each pixel to the corresponding pixel on the depth image to obtain an aligned depth map.

[0009] In some embodiments, before performing the volume calculation in step S4, image segmentation processing is required to obtain a mask image of each aggregate in the image through a trained Maskrcnn image segmentation algorithm.

[0010] In some embodiments, all pixels within the aggregate outline are obtained according to the mask image of the aggregate, the total number of all pixels of each aggregate is recorded as P, and the cross-sectional area S of the aggregate is calculated by the pixels within the aggregate outline. The calculation formula of the cross-sectional area S is: Where dx is the unit size of the pixel in the X-axis direction, and dy is the unit size of the pixel in the Y-axis direction; The pixel points within each aggregate outline are integrated at height to obtain the volume information of the aggregate. The calculation formula of the aggregate volume V is: Among them, dH is the height of the pixel point, and the height of the pixel point comes from the collected depth image. Since the height information of the aggregate is stored in the depth camera coordinate system and the pixel coordinates are mapped, the corresponding height value is saved in each pixel point.

[0011] A device for calculating aggregate volume by visual image detection, comprising: a conveyor belt on which the aggregate is placed; A depth camera is disposed above the conveyor belt and collects a depth image of the aggregate; an RGB camera, which is disposed above the conveyor belt and captures an RGB image of the aggregate; The PC is electrically connected to the conveyor belt, the depth camera and the RGB camera.

[0012] By adopting the above technical solution, the beneficial effects of the present invention are: The present invention obtains aggregate image data collected by a depth camera and an RGB camera, and calculates the volume of the aggregate based on an improved Maskrcnn network. The aggregate images can be collected and processed quickly and continuously. Compared with traditional manual measurement and physical screening methods, the speed of aggregate volume calculation and the overall work efficiency are greatly improved. By using high-precision image processing technology and advanced algorithms, the outline of the aggregate can be more accurately identified and its volume can be calculated, thereby reducing human errors and improving the accuracy of the measurement results.

[0013] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

[0014] Undoubtedly, these and other objects of the present invention will become more apparent after the following detailed description of the preferred embodiment described with reference to various figures and drawings.

[0015] In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, one or several preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the present invention and do not constitute a limitation of the present invention.

[0017] In the drawings, the same reference numerals are used for the same components and the drawings are schematic and not necessarily drawn to scale.

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only one or several embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on such drawings without paying creative work.

[0019] Figure 1 is a schematic flow chart of a method for calculating the volume of coarse aggregate based on visual image detection in some embodiments of the present invention; Figure 2 Schematic diagram of aggregate data collection platform in some embodiments of the present invention; Figure 3 Schematic diagram of the calculation principle of the aggregate volume calculation method in some embodiments of the present invention.

[0020] Description of main reference numerals: 1. Conveyor belt; 2. Depth camera; 3. RGB camera; 4. PC. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but are not used to limit the present invention.

[0022] In addition, in the description of the present invention, it should be understood that the terms "center", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "axial", "radial", "circumferential" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0023] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral body; it can be directly connected or indirectly connected through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. However, if it is indicated as a direct connection, it means that the two connected bodies are not connected through a transition structure, but are connected to form a whole through a connecting structure. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0024] In the present invention, unless otherwise clearly specified and limited, the first feature "on" or "under" the second feature may be that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in an appropriate manner in any one or more embodiments or examples.

[0025] Reference Figure 1-Figure 3 , Figure 1 is a schematic flow chart of a method for calculating the volume of coarse aggregate based on visual image detection in some embodiments of the present invention; Figure 2 Schematic diagram of aggregate data collection platform in some embodiments of the present invention; Figure 3 Schematic diagram of the calculation principle of the aggregate volume calculation method in some embodiments of the present invention.

[0026] According to some embodiments of the present invention, the present invention provides a method for calculating aggregate volume based on visual image detection, comprising: S1, image acquisition: using the depth camera 2 and the RGB camera 3 to detect the visual image data of the aggregate on the conveyor belt in real time, obtaining the depth image and the RGB image respectively, and inputting them into the PC 4; Depth camera 2 uses a binocular stereo vision system to obtain depth information through parallax calculation. The two infrared images captured by the binocular stereo vision system have a parallax, that is, the position difference of the same object in the two images. The distance from the object to the camera, that is, the depth, can be calculated through the parallax.

[0027] Since the depth camera 2 and the RGB camera 3 are at a distance above the conveyor belt 1, the fields of view captured are different. Therefore, the depth camera 2 will take pictures T seconds after the RGB camera 3 collects the pictures. The time T is calculated as T=S / V, where S is the distance between the depth camera 2 and the RGB camera 3, and V is the speed of the belt drive.

[0028] S2, image preprocessing: PC 4 preprocesses the visual image data and repairs the missing values ​​in the depth image; Since there are a large number of missing values ​​in the depth image, the depth image is repaired. The specific repair steps are as follows: S21, creating a mask image for the depth image, marking pixels with a value of 0 in the depth image as pixels that need to be repaired; S22, replacing the pixels that need to be repaired in the depth image with NaN; S23, using the nearest neighbor interpolation method to fill the NaN value, by finding the nearest pixel value of the target pixel in the original image to assign it to the target pixel; S24. Call the function in opencv to perform patching.

[0029] S3, aggregate three-dimensional data acquisition: align the depth image with the RGB image to obtain the three-dimensional data of the aggregate; The present invention chooses to align the depth image to the RGB image. Since the pixels in the depth image are two-dimensional data, the two-dimensional points in the depth image are mapped to three-dimensional points in the world coordinate system, and then the mapped three-dimensional points in the world coordinate system are projected onto the RGB image to obtain the coordinates of the pixels on the RGB image. Then, the values ​​of the RGB three channels of the pixels are extracted, and the RGB values ​​of the pixels are assigned to the depth image, and each pixel on the depth image is assigned a corresponding RGB three-channel value. The specific steps for aligning the depth image with the RGB image are: S31. Construct a three-dimensional vector in the coordinate system of depth camera 2 , where (x, y) is the horizontal and vertical coordinates of the pixel point, and z is the depth value of the pixel point; S32, converting the coordinates of the pixel point into the coordinates of the space point, the spatial point coordinate calculation formula is: in, is the intrinsic parameter matrix of depth camera 2, is the three-dimensional coordinate of the pixel point in the coordinate system of depth camera 2; S33. Due to is the coordinate of the pixel in the coordinate system of depth camera 2, so it needs to be converted to the coordinate system of RGB camera 3. The conversion formula is: in, is the pixel coordinate in the RGB camera 3-coordinate system, R is the rotation matrix, and T is the translation vector; S34, will Multiply by the intrinsic parameter matrix of RGB camera 3 Get the point coordinates of the depth image corresponding to the RGB image , For three-dimensional coordinates, the conversion formula is: (x, y) is the pixel coordinate of the pixel in the RGB image, and the color of the pixel is the color of the corresponding pixel in the depth image; S35, assigning the color of each pixel to the corresponding pixel on the depth image to obtain an aligned depth map; Before performing the volume calculation in step S4, image segmentation processing is required. The mask map of each aggregate in the image is obtained through the trained Maskrcnn image segmentation algorithm.

[0030] S4, volume calculation: use the Maskrcnn network to obtain the mask image of the aggregate, obtain the contour information based on the mask image of the aggregate and obtain the pixel points within the aggregate contour, perform height integration and accumulation on each pixel point, and fit to obtain the volume of the aggregate; like Figure 3 As shown in FIG. 1 , according to the mask map of the aggregate, all the pixels within the outline of the aggregate are obtained. The total number of all the pixels of each aggregate is recorded as P. The cross-sectional area S of the aggregate is calculated through the pixels within the outline of the aggregate. The calculation formula of the cross-sectional area S is: Where dx is the unit size of the pixel in the X-axis direction, and dy is the unit size of the pixel in the Y-axis direction; The pixel points within each aggregate outline are integrated at height to obtain the volume information of the aggregate. The calculation formula of the aggregate volume V is: Among them, dH is the height of the pixel point, and the height of the pixel point comes from the collected depth image. Since the height information of the aggregate is stored in the coordinate system of the depth camera 2 and the pixel coordinates are mapped, the corresponding height value is saved in each pixel point.

[0031] The present invention also provides an aggregate volume calculation device based on visual image detection, comprising: a conveyor belt 1 on which aggregate is placed; A depth camera 2 is disposed above the conveyor belt 1 and collects a depth image of the aggregate; An RGB camera 3 is disposed above the conveyor belt 1 and collects RGB images of the aggregate; The PC 4 is electrically connected to the conveyor belt 1, the depth camera 2 and the RGB camera 3. Figure 2 As shown, the depth camera 2 and the RGB camera 3 need to cooperate with the conveyor belt 1 and the PC 4. The depth camera 2 and the RGB camera 3 are respectively arranged above the conveyor belt 1 and connected to the PC 4. The conveyor belt 1 is used to transport aggregates. The RGB camera 3 and the depth camera 2 are used in turn to photograph the aggregates on the conveyor belt 1 and transmit them back to the PC 4 to obtain the three-dimensional data of the coarse aggregate.

[0032] It should be understood that the embodiments disclosed in the present invention are not limited to the specific processing steps or materials disclosed herein, but should be extended to equivalent substitutions of such features understood by ordinary technicians in the relevant field. It should also be understood that the terms used herein are only used for the purpose of describing specific embodiments and are not meant to be limiting.

[0033] The "embodiment" mentioned in the specification means that the specific features or characteristics described in conjunction with the embodiment are included in at least one embodiment of the present invention. Therefore, the phrases or "embodiment" appearing in various places throughout the specification do not necessarily refer to the same embodiment.

[0034] In addition, the described features or characteristics may be combined in one or more embodiments in any other suitable manner. In the above description, some specific details, such as thickness, quantity, etc., are provided to provide a comprehensive understanding of the embodiments of the present invention. However, those skilled in the relevant art will understand that the present invention can be implemented without one or more of the above specific details or can also be implemented using other methods, components, materials, etc.

Claims

1. A method for calculating aggregate volume based on visual image detection, characterized in that: include: S1. Image acquisition: Use the depth camera and RGB camera to detect the visual image data of aggregates on the conveyor belt in real time, obtain the depth image and RGB image respectively, and input them into the PC; S2, image preprocessing: PC preprocesses the visual image data and repairs the missing values ​​in the depth image; S3, aggregate three-dimensional data acquisition: align the depth image with the RGB image to obtain the three-dimensional data of the aggregate; S4. Volume calculation: The mask image of the aggregate is obtained using the Maskrcnn network. The contour information is obtained based on the mask image of the aggregate and the pixel points within the aggregate contour are obtained. The height of each pixel is highly integrated and accumulated to obtain the volume of the aggregate by fitting.

2. The method for calculating aggregate volume based on visual image detection according to claim 1, characterized in that: Since there are a large number of missing values ​​in the depth image, the depth image is repaired. The specific repair steps are as follows: S21, creating a mask image for the depth image, marking pixels with a value of 0 in the depth image as pixels that need to be repaired; S22, replacing the pixels marked in the depth image that need to be repaired with NaN; S23, using the nearest neighbor interpolation method to fill the NaN value, by finding the nearest pixel value of the target pixel in the original image to assign it to the target pixel; S24. Call the function in opencv to perform patching.

3. The method for calculating aggregate volume based on visual image detection according to claim 1, characterized in that: In step S3, the specific steps for aligning the depth image with the RGB image are: S31. Construct a three-dimensional vector in the coordinate system of the depth camera , where (x, y) is the horizontal and vertical coordinates of the pixel point, and z is the depth value of the pixel point; S32, converting the coordinates of the pixel point into the coordinates of the space point, the spatial point coordinate calculation formula is: in, is the intrinsic parameter matrix of the depth camera, is the three-dimensional coordinate of the pixel point in the depth camera coordinate system; S33. Due to is the coordinate of the pixel in the depth camera coordinate system, so it needs to be converted to the RGB camera coordinate system. The conversion formula is: in, is the pixel coordinate in the RGB camera coordinate system, R is the rotation matrix, and T is the translation vector; S34, will Multiply by the intrinsic parameter matrix of the RGB camera Get the point coordinates of the depth image corresponding to the RGB image , For three-dimensional coordinates, the conversion formula is: (x, y) is the pixel coordinate of the pixel in the RGB image, and the color of the pixel is the color of the corresponding pixel in the depth image; S35 . Assign the color of each pixel to the corresponding pixel on the depth image to obtain an aligned depth map.

4. The method for calculating aggregate volume based on visual image detection according to claim 1, characterized in that: Before performing the volume calculation in step S4, image segmentation processing is required. The mask map of each aggregate in the image is obtained through the trained Maskrcnn image segmentation algorithm.

5. The method for calculating aggregate volume based on visual image detection according to claim 4, characterized in that: According to the mask map of the aggregate, all the pixels within the outline of the aggregate are obtained. The total number of all the pixels of each aggregate is recorded as P. The cross-sectional area S of the aggregate is calculated through the pixels within the outline of the aggregate. The calculation formula of the cross-sectional area S is: Where dx is the unit size of the pixel in the X-axis direction, and dy is the unit size of the pixel in the Y-axis direction; The pixel points within each aggregate outline are integrated at height to obtain the volume information of the aggregate. The calculation formula of the aggregate volume V is: Among them, dH is the height of the pixel point, and the height of the pixel point comes from the collected depth image. Since the height information of the aggregate is stored in the depth camera coordinate system and the pixel coordinates are mapped, the corresponding height value is saved in each pixel point.

6. A device for calculating aggregate volume by visual image detection, characterized in that: The method for calculating the aggregate volume by visual image detection according to any one of claims 1 to 5 above is applied, and the device comprises: a conveyor belt on which the aggregate is placed; A depth camera is disposed above the conveyor belt and collects a depth image of the aggregate; an RGB camera, which is disposed above the conveyor belt and captures an RGB image of the aggregate; The PC is electrically connected to the conveyor belt, the depth camera and the RGB camera.

Images