Bulk material flow measurement detection method and system based on depth camera

Abstract

The invention discloses a bulk material flow measurement detection method and system based on a depth camera, and belongs to the field of material metrology, and the method comprises the steps: S1, training a neural network through employing a collected training color image; s2, acquiring a no-load depth image of the belt by using a depth camera, and further obtaining the belt height of each pixel in the belt row pixels; s3, when the belt conveys the bulk materials, collecting a color image and a depth image of the belt, inputting the color image of the current frame into the neural network, and further obtaining the charge level height of each pixel in charge level row pixels; s4, obtaining the material thickness of each pixel in the charge level row pixels, calculating the cross sectional areas of all materials in the charge level row pixels, and further obtaining the volumes of all materials in the charge level row pixels; and S5, the step S3 to the step S4 are repeated for set times, so that the total weight of the materials passing through the belt in the time period is obtained. Defects of a traditional weight belt scale can be overcome so that reliability and precision of measurement can be enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of material metering, and in particular relates to a bulk material flow metering detection method and system based on a depth camera. Background Art

[0002] In the application scenario of belt conveying materials, accurate monitoring and measurement of material flow is a very common measurement requirement. In the existing technology, traditional weight belt scales are usually used to mechanically measure material flow, such as measuring the weight of the material on the belt through a pressure sensor, or calculating the speed and flow of the material through a speed sensor and length calculation. However, traditional belt scales have the following defects: mechanical sensors are susceptible to wear, environmental influences and accuracy limitations; traditional belt scales need to directly contact the material and are easily affected by factors such as material properties, humidity, particle size, etc., resulting in inaccurate measurements or failures; traditional belt scales have high maintenance costs and require frequent calibration and maintenance, which affects production efficiency and equipment reliability. Summary of the invention

[0003] The purpose of the present invention is to propose a bulk material flow measurement detection method and system based on a depth camera, which can overcome the defects of traditional weight belt scales and thus improve the reliability and accuracy of measurement.

[0004] The present invention is achieved through the following technical solutions: A bulk material flow measurement and detection method based on a depth camera comprises the following steps: Step S1, using a depth camera to collect a training color image of a belt that is conveying bulk materials, and using the collected training color image to train a neural network so that the neural network outputs an image with the surface contours of all materials; Step S2: when the belt is unloaded, a depth camera is used to collect an unloaded depth image of the belt, a row in the unloaded depth image is intercepted to obtain belt row pixels, and the belt height of each pixel in the belt row pixels is obtained; Step S3, when the belt is conveying bulk materials, a color image and a depth image of the belt are collected by using a depth camera, and the current frame color image is input into the neural network trained in step S1 to obtain an image with the material surface contours of all materials, and a row corresponding to the interception position in the unloaded image is intercepted in the image and the current frame depth image to obtain material surface row pixels and depth row pixels, and the material surface height of each pixel in the material surface row pixels is obtained according to the depth row pixels; Step S4, obtaining the material thickness of each pixel in the material surface row pixels according to the material surface height of each pixel in the material surface row pixels and the belt height of each pixel in the belt row pixels, and calculating the cross-sectional area of ​​all materials in the material surface row pixels in combination with the distance between two adjacent pixels in the material surface row pixels, and then obtaining the volume of all materials in the material surface row pixels of the current frame color image; Step S5, repeating steps S3 to S4 for a set number of times, adding up the volumes of all materials obtained in each repetition to obtain the total volume of the materials passing through the belt within a set time period, and then obtaining the total weight of the materials passing through the belt within the time period.

[0005] Furthermore, in step S1, a depth camera is used to capture multiple training color images, and the neural network is trained after the material surface contour is marked in each training color image.

[0006] Furthermore, in step S2, when the belt is unloaded, a depth camera is used to collect n An empty depth image, and according to the formula Calculate the average image , and take the average image to obtain the belt height of each pixel in the belt row pixel, where express n The total depth image is obtained by superimposing the unloaded depth images.

[0007] Furthermore, in step S2, a row intercepted from the average image is a row where the center of the average image is located, a row with the smallest sinking amount in the average image, or a row in the average image corresponding to the belt roller.

[0008] Furthermore, in step S3, before the current frame color image is input into the neural network, the current frame color image and the current frame depth image collected by the depth camera are matched and aligned, and the first pixel in the material surface row is i Pixel surface height That is, the depth data of the corresponding pixel in the depth row pixel.

[0009] Furthermore, in step S4, according to the formula Get the first pixel in the material surface row i Material thickness in pixels , and according to the formula Get the volume of all materials in the material surface row pixels of the current frame color image, where: is the cross-sectional area of ​​all materials, The first pixel in the material row i The cross-sectional area of ​​pixels, L is the distance between two adjacent pixels, v Indicates belt speed, 1≤ i ≤ m , m is the number of pixels in the row of pixels on the surface, Indicates the first pixel in the material row i The height of the material surface in pixels, Indicates the number of pixels in the belt row corresponding to the number of pixels in the material row. i Belt height in pixels.

[0010] Furthermore, the belt speed is acquired through a belt encoder.

[0011] Furthermore, the neural network includes a fully convolutional network.

[0012] The present invention is also achieved through the following technical solutions: A detection system based on any of the above-described bulk material flow measurement detection methods based on a depth camera comprises: Acquisition module: including a depth camera and a belt encoder. The depth camera is used to collect training color images, no-load depth images, and multi-frame color images and depth images of the belt when the belt is conveying bulk materials. The belt encoder is used to collect the belt speed. Calculation module: connected to the acquisition module, trains the neural network according to the training color image acquired by the acquisition module, so that the neural network outputs an image with the material surface contours of all materials; obtains the belt height of each pixel in the belt row pixels according to the no-load depth image acquired by the acquisition module; inputs the current frame color image into the neural network trained by step S1 to obtain an image with the material surface contours of all materials, obtains the material surface height of each pixel in the material surface row pixels according to the current frame depth image, obtains the material thickness of each pixel in the material surface row pixels according to the material surface height of each pixel in the material surface row pixels and the belt height of each pixel in the belt row pixels, and calculates the cross-sectional area of ​​all materials in the material surface row pixels in combination with the distance between two adjacent pixels in the material surface row pixels, thereby obtaining the volume of all materials in the material surface row pixels of the current frame color image, and obtaining the total weight of the material passing through the belt within a set time period.

[0013] The present invention has the following beneficial effects: 1. The present invention first trains a neural network using a training color image, and obtains the belt height of each pixel in the belt row pixels according to the no-load depth image, then inputs the current frame color image into the trained neural network to obtain an image with the material surface contours of all materials, and respectively intercepts a row in the image and the current frame depth image corresponding to the interception position in the no-load image to obtain material surface row pixels and depth row pixels, and obtains the material surface height of each pixel in the material surface row pixels according to the depth row pixels, then obtains the material thickness of each pixel in the material surface row pixels according to the material surface height of each pixel in the material surface row pixels and the belt height of each pixel in the belt row pixels, and calculates the cross-sectional area of ​​all materials in the material surface row pixels in combination with the distance between two adjacent pixels in the material surface row pixels, and then obtains the volume of all materials in the material surface row pixels of the current frame color image, and obtains the total volume of the material passing through the belt within a set time period by repeatedly collecting images and calculating for a set number of times, and then obtains the total weight of the material passing through the belt within the time period. In this process, the image is collected by a depth camera, which is an optical scanning method and a non-contact measurement. It does not require direct contact with the material, thus avoiding the problem of easy wear of traditional belt weighing mechanical sensors and the adverse effects of the sensor on the material, while also reducing maintenance costs. Moreover, the detection system of the present invention is smaller in size and easier to install than traditional belt weighing scales. On the other hand, the present invention combines a depth camera with a neural network, which has high precision and stability, can effectively handle complex material flow conditions, and does not require complex calculations during the measurement process, resulting in a smaller amount of calculations and a faster calculation speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The present invention is further described in detail below with reference to the accompanying drawings.

[0015] Figure 1 The figure is a functional block diagram of the system of the present invention.

[0016] Figure 2 Flow chart of the method of the present invention.

[0017] Figure 3 This is a color image captured by the depth camera when the belt of the present invention conveys bulk materials.

[0018] Figure 4 It is an image with the surface contours of all materials output by the neural network of the present invention.

[0019] Figure 5 It is a schematic diagram of the overall material cross-section of a single-frame image of the present invention. DETAILED DESCRIPTION

[0020] like Figure 1As shown, the bulk material flow measurement detection system based on the depth camera includes a collection module and a calculation module connected to the collection module. The collection module includes a depth camera and a belt encoder. The depth camera can be installed directly above the belt. The specific installation structure is the existing technology. The calculation module includes a data analysis calculation and processing unit, which is used to implement the bulk material flow measurement detection method based on the depth camera, that is, to calculate according to the data collected by the collection module to obtain the weight of the material. Specifically, Figure 2 As shown, the bulk material flow measurement detection method based on the depth camera includes the following steps: Step S1, using a depth camera to collect a training color image of a belt that is conveying bulk materials, and using the collected training color image to train a neural network so that the neural network outputs an image with the surface contours of all materials, thereby filtering out false detections caused by the belt or dust.

[0021] Specifically, a depth camera is used to collect multiple training color images, and the material surface contour is marked in each training color image before training the neural network. The neural network of this embodiment selects FCN (Fully Convolutional Networks), and the process of marking the material surface contour is the existing technology.

[0022] Step S2: when the belt is unloaded, a depth camera is used to collect an unloaded depth image of the belt, a row in the unloaded depth image is intercepted to obtain belt row pixels, and the belt height of each pixel in the belt row pixels is obtained; Specifically, using a depth camera to collect n An empty depth image, and according to the formula Calculate the average image , and take the average image , to obtain the belt height of each pixel in the belt row pixel, and the belt height is the depth data of the corresponding pixel, where express n The total depth image obtained by superimposing the unloaded depth images is n ≥1.

[0023] The image overlay process can use OpenCv tools or be programmed in C language, which is a prior art.

[0024] When intercepting the average image, a row where the center of the average image is located, a row with the smallest sinking amount in the average image, or a row corresponding to the belt roller in the average image can be intercepted. The specific row to be intercepted can be preset according to the on-site situation. The specific process of intercepting a row in the image is the existing technology. The depth value of each pixel in the intercepted row of pixels is used as the belt height of the pixel.

[0025] Step S3, when the belt is conveying bulk materials, a color image and a depth image of the belt are collected by using a depth camera, and the current frame color image is input into the neural network trained in step S1 to obtain an image with the material surface contours of all materials, and a row corresponding to the interception position in the unloaded image is intercepted in the image and the current frame depth image to obtain material surface row pixels and depth row pixels, and the material surface height of each pixel in the material surface row pixels is obtained according to the depth row pixels; Specifically, before the current frame color image is input into the neural network, the current frame color image and the current frame depth image collected by the depth camera are matched and aligned. The matching and alignment process can be achieved through the tools provided by the depth camera. i Pixel surface height That is, the depth data of the corresponding pixel in the depth row pixel.

[0026] Step S4, obtaining the material thickness of each pixel in the material surface row pixels according to the material surface height of each pixel in the material surface row pixels and the belt height of each pixel in the belt row pixels, and calculating the cross-sectional area of ​​all materials in the material surface row pixels in combination with the distance between two adjacent pixels in the material surface row pixels, and then obtaining the volume of all materials in the material surface row pixels of the current frame color image; Specifically, according to the formula Get the first pixel in the material surface row i Material thickness in pixels , and according to the formula Get the volume of all materials corresponding to the material surface row pixels of the current frame color image, where: is the cross-sectional area of ​​all materials corresponding to the material surface row pixels of the current frame color image. The cross-sectional area of ​​all materials corresponding to the material surface row pixels of the current frame color image is as follows: Figure 5 As shown, The first pixel in the material row i The cross-sectional area of ​​pixels, L is the distance between two adjacent pixels, v Indicates the belt speed, which is obtained through the belt encoder, 1≤ i ≤ m , m is the number of pixels in the row of pixels on the surface, Indicates the number of pixels in the belt row corresponding to the number of pixels in the material row. i Belt height in pixels.

[0027] Step S5, repeat steps S3 to S4 a set number of times, add up the volumes of all materials obtained in each repetition to obtain the total volume of the material passing through the belt within the set time period, and multiply the total volume by the density of the material to obtain the total weight of the material passing through the belt within the time period. The density of the material is the stacking density of the material stacked to the material thickness. More specifically, the material density can be pre-tested according to the on-site conditions. This process is a prior art.

[0028] The above description is only a preferred embodiment of the present invention, and therefore cannot be used to limit the scope of implementation of the present invention. That is, equivalent changes and modifications made according to the scope of the patent application of the present invention and the contents of the specification should still fall within the scope covered by the patent of the present invention.

Claims

1. A bulk material flow measurement and detection method based on a depth camera, characterized in that: The steps include: Step S1, using a depth camera to collect a training color image of a belt that is conveying bulk materials, and using the collected training color image to train a neural network so that the neural network outputs an image with the surface contours of all materials; Step S2: when the belt is unloaded, a depth camera is used to collect an unloaded depth image of the belt, a row in the unloaded depth image is intercepted to obtain belt row pixels, and the belt height of each pixel in the belt row pixels is obtained; Step S3, when the belt is conveying bulk materials, a color image and a depth image of the belt are collected by using a depth camera, and the current frame color image is input into the neural network trained in step S1 to obtain an image with the material surface contours of all materials, and a row corresponding to the interception position in the unloaded image is intercepted in the image and the current frame depth image to obtain material surface row pixels and depth row pixels, and the material surface height of each pixel in the material surface row pixels is obtained according to the depth row pixels; Step S4, obtaining the material thickness of each pixel in the material surface row pixels according to the material surface height of each pixel in the material surface row pixels and the belt height of each pixel in the belt row pixels, and calculating the cross-sectional area of ​​all materials in the material surface row pixels in combination with the distance between two adjacent pixels in the material surface row pixels, and then obtaining the volume of all materials in the material surface row pixels of the current frame color image; Step S5, repeating steps S3 to S4 for a set number of times, adding up the volumes of all materials obtained in each repetition to obtain the total volume of the materials passing through the belt within a set time period, and then obtaining the total weight of the materials passing through the belt within the time period.

2. The method for measuring and detecting bulk material flow based on a depth camera according to claim 1, characterized in that: In the step S1, a depth camera is used to collect a plurality of training color images, and the neural network is trained after marking the material surface contour in each training color image.

3. The method for measuring and detecting bulk material flow based on a depth camera according to claim 1, characterized in that: In step S2, when the belt is unloaded, a depth camera is used to collect n An empty depth image, and according to the formula Calculate the average image , and take the average image to obtain the belt height of each pixel in the belt row pixel, where express n The total depth image is obtained by superimposing the unloaded depth images.

4. The method for measuring and detecting bulk material flow based on a depth camera according to claim 3 is characterized in that: In step S2, a row intercepted from the average image is a row where the center of the average image is located, a row with the smallest sinking amount in the average image, or a row in the average image corresponding to the belt roller.

5. A bulk material flow measurement and detection method based on a depth camera according to claim 1, 2, 3 or 4, characterized in that: In the step S3, before the current frame color image is input into the neural network, the current frame color image and the current frame depth image collected by the depth camera are matched and aligned, and the first pixel in the material surface row is i Pixel surface height That is, the depth data of the corresponding pixel in the depth row pixel.

6. A bulk material flow measurement and detection method based on a depth camera according to claim 1, 2, 3 or 4, characterized in that: In step S4, according to the formula Get the first pixel in the material surface row i Material thickness in pixels , and according to the formula Get the volume of all materials in the material surface row pixels of the current frame color image, where: is the cross-sectional area of ​​all materials, The first pixel in the material row i The cross-sectional area of ​​pixels, L is the distance between two adjacent pixels, v Indicates belt speed, 1≤ i ≤ m , m is the number of pixels in the row of pixels on the surface, Indicates the first pixel in the material row i The material surface height in pixels, Indicates the number of pixels in the belt row corresponding to the number of pixels in the material row. i Belt height in pixels.

7. The method for measuring and detecting bulk material flow based on a depth camera according to claim 6, characterized in that: The belt speed is obtained through a belt encoder.

8. A bulk material flow measurement and detection method based on a depth camera according to claim 1, 2, 3 or 4, characterized in that: The neural network includes a fully convolutional network.

9. A detection system for bulk material flow measurement detection method based on a depth camera according to any one of claims 1 to 8, characterized in that: include: Acquisition module: including a depth camera and a belt encoder. The depth camera is used to collect training color images, no-load depth images, and multi-frame color images and depth images of the belt when the belt is conveying bulk materials. The belt encoder is used to collect the belt speed. Calculation module: connected to the acquisition module, trains the neural network according to the training color image acquired by the acquisition module, so that the neural network outputs an image with the material surface contours of all materials; obtains the belt height of each pixel in the belt row pixels according to the no-load depth image acquired by the acquisition module; inputs the current frame color image into the neural network trained by step S1 to obtain an image with the material surface contours of all materials, obtains the material surface height of each pixel in the material surface row pixels according to the current frame depth image, obtains the material thickness of each pixel in the material surface row pixels according to the material surface height of each pixel in the material surface row pixels and the belt height of each pixel in the belt row pixels, and calculates the cross-sectional area of ​​all materials in the material surface row pixels in combination with the distance between two adjacent pixels in the material surface row pixels, thereby obtaining the volume of all materials in the material surface row pixels of the current frame color image, and obtaining the total weight of the material passing through the belt within a set time period.

Images