A method for mapping bulk materials in reservoirs using machine learning

Abstract

A method for mapping bulk material within a storage facility containing the bulk material and having a wall is disclosed. The method involves using a depth map acquisition sensor, such as a time-of-flight camera, to acquire multiple training images, each defined by an array of pixels, of bulk material of various topologies within the reservoir. An artificial neural network is trained on these training images to develop an artificial intelligence model that can distinguish between walls and surfaces of bulk materials. Further, the method involves using a depth map acquisition sensor (e.g., a time-of-flight camera) to acquire one or more images of the bulk material to be mapped within the reservoir or another reservoir of similar shape, and then using an artificial intelligence model to map the bulk material within the reservoir or another reservoir of similar shape to identify walls and bulk material within the reservoir or another reservoir of similar shape.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 485,701, filed February 17, 2023.

[0002] Technical Field The present invention relates to techniques for detecting bulk material within a bulk storage vessel, and more particularly to techniques for mapping the topology of bulk material within a bulk storage vessel. [Background technology]

[0003] Various techniques are known for monitoring and / or measuring bulk materials stored within bulk storage vessels. In the agricultural industry, there is a strong desire to monitor various attributes of grain, seeds, or other agricultural products in storage. For example, it is known to measure the temperature and / or moisture of grain or seeds to prevent spoilage and to know when the grain is ready for delivery to an end user. It is also highly desirable to accurately measure the volume of grain or feed in a storage bin. A variety of LIDAR and camera-based sensor techniques have been disclosed for mapping the topology of bulk materials within a reservoir. An example of a storage level monitoring system is disclosed in US Pat. No. 6,223,999, which is incorporated herein by reference. The storage level monitoring system of Patent Document 1 includes an optical sensor such as a LIDAR sensor or a time-of-flight (TOF) camera for detecting the feed level in the feed storage, a circuit board communicatively connected to the sensor for receiving a level signal from the sensor and processing the level signal to generate storage level data, a battery for supplying power to the circuit board and the sensor, a housing for enclosing the circuit board, and a wireless transmitter for transmitting the storage level data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2020102879A1 Brochure Summary of the Invention [Problem to be solved by the invention]

[0005] Distinguishing between the wall of a container and the top surface of a bulk material remains a technical challenge that has thus far limited the accuracy of these conventional sensor technologies. A technological solution to this problem is highly desirable to more accurately monitor bulk materials within the storage facility. [Means for solving the problem]

[0006] The present invention provides a method and system for mapping bulk material within a reservoir by using machine learning to develop an artificial intelligence model that can distinguish between reservoir walls and bulk material.

[0007] An aspect of the present invention is a method for mapping bulk material within a storage facility containing the bulk material and having a wall. That is, the method of mapping bulk material within a reservoir of the present invention involves using a depth map acquisition sensor, such as a time-of-flight camera, to acquire multiple training images, each defined by an array of pixels, of bulk material of various topologies within the reservoir. An artificial neural network is trained on these multiple training images to develop an artificial intelligence model that can distinguish between walls and surfaces of bulk materials. Furthermore, the method of the present invention involves using a depth map acquisition sensor (e.g., a time-of-flight camera) to acquire one or more images of the bulk material to be mapped within the reservoir or another reservoir of similar shape, and then using an artificial intelligence model to map the bulk material within the reservoir or another reservoir of similar shape and identify the walls and bulk material within the reservoir or another reservoir of similar shape.

[0008] Another aspect of the present invention is a system for mapping bulk material within a storage facility containing the bulk material and having a wall. That is, a system for mapping bulk material within a storage facility having walls and containing the bulk material includes a depth map acquisition sensor, such as a time-of-flight camera, for acquiring multiple learning images, each defined by a pixel array, of bulk material of various topologies within the storage facility. The system also includes a processor for training an artificial neural network with the plurality of training images to develop an artificial intelligence model capable of distinguishing between walls and surfaces of bulk materials. A depth map acquisition sensor (eg, a time-of-flight camera) then acquires one or more images of the bulk material to be mapped within the vault or another vault of similar shape. The processor is further configured to use the artificial intelligence model to map the bulk material within the reservoir or another reservoir of similar shape to identify walls and bulk material within the reservoir or another reservoir of similar shape.

[0009] The above has been presented as a simplified summary of the invention in order to provide a basic understanding of the invention. This summary is not an exhaustive overview of the invention. It is not intended to identify essential, critical, or critical elements of the invention or to delineate the scope of the invention. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later. Other aspects of the invention are described below in conjunction with the accompanying drawings.

[0010] Further features and advantages of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0011] [Figure 1]1 is a schematic diagram of a system for mapping bulk material in a storage facility in accordance with an embodiment of the present invention; [Figure 2] 1 is a flow chart illustrating a method for mapping bulk material in a storage facility according to an embodiment of the present invention. [Figure 3] 4 is a flow chart illustrating optional further steps of the method. [Figure 4] 1 is an example of an image in which pixels represent amplitude data. [Figure 5] An example of an image where pixels represent distance data. DETAILED DESCRIPTION OF THE INVENTION

[0012] It should be noted that throughout the accompanying drawings, like features are identified with like reference numerals.

[0013] FIG. 1 illustrates a schematic of a novel system for mapping bulk materials in a repository, in accordance with an embodiment of the present invention. In the embodiment shown in FIG. 1, the system, designated by reference numeral 10, is designed to map bulk material 20 within a storage facility 30 having walls 32 for containing the bulk material. Bin 30 can be any container, silo, receptacle, or storage structure used to store solid particulate or bulk materials, such as grain, seeds, or other such materials, whether agricultural or not. The system 10 includes a depth map acquisition sensor, such as a time-of-flight camera 40, for acquiring multiple training images, each defined by an array of pixels, of bulk material of various topologies within the reservoir. Camera or sensor image data from the depth map acquisition sensor (e.g., time-of-flight camera) 40 may be transmitted to a server or other computing device for image processing as further described below. Camera or sensor image data may be transmitted wirelessly, for example, over a cellular data network using an antenna 50 on the depot. Alternatively, the camera or sensor image data may be transmitted via any other wired or wireless communication means. In this example, the camera or sensor image data is transmitted to a cellular base station 60 which has a gateway to the Internet 70 . The data packets of camera or sensor image data are then sent to a server or other computing device for image processing. In the example of FIG. 1, a server cluster 80 or cloud implementation is shown, although in simpler implementations a single server or computing device may be used. The system 10 of the present invention then includes at least one server or computing device 100 having a processor or CPU 110 (or multiple processors or CPUs) for training an artificial neural network 150 by using multiple training images to develop an artificial intelligence model capable of distinguishing between walls and surfaces of bulk materials. For the sake of completeness, the server or computing device 100 also comprises a memory 120, a communication port 130, and input / output (I / O) devices 140 that cooperate with the processor 110 to implement an artificial neural network for image processing. Once the artificial neural network has been properly trained using the training images, a time-of-flight (TOF) camera 40 then acquires one or more images of the volume of bulk material being mapped within the reservoir. The processor is further configured to use an artificial intelligence model to map the volume of the bulk material within the reservoir to identify walls and bulk material within the reservoir or another reservoir of a similar shape. It should be understood that in alternative embodiments, the calculations for training the model need not be performed by computing device 100; rather, in alternative embodiments, these calculations may be performed in whole or in part by a processor or computer located within the repository. Alternatively, an edge computing paradigm may be adopted, with all or part of the computation being performed by computing resources closer to the depot. As a further alternative, the cloud computing paradigm may be adopted in whole or in part for training the model.

[0014] In one embodiment of the present invention, the system uses a TOF camera 40 to measure the surface of the bulk material and the walls of the reservoir. The system of the present invention uses machine learning (i.e., artificial intelligence) to distinguish between bulk materials and reservoir walls. In one embodiment, the processor is configured to computationally remove the walls of the reservoir and fit the surface of the bulk material to a 3D model of the reservoir.

[0015] The depth map acquisition sensor is a time-of-flight (TOF) camera 40 . The TOF camera 40 can be any suitable device that emits modulated light to illuminate the feed within the bin and the interior surfaces of the bin walls. The TOF camera 40 is configured to capture light reflected from the feed within the bin and from the inner surfaces of the bin walls. The TOF camera 40 is configured to take advantage of the phase shift between the illumination and the reflected light. By measuring the phase shift, the TOF camera 40 can calculate the distance. The TOF camera 40 may use, for example, a solid-state laser or a light-emitting diode (LED) operating in the near-infrared range (ie, approximately 850 nm). The TOF camera 40 may include an imaging sensor having an array of pixels that generate a current for each pixel in response to reflected infrared (IR) light. To detect the phase difference between the illumination and the reflected light, the light source may be pulsed or otherwise modulated with a continuous wave, such as a sine wave or square wave. The TOF camera 40 may have its own illumination source, for example in the form of an LED matrix emitting modulated infrared light. Based on the detection of the reflected waves, two types of images can be formed: (i) a distance image, which is calculated based on the phase difference between the transmitted signal and the reflected signal, and (ii) an amplitude image, which is calculated based on the amplitude of the reflected signal at each pixel location. Distances can be measured for every pixel in a two-dimensional addressable array to generate a depth map. A depth map can be conceptualized as a collection of three-dimensional points, where the more intense a pixel is, the closer the location of the corresponding physical point in three-dimensional space. Alternatively, the depth map can be rendered as a collection of points (point cloud) in three-dimensional space. Three-dimensional points can be mathematically connected to form a mesh onto which a surface can be mapped.

[0016] This system enables a novel method for mapping bulk materials within a repository, as shown in the flow chart of Figure 2. Method 200 involves acquiring 210 multiple training images, each defined by an array of pixels, of bulk material of various topologies within the reservoir using a depth map acquisition sensor, such as a time-of-flight camera. The method 200 further requires step 220 of training an artificial neural network with a plurality of training images to develop an artificial intelligence model capable of distinguishing between walls and surfaces of bulk materials. The method 200 further requires step 230 of acquiring one or more images of the bulk material to be mapped within the vault or another vault of similar shape using a depth map acquisition sensor (e.g., a time-of-flight camera). The method 200 further requires step 240 of using an artificial intelligence model to map the bulk material within the storage bin or another storage bin of similar shape to identify walls and bulk material within the storage bin or another storage bin of similar shape. For clarity, a model can be trained using a first reservoir, and then the model can be used in a second reservoir to identify the topology of the bulk material in the second reservoir. In a variant, the model can be trained using a first set of repositories, and then the model can be deployed for use with one or more other repositories, i.e., for use with a single repositories or a second set of repositories.

[0017] In one embodiment, the method may include optional further steps, as shown in FIG. The step 240 of mapping the volume of the bulk material may include a step 250 of identifying the walls of the reservoir from the bulk material within the reservoir or another reservoir of similar shape, followed by a step 260 of removing the pixels representing the walls and fitting the pixels representing the bulk material to a predetermined 3D model of the reservoir or another reservoir of similar shape.

[0018] In one embodiment, training of an artificial neural network on a plurality of training images is performed using both pixel amplitude and distance data. In another embodiment, training of the artificial neural network on the plurality of training images is performed using pixel amplitude data only. In another embodiment, training of the artificial neural network on the training images is done using pixel range data only.

[0019] This method can be performed using a deep convolutional neural network (DCNN), although other types of artificial neural networks may be used to achieve similar results. The deep convolutional neural network may be, for example, a region-based convolutional neural network (R-CNN), Fast R-CNN, GoogleNet, VGGNet, or ResNet (residual neural network). DCNN employs a layer structure to process the depth and amplitude data of acquired TOF camera images. Deep convolutional neural networks take training images (i.e., human-marked images that serve as ground truth) as input and use them to train a classifier. Typically, a DCNN has four types of layers: convolutional layers, activation layers, pooling layers, and fully connected layers. In the convolutional layer, DCNN applies convolutional filters to the image to extract image features. Convolution is performed by multiplying the input values ​​from the neural network by weights. During this multiplication, the kernel (in the case of a 2D weight array) or filter (in the case of a 3D matrix) moves over the image. During convolution, each filter multiplies a different input value by a weight and sums them to produce a particular value at each filter position. This convolution map is then processed with a nonlinear activation layer, e.g., rectified linear units (ReLu), which replaces all negative values ​​in the image with zeros. In the pooling layer, the image size is reduced successively. For example, for each group of four pixels, either keep the pixel with the maximum value (called max pooling) or keep only the average value (average pooling). After multiple convolutions and pooling, the result is a multi-layer perceptron, a fully connected neural network. The DCNN can then receive a newly acquired image from the TOF camera, including amplitude and depth data for the pixels of the newly acquired image. This allows the fully connected neural network to learn to recognize whether a pixel represents the surface of a bulk material or the wall of a reservoir.

[0020] FIG. 4 is an example of an image acquired by a TOF camera, where pixels represent amplitude data. Using this amplitude data, the DCNN can distinguish between the walls of the reservoir 310 and the bulk material 300.

[0021] FIG. 5 is an example of an image captured by a TOF camera, where pixels represent distance data. Using this depth data, the DCNN can distinguish between the walls of the reservoir 310 and the bulk material 300.

[0022] The reservoir in the illustrated embodiment is a cylindrical reservoir having a cylindrical wall defining a fixed radius of curvature. The concepts of the present invention may be adapted or modified for use with other shapes of reservoirs.

[0023] In one embodiment, the artificial intelligence model includes multiple material-specific sub-models for different types of bulk materials. For example, one AI sub-model may be developed for grains and another for seeds. The processor can receive data indicative of the type of bulk material and apply material-specific sub-models to the image to better distinguish between walls and bulk material. Bulk material types can be characterized by the color, granularity, reflectivity, or other properties of the material. A type of bulk material can also be characterized by its angle of repose, a bulk material property that indicates how a particular bulk material will pile up depending on its particle shape and coefficient of friction, for example.

[0024] The method can be implemented in hardware, software, firmware, or any suitable combination thereof. That is, when implemented as software, the computer-readable medium includes coded instructions that, when loaded into memory and executed on a processor of a computing device, cause the computing device to perform any of the method steps described above. These method steps are implemented as software, i.e., as coded instructions stored on a computer-readable medium, which when loaded into memory and executed by a microprocessor of a mobile device, perform the aforementioned steps. A computer-readable medium may be any means of containing, storing, communicating, propagating or transporting a program for use by or in connection with an instruction execution system, apparatus or device. The computer readable medium can be an electronic, magnetic, optical, electromagnetic, infrared, or any semiconductor system or device. For example, computer executable code for performing the methods disclosed herein may be tangibly recorded on a computer readable medium, including, but not limited to, a floppy disk, a CD-ROM, a DVD, RAM, ROM, EPROM, flash memory, or any suitable memory card. The method can also be implemented in hardware. Hardware implementations may employ discrete logic circuits having logic gates for performing logical functions on data signals, application specific integrated circuits (ASICs) having appropriate combinatorial logic gates, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0025] For purposes of interpreting this specification, when referring to elements of various embodiments of the invention, the articles "a," "an," "the," and "said" are intended to mean that there are one or more of the element. The terms “comprising,” “including,” “having,” “entailing,” and “involving,” and their verb tense variations, are intended to be inclusive and open-ended, meaning that there may be additional elements other than the listed elements.

[0026] This new technology has been described with reference to specific implementations and configurations that are intended to be illustrative only. Those skilled in the art will recognize that many obvious variations, improvements, and modifications may be made without departing from the inventive concepts presented in this application. Accordingly, it is intended that the scope of the exclusive rights sought by applicants be limited only by the appended claims.

Claims

1. 1. A method of mapping bulk material within a storage facility containing the bulk material and having a wall, comprising: acquiring a plurality of training images, each defined by an array of pixels, of bulk material of various topologies within the reservoir using a depth map acquisition sensor; training an artificial neural network with the plurality of training images to develop an artificial intelligence model capable of distinguishing between the wall and a surface of the bulk material; acquiring one or more images of the bulk material to be mapped within the storage facility using the depth map acquisition sensor; using the artificial intelligence model to map the bulk material within the bin to identify the walls and the bulk material within the bin or another bin of similar shape; 1. A method for mapping bulk material in a storage facility containing the bulk material and having a wall, comprising:

2. Mapping the volume of the bulk material comprises: identifying the walls of the reservoir from the volume of the bulk material within the reservoir or another reservoir of similar shape, and then removing pixels representing the walls; fitting pixels representing said bulk material to a predetermined 3D model of said reservoir or another reservoir of similar shape; 10. A method for mapping bulk material within a storage facility containing the bulk material and having a wall, comprising:

3. 2. The method of mapping bulk material within a walled storage facility containing bulk material of claim 1, wherein training the artificial neural network on the plurality of training images is performed using both amplitude data and distance data for the pixels.

4. 3. The method of mapping bulk material within a walled storage facility containing bulk material as described in claim 2, wherein training the artificial neural network on the plurality of training images is performed using both amplitude data and distance data for the pixels.

5. 2. The method of mapping bulk material within a walled storage facility containing bulk material of claim 1, wherein training the artificial neural network on the plurality of training images is performed using amplitude data of the pixels.

6. 2. The method of mapping bulk material within a walled storage facility containing bulk material of claim 1, wherein said training of said artificial neural network on said plurality of training images is performed using said pixel distance data.

7. 2. The method of mapping bulk material within a storage facility containing the bulk material and having a wall, as set forth in claim 1, wherein the neural network is a deep convolutional neural network.

8. 2. The method of mapping bulk material within a storage facility containing the bulk material and having walls, as set forth in claim 1, wherein the depth map acquisition sensor is a time-of-flight (TOF) camera.

9. 10. The method of mapping bulk material within a reservoir containing bulk material and having a wall, as set forth in claim 1, wherein the reservoir is a cylindrical reservoir having a cylindrical wall defining a fixed radius of curvature.

10. 2. The method of mapping bulk material within a walled storage facility containing bulk material of claim 1, wherein the artificial intelligence model includes multiple material-specific sub-models for different types of bulk material.

11. 1. A system for mapping bulk material within a storage facility containing the bulk material and having a wall, comprising: a depth map acquisition sensor for acquiring a plurality of training images of bulk material of various topologies within the reservoir, each training image being defined by an array of pixels; a processor for training an artificial neural network with the plurality of training images to develop an artificial intelligence model capable of distinguishing between the wall and a surface of the bulk material; Equipped with the depth map acquisition sensor then acquires one or more images of the bulk material to be mapped within the storage bin or another storage bin of similar configuration; The system for mapping bulk material within a storage facility containing bulk material and having walls, wherein the processor is further configured to use the artificial intelligence model to map the bulk material within the storage facility or another storage facility of similar shape to identify the walls and the bulk material within the storage facility or another storage facility of similar shape.

12. The processor: identifying the walls of the reservoir from the bulk material within the reservoir or another reservoir of similar shape, and then removing pixels representing the walls; fitting pixels representing said bulk material to a predetermined 3D model of said reservoir or another reservoir of similar shape; 12. The system for mapping bulk material in a storage facility containing bulk material and having a wall as recited in claim 11, configured to map the bulk material by:

13. 12. The system for mapping bulk material within a storage facility containing bulk material and having walls, as described in claim 11, wherein the processor trains the artificial neural network on the plurality of training images using both amplitude data and distance data for the pixels.

14. 13. The system for mapping bulk material within a walled storage facility containing bulk material as described in claim 12, wherein the processor trains the artificial neural network on the plurality of training images using both amplitude data and distance data for the pixels.

15. 12. The system for mapping bulk material within a walled storage facility containing bulk material as described in claim 11, wherein the processor trains the artificial neural network on the plurality of training images using amplitude data of the pixels.

16. 12. The system for mapping bulk material within a storage facility containing bulk material and having walls, as described in claim 11, wherein the processor trains the artificial neural network on the plurality of training images using the pixel distance data.

17. 12. The system for mapping bulk material within a storage facility containing bulk material and having walls as described in claim 11, wherein the neural network is a deep convolutional neural network.

18. 12. The system for mapping bulk material within a storage facility containing the bulk material and having walls, as set forth in claim 11, wherein the depth map acquisition sensor is a time-of-flight (TOF) camera.

19. 12. The system for mapping bulk material within a walled reservoir containing bulk material as set forth in claim 11, wherein the reservoir is a cylindrical reservoir having a cylindrical wall defining a fixed radius of curvature.

20. 12. The system for mapping bulk materials within a walled storage facility containing bulk materials of claim 11, wherein the artificial intelligence model includes multiple material-specific sub-models for different types of bulk materials.

Images