Determination of material volume and density based on sensor data

Abstract

A controller (145) can receive information identifying regions of interest from a plurality of candidate regions of interest (140, 175), including locations on and outside a machine (105). The controller (145) can use one or more first sensor devices (160) to obtain data identifying material located at the region of interest; and generate a graphical representation based on the data. The controller (145) can use one or more second sensor devices (150) to determine at least one of the positions or orientations of one or more portions (120, 130, 135) of the machine (105); and identify a portion of the graphical representation based on at least one of the positions or orientations of the one or more portions (120, 130, 135). The portion may correspond to material located at the region of interest. The controller (145) can use one or more computational models to determine the volume of the material based on the portion of the graphical representation.

Description

Technical Field

[0001] This disclosure generally relates to a controller, and for example to a controller for determining the volume and density of a material moved by a machine. Background Technology

[0002] When a dump truck is positioned where the excavator's bucket can dump material into the dump truck's cargo bed, the excavator can load material into the dump truck. The excavator can make one or more passes to load material into the truck cargo bed. Individuals (e.g., excavator operators and / or excavator owners) may wish to determine measurements of the excavator's productivity (e.g., during work shifts).

[0003] For example, an individual might want information about the volume and / or density of material loaded into a truck bed, the volume and / or density of material removed by the bucket (e.g., from the ground), and / or the volume and / or density of material in a pile near the excavator. Excavators and dump trucks cannot provide this information to an individual. Therefore, the individual may resort to manual measurement of such volume and / or density. Obtaining such manual measurement is a time-consuming process. Additionally, manual measurements may be inaccurate. Inaccuracy in manual measurements may lead to inaccurate changes and / or adjustments to the excavator (e.g., inaccurate changes and / or adjustments to the excavator configuration, inaccurate changes and / or adjustments to excavator components, etc.).

[0004] U.S. Patent Application Publication No. 20200087893 ('893 Publication) discloses a mobile working machine comprising a container movably supported by a frame. The '893 Publication also discloses that the container is configured to receive contents, and that actuators are configured to controllably drive movement of the container relative to the frame. The '893 Publication further discloses determining the density, volume, or weight of soil in the container of the working machine (e.g., the bucket of an excavator).

[0005] Although the '893 publication discloses the determination of the density, volume, or weight of soil in the excavator bucket, the '893 publication does not disclose the determination of the density or volume of soil in the truck bed of a dump truck, nor does it disclose the determination of the density or volume of soil in a pile near the excavator.

[0006] The controller disclosed herein addresses one or more of the problems described above and / or other problems in the art. Summary of the Invention

[0007] In some embodiments, a method performed by a controller of a machine includes: receiving information identifying a region of interest from a plurality of candidate regions of interest, wherein the plurality of candidate regions of interest include locations on and outside the machine; acquiring an image identifying material located at the region of interest using one or more first sensor devices associated with the machine; generating a three-dimensional graphical representation based on the image; determining at least one of the positions or orientations of one or more portions of the machine using one or more second sensor devices of the machine; determining the coordinates of the material at the region of interest relative to the machine based on at least one of the positions or orientations of the one or more portions; identifying a portion of the three-dimensional graphical representation based on the coordinates, wherein the portion corresponds to the material located at the region of interest; determining the volume of the portion using one or more computational models; determining the volume of the material based on the volume of the portion; and performing an action based on the volume of the material.

[0008] In some embodiments, a machine includes one or more memories; and one or more processors configured to: receive information identifying regions of interest from a plurality of candidate regions of interest, wherein the plurality of candidate regions of interest include locations on and outside the machine; obtain data identifying material located at the region of interest using one or more first sensor devices associated with the machine; generate a three-dimensional graphical representation of the material based on the data; determine at least one location or orientation of one or more portions of the machine using one or more second sensor devices of the machine; identify a portion of the three-dimensional graphical representation based on at least one location or orientation of the one or more portions, wherein the portion corresponds to material located at the region of interest; determine the volume of the portion using one or more computational models; and determine the volume of the material based on the volume of the portion.

[0009] In some embodiments, a system includes one or more first sensor devices associated with a machine; one or more second sensor devices associated with the machine; and a controller for the machine, the controller being configured to: receive information identifying a region of interest from a plurality of candidate regions of interest, wherein the plurality of candidate regions of interest include locations on and outside the machine; use the one or more first sensor devices to obtain data identifying material located at the region of interest; generate a graphical representation based on the data; use the one or more second sensor devices to determine at least one of the locations or orientations of one or more parts of the machine; identify a portion of the graphical representation based on at least one of the locations or orientations of the one or more parts, wherein the portion corresponds to material located at the region of interest; and determine the volume of the material based on the portion of the graphical representation using one or more computational models. Attached Figure Description

[0010] Figure 1 This is a diagram of the example implementation described in this article.

[0011] Figure 2 This is a diagram of the example system described in this article, which can be compared with... Figure 1 The machines are implemented in conjunction with each other.

[0012] Figure 3 This is a flowchart illustrating an example process for determining the volume and density of a material based on sensor data. Detailed Implementation

[0013] This disclosure relates to a controller for a machine that determines the volume and / or density of material located in multiple regions of interest (e.g., locations on and outside the machine). The term "machine" can refer to any machine that performs operations associated with, for example, mining, construction, agriculture, transportation, or other industries. Furthermore, one or more tools may be attached to the machine.

[0014] Figure 1 This is a diagram of the example implementation scheme 100 described in this article. Figure 1 Example implementation 100 includes machines 105 and 110. For example... Figure 1 As shown, machine 105 is embodied as a load machine, such as an excavator. Alternatively, machine 105 can be another type of load machine, such as a bulldozer, a wheeled load machine, and / or similar machines. Figure 1 As shown, machine 110 is embodied as a hauling machine, such as a mining truck, hauling truck, dump truck, and / or similar machine. In some instances, machine 105 can load (or move) materials into machine 110 (e.g., load (or move) them into the truck bed of machine 110).

[0015] like Figure 1 As shown, machine 105 includes a ground engagement component 115, a cab 120, and a machine body 125. The ground engagement component 115 can be configured to propel machine 105. The ground engagement component 115 may include tracks (such as...). Figure 1 (As shown). Alternatively, the ground engagement component 115 may include wheels, rollers, etc. The ground engagement component 115 may be mounted on the machine body 125 and driven by one or more engines and transmissions (not shown).

[0016] The operator's cab 120 is supported by the machine body 125 and a rotating frame (not shown). The operator's cab 120 includes an integrated display 122 and operator controls 124, such as an integrated joystick. The operator controls 124 may include one or more input components to generate signals that control the movement of the machine 105.

[0017] For autonomous machines, the operator control unit 124 may not be designed for operator use, but rather can be designed to operate independently of an operator. In this case, for example, the operator control unit 124 may include one or more input components that provide input signals for use by another component without any operator input. The machine body 125 is mounted on a rotating frame (not shown).

[0018] like Figure 1 As shown, machine 105 includes a boom 130, a joystick 135, and a tool 140. The boom 130 is pivotally mounted at the proximal end of machine body 125 and is hinged relative to machine body 125 via one or more fluid-actuated cylinders (e.g., hydraulic or pneumatic cylinders), electric motors, and / or other electromechanical components. The joystick 135 is pivotally mounted at the distal end of boom 130 and is hinged relative to boom 130 via one or more fluid-actuated cylinders, electric motors, and / or other electromechanical components. The tool 140 is mounted at the distal end of joystick 135 and is hinged relative to joystick 135 via one or more fluid-actuated cylinders, electric motors, and / or other electromechanical components. The tool 140 may be a bucket (e.g., a shovel). Figure 1 (as shown) or any other tool that can be mounted on the joystick 135.

[0019] like Figure 1 As shown, machine 105 includes controller 145 (e.g., electronic control module (ECM)), one or more inertial measurement units (IMUs) 150 (hereinafter referred to individually as "one IMU 150" and collectively as "a plurality of IMUs 150"), load sensor device 155, one or more stereo camera devices 160 (hereinafter referred to individually as "one stereo camera device 160" and collectively as "a plurality of stereo camera devices 160"), and wireless communication component 165.

[0020] The controller 145 can control and / or monitor the operation of the machine 105. For example, the controller 145 can control and / or monitor the operation of the machine 105 based on signals from the operator control unit 124, from the IMU 150, from the payload sensor device 155, and / or from the stereo camera device 160.

[0021] IMU 150 includes one or more devices capable of receiving, generating, storing, processing, and / or providing signals indicating the position and orientation of components of machine 105 on which IMU 150 is mounted. For example, IMU 150 may include one or more accelerometers and / or one or more gyroscopes. The one or more accelerometers and / or one or more gyroscopes generate and provide signals that can be used to determine the position and / or orientation of IMU 150 relative to a reference frame, and thus determine the position and / or orientation of components. Figure 1 As shown, the IMU 150 is installed in different locations on components or parts of the machine 105, such as on the cab 120, boom 130, joystick 135, and tool 140.

[0022] The load sensor device 155 may include one or more sensor devices capable of sensing the mass (or weight) of a material (e.g., material loaded in tool 140) and generating a signal indicating that mass. The load sensor device 155 may include strain gauges, piezoelectric sensors, pressure sensors, pressure transducers, and / or similar sensor devices. Figure 1 As shown, the load sensor device 155 can be mounted on the tool 140.

[0023] The stereo camera device 160 may include one or more sensor devices capable of acquiring data that can be used (e.g., by the controller 145) to generate a three-dimensional graphical representation of a region associated with machine 105. As an example, the stereo camera device 160 can acquire an image of the region associated with machine 105. Figure 1 As shown, the stereo camera device 160 is mounted at different locations on components or parts of the machine 105, such as on the cab 120, boom 130, and joystick 135. As an alternative to or supplement to the stereo camera device 160, the machine 105 may include light detection and ranging (LIDAR) devices, sensing sensors, and / or similar devices.

[0024] Wireless communication component 165 may include one or more devices capable of communicating with one or more other machines (e.g., machine 110) and / or one or more other devices, as described herein. Wireless communication component 165 may include a transceiver, separate transmitters and receivers, antennas, etc. Wireless communication component 165 may communicate with one or more machines using short-range wireless communication protocols, such as Bluetooth. Low power consumption, Bluetooth Wi-Fi, Near Field Communication (NFC), Z-wave, ZigBee, Institute of Electrical and Electronics Engineers (IEEE) 802.154, etc.

[0025] Additionally or alternatively, the wireless communication component 165 may communicate with one or more machines via a network, which includes one or more wired and / or wireless networks, such as wireless local area networks (LANs), cellular networks (e.g., Long Term Evolution (LTE) networks, Code Division Multiple Access (CDMA) networks, 3G networks, 4G networks, 5G networks, or another type of cellular network), public land mobile networks (PLMNs), wide area networks (WANs), metropolitan area networks (MANs), telephone networks (e.g., public switched telephone networks (PSTNs)), private networks, self-organizing networks, intranets, the Internet, fiber-optic-based networks, cloud computing networks, and / or combinations of these or other types of networks.

[0026] like Figure 1 As shown, machine 110 includes a wireless communication component 170 and a truck bed 175. The wireless communication component 170 may be similar to the wireless communication component 165 described above. The truck bed 175 may be used to receive material 180 (e.g., loaded into the truck bed 175 using tool 140 of machine 105). Material 180 may include ground material (e.g., material obtained from soil).

[0027] As mentioned above, providing Figure 1 As an example. Other instances may differ from the combination. Figure 1 As described.

[0028] Figure 2 This is a diagram of the example system 200 described in this article, which can be used with... Figure 1 The machine (e.g., machine 105) is associated with it. Figure 2 As shown, system 200 includes a controller 145, one or more IMUs 150 (e.g., IMUs 150-1 to 150-M (M≥1)), a load sensor device 155, a stereo camera device 160 (e.g., stereo camera devices 160-1 to 160-N (N≥1)), and a device 230. As an alternative to or supplement to the stereo camera device 160, system 200 may include a LiDAR device, a sensing sensor, and / or similar devices.

[0029] Controller 145 may include one or more processors 210 (hereinafter referred to individually as "a processor 210" and collectively as "a plurality of processors 210") and one or more memories 220 (hereinafter referred to individually as "a memory 220" and collectively as "a plurality of memories 220"). Processor 210 is implemented in hardware, firmware, and / or a combination of hardware and software. Processor 210 includes a central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), microprocessor, microcontroller, digital signal processor (DSP), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), or another type of processing unit. Processor 210 can be programmed to perform functions.

[0030] Memory 220 includes random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic storage, and / or optical storage) for storing information and / or instructions used by processor 210 to execute functions. For example, when executing a function, controller 145 (e.g., using processor 210 and memory 220) can acquire data from one or more IMUs 150, load sensor devices 155, and / or stereo camera devices 160, and can determine the volume and / or density of material 180 moved by or to be moved by machine 105 in real time or near real time.

[0031] IMU 150 can be configured to transmit attitude information to controller 145 so that controller 145 can determine the volume of material 180. The attitude information may include information about the position and / or orientation of one or more parts of machine 105. For example, the attitude information may include information identifying the swing angle of cab 120, the angle of boom 130, and / or the angle of joystick 135.

[0032] IMU 150 can be configured to periodically (e.g., every second, every minute, upon triggering, etc.) transmit attitude information to controller 145. In some instances, IMU 150 may be pre-configured with a time period for transmitting attitude information. Alternatively, the time period for transmitting attitude information can be determined by an operator associated with machine 105. Alternatively, the time period for transmitting attitude information can be determined by controller 145 (e.g., based on historical attitude transmission data about machine 105).

[0033] Historical posture transmission data may include historical data regarding the time period used to transmit posture information, the frequency of movement of machine 105, and / or similar information. In some cases, IMU 150 may be configured to transmit posture information to controller 145 based on a request from controller 145. For example, controller 145 may transmit a request for posture information based on detecting movement of machine 105, based on a request for posture information from the operator of machine 105, and / or based on a request to determine the volume and / or density of material 180.

[0034] The load sensor device 155 can be configured to transmit load information to the controller 145 so that the controller 145 can determine the density of the material 180. The load information may include information identifying the weight and / or mass of the material 180 (e.g., loaded into the tool 140 of machine 105 and / or loaded into the truck bed 175 of machine 110).

[0035] Load sensor device 155 can be configured to periodically (e.g., every second, every minute, upon triggering, etc.) transmit load information to controller 145. In some instances, load sensor device 155 may be pre-configured with a time period for transmitting load information. Alternatively, the time period for transmitting load information may be determined by an operator associated with machine 105. Alternatively, the time period for transmitting load information may be determined by controller 145 (e.g., based on historical load transmission data about machine 105).

[0036] Historical load transfer data may include historical data regarding the time period used to transfer load information and / or the frequency of movement of machine 105. Load sensor device 155 may be configured to transfer load information to controller 145 based on a request from controller 145. For example, controller 145 may transfer a request for load information based on receiving a request to determine density, based on detecting movement of machine 105 (e.g., based on information from IMU 150), and / or based on a request for load information.

[0037] The stereo camera device 160 can be configured to acquire image data and transmit the image data to the controller 145 so that the controller 145 can determine the volume and / or density of the material 180. The image data may include images of the area associated with machine 105 (e.g., the area surrounding machine 105). The image data can identify regions of interest associated with machine 105. Regions of interest may include the location of the tool 140 of machine 105, the location of the truck bed 175 of machine 110, and / or the location of the material pile 180.

[0038] The stereo camera device 160 can be configured to periodically (e.g., every second, every minute, upon triggering, etc.) transmit image data to the controller 145. In some instances, the stereo camera device 160 may be pre-configured with a time period for transmitting image data. Alternatively, the time period for transmitting image data may be determined by an operator associated with machine 105. Alternatively, the time period for transmitting image data may be determined by the controller 145 (e.g., based on historical image data about machine 105).

[0039] Historical image transmission data may include historical data regarding the time period for transmitting image data, the frequency of movement of machine 105, and the frequency of requests to determine volume and / or density. In some instances, stereo camera device 160 may be configured to transmit image data to controller 145 based on requests from controller 145. For example, controller 145 may transmit a request for image data based on detected movement of machine 105 (e.g., movement associated with moving material into truck bed 175 of machine 110), a request for image data, and / or a request to determine the volume and / or density of material 180.

[0040] In some instances, the stereo camera device 160 may perform one or more object detection (or object recognition) operations to identify regions of interest in the image data. For example, the controller 145 may provide the stereo camera 160 with information to identify regions of interest, and the stereo camera device 160 may perform one or more object detection operations to identify regions of interest in the image data.

[0041] Device 230 may include one or more devices capable of monitoring the volume and / or density of material moved by different machines (e.g., machine 105, machine 110, and / or other machines). Device 230 may include server equipment (e.g., host server, web server, application server, etc.), computers (e.g., laptop computers, desktop computers, etc.), user equipment (e.g., mobile devices, laptop computers, etc.), cloud equipment, etc. In some instances, device 230 may be included within machine 110.

[0042] Controller 145 may acquire data from one or more IMUs 150, load sensor devices 155, and / or stereo camera devices 160 to determine the volume of material 180 and / or the volume of material 180 moved by machine 105, as described in more detail below. In some instances, controller 145 may receive requests to determine the volume (and / or density) of material moved by machine 105 and / or to be moved (e.g., the volume and / or density of material 180 loaded into truck bed 175, the volume and / or density of material 180 in tool 140, and / or the volume and / or density of material pile 180 within a threshold distance of machine 105 and / or machine 110).

[0043] Controller 145 can receive requests from the operator of machine 105, the operator of machine 110, and / or the user of device 230. For example, the operator of machine 105 can submit a request using integrated display 122, and controller 145 can receive the request from integrated display 122. Alternatively, the operator of machine 110 can transmit the request via wireless communication component 170 (of machine 110), and controller 145 can receive the request via wireless communication component 165 (of machine 105). Alternatively, controller 145 can receive the request from device 230 via wireless communication component 165 (of machine 105). In some instances, controller 145 may provide a user interface (e.g., a graphical user interface for display), and the user interface can be used to submit requests.

[0044] In some instances, controller 145 may receive (as part of a request) information identifying regions of interest from multiple candidate regions of interest. Regions of interest may correspond to locations, regions, areas, and / or similar geographical information. Multiple candidate regions of interest may include locations on and outside the machine 105. For example, multiple candidate regions of interest may include the truck bed 175 of machine 110, the tool 140 of machine 105, and / or the material pile 180 within a threshold distance of machine 105 and / or machine 110. Therefore, the information identifying the regions of interest (hereinafter referred to as "region of interest information") may identify the truck bed 175, the tool 140, and / or the material pile 180.

[0045] The region of interest information may include information identifying machine 110 (e.g., type of machine 110 and / or size of machine 110), information identifying truck bed 175 (e.g., type of truck bed 175, size of truck bed 175 and / or position of truck bed 175 relative to machine 110), information identifying tool 140 (e.g., type of tool 140 and / or size of tool 140) and / or information identifying material pile 180 (e.g., position of material pile 180 (e.g., relative to machine 105 and / or machine 110), size of material pile 180 and shape of material pile 180).

[0046] In some instances, controller 145 may obtain information identifying truck bed 175 from one or more memories (e.g., memory 220) associated with machine 105 based on information identifying machine 110. Controller 145 may also obtain information identifying tool 140 from one or more memories based on a request to identify tool 140 as a region of interest.

[0047] In some instances, based on a received request, controller 145 can acquire data associated with a region of interest identified in the request. For example, based on receiving the request, controller 145 can direct one or more stereo camera devices 160 to the region of interest and acquire data (e.g., an image) including the region of interest. The image can identify material 180 located at the region of interest. In some instances, controller 145 can store the image in one or more memories.

[0048] In some cases, controller 145 may cause one or more stereo camera devices 160 to acquire images each time movement of machine 105 is detected (e.g., movement associated with machine 105 moving material 180). Controller 145 may detect movement of machine 105 based on information obtained from one or more IMUs 150.

[0049] In some embodiments, machine 110 may transmit arrival information via wireless communication component 170, indicating that machine 110 will arrive to obtain material 180 (from machine 105), indicating an estimated arrival time, and / or indicating the expected location of machine 110 upon arrival (e.g., relative to machine 105). Alternatively, arrival information may indicate that machine 110 has arrived (and is ready to obtain material 180 from machine 105) and indicates the actual location of machine 110 (e.g., relative to machine 105).

[0050] The controller 145 can receive arrival information via the wireless communication component 165 and, based on the arrival information, enable one or more stereoscopic imaging devices 160 to acquire images. For example, the controller 145 can enable one or more stereoscopic camera devices 160 to acquire an image including the expected location at an estimated time, or to acquire an image including the actual location based on the received arrival information.

[0051] The controller 145 (and / or one or more stereo camera devices 160) can analyze data (e.g., images) to identify regions of interest. For example, the controller 145 (and / or one or more stereo camera devices 160) can analyze the image using one or more object detection techniques (e.g., Single Lens Detector (SSD) technology, One-View-Only (YOLO) technology, etc.) to identify regions of interest. In some instances, the controller 145 can process the image using one or more image processing techniques before analyzing it. For example, the controller 145 can combine the image using one or more image processing techniques before analyzing it.

[0052] In some instances, arrival information may indicate that one or more identification elements (e.g., one or more machine-readable optical markers) are equipped with truck bed 175 (e.g., located at one or more corners of truck bed 175). The identification elements enable controller 145 (and / or one or more stereo camera devices 160) to identify truck bed 175 during target detection operations performed on the image by controller 145 (and / or one or more stereo camera devices 160). Alternatively, controller 145 (and / or one or more stereo camera devices 160) may analyze the image using one or more target detection techniques (e.g., discussed above) to identify truck bed 175 without using identification elements.

[0053] Controller 145 can generate a three-dimensional (3D) graphical representation based on an image. For example, as a result of identifying a region of interest in data (e.g., an image), controller 145 can generate a 3D graphical representation. The 3D graphical representation can represent the area (or a portion of the area around machine 105) including the region of interest. In this respect, the 3D graphical representation enables controller 145 to determine the 3D features of the area around machine 105, including the 3D features of the region of interest (e.g., the volume of the region of interest).

[0054] Controller 145 can generate a disparity map based on an image. Controller 145 can generate the disparity map using one or more data processing techniques (e.g., one or more data processing techniques for generating the disparity map). Controller 145 can also generate a 3D graphical representation based on the disparity map using one or more image processing techniques (e.g., for generating a 3D graphical representation based on the disparity map). As an example, the 3D graphical representation may include a 3D point cloud.

[0055] Controller 145 can determine the position and / or orientation of one or more parts of machine 105. For example, controller 145 can obtain attitude information from one or more IMUs 150 in a manner similar to that described above. Controller 145 can determine the position and / or orientation of one or more parts of machine 105 based on the attitude information. For example, controller 145 can determine the swing angle of cab 120, the angle of boom 130, and / or the angle of joystick 135 based on the attitude information.

[0056] The controller 145 can determine the coordinates of the region of interest relative to the machine 105 (e.g., the coordinates of the material 180 at the region of interest) based on the position and / or orientation of one or more portions. For example, the controller 145 can determine the coordinates of the region of interest relative to a specific portion of the machine 105. For example, the controller 145 can consider a specific portion of the machine 105 as the center point of a 3D graphical representation (e.g., coordinates "0, 0, 0" in the 3D graphical representation), and the coordinates of the region of interest can correspond to the 3D coordinates of the 3D graphical representation relative to the center point.

[0057] The controller 145 can identify a portion of the 3D graphical representation corresponding to the region of interest (hereinafter referred to as the "3D portion") based on coordinates. In some instances, the 3D portion may correspond to a truck bed 175 including material 180. Alternatively or additionally, the 3D portion may correspond to a tool 140 including material 180. Alternatively or additionally, the 3D portion may correspond to a material pile 180. The controller 145 can identify the 3D portion to conserve computational resources that will be additionally used to process the entire 3D graphical representation to determine the volume of material 180 at the region of interest.

[0058] The controller 145 may determine the volume of the 3D portion using one or more computational models, one or more computational algorithms, and / or other machine algorithms that can be used to determine the volume of the 3D graphical representation. In some cases, the 3D portion may correspond to a material stack 180. Therefore, the volume of the 3D portion may correspond to the volume of the material 180.

[0059] The controller 145 can determine the volume of the 3D portion in real time or near real time. In this respect, the controller 145 can determine the volume of the 3D portion periodically (e.g., per second, per minute, and / or similar time intervals). Additionally or alternatively, the controller 145 can determine the volume of the 3D portion each time movement of the machine 105 is detected (e.g., movement associated with the movement of material 180 by the machine 105). The controller 145 can detect the movement of the machine 105 as described above.

[0060] Because the controller 145 can determine the volume of the 3D portion in real time, the volume of the 3D portion can change over a period of time. For example, suppose the region of interest is the truck bed 175. At the first moment, before the machine 105 loads any material 180 into the truck bed 175, the volume of the 3D portion can correspond to the volume of the truck bed 175 when it is empty.

[0061] Assume that machine 105 loads a first portion of material 180 into truck bed 175 after a first time. At a second time (after the first time), the volume of the 3D portion may correspond to the volume of truck bed 175 including the first portion of material 180 (e.g., the volume of the first portion of material 180 other than the volume of truck bed 175 when it is truck bed 175), etc. Controller 145 may store (e.g., in one or more memories) information identifying the different volumes of the 3D portion during that time period (e.g., during a work shift).

[0062] The controller 145 can determine the volume of the material 180 based on the volume of the 3D portion. In some cases, the volume of the 3D portion can correspond to the volume of the material 180, as described above. In some cases, the 3D portion can correspond to a truck bed 175 that includes the material 180 (if the region of interest is the truck bed 175), or it can correspond to a tool 140 that includes the material 180 (if the region of interest is the tool 140).

[0063] Assuming the 3D portion corresponds to a truck bed 175 including material 180, controller 145 can determine the volume of material 180 in the truck bed 175 based on the previous volume of the truck bed 175. For example, controller 145 can determine the volume of material 180 in the truck bed 175 based on the volume of the truck bed 175 when it is empty. For instance, controller 145 can obtain information identifying the volume of the truck bed 175 when it is empty from one or more memories. Controller 145 can determine the volume of material 180 by subtracting the volume of the truck bed 175 from the volume of the 3D portion. Controller 145 can determine the volume of material 180 in tool 140 in a manner similar to that described above regarding the volume of material 180 in truck bed 175.

[0064] Controller 145 can determine the volume of material 180 in real time or near real time. In this respect, controller 145 can determine the volume of material 180 periodically (e.g., per second, per minute, and / or similar time intervals). Additionally or alternatively, controller 145 can determine the volume of material 180 each time movement of machine 105 is detected (e.g., movement associated with machine 105 moving material 180). Controller 145 can detect movement of machine 105 as described above.

[0065] The controller 145 can perform actions based on the volume of the material 180. For example, the action may include the controller 145 transmitting information about the volume of the material 180 to the machine 110, the device 230, and / or another similar receiver. For example, the controller 145 may use a wireless communication component 165 to transmit the volume information.

[0066] Additionally or alternatively, the action may include controller 145 determining the density of material 180. For example, controller 145 may determine the density of material 180 based on the volume and mass of material 180. For example, controller 145 may determine the density of material 180 based on one or more mathematical operations involving the mass and volume of material 180. In some cases, controller 145 may determine the density of material 180 based on the following formula:

[0067] p = m / V

[0068] Where p corresponds to the density of material 180, m corresponds to the mass of material 180, and V corresponds to the volume of material 180.

[0069] In some instances, controller 145 can determine the density of material 180 in truck bed 175. Controller 145 can obtain information from machine 110 via wireless communication component 165 to identify the mass of material 180 in truck bed 175. For example, controller 145 can transmit a request for information to identify the mass of material 180 in truck bed 175 to machine 110 via wireless communication component 165, and controller 145 can receive information from machine 110 to identify the mass of material 180 in truck bed 175 via wireless communication component 165. In some cases, machine 110 can determine the mass of material 180 based on data from one or more sensor devices associated with truck bed 175. The one or more sensor devices may be similar to load sensor device 155 and may be located at one or more portions of truck bed 175. Data from the one or more sensor devices can identify the mass of material 180 in truck bed 175.

[0070] Additionally or alternatively, controller 145 may receive, via wireless communication component 165, a request to determine the density of material 180 in truck bed 175 (e.g., based on information received by machine 110 identifying the volume of material 180). This request may include information identifying the mass of material 180 in truck bed 175 (e.g., obtained from one or more sensor devices associated with truck bed 175). Controller 145 may determine the density of material 180 based on the mass of material 180 (e.g., identified in load information) and the volume of material 180.

[0071] Additionally or alternatively, the controller 145 can determine the density of the material 180 in the tool 140. The controller 145 can obtain load information from the load sensor device 155 in a manner similar to that described above (with regard to the load sensor device 155). The load information can identify the mass of the material 180 in the tool 140. The controller 145 can determine the density of the material 180 (in the tool 140) based on the mass and volume of the material 180 (in the tool 140) determined above.

[0072] Additionally or alternatively, controller 145 may determine the mass of material 180 in material pile 180. In some instances, controller 145 may obtain information identifying the mass of material 180 from the machine that deposits the pile, from the user equipment of the operator that causes the pile to be deposited, from device 230, and / or other similar information sources. The machine may provide information identifying the mass of material 180 in a manner similar to that described above with respect to machine 110. In some cases, the mass of material 180 may be predetermined (e.g., predetermined by the user equipment, by device 230, etc.). Controller 145 may determine the density of material 180 (in the pile) based on the mass and volume of material 180 (in the pile) determined above.

[0073] In some instances, controller 145 can determine the type of material 180 (e.g., in truck bed 175 and / or in a stack) and can determine the mass of material 180 based on the type and volume of material 180. For example, controller 145 can determine the mass of material 180 based on one or more mathematical operations involving the volume and type of material 180. Additionally or alternatively, controller 145 can determine the mass of material 180 using one or more computational models, one or more computational algorithms, and / or other machine algorithms that can be used to determine the mass of material based on the volume and type of material.

[0074] The controller 145 can obtain information identifying the type of material 180 from the operator of machine 105, from one or more memories, from the operator of machine 110, from the controller of machine 110, from the user of device 230, and / or from device 230. The type of material 180 may include soil, rock, mineral, bitumen, and / or other types of materials. The quality of one type of material 180 may differ from that of another type of material 180.

[0075] The controller 145 can transmit information about the density of the material 180 (e.g., in the truck bed 175, in the tool 140, and / or in the stack) to the device 230. For example, the controller 145 can use the wireless communication component 165 to transmit information about the density (and / or volume) of the material 180.

[0076] Additionally or alternatively, actions may include the controller 145 automatically updating the operation of the machine 105 to improve the productivity of the machine 105. For example, the controller 145 may automatically cause updates to the configuration of the machine 105 (e.g., updates to the machine 105 software), updates to one or more components of the machine 105 (e.g., updates to the calibration of one or more components, replacement of one or more components and / or similar updates to one or more components), and / or another similar update.

[0077] In some instances, region of interest (ROI) information can identify multiple ROIs. For example, multiple ROI information can identify combinations (e.g., two or more) of truck bed 175, another machine's truck bed (or container), tool 140, material pile 180, and / or another material pile. Controller 145 can determine the volume (and / or density) of the material at multiple ROIs in real time or near real time in a manner similar to that described above regarding the determination of the volume (and / or density) of material 180 in real time or near real time.

[0078] Figure 2 The number and arrangement of devices and networks shown are provided as examples. In reality, there may be additional devices, fewer devices, different devices, or connections. Figure 2 The equipment shown is arranged differently. Furthermore, Figure 2 The two or more devices shown can be implemented within a single device, or Figure 2 The single device shown can be implemented as multiple distributed devices. Additionally or alternatively, a group of devices in system 200 (e.g., one or more devices) can perform one or more functions described as being performed by another group of devices in system 200.

[0079] Figure 3This is a flowchart of an example process 300 associated with the determination of material volume and density based on sensor data. In some implementations, Figure 3 One or more processing blocks can be executed by a controller (e.g., controller 145). In some implementations, Figure 3 One or more processing blocks may be performed by another device or a group of devices that are separate from or include the controller, such as sensor devices (e.g., stereo cameras 160-1 to 160-M, one or more LiDAR devices and / or one or more sensing sensors), IMUs (e.g., IMUs 150-1 to 150-N), wireless communication components 165 and / or devices (e.g., device 230).

[0080] like Figure 3 As shown, process 300 may include receiving information identifying a region of interest from a plurality of candidate regions of interest, wherein the plurality of candidate regions of interest include locations on and outside the machine (block 310). For example, the controller may receive information identifying a region of interest from a plurality of candidate regions of interest, wherein the plurality of candidate regions of interest include locations on and outside the machine, as described above. The plurality of candidate regions of interest include locations on and outside the machine.

[0081] like Figure 3 As further shown, process 300 may include obtaining an image (box 320) of material located at the region of interest using one or more first sensor devices associated with the machine. For example, the controller may use one or more stereo camera devices associated with the machine to obtain an image of material located at the region of interest, as described above.

[0082] like Figure 3 As further shown, process 300 may include generating a three-dimensional graphical representation based on an image (box 330). For example, a controller may generate a three-dimensional graphical representation based on an image, as described above. The controller may generate a disparity map based on a first image and a second image among a plurality of images: and generate a three-dimensional (3D) point cloud based on the disparity map.

[0083] like Figure 3 As further shown, process 300 may include determining at least one of the positions or orientations of one or more parts of the machine using one or more second sensor devices of the machine (block 340). For example, the controller may use one or more sensor devices of the machine to determine at least one of the positions or orientations of one or more parts of the machine, as described above. Determining at least one of the positions or orientations of one or more parts of the machine includes determining at least one of the positions or orientations of the machine's cab, the machine's joystick, or the machine's boom.

[0084] like Figure 3 As further shown, process 300 may include determining the coordinates of the material relative to the machine at the region of interest based on at least one of the positions or orientations of one or more portions (box 350). For example, the controller may determine the coordinates of the material relative to the machine at the region of interest based on at least one of the positions or orientations of one or more portions, as described above.

[0085] like Figure 3 As further shown, process 300 may include identifying a portion of a 3D graphical representation based on coordinates, wherein the portion corresponds to material located at a region of interest (box 360). For example, the controller may identify a portion of a 3D graphical representation based on coordinates, wherein the portion corresponds to material located at a region of interest, as described above. This portion corresponds to material located at a region of interest.

[0086] like Figure 3 As further shown, process 300 may include determining the volume of the portion using one or more computational models (box 370). For example, the controller may use one or more computational models to determine the volume of the portion, as described above.

[0087] like Figure 3 As further shown, process 300 may include determining the volume of the material based on the volume of the portion (box 380). For example, the controller may determine the volume of the material based on the volume of the portion, as described above.

[0088] The machine can be a first machine, and the process can also include identifying the truck bed of a second machine as a region of interest based on images. In some embodiments, determining the volume of material includes determining the volume of material moved from the first machine into the truck bed.

[0089] like Figure 3 As further shown, process 300 may include performing actions based on the volume of the material (block 390). For example, the controller may perform actions based on the volume of the material, as described above.

[0090] One or more first sensor devices may include at least one of one or more stereo camera devices, one or more LiDAR devices, or one or more sensing sensors, and the process may further include using one or more third sensor devices of the machine to obtain information identifying the mass of the material, and determining the density of the material based on the mass and volume of the material. Determining the volume of the material may include: determining a first volume of the truck bed when it is empty based on data; determining a second volume of the truck bed after the material has been moved into the truck bed based on the data; and determining the volume of the material at the region of interest based on the difference between the first volume and the second volume.

[0091] In some instances, the action involves transmitting information about at least one of volume or density to one or more devices that monitor at least one of the volume or density of the material being moved by the machine.

[0092] Although Figure 3 An example block of process 300 is shown, but in some embodiments, process 300 may include... Figure 3 The boxes depicted in the diagram are compared to additional boxes, fewer boxes, different boxes, or boxes arranged differently. Alternatively, two or more boxes of process 300 can be executed in parallel.

[0093] Industrial applicability

[0094] This disclosure relates to a controller for determining the volume and / or density of a material located at multiple regions of interest (e.g., locations on and outside the machine). The disclosed process for determining the volume and / or density of a material located at multiple regions of interest can prevent the problems associated with manual measurement of the material's volume and / or density. Such manual measurement can waste computational resources that are used to remedy problems associated with inaccurate manual measurements (e.g., remedy problems associated with inaccurate changes and / or adjustments to the excavator, such as inaccurate changes and / or adjustments to the excavator configuration, inaccurate changes and / or adjustments to excavator components, etc.).

[0095] The disclosed process addresses the aforementioned problems related to manual measurement. The disclosed process for determining the volume and / or density of material located in multiple regions of interest can offer several advantages. For example, the process can determine the volume and / or density of material at locations both on and outside the machine (e.g., the machine's bucket, another machine's truck bed, and / or a material pile within a threshold distance of the machine). Additionally, by determining the material's volume and / or density using sensor data, the process prevents manual measurement of the material's volume and / or density.

[0096] By preventing such manual measurements, the process can conserve computational or machine resources that would otherwise be used to remedy problems associated with inaccurate manual measurements, problems related to inaccurate changes and / or adjustments to the excavator (e.g., problems associated with inaccurate changes and / or adjustments to excavator configuration, inaccurate changes and / or adjustments to excavator components, etc.).

[0097] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the embodiments to the precise forms disclosed. Modifications and variations can be made based on the foregoing disclosure, or can be derived from practice of the embodiments. Furthermore, any embodiments described herein can be combined unless the foregoing disclosure expressly provides for reasons why one or more embodiments cannot be combined. Even if specific combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of various embodiments. Although each dependent claim listed below may be directly subordinate to only one claim, the disclosure of various embodiments includes each dependent claim in combination with all other claims in the group of claims.

[0098] As used herein, the terms “a,” “an,” and “a group” are intended to include one or more things and are interchangeable with “one or more.” Furthermore, as used herein, the article “the” is intended to include one or more things referred to in conjunction with the article “the” and is interchangeable with “the one or more.” Additionally, the phrase “based on” is intended to mean “at least partially based on,” unless otherwise expressly stated. Furthermore, as used herein, the term “or,” when used in series, is intended to be inclusive and is interchangeable with “and / or” unless otherwise expressly stated (e.g., if used in combination with “either” or “only one”). Additionally, for ease of description, spatially relative terms such as “below,” “under,” “above,” “on,” etc., may be used herein to describe the relationship between one element or feature and another element or feature as illustrated in the figures. Spatially relative terms are intended to cover different orientations of devices, apparatuses, and / or elements in use or operation other than those depicted in the figures. Devices may be oriented in other ways (rotated 90 degrees or otherwise), and the spatially relative descriptors used herein may be interpreted accordingly.

Claims

1. A method executed by a controller (145) of a machine (105), the method comprising: Receive information identifying the region of interest from multiple candidate regions of interest (140, 175). The plurality of candidate regions of interest (140, 175) include locations on and outside the machine (105); One or more first sensor devices (160) associated with the machine (105) are used to obtain images that identify the material located in the region of interest; A three-dimensional graphic representation is generated based on the image; The position or orientation of one or more parts (120, 130, 135) of the machine (105) is determined using one or more second sensor devices (150) of the machine (105); The coordinates of the material at the region of interest relative to the machine (105) are determined based on at least one of the positions or orientations of the one or more portions (120, 130, 135); Identify a portion of the 3D graphic representation based on the coordinates. The portion thereunder corresponds to the material located in the region of interest; The volume of the portion is determined using one or more computational models; The volume of the material is determined based on the volume of the portion; as well as The action is performed based on the volume of the material.

2. The method of claim 1, wherein the one or more first sensor devices (160) comprises at least one of one or more stereo camera devices, one or more light detection and ranging (LIDAR) devices, or one or more sensing sensors; and The method further includes: One or more third sensor devices (155) of the machine (105) are used to obtain information that identifies the quality of the material; as well as The density of the material is determined based on the mass and volume of the material.

3. The method of claim 2, wherein performing the action comprises: Information about at least one of the volume or the density is transmitted to one or more devices (110, 230) that monitor at least one of the volume or density of the material being moved by the machine (105).

4. The method according to claim 1, wherein the machine (105) is a first machine (105); The method further includes: Based on the image, the truck bed (175) of the second machine (110) is identified as the region of interest; and Determining the volume of the material includes: Determine the volume of the material moved by the first machine (105) into the truck bed (175).

5. The method of claim 1, wherein determining at least one of the positions or orientations of the one or more portions (120, 130, 135) of the machine (105) comprises: Determine the position or orientation of at least one of the cab (120) of the machine (105), the control lever (135) of the machine (105), or the boom (130) of the machine (105).

6. A machine (105), comprising: One or more memory units (220); as well as One or more processors (210), said one or more processors (210) being configured to: Receive information identifying the region of interest from multiple candidate regions of interest (140, 175). The plurality of candidate regions of interest (140, 175) include locations on and outside the machine (105); Data identifying material located in the region of interest is obtained using one or more first sensor devices (160) associated with the machine (105); A three-dimensional graphical representation of the material is generated based on the data; The position or orientation of one or more parts (120, 130, 135) of the machine (105) is determined using one or more second sensor devices (150) of the machine (105); A portion of the three-dimensional graphic representation is identified based on at least one of the positions or orientations of the one or more portions (120, 130, 135). The portion thereunder corresponds to the material located in the region of interest; The volume of the portion is determined using one or more computational models; as well as The volume of the material is determined based on the volume of the portion.

7. The machine (105) according to claim 6, wherein the one or more first sensor devices (160) include one or more stereo cameras (160) located at the one or more portions (120, 130, 135) of the machine (105). The one or more parts (120, 130, 135) include at least one of the machine (105) cab (120), the machine (105) joystick (135) or the machine (105) boom (130); The data mentioned above includes multiple images; and in, When generating the three-dimensional graphical representation, the one or more processors (210) are configured to: A disparity map is generated based on the first and second images from the plurality of images; and A three-dimensional (3D) point cloud is generated based on the disparity map.

8. The machine (105) according to claim 6, wherein the one or more first sensor devices (160) comprise one or more light detection and ranging (LIDAR) devices; and in, When generating the three-dimensional graphical representation, the one or more processors (210) are configured to: A three-dimensional (3D) point cloud is generated based on the data.

9. The machine (105) according to claim 6, wherein the data is first data; The region of interest is the first region of interest; and The one or more processors (210) are further configured to: Second data is obtained using the one or more first sensor devices of the machine (105). The second data identifies the second region of interest among the plurality of candidate regions of interest (140, 175); The second region of interest is identified based on the second data; as well as The volume of the material at the second region of interest is determined based on the second data.

10. The machine (105) according to claim 6, wherein the one or more processors (210) are further configured to: Using one or more object detection techniques, the truck bed (175) of another machine (110) is identified as the region of interest in the data; and in, In determining the volume of the material, the one or more processors (210) are further configured to: Based on the data, determine the first volume of the truck bed (175) when it is empty; Based on the data, a second volume of the truck bed (175) is determined after the material has been moved into the truck bed (175); as well as The volume of the material at the region of interest is determined based on the difference between the first volume and the second volume.

Images