Provision of Control Information

Abstract

An apparatus, method and computer program product for receiving information regarding a first position of an element positioned by a boom (450) based on a pre-programmed automation sequence, receiving image information including a target position (470) of the element and a representation of at least a part of the element, determining a second position of the element and the target position of the element based on the image information, determining a position error based on the second position of the element and the target position of the element, and providing control information for adjusting the first position of the element until at least one criterion for advancing the pre-programmed automation sequence is satisfied based on the position error.

Description

Technical Field

[0001] This application generally relates to providing control information. More particularly, this application relates to providing control information for adjusting the position of an element.

Background Art

[0002] Different types of mobile mining machines are configured to perform different types of operations. Accurate control of the operation of mobile mining machines is necessary to enhance safety and enable efficient production in difficult environments.

[0003] For example, a drilling rig includes a carrier and at least one boom for positioning a drilling unit. Accurate positioning of the drilling unit plays an important role in drilling because accurate positioning enables accurate drilling.

Summary of the Invention

[0004] Various aspects of examples of the present invention are set forth in the claims. The scope of protection sought for various embodiments of the present invention is indicated by the independent claims. If there are examples and features described herein that are not included within the scope of the independent claims, they should be construed as useful examples for understanding the various embodiments of the present invention.

[0005] According to a first aspect of the present invention, there is provided an apparatus comprising means for receiving information regarding a first position of an element positioned by a boom based on a pre-programmed automation sequence, means for receiving image information including a target position of the element and a representation of at least a part of the element, means for determining a second position of the element and the target position of the element based on the image information, means for determining a position error based on the second position of the element and the target position of the element, and means for providing control information for adjusting the first position of the element until at least one criterion for advancing the pre-programmed automation sequence is satisfied based on the position error.

[0006] According to a second aspect of the present invention, there is provided a method comprising receiving information regarding a first position of an element positioned by a boom based on a pre-programmed automation sequence; receiving image information including a target position of the element and a representation of at least a part of the element; determining a second position of the element and the target position of the element based on the image information; determining a position error based on the second position of the element and the target position of the element; and providing control information for adjusting the first position of the element until at least one criterion for advancing the pre-programmed automation sequence is satisfied based on the position error.

[0007] According to a third aspect of the present invention, there is provided a computer program including instructions for causing an apparatus to at least receive information regarding a first position of an element positioned by a boom based on a pre-programmed automation sequence; receive image information including a target position of the element and a representation of at least a part of the element; determine a second position of the element and the target position of the element based on the image information; determine a position error based on the second position of the element and the target position of the element; and provide control information for adjusting the first position of the element until at least one criterion for advancing the pre-programmed automation sequence is satisfied based on the position error.

[0008] According to a fourth aspect of the present invention, there is provided an apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured to cause the at least one processor to cause the apparatus to at least receive information regarding a first position of an element positioned by a boom based on a pre-programmed automated sequence, receive image information including a target position of the element and a representation of at least a part of the element, determine a second position of the element and the target position of the element based on the image information, determine a position error based on the second position of the element and the target position of the element, and provide control information for adjusting the first position of the element until at least one criterion for advancing the pre-programmed automated sequence is satisfied based on the position error.

[0009] According to a fifth aspect of the present invention, there is provided a non-transitory computer-readable medium including program instructions for causing at least an apparatus to perform at least receiving information regarding a first position of an element positioned by a boom based on a pre-programmed automated sequence, receiving image information including a target position of the element and a representation of at least a part of the element, determining a second position of the element and the target position of the element based on the image information, determining a position error based on the second position of the element and the target position of the element, and providing control information for adjusting the first position of the element until at least one criterion for advancing the pre-programmed automated sequence is satisfied based on the position error.

[0010] According to a sixth aspect of the present invention, receiving information regarding a first position of an element positioned by a boom based on a pre-programmed automation sequence, receiving image information including a target position of the element and a representation of at least a part of the element, determining a second position of the element and the target position of the element based on the image information, determining a position error based on the second position of the element and the target position of the element, and providing control information for adjusting the first position of the element until at least one criterion for advancing the pre-programmed automation sequence is satisfied based on the position error. A computer-readable medium is provided that includes program instructions for causing at least an apparatus to execute the above.

Brief Description of Drawings

[0011] Here, with reference to the accompanying drawings, some exemplary embodiments will be described.

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Modes for Carrying Out the Invention

[0012] The following embodiments are illustrated. This specification may refer to "an", "one", or "some" embodiments (plural) at several places in the text, but this does not necessarily mean that each reference is made to the same embodiment (plural), or that a particular feature applies only to a single embodiment. It is also possible to combine single features of different embodiments to provide other embodiments.

[0013] An exemplary embodiment relates to a mobile mining machine configured to execute at least one pre-programmed automation sequence. A mobile mining machine comprising a carrier and at least one boom may be configured to position at least one element by means of the at least one boom based on a pre-programmed automation sequence. The pre-programmed automation sequence may include, for example, a sequence for changing a drill bit, a bolting sequence, or a sequence for installing a lifter tube. Each of these automation sequences requires accurate positioning of elements associated with the sequence, such as a drill bit, a bolt, or a lifter tube.

[0014] Currently, the element is positioned based on a kinematic model of the mobile mining machine using measurement information from at least one sensor associated with the mobile mining machine. Based on the kinematic model and the measurement information, the position of the element relative to the mobile machine is calculated. However, the accuracy of such a system depends on the accuracy of the sensors, the calibration of the sensors, the accuracy of the boom sensors, the clearances and wear of the joints. Furthermore, there is no feedback between the actual position of the element and the position determined based on the kinematic model. Incorrect positioning also affects, for example, drilling accuracy and production.

[0015] An exemplary embodiment relates to an apparatus for controlling a mobile mining machine comprising at least one boom configured to position at least one element based on a carrier and a pre-programmed automation sequence, the apparatus receiving information regarding a first position of an element positioned by the boom based on the pre-programmed automation sequence, receiving image information including a target position of the element and a representation of at least a part of the element, determining a second position of the element and the target position of the element based on the image information, determining a position error based on the second position of the element and the target position of the element, and providing control information for adjusting the first position of the element until at least one criterion for advancing the pre-programmed automation sequence is met, based on the position error.

[0016] FIG. 1 is a block diagram showing an apparatus 100 operating according to an exemplary embodiment of the present invention. The apparatus 100 may be an electronic device such as a module, chip, or chip set constituted by, for example, an automation or control system. The apparatus 100 includes at least one processor 110 and at least one memory 160 including one or more algorithms such as computer program instructions 120, and the at least one memory 160 and the computer program instructions are configured to cause the at least one processor 110 to execute any of the exemplary functions described later for the apparatus.

[0017] In the example of FIG. 1, the processor 110 is a control unit operably connected to read and write with the memory 160. The processor 110 may also be configured to receive control signals received via an input interface, and / or the processor 110 may be configured to output control signals via an output interface. In an exemplary embodiment, the processor 110 can be configured to convert the received control signals into appropriate commands for controlling the functions of the apparatus.

[0018] When at least one memory 160 is loaded into the processor 110, it stores computer program instructions 120 that control the operation of the device 100 as described below. In other examples, the device 100 can include a plurality of memories 160 or different types of storage devices.

[0019] The computer program instructions 120 or a portion of such computer program instructions for enabling the implementation of the exemplary embodiments of the present invention may be loaded into the device 100 by the manufacturer of the device 100, by the user of the device 100, or by the device 100 itself based on a download program, or the instructions may be pushed into the device 100 by an external device. The computer program instructions may reach the device 100 via an electromagnetic carrier signal, or may be copied from a physical entity such as a memory device or a recording medium such as a computer program product, a USB stick, a compact disc (CD), a compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), or a Blu-ray disc.

[0020] FIG. 2 is a block diagram showing a device 200 according to an exemplary embodiment of the present invention. The device 200 may be an electronic device such as a module configured by an automation system or a control system, a personal computer (PC), a laptop, a desktop, a wireless terminal, a communication terminal, a computing device, and the like. In the following examples, it is assumed that the device 200 is a computing device.

[0021] In the exemplary embodiment of FIG. 2, the apparatus 200 is shown as including the apparatus 100, a display 210, a user interface 220 for interacting with the computing device 200, and a communication module 230. The display 210 may also be configured to function as a user interface. For example, the display may be a touch screen display. In the exemplary embodiment, the display 210 and / or the user interface 220 may be external to the apparatus 200 but communicate therewith.

[0022] Additionally or alternatively, the user interface may also include a manually operable control such as buttons, keys, touch pads, joysticks, styli, pens, rollers, rockers, keypads, keyboards, or any suitable input mechanism for inputting and / or accessing information.

[0023] The communication module 230 may be configured to communicate with one or more other devices such as one or more sensors, a control unit, and a work machine such as a mobile mining vehicle. Communicating with one or more devices can include receiving information from one or more devices via a wired or wireless connection and / or transmitting information to one or more devices.

[0024] The apparatus 200 of the example of FIG. 2 may also be configured to establish wireless communication with another device using, for example, a cellular network, Bluetooth, or a WiFi connection. Alternatively or additionally, the apparatus 200 may be configured to communicate with another device using a wired connection.

[0025] According to an exemplary embodiment, the apparatus 200 is configured to communicate with at least one camera. The apparatus 200 may be configured to communicate with at least one camera via, for example, the communication module 230. The at least one camera can include, for example, a video camera, a stereo camera, or a monocular camera.

[0026] Communicating with at least one camera can include, for example, receiving image information captured or streamed by at least one camera. As another example, communicating with at least one camera can include, for example, instructing the camera to capture or stream image information. The image information can include image data such as one or more data frames including a visual representation of one or more objects.

[0027] According to an exemplary embodiment, apparatus 200 includes a module operably connected to a mobile mining machine. For example, apparatus 200 may be implemented in a control system of the mobile mining machine, or apparatus 200 may be configured to communicate with a control system of the mobile mining machine.

[0028] According to an exemplary embodiment, apparatus 200 is configured to control one or more functions and / or actuators of a mobile mining machine. Apparatus 200 may be configured to control one or more functions and / or actuators directly or via one or more control units of a control system of the mobile mining machine. Controlling one or more functions and / or actuators via one or more control units can include, for example, communicating with one or more control units via a controller area network (CAN) bus or an Ethernet network.

[0029] According to an exemplary embodiment, the mobile mining machine includes a drilling rig. The drilling rig can include a drilling rig for underground mining or a drilling rig for surface mining. The drilling rig can include, for example, a rock drilling rig.

[0030] According to an exemplary embodiment, the mobile mining machine comprises a mobile carrier and at least one boom. The at least one boom may comprise different types of operating units at the distal end of the at least one boom. For example, the at least one boom can be equipped with a rock drilling unit, a bolting unit, a lifter tube fixture, a casing fixture, a rock support fixture, etc.

[0031] The rock drilling unit is configured to drill holes in the rock. The rock drilling unit comprises a rock drill and a feed beam configured to provide a linear motion to the rock drill along the feed beam when drilling the hole.

[0032] The bolting unit is configured to reinforce the rock with bolts or elongated reinforcement materials, resin, cement, etc. The bolting unit comprises a rock bolting machine and a feed beam for feeding a rock bolt or a corresponding elongated reinforcement material into the drilled hole.

[0033] According to an exemplary embodiment, the mobile mining machine is configured to execute at least one pre-programmed automation sequence. The mobile mining machine may be configured to start at least one pre-programmed automation sequence, for example, based on an input received from an operator of the mobile mining machine or automatically as part of the operation of the mobile mining machine.

[0034] The pre-programmed automation sequence can include a pre-programmed series of actions for performing at least one task. The pre-programmed series of actions can include, for example, a plurality of consecutive actions performed by the mobile mining machine to perform the task. The plurality of consecutive actions performed by the mobile mining machine can include, for example, controlling one or more functions and / or actuators of the mobile mining machine.

[0035] Pre-programmed automation sequences can include, for example, sensor-based and / or time-based automation sequences.

[0036] Sensor-based automation sequences can be configured to use data received from at least one sensor to start, execute, and / or end an automation sequence. For example, a mobile mining machine may be configured to execute a sensor-based automation sequence for controlling a boom in response to determining that the boom of the mobile mining machine is in a particular position.

[0037] Time-based automation sequences can be configured to determine specific time instances or durations related to the operation of a mobile mining machine to start, execute, and / or end an automation sequence. For example, a mobile mining machine may be configured to execute a time-based automation sequence for controlling a boom in response to, for example, determining that a particular period of time has elapsed since moving the boom to a particular position.

[0038] Pre-programmed automation sequences can also include a combination of sensor-based automation sequences and time-based automation sequences. For example, one or more pre-programmed operations of an automation sequence may be sensor-based, and one or more pre-programmed operations may be time-based.

[0039] Pre-programmed automation sequences executed by a mobile mining machine can include, for example, sequences for changing drill bits, bolt tightening sequences, sequences for installing lifter tubes, sequences for changing drill rods, or any other applicable sequences.

[0040] As described above, a pre-programmed sequence includes pre-programmed actions for performing operations. As an example, a sequence for changing a drill bit can include actions such as loosening the drill bit, retracting the drill, closing the rod retainer, tightening the shank threads, opening the rod retainer, controlling the position of the drill bit to the bit changer position, actuating the position adjustment of the drill bit, depositing or retrieving the drill bit, retracting the drill, and verifying that the operation was successful.

[0041] According to an exemplary embodiment, the position adjustment includes adjusting the position of an element based on image information. For example, the position adjustment of elements such as drill bits, bolts, lifter tubes, or drill rods will be described in more detail in the following examples.

[0042] A pre-programmed automation sequence can include one or more sub-sequences that are executed as part of the automation sequence. For example, actuating the position adjustment included in the sequence for changing a drill bit can include a sub-sequence that includes a plurality of actions for adjusting the position of the drill bit / boom.

[0043] Note that a sequence for installing a lifter tube or a bolt tightening sequence may include actions different from those of a sequence for changing a drill bit. Alternatively, a sequence for installing a lifter tube or a bolt tightening sequence may have actions that are partially the same as those of a sequence for changing a drill bit, such as verifying that the operation was successful.

[0044] As described above, the apparatus 200 is configured to communicate with different devices such as different sensors and / or control units.

[0045] According to an exemplary embodiment, the apparatus 200 is configured to receive information regarding a first position of an element positioned by a boom based on a pre-programmed automation sequence.

[0046] The apparatus 200 may be configured to receive information regarding the first position from a separate device such as a sensor, or the apparatus 200 may be configured to determine the first position based on measurement data from, for example, one or more sensors.

[0047] The information regarding the first position of the element can include information regarding the position of the element relative to the mobile mining machine. The position of the element relative to the mobile mining machine can include, for example, the position of the element relative to a predetermined reference point associated with the mobile mining machine. As another example, the information regarding the first position of the element can include the position of the element in a coordinate system such as the position of the element in mine coordinates.

[0048] The information regarding the first position can include measurement data indicating the first position, a simulated position, or a position determined based on a model of the mobile mining machine, and measurement data from one or more sensors associated with the mobile mining machine.

[0049] The model of the mobile mining machine can include, for example, a kinematic model of the mobile mining machine.

[0050] The kinematic model of the mobile mining machine includes a mathematical description of at least a part of the mobile mining machine. The kinematic model describes the motion of the machine without considering the forces that cause the motion. The kinematic model can be used to estimate the position of the machine based on measurement data from one or more sensors associated with the machine or the movement of the machine caused by a given control input. The kinematic model of the mobile mining machine includes at least the dimensions of the mobile mining machine and / or the reach of the mobile mining machine such as the movement range of at least one boom of the mobile mining machine.

[0051] According to an exemplary embodiment, the first position includes a position determined using a kinematic model of the mobile mining machine and measurement data from at least one sensor associated with the mobile mining machine.

[0052] The element may include elements related to a pre-programmed automation sequence. The element can include an exchangeable element or a fixed element. An exchangeable element refers to an element that can be removed or replaced due to wear or different operating stages, and a fixed element means that it lasts over a longer period, but can be replaced, for example, when damaged. Exchangeable elements can include, for example, drill bits, lifter tubes, bolts, drill rods, etc. Fixed elements can include, for example, booms, feed beams, etc.

[0053] According to an exemplary embodiment, at least one element includes a drill bit, a lifter tube, or a bolt.

[0054] Thus, the first position of the element can include, for example, the position of the drill bit relative to the mobile mining machine, the position of the lifter tube relative to the mobile mining machine, or the position of the bolt relative to the mobile mining machine.

[0055] As described above, the apparatus 200 is configured to communicate with at least one camera. The at least one camera may be associated with the mobile mining machine, or the at least one camera may be disposed within the environment of the mobile mining machine.

[0056] The device 200 may be configured to receive image information from a single camera or multiple cameras. The image information may include information about the environment of the mobile mining machine and / or information about the mobile mining machine. The information about the environment of the mobile machine may include a representation of the environment, and the information about the mobile mining machine may include a representation of at least a part of the mobile mining machine, elements related to the mobile mining machine, etc. According to an exemplary embodiment, the device 200 is configured to receive image information including the target position of an element and a representation of at least a part of the element.

[0057] The target position includes the target position of the element. For example, the target may include, for example, a cavity configured to receive the element.

[0058] According to an exemplary embodiment, the target position of the element includes an existing hole, and the element is placed in the hole.

[0059] According to an exemplary embodiment, the existing holes include drilled holes or artificial holes. Drilled holes may include, for example, holes drilled in rock, and artificial holes may include, for example, holes in drill bit holders.

[0060] The target position of the element may include the target position of the element associated with a pre-programmed automation sequence. For example, if the element is a drill bit, the target position may include a position that enables changing the drill bit. As another example, if the element is a lifter tube, the target position may include a position that enables installing the lifter tube. As a further example, if the element is a bolt, the target position may include a position that enables attaching the bolt.

[0061] The image information may include, for example, a video feed from a camera. The video feed includes a series of images processed electronically into a predetermined format. As another example, the image information may include one or more images captured by a camera.

[0062] According to an exemplary embodiment, the image information includes image data such as one or more data frames.

[0063] According to an exemplary embodiment, the apparatus 200 is configured to determine the presence and / or the state of one or more elements within the image information based on the image information.

[0064] Determining the presence of an element can include determining whether a representation of at least a portion of the element is included in the image information. Determining the presence of an element can include determining that a representation of the element is included in the image information or that a representation of the element is not included in the image information.

[0065] Determining the state of an element can include determining the state of the element based on a representation of the element within the image information. Determining the state of an element can include comparing the representation of the element with reference information. Determining the state of an element can include, for example, determining that the element is of the correct type, determining that the element is not damaged, and / or determining that the element is correctly positioned.

[0066] Determining the presence and / or the state of one or more elements can include, for example, processing the image information. Processing the image information can include, for example, filtering the image information, analyzing the image information, object detection, or other suitable processing methods.

[0067] The apparatus 200 may be configured to process the image information using, for example, artificial intelligence (AI). The artificial intelligence can comprise, for example, different types of mechanisms for analyzing the image information. The AI can include, for example, one or more machine vision algorithms such as one or more algorithms for object detection and / or object tracking.

[0068] The machine vision algorithm is configured to extract information from the image information and obtain the desired data for performing an action.

[0069] According to an exemplary embodiment, the apparatus 200 is configured to analyze the image information using a machine vision algorithm. In the example of FIG. 2, the machine vision algorithm includes an algorithm for automatic acquisition and analysis of an image to determine an element and a target position of at least a part of the element. The machine vision algorithm may be stored, for example, in the memory 160.

[0070] The machine vision algorithm can include, for example, a neural network. The neural network comprises a plurality of algorithms configured to recognize underlying relationships and / or patterns within a set of data.

[0071] The neural network includes a plurality of node layers such as an input layer, one or more hidden layers, and an output layer. Each node has an associated weight and threshold, and each node is connected to a node in the next layer. If the output of an individual node exceeds the threshold, the node is activated and data is transmitted to the next layer of the network.

[0072] The neural network is configured to learn based on training data, and the neural network improves its accuracy over time. The learning algorithm of the neural network can include supervised learning or unsupervised learning. Supervised learning uses labeled input-output data, which means that the desired output is known, while unsupervised learning algorithms do not use labeled input-output data.

[0073] According to an exemplary embodiment, the apparatus 200 is configured to determine a second position of the element and a target position of the element based on the image information.

[0074] Determining the second position of the element and the target position of the element based on the image information can include identifying the representation of the element and the target of the element within the image information using a machine vision algorithm. The machine vision algorithm can, for example, recognize the shape of the element and the shape of the target, and identify the element and the target based on the shape of each element. For example, the machine vision algorithm can recognize the shape of a drill bit, a bolt, or a lifter tube, and the shape of a hole.

[0075] Determining the second position of the element and the target position of the element based on the image information can further include determining the position of the element and the position of the target relative to a reference point.

[0076] The reference point can include a static reference point or a dynamic reference point. The static reference point can include, for example, markings within a tunnel wall. The dynamic reference point can include, for example, a reference point associated with a mobile mining machine such as a point within a movable boom.

[0077] According to an exemplary embodiment, the apparatus 200 is configured to perform at least one operation that affects at least one of the second position of the element or the target position of the element simultaneously with the reception of the image information.

[0078] According to an exemplary embodiment, the at least one operation includes an operation that causes a change in the position of the element and / or the target position. The at least one operation can include, for example, causing a controlled collision between the element and the target position. For example, the apparatus 200 may be configured to cause a controlled collision between a drill bit or a drill rod and a bit holder. The controlled collision includes an intentionally caused collision.

[0079] A controlled collision can include a collision with a defined collision position. The defined collision position can include, for example, a selected position such that an element and a target position collide while an unobstructed view of the representation of a selected object included in the image information is enabled. For example, assuming that the device 200 is configured to cause a controlled collision between a drill bit and a bit holder on the right side of a mobile mining machine, the defined position can include the lower left corner of the bit holder such that the hole in the bit holder is not blocked by the drill bit.

[0080] According to an exemplary embodiment, the device 200 is configured to detect a change in the position of at least one representation of an object included in the image information. The object can include, for example, at least one element such as a drill bit, a bolt, a drill rod, or a boom, a target position of an element such as a bit holder, or some other object.

[0081] According to an exemplary embodiment, the device 200 is configured to detect a change in the position of a plurality of representations of an object included in the image information.

[0082] Without limiting the claims, the advantage of detecting a change in the position of a plurality of objects is that the detected change in the position of an object can be confirmed by the detected changes of one or more other objects.

[0083] According to an exemplary embodiment, the device 200 is configured to store information regarding an operation that affects a second position of an element and / or a target position of an element.

[0084] Information regarding the second position of the element and / or an operation affecting the target position of the element can include, for example, information regarding the position of the unit causing the change in position. For example, if the operation affecting the second position and / or the target position of the element includes a controlled collision between the drill bit / drill rod and the bit holder, the position of the rock drilling machine can be memorized when the rock drilling unit is controlled forward and when the rock drilling machine is retracted.

[0085] According to an exemplary embodiment, the apparatus 200 is configured to determine the second position of the element based on image information and information regarding at least one operation affecting the second position of the element and / or the target position of the element.

[0086] Without limiting the claims, the advantage of determining the second position of the element based on the image information and the information regarding at least one operation affecting the second position of the element and / or the target position of the element is that more accurate information regarding the second position of the element can be received. For example, if the operation causing the change in the position of the element and / or the target position includes an intentional collision between the drill bit / drill rod and the bit holder, the apparatus 200 can be configured to determine the three-dimensional (3D) position information of the second position of the element.

[0087] The apparatus 200 may be configured to determine the second position of the element and the target position of the element based on a predetermined condition. For example, the apparatus 200 may be configured to determine the second position and the target position of the element in response to detecting that a pre-programmed automation sequence has reached a specific state.

[0088] According to an exemplary embodiment, the apparatus 200 is configured to determine the second position in response to a determination that the first position is within a threshold distance from the target position. The threshold distance may include, for example, 1 to 30 cm, or more, such as 5, 10, 15, 20, or 25 cm.

[0089] Although not limiting the claims, the advantage of determining the second position of the element based on the image information is that the position of the element can be determined independently of sensors associated with mobile mining machines such as boom sensors. This enables accurate determination of the element regardless of various sensor accuracies, sensor calibrations, boom structural accuracies, and / or wear.

[0090] According to an exemplary embodiment, the apparatus 200 is configured to determine a position error based on the second position of the element and the target position of the element.

[0091] According to an exemplary embodiment, the position error includes a parameter value indicating a misplacement of the element relative to the target position of the element.

[0092] The misplacement of the element can include, for example, distance information and / or orientation information. The distance information can include, for example, the distance between the second position and the target position. The orientation information can include, for example, the angle between the second position and the target position.

[0093] According to an exemplary embodiment, the apparatus 200 is configured to determine control information for adjusting the first position of the element. The control information can include one or more instructions for controlling the position of the element to reach the target position.

[0094] According to an exemplary embodiment, the apparatus 200 is configured to determine the control information based on the image information.

[0095] The control information can include, for example, control information for moving the element a specific distance and / or in a specific direction from the first position. Alternatively, the control information can include control information corresponding to the image information, such as the coordinates of the target position or the number of pixels moved by the element.

[0096] The control information may include an absolute value or a relative value. For example, the control information can include control information for moving an element 5 cm to the left or 10 cm down from the current position.

[0097] According to an exemplary embodiment, the apparatus 200 is configured to provide control information for adjusting a first position of an element until at least one criterion for proceeding with a pre-programmed automation sequence is satisfied based on a position error.

[0098] According to an exemplary embodiment, the apparatus 200 is configured to determine that at least one criterion is satisfied based on image information.

[0099] The at least one criterion may be related to the position of the element or the purpose of a pre-programmed automation sequence.

[0100] According to an exemplary embodiment, the at least one criterion includes that the position of the element corresponds to a target position or that an operation has been completed.

[0101] Completed operations can include, for example, a drill bit that has been successfully replaced, a lift tube that has been successfully installed, or a bolt that has been successfully tightened.

[0102] The apparatus 200 may be configured to provide control information to a control unit configured to control at least one actuator of a mobile mining machine, or the apparatus 200 may be configured to directly control at least one actuator of a mobile mining machine.

[0103] According to an exemplary embodiment, the apparatus 200 is configured to control at least one actuator using the control information.

[0104] Note that the device 200 may be configured to determine a plurality of position errors of a mobile mining machine having, for example, a plurality of booms. Assuming that the element controlled by the boom is the boom itself, the device 200 may be configured to determine a first position error for a first boom and a second position error for a second boom based on the image information. The device 200 may be further configured to provide first control information for adjusting the position of the first boom based on the first position error and to provide second control information for adjusting the position of the second boom based on the second position error. The device 200 can be further configured to use the position of the boom to compensate for the error of the kinematic model.

[0105] According to an exemplary embodiment, the device 200 comprises means for performing the functions of the device 200, the means for performing comprising at least one processor 110 and at least one memory 160 including computer code 120 configured to cause the execution of the device 200 using the at least one processor 110.

[0106] According to an exemplary embodiment, the device 200 comprises means for receiving information regarding a first position of an element positioned by a boom based on a pre-programmed automation sequence, means for receiving image information including a target position of the element and a representation of at least a portion of the element, means for determining a second position of the element and the target position of the element based on the image information, means for determining a position error based on the second position of the element and the target position of the element, and means for providing control information for adjusting the first position of the element until at least one criterion for advancing the pre-programmed automation sequence is met based on the position error.

[0107] According to an exemplary embodiment, the apparatus 200 can further comprise means for determining a second position in response to a determination that the first position is within a threshold distance from the target position. According to an exemplary embodiment, the apparatus 200 can further comprise means for analyzing image information using a machine vision algorithm and / or means for determining that at least one criterion is met based on the image information. According to an exemplary embodiment, the apparatus 200 can further comprise means for determining that control information includes one or more parameters for controlling the direction of the boom or coordinate information for positioning the boom based on the image information and / or means for controlling at least one actuator using the control information.

[0108] FIG. 3 shows an exemplary system 300 incorporating aspects of an exemplary embodiment. In the example of FIG. 3, the system 300 is constituted by a mobile mining machine, and the apparatus 200 corresponds to the apparatus 200 in the example of FIG. 2.

[0109] In the example of FIG. 3, the mobile mining machine comprises a drilling rig having at least one boom provided with a rock drilling unit at the distal end of at least one boom. The rock drilling unit comprises a feeding system configured to keep the drill bit in contact with the rock during drilling.

[0110] The system 300 comprises the apparatus 200, an image processing unit 310 operably connected to a camera 320, a position control unit 330 operably connected to a boom control actuator 340, and a boom kinematic model 350 configured to receive information from a boom sensor 360. It should be noted that the image processing unit 310 may be constituted by the apparatus 200 as shown by the dashed line, or the image processing unit 310 may be operably connected to the apparatus 200 but separate from the apparatus 200.

[0111] The device 200 is configured to receive information from the kinematic model 350 and the image processing unit 310. In the example of FIG. 3, the information from the kinematic model 350 includes information regarding the first position of an element positioned by a boom based on a pre-programmed automation sequence, and the information from the image processing unit 310 includes image information including the target position of the element and a representation of at least a portion of the element.

[0112] The pre-programmed automation sequence can include, for example, an automation sequence for changing a drill bit executed by a mobile mining machine, a bolt tightening sequence, or a sequence for installing a lifter tube.

[0113] The device 200 is further configured to determine a second position of the element and the target position of the element based on information from the image processing unit 310 and to determine a position error based on the second position of the element and the target position of the element.

[0114] The device 200 is further configured to provide control information to the position control unit 330 to adjust the first position of the element until at least one criterion for advancing the pre-programmed automation sequence is met based on the position error. In the example of FIG. 3, the control information is used to control at least one boom control actuator 340 configured to control the position of the boom.

[0115] FIG. 4 shows an exemplary arrangement 400 for determining a position error for a mobile mining machine 410 comprising the device 200. The element 440 is assumed to be positioned by the boom 450 based on a pre-programmed automation sequence.

[0116] The pre-programmed automation sequence executed by the mobile mining machine can include, for example, a sequence for changing a drill bit, a bolt tightening sequence, a sequence for installing a lifter tube, or any other applicable sequence.

[0117] The device 200 is configured to receive information regarding a first position 430 of an element positioned by the boom 450. The first information can include the position of the element determined using a kinematic model of the mobile mining machine.

[0118] In the example of FIG. 4, the mobile mining machine 410 further comprises a camera 320, and the device 200 is configured to receive image information including a target position 470 of the element 440 and a representation of at least a part of the element 440.

[0119] The device 200 is further configured to determine a second position 420 of the element 440 and a target position 470 of the element based on the image information, and to determine a position error 460 based on the second position 420 of the element and the target position 470 of the element. The position error 460 includes a parameter value indicating a misplacement of the element relative to the target position of the element.

[0120] The example of FIG. 4 shows a mobile mining machine with one boom, but the position error may also be determined in a multi-boom system. Assuming that the element controlled by the boom is the boom itself, the device 200 may be configured to determine a first position error for the first boom and a second position error for the second boom based on the image information. The device 200 may be further configured to provide first control information for adjusting the position of the first boom based on the first position error, and to provide second control information for adjusting the position of the second boom based on the second position error. The device 200 can be further configured to use the position of the boom to compensate for errors in the kinematic model.

[0121] Figure 5 shows an exemplary method 500 that incorporates aspects of the previously disclosed embodiments. More specifically, exemplary method 500 shows a method of controlling a mobile mining machine comprising a carrier configured to position at least one element based on a pre-programmed automation sequence and at least one boom. Method 500 can include a computer-implemented method executed by apparatus 200.

[0122] The method begins at 505 by receiving information regarding a first position of an element positioned by a boom based on a pre-programmed automation sequence.

[0123] In the example of Figure 5, the first position includes a position determined using a kinematic model of the mobile mining machine and measurement data from at least one sensor associated with the mobile mining machine.

[0124] The element may include an element related to a pre-programmed automation sequence such as a drill bit, a lifter tube, or a bolt.

[0125] The method continues at 510 by receiving image information including a target position of the element and a representation of at least a portion of the element. The image information can include, for example, a video feed from a camera or one or more images captured by a camera.

[0126] Receiving the image information can also include processing the image information. In the example of Figure 5, receiving the image information includes analyzing the image information using a machine vision algorithm. The machine vision algorithm can include, for example, a neural network.

[0127] In the example of Figure 5, the target position of the element includes an existing hole and the element is placed within the hole. The existing hole can include a drilled hole or an artificial hole. The existing hole can include, for example, a hole drilled in rock, and the artificial hole can include, for example, a hole in a drill bit holder.

[0128] The method then continues with determining a second position of the element and a target position of the element based on the image information. The apparatus 200 can be configured to determine the second position in response to a determination that the first position is within a threshold distance from the target position.

[0129] The method then continues with determining a position error based on the second position of the element and the target position of the element.

[0130] In the example of FIG. 5, the position error includes a parameter value indicating a misplacement of the element relative to the target position of the element.

[0131] The method then further continues with providing control information for adjusting the first position of the element until at least one criterion for advancing a pre-programmed automation sequence based on the position error is satisfied.

[0132] The at least one criterion can be related to the position of the element or the purpose of the pre-programmed automation sequence.

[0133] In the example of FIG. 5, the at least one criterion includes that the position of the element corresponds to the target position or that the operation has been completed. The completed operation can include, for example, a drill bit that has been successfully replaced, a lift tube that has been successfully installed, or a bolt that has been successfully tightened.

[0134] Without limiting the claims, an advantage of determining a position error based on the second position of the element and the target position of the element and providing control information for adjusting the first position of the element based on the position error is that it can provide feedback between the calculated position and the actual position.

[0135] Without in any way limiting the scope, interpretation, or application of the claims set forth below, one or more technical effects of the exemplary embodiments disclosed herein are that surface excavation, cross-cut excavation, and the installation accuracy of lifter tubes are improved. Another technical effect is that position errors can be used to compensate for errors in the kinematic model.

[0136] As used in this application, the term "circuit" refers to (a) a hardware-only circuit implementation (such as an implementation with only analog and / or digital circuits), and (b) (if applicable) (i) a combination of analog and / or digital hardware circuit(s) and software / firmware, and (ii) any portion of a hardware processor(s) having software (including a digital signal processor(s)) that cooperate to cause a device such as a mobile phone or server to perform various functions, software, and memory(ies), i.e., a combination of a hardware circuit and a software circuit(s), (c) one or more or all of a hardware circuit(s) and a processor(s) such as a microprocessor(s) or a portion of a microprocessor(s) that requires software (e.g., firmware) for operation, where the software may not be present if it is not required for operation.

[0137] This definition of circuit applies to all uses of this term in this application, including any claims. As a further example, as used in this application, the term "circuit" also encompasses simply a hardware circuit or processor (or processors), or a portion of a hardware circuit or processor, and the accompanying software and / or firmware implementation. The term "circuit" also encompasses, for example, if applicable, a particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device, or a similar integrated circuit within a server, cellular network device, or other computing or network device.

[0138] Embodiments of the present invention can be implemented in software, hardware, application logic, or a combination of software, hardware, and application logic. The software, application logic, and / or hardware may reside on an apparatus, a separate device, or multiple devices. Optionally, some of the software, application logic, and / or hardware may reside on an apparatus, some of the software, application logic, and / or hardware may reside on a separate device, and some of the software, application logic, and / or hardware may reside on multiple devices. In an exemplary embodiment, the application logic, software, or instruction set is maintained on any one of a variety of conventional computer-readable media. In the context of this specification, a "computer-readable medium" can be any medium or means that can contain, store, communicate, propagate, or transport instructions for use by or in connection with an instruction execution system, apparatus, or device such as a computer, using an example of a computer described and depicted in FIG. 2. The computer-readable medium can include a computer-readable storage medium that can be any medium or means that can contain or store instructions for use by or in connection with an instruction execution system, apparatus, or device such as a computer.

[0139] Optionally, the different functions described herein may be performed in a different order and / or simultaneously with each other. Also, one or more of the functions described above may be optional and may be combined as desired.

[0140] Although various aspects of the present invention are set forth in the independent claims, other aspects of the present invention include other combinations of features from the described embodiments and / or the dependent claims and the features of the independent claims, and not only the combinations explicitly set forth in the claims.

[0141] As technology advances, it will be apparent to those skilled in the art that the concepts of the present invention can be implemented in various ways. The present invention and its embodiments are not limited to the examples described above and may vary within the scope of the claims.

Claims

1. An apparatus for controlling a mobile mining machine comprising a carrier and at least one boom configured to position at least one element based on a pre-programmed automation sequence, wherein the apparatus comprises at least one processor and at least one memory containing computer program code, and the at least one memory and the computer program code use the at least one processor to provide the apparatus with at least one Information regarding the first position of an element positioned by the boom based on a pre-programmed automated sequence is received. The system receives image information including the target position of the element and a representation of at least a part of the element. Based on the aforementioned image information, the second position of the element and the target position of the element are determined. Based on the second position of the element and the target position of the element, a position error is determined, including a parameter value indicating an incorrect placement of the element relative to the target position. A device configured to provide control information for adjusting the first position of the element based on the position error until at least one criterion for advancing the pre-programmed automated sequence is met.

2. The apparatus according to claim 1, wherein the at least one memory and the computer program code are further configured to cause the apparatus to determine the second position in response to a determination that the first position is within a threshold distance from the target position, using the at least one processor.

3. The apparatus according to claim 1, wherein the target position of the element includes an existing hole, and the element is positioned within the hole.

4. The apparatus according to claim 3, wherein the existing hole includes a drilled or artificial hole.

5. The apparatus according to claim 1, wherein the at least one memory and the computer program code are further configured to cause the apparatus to analyze image information using a machine vision algorithm, using the at least one processor.

6. The apparatus according to claim 1, wherein the first position includes a position determined using a kinematic model of the mobile mining machine and measurement data from at least one sensor associated with the mobile mining machine.

7. The apparatus according to claim 1, wherein the at least one element includes a drill bit, a lifter tube, or a bolt.

8. The apparatus according to claim 1, wherein the at least one memory and the computer program code are further configured to cause the apparatus to determine the control information based on the image information using the at least one processor.

9. The apparatus according to claim 1, wherein the at least one memory and the computer program code are further configured to cause the apparatus to control at least one actuator using the control information, using the at least one processor.

10. The apparatus according to claim 1, wherein the at least one memory and the computer program code are further configured to cause the apparatus to determine, using the at least one processor, that the at least one criterion is met based on the image information.

11. The apparatus according to claim 1, wherein the at least one criterion includes the position of the element corresponding to the target position or the completion of the operation.

12. The apparatus according to claim 1, wherein the at least one memory and the computer program code are further configured to cause the apparatus to perform at least one operation that affects at least one of the second position of the element or the target position of the element, simultaneously with the reception of the image information, using the at least one processor.

13. A drilling rig comprising the apparatus described in any one of claims 1 to 12.

14. A method for controlling a mobile mining machine comprising a carrier and at least one boom configured to position at least one element based on a pre-programmed automation sequence, Receiving information regarding the first position of an element positioned by the boom based on a pre-programmed automated sequence, Receiving image information including the target position of the element and a representation of at least a part of the element, Based on the aforementioned image information, the second position of the element and the target position of the element are determined. Based on the second position of the element and the target position of the element, a position error is determined, including a parameter value indicating an incorrect placement of the element relative to the target position. A method comprising providing control information for adjusting the first position of the element based on the position error until at least one criterion for advancing the pre-programmed automation sequence is met.

15. By causing at least one processor to execute instructions, the device can provide at least one, Receiving information regarding the first position of an element positioned by the boom based on a pre-programmed automated sequence, Receiving image information including the target position of the element and a representation of at least a part of the element, Based on the aforementioned image information, the second position of the element and the target position of the element are determined. Based on the second position of the element and the target position of the element, a position error is determined, including a parameter value indicating an incorrect placement of the element relative to the target position of the element. A computer program for causing a computer to perform the following actions: providing control information for adjusting the first position of the element based on the position error until at least one criterion for advancing the pre-programmed automated sequence is met.

Images