Weight prediction device, crane control device, crane, weight prediction method, and weight prediction program

Abstract

To improve transport efficiency of sediment transport by a crane with a bucket.SOLUTION: A weight prediction device (1) comprises an image acquisition part (101) to acquire the gripping state image of an object captured in the period from a gripping start to a gripping completion, and a weight prediction part (102) to predict the object weight using the prediction model (112) constructed by performing the machine-learning of the relationship between the gripping state image captured during the previous lift of the object and the lifted object weight.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a weight prediction device that predicts the weight of an object to be lifted by a crane equipped with a bucket. [Background technology]

[0002] Waste brought into a waste treatment plant is temporarily stored in a storage facility called a waste pit, and then sent to an incinerator for incineration. A crane equipped with a bucket is generally used to transport the waste inside the waste pit. Technology for automating the operation of such cranes has been developed. For example, Patent Document 1 listed below discloses technology for automatically controlling a crane equipped with a bucket to stir the waste inside the waste pit. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 5185197 Summary of the Invention [Problem to be solved by the invention]

[0004] The automatic control in Patent Document 1 is based on the premise that the crane bucket always lifts the same volume of waste. However, when a crane is actually made to grab waste in a waste pit, the volume and weight of the waste lifted are often not constant. For example, if the waste at the position where the bucket is lowered is tilted or is compacted and hardened, the volume and weight of the waste lifted by the bucket can be significantly reduced.

[0005] In automatic control, it is only when the weight of the lifted trash is determined by a weighing scale attached to the wire that suspends the bucket, that is, after the trash has been lifted completely, that it becomes clear that the amount of trash lifted by the bucket is small. After it is determined that the amount of trash is small, the bucket is opened to release the lifted trash, and then the bucket is lowered again to re-grab the trash. Thus, in the prior art, the amount of trash gripped by the bucket could not be determined until the trash was lifted, which hindered the efficient transportation of trash. This is a problem that arises not only when transporting trash, but also when transporting any piled up object with a crane equipped with a bucket.

[0006] One aspect of the present invention aims to provide a technique that enables efficient transportation of piled objects by a crane equipped with a bucket. [Means for solving the problem]

[0007] In order to solve the above problems, a weight prediction device according to one embodiment of the present invention includes an image acquisition unit that acquires images of an object being gripped by a bucket, the images being taken during the period from the start of gripping of the object to the completion of gripping when a portion of a piled object is gripped and lifted by a crane equipped with a bucket, and a weight prediction unit that predicts the weight of the object being lifted by the bucket from the images being gripped using a prediction model constructed by machine learning of the relationship between the images being gripped that were taken when the object was previously lifted by the crane or another crane and the weight of the object lifted.

[0008] In addition, in order to solve the above-mentioned problems, a crane according to one embodiment of the present invention is provided with an imaging device inside a bucket for lifting an object to be transported, for taking an image during gripping, which is an image of the object being gripped by the bucket, during the period from the start of gripping the object to the completion of gripping when the bucket grips and lifts a portion of the object stacked up.

[0009] In addition, in order to solve the above-mentioned problems, a weight prediction method according to one embodiment of the present invention is a weight prediction method executed by one or more information processing devices, and includes: an image acquisition step of acquiring an image during gripping, which is an image of the object being gripped by the bucket, taken during a period from the start of gripping of the object to the completion of gripping, when a portion of a piled object is gripped and lifted by a crane equipped with a bucket; and a weight prediction step of predicting the weight of the object to be lifted by the bucket from the image during gripping, using a prediction model constructed by machine learning the relationship between the image during gripping taken when the object was lifted in the past by the crane or another crane, and the weight of the object lifted. Effect of the Invention

[0010] According to one aspect of the present invention, it is possible to efficiently transport piled objects using a crane equipped with a bucket. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing an example of the configuration of a weight prediction device and a crane control device according to a first embodiment of the present invention. FIG. [Diagram 2] FIG. 2 is a diagram showing a configuration of a crane control system including the weight prediction device and the crane control device. [Diagram 3] FIG. 13 is a diagram showing how a bucket grips and lifts a portion of the piled waste. [Figure 4] FIG. 2 is a diagram showing an example of the configuration of a prediction model used by the weight prediction device for weight prediction. [Diagram 5] 4 is a flowchart showing an example of a process executed by the weight prediction device and the crane control device. [Figure 6] FIG. 11 is a block diagram showing an example of the configuration of a weight prediction device according to a second embodiment of the present invention. [Figure 7] 4 is a flowchart illustrating an example of a process executed by the weight prediction device. [Figure 8] FIG. 2 is a diagram showing the appearance of a mock crane. [Figure 9] 1A and 1B are diagrams showing the state in which simulated trash is grabbed by the simulated crane, and an image of the simulated trash being grabbed. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] [Embodiment 1] (System Configuration) 2 is a diagram showing the configuration of a crane control system 7 according to this embodiment. As shown in the figure, the crane control system 7 includes a weight prediction device 1, a crane control device 3, and a crane 5. The crane control system 7 is a system that controls the crane 5 by the weight prediction device 1 and the crane control device 3.

[0013] FIG. 2 shows an example in which the crane control system 7 is applied to the transportation of waste stored in a pit P, which is a waste storage facility. The waste may be anything that can be transported by the crane 5, and may be, for example, household garbage or industrial waste. The crane control system 7 may also be used to transport any object other than waste. Therefore, "waste" in the following description may be read as any object.

[0014] The crane 5 includes a bucket 52 for lifting waste and a wire 51 for hoisting the bucket 52, and the bucket 52 includes a base 521 and claws 522. The base 521 of the bucket 52 is provided with a camera 6. The claws 522 are for gripping the waste and are attached rotatably to the base 521. Note that, for convenience of illustration, only two claws 522 are shown in Fig. 2, but it is sufficient to provide as many claws 522 as are necessary to stably grip the waste.

[0015] The image capturing device 6 captures images of waste being held by the bucket 52 during the period from the start of holding the waste to the completion of holding the waste when the bucket 52 holds and lifts a portion of the piled up waste. The image capturing device 6 may be, for example, a general camera that captures RGB (Red Green Blue) color images. The image capturing device 6 may also be, for example, an infrared camera or the like that is suitable for taking pictures in dark places.

[0016] The weight prediction device 1 uses the grasping image to predict the weight of the waste to be lifted by the bucket 52. The crane control device 3, which is a device for controlling the operation of the crane 5, obtains the weight prediction result by the weight prediction device 1 and controls the operation of the crane 5 in accordance with the prediction result.

[0017] (advantage) The advantages of the crane control system 7 will be described with reference to Fig. 3. Fig. 3 is a diagram showing how the bucket 52 grips and lifts a portion of the piled up waste. As shown in Fig. 3, when gripping and lifting a portion of the piled up waste, the crane control device 3 first opens the claws 522 of the bucket 52 (ST1). By opening the claws 522, the tips of the claws 522 face downward. The opening angle of the claws 522 is represented by the inclination angle of the base of the claws 522 with respect to the horizontal plane (θ in the figure). θ when the claws 522 are closed is a larger value than when the claws 522 are open.

[0018] Next, the crane control device 3 lowers the bucket 52 with the claws 522 open onto the waste (ST2). This causes the tips of the claws 522 to bite into the waste. Then, the crane control device 3 closes the claws 522 (ST3). When the opening angle of the claws 522 reaches or exceeds a predetermined threshold, the gripping operation is completed. In the example of FIG. 3, the gripping operation is completed when the opening angle of the claws 522 reaches approximately 90 degrees.

[0019] Here, the photographing device 6 photographs the waste during the period from the start of gripping the waste to the completion of gripping, that is, the period from ST2 to ST3. The images photographed during this period are the above-mentioned images during gripping. Note that, in the example of FIG. 3, the image is photographed at ST3, but the image may be photographed at a stage before the completion of gripping. In other words, the images during gripping may be photographed during the period when the movement of closing the claws 522 of the bucket 52 is affecting the appearance of the waste. For example, the image may be photographed when the opening angle of the claws 522 is 50% or 80% of the opening angle when completely closed.

[0020] At the ST3 stage, the waste has not been lifted, so its weight cannot be measured. However, the images during gripping provide clues for predicting the weight of the waste to be lifted by the bucket 52. In other words, there is a correlation between the images during gripping taken during gripping and the weight of the waste to be lifted once gripping is complete. For example, if the amount of waste shown in the images during gripping is small, the weight of the waste to be lifted will also be small, and conversely, if the amount of waste shown in the images during gripping is large, the weight of the waste to be lifted will tend to be large.

[0021] Therefore, the weight prediction device 1 predicts the weight of the waste to be lifted by the bucket 52 from the captured grasping image using a prediction model constructed by machine learning of the relationship between the grasping image taken when lifting waste in the past by the crane 5 or another crane and the weight of the lifted waste.

[0022] With this configuration, the weight of the waste can be predicted before it is lifted, reducing the time lost from performing a lifting operation before the desired weight of waste has been grasped, thereby making it possible to make waste transportation more efficient.

[0023] For example, the crane control device 3 may cause the crane 5 to re-gripping the waste when the weight predicted by the weight prediction device 1 is equal to or less than a predetermined lower limit. Similarly, the crane control device 3 may cause the crane 5 to re-gripping the waste when the weight predicted by the weight prediction device 1 is equal to or more than a predetermined upper limit. Then, the crane control device 3 may wind up the wire 51 to lift the waste when the weight predicted by the weight prediction device 1 is within an allowable range (ST4).

[0024] As described above, the crane control system 7 includes the camera 6 that captures the above-mentioned images during gripping during the period from the start of gripping the waste to the completion of gripping, the weight prediction device 1 that predicts the weight of the waste to be lifted by the bucket 52 from the images during gripping captured by the camera 6 using the above-mentioned prediction model, and the crane control device 3 that controls the operation of the crane 5 in accordance with the weight prediction result by the weight prediction device 1. This makes it possible to reduce time loss caused by performing a lifting operation when an object of the desired weight has not been gripped in the automatic control of the crane, and to make the transportation of the object more efficient.

[0025] As described above, the crane 5 is configured to have a camera device 6 provided inside the bucket 52 for taking images of the waste being gripped by the bucket 52 during the period from the start of gripping the waste to the completion of gripping when the bucket 52 grips and lifts a portion of the accumulated waste.

[0026] According to the above configuration, the state of the waste being gripped by the bucket 52 can be photographed without being obstructed by the bucket 52. Therefore, the crane 5 can be suitably applied to generating teacher data used for learning the above-mentioned prediction model. In addition, the weight prediction device 1 can predict the weight of the waste transported by the crane 5 with high accuracy.

[0027] However, depending on the type of object being transported by the crane 5, it may be difficult to capture images using the camera 6 installed inside the bucket 52. For example, when transporting general household waste, flying dust or the like may adhere to the camera 6, making it impossible to capture clear images during grabbing. For this reason, images during grabbing may be captured using a camera installed outside the bucket 52. For example, the camera may be installed inside a monitoring room that monitors the pit P.

[0028] (Detailed configuration of the device) The configurations of the weight prediction device 1 and the crane control device 3 will be described in more detail with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of the weight prediction device 1 and the crane control device 3.

[0029] As shown in the figure, the weight prediction device 1 includes a control unit 10 that controls each unit of the weight prediction device 1, and a storage unit 11 that stores various data used by the weight prediction device 1. The weight prediction device 1 also includes a communication unit 12 that allows the weight prediction device 1 to communicate with other devices (e.g., crane control device 3), an input unit 13 that accepts input of various data to the weight prediction device 1, and an output unit 14 that allows the weight prediction device 1 to output various data. The control unit 10 also includes an image acquisition unit 101 and a weight prediction unit 102. The storage unit 11 stores a gripping image 111 and a prediction model 112.

[0030] The image acquisition unit 101 acquires grasping images 111, which are images of an object grasped by the bucket 52, captured during a period from the start of grasping the object to the completion of grasping when a part of a piled object is grasped and lifted by a crane 5 equipped with a bucket 52. For example, the image acquisition unit 101 may acquire the grasping images 111 from the photographing device 6 by communication via the communication unit 12, or may acquire the grasping images 111 input by a user of the weight prediction device 1 or the like via the input unit 13. The acquired grasping images 111 are stored in the storage unit 11.

[0031] It is preferable that the image acquiring unit 101 acquires an image during gripping captured by a camera attached to the inside of the bucket 52 (the side facing the object to be gripped), such as the camera 6 shown in Figures 2 and 3. Since the image during gripping captured by the camera attached to the inside of the bucket 52 shows the state of the object to be gripped by the bucket 52 without being obstructed by the bucket 52, a highly accurate prediction can be made by predicting the weight using this image during gripping.

[0032] The weight prediction unit 102 predicts the weight of the object to be lifted by the bucket 52 from the during-grasping image 111 using a prediction model 112. The prediction model 112 is constructed by machine learning the relationship between the during-grasping image taken when an object was lifted in the past by the crane 5 or another crane and the weight of the lifted object. Details of the prediction model 112 will be described later in the section "Example of the configuration of a prediction model."

[0033] As described above, the weight prediction device 1 includes an image acquisition unit 101 that acquires images during gripping 111, which are images of an object gripped by the bucket 52, taken during a period from the start of gripping the object to the completion of gripping when a part of a piled object is gripped and lifted by the crane 5 equipped with the bucket 52, and a weight prediction unit 102 that predicts the weight of the object to be lifted by the bucket 52 from the images during gripping 111 acquired by the image acquisition unit 101, using a prediction model 112 constructed by machine learning the relationship between images during gripping taken when lifting an object in the past by the crane 5 or another crane and the weight of the lifted object. According to this configuration, the weight of the object can be predicted at a stage before it is lifted, so that the time loss caused by performing a lifting operation without gripping an object of the desired weight can be reduced, and the transportation of the object can be made more efficient.

[0034] 1, the crane control device 3 includes a control unit 30 that controls each unit of the crane control device 3, and a storage unit 31 that stores various data used by the crane control device 3. The crane control device 3 also includes a communication unit 32 that enables the crane control device 3 to communicate with other devices (e.g., weight prediction device 1), an input unit 33 that accepts input of various data to the crane control device 3, and an output unit 34 that enables the crane control device 3 to output various data. The control unit 30 also includes a crane control unit 301, an imaging control unit 302, and a prediction result acquisition unit 303.

[0035] The crane control unit 301 controls the operation of the crane 5. For example, the crane control unit 301 performs a process of moving the bucket 52 of the crane 5 to an arbitrary position in the pit P, a process of opening and closing the claws 521 of the bucket 52, a process of winding up and winding down the wire 51, and the like. In performing these processes, the crane control unit 301 refers to various information necessary for control, such as the measured value of the opening angle of the claws 521.

[0036] The photography control unit 302 controls the photography device 6 to photograph the gripping image 111. For example, the photography control unit 302 may cause the photography device 6 to photograph the gripping image 111 when the opening angle of the claws 521 becomes equal to or greater than a predetermined threshold value after the crane control unit 301 performs an operation to close the claws 521 of the bucket 52 (after the processing of ST in FIG. 3). The control of the photography device 6 may be performed by communication via the communication unit 32. Note that the communication may be wired communication or wireless communication.

[0037] The prediction result acquisition unit 303 acquires the weight prediction result obtained by the weight prediction device 1. The crane control unit 301 may cause the crane 5 to perform an operation of gripping the target object again when the prediction result acquired by the prediction result acquisition unit 303 is equal to or lower than a predetermined lower limit value and / or when the prediction result is equal to or higher than a predetermined upper limit value.

[0038] As described above, the crane control device 3 includes the prediction result acquisition unit 303 that acquires the weight prediction result from the weight prediction device 1, and the crane control unit 301 that causes the crane 5 to perform an operation to re-grab the object when the prediction result acquired by the prediction result acquisition unit 303 is equal to or lower than a predetermined lower limit value, or when the prediction result is equal to or higher than a predetermined upper limit value. This configuration reduces the time loss caused by performing a lifting operation when an object of the desired weight cannot be gripped in the automatic control of the crane 5, and makes it possible to transport the object more efficiently.

[0039] (Example of predictive model configuration) Fig. 4 is a diagram showing an example of the configuration of the prediction model 112. Fig. 4 shows two examples of the configuration of the prediction model 112. The upper part of Fig. 4 shows a prediction model 112A that is a convolutional neural network model.

[0040] Prediction model 112A is configured to use holding image 111 as input data, generate a feature vector of holding image 111 by repeating convolution and pooling, and predict weight from the generated feature vector. More specifically, prediction model 112A is configured to generate a 10-dimensional feature vector from holding image 111, which is a grayscale image of 64 pixels by 64 pixels, and calculate a predicted value of weight from the feature vector.

[0041] Prediction model 112A is a deep neural network model including multiple intermediate layers. Of course, the vertical and horizontal sizes (number of pixels) of input gripping image 111, the number of convolutional layers, the number of intermediate layers, and the like are not limited to the illustrated example, and may be appropriately designed according to the required prediction accuracy, etc. Also, a color image may be input instead of a grayscale image. This also applies to prediction model 112B described below.

[0042] The difference between prediction model 112B and prediction model 112A is that prediction model 112B combines a convolutional neural network with GP, i.e., a Gaussian process. Prediction model 112B shown in the figure is configured such that a 10-dimensional feature vector generated from a 64 pixel x 64 pixel grayscale gripping image 111 in the same manner as prediction model 112A is input to a Gaussian process. Prediction model 112B is called a deep kernel learning (DKL) model.

[0043] Before explaining DKL, we will briefly explain Gaussian processes, a non-parametric, machine learning technique used to infer functional relationships between input and output data.

[0044] N-point input data X=[x 1 ,…,x N ] T and the output data y=[y 1 ,…,y N ] T Given, we assume that the following function holds between these input and output data. We also place a Gaussian process on the function f.

[0045]

number

[0046] Here, ε n 〜Ν(0,β) is the observation noise. This means that the function value corresponding to the given data follows the Gaussian distribution:

[0047]

number

[0048] Here, f=[f(x 1 ),…,f(x N )] Twhere K is an N×N kernel Gram matrix. The element in row i and column j of the kernel Gram matrix K is expressed as [K] using the kernel function k( , ). ij =k(x i ,x j )

[0049] In a Gaussian process, the unknown input data x * Output data for y * is probabilistically predicted using a Gaussian distribution.

[0050]

number

[0051] where k *N =[k(x * ,x 1 ),…,f(x * ,x N )] T Katsuk * =(x * ,x * ) and m(x * ) is x * The average of σ(x * ) is x * is the variance of.

[0052] The kernel function used in Gaussian process regression affects regression performance, so it must be designed to suit the data used in regression. When the kernel function has a parameter θ, the optimal kernel parameter θ according to the data can be determined by optimizing the log marginal likelihood lnp(y|θ) shown in the following formula (7) as the objective function. * (See equation (8) below) can be learned.

[0053]

number

[0054] DKL is a method in which a neural network is incorporated into a kernel function used in a Gaussian process. In this method, the neural network incorporated into the kernel function is trained in a framework of optimizing hyperparameters using the marginal likelihood of the Gaussian process. In the prediction model 112B, as shown in FIG. 4, the neural network incorporated into the kernel function is a convolutional neural network that excels in image processing. This makes it possible to handle high-dimensional data such as images, which have been difficult to handle in the Gaussian process, as input. In addition, when constructing the prediction model 112B, parameters are updated based on the marginal likelihood, so that the prediction model 112B has the advantage of being less susceptible to overlearning even when a small amount of training data is used for learning, compared to the prediction model 112A.

[0055] The kernel function used in DKL is expressed as follows:

[0056]

number

[0057] Here, g(·;w) denotes a convolutional neural network with parameter w that converts the input image into features. The parameter w of the convolutional neural network can be regarded as a parameter of the kernel function, and is trained by the above-mentioned formulas (7) and (8) in the same way as the parameter θ.

[0058] The weight prediction unit 102 may predict the weight of the object using the prediction model 112B described above, that is, the prediction model 112B that combines a convolutional neural network that extracts the feature amount of the holding image 111 and a Gaussian process that infers the weight of the object from the feature amount. By using the prediction model 112B, it becomes possible to handle the holding image 111, which is high-dimensional data, in a Gaussian process. In addition, since the prediction model 112B is constructed by calculation based on the probability statistics of the Gaussian process, it has an advantage that overlearning is unlikely to occur even if the amount of training data used for learning is small. In other words, when the prediction model 112B is used, high prediction accuracy can be expected even if the amount of training data used for the learning is small.

[0059] Of course, the prediction model 112 is not limited to the above example as long as it can calculate a predicted value of weight from the gripping image 111 or a feature amount extracted from the gripping image 111.

[0060] (Processing flow) The flow of the process executed by the weight prediction device 1 and the crane control device 3 will be described with reference to Fig. 5. Fig. 5 is a flowchart showing an example of the process executed by the weight prediction device 1 and the crane control device 3. Note that S1 to S6 in Fig. 5 are processes executed by the crane control device 3, and S11 to S13 are processes executed by the weight prediction device 1 (of which S11 and S12 are weight prediction methods). Fig. 5 shows an example in which the object to be grasped by the crane 5 is garbage (waste) stored in a pit P.

[0061] In S1, the crane control unit 301 causes the crane 5 to execute an operation of grabbing the waste. Specifically, as shown in FIG. 3, the crane control unit 301 causes the crane 5 to lower the claws 522 of the bucket 52 of the crane 5 onto the waste with the claws 522 in an open state, and then causes the crane 5 to execute an operation of closing the claws 522.

[0062] In S2, the photography control unit 302 causes the photography device 6 to photograph the gripping image 111. The photography may be performed during the period from the start of gripping to the completion of gripping, that is, during the period from when the claws 522 of the bucket 52 start to close to when they finish closing. The gripping image 111 photographed by the photography device 6 is transmitted to the weight prediction device 1.

[0063] In the weight prediction device 1 that has received the gripping image 111, the image acquisition unit 101 acquires the gripping image 111 (S11). Next, the weight prediction unit 102 predicts the weight of the trash to be lifted by the bucket 52 from the gripping image 111 acquired in S11 using the prediction model 112. More specifically, the weight prediction unit 102 predicts the weight of the trash based on an output value obtained by inputting the gripping image 111 acquired in S11 to the prediction model 112 (S12).

[0064] Then, the weight prediction unit 102 transmits the prediction result of S12 to the crane control device 3 (S13). This ends the processing of the weight prediction device 1.

[0065] In the crane control device 3 that has received the prediction result, the prediction result acquisition unit 303 acquires the prediction result (S3). Then, the crane control unit 301 judges whether or not the weight indicated by the prediction result acquired by the prediction result acquisition unit 303 is equal to or greater than a predetermined threshold (in other words, whether the predicted weight of the garbage is too little) (S4). Note that the crane control unit 301 may judge whether or not the weight indicated by the prediction result is equal to or greater than a predetermined threshold (in other words, whether the predicted weight of the garbage is too much), or may judge whether or not it is within a predetermined appropriate range.

[0066] If the answer in S4 is YES, the process proceeds to S5, where the crane control unit 301 causes the crane 5 to lift the waste grabbed by the bucket 52. Specifically, the crane control unit 301 performs a process of winding up the wire 51 to a predetermined length. This ends the process of the crane control device 3 shown in FIG. 5. It should be noted that any method can be used to dispose of the lifted waste. For example, the waste may be dropped at the point where it is lifted and loosened, or it may be transported to another point and reloaded, or it may be scattered and agitated within the pit P, or it may be thrown into a hopper connected to an incinerator.

[0067] On the other hand, if the result of S4 is NO, the process proceeds to S6, where the crane control unit 301 opens the claws 521 of the bucket 52 and winds up the wire 51 to a predetermined length. After this, the process returns to S1. That is, in this case, the garbage that was once gripped is released and then gripped again. Note that when a new gripping image 111 is captured, the processes of S11 to S13 are performed again.

[0068] As described above, the weight prediction method according to this embodiment includes an image acquisition step (S11) of acquiring a gripping image 111, which is an image of an object to be gripped by the bucket 52, taken during a period from the start of gripping the object to the completion of gripping when a part of a piled object (garbage in the above example) is gripped and lifted by the crane 5 equipped with the bucket 52, and a weight prediction step (S12) of predicting the weight of the object to be lifted by the bucket 52 from the gripping image 111 acquired in S11 using a prediction model 112 constructed by machine learning the relationship between the gripping image taken when lifting an object by the crane 5 or another crane in the past and the weight of the lifted object. This makes it possible to predict the weight of the object before it is lifted, thereby reducing the time loss caused by performing a lifting operation in a state where an object of the desired weight has not been gripped, and making it possible to efficiently transport the object.

[0069] [Embodiment 2] Another embodiment of the present invention will be described below. For convenience of explanation, the same reference numerals are given to members having the same functions as those described in the above embodiment, and the explanations thereof will not be repeated. In this embodiment, a weight prediction device 1A that combines the functions of the weight prediction device 1 and the crane control device 3 in the first embodiment will be described.

[0070] (Device configuration) Fig. 6 is a block diagram showing a configuration example of the weight prediction device 1A. The weight prediction device 1A is different from the weight prediction device 1 shown in Fig. 1 in that the control unit 10 and the storage unit 11 are replaced by a control unit 10A and a storage unit 11, respectively. The control unit 10A includes an image acquisition unit 101 and a weight prediction unit 102 included in the weight prediction device 1, as well as a photography control unit 103, a weight specification unit 104, a teacher data generation unit 105, a learning unit 106, and a crane control unit 107. The storage unit 11A stores teacher data 113 in addition to a gripping image 111 and a prediction model 112.

[0071] The photography control unit 103, like the photography control unit 302 included in the crane control device 3 of the embodiment 1, controls the photography device 6 to photograph the grasping image 111. For example, the photography control unit 103 may transmit a photography instruction to the photography device 6 by communication via the communication unit 12 during the period from the start of grasping the object by the crane 5 to the completion of grasping, thereby causing the photography device 6 to photograph the grasping image 111.

[0072] The weight specifying unit 104 specifies the actual weight of the lifted object after the period from when the crane 5 starts to grip the object to when it finishes gripping the object. For example, if a weighing scale is provided at the end of the wire 51, the actual weight of the lifted object can be measured. In this case, the weight specifying unit 104 may specify the measurement value of the weighing scale. The measurement value may be input by the user via the input unit 13, or may be input from another device via the communication unit 12.

[0073] The teacher data generating unit 105 associates the holding image 111 captured by the photography control unit 103 with the weight identified by the weight identifying unit 104 to generate teacher data 113 used for learning the prediction model 112. The generated teacher data 113 is stored in the storage unit 11A.

[0074] In this way, the weight prediction device 1A includes an imaging control unit 103 that controls the imaging device 6 to capture images 111 during gripping during the period from when the crane 5 starts to grip the object to when the crane 5 finishes gripping, a weight specifying unit 104 that specifies the actual weight of the object lifted after the above period, and a teacher data generation unit 105 that generates teacher data 113 to be used for learning the prediction model 112 by associating the images 111 during gripping captured by the imaging control unit 103 with the weight specified by the weight specifying unit 104. This makes it possible to automatically generate and accumulate teacher data 113 during the period when the crane 5 is being made to perform the operation of lifting and transporting the object.

[0075] The learning unit 106 generates a prediction model 112 by performing machine learning using the teacher data 113. The prediction model 112 may be, for example, a convolutional neural network model such as the prediction model 112A shown in Fig. 4, a DKL model such as the prediction model 112B, or another model.

[0076] The crane control unit 107 controls the operation of the crane 5, similar to the crane control unit 301 included in the crane control device 3 of the first embodiment. Furthermore, when the weight prediction unit 102 predicts the weight, the crane control unit 107 controls the operation of the crane 5 in accordance with the prediction result. For example, the crane control unit 107 may cause the crane 5 to perform an operation of gripping the object again at least in one of the cases where the prediction result of the weight prediction unit 102 is equal to or lower than a predetermined lower limit value and where the prediction result is equal to or higher than a predetermined upper limit value.

[0077] (Processing flow) The flow of the process (weight prediction method) executed by the weight prediction device 1A will be described with reference to Fig. 7. Fig. 7 is a flowchart showing an example of the process executed by the weight prediction device 1A. Note that Fig. 7 shows an example in which the object to be gripped by the crane 5 is garbage (waste) stored in a pit P, similar to Fig. 5.

[0078] In S21, the crane control unit 107 lowers the bucket 52 of the crane 5 onto the waste stored in the pit P. At this time, the claws 522 of the bucket 52 of the crane 5 are kept in an open state (see ST1 in FIG. 3).

[0079] In S22, crane control unit 107 causes bucket 52 to execute a closing operation. More specifically, crane control unit 107 causes claws 522 of bucket 52 to close until the closing operation is stopped by pressure from the gripped waste. For this reason, the open angle of claws 521 after the process of S22 is completed may differ depending on the amount, quality, etc. of the gripped waste.

[0080] Therefore, in S23, the crane control unit 107 judges whether the opening angle of the claws 521 is equal to or greater than a predetermined threshold, in other words, whether the bucket 52 is closed to a sufficient degree to capture the gripping image 111. If the judgment in S23 is YES, the process proceeds to S26. On the other hand, if the judgment in S23 is NO, the process proceeds to S24.

[0081] In S24, the crane control unit 107 judges whether the cumulative number of times the closing operation has been performed in the most recent S22 has reached a predetermined upper limit. If the answer to S24 is YES, the process proceeds to S26. On the other hand, if the answer to S24 is NO, the process proceeds to S25.

[0082] In S25, the crane control unit 107 slightly lifts up the bucket 52 so that the claws 521 can dig more firmly into the garbage held by the bucket 52. Thereafter, the process returns to S21, the bucket 52 is lowered back onto the garbage, and the closing operation is executed again. In this manner, in the weight prediction method of FIG. 7, the crane control unit 107 repeats the closing operation up to a predetermined upper limit number of times until the opening angle of the claws 521 holding the garbage becomes equal to or greater than a predetermined threshold value. This allows the opening angle of the claws 521 at the time of capturing the gripping image 111 to be unified at a high level, so that the weight can be predicted stably. In addition, since an upper limit is set for the number of times the closing operation can be repeated, it is possible to prevent excessive time loss caused by the closing operation being repeated endlessly.

[0083] In S26, the photography control unit 103 controls the photography device 6 to photograph the gripping image 111. If the determination in S23 is YES and the process proceeds to S26, the gripping image 111 is photographed in a state where the open angle of the claws 521 gripping the garbage is equal to or greater than a predetermined threshold. On the other hand, if the determination in S24 is YES and the process proceeds to S26, the gripping image 111 is photographed in a state where the open angle of the claws 521 gripping the garbage is less than the predetermined threshold.

[0084] In S27, the image acquisition unit 101 acquires the gripping image 111 captured in S26. Then, in S28, the weight prediction unit 102 uses the prediction model 112 to predict the weight of the trash to be lifted by the bucket 52 from the gripping image 111 acquired in S27.

[0085] In S29, the crane control unit 107 judges whether the predicted weight is equal to or greater than a predetermined threshold (in other words, whether the predicted weight of the waste is too little). Note that the crane control unit 107 may judge whether the weight indicated by the prediction result is equal to or greater than a predetermined threshold (in other words, whether the predicted weight of the waste is too much), or may judge whether it is within a predetermined appropriate range.

[0086] If the determination in S29 is YES, the process proceeds to S30, where the crane control unit 107 causes the crane 5 to lift the waste grabbed by the bucket 52. Specifically, the crane control unit 107 performs a process of winding up the wire 51 to a predetermined length. This ends the process in FIG. 7. It is possible to decide how to dispose of the lifted waste.

[0087] On the other hand, if the result of S29 is NO, the process proceeds to S31, where the crane control unit 107 opens the claws 521 of the bucket 52 and winds up the wire 51 to a predetermined length. After this, the process returns to S21.

[0088] When generating the teacher data 113, after the processing of S31, the weight identification unit 104 identifies the actual weight of the garbage lifted by the crane 5. Then, the teacher data generation unit 105 generates the teacher data 113 by associating the holding image 111 acquired in S27 with the weight identified by the weight identification unit 104.

[0089] [Demonstration experiment] An experiment was conducted using a simulated crane to verify the accuracy of weight prediction by the weight prediction device 1A. This experiment will be described with reference to Fig. 8 and Fig. 9. Fig. 8 is a diagram showing the appearance of the simulated crane 5A, and Fig. 9 is a diagram showing the state in which the simulated garbage G is being gripped by the simulated crane 5A, and an image of the simulated garbage G being gripped.

[0090] FIG. 8 shows the appearance of the simulated crane 5A as seen from diagonally above, and as seen from the side where the camera device 6A is provided. The simulated crane 5A has four claws 522A attached to a base 521A, to which the camera device 6A is attached and a protective member 61A that surrounds the camera device 6A on all four sides. Although it is not essential to provide the protective member 61A, providing the protective member 61A has the advantage that the object to be grasped can be prevented to some extent from directly hitting the camera device 6A. As with a general bucket crane, the simulated crane 5A can grasp and transport an object with the four claws 522A.

[0091] Fig. 9 shows four images, IMG1 to IMG4. IMG1 shows the state in which the simulated crane 5A is grabbing the simulated garbage G in the simulated pit P. The simulated garbage G that is the object to be transported by the simulated crane 5A is rubber balls. IMG2 shows the state in which the claws 522A of the simulated crane 5A are closed to grab the simulated garbage G.

[0092] IMG3 and IMG4 are example images taken while catching imitation trash G as shown in IMG2. IMG3 is an image taken when catching a large amount of imitation trash G. From IMG3, it can be seen that the entire image is packed with imitation trash G with few gaps.

[0093] On the other hand, IMG4 is an image taken when only a small amount of imitation garbage G was grasped. In IMG4, there are gaps in the imitation garbage G in the upper left and lower right areas of the image, and the entire imitation garbage G appears brighter than in IMG3. This is because external light was entering through the gap between the camera 6A and the imitation garbage G and hitting the imitation garbage G.

[0094] In the experiment, the process of Fig. 7, the process of identifying the actual weight of the garbage by the weight identifying unit 104, and the process of generating the training data 113 by the training data generating unit 105 were repeated while randomly changing the position at which the simulated garbage G was grasped. The threshold value of S23 in Fig. 7 was set to 80 degrees. Then, the training data 113 generated by these processes was used to generate the prediction models 112A and 112B shown in Fig. 4.

[0095] Moreover, the threshold value of S23 in Fig. 7 was changed to 0 degrees to generate prediction models 112A and 112B, and the threshold value of S23 in Fig. 7 was changed to 45 degrees to generate prediction models 112A and 112B. In other words, prediction models 112A and 112B corresponding to the threshold values ​​of S23 of 0 degrees, 45 degrees, and 80 degrees, respectively, were generated.

[0096] Then, test data (data that correlates the holding image 111 with weight, similar to the teacher data 113, and that was not used for learning) was input into the generated predictive models 112A and 112B, and the predictive accuracy of these models was evaluated based on the output values ​​obtained.

[0097] As a result of this experiment, when prediction models 112A and 112B generated with the threshold of S23 set to 45 degrees or 80 degrees were used, prediction results that followed the actual weight were obtained. On the other hand, when the threshold of S23 was set to 0 degrees, that is, when prediction models 112A and 112B were trained using an image captured with claws 522A fully open as shown in IMG1 of FIG. 9, prediction results that followed the actual weight were not obtained. This experimental result supports the need to use grasping image 111 captured during the period from the start of grasping to the completion of grasping, more specifically, after claws 522A began to close, for learning and weight prediction.

[0098] In addition, the results of this experiment showed that the prediction accuracy was higher when the threshold value of S23 was set to 80 degrees than when it was set to 45 degrees. The results of this experiment show that the prediction accuracy is higher as the opening angle of the bucket 52A becomes larger, that is, the closer the bucket 52A is closed. However, when the opening angle of the claws 522A becomes 90 degrees, that is, the bucket 52A is completely closed, regardless of the amount of the simulated garbage G being gripped, the bucket 52A is filled with the simulated garbage G and there is no or little space, so it is expected that the prediction accuracy will actually decrease. For this reason, it is preferable to capture the gripping image 111 with the opening angle of the claws 522A of the bucket 52A set to about 80 degrees (a state slightly open from the completely closed state where the opening angle of the claws 522A is 90 degrees).

[0099] Furthermore, the results of this experiment showed that prediction model 112B had higher prediction accuracy than prediction model 112A. Of course, prediction model 112A and prediction model 112B were trained using the same set of training data 113.

[0100] In addition, an experiment was also conducted to trial a task in which the process of Figure 7 was performed using the predictive model 112B, and after S30 in Figure 7, the series of processes of transporting the simulated garbage G grasped by the bucket 52A to a predetermined position was repeated until the amount of transported simulated garbage G reached a predetermined amount.

[0101] The results of this experiment demonstrated that it is possible to determine the need for re-gripping at a timing sufficiently earlier than measuring the weight of the imitation garbage G with a weighing scale attached to the rope suspending the bucket 52A after gripping up the imitation garbage G and determining whether or not re-gripping is necessary from this measurement result. This experiment also demonstrated that it is possible to transport the imitation garbage G more efficiently than in the case where the need for re-gripping is determined from the weight of the imitation garbage G actually measured with a weighing scale.

[0102] [Modifications] The execution entity of each process described in each of the above-mentioned embodiments is arbitrary and is not limited to the above-mentioned examples. In other words, as long as the execution entity of each process described in each of the above-mentioned embodiments can be executed, the devices constituting the crane control system 7 can be appropriately changed.

[0103] For example, in the crane control system 7 in Fig. 1, the weight estimation device 1 and the crane control device 3 are separate devices, but the crane control system 7 can also be configured using a weight estimation device 1A that combines them into one device. Also, the function of generating teacher data 113 for the weight estimation device 1A and the function of generating the prediction model 112 by machine learning may be provided in another device. The respective processes shown in Fig. 5 and the respective processes shown in Fig. 7 may be executed by one or more arbitrary information processing devices (computers).

[0104] [Software implementation example] The functions of the weight prediction devices 1 and 1A (hereinafter referred to as the "devices") can be realized by a program for causing a computer to function as the device, and a program (weight prediction program) for causing a computer to function as each control block of the device (particularly each part included in the control unit 10 or 10A).

[0105] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program to realize each function described in each of the above embodiments.

[0106] The program may be non-transitory and may be recorded in one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be provided to the device via any wired or wireless transmission medium.

[0107] In addition, some or all of the functions of each of the control blocks can be realized by a logic circuit. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of each of the control blocks can be realized by, for example, a quantum computer.

[0108] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0109] 1. 1A Weight Prediction Device 101 Image acquisition unit 102 Weight Prediction Department 103 Shooting control section 104 Weight identification section 105 Teacher Data Generation Unit 106 Learning Department 107 Crane control unit 111 Grasping Image 112 Predictive Models 113 Teacher Data 3 Crane control device 301 Crane control unit 302 Shooting control section 303 Prediction result acquisition unit 5. Crane 52 Bucket 6. Imaging Device

Claims

1. an image acquisition unit that acquires images of the object being grasped by the bucket, the images being captured during the period from when the object is grasped and lifted by a crane equipped with a bucket, to when the object is grasped and lifted by the crane; and a weight prediction unit that predicts the weight of the object to be lifted by the bucket from the grasping image using a prediction model constructed by machine learning of the relationship between the grasping image taken when the object was previously lifted by the crane or another crane and the weight of the lifted object.

2. 2. The weight prediction device according to claim 1, wherein the weight prediction unit predicts the weight of the object using the prediction model that combines a convolutional neural network that extracts features of the gripping image and a Gaussian process that infers the weight of the object from the features.

3. The weight prediction device according to claim 1 , wherein the image acquisition unit acquires the gripping image captured by an image capture device attached to an inside of the bucket.

4. an imaging control unit that controls the imaging device during the period to capture the during-grasping image; a weight identifier that identifies an actual weight of the object lifted after the period of time; The weight prediction device according to any one of claims 1 to 3, further comprising a teacher data generation unit that generates teacher data to be used in learning the prediction model by associating the holding image captured by the photography control unit with the weight identified by the weight identification unit.

5. A prediction result acquisition unit that acquires a weight prediction result obtained by the weight prediction device according to any one of claims 1 to 4; a crane control unit that causes the crane to perform a re-gripping operation of the object when the prediction result is equal to or lower than a predetermined lower limit value and / or when the prediction result is equal to or higher than a predetermined upper limit value.

6. A crane is provided with a photographing device inside a bucket for lifting an object to be transported, for taking an image during gripping, which is an image of the object being gripped by the bucket, from the start of gripping the object to the completion of gripping when the bucket grips and lifts a portion of the object stacked up.

7. A weight prediction method executed by one or more information processing devices, comprising: an image acquiring step of acquiring images during gripping, which are images of the object being gripped by the bucket, taken during a period from when gripping of the object is started to when gripping of the object is completed, when a part of the object being piled up is gripped and lifted by a crane equipped with a bucket; and a weight prediction step of predicting the weight of the object to be lifted by the bucket from the grasping image using a prediction model constructed by machine learning of the relationship between the grasping image taken when the object was previously lifted by the crane or another crane and the weight of the lifted object.

8. A weight prediction program for causing a computer to function as the weight prediction apparatus according to claim 1, the weight prediction program causing a computer to function as the image acquisition unit and the weight prediction unit.

Images