Object input management device, program, and method

Abstract

To more accurately identify a loading position of an object such as stone by simply and accurately detecting the open / closed state of a bucket.SOLUTION: An image data acquisition unit 111 acquires operation video data capturing an image of an operation of an operator in a control room 7 from a camera 8a, and acquires opening / closing video data capturing an image of a bucket 5, the open / closed state of which changes in response to the operation of the operator, from a camera 8b. A coordinate data acquisition unit 112 acquires two-dimensional coordinate values indicating the position of the bucket 5. A machine learning unit 113 performs machine learning using the operation video data and the opening / closing video data acquired by the image data acquisition unit 111 in the machine learning process to generate a learning model. A calculation unit 117 calculates a throwing position and the number of throws of the stone material carried out and thrown by the bucket 5, based on the two-dimensional coordinate values acquired by the coordinate data acquisition unit 112 and the opening / closing data acquired by an acquisition unit 115.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a technique for throwing objects such as stones into water. [Background technology]

[0002] When dumping stone into water from a crane ship during port construction work, a system is known that uses positioning from a GNSS (Global Navigation Satellite System) device installed at the tip of the crane boom to determine the location where the stone has been dumped from a bucket 5 suspended from the crane boom (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-24400 Summary of the Invention [Problem to be solved by the invention]

[0004] However, on crane ships, stone is dropped by opening a bucket suspended by a wire from the tip of the crane boom while the boom is stopped. Therefore, even if the stone dropping position can be managed using GNSS, in order to determine the number of times the stone is dropped, the open / closed state of the bucket must be checked and recorded each time, and the number of times and amount of stone dropped within a specified range within the target area for stone dropping must be counted and managed separately.

[0005] Therefore, the present invention aims to more accurately identify not only the loading position of an object such as stone, but also the number of times and the amount of loading based on that number of times, by simply and accurately detecting the opening and closing state of the bucket using image data of the bucket opening and closing and image data of the bucket operation. [Means for solving the problem]

[0006] In order to solve the above problem, the object input management device of the present invention comprises an image data acquisition unit that acquires operation video data that captures an operator's operation on a carry-out tool and opening and closing video data that captures the open and closed state of the carry-out tool, the open and closed state of which changes in response to the operator's operation; a coordinate data acquisition unit that acquires two-dimensional coordinate values ​​that indicate the position of the carry-out tool; a memory unit that stores a learning model generated by machine learning using the operation video data as an explanatory variable and opening and closing data that indicates the open and closed state of the carry-out tool identified using the opening and closing video data as an objective variable; an acquisition unit that inputs the operation video data acquired when the carry-out tool is carrying out and inputting objects into the stored learning model to acquire the opening and closing data; and a calculation unit that calculates the input position and number of times the object is inputted by the carry-out tool based on the two-dimensional coordinate values ​​acquired when the opening and closing data is acquired and the acquired opening and closing data.

[0007] The carrying-out device may be provided with a mesh division unit that divides the target area into which the object is to be thrown into a plurality of meshes of a predetermined size, and the calculation unit may calculate the throw-in position and the number of times the object is thrown in for each mesh divided by the mesh division unit.

[0008] The unloading implement may be a bucket of a construction machine.

[0009] The calculation unit may further calculate, for each mesh divided by the mesh dividing unit, an approximate amount of objects fed in by the carrying-out appliance based on the capacity of the carrying-out appliance.

[0010] The calculation unit may calculate the throw-in distribution and throw-in pile height of the object based on the throw-in position, throw-in number, and approximate throw-in amount of the object for each mesh calculated each time the object is thrown from the carrying-out device.

[0011] In addition, the program of the present invention is a program for causing a computer to execute the following steps: an image data acquisition step for acquiring operation video data capturing an operator's operation on a discharge device and opening / closing video data capturing the open / closed state of the discharge device, the open / closed state of which changes in response to the operator's operation; a coordinate data acquisition step for acquiring two-dimensional coordinate values ​​indicating the position of the discharge device; a storage step for storing a learning model generated by machine learning using the operation video data as an explanatory variable and opening / closing data indicating the open / closed state of the discharge device identified using the opening / closing video data as an objective variable; an acquisition step for inputting the operation video data acquired when the discharge device is discharging and inserting objects into the stored learning model to acquire the opening / closing data; and a calculation step for calculating the insertion position and number of insertions of the object transported and inserted by the discharge device based on the two-dimensional coordinate values ​​acquired when the opening / closing data was acquired and the acquired opening / closing data.

[0012] The method may include a division step in which the target area into which the carrying device throws the object is divided into a plurality of meshes of a predetermined size, and in the calculation step, the throwing position and the number of throwing times of the object are calculated in units of meshes divided by the division step.

[0013] In the calculation step, an approximate amount of objects carried out by the carrying-out device may be calculated for each mesh divided by the dividing step using the capacity of the carrying-out device.

[0014] In the calculation step, the throw-in distribution and throw-in pile height of the object may be calculated based on the throw-in position, throw-in number and approximate throw-in amount of the object for each mesh calculated each time the object is thrown from the carrying-out device.

[0015] The present invention may also be an object input management method for managing the input positions and the number of times objects are input using the object input management device described above. [Effects of the Invention]

[0016] According to the present invention, by simply and accurately detecting the open / closed state of the bucket, it is possible to more accurately determine the throwing position, number of throwings, and throwing amount of objects such as stone for each mesh within a specified range. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a plan view illustrating a main part of a crane ship 1 according to an embodiment of the present invention. [Figure 2] 1 is a side view illustrating a main part of a crane ship 1 according to an embodiment of the present invention. [Figure 3] FIG. 1 is a block diagram showing the hardware configuration of a stone input management system 10. [Figure 4] FIG. 2 is a block diagram showing an example of the hardware configuration of a computer 11. [Figure 5] FIG. 2 is a block diagram showing an example of the functional configuration of a computer 11. [Figure 6] 10 is a diagram illustrating a video captured by a camera 8a installed in the wheelhouse 7. FIG. [Figure 7] 1 is a plan view illustrating a mesh M set around a crane ship 1. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of the present invention will be described. FIG. 1 is a plan view illustrating the main components of a crane ship 1 according to one embodiment of the present invention, and FIG. 2 is a side view illustrating the main components of the crane ship 1. As shown in FIGS. 1 and 2, the crane ship 1, which is a construction machine, includes a slewing mechanism 2 supported at a predetermined position on the deck so as to be rotatable in the direction of arrow a, a boom 3 supported on the slewing mechanism 2 so as to be rotatable up and down, a wire 4 reeled out from the tip of the boom 3, a bucket 5 serving as a carrying tool suspended from the wire 4, a hold 6 in which objects to be thrown into water (stones in this embodiment) are loaded, a wheelhouse 7 equipped with operating tools for operating the boom 3 and bucket 5, a camera 8a installed in the wheelhouse 7, a camera 8b installed at a predetermined position, such as near the connection between the slewing mechanism 2 and the boom 3, and two GNSS devices 9a, 9b installed at different positions along the length of the boom 3. The camera 8a captures images of the interior space of the wheelhouse 7, particularly the space including the operating tools for operating the bucket 5 to open and close it. Camera 8b captures an image of the space including bucket 5 suspended from the tip of boom 3 at a position that allows the open / closed state of bucket 5 to be determined. GNSS devices 9a and 9b perform positioning using GNSS signals emitted from artificial satellites.

[0019] In the crane ship 1, stones in the hold 6 are grabbed with a bucket 5 suspended from a wire 4 reeled out from the tip of a boom 3, and the bucket 5 is moved outward by crane operation and released within the desired range above the water surface to dump the stones into the water. By repeating this dumping operation multiple times, a mound of the desired shape is created on the water bottom. Note that in Figures 1 and 2, the Y axis is an axis parallel to the overall length of the crane ship 1, with its positive direction running from the stern to the bow, the X axis is an axis parallel to the overall width of the ship, with its positive direction running from starboard to port, and the Z axis is an axis perpendicular to the X and Y axes, with its positive direction running from bottom to top (the same applies below).

[0020] Figure 3 is a diagram showing the hardware configuration of the stone input management system 10 installed on the crane ship 1. The stone input management system 10 is a system in which a computer 11 functioning as the object input management device according to the present invention, cameras 8a and 8b, and GNSS devices 9a and 9b are networked together via communication lines such as Ethernet or optical fiber.

[0021] FIG. 4 is a diagram showing the hardware configuration of a computer 11 that functions as an object input management device. The computer 11 is installed, for example, in the wheelhouse 7. The computer 11 physically includes a processor 1101, a memory 1102, a storage 1103, a communication device 1104, an input device 1105, an output device 1106, and a bus connecting these. Each of these devices operates using power supplied from a power source (not shown). The hardware configuration of the computer 11 may be configured to include one or more of the devices shown in FIG. 4, or may be configured without including some of the devices. Furthermore, the computer 11 may be externally attached to the outside of the computer 11.

[0022] Each function in computer 11 is realized by loading specified software (programs) onto hardware such as processor 1101 and memory 1102, causing processor 1101 to perform calculations, control communications via communication device 1104, acquire data transmitted from other devices, and control at least one of reading and writing data in memory 1102 and storage 1103.

[0023] The processor 1101 controls the entire computer by running, for example, an operating system, and may be configured as a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc.

[0024] The processor 1101 reads programs (program codes), software modules, data, etc. from at least one of the storage 1103 and the communication device 1104 into the memory 1102, and executes various processes in accordance with these.

[0025] The memory 1102 is a computer-readable recording medium and may be configured by, for example, at least one of a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), etc. The memory 1102 may also be called a register, a cache, a main memory (primary storage device), etc. The memory 1102 can store executable programs (program codes), software modules, etc. for implementing the method according to this embodiment.

[0026] The storage 1103 is a computer-readable recording medium such as a hard disk drive, and the storage 1103 may also be called an auxiliary storage device.

[0027] The communication device 1104 is hardware (transmitting / receiving device) for communicating with other devices such as cameras 8a, 8b and GNSS devices 9a, 9b via at least one of wired and wireless connections, and is also referred to as, for example, a network device, network controller, network card, or communication module.

[0028] The input device 1105 is an input device (for example, a keyboard, a mouse, a microphone, a switch, a button, etc.) that accepts input from the outside. The output device 1106 is an output device (for example, a display, a speaker, an LED lamp, etc.) that outputs to the outside.

[0029] 5 is a diagram showing an example of the functional configuration of the computer 11. The image data acquisition unit 111 acquires operation video data from the camera 8a capturing images of operations performed on the bucket 5 by the operator in the wheelhouse 7, and acquires opening / closing video data from the camera 8b capturing images of the bucket 5, the open / close state of which changes in response to the operator's operation. The operation video data from the camera 8a and the opening / closing video data from the camera 8b are linked based on the time of capture. In this embodiment, there are broadly two types of processing: machine learning processing and loading management processing. In the machine learning processing, the image data acquisition unit 111 acquires both the operation video data and the opening / closing video data described above, and in the loading management processing, it acquires and uses only the operation video data described above.

[0030] The coordinate data acquisition unit 112 acquires two-dimensional coordinate values ​​(X, Y) indicating the position of the bucket 5. Specifically, the coordinate data acquisition unit 112 acquires three-dimensional coordinate values ​​(X, Y, Z) of each of the GNSS devices 9a, 9b from the GNSS devices 9a, 9b installed at known positions on the boom 3, and calculates the angle between the longitudinal direction of the boom 3 and the horizontal plane and the direction indicated by the horizontal component of the longitudinal direction of the boom 3 based on the difference between these three-dimensional coordinate values ​​(X, Y, Z). Furthermore, the coordinate data acquisition unit 112 calculates the two-dimensional coordinate value (X, Y) of the tip of the boom 3 based on the calculated angle and direction, the previously measured distance between the GNSS devices 9a, 9b, the length of the boom 3 (here, the distance from the GNSS device 9b to the tip of the boom 3), and the three-dimensional coordinate of the GNSS device 9b. Because the bucket 5 is located directly below the tip of the boom 3 when the object to be thrown into water is thrown, the calculated two-dimensional coordinate value corresponds to the two-dimensional coordinate value indicating the position of the bucket 5.

[0031] The machine learning unit 113 generates a learning model by performing machine learning using the operation video data and opening / closing video data acquired by the image data acquisition unit 111 in the machine learning process. Specifically, the machine learning unit 113 performs machine learning using training data in which the operation video data acquired by the image data acquisition unit 111 and stored in the storage unit 114 is used as an explanatory variable and the open / closed state of the bucket 5 identified from the opening / closing video data acquired by the image data acquisition unit 111 (two categories, assuming that the open state is 1 or the closed state is 2) is used as a target variable. The learning model generated by the machine learning unit 113 is stored in the storage unit 114.

[0032] FIG. 6 is a diagram illustrating a video (operation video data) captured by camera 8a installed in the wheelhouse 7. FIG. 6 illustrates the back of operator H, lever R, which is an operating tool for changing the open / closed state of bucket 5, and glass G installed in the front of wheelhouse 7 (the direction from which operator H can see hold 6). Arrow A indicates the operating direction of lever R; when operator H moves lever R in a certain direction, bucket 5 opens, and when operator H moves lever R in another direction, bucket 5 closes. The operation video data is a video showing how operator H operates lever R, which is an operating tool for opening and closing bucket 5. Note that the configuration of the operation video data does not have to be the configuration described above; it is sufficient if at least the operation status of lever R by operator H can be acquired. Furthermore, the number, position, operating direction, and operating method of the operating tool described using lever R as an example are not limited to the above example.

[0033] When identifying the open / closed state of the bucket 5 from the opening / closing video data in machine learning, for example, it is possible to automatically identify whether the bucket 5 is open or closed by image processing using pattern matching or the like, or it is also possible to use the results of a human determining whether the bucket 5 is open or closed by viewing the opening / closing video data. When generating such training data, it is desirable to fix the position of the bucket 5 at a position where the difference in shape before and after opening / closing can be clearly captured by camera 8b.

[0034] After the learning model is stored in the memory unit 114, the operator in the wheelhouse 7 operates the lever R to load the stones. The computer 11 performs loading management processing throughout this operation. The acquisition unit 115 inputs the operation video data acquired by the image data acquisition unit 111 while the bucket 5 is transporting and loading the stones into the learning model stored in the memory unit 114 as explanatory variables, and acquires open / close data indicating the open / closed state of the bucket 5 as a target variable. In addition, the coordinate data acquisition unit 112 sequentially acquires two-dimensional coordinate values ​​indicating the position of the bucket 5.

[0035] The calculation unit 117 calculates the stone throwing position and throwing count of the stone carried out by the bucket 5 at the time of acquiring the opening / closing data, based on the two-dimensional coordinate values ​​of the bucket 5 acquired by the coordinate data acquisition unit 112 and the opening / closing data of the bucket 5 acquired by the acquisition unit 115. Specifically, the calculation unit 117 sets the two-dimensional coordinate value of the bucket 5 when the opening / closing data changes from closed to open as the stone throwing position, and accumulates the number of stone throws in the mesh that includes that position. When the opening / closing data changes from open to closed, the calculation unit 117 resets the processing performed up to that point regarding the stone throwing position and number of stone throws per one gripping of the bucket 5, and starts processing regarding the stone throwing position and number of stone throws per next gripping.

[0036] The stone throwing positions and the number of throwing times are calculated for each mesh divided into a predetermined size in the target area into which the stone is thrown. The mesh dividing unit 116 divides the target area (XY plane) into which the bucket 5 throws the stone into a plurality of meshes M of a predetermined size, as shown in Fig. 7 for example. The size and shape of the meshes M are arbitrary and may be any. The calculation unit 117 calculates the stone throwing positions and the cumulative number of throwing times for each mesh divided by the mesh dividing unit 116.

[0037] The calculation unit 117 further calculates the approximate amount of stone carried out by the bucket 5 based on the capacity of the bucket 5 for each mesh divided into predetermined sizes by the mesh division unit 116. Specifically, the calculation unit 117 preliminarily specifies and stores the amount of stone that the bucket 5 can carry out in one go, and calculates the approximate amount of stone carried out by multiplying the amount of stone that can be carried out in one go by the number of times the bucket 5 carries out the stone at each carry-out position. Alternatively, a LiDAR (Light Detection and Ranging) device (not shown) may be installed at a location where it can measure the entire interior of the hold 6 where the stone is stored, and the LiDAR device may measure the overall shape of the stone in the hold 6 before and after the stone is gripped by the bucket 5. The difference between the measurement results may be used to calculate the volume of the change in the overall shape of the stone. The calculated difference in volume corresponds to the amount of stone carried out from the hold 6 by one gripping of the bucket 5. Then, the calculation unit 117 calculates the distribution of stones to be thrown and the height of the stones to be piled up based on the throwing position, the number of throwing times and the approximate throwing amount for each mesh M calculated each time stones are thrown from the bucket 5.

[0038] The display unit 118 displays various information calculated by the calculation unit 117. By referring to such information, the operator can perform the stone dumping work while understanding the distribution of stone dumped on the water bottom and the dumping pile height. The display unit 118 does not have to be installed together with the computer 11, and may be a portable terminal such as a smartphone or tablet, and may be installed in a location in the wheelhouse 7 that is easy for the operator to see. This allows the operator to understand the amount of dumping and the number of times of dumping on a mesh-by-mesh basis, making it easier to dump the object to be dumped into the water as designed.

[0039] According to the embodiment described above, by simply and accurately detecting the open / closed state of the bucket, it becomes possible to more simply and accurately determine the loading position, number of loading times, and loading amount of objects such as stone.

[0040] [Variations] The present invention is not limited to the above-described embodiment, but may be modified as follows.

[0041] [Variation 1] The object to be thrown into the water is not limited to the stone material exemplified in the embodiment, and may be any object.

[0042] [Variation 2] The display unit 118 may display the time-series position history of the bucket 5 (i.e., the history of the movement trajectory of the bucket 5) acquired by the coordinate data acquisition unit 112, in addition to the various information calculated by the calculation unit 117. The operator can perform stone loading work while referring to the various information calculated by the calculation unit 117 and the position of the bucket 5.

[0043] [Variation 3] In addition to the various information calculated by the calculation unit 117, the display unit 118 may also display the range of the water bottom positions of the stones specified in the construction design stage and the difference between the planned quantity to be added. In this case, the computer 11 pre-stores design shape data for the water bottom to be formed by stone addition and planned quantity data for stones to be added to realize the design shape. The display unit 118 displays the range of the water bottom positions of the stones specified in the construction design stage based on this design shape data, and also displays the planned quantity of stones to be added in the future based on the planned quantity data. The operator can perform stone addition work while referring to the various information calculated by the calculation unit 117 and the differences between the range of the water bottom positions of the stones specified in the construction design stage and the planned quantity to be added.

[0044] The present invention can also be implemented as an object input management method including: an image data acquisition step of acquiring operation video data capturing an operator's operation of a carrying-out device and opening / closing video data capturing an open / closed state of the carrying-out device, the open / closed state of which changes in response to the operator's operation; a coordinate data acquisition step of acquiring two-dimensional coordinate values ​​indicating the position of the carrying-out device; a storage step of storing a learning model generated by machine learning using the operation video data as an explanatory variable and opening / closing data indicating the open / closed state of the carrying-out device identified using the opening / closing video data as an objective variable; an acquisition step of inputting the operation video data acquired while the carrying-out device is carrying out and inserting objects into the stored learning model to acquire the opening / closing data; and a calculation step of calculating the insertion positions and the number of insertions of the objects carried out and inserted by the carrying-out device based on the two-dimensional coordinate values ​​acquired when the opening / closing data was acquired and the acquired opening / closing data. The present invention may also be a program for operating the object input management device (computer 11) described in the embodiment. [Explanation of symbols]

[0045] 1: crane vessel, 2: slewing device, 3: boom, 4: wire, 5: bucket, 6: hold, 7: wheelhouse, 8a, 8b: camera, 9a, 9b: GNSS device, 10: stone input management system, 11: computer, 1101: processor, 1102: memory, 1103: storage, 1104: communication device, 1105: input device, 1106: output device, 111: image data acquisition unit, 112: coordinate data acquisition unit, 113: machine learning unit, 114: memory unit, 115: acquisition unit 116: mesh division unit, 117: calculation unit, 118: display unit, R: lever, H: operator, G: glass, M: mesh

Claims

1. an image data acquisition unit that acquires operation video data that captures an operation of an operator on a carrying-out tool and opening / closing video data that captures an opening / closing state of the carrying-out tool, the opening / closing state of which changes in response to the operation of the operator; a coordinate data acquisition unit that acquires two-dimensional coordinate values ​​indicating the position of the carrying-out tool; a storage unit that stores a learning model generated by machine learning using the operation video data as an explanatory variable and opening / closing data indicating the open / closed state of the carrying-out tool identified using the opening / closing video data as a target variable; and an acquisition unit that inputs the operation video data acquired when the carrying-out tool carries out and puts in the object into the stored learning model, and acquires the opening and closing data; a calculation unit that calculates the insertion position and the number of insertions of the object carried out and inserted by the carrying-out tool based on the two-dimensional coordinate value acquired when the opening / closing data is acquired and the acquired opening / closing data; An object input management device comprising:

2. a mesh dividing unit that divides a target area into which the object is to be thrown by the carrying-out tool into a plurality of meshes of a predetermined size; The calculation unit calculates the throwing position and the number of throwing times of the object for each mesh divided by the mesh dividing unit. The object input management device according to claim 1 .

3. The carrying implement is a bucket of a construction machine. The object input management device according to claim 1 .

4. The calculation unit further calculates, for each mesh divided by the mesh dividing unit, an approximate amount of objects discharged by the discharge device based on the capacity of the discharge device. The object input management device according to claim 2 .

5. The calculation unit calculates a throw-in distribution and a throw-in pile height of the object based on the throw-in position, the throw-in count, and the approximate throw-in amount of the object for each mesh, which are calculated each time the object is thrown from the carrying-out device. The object input management device according to claim 4.

6. On the computer, an image data acquisition step of acquiring operation video data capturing an operation of an operator on a carrying-out tool and opening / closing video data capturing an opening / closing state of the carrying-out tool, the opening / closing state of which changes in response to the operation of the operator; a coordinate data acquisition step of acquiring two-dimensional coordinate values ​​indicating the position of the carrying-out tool; a storage step of storing a learning model generated by machine learning using the operation video data as an explanatory variable and opening / closing data indicating the open / closed state of the carrying-out tool identified using the opening / closing video data as a target variable; an acquisition step of inputting the operation video data acquired when the carrying-out tool is carrying out and putting in the object into the stored learning model, and acquiring the opening / closing data; a calculation step of calculating the insertion position and the number of insertions of the object carried out and inserted by the carrying-out tool based on the two-dimensional coordinate value acquired when the opening / closing data is acquired and the acquired opening / closing data; A program to execute.

7. a dividing step of dividing a target area into which the object is to be thrown by the carrying-out tool into a plurality of meshes of a predetermined size; In the calculation step, the throwing position and the number of throwing times of the object are calculated for each mesh divided by the division step. The program according to claim 6.

8. In the calculation step, an approximate amount of objects carried out by the carrying-out device is calculated for each mesh divided by the dividing step using the capacity of the carrying-out device. The program according to claim 7.

9. In the calculation step, a distribution of the objects to be thrown and a piled-up height of the objects are calculated based on the throwing positions, the number of throwing times, and the approximate throwing amounts of the objects for each mesh, which are calculated each time the objects are thrown from the carrying-out device. The program according to claim 8.

10. 6. An object input management method for managing the input positions and the number of times an object is input, using the object input management device according to claim 1.

Images