Crane boom automation device, crane boom control method and program

Abstract

To provide a crane boom automation device, a crane boom automation method, and a program capable of moving a boom head above an object to be lifted by a boom through simple calculation.SOLUTION: A crane boom automation device 1 comprises: imagers 10R and 10L; an identification unit 25 that identifies an object to be lifted by a crane in an omnidirectional image captured by the imagers 10R and 10L and calculates a position in the image; and a control unit 26 that calculates the position of a horizontal line indicating a horizontal plane in the omnidirectional image and the position of a vertical line indicating a vertical plane vertical to a horizontal direction in the omnidirectional image on the basis of the position of a boom head with respect to a support and the positions of the imagers 10R and 10L with respect to the boom head, raises a boom when the position of the object in the image calculated by the identification unit 25 is higher than the position of the horizontal line, and turns the boom when the position of the object in the image calculated by the identification unit 25 is lower than the position of the horizontal line and higher than the position of the vertical line.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a crane boom automation device, a crane boom control method, and a program. [Background technology]

[0002] In a crane, a camera is provided on the boom head to prevent load swing, and images captured by the camera are sometimes used for operations such as raising and lowering the boom and rotating it.

[0003] Controlling a machine using images captured by such a camera is called visual servoing, and development of visual servoing is underway in a variety of fields.

[0004] For example, Patent Document 1 discloses a robot system that includes a surgical robot equipped with an endoscope, a camera connected to the endoscope, and a controller that controls the surgical robot using images captured by the camera. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2011 / 083374 Summary of the Invention [Problem to be solved by the invention]

[0006] In a crane, in order to move the boom head above the object to be lifted and lift it, the rotating body is rotated and the boom is raised and lowered / extended. In order for the crane itself to autonomously perform the operation to move the boom head above the object to be lifted, it is conceivable to apply the technology of the robot system described in Patent Document 1 to the crane.

[0007] However, the robot system described in Patent Document 1 extracts specific features from each video frame to determine a target point and then maps the trajectory vector of the target point. This requires complex calculations. As a result, it is not easy to apply the technology of the robot system described in Patent Document 1 to move the boom head of the crane above the object to be lifted.

[0008] The present invention has been made to solve the above-mentioned problems, and aims to provide a crane boom automation device, a crane boom control method, and a program that can move the boom head above an object to be lifted by the crane with simple calculations. [Means for solving the problem]

[0009] In order to achieve the above object, a crane boom automation device according to a first aspect of the present invention comprises: A boom automation device for a crane including a rotating body that can rotate relative to a support body, and a boom that is provided on the rotating body and can be raised and lowered relative to the rotating body, an omnidirectional camera unit provided on a boom head of the boom, which captures an omnidirectional image with a field of view in all directions including the front-rear direction, the left-right direction, and the up-down direction of the boom head; an identification unit that identifies an object to be lifted by the crane in the omnidirectional image captured by the omnidirectional camera unit and determines the position of the identified object in the image; a control unit that determines the position of a horizontal line indicating a horizontal plane in the omnidirectional image captured by the omnidirectional camera unit and the position of a vertical line indicating a vertical plane perpendicular to the left-right direction in the omnidirectional image based on the position of the boom head relative to the support body and the position of the omnidirectional camera unit relative to the boom head, and raises the boom when the position of the object in the image determined by the identification unit is above the position of the horizontal line, and rotates the boom when the position of the object in the image determined by the identification unit is below the position of the horizontal line and above the position of the vertical line; The present invention is characterized by comprising:

[0010] The omnidirectional camera unit a right hemispherical camera provided on a right side surface of the boom head and configured to capture an image to the right of the boom head; a left hemispherical camera provided on a left side surface of the boom head and configured to capture an image to the left of the boom head; Equipped with The omnidirectional image may be generated from a right image portion captured by the right hemispherical camera and a left image portion captured by the left hemispherical camera.

[0011] Furthermore, the control unit may rotate the boom to the right when the position of the object in the image determined by the identification unit is below the horizontal line and above the vertical line and the object is located in a right-side image portion captured by the right-side hemispherical camera, and may rotate the boom to the left when the position of the object in the image determined by the identification unit is below the horizontal line and above the vertical line and the object is located in a left-side image portion captured by the left-side hemispherical camera.

[0012] The boom is extendable and retractable, The control unit may determine the position of a left-right line indicating a left-right plane perpendicular to the front-to-back direction in the omnidirectional image captured by the omnidirectional camera unit based on the position of the boom head relative to the support body and the position of the omnidirectional camera unit relative to the boom head, and may extend the boom when the position of the object in the image determined by the identification unit is in front of the position of the left-right line, and may retract the boom when the position of the object in the image determined by the identification unit is behind the position of the left-right line.

[0013] A crane boom control method according to a second aspect of the present invention comprises: A boom control method for a crane including a rotating body that can rotate relative to a support body, and a boom that is provided on the rotating body and can be raised and lowered relative to the rotating body, comprising: an omnidirectional camera unit provided on a boom head of the boom, which captures an omnidirectional image with a field of view in all directions including the front-to-back, left-to-right, and up-to-down directions of the boom head, identifies an object to be lifted by the crane in the omnidirectional image captured by the omnidirectional camera unit, and determines the position of the identified object in the image; determining the position of a horizontal line indicating a horizontal plane in the omnidirectional image captured by the omnidirectional camera unit and the position of a vertical line indicating a vertical plane perpendicular to the left-right direction in the omnidirectional image based on the position of the boom head relative to the support body and the position of the omnidirectional camera unit relative to the boom head, and raising the boom when the position of the object in the image determined in the step of determining the position of the object in the image is above the position of the horizontal line, and rotating the boom when the position of the object in the image determined in the step of determining the position of the object in the image is below the position of the horizontal line and above the position of the vertical line; The present invention is characterized by comprising:

[0014] A program according to a third aspect of the present invention comprises: A program for controlling a boom of a crane including a rotating body that can rotate relative to a support body and a boom that is provided on the rotating body and can be raised and lowered relative to the rotating body, On the computer, an omnidirectional camera unit provided on a boom head of the boom, which captures an omnidirectional image with a field of view in all directions including the front-to-back, left-to-right, and up-to-down directions of the boom head, identifies an object to be lifted by the crane in the omnidirectional image captured by the omnidirectional camera unit, and determines the position of the identified object in the image; determining the position of a horizontal line indicating a horizontal plane in the omnidirectional image captured by the omnidirectional camera unit and the position of a vertical line indicating a vertical plane perpendicular to the left-right direction in the omnidirectional image based on the position of the boom head relative to the support body and the position of the omnidirectional camera unit relative to the boom head, and raising the boom when the position of the object in the image determined in the step of determining the position of the object in the image is above the position of the horizontal line, and rotating the boom when the position of the object in the image determined in the step of determining the position of the object in the image is below the position of the horizontal line and above the position of the vertical line; The program is characterized by being a program for executing the above. [Effects of the Invention]

[0015] According to the configuration of the present invention, the control unit determines the position of a horizontal line indicating a horizontal plane in an omnidirectional image captured by the omnidirectional camera unit and the position of a vertical line indicating a vertical plane perpendicular to the left-right directions in the omnidirectional image based on the position of the boom head relative to the support body and the position of the omnidirectional camera unit relative to the boom head, and raises the boom when the position of the object in the image determined by the identification unit is above the horizontal line, and rotates the boom when the position of the object in the image determined by the identification unit is below the horizontal line and above the vertical line. As a result, the boom automation device can move the boom head above the object to be lifted by the crane with simple calculations. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a side view of a loaded truck crane equipped with a boom automation device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a hardware configuration diagram of the boom automation device. [Figure 3] FIG. 2 is a front view of a boom head on which an imaging device provided in a boom automation device is installed. [Figure 4] FIG. 2 is a block diagram of a boom automation device. [Figure 5]1A and 1B are conceptual diagrams showing an example of an image captured by an imaging device installed on the left and right sides of a boom head, respectively. [Figure 6] 10 is a flowchart of a boom automation process performed by the boom automation device. [Figure 7] (A) A conceptual diagram showing an example of a case where left and right lines calculated by a modified example of the control unit are displayed on an image captured by an imaging device installed on the left side of the boom head. (B) A conceptual diagram showing an example of a case where left and right lines calculated by a modified example of the control unit are displayed on an image captured by an imaging device installed on the right side of the boom head. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, a crane boom automation device, a crane boom control method, and a program according to an embodiment of the present invention will be described in detail with reference to the drawings. In the drawings, the same or equivalent parts are designated by the same reference numerals.

[0018] The crane boom automation device according to the embodiment is a device that controls each part of the crane and automatically moves the boom head in accordance with the position of the object to be lifted in an image captured by a hemispherical camera in order to facilitate crane operation. The configuration of this boom automation device will be described below using an example in which it is installed on a loaded truck crane (also called a cargo crane). First, the configuration of a loaded truck crane equipped with the boom automation device will be described with reference to FIG. 1.

[0019] FIG. 1 is a side view of a loaded truck crane 100 equipped with a crane boom automation system 1 according to an embodiment of the present invention.

[0020] As shown in FIG. 1, the loaded truck crane 100 comprises a vehicle 110 having a traveling function, a rotating body 120 provided on the vehicle 110, and a boom 130 provided on the rotating body 120.

[0021] The vehicle 110 is a truck that travels on a road. The vehicle 110 has a cab 111 for a driver to sit in and a loading platform 112 for loading luggage. A crane rotating body 120 is provided between the cab 111 and the loading platform 112 to load luggage onto the loading platform 112.

[0022] The revolving unit 120 is attached to a frame 121 fixed to the vehicle 110. The revolving unit 120 is driven by a revolving motor 122, and is capable of revolving about the vertical direction VD on the frame 121. Meanwhile, the revolving unit 120 extends vertically like a pillar, and a boom 130 is provided at the upper end thereof.

[0023] The frame 121 is provided with outriggers 160 that come into contact with the ground and stabilize the vehicle 110. The frame 121 supports the rotating body 120. For this reason, the frame 121 is also called a support body. The frame 121 is a member that corresponds to the support body in the claims.

[0024] The lower end of the boom 130 is attached to the revolving unit 120 by a foot pin (not shown). The middle portion of the boom 130, which is located further upward than the lower end, is supported by a hoisting cylinder 131 provided on the revolving unit 120. As a result, the boom 130 can be raised and lowered by the extension and contraction of the hoisting cylinder 131.

[0025] The boom 130 is formed in a telescopic manner and can be extended and retracted by a telescopic cylinder 132. A boom head 133 is provided at the tip of the boom 130.

[0026] Although not shown, a sheave is built into the boom head 133. A wire rope 140 is wound around the sheave. As a result, one end of the wire rope 140 hangs down from the boom head 133. A hook 150 is mechanically connected to one end of the hanging wire rope 140.

[0027] As described above, the boom 130 is capable of being raised and lowered, and is also extendable and rotatable as a result of being mounted on the rotating unit 120. As a result, the boom head 133 can move the hook 150 to a desired position by raising and lowering the boom 130 and rotating the rotating unit 120.

[0028] The other end of the wire rope 140 is wound around a winch drum 141 provided on the rotating body 120. As a result, the wire rope 140 is wound up or down by the rotation of the winch drum 141. As described above, the hook 150 is connected to one end of the wire rope 140, and therefore rises or falls by the winding up or down of the wire rope 140. As described above, the hook 150 is moved to a desired position by the raising and lowering of the boom 130 and the rotation of the rotating body 120, and further rises or falls to a desired height by the rotation of the winch drum 141.

[0029] In the loaded truck crane 100, an operator operates the boom 130, the rotating bed 120, and the winch drum 141 to move the hook 150 to a desired position. However, this operation requires skill and is not particularly easy. Therefore, to facilitate the operation of moving the hook 150 to a desired position, the loaded truck crane 100 is equipped with a boom automation system 1 that automatically moves the hook 150 to the position of the object to be lifted, i.e., the position of the suspended load. Next, the configuration of the boom automation system 1 will be described with reference to Figures 2 to 4.

[0030] Fig. 2 is a hardware configuration diagram of the boom automation device 1. Fig. 3 is a front view of a boom head 133 on which imaging devices 10R and 10L are installed, which are provided in the boom automation device 1. Fig. 4 is a block diagram of the boom automation device 1.

[0031] 2 and 4, in order to facilitate understanding, in addition to the configuration of the boom automation device 1, the configuration of a portion of the loaded truck crane 100 connected to the boom automation device 1 is also shown.

[0032] As shown in FIG. 2, the boom automation device 1 includes imaging devices 10R and 10L, an I / O port (Input / Output Port) 20, a CPU (Central Processing Unit) 21, a ROM (Read-Only Memory) 22, and a RAM (Random Access Memory) 23.

[0033] The imaging devices 10R and 10L are hemispherical cameras that can capture a hemispherical field of view of 360° around the direction in which the lens is pointed and 180° on the side in which the lens is pointed. As shown in FIG. 3, the imaging devices 10R and 10L are installed on the right and left sides of the boom head 133, respectively. Furthermore, the lenses of the imaging devices 10R and 10L are directed obliquely downward, rather than in the horizontal direction H or the vertical direction V. Images I captured by the imaging devices 10R and 10L are shown in FIGS. 5(A) and (B). R , I L A conceptual diagram is shown below.

[0034] FIG. 5A shows an image I captured by the imaging device 10L installed on the left side of the boom head 133. L 5B is a conceptual diagram showing an example of an image I captured by the imaging device 10R installed on the right side of the boom head 133. R FIG. 1 is a conceptual diagram showing an example of the above.

[0035] In order to facilitate understanding, in FIGS. 5(A) and (B), when it is assumed that a horizontal plane is imaged, the horizontal plane is represented by a horizontal line H L , H R In addition, if we assume that a vertical plane perpendicular to the left and right direction is captured, the vertical plane is represented as a vertical line V L , V R It displays:

[0036] As shown in FIG. 5(A), imaging device 10R captures an image of the right side of boom head 133 and point P0 vertically from the center of boom head 133 in a top view, i.e., point P0 directly below boom head 133. Furthermore, imaging device 10L captures an image of the left side of boom head 133 and point P0 directly below boom head 133, as shown in FIG. 5(B). In this way, imaging devices 10R and 10L each orient their lenses diagonally downward, thereby enabling imaging of point P0 directly below in addition to the right and left fields of view. As a result, imaging device 10L can capture an image of the suspended load L0 shown in FIG. 3 when the load is located to the side of boom head 133, as well as when the load is located directly below boom head 133, regardless of the thickness of the boom head 133.

[0037] Both imaging devices 10R and 10L capture omnidirectional images with a field of view in all directions, including the front-back, left-right, and up-down directions. Therefore, both imaging devices 10R and 10L are collectively referred to as an omnidirectional camera or celestial sphere camera. Furthermore, both imaging devices 10R and 10L are devices that correspond to the omnidirectional camera unit defined in the claims.

[0038] Returning to FIG. 2, the I / O port 20 is connected to a bus line 24 provided for connecting various devices. Also connected to the bus line 24 are a CPU 21, a ROM 22, and a RAM 23 that constitute a computer. In the boom automation device 1, the CPU 21 reads various programs stored in the ROM 22 into the RAM 23 and executes them to perform various processes. The I / O port 20 is electrically connected to the imaging devices 10R and 10L described above for performing image processing, which is one of the various processes. As a result, image data captured by the imaging devices 10R and 10L can be transmitted to the CPU 21, etc.

[0039] Furthermore, components in each part of the loaded truck crane 100 are connected to the I / O port 20 so that the CPU 21 can control each part of the loaded truck crane 100. In detail, the I / O port 20 is connected to a swing angle detection sensor 123, a boom angle detection sensor 134, and a boom length detection sensor 135.

[0040] Here, the swing angle detection sensor 123 is a sensor that detects the angle at which the swing body 120 is swung by the swing motor 122. The boom angle detection sensor 134 is a sensor that detects the angle at which the boom 130 is raised or lowered by the extension or contraction of the hoisting cylinder 131. The boom length detection sensor 135 is a sensor that detects the length of the boom 130 when the boom 130 is extended or lowered by the extension or contraction of the telescopic cylinder 132.

[0041] The rotation angle detection sensor 123, the boom angle detection sensor 134, and the boom length detection sensor 135 transmit the detected rotation angle, the boom 130 elevation angle, and the length of the boom 130 to the CPU 21 via the I / O port 20.

[0042] The I / O port 20 is also electrically connected to a display device 125 on a control panel (not shown) of the loading truck crane 100. As a result, various data are input to the display device 125 via the I / O port 20, and the various data are displayed. For example, the display device 125 displays the images I captured by the image capture devices 10R and 10L. R , I L Or, the image I that the CPU 21 has processed R , I L is displayed.

[0043] Furthermore, the I / O port 20 is electrically connected to electromagnetic hydraulic control valves (not shown) provided on the swing motor 122, the hoisting cylinder 131, and the telescopic cylinder 132 in order to operate each part of the loaded truck crane 100. The CPU 21 described above receives the images I captured by the imaging devices 10R and 10L. R , IL and data detected by the swing angle detection sensor 123, boom angle detection sensor 134, and boom length detection sensor 135, the CPU 21 calculates various data for controlling the loaded truck crane 100. The swing motor 122, derrick cylinder 131, and telescopic cylinder 132 receive the various data calculated by the CPU 21 via the I / O port 20. The swing motor 122, derrick cylinder 131, and telescopic cylinder 132 operate based on the various data by adjusting the opening of their respective electromagnetic hydraulic control valves.

[0044] Next, the processing performed by the boom automation device 1 to control the swing motor 122, derrick cylinder 131, and telescopic cylinder 132 will be described in detail. As shown in Fig. 2, a storage device 30 is connected to the I / O port 20. The boom automation device 1 performs boom automation processing to control the swing motor 122, derrick cylinder 131, and telescopic cylinder 132 by having the CPU 21 read out the boom automation program 31 stored in the storage device 30 into the RAM 23 and execute it. To perform this boom automation processing, the boom automation device 1 comprises various blocks configured as software, as shown in Fig. 4. In more detail, the boom automation device 1 comprises an identification unit 25 and a control unit 26.

[0045] The recognition unit 25 recognizes the image I captured by the imaging devices 10R and 10L. R , I L The lifted load portion L1 (see FIG. 5(A)) is identified from the image data. In detail, the imaging devices 10R and 10L capture images of the area around the boom head 133 at regular intervals, for example, every 1 / 30 seconds, while the boom automation device 1 is running. Then, the imaging devices 10R and 10L transmit the captured image data to the recognition unit 25 every time an image is captured. The recognition unit 25 receives the captured image data and identifies the image I obtained from the captured image data. R , I L Feature extraction is performed on the

[0046] On the other hand, the storage device 30 stores the image I R , IL The identification dictionary 32 for identifying the suspended load portion L1 included in the image I is stored in the storage device 30. After extracting the features, the identification unit 25 reads the identification dictionary 32 from the storage device 30 and identifies the image I from the read identification dictionary 32 and the extracted feature data. R , I L The recognition unit 25 then identifies the suspended load portion L1 contained within the image data. The recognition unit 25 then identifies a representative point, for example, the center of the suspended load portion L1, from the identified suspended load portion L1, and sets the coordinates of the identified center in the image as the coordinates of the suspended load portion L1. In this way, the recognition unit 25 obtains the position of the suspended load portion L1 in the image from the imaging data of the imaging devices 10R and 10L. The recognition unit 25 transmits the obtained position of the suspended load portion L1 in the image to the control unit 26.

[0047] The control unit 26 operates the revolving structure 120 and the boom 130 of the loaded truck crane 100 based on the position of the above-mentioned suspended load portion L1 in the image. In detail, when the control unit 26 receives the position of the suspended load portion L1 in the image from the identification unit 25, the control unit 26 causes the revolving angle detection sensor 123, the boom angle detection sensor 134, and the boom length detection sensor 135 to transmit data on the revolving angle of the revolving structure 120, the hoisting angle of the boom 130, and the length of the boom 130, and obtains each of these data.

[0048] The rotation angle of the rotating body 120, the angle at which the boom 130 is raised and lowered, and the length of the boom 130 are physical quantities that correspond to the position of the boom head relative to the support body as referred to in the claims.

[0049] Meanwhile, the storage device 30 stores position information data 33 including data on the position of each of the imaging devices 10R and 10L relative to the boom head 133, for example, data on the position relative to the center of gravity of the boom head 133 and the orientation of the lens. The control unit 26 reads out the position information data 33 from the storage device 30 and, based on each data item including (1) the read-out position information data 33, (2) the swing angle obtained from the swing angle detection sensor 123, (3) the boom 130 hoisting angle obtained from the boom angle detection sensor 134, and (4) the length of the boom 130 obtained from the boom length detection sensor 135, calculates the image I captured by the imaging devices 10R and 10L. R , IL In Fig. 5(A) and (B), the horizontal line H L , H R and the vertical line V L , V R Find the position of .

[0050] The control unit 26 determines the horizontal line H L , H R Position of and vertical line V L , V R 4 in accordance with the position of the suspended load portion L1 in the image obtained from the recognition unit 25 relative to the position of the horizontal line H. In detail, the control unit 26 controls the swing motor 122 and the elevation cylinder 131 shown in FIG. L , H R If the load L0 is located above this position, the controller 26 determines that the load L0 is located higher than the boom head 133, and controls the hoisting cylinder 131 to raise the boom 130. As a result, the controller 26 orients the direction in which the boom 130 tilts according to the hoisting angle toward the load L0.

[0051] Furthermore, the control unit 26 determines whether the position of the suspended load portion L1 in the image obtained from the recognition unit 25 is on the horizontal line H L , H R Below the position of and vertical line V L , V R If the load L0 is above the position, the control unit 26 determines that the load L0 is not directly below the boom head 133. Then, the control unit 26 determines that the position of the load portion L1 in the image is within the range of the image I captured by the right imaging device 10R. R In response to this, the control unit 26 determines whether the position of the suspended load portion L1 in the image is the same as the position of the suspended load portion L1 in the image I captured by the left imaging device 10L. L If the position in the image is

[0053] , the control unit 26 controls the rotation motor 122 to rotate the rotating body 120 to the left and turn the boom 130 to the left. By this operation, the control unit 26 turns the boom 130 toward the load L0.

[0052] In this way, the boom automation device 1 uses the images I captured by the imaging devices 10R and 10L. R , I L The position of the suspended load part L1 in the image is the horizontal line H L , H R or vertical line V L , V R Depending on the position of the load L0 relative to the load L0, the boom automation device 1 controls the derrick cylinder 131 or the swing motor 122 to point the boom 130 toward the load L0. In this way, the boom automation device 1 brings the boom head 133 to approximately the same height as the load L0 and automates the operation of pointing the boom 130 toward the load L0.

[0053] Next, the boom automation process performed by the boom automation device 1 will be described in detail with reference to Figure 6. In the following description, it is assumed that the boom automation device 1 is activated by pressing a start button on a control panel provided on the loaded truck crane 100. It is also assumed that a small lighting device that emits light of a specific wavelength, for example, red light, called a marker is attached to the suspended load L0 to make it easier to identify the suspended load L0. It is also assumed that an identification dictionary 32 for identifying the small lighting device is stored in the storage device 30.

[0054] FIG. 6 is a flowchart of the boom automation process performed by the boom automation device 1.

[0055] First, as a preparation step, the operator operating the loaded truck crane 100 extends the outriggers 160 and places their tips on the ground. This stabilizes the vehicle 110. Next, the operator attaches the small lighting device described above to the suspended load L0 and turns on the small lighting device. After that, to prevent the boom 130 from operating and interfering with buildings or the like during the boom automation process, the operator confirms that there are no objects, such as buildings, around the loaded truck crane 100 that the boom 130 may interfere with. Then, the operator presses the start button described above. As a result, the boom automation device 1 is started.

[0056] When the boom automation device 1 is started up, the CPU 21 executes the boom automation program 31 stored in the storage device 30, and as a result, the flow of the boom automation process shown in FIG. 6 is started.

[0057] In the boom automation process, first, the recognition unit 25 provided in the boom automation device 1 acquires the image I from the imaging devices 10R and 10L on the right and left sides of the boom head 133. R , I L In detail, the image recognition unit 25 acquires the image I immediately before the image capture devices 10R and 10L periodically capture images (step S1). R , I L Data is acquired from the imaging devices 10R and 10L.

[0058] Next, the recognition unit 25 recognizes the acquired image I R , I L The recognition unit 25 extracts features from the data and performs a process of identifying the suspended load portion L1 (step S2). R , I L The high-brightness portion is extracted from the brightness distribution of the suspended load L0. The storage device 30 pre-stores an identification dictionary 32 containing data such as the brightness, color, and shape of the marker portion, for identifying the image portion of the small lighting device attached to the suspended load L0 (hereinafter referred to as the marker portion). The identification unit 25 reads out the identification dictionary 32 and uses it to determine whether the extracted high-brightness portion corresponds to the marker portion. If there is a portion corresponding to the marker portion, the identification unit 25 determines that the suspended load portion L1 has been identified. Furthermore, the image coordinates of a representative point of that portion, for example, the image coordinates of the center point, are set as the image coordinates of the suspended load portion L1. The identification unit 25 sends the result data of the identification process to the control unit 26.

[0059] When the control unit 26 receives the result data of the identification process from the identification unit 25, it determines whether the suspended load part L1 has been identified (step S3). In detail, the control unit 26 determines that the suspended load part L1 has been identified if the image coordinates of the suspended load part L1 are included in the result data of the identification process from the identification unit 25. When the control unit 26 determines that the suspended load part L1 has been identified (Yes in step S3), it proceeds to step S4.

[0060] On the other hand, if the image coordinates of the suspended load portion L1 are not included in the identification processing result data from the identification unit 25, the control unit 26 determines that the suspended load portion L1 has not been identified. If the control unit 26 determines that the suspended load portion L1 has not been identified (No in step S3), it considers that there is no suspended load L0 around the loading truck crane 100, and as a result, the imaging devices 10R, 10L have not been able to capture an image of the suspended load L0. The control unit 26 determines that there is no suspended load L0 and terminates the boom automation processing.

[0061] Before terminating the boom automation process, the control unit 26 may send a signal to the above-mentioned display device 125 indicating that the suspended load portion L1 was not identified, and cause the display device 125 to display a message indicating that the suspended load L0 was not found.

[0062] When the control unit 26 determines that the suspended load portion L1 has been identified and proceeds to step S4, the control unit 26 R , I L Horizontal line H L , H R , vertical line V L , V R The position of the object is calculated (step S4).

[0063] Here, the horizontal line H L , H R is an image I representing a horizontal plane when the imaging devices 10R and 10L capture the image of the horizontal plane. R , I L (See Figure 5(A) and (B)). Vertical line V L , V RThe image I represents a vertical plane perpendicular to the left-right direction when the image capturing devices 10R and 10L capture the vertical plane. R , I L This refers to the line inside (see the same diagram).

[0064] To explain step S4 in more detail, the storage device 30 stores position information data 33 including data on the position of each of the imaging devices 10R and 10L relative to the center of gravity of the boom head 133 and data on the orientation of the lens. The control unit 26 reads out the position information data 33. The control unit 26 also acquires sensor data on the rotation angle of the rotating body 120, the hoisting angle of the boom 130, and the length of the boom 130 from the rotation angle detection sensor 123, the boom angle detection sensor 134, and the boom length detection sensor 135 provided in the loaded truck crane 100. The control unit 26 then generates the image I from the read position information data 33 and the acquired sensor data. R , I L The control unit 26 calculates the direction of the image captured by each pixel part of each coordinate in the horizontal line H defined above, based on the calculated correspondence between each pixel part and the direction. L , H R and vertical line V L , V R Image I R , I L Calculate the inner position.

[0065] Next, the control unit 26 determines whether the suspended load portion L1 identified in step S2 is aligned with the vertical line V L , V R In detail, the control unit 26 determines whether the coordinates of the suspended load portion L1 in the image are on the vertical line V L , V R corresponds to the coordinates in the image through which the vertical line V passes. L , V R By determining whether the coordinates in the image are only a small distance away from the vertical line V, the suspended load part L1 is L , V RHere, the minute distance is determined as the coordinates of the suspended load portion L1 in the image that are substantially on the vertical line V L , V R This refers to the distance through which the image coordinates can be treated as passing.

[0066] The control unit 26 determines whether the suspended load portion L1 is aligned with the vertical line V L , V R If it is determined that the boom 130 is located directly below the boom head 133 or on an extension of the boom 130 (Yes in step S5), it is determined that the boom 130 is already pointed toward the load L0. In other words, it is determined that there is no need to perform operations to rotate the revolving body 120 or to raise or lower the boom 130. As a result, the control unit 26 ends the boom automation process.

[0067] On the other hand, the control unit 26 determines whether the suspended load portion L1 is aligned with the vertical line V L , V R If it is determined that the suspended load portion L1 identified in step S2 is not located on the horizontal line H L , H R It is determined whether the image is above the image I (step S6). R , I L In step S6, the control unit 26 determines whether the coordinates of the suspended load portion L1 in the image are on the horizontal line H L , H R The determination is made by checking whether the point is above the coordinates in the image through which the point passes.

[0068] The control unit 26 determines whether the suspended load portion L1 is parallel to the horizontal line H L , H R If it is determined that the load L0 is higher than the boom head 133 (Yes in step S6), it is determined that the load L0 is higher than the boom head 133 and that the boom 130 needs to be raised. As a result, the control unit 26 extends the hoisting cylinder 131 (step S7). The length by which the hoisting cylinder 131 is extended is a predetermined fixed length. As a result, the control unit 26 raises the boom 130 by only a small angle.

[0069] After raising the boom 130 by a small angle, the control unit 26 returns to step S1 to check whether the orientation of the boom 130 needs to be readjusted.

[0070] On the other hand, the control unit 26 determines whether the suspended load portion L1 is on the horizontal line H L , H R That is, the suspended load portion L1 is not above the horizontal line H L , H R If it is determined that the load L0 is at a height equal to or lower than the boom head 133 (No in step S6), it is determined that there is no need to raise the boom 130. Then, in step S5, the control unit 26 determines that the load L1 has already been raised along the vertical line V L , V R Since it is determined that the load L0 is not located in the line of extension of the boom 130, it is determined that the load L0 has deviated from the line of extension of the boom 130 and that it is necessary to rotate the rotating body 120. As a result, the control unit 26 rotates the rotation motor 122 (step S8).

[0071] The rotation direction is the same as that of the image I captured by the left imaging device 10L. L Horizontal line H at L and vertical line V L When the suspended load portion L1 is located between the left and right imaging devices 10R and 10R, the direction is the direction in which the rotating body 120 rotates to the left. R Horizontal line H at R and vertical line V R When the suspended load portion L1 is located between these two points, the direction is to rotate the rotating body 120 to the right. The amount of rotation is a predetermined fixed amount. As a result, the control unit 26 rotates the rotating body 120 by a fixed angle.

[0072] After rotating the rotating body 120 by a certain angle, the control unit 26 returns to step S1 to check whether the rotation position of the rotating body 120 needs to be readjusted.

[0073] In step S5, the control unit 26 determines whether the suspended load portion L1 is aligned with the vertical line VL , V R Steps S1 to S8 are repeated until it is determined that the revolving body 120 is positioned at the load L0 (Yes in step S5) and it is no longer necessary to perform the operations of revolving the revolving body 120 and raising and lowering the boom 130. As a result, the control unit 26 directs the boom 130 toward the load L0.

[0074] It is preferable that the control unit 26 presets a limit value for raising and lowering the boom 130, and forcibly terminates the boom automation process when the limit value is reached. Alternatively, the control unit 26 may forcibly terminate the boom automation process in accordance with the output of the overload prevention device.

[0075] As described above, the boom automation device 1 uses the images I captured by the imaging devices 10R and 10L, which are hemispherical cameras. R , I L Within this, the suspended load part L1 is on the horizontal line H L , H R and vertical line V L , V R The boom automation system 1 controls the rotation of the rotating body 120 and the elevation of the boom 130 depending on the relative position of the horizontal line H of the lifted load part L1. L , H R and vertical line V L , V R The calculation is simple because the revolving body 120 and the boom 130 are controlled based on the relative position of the load L0. Furthermore, with this simple calculation, the boom head 133 can be moved to above the load L0.

[0076] Furthermore, the boom automation device 1 can be installed on the loading truck crane 100 simply by attaching the imaging devices 10R and 10L, which are semi-spherical cameras, to the boom head 133. As a result, the cost required for installation is low.

[0077] Although the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments. For example, in the above embodiments, the imaging devices 10R and 10L are hemispherical cameras, but the present invention is not limited to this. In the present invention, the imaging devices 10R and 10L may be omnidirectional camera units that capture omnidirectional images with a field of view in all directions, including the front-to-back, left-to-right, and up-to-down directions of the boom head 133. Therefore, in the present invention, the imaging devices 10R and 10L may be cameras that satisfy this condition. For example, the imaging devices 10R and 10L may be a single omnidirectional camera. Alternatively, the imaging devices 10R and 10L may be a camera system composed of three or more cameras.

[0078] In the above embodiment, the identification unit 25 identifies a small lighting device called a marker attached to the suspended load L0. However, the present invention is not limited to this. In the present invention, the identification unit 25 may identify an object to be lifted by the loaded truck crane 100 in an omnidirectional image captured by the omnidirectional camera unit. The identification unit 25 may also determine the position of the identified object within the image. For example, if the suspended load L0 is a specific object such as steel, the identification unit 25 may directly identify the specific object from the image. Furthermore, if a specific plate is attached to the suspended load L0 or is painted a specific color, the identification unit 25 may identify the specific plate or the portion painted a specific color.

[0079] In the above embodiment, the control unit 26 R , I L Horizontal line H of the hanging load part L1 L , H R and vertical line V L , V R The control unit 26 controls the rotation of the rotating body 120 and the elevation of the boom 130 based on the relative position of the boom 130. However, the present invention is not limited to this. In the present invention, the control unit 26 may further control the extension and contraction of the boom 130.

[0080] FIG. 7A shows an image I captured by the imaging device 10L installed on the left side of the boom head 133. L The left and right lines LR calculated by the modified example of the control unit 26 areL 7B is a conceptual diagram showing an example of a case where an image I captured by the imaging device 10R installed on the right side of the boom head 133 is displayed. R The left and right lines LR calculated by the modified example of the control unit 26 are R FIG. 10 is a conceptual diagram showing an example of a case where the following is displayed.

[0081] The control unit 26 generates the image I shown in FIGS. R , I L The left and right line LR indicates the left and right plane perpendicular to the front and rear direction within the vehicle. R , L.R. L Then, the control unit 26 calculates the positions of the left and right lines LR in the image. R , L.R. L In the case where the load portion L1 described in the embodiment is located forward, the control unit 26 may extend the boom 130. R , L.R. L If the lifting load portion L1 is at the rear, it is better to retract the boom 130. R , I L Although we know the direction of each pixel in the image, the distance is R , I L Therefore, it is advisable to determine in advance the length to which the boom 130 is extended and the length to which the boom 130 is retracted. Then, in the same manner as in the flow described in the embodiment, after the boom 130 is extended or retracted, the image I is displayed again. R , I L and then take that image I R , I L The left and right lines LR of the hanging load part L1 R , L.R. L It is advisable to determine the position of the boom 130 relative to the target position to see if further length adjustment of the boom 130 is required.

[0082] In addition, when the suspended load L0 is located near the bottom of the boom 130 and is captured by both the image capturing devices 10R and 10L, the control unit 26 R , I LThe distance from the imaging devices 10R and 10L to the load L0 may be calculated by applying a triangulation method to the coordinates of the suspended load portion L1 in the image and the distance between the imaging devices 10R and 10L. Then, the control unit 26 may determine the length to extend or retract the boom 130 based on the calculated distance.

[0083] In the above embodiment, the boom automation device 1 is equipped on the loaded truck crane 100, but the present invention is not limited to this. In addition to the loaded truck crane 100, the present invention is applicable to all cranes, such as wheel cranes and crawler cranes, that have a rotating body that can rotate relative to a support body and a boom that is attached to the rotating body and can be raised and lowered relative to the rotating body. [Explanation of symbols]

[0084] 1...Boom automation device, 10R, 10L...imaging device, 20...I / O port, 21...CPU, 22...ROM, 23...RAM, 24...bus line, 25...identification unit, 26...control unit, 30...storage device, 31...boom automation program, 32...identification dictionary, 33...position information data, 100...loaded truck crane, 110...vehicle, 111...operator's cab, 112...bed, 120...rotating body, 121...frame, 122...rotation motor, 123...rotation angle detection sensor, 125...display device, 130...boom, 131...derailing cylinder, 132...telescopic cylinder, 133...boom head, 134...boom angle detection sensor, 135...boom length detection sensor, 140...wire rope, 141...winch drum, 150...hook, 160...outrigger, H...horizontal direction, H L ,H R …Horizontal line, I R , I L ...image, L0...hanging load, L1...hanging load part, LR L ,LR R ...left and right lines, P0...point, V...vertical direction, V L ,V R …Vertical line, VD…Vertical direction

Claims

1. A boom automation device for a crane including a rotating body that can rotate relative to a support body, and a boom that is provided on the rotating body and can be raised and lowered relative to the rotating body, an omnidirectional camera unit provided on a boom head of the boom, which captures an omnidirectional image with a field of view in all directions including the front-rear direction, the left-right direction, and the up-down direction of the boom head; an identification unit that identifies an object to be lifted by the crane in the omnidirectional image captured by the omnidirectional camera unit and determines the position of the identified object in the image; a control unit that determines the position of a horizontal line indicating a horizontal plane in the omnidirectional image captured by the omnidirectional camera unit and the position of a vertical line indicating a vertical plane perpendicular to the left-right direction in the omnidirectional image based on the position of the boom head relative to the support body and the position of the omnidirectional camera unit relative to the boom head, and raises the boom when the position of the object in the image determined by the identification unit is above the position of the horizontal line, and rotates the boom when the position of the object in the image determined by the identification unit is below the position of the horizontal line and above the position of the vertical line; A crane boom automation device comprising:

2. The omnidirectional camera unit a right hemispherical camera provided on a right side surface of the boom head and configured to capture an image to the right of the boom head; a left hemispherical camera provided on a left side surface of the boom head and configured to capture an image to the left of the boom head; Equipped with generating the omnidirectional image from a right image portion captured by the right hemispherical camera and a left image portion captured by the left hemispherical camera; The crane boom automation system according to claim 1.

3. the control unit rotates the boom to the right when the position of the object in the image determined by the identification unit is below the horizontal line and above the vertical line and is located in a right-side image portion captured by the right hemispherical camera, and rotates the boom to the left when the position of the object in the image determined by the identification unit is below the horizontal line and above the vertical line and is located in a left-side image portion captured by the left hemispherical camera. The crane boom automation device according to claim 2.

4. The boom is extendable and retractable, the control unit determines the position of a left-right line indicating a left-right plane perpendicular to a front-to-rear direction in the omnidirectional image captured by the omnidirectional camera unit based on the position of the boom head relative to the support body and the position of the omnidirectional camera unit relative to the boom head, and extends the boom when the position of the object in the image determined by the identification unit is forward of the position of the left-right line, and retracts the boom when the position of the object in the image determined by the identification unit is behind the position of the left-right line. The crane boom automation device according to any one of claims 1 to 3.

5. A boom control method for a crane including a rotating body that can rotate relative to a support body, and a boom that is provided on the rotating body and can be raised and lowered relative to the rotating body, comprising: an omnidirectional camera unit provided on a boom head of the boom, which captures an omnidirectional image with a field of view in all directions including the front-to-back, left-to-right, and up-to-down directions of the boom head, identifies an object to be lifted by the crane in the omnidirectional image captured by the omnidirectional camera unit, and determines the position of the identified object in the image; determining the position of a horizontal line indicating a horizontal plane in the omnidirectional image captured by the omnidirectional camera unit and the position of a vertical line indicating a vertical plane perpendicular to the left-right direction in the omnidirectional image based on the position of the boom head relative to the support body and the position of the omnidirectional camera unit relative to the boom head, and raising the boom when the position of the object in the image determined in the step of determining the position of the object in the image is above the position of the horizontal line, and rotating the boom when the position of the object in the image determined in the step of determining the position of the object in the image is below the position of the horizontal line and above the position of the vertical line; A method for controlling a boom of a crane comprising:

6. A program for controlling a boom of a crane including a rotating body that can rotate relative to a support body and a boom that is provided on the rotating body and can be raised and lowered relative to the rotating body, On the computer, an omnidirectional camera unit provided on a boom head of the boom, which captures an omnidirectional image with a field of view in all directions including the front-to-back, left-to-right, and up-to-down directions of the boom head, identifies an object to be lifted by the crane in the omnidirectional image captured by the omnidirectional camera unit, and determines the position of the identified object in the image; determining the position of a horizontal line indicating a horizontal plane in the omnidirectional image captured by the omnidirectional camera unit and the position of a vertical line indicating a vertical plane perpendicular to the left-right direction in the omnidirectional image based on the position of the boom head relative to the support body and the position of the omnidirectional camera unit relative to the boom head, and raising the boom when the position of the object in the image determined in the step of determining the position of the object in the image is above the position of the horizontal line, and rotating the boom when the position of the object in the image determined in the step of determining the position of the object in the image is below the position of the horizontal line and above the position of the vertical line; A program to execute.

Images