Crane device

Abstract

[Problem] To provide a crane device capable of generating a bird's-eye view image in a wide range at a low cost. [Solution] A crane device 12 includes a sensor group 26, wide-angle cameras 17, 18, a display 84, a rotation base 21, a boom 22, and a winch 23. The wide-angle camera 17 is attached to the lower surface of the boom 22 in a collapsed posture, and the wide-angle camera 18 is attached to the upper surface of the boom 22 in the collapsed posture. The angles of view of the wide-angle camera 17 and the wide-angle camera 18 are both 210 degrees. The wide-angle camera 17 captures an image of the front side of the crane vehicle 10 from above. The wide-angle camera 18 captures an image of the rear side of the crane vehicle 10 from above. That is, the wide-angle cameras 17, 18 capture images of the entire circumference of the crane vehicle 10 from above. Bird's-eye view image data is generated on the basis of the first captured image data generated by the wide-angle camera 17 and the second captured image data generated by the wide-angle camera 18. A bird's-eye view image represented by the bird's-eye view image data is displayed on a display 84.

Description

Crane equipment

[0001] The present invention relates to a crane apparatus including a swivel base and a boom.

[0002] Patent Document 1 discloses a crane apparatus equipped with a swivel base and a boom. Four peripheral cameras facing forward, backward, left, and right are mounted on the swivel base. The four peripheral cameras have wide-angle lenses and capture images of the entire periphery of the crane apparatus. The four images captured by the four peripheral cameras are combined to generate an overhead image.

[0003] JP 2011-151742 A

[0004] The peripheral camera described in Patent Document 1 is mounted on a swivel base. Therefore, the height of the swivel base is used as a reference for the generated overhead image. However, because a crane apparatus has a wide working range (working radius), the crane apparatus described in Patent Document 1 may not be able to obtain an overhead image that covers the entire working range.

[0005] Furthermore, the crane apparatus described in Patent Document 1 requires four peripheral cameras to generate an overhead image, which increases costs.

[0006] The present invention has been made in view of the above-mentioned circumstances, and an object of the present invention is to provide a crane apparatus that can generate a wide-range bird's-eye image at low cost.

[0007] (1) A crane apparatus according to the present invention includes a swivel base, a boom that is extendable and foldable on the swivel base, a first wide-angle camera attached to a first side of the boom, a second wide-angle camera attached to a second side opposite the first side, and a controller that executes a correction process to correct first captured image data generated by the first wide-angle camera and second captured image data generated by the second wide-angle camera, and an overhead image generation process to generate overhead image data showing an overhead image based on the corrected first captured image data and second captured image data.

[0008] The first wide-angle camera and the second wide-angle camera are arranged facing opposite directions across the boom. The first wide-angle camera captures images of the surroundings in front of the crane apparatus, and the second wide-angle camera captures images of the surroundings in rear of the crane apparatus. Alternatively, the first wide-angle camera captures images of the surroundings in the left part of the crane apparatus, and the second wide-angle camera captures images of the surroundings in the right part of the crane apparatus. In other words, the first wide-angle camera and the second wide-angle camera capture images of the surroundings in the crane apparatus that are wider than when cameras are mounted on the swivel base. Based on the first captured image data and the second captured image data generated by the first wide-angle camera and the second wide-angle camera, overhead image data showing an overhead image is generated. As a result, the crane apparatus according to the present invention can generate overhead image data showing a wide range of overhead images using two wide-angle cameras. In other words, the crane apparatus according to the present invention can generate wide range of overhead images at low cost.

[0009] (2) The boom may have a plurality of cylindrical bodies including a base boom, and the first wide-angle camera and the second wide-angle camera may be provided on the base boom.

[0010] Because the first and second wide-angle cameras are mounted on the base boom, cable routing is easier than when the first and second wide-angle cameras are mounted on other cylindrical bodies that slide relative to the base boom. Furthermore, the height positions of the first and second wide-angle cameras do not change when the boom is extended or retracted. Because the height positions of the first and second wide-angle cameras do not change, changes in the height position of the viewpoint of the overhead image due to the extension or retraction of the boom can be suppressed.

[0011] (3) The first side surface may be a lower surface of the boom when the boom is in a lying position in which the boom extends along the horizontal direction, and the second side surface may be a top surface of the boom when the boom is in the lying position.

[0012] If the first wide-angle camera and the second wide-angle camera were provided on the left and right sides of the boom in the lowered position, the load hook hanging from the tip of the boom and the load attached to the load hook might appear in both the image captured by the first wide-angle camera and the image captured by the second wide-angle camera. If the load hook and the load appear in two images, the operator might feel uncomfortable seeing the images of the load hook and the load in the overhead image generated by combining the two images. By attaching the first wide-angle camera to the underside (first side) of the boom in the lowered position, the load hook and the load will only appear in the image captured by the first wide-angle camera. As a result, the operator is prevented from feeling uncomfortable seeing the load hook and the load in the overhead image.

[0013] (4) The crane apparatus according to the present invention may further include a control room in which a control device is installed, and a display installed in the control room and displaying the overhead image.

[0014] An overhead image is provided to an operator operating a crane device in a control room.

[0015] (5) The crane apparatus according to the present invention may further include a control room in which a control device is installed, a display installed in the control room and displaying the overhead image, a boom hoisting angle sensor that outputs a first detection value indicating a boom hoisting angle, a boom length sensor that outputs a second detection value indicating an extended length of the boom, a swing angle sensor that outputs a third detection value indicating a swing position of the swivel base, and a memory. The controller further executes a specific coordinate storage process that acquires specific coordinates corresponding to a position of the boom tip in an actual coordinate system based on the crane apparatus and stores the specific coordinates in the memory based on the first detection value, the second detection value, and the third detection value, a specific display position generation process that generates position information on the overhead image corresponding to the specific coordinates based on the first detection value, the second detection value, the third detection value, and the specific coordinates, and a display process that displays the overhead image with an emphasis object located at a position indicated by the position information on the display.

[0016] The specific coordinates are coordinates that indicate a special position, such as the position where the crane lifts a load, the position where the crane lowers a load, or a position specified by an operator. For example, when the swivel base rotates, the overhead image changes (rotates). Even if the overhead image changes, the highlight object continues to indicate the special position. Therefore, the crane apparatus according to the present invention allows the operator to easily recognize the current status of the special position.

[0017] (6) The crane apparatus according to the present invention may further include a load hook suspended from the tip of the boom by a wire rope, and a load sensor that outputs a fourth detection value corresponding to a load applied to the wire rope. The controller executes the specific coordinate storage process based on a change in the load indicated by the fourth detection value.

[0018] When a load is lifted or lowered, the load weight changes and specific coordinates are stored in memory. In other words, the specific coordinates indicate the position where the load was placed before transport and the position where the load will be transported. Therefore, the crane apparatus according to the present invention can easily allow the operator to recognize the position where the load was placed before transport and the position where the load will be transported by using an emphasis object in the overhead image.

[0019] (7) The crane apparatus according to the present invention may further include a load hook suspended from the tip of the boom by a wire rope, a boom hoisting angle sensor that outputs a first detection value indicating the boom hoisting angle, a boom length sensor that outputs a second detection value indicating the extended length of the boom, and a wire sensor that outputs a fifth detection value corresponding to the distance from the tip of the boom to the load hook. The controller further executes a hook height calculation process that calculates a hook height indicating the height position of the load hook based on the first detection value, the second detection value, and the fifth detection value, and a hook image generation process that generates hook image data by correcting at least one of the first captured image data or the second captured image data based on the hook height. The correction process includes a process of correcting the first captured image data based on a predetermined position corresponding to the base end of the boom shown in the first captured image indicated by the first captured image data, and a process of correcting the second captured image data based on a predetermined position corresponding to the base end of the boom shown in the second captured image indicated by the second captured image data.

[0020] The controller may generate both the overhead image data and the hook image data.

[0021] (8) The crane apparatus according to the present invention may further include a load hook suspended from the tip of the boom by a wire rope, a boom hoisting angle sensor that outputs a first detection value indicating the boom hoisting angle, a boom length sensor that outputs a second detection value indicating the extended length of the boom, a load load sensor that outputs a fourth detection value corresponding to the load applied to the wire rope, and a wire sensor that outputs a fifth detection value corresponding to the distance from the tip of the boom to the load hook. The controller further executes an elevation difference calculation process that calculates an elevation difference between the height of a loading surface on which the load is placed before transport or the height of a loading surface on which the transported load is lowered and the crane installation surface based on the first, second, fourth, and fifth detection values. The correction process is further executed based on the elevation difference.

[0022] The correction process is performed based on the height position of the loading surface on which the load is placed before transportation or the height position of the loading surface on which the transported load is lowered and the height difference between the crane installation surface, so that an overhead image with even less distortion can be generated.

[0023] (9) The sum of the angle of view of the first wide-angle camera and the angle of view of the second wide-angle camera may be 360 ​​degrees or more.

[0024] When the sum of the angle of view of the first wide-angle camera and the angle of view of the second wide-angle camera is 360 degrees or more, an overhead image including the entire periphery of the crane apparatus is generated.

[0025] (10) The crane apparatus according to the present invention may further include a boom hoisting angle sensor that outputs a first detection value indicating a boom hoisting angle, and a memory that stores a threshold angle. The controller executes the overhead image generation process based on whether the boom hoisting angle indicated by the first detection value is equal to or greater than the threshold angle.

[0026] When the boom hoisting angle is equal to or greater than the threshold angle, overhead image data is generated, so that only overhead images with minimal distortion can be shown to the operator.

[0027] (11) The crane apparatus according to the present invention may further include a display that displays the overhead image, a boom hoisting angle sensor that outputs a first detection value indicating a boom hoisting angle, a boom length sensor that outputs a second detection value indicating an extended length of the boom, a swing angle sensor that outputs a third detection value indicating a swing position of the swivel base, and a memory. The controller executes the following processes: a specific coordinate storage process that acquires specific coordinates corresponding to a position of the boom tip in an actual coordinate system based on the crane apparatus, based on the first detection value, the second detection value, and the third detection value, and stores the specific coordinates in the memory; a specific display position generation process that generates position information on the overhead image corresponding to the specific coordinates, based on the first detection value, the second detection value, the third detection value, and the specific coordinates; a specific image generation process that generates specific image data indicating a specific image that is an image of a portion of the overhead image indicated by the position information; and a display process that displays the overhead image and the specific image on the display.

[0028] In a specific coordinate storage process, the controller acquires specific coordinates indicating a specific position, such as a load pick-up position or a load drop-off position, and stores the coordinates in memory. The controller generates position information on the overhead image corresponding to the specific coordinates and displays a specific image, which is an image of the portion of the overhead image indicated by the position information, on the display. At the load pick-up position or load drop-off position, a worker is present to hang or lower a load (suspended load). The worker may give instructions to the operator using body language such as gestures, hand movements, and pointing, even during rotation. When the overhead image rotates in accordance with the rotation of the swivel, the position on the overhead image of the worker who is communicating instructions to the operator changes. With only the overhead image, the operator needs to search for the worker in the overhead image. Because an image of the specific position, such as the load pick-up position or load drop-off position, i.e., an image of the worker, is displayed on the display regardless of the rotation of the swivel, the operator does not need to search for the worker in the overhead image and can reliably receive the worker's instructions.

[0029] (12) The controller may further execute a generation process of generating forward expanded image data showing a forward expanded image including an area from the tip of the boom to a load hook, based on the corrected first captured image data and the corrected second captured image data. The display process includes a process of displaying the forward expanded image on a display.

[0030] An expanded forward image including the area from the tip of the boom to the load-lifting hook is displayed on the screen, making it even easier for the operator to check safety when carrying out tasks such as transporting cargo.

[0031] The crane apparatus according to the present invention can generate a wide-range bird's-eye view image at low cost.

[0032] FIG. 1 is a functional block diagram of the crane apparatus 12. FIG. 2 is a schematic side view of the crane vehicle 10 with the boom 22 raised. FIG. 3 is a schematic side view of the crane vehicle 10 with the boom 22 in a lowered position. FIG. 4 is a diagram showing the control device 29 in the operator's cabin 13. FIG. 5 is a flowchart of the process for displaying overhead images, etc. FIG. 6(A) is a schematic diagram of a first captured image taken by the first wide-angle camera 17, FIG. 6(B) is a schematic diagram of a second captured image taken by the second wide-angle camera 18, and FIG. 6(C) is a schematic diagram of a synthesized overhead image. FIG. 7(A) is a schematic diagram of an overhead image during crane operation, and FIG. 7(B) is a schematic diagram of an overhead image in a state where a suspended load 35 is being lowered to a load lowering position. FIG. 8 is a flowchart of the process for calculating an elevation difference in Modification 1. FIG. 9 is a flowchart of the process for displaying overhead images, etc. in Modification 1. Fig. 10 is a schematic plan view of the crane vehicle 10 according to Modification 2. Fig. 11 is a functional block diagram of the crane apparatus 12 according to Modification 3. Fig. 12 is a flowchart of the process for displaying an overhead image, etc., according to Modification 3. Fig. 13 is an explanatory diagram illustrating an image displayed on the display 84.

[0033] An embodiment of the present invention will be described below. Note that the embodiment described below is merely one example of the present invention, and it goes without saying that the embodiment can be modified as appropriate without departing from the spirit of the present invention. For example, the execution order of each process described below can be modified as appropriate without departing from the spirit of the present invention. Alternatively, part of the process described below can be omitted as appropriate without departing from the spirit of the present invention.

[0034] Fig. 1 is a configuration diagram and functional block diagram of the crane apparatus 12. Fig. 2 is a side view of the mobile crane 10 with the boom 22 raised. Fig. 3 is a schematic side view of the mobile crane 10 with the boom 22 lowered.

[0035] In this embodiment, a crane vehicle 10 (see FIG. 2) equipped with a crane apparatus 12 shown in FIG. 1 will be described.

[0036] 2 and 3, the mobile crane 10 is a rough terrain crane. However, the mobile crane 10 may also be an all-terrain crane or a crawler crane.

[0037] The crane vehicle 10 comprises a running body 11, an outrigger device 14 attached to the running body 11, a crane device 12 mounted on the running body 11, and a control room 13.

[0038] The outrigger device 14 has a plurality of jacks 24 that contact the ground to stabilize the position of the mobile crane 10. The crane device 12 is used with the jacks 24 extended and the position of the mobile crane 10 stabilized.

[0039] The crane apparatus 12 has a swivel base 21, a boom 22, a winch 23, a group of sensors 26 (see Figure 1), a group of hydraulic actuators 27 (see Figure 1), a hydraulic supply device 28 (see Figure 1), a control device 29 (see Figure 1), a control monitor device 80 (see Figure 1), a display 84 (see Figure 1), a hook block 32, a first wide-angle camera 17, a second wide-angle camera 18, and a controller 70 (see Figure 1).

[0040] The swivel base 21 is rotatably held by the traveling body 11. The rotation axis of the swivel base 21 is aligned in the vertical direction (up-down direction) and is located approximately in the center of the traveling body 11.

[0041] The boom 22 is supported by the swivel base 21 so as to be able to rise and fall. That is, the boom 22 is able to rise and fall and to swing.

[0042] The boom 22 includes a base boom 42, an intermediate boom 43, and a top boom 44. The base boom 42, the intermediate boom 43, and the top boom 44 are each shaped like a rectangular tube. The intermediate boom 43 is positioned inside the base boom 42, and the top boom 44 is positioned inside the intermediate boom 43. In other words, the base boom 42, the intermediate boom 43, and the top boom 44 are nested within each other, forming a so-called telescopic structure. The boom 22 extends and retracts as the intermediate boom 43 slides relative to the base boom 42 and the top boom 44 slides relative to the intermediate boom 43. The base boom 42, the intermediate boom 43, and the top boom 44 correspond to the "cylinder" described in the claims. The boom 22 may be composed of two cylinders or four or more cylinders.

[0043] The base boom 42 is located on the outermost side. Therefore, the outer surface of the base boom 42 is always exposed to the outside. The rectangular cylindrical base boom 42 has four side surfaces 45, 46, 47, and 48. The side surface 45 is the underside when the boom 22 is in the horizontally-oriented lying position (see FIG. 3 ). The side surface 46 is the upper side when the boom 22 is in the lying position. The side surfaces 47 and 48 are the left and right sides when the boom 22 is in the lying position. The side surface 45 corresponds to the "first side surface" in the claims. The side surface 46 corresponds to the "second side surface" in the claims.

[0044] The boom 22 is supported by the swivel base 21 at a base boom 42 so that it can be raised and lowered. In other words, the base boom 42 does not move (slide) when the boom 22 is extended or retracted. The wide-angle cameras 17 and 18 are attached to the base boom 42, which does not move when the boom 22 is extended or retracted. In other words, the positions of the wide-angle cameras 17 and 18 do not change when the boom 22 is extended or retracted.

[0045] The winch 23 is attached to the base end of the boom 22 or the swivel base 21. The winch 23 has a drum 56 around which a wire rope 41 (hereinafter referred to as "wire 41") is wound, and a sheave 57 around which the wire 41 is wound. The wire 41 is wound onto or paid out from the drum 56 when the winch 23 is driven.

[0046] The wire 41 is wound around a sheave 57 provided at the base end of the boom 22, a sheave 58 provided at the tip end of the boom 22, and a pulley device (not shown). The pulley device has a plurality of first sheaves (not shown) provided at the tip end of the boom 22 and a plurality of second sheaves (not shown) provided on the hook block 32.

[0047] The hook block 32 includes a hook body 33, a plurality of the second sheaves rotatably held within the hook body 33, and a hook 34 provided on the underside of the hook body 33. The hook block 32 or the hook 34 corresponds to the "load lifting hook" set forth in the claims.

[0048] The number of times the wire 41 is wound around the first sheave and the second sheave is the so-called "number of times the wire is wound." The greater the number of times the wire is wound, the greater the maximum lifting load, which is the maximum amount of cargo that can be hoisted by the crane apparatus 12. In other words, the greater the number of times the wire is wound, the greater the lifting performance of the crane apparatus 12. In the example shown in FIG. 2, four wires 41 are visible between the tip of the boom 22 and the hook block 32, and the number of times the wire is wound is "4." The number of times the wire is wound is input to the controller 70, for example, via the control monitor device 80 (see FIG. 1) and stored in the memory 72 (see FIG. 1). The number of times the wire is wound is used to calculate the suspension length, which is the distance from the tip of the boom 22 to the suspended load 35 (see FIG. 4).

[0049] Two winches 23 may be provided on the crane apparatus 12. A hook block 32, which is a so-called main hook, is connected to a wire 41 of one of the winches 23, and a sub-hook (not shown) is connected to a wire 41 of the other winch 23. The hook block 32, which is the main hook, and the sub-hook correspond to the "load-lifting hook" described in the claims.

[0050] 1 , the crane apparatus 12 includes a hydraulic supply unit 28 and a hydraulic actuator group 27. The hydraulic actuator group 27 includes a swing motor 51, a derricking cylinder 52, a telescopic cylinder 53, a hydraulic motor 54, and the jack 24.

[0051] The swing motor 51 is a hydraulic motor that rotates via hydraulic oil supplied from the hydraulic supply unit 28, and rotates the swing base 21. The hoisting cylinder 52 is a hydraulic cylinder that extends and retracts via hydraulic oil supplied from the hydraulic supply unit 28, and raises and lowers the boom 22. The telescopic cylinder 53 is a hydraulic cylinder that extends and retracts via hydraulic oil supplied from the hydraulic supply unit 28, and extends and lowers the boom 22. The hydraulic motor 54 rotates via hydraulic oil supplied from the hydraulic supply unit 28, and rotates the drum 56 of the winch 23.

[0052] The hydraulic supply device 28 includes a hydraulic pump driven by the engine 15 mounted on the traveling vehicle 11, piping connecting the hydraulic pump to the swing motor 51 and other components of the hydraulic actuator group 27, and a hydraulic selector valve and other components provided on the piping. The hydraulic selector valve may be a so-called electromagnetic valve, which is driven by a drive signal input from the controller 70. Driving the electromagnetic valve drives the swing motor 51, the hoisting cylinder 52, the telescopic cylinder 53, the hydraulic motor 54, and the jack 24. In other words, by outputting drive signals, the controller 70 can swing, hoist, and extend the boom 22, reel in or pay out the wire 41, and extend or retract the jack 24.

[0053] The sensor group 26 includes a swing angle sensor 61 , a boom length sensor 62 , a hoisting angle sensor 63 , a drum sensor 64 , and a suspended load sensor 65 .

[0054] The swivel angle sensor 61 outputs a detection value corresponding to the swivel angle of the swivel base 21 from a swivel reference position. The swivel reference position is the position shown in FIG. 3 where the boom 22 protrudes forward. The swivel angle sensor 61 is, for example, a rotary encoder provided on the swivel shaft of the swivel base 21. The swivel angle sensor 61 outputs, as detection values, a number of pulse signals corresponding to the swivel angle of the swivel base 21. The detection value output by the swivel angle sensor 61 corresponds to the "third detection value" recited in the claims.

[0055] The boom length sensor 62 outputs a detection value corresponding to the length of the boom 22. The boom length sensor 62 may be a sensor that outputs a detection value indicating the length of the boom 22, a sensor that outputs a detection value indicating the extended length of the telescopic cylinder 53, or a sensor that outputs a detection value indicating the operating time of the telescopic cylinder 53. The detection value output by the boom length sensor 62 corresponds to the "second detection value" recited in the claims.

[0056] The hoisting angle sensor 63 outputs a detection value corresponding to the hoisting angle of the boom 22. The hoisting angle sensor 63 is, for example, an inclination sensor or a horizontal sensor that outputs the angle relative to the horizontal plane. Alternatively, the hoisting angle sensor 63 may be a sensor that outputs a detection value indicating the extension length or operating time of the hoisting cylinder 52. The detection value output by the hoisting angle sensor 63 corresponds to the "first detection value" recited in the claims.

[0057] The drum sensor 64 outputs a detection value corresponding to the length of the wire 41 unwound from the winch 23. The drum sensor 64 is, for example, a rotary encoder attached to the shaft of the drum 56. The drum sensor 64 outputs a pulse signal corresponding to the unwound and reeled-out of the wire 41. The number of pulses per unit time indicates the rotation speed of the drum 56, i.e., the unwound and reeled-out speeds of the wire 41, and the total number of pulses indicates the amount of rotation of the drum 56, i.e., the unwound and reeled length of the wire 41. The drum sensor 64 corresponds to the "wire sensor" recited in the claims. The pulse signal output by the drum sensor 64 corresponds to the "fifth detection value" recited in the claims.

[0058] The load sensor 65 outputs a detection value corresponding to the load applied to the wire 41 by the load 35 (see FIG. 4 ). The load sensor 65 is, for example, a hydraulic sensor that measures the hydraulic pressure supplied to the hoisting cylinder 52. The memory 72 pre-stores an arithmetic expression for calculating the load applied to the wire 41 by the load 35 based on the boom length measured by the boom length sensor 62, the hoisting angle measured by the hoisting angle sensor 63, and the hydraulic pressure measured by the hydraulic sensor serving as the load load sensor 65. Alternatively, the memory 72 pre-stores a correspondence table that associates the boom length measured by the boom length sensor 62, the hoisting angle measured by the hoisting angle sensor 63, the hydraulic pressure measured by the hydraulic sensor serving as the load load sensor 65, and the load applied to the wire 41 by the load 35. The load applied to the wire 41 by the load 35 is calculated using the arithmetic expression or the correspondence table. The detection value output by the load load sensor 65 corresponds to a "fourth detection value" recited in the claims.

[0059] The suspended load sensor 65 may be a tension meter or load cell that directly measures the load that the suspended load 35 applies to the wire 41, or a pressure sensor that measures the hydraulic pressure supplied to the winch 23, or the like.

[0060] In the following, the detection values ​​output by sensors such as the rotation angle sensor 61, boom length sensor 62, elevation angle sensor 63, drum sensor 64, and suspended load sensor 65 will be referred to as "sensor detection values" and explained.

[0061] The first wide-angle camera 17 and the second wide-angle camera 18 each include a condenser lens, an optical system, multiple image sensors, a control circuit, and a communication interface. The condenser lens is a so-called wide-angle lens or a fisheye lens, and the first wide-angle camera 17 and the second wide-angle camera 18 are so-called wide-angle cameras or fisheye cameras.

[0062] The optical system has, for example, multiple lenses, and guides light focused by a focusing lens to an imaging element. The multiple imaging elements are image sensors such as CCDs or CMOSs. The multiple imaging elements are arranged in a matrix (grid) pattern. The multiple imaging elements output image data according to the incident light. The image data consists of multiple pixel data and pixel position data. If the captured image is a color image, one pixel data consists of, for example, three color pixel data. The control circuit captures images using the first wide-angle camera 17 and the second wide-angle camera 18 and generates image data. The communication interface converts the image data into transmission data and outputs it. The control circuit is, for example, a driver IC, and the communication interface is, for example, a communication IC.

[0063] 2 and 3 , the first wide-angle camera 17 is attached to a side surface 45 of the base boom 42, and the second wide-angle camera 18 is attached to a side surface 46 of the base boom 42. The first wide-angle camera 17 and the second wide-angle camera 18 are disposed at a distance from the base end of the base boom 42. That is, the first wide-angle camera 17 and the second wide-angle camera 18 are positioned higher than the swivel base 21 when the boom 22 is raised (crane operation state). Therefore, the range of the overhead image (see FIG. 7A ) generated from the images captured by the first wide-angle camera 17 and the second wide-angle camera 18 is wider than when the cameras are attached to the swivel base 21. That is, by attaching the first wide-angle camera 17 and the second wide-angle camera 18 to the boom 22, it is possible to generate an overhead image with a wider range than when the cameras are attached to the swivel base 21.

[0064] 2 and 3, the first wide-angle camera 17 and the second wide-angle camera 18 are positioned closer to the tip than the center of the base boom 42 so that a bird's-eye view image of a wider range can be generated. However, the first wide-angle camera 17 and the second wide-angle camera 18 are positioned slightly away from the tip of the base boom 42 so that the first wide-angle camera 17 and the second wide-angle camera 18 are appropriately spaced apart from the wire 41.

[0065] In addition, the first wide-angle camera 17 and the second wide-angle camera 18 may be positioned in the center of the base boom 42 or closer to the base end than the center of the base boom 42, as long as the range of the overhead image they generate covers the maximum working radius of the crane apparatus 12 (see the dotted line in Figure 7 (A)).

[0066] The first wide-angle camera 17 and the second wide-angle camera 18 may be monochrome cameras that output image data representing monochrome images, or color cameras that output image data representing color images. Furthermore, the image data output by the first wide-angle camera 17 and the second wide-angle camera 18 may be data representing a still image or video data consisting of multiple frames.

[0067] 2, the first wide-angle camera 17 and the second wide-angle camera 18 are attached to the base boom 42 so that, for example, when the boom 22 is at a predetermined angle, the first wide-angle camera 17 and the second wide-angle camera 18 are at the same height. In other words, the first wide-angle camera 17 and the second wide-angle camera 18 are aligned in a generally horizontal direction during crane operation.

[0068] The dashed-dotted line in Fig. 2 indicates the imaging range (angle of view) of the first wide-angle camera 17, and the dashed-two-dotted line in Fig. 2 indicates the imaging range (angle of view) of the second wide-angle camera 18. As shown in Fig. 2, the mounting positions of the first wide-angle camera 17 and the second wide-angle camera 18 and the angles of view of the first wide-angle camera 17 and the second wide-angle camera 18 are determined so that at least the front of the crane truck 10 falls within the imaging range of the first wide-angle camera 17, and at least the rear of the crane truck 10 falls within the imaging range of the second wide-angle camera 18. In other words, the crane truck 10 is always included in the overhead image (see Fig. 6(C)) generated by the images captured by the first wide-angle camera 17 and the second wide-angle camera 18.

[0069] In order to generate an overhead image capturing the entire periphery of the crane truck 10, the sum of the angles of view of the first wide-angle camera 17 and the second wide-angle camera 18 is set to 360 degrees or more. In the example shown in FIG. 2 , the angles of view of the first wide-angle camera 17 and the second wide-angle camera 18 are both 210 degrees. That is, the sum of the angles of view of the first wide-angle camera 17 and the second wide-angle camera 18 is 420 degrees. As long as the sum of the angles of view of the first wide-angle camera 17 and the second wide-angle camera 18 is 360 degrees or more, the angles of view of the first wide-angle camera 17 and the second wide-angle camera 18 may be different. For example, the angle of view of the first wide-angle camera 17 may be 140 degrees, and the angle of view of the second wide-angle camera 18 may be 220 degrees. Since the angle of view of a wide-angle camera is generally up to 220 degrees, the sum of the angle of view of the first wide-angle camera 17 and the angle of view of the second wide-angle camera 18 is 440 degrees or less.

[0070] The swing angle sensor 61, boom length sensor 62, hoisting angle sensor 63, drum sensor 64, and suspended load sensor 65 of the sensor group 26 shown in Fig. 1 are connected to the controller 70 by cables. That is, the sensor detection values ​​output by the swing angle sensor 61 and the like are input to the controller 70. Note that the sensor detection values ​​output by each sensor may also be input to the controller 70 via wireless communication.

[0071] The first wide-angle camera 17 and the controller 70 are connected by a cable 19, and the second wide-angle camera 18 and the controller 70 are connected by a cable 20. That is, the captured image data output by the first wide-angle camera 17 and the second wide-angle camera 18 is input to the controller 70 via the cables 19 and 20. However, the captured image data may also be input to the controller 70 via wireless communication. In this case, a battery and a transmitting antenna are attached to the boom 22, and a receiving antenna connected to the controller 70 by a cable is installed in the cockpit 13. The first wide-angle camera 17 and the second wide-angle camera 18 capture images using power supplied from the battery and transmit the captured image data from the transmitting antenna. The transmitted captured image data is received by the receiving antenna and input to the controller 70.

[0072] Each sensor, such as the turning angle sensor 61, continuously performs detection at predetermined time intervals (so-called sampling periods / sampling cycles) and outputs sensor detection values ​​at the predetermined time intervals. The first wide-angle camera 17 and the second wide-angle camera 18 continuously capture images at the predetermined time intervals and output captured image data at the predetermined time intervals. Alternatively, the first wide-angle camera 17 and the second wide-angle camera 18 output video data consisting of multiple frames.

[0073] FIG. 4 is a diagram showing the control device 29 in the cockpit 13.

[0074] 4, the control device 29 is disposed in the cockpit 13. The control device 29 includes an operation lever, a foot pedal, an operation button, and the like, which are operated by the operator.

[0075] The control device 29 is connected to the controller 70 via a signal line (not shown). An operator operates the control device 29 to input instructions to the controller 70 and control the crane apparatus 12.

[0076] 1, the control monitor device 80 includes a display 81, a transparent sheet-like touch sensor 82 overlaid on the display 81, and a speaker 83. In other words, the control monitor device 80 is a so-called AML. An operator uses the touch sensor 82 to input various information such as the number of wire hooks.

[0077] The display 84 displays an overhead image, or an overhead image and a hook image. The display 84 corresponds to the "display" described in the claims.

[0078] The controller 70 includes a CPU 71, which is a central processing unit, a memory 72, a communication interface 73, a power supply circuit 74, and a communication bus (not shown). The controller 70 is realized by an IC, a computer (or a microcomputer), resistors, diodes, coils, capacitors, etc. mounted on a control board. The control board is disposed, for example, in a control box disposed in the cockpit 13. The controller 70 corresponds to the "controller" recited in the claims.

[0079] The CPU 71, memory 72, hydraulic supply device 28, swing angle sensor 61 and the like of the sensor group 26, wide-angle cameras 17 and 18, and control device 29 are connected to a communication bus (not shown). A control program 75 (described below) executed by the CPU 71 reads data and information from memory 72, stores the data and information in memory 72, and controls the operation of the hydraulic actuator group 27. The control program 75 also acquires sensor detection values ​​output by the sensor group 26, captured image data output by the wide-angle cameras 17 and 18, and inputs made to the control device 29 by the operator.

[0080] The memory 72 stores in advance an OS 77, which is an operating system, and a control program 75 executed by the CPU 71. The memory 72 also stores in advance object data, a first correction function, a second correction function, a third correction function, a first specified function, a second specified function, and a threshold angle.

[0081] The object data is data representing highlighting objects 86, 87, and 88 (see FIG. 7 ) that are displayed superimposed on the overhead image. In the example shown in FIG. 7 , the highlighting objects 86, 87, and 88 have the same shape, which is a dashed rectangle. However, the shapes of the highlighting objects 86, 87, and 88 may be different, or may be other shapes such as a circle or an ellipse, or may be other line types such as a solid line or a dashed line.

[0082] The first correction function is a function for generating first corrected image data by correcting (converting) the first captured image data output by the first wide-angle camera 17. The second correction function is a function for generating second corrected image data by correcting (converting) the second captured image data output by the second wide-angle camera 18. The third correction function is a function for generating hook image data by correcting (converting) the first captured image data output by the first wide-angle camera 17.

[0083] The first specification function is a function that specifies (calculates) the position on the image of the hook block 32 that appears in the first captured image captured by the first wide-angle camera 17, based on the actual position of the hook block 32. The second specification function is a function that specifies (calculates) the load pick-up position and the load unloading position on the overhead image. In other words, the second specification function is a function that acquires the display positions of the highlight objects 86, 87 that are superimposed on the overhead image. The load pick-up position refers to the location (position) where the suspended load 35 (see FIG. 4 ) is placed together before being transported by the crane apparatus 12. The unloading position refers to the location (position) where the suspended load 35 is transported by the crane apparatus 12.

[0084] The threshold angle is, for example, the minimum angle at which the boom 22 can be raised or lowered at which an overhead image can be generated with high accuracy.

[0085] The control program 75 is a program that controls the driving of the hydraulic actuator group 27 and the like based on operation signals input from the control device 29, and also executes a process for displaying an overhead image and the like (see FIG. 5), which will be described later.

[0086] The communication interface 73 is connected to the Internet 99 via a mobile communication network (not shown). The communication interface 73 converts transmission data conforming to the communication standards of the mobile communication network into data in a format that can be input to the controller 70, or converts data output by the controller 70 into the transmission data. The communication interface 73 is, for example, a communication modem.

[0087] The power supply circuit 74 converts the DC voltage supplied from the battery 16 mounted on the running object 11 into a DC voltage of a predetermined voltage value, such as 3.3 V or 5 V, and outputs the converted DC voltage. The power supply circuit 74 is, for example, a DC / DC converter (power supply IC) such as a switching regulator. The battery 16 is charged by the engine 15 of the running object 11. The DC voltage output by the power supply circuit 74 is supplied to the CPU 71, the control device 29, the sensor group 26, the wide-angle cameras 17 and 18, etc. In FIG. 1 , power supply lines from the power supply circuit 74 to the sensor group 26, etc. are omitted from the illustration.

[0088] The following describes the processing executed by the controller 70 of the crane truck 10. The processing executed by the controller 70 is processing that the control program 75 causes the CPU 71 to execute.

[0089] Fig. 5 is a flowchart of the process of displaying an overhead image, etc., executed by the controller 70. Fig. 6(A) is a schematic diagram of a first captured image captured by the first wide-angle camera 17. Fig. 6(B) is a schematic diagram of a second captured image captured by the second wide-angle camera 18. Fig. 6(C) is a schematic diagram of an overhead image generated based on the first captured image and the second captured image. Fig. 7(A) is a schematic diagram of an overhead image during crane operation, and Fig. 7(B) is a schematic diagram of an overhead image in a state where the crane truck 10 has lowered the load 35 to the load unloading position.

[0090] The process for displaying an overhead image, etc., will be described with reference to Figure 5. The controller 70 starts the process for displaying an overhead image, etc., based on, for example, the power supply of the control device 29 being turned on, or based on the operator making a predetermined input to the control device 29 or the control monitor device 80. Note that the following description will be given assuming that the crane installation surface (e.g., the ground surface) on which the crane vehicle 10 is installed and the load pick-up position and load drop-off position where the suspended load 35 is placed are at the same height.

[0091] The controller 70 acquires the sensor detection values ​​output by each sensor of the sensor group 26 and stores them in the memory 72 (S11). The controller 70 repeatedly acquires the sensor detection values ​​at a predetermined sampling period and stores them in the memory 72.

[0092] The controller 70 determines whether the amount of the suspended load has changed (increased or decreased) based on the latest sensor detection value output by the suspended load sensor 65 and the previous sensor detection value output by the suspended load sensor 65 (S12). For example, the controller 70 determines that the amount of the suspended load has not changed (increased or decreased) based on the absolute value of the difference between the suspended load indicated by the latest sensor detection value and the suspended load indicated by the previous sensor detection value being less than a predetermined value pre-stored in the memory 72 (S12: No), and determines that the suspended load has changed (increased or decreased) based on the absolute value being equal to or greater than a predetermined value (S12: Yes). The suspended load changes when the crane apparatus 12 suspends or lowers the suspended load 35 (see FIG. 4). That is, in step S12, it is determined whether the suspended load 35 has been suspended or lowered.

[0093] When the controller 70 determines that the amount of the lifted load has changed (S12: Yes), it calculates the coordinates (x, y) of the load pick-up position or load unloading position in the actual coordinate system as specific coordinates based on the latest sensor detection values ​​stored in the memory 72 in step S11, and stores them in the memory 72 (S13). The processing in step S13 corresponds to the "specific coordinate storage processing" set forth in the claims.

[0094] The real coordinate system is a two-dimensional coordinate system in which, for example, the center of rotation of the swivel platform 21 or the base end (hoisting center) of the boom 22 is the origin (0,0), the direction along the longitudinal direction of the mobile crane 10 is the x-axis direction, and the direction along the lateral direction of the mobile crane 10 is the y-axis direction. For example, if the center of rotation and the starting point of the hoisting of the boom 22 are roughly the same, the length of the boom 22 detected by the boom length sensor 62 is L, the rotation angle detected by the swivel angle sensor 61 is α, and the hoisting angle detected by the hoisting angle sensor 63 is θ, then in the two-dimensional coordinate system (x, y), x = L cos θ cos α and y = L cos θ sin α.

[0095] The real coordinate system may be a three-dimensional coordinate system. In other words, coordinates (x, y, z) may be calculated as specific coordinates. In that case, z = L sin θ + A - B. A is a value stored in advance in memory 72 and indicates the height from the crane installation surface to the base end of the boom 22. B is the distance from the tip of the boom 22 to the suspended load 35, and is a length (suspending length) corresponding to the detection value of the drum sensor 64 and the number of wire hooks.

[0096] The real coordinate system may also be a circular coordinate system (L, θ) or a cylindrical coordinate system (L, θ, z).

[0097] In step S12, an increase in the weight of the lifting load and a decrease in the weight of the lifting load may be determined separately. For example, if the shape and size of an emphasis object 86 (see FIG. 7) placed at the position where the suspended load before transportation is placed in the bird's-eye image are made different from the shape and size of an emphasis object 87 (see FIG. 7) placed at the position where the suspended load 35 is transported, an increase in the weight of the lifting load and a decrease in the weight of the lifting load are determined separately in step S12. Then, in step S13, specific coordinates indicating the load pick-up position and specific coordinates indicating the load unloading position are stored separately in memory 72.

[0098] When the controller 70 determines that the lifting load amount has not increased or decreased (S12: No), or after executing the process of step S13, the controller 70 determines whether a designation instruction has been input via the control monitor device (AML) 80 (S14). For example, if there is a location (position) that the operator wants to highlight in the overhead image (see FIG. 7), the operator inputs the designation instruction when the tip of the boom 22 reaches directly above that location. The location (position) that the operator wants to highlight is, for example, a location where an obstacle 89 (see FIG. 7) is located.

[0099] When the controller 70 determines that a designation instruction has been input (S14: Yes), it calculates the coordinates (x, y) in the real coordinate system as specific coordinates based on the sensor detection values ​​stored in memory 72 when the designation instruction was input, and stores the specific coordinates in memory 72 (S13).

[0100] If the controller 70 determines that a designation instruction has not been input (S14: No), it determines whether the hoisting angle of the boom 22 indicated by the sensor detection value acquired in step S11 is equal to or greater than a threshold angle pre-stored in the memory 72 (S15). That is, in step S15, it is determined whether the boom 22 has been raised to an angle at which an overhead image can be generated with high accuracy. The controller 70 repeatedly executes the processes from step S11 onwards until it determines that the hoisting angle of the boom 22 is equal to or greater than the threshold angle (S15: No, S27: No).

[0101] If the controller 70 determines that the boom hoisting angle is equal to or greater than the threshold angle (S15: Yes), it acquires the first captured image data output by the first wide-angle camera 17 and the second captured image data output by the second wide-angle camera 18 (S16). The controller 70 generates first corrected image data by correcting the first captured image data based on the acquired first captured image data, the first correction function stored in the memory 72, and the boom hoisting angle indicated by the sensor detection value acquired in step S11 (S17). Specifically, the controller 70 calls the first correction function, passes the first captured image data and the boom hoisting angle as arguments to the first correction function, and receives the first corrected image data as a return value. Note that the first correction function may be a class that generates the first corrected image data as an instance in response to input of the first captured image data and the boom hoisting angle. The processing in step S17 corresponds to the "correction processing" described in the claims.

[0102] The generation of the first corrected image data will be described in detail with reference to FIG. 6A . The first wide-angle camera 17 is a wide-angle camera with a 210-degree angle of view. Therefore, the first captured image represented by the first captured image data captures a wide range, but has distortion that increases the farther away from the center of the image capture. This distortion depends on the angle of view of the first wide-angle camera 17 and the height of the first wide-angle camera 17 from the crane installation surface. The angle of view of the first wide-angle camera 17 is constant. The height of the first wide-angle camera 17 is determined by the distance from the base end of the base boom 42 to the first wide-angle camera 17 and the boom hoisting angle of the boom 22. The distance from the base end of the base boom 42 to the first wide-angle camera 17 is a constant value. Therefore, the distortion of the first captured image depends on the boom hoisting angle of the boom 22. Furthermore, the image required to generate the overhead image is the image of the lower portion of the first captured image. The first correction function is a function that corrects distortion in a predetermined range around a predetermined correction reference located on the lower side in the first captured image. The predetermined correction reference is a position corresponding to the crane truck 10 shown in the first captured image. More specifically, the predetermined correction reference is a position corresponding to the base end of the boom 22 in the first captured image. The position corresponding to the base end of the boom 22 includes the base end of the boom 22, the rotation center of the swivel base 21, the front end of the crane truck 10, the rear end of the crane truck 10, the center of the crane truck 10, etc. The predetermined correction reference corresponds to the "predetermined position" recited in the claims.

[0103] Specifically, the first correction function performs correction to change the positions of pixels contained in the first captured image data received as an argument, and the undulation angle θ is used as a variable parameter in changing the pixel positions.

[0104] Using the first correction function, the first captured image is corrected to a first corrected image that is an image taken from a viewpoint looking down on the crane truck 10 from the first wide-angle camera 17 and in which distortion has been corrected.

[0105] In step S17 of the overhead image display process shown in Figure 5, the controller 70 generates second corrected image data by correcting the second captured image data based on the second captured image data acquired in step S16, the second correction function stored in memory 72, and the angle of elevation of the boom 22 indicated by the sensor detection value acquired in step S11.

[0106] The generation of the second corrected image data will be described in detail with reference to FIG. 6B . The second wide-angle camera 18 is a wide-angle camera with a 210-degree angle of view. Therefore, the second captured image represented by the second captured image data captures a wide range, but has distortion that increases the farther away from the center of the image capture. This distortion depends on the angle of view of the second wide-angle camera 18 and the height of the second wide-angle camera 18 from the crane installation surface. The angle of view of the second wide-angle camera 18 is constant. The height of the second wide-angle camera 18 is determined by the distance from the base end of the base boom 42 to the second wide-angle camera 18 and the boom hoisting angle of the boom 22. The distance from the base end of the base boom 42 to the second wide-angle camera 18 is constant. Therefore, the distortion of the second captured image depends on the boom hoisting angle of the boom 22. Furthermore, the image required to generate the overhead image is the image of the upper portion of the second captured image. The second correction function is a function that corrects distortion in a predetermined range around a predetermined correction reference located at the top of the second captured image. The predetermined correction reference is a position corresponding to the crane truck 10 shown in the second captured image. More specifically, the predetermined correction reference is a position corresponding to the base end of the boom 22 in the second captured image. The position corresponding to the base end of the boom 22 includes the rotation center of the swivel base 21, the front end of the crane truck 10, the rear end of the crane truck 10, the center of the crane truck 10, etc. The predetermined correction reference corresponds to the "predetermined position" recited in the claims.

[0107] Specifically, the second correction function performs correction to change the positions of pixels contained in the second captured image data received as an argument, using the undulation angle θ as a variable parameter in changing the pixel positions.

[0108] Using the second correction function, the second captured image is corrected to a second corrected image that is an image taken from a viewpoint looking down on the crane truck 10 from the second wide-angle camera 18 and in which distortion has been corrected.

[0109] 5, the controller 70 combines the first corrected image represented by the first corrected image data with the second corrected image represented by the second corrected image data to generate overhead image data representing the overhead image (S18). The processing of step S18 corresponds to the "overhead image generation processing" set forth in the claims.

[0110] Fig. 6(C) is an overhead image shown by the synthesized overhead image data. In the overhead image, the image of the crane truck 10 is located at the center of the overhead image. A wide area around the crane truck 10 is displayed in the overhead image. As shown in Fig. 7(A), the overhead image displays an area wider than the maximum working area (working radius) of the crane truck 10. In Fig. 7(A), the maximum working area is indicated by a two-dot chain line.

[0111] 5 , the controller 70 determines whether a hook image display instruction has been input via the control monitor device (AML) 80 (S19). When the operator wants to display a hook image on the display 84 in addition to or instead of an overhead image, the operator inputs the hook image display instruction via the control monitor device (AML) 80. Although not shown in the flowchart, when the hook image display instruction is input, the controller 70 stores information indicating that the hook image display instruction has been input in the memory 72. The controller 70 determines whether the hook image display instruction has been input based on whether the information is stored in the memory 72 (S19).

[0112] When the controller 70 determines that a hook image display command has been input (S19: Yes), it identifies the position of the hook block 32 (or sub-hook) in the first captured image (see FIG. 6A) based on the sensor detection values ​​acquired in step S11 (S20). More specifically, the controller 70 calculates the suspension length, which is the distance from the tip of the boom 22 to the hook block 32, based on the boom length detected by the boom length sensor 62, the payout length of the wire 41 detected by the drum sensor 64, and the number of wire loops. For example, the suspension length is calculated by subtracting the boom length from the payout length of the wire 41 and then dividing the result by the number of wire loops. The controller 70 also calculates the height of the tip of the boom 22 relative to the crane installation surface based on the hoisting angle detected by the hoisting angle sensor 63, the boom length, and the A. The controller 70 calculates the hook height, which is the height from the crane installation surface to the hook block 32, by subtracting the suspension length from the height. The controller 70 calls a first specification function stored in memory 72, passes the hook height as an argument to the called first specification function, and receives pixel position information indicating the position (hook position) of the hook block 32 in the first captured image as a return value. The pixel position information is, for example, a pixel number. In other words, in step S20, the position on the image of the hook block 32 that serves as the center (reference) of correction for generating hook image data from the first captured image data is identified. The processing of step S20 corresponds to the "hook height calculation processing" recited in the claims.

[0113] The controller 70 acquires hook image data based on the pixel position information indicating the correction standard, the first captured image data, and the third correction function (S21). More specifically, the controller 70 passes the first captured image data acquired in step S16 and the pixel position information indicating the hook position acquired in step S20 as arguments to the third correction function, and receives the corrected hook image data as a return value. The hook image indicated by the hook image data is an image showing the hook block 32, the suspended load 35, and their surroundings. The range of the surroundings is determined by the third correction function. Note that the first specific function and the third correction function may be a single function. The processing of step S21 corresponds to the "hook image generation processing" set forth in the claims.

[0114] The height positions of the hook block 32 and the suspended load 35 change due to the operation of the winch 23. Therefore, the positions at which the hook block 32 and the suspended load 35 appear in the first captured image are indefinite. On the other hand, to obtain a hook image with corrected distortion, it is necessary to perform correction using the hook block 32 and the suspended load 35 on the image as a correction reference. The positions (pixel position information) of the hook block 32 and the suspended load 35 on the image are identified based on the sensor detection values, and hook image data is generated based on the identified pixel position information. Therefore, the hook image represented by the hook image data acquired in step S21 is an image with appropriately corrected distortion.

[0115] In this way, the correction standard for generating the overhead image data is the above-mentioned predetermined position and is fixed, and the correction standard for generating the hook image data is the variable hook block 32. By appropriately setting the correction standard for generating the overhead image data and the correction standard for generating the hook image data, overhead image data and hook image data with appropriately corrected distortion are generated.

[0116] When the controller 70 determines in step S19 that a hook image display instruction has not been input (S19: No), the controller 70 skips the processes of steps S20 and S21.

[0117] After executing the process of step S21, or if the controller 70 determines in step S19 that a hook image display instruction has not been input (S19: No), the controller 70 determines whether specific coordinates are stored in the memory 72 (S22). If the controller 70 determines that specific coordinates are stored in the memory 72 (S22: Yes), the controller 70 acquires a position (specific display position) corresponding to the specific coordinates in the overhead image (see FIG. 7 ) based on the sensor detection values ​​acquired in step S11 and the specific coordinates stored in the memory 72 (S23). More specifically, the controller 70 passes the rotation angle detected by the rotation angle sensor 61, the boom hoisting angle detected by the hoisting angle sensor 63, and the specific coordinates stored in the memory 72 as arguments to the second specification function, and receives the specific display position indicating the position on the overhead image as a return value. The specific display position is, for example, pixel position information (pixel number) indicating a load pick-up position, a load unloading position, or a designated position. The specific display position indicates the position where the highlight objects 86, 87, and 88 (see FIG. 7) are to be displayed in the overhead image. The process of step S23 corresponds to the "specific display position generation process" described in the claims. The specific display position corresponds to the "position information on the overhead image corresponding to specific coordinates" described in the claims.

[0118] After executing the process of step S23, the controller 70 displays the overhead image on which the highlight objects 86, 87, and 88 are superimposed, and the hook image on the display 84 (S24). Specifically, the controller 70 inputs the overhead image data generated in step S18, the hook image data acquired in step S21, and the object data with the specific display position added acquired in step S23 to the display 84. Note that if a hook image display instruction has not been input (S19: No) and the hook image data has not been generated, only the overhead image data and the object data are input to the display 84. Note that if the specific coordinates have not been stored in the memory 72 (S22: No) and the specific display position has not been acquired, the overhead image data without the object data is input to the display 84. The process of step S25 corresponds to the "display process" set forth in the claims.

[0119] An example of superimposing objects 86, 87, and 88 on a bird's-eye view image is superimposing a layer displaying objects 86, 87, and 88 on a layer displaying the bird's-eye view image. In this case, the layer displaying objects 86, 87, and 88 is generated using object data whose display positions are determined by the specific display positions. Another example of superimposing objects 86, 87, and 88 on a bird's-eye view image is changing the pixel indicated by the specific display position in the pixels of the bird's-eye view image data to the object data indicating a dashed line.

[0120] The display 84 displays the overhead image and the hook image. The overhead image and the hook image are displayed side by side, either vertically or horizontally. A display changeover switch may be provided on the display 84, or the control monitor device (AML) 80 may receive a display changeover instruction. In this case, the display 84 displays either the overhead image or the hook image, whichever image the operator selects using the display changeover switch or in a display changeover instruction.

[0121] Referring to FIG. 7, the highlighting of pick-up locations, drop-off locations, and obstacles 89 by highlighting objects 86, 87, and 88 will be described in detail.

[0122] As shown in FIG. 7 , when the boom 22 is rotated by the swivel base 21, the wide-angle cameras 17 and 18 attached to the boom 22 also rotate, and as a result, the overhead image also rotates. The display positions of the highlight objects 86, 87, and 88 in the overhead image are calculated based on the actual coordinates of the pickup position, the unloading position, and the obstacle 89. Therefore, even if the overhead image rotates, the highlight objects 86, 87, and 88 do not move from the pickup position, the unloading position, and the obstacle 89 in the overhead image. For example, if the pickup position, the unloading position, and the obstacle 89 in the overhead image were identified by image recognition, the processing would take a long time and there would be a risk of image recognition errors. By calculating the display positions of the highlight objects 86, 87, and 88 in the overhead image based on the actual coordinates of the pickup position, the unloading position, and the obstacle 89, the pickup position, the unloading position, and the obstacle 89 can be reliably highlighted.

[0123] 5, after executing the process of step S25, the controller 70 determines whether or not to end the process of displaying an overhead image, etc. (S25). For example, the controller 70 determines to end the process based on the fact that the control device 29 of the crane apparatus 12 is powered off or an end instruction is input to the control device 29 (S25: Yes). If the controller 70 determines to end the process (S25: Yes), it ends the process of displaying an overhead image, etc. (END). If the controller 70 determines not to end the process (S25: No), it executes the processes from step S11 onwards again.

[0124] [Effects of the embodiment]

[0125] The first wide-angle camera 17 and the second wide-angle camera 18 are disposed facing opposite directions across the boom 22, and the sum of the angles of view of the first wide-angle camera 17 and the second wide-angle camera 18 is equal to or greater than 360 degrees. Therefore, the two cameras can capture images of the entire periphery of the crane apparatus 12. Furthermore, because the first wide-angle camera 17 and the second wide-angle camera 18 are attached to the boom 22, they can capture images of a wider area than when a camera for generating an overhead image is provided on the swivel base 21 or the traveling body 11. As a result, the crane apparatus 12 can generate wide-area overhead images at low cost.

[0126] Because the first wide-angle camera 17 and the second wide-angle camera 18 are attached to the boom 22, the crane apparatus 12 can generate an overhead image that covers the entire maximum working area (working radius). In other words, regardless of where the loading and unloading positions are located in the working area, the loading and unloading positions can be displayed in the overhead image.

[0127] The first correction function corrects distortion in the first captured image from a viewpoint looking down on the mobile crane 10 from the first wide-angle camera 17. The second correction function corrects distortion in the second captured image from a viewpoint looking down on the mobile crane 10 from the second wide-angle camera 18. Therefore, the crane apparatus 12 can generate overhead image data that shows an undistorted overhead image.

[0128] Since the first wide-angle camera 17 and the second wide-angle camera 18 can capture images of the surroundings of the mobile crane 10 from a higher position than when they are attached to the swivel base 21, overhead image data can be generated without performing a transformation that changes the viewpoint position, such as a projective transformation, thereby reducing the processing load on the controller 70.

[0129] Since the first wide-angle camera 17 and the second wide-angle camera 18 are provided on the base boom 42, wiring of the cables 19 and 20 is easier than when the first wide-angle camera 17 and the second wide-angle camera 18 are attached to the intermediate boom 43 or the top boom 44, which are other cylindrical bodies that slide relative to the base boom 42.

[0130] Because the first wide-angle camera 17 and the second wide-angle camera 18 are mounted on the base boom 42, the height positions of the wide-angle cameras 17, 18 do not change with the extension and retraction of the boom 22. Because the height positions of the wide-angle cameras 17, 18 do not change, it is possible to suppress changes in the height position of the viewpoint of the overhead image caused by the extension and retraction of the boom 22. In other words, even if the boom 22 extends or retracts, the range captured in the overhead image and the size of objects captured in the overhead image, such as the suspended load 35 or obstacles, do not change. As a result, it becomes easier for the operator to operate the crane apparatus 12.

[0131] If the first wide-angle camera 17 and the second wide-angle camera 18 were provided on the left and right side surfaces 47, 48 of the boom 22, respectively, there is a possibility that the hook block 32 and the load 35 attached to the hook block 32 would straddle both the image captured by the first wide-angle camera 17 and the image captured by the second wide-angle camera 18. In this case, there is a risk that the operator would feel uncomfortable seeing the image of the hook block 32 and the load 35 in the overhead image generated by combining both images. By attaching the first wide-angle camera 17 to the side surface 45 of the boom 22, the hook block 32 and the load 35 will only be visible in the image captured by the first wide-angle camera 17. As a result, the operator is prevented from feeling uncomfortable seeing the hook block 32 and the load 35 visible in the overhead image.

[0132] The display positions of the highlight objects 86, 87, 88 in the overhead image are calculated based on the actual coordinates of the loading position, the loading position, and the obstacle 89, so that even when the boom 22 is rotated, the loading position, the loading position, and the obstacle 89 can be reliably highlighted by the highlight objects 86, 87, 88.

[0133] The load of the suspended load changes when the suspended load 35 is lowered or lifted. The controller 70 stores specific coordinates in the memory 72 based on the change in the suspended load. Therefore, the coordinates (specific coordinates) of the load pick-up position and the load unloading position can be stored in the memory 72. As a result, the crane apparatus 12 can allow the operator to easily recognize, by means of the highlight objects 86, 87, 88 in the overhead image, the load pick-up position where the suspended load 35 is placed before transport and the load unloading position where the transported load 35 is lowered.

[0134] The position of the hook block 32 in the first captured image (hook position) is identified based on the actual position of the hook block 32, and a hook image is generated based on the hook position. Therefore, in addition to the overhead image, a hook image with corrected distortion that shows the hook block 32 and the suspended load 35 can also be displayed on the display 84 of the crane apparatus 12.

[0135] When the angle of the boom 22 is equal to or greater than the threshold angle, overhead image data is generated. Therefore, only overhead images with little distortion can be shown to the operator.

[0136] [Modification 1]

[0137] In the above-described embodiment, a case has been described in which the height positions of the loading and unloading positions are the same as the crane installation surface. In this modified example, an example will be described in which there is an elevation difference between the elevation positions of the loading and unloading positions and the crane installation surface, and the overhead image data is generated (corrected) according to this elevation difference. Note that the configurations and processes other than those described below are the same as the configurations and processes described in the embodiment. The same reference numerals and step numbers as in the embodiment will be used for the same configurations and processes, and descriptions thereof will be omitted.

[0138] The memory 72 of the crane apparatus 12 further stores a fourth correction function and a fifth correction function, which are indicated by dashed lines in Fig. 1. The fourth correction function is a function that receives the first captured image data, the hoisting angle, and the elevation difference as arguments, and outputs the third corrected image data as a return value. The fifth correction function is a function that receives the second captured image data, the hoisting angle, and the elevation difference as arguments, and outputs the fourth corrected image data as a return value.

[0139] 8 is a flowchart of the elevation difference calculation process executed by the controller 70 of the crane apparatus 12. FIG. 9 is a flowchart of the overhead image display process executed by the controller 70.

[0140] The controller 70 executes the elevation difference calculation process in parallel with the overhead image display process (see FIG. 9 ). For example, the controller 70 repeatedly executes the elevation difference calculation process at predetermined time intervals based on the power being supplied to the crane apparatus 12.

[0141] In the elevation difference calculation process shown in FIG. 8, first, the controller 70 acquires sensor detection values ​​and stores them in the memory 72 (S41).

[0142] The controller 70 determines whether the amount of the suspended load detected by the suspended load sensor 65 has changed (S42). Specifically, the controller 70 determines whether the amount of the suspended load detected by the suspended load sensor 65 has changed from less than a first predetermined value to a second predetermined value or more, and whether the amount of the suspended load has changed from the second predetermined value or more to less than the first predetermined value. The first predetermined value and the second predetermined value are stored in advance in the memory 72. Note that the first predetermined value and the second predetermined value may be the same.

[0143] When the controller 70 determines that the amount of the hoisting load has changed (S42: Yes), it determines whether the position is the load-receiving position or the load-unloading position. Specifically, the controller 70 determines that the position is the load-receiving position based on the fact that the amount of the hoisting load detected by the hoisting load sensor 65 has changed from less than the first predetermined value to equal to or greater than the second predetermined value (S43: load-receiving position), and determines that the position is the load-unloading position based on the fact that the amount of the hoisting load has changed from equal to or greater than the second predetermined value to less than the first predetermined value (S43: unloading position).

[0144] When the controller 70 determines that the load 35 is at the load pickup position (S43: load pickup position), it calculates the load pickup position elevation difference based on the sensor detection values ​​stored in memory 72 in step S41 and stores the calculated value in memory 72 (S44). Specifically, the controller 70 calculates the height H = L sin θ + A from the crane installation surface to the tip of the boom 22 at the moment the load 35 is suspended based on the boom length L detected by the boom length sensor 62 and the hoisting angle θ detected by the hoisting angle sensor 63. A is a value pre-stored in memory 72 and indicates the height from the crane installation surface to the base end of the boom 22. The controller 70 subtracts the suspension length calculated based on the detection value of the drum sensor 64 from the height H to calculate the load pickup position elevation difference E, which is the difference between the crane installation surface and the load pickup position (load placement surface). The load pickup position elevation difference E corresponds to the "load placement surface on which the suspended load is placed before transportation" as defined in the claims.

[0145] When the controller 70 determines that the position is the unloading position (S43: unloading position), it calculates the unloading position elevation difference F, which is the difference in elevation between the crane installation surface and the unloading position, based on the sensor detection values ​​stored in the memory 72 in step S41, and stores the calculated elevation difference F in the memory 72 (S45). The method for calculating the unloading position elevation difference F is the same as the method for calculating the loading position elevation difference E. The elevation of the unloading position (placing surface) corresponds to the "placing surface onto which the transported suspended load is lowered" recited in the claims.

[0146] If the height of the load-receiving position is lower than the crane installation surface, the load-receiving position height difference E will be a negative value. Also, if the height of the load-unloading position is lower than the crane installation surface, the load-unloading position height difference F will be a negative value.

[0147] After executing the process of step S44 or step S45, the controller 70 determines whether or not to end the process (S46) in the same manner as step S27 (see FIG. 5), and if it determines not to end the process (S46: No), it executes the processes from step S41 onward. If it determines to end the process (S46: Yes), the controller 70 ends the elevation difference calculation process (END).

[0148] The controller 70 executes the overhead image display process shown in FIG. 9 instead of the overhead image display process shown in FIG. 5 .

[0149] First, the controller 70 acquires a sensor detection value and stores it in the memory 72 (S51). Note that the processing of step S51 may be omitted and replaced with the processing of step S41 (see FIG. 8). Next, the controller 70 determines whether the suspended load detected by the suspended load sensor 65 has changed from less than the first predetermined value to equal to or greater than the second predetermined value, and whether it has changed from equal to or greater than the second predetermined value to less than the first predetermined value (S52). Note that the processing of step S52 may be omitted and replaced with the processing of step S42 (see FIG. 8).

[0150] When the controller 70 determines that the suspended load has changed (S52: Yes), it calculates specific coordinates indicating the load pick-up position or the load unloading position based on the sensor detection value and stores them in the memory 72 (S53).

[0151] If the controller 70 determines that the amount of the suspended load has not changed (S52: No), or after executing the process of step S53, it determines whether or not the above-mentioned designation instruction has been input (S14). If the controller 70 determines that the designation instruction has been input (S14: Yes), it calculates specific coordinates indicating the position of the obstacle or the like based on the sensor detection value and stores the coordinates in the memory 72 (S53).

[0152] If the controller 70 determines that a designation instruction has not been input (S14: No), it determines whether the hoisting angle detected by the hoisting angle sensor 63 is equal to or greater than the threshold angle (S15). The controller 70 repeatedly executes the processes from step S51 onwards until it determines that the hoisting angle of the boom 22 is equal to or greater than the threshold angle (S15: No, S27: No).

[0153] If the controller 70 determines that the hoisting angle detected by the hoisting angle sensor 63 is equal to or greater than the threshold angle (S15: Yes), it identifies an image area with a height difference and stores the image area in the memory 72 (S54). Specifically, the controller 70 identifies the boundary line between the crane installation surface and the load-receiving position (or load-unloading position) using the brightness difference or the lightness difference in the first captured image and the second captured image. The boundary line is, for example, data consisting of multiple pixel numbers. The image area may be stored in advance in the memory 72 as a predetermined range, such as a circle centered on the load-receiving position or the load-unloading position.

[0154] The controller 70 passes the first captured image data and the elevation angle as arguments to the first correction function and obtains the first corrected image data as a return value. The controller 70 also passes the second captured image data and the elevation angle as arguments to the second correction function and obtains the second corrected image data as a return value. In other words, the controller 70 obtains corrected image data for the crane installation surface (S55).

[0155] The controller 70 also passes the first captured image data, the elevation angle, and the elevation difference as arguments to a fourth correction function, and obtains the third corrected image data as a return value. The third corrected image data is data obtained by correcting the first captured image data according to the height of the loading position or the unloading position. The controller 70 also passes the second captured image data, the elevation angle, and the elevation difference as arguments to a fifth correction function, and obtains the fourth corrected image data as a return value. The fourth corrected image data is data obtained by correcting the second captured image data according to the height of the loading position or the unloading position. In other words, the controller 70 obtains corrected image data corrected according to the height of the loading position or the unloading position (S56).

[0156] The controller 70 generates overhead image data based on the first, second, third, and fourth corrected image data and the image range (S57). Specifically, within the range indicated by the image range, the controller 70 uses the third or fourth corrected image data to generate overhead image data, and outside the range indicated by the image range, the controller 70 uses the first and second corrected image data to generate overhead image data. In other words, the overhead image data is generated using corrected image data that has been appropriately corrected according to the height from the crane installation surface.

[0157] The controller 70 executes the processes from step S19 to step S24 and the process of step S25 in the same manner as in the embodiment, and ends the process of displaying an overhead image or the like (END).

[0158] [Effects of Modification Example 1]

[0159] In this modified example, even if the height of the loading position or loading position is higher or lower than the crane installation surface, overhead image data that is in focus across the entire overhead image and has distortion corrected can be generated.

[0160] [Modification 2] In the embodiment, an example was described in which the wide-angle cameras 17, 18 were attached to the side surfaces 45, 46 of the boom 22. In this modification, an example is described in which the wide-angle cameras 17, 18 are provided on the left and right side surfaces 47, 48 (see FIG. 10 ) of the boom 22. Note that configurations and processes other than those described below are the same as the configurations and processes described in the embodiment. The same reference numerals and step numbers as in the embodiment are used for the same configurations and processes, and descriptions thereof will be omitted.

[0161] FIG. 10 is a schematic plan view of the mobile crane 10 with the boom 22 in the lowered position.

[0162] The first wide-angle camera 17 is attached to a side surface 47, which is the left side surface of the base boom 42. The second wide-angle camera 18 is attached to a side surface 48, which is the right side surface of the base boom 42. In other words, the two wide-angle cameras 17, 18 are attached to the left and right sides of the boom 22, respectively. The side surface 47 corresponds to the "first side surface" in the claims. The side surface 48 corresponds to the "second side surface" in the claims.

[0163] The first wide-angle camera 17 captures an image of the left side of the crane truck 10, and the second wide-angle camera 18 captures an image of the right side of the crane truck 10. In the overhead image display process (see FIG. 5), the first corrected image data showing the left side of the crane truck 10 and the second corrected image data showing the right side of the crane truck 10 are combined to generate overhead image data (S18). In addition, hook image data is generated and acquired based on the first captured image data and the second captured image data (S21).

[0164] [Effects of Modification 2]

[0165] Even if two wide-angle cameras 17, 18 are attached to the left and right sides 47, 48 of the boom 22, it is possible to generate wide-area overhead image data that covers the maximum working area (working radius) of the crane apparatus 12.

[0166] [Modification 3]

[0167] At the load-receiving position, a worker is present who suspends the load 35 from the hook block 32. At the load-unloading position, a worker is present who removes the load 35 from the hook block 32. The worker may give instructions to the operator using body language such as gestures, hand movements, and pointing. In the embodiment, an example has been described in which emphasis objects 86, 87 (see FIG. 7 ) are placed at the load-receiving position and the load-unloading position in the overhead-view image. In this modified example, an example will be described in which, instead of placing the emphasis objects 86, 87 in the overhead-view image, or in addition to placing the emphasis objects 86, 87 in the overhead-view image, an image of a specific position in the overhead-view image, such as the load-receiving position or the load-unloading position (an image that is a part of the overhead-view image), is displayed on the display 84 separately from the overhead-view image.

[0168] The specific position on the overhead image changes as the swivel base 21 rotates. In this modified example, instead of simply extracting a portion of the overhead image and displaying it on the display 84, the specific position on the overhead image is determined based on the absolute coordinates of the specific position on the crane truck 10, and the image of the determined specific position is displayed on the display 84 as the specific image.

[0169] The configurations and processes other than those described below are the same as those described in the embodiment. The same configurations and processes as those in the embodiment are denoted by the same reference numerals and step numbers as those in the embodiment, and descriptions thereof will be omitted.

[0170] Fig. 11 is a functional block diagram of the crane apparatus 12 according to this modification. Fig. 12 is a flowchart of the process of displaying an overhead image, etc., executed by the controller 70 (control program 75) in this modification. Fig. 13 is an explanatory diagram illustrating a screen displayed on the display 84 in this modification.

[0171] As shown in FIG. 11 , the crane apparatus 12 further includes a boom top camera 101 in addition to the first wide-angle camera 17 and the second wide-angle camera 18 .

[0172] The boom top camera 101 is attached to the tip of the boom 22 with its lens facing downward. The boom top camera 101 generates image data showing an image (hanging load image) of the load 35 (or the hook block 32) suspended from the tip of the boom 22 captured from above. The boom top camera 101 is connected to the controller 70 via a cable (wired) or wirelessly so as to be able to communicate with the controller 70.

[0173] The memory 72 of the controller 70 stores the control program 75, correction functions, etc., as well as an enlargement function and a rotation function for generating specific image data. The enlargement function is a function for enlarging an image. The rotation function is a function for rotating an image. The enlargement function and the rotation function are, for example, affine transformation matrices.

[0174] The memory 72 also stores a display screen format. The display screen format is format data for a screen (see FIG. 13) to be displayed on the display 84. The controller 70 inputs each image data into the display screen format to generate screen data to be displayed on the display 84.

[0175] 13, the display device has a first display area 111, a second display area 112, a third display area 113, and a fourth display area 114. The first display area 111 is an area where the overhead image described in the embodiment is displayed.

[0176] The second display area 112 is an area where a suspended load image is displayed. The suspended load image is an image captured by the boom top camera 101.

[0177] The third display area 113 is an area where the specific image is displayed. The third display area 113 is located within the second display area 112. In other words, the specific image is displayed superimposed on the suspended load image.

[0178] The fourth display area 114 is an area where a forward expanded image is displayed. The forward expanded image is an image showing the area from the tip of the boom 22 to the hook block 32 (suspended load 35). The fourth display area 114 is located above the first display area 111 where the overhead image is displayed. In other words, the forward expanded image is displayed on top of the overhead image as a continuous image with the overhead image. However, the forward expanded image and the overhead image may be displayed as a single image. In FIG. 13 , the overhead image displayed in the first display area 111 employs a schematic diagram such as that shown in FIG. 7(B), and therefore does not have a sense of unity with the forward expanded image displayed in the fourth display area 114. However, when displayed as a single image, it is preferable to use an image that combines the overhead image (see FIG. 6(C)) and the first captured image (see FIG. 6(A)). The area from the tip of the boom 22 to the hook block 32 is captured by the wide-angle camera 17.

[0179] The controller 70 (control program 75) executes a process for displaying an overhead image, etc., shown in FIG. 12, instead of the process for displaying an overhead image, etc., shown in FIG.

[0180] The controller 70 executes steps S11 to S14 in the process of displaying an overhead image, etc. That is, the controller 70 acquires sensor detection values ​​and instructions from the operator (S11), and stores in the memory 72 specific coordinates indicating a specific position, such as a loading position, a loading position, or a position designated by the operator (S13).

[0181] Next, the controller 70 determines whether the boom 22 has been raised to an angle (threshold angle) at which overhead image data can be generated (S15). If the controller 70 determines that the boom 22 has not been raised to an angle (threshold angle) at which overhead image data can be generated (S15: No), that is, if the controller 70 determines that overhead image data cannot be generated, the controller 70 skips the process up to step S64 for generating image data such as overhead image data.

[0182] When the controller 70 determines that the boom 22 has been raised to an angle (threshold angle) at which overhead image data can be generated (S15: Yes), the controller 70 acquires the captured image data input from the cameras 17, 18, and 101 (S61).

[0183] Next, the controller 70 executes the processes of S17, S18, and S20 to S23. That is, the controller generates overhead image data (S18) and hook image data (S21), and acquires the specific display position (S23).

[0184] Next, the controller 70 generates image data showing an image around the specific position based on the overhead image data acquired in step S18 and the specific display position acquired in step S23 (S62). For example, the controller 70 passes the overhead image data and the specific display position as arguments to a specific function (not shown) stored in advance in the memory 72, and acquires the image data as a return value.

[0185] The controller 70 acquires specific image data representing a specific image based on the image data acquired in step S62, the rotation angle indicated by the sensor detection value acquired in step S11, and the rotation function and enlargement function stored in memory 72 (S63). Specifically, the controller 70 passes the image data and the rotation angle acquired in step S62 as arguments to the rotation function, and acquires, as a return value, image data representing an image around the specific position that has been rotated according to the rotation angle. The controller 70 also passes the image data and the enlargement factor as arguments to the enlargement function, and acquires the specific image data as a return value.

[0186] Here, the "swing angle" passed to the specification function is, for example, the swing angle of the swivel base 21 relative to the swing position when the boom 22 is facing a specific position. In other words, the specific image data generated in step S63 is an image of the boom 22 facing the specific position (load pick-up position or load unloading position) regardless of the swing angle of the boom 22, as shown in Fig. 13 .

[0187] The magnification factor passed to the specific function may be a magnification factor specified by the operator, or may be a magnification factor calculated from the hoisting angle of the boom 22. For example, the controller 70 calculates the distance between the wide-angle cameras 17, 18 and the specific position from the hoisting angle of the boom 22, and calculates the magnification factor according to that distance. Note that if the wide-angle cameras 17, 18 are provided at positions whose heights change depending on the extended length of the boom 22, such as the tip of the boom 22, the magnification factor may be calculated further depending on the extended length of the boom 22.

[0188] The processing in steps S62 and S63 corresponds to the "specific image generation processing" described in the claims. This specific image generation processing (S62 and S63) includes a rotation processing for rotating the specific image by an angle corresponding to the third detection value indicating the rotation angle.

[0189] Next, the controller 70 generates forward extended image data representing a forward extended image based on the corrected image data acquired in step S17 and a specific function (not shown) previously stored in the memory 72 (S64). Specifically, the controller 70 passes the corrected image data, the image data generated by the wide-angle camera 17, and the boom hoisting angle acquired in step S11 as arguments to the specific function, and acquires the forward extended image data as a return value. The processing in step S64 corresponds to the "generation processing" set forth in the claims.

[0190] The controller 70 generates display screen data to be displayed on the display 84 based on the generated image data and the display screen format stored in the memory 72 (S65). The controller 70 inputs the generated display screen data to the display 84, and causes the display 84 to display a screen (see FIG. 13) (S65). The processing of step S65 corresponds to the "display processing" set forth in the claims.

[0191] In step S15, if it is determined that the boom 22 has not been raised to an angle at which an overhead image can be generated (S15: No), the processes from step S61 to S65 are skipped.

[0192] The controller 70 determines whether to end the process (S25), and if it determines not to end the process (S25: No), it repeatedly executes the processes from step S11 onward at a predetermined cycle of several hundred milliseconds to several seconds. If it determines to end the process (S25: Yes), the controller 70 ends the process of displaying an overhead image or the like (END).

[0193] [Effects of Modification 3]

[0194] At the loading position or unloading position, there is a worker who suspends a load (a suspended load 35) from the hook block 32 or unsuspends the suspended load from the hook block 32. The worker may give instructions to the operator using body language even during rotation. As the overhead image rotates in response to the rotation of the boom 22 (swivel base 21), the position of the worker giving instructions to the operator changes in the overhead image. Furthermore, the worker may not appear in the overhead image due to the extension and retraction and elevation of the boom 22 or the enlargement of the overhead image. With only the overhead image, the operator must search for the worker in the overhead image, or may not be able to see the worker. Because images of specific positions, such as the loading position or unloading position, are displayed on the display regardless of the rotation of the boom 22, the operator does not need to search for the worker in the overhead image and can reliably receive the worker's instructions. Furthermore, there may be other workers besides the worker giving instructions at the loading position or unloading position. The operator can also constantly check the status of workers at the loading and unloading positions.

[0195] The specific image is displayed on the display 84 after being rotated by an angle corresponding to the rotation angle of the boom 22 (swivel base 21). Therefore, the orientation of the worker at the specific position (load pick-up position, load unloading position) is always constant regardless of the rotation of the boom 22. Therefore, it is easier for the operator to recognize the instructions of the worker compared to when the orientation of the worker changes according to the rotation of the boom 22.

[0196] Since a forward expanded image including the area from the tip of the boom 22 to the hook block 32 is displayed on the display 84, it becomes even easier for the operator to check for safety when carrying loads, etc.

[0197] In the third modification, an example in which a suspended load image is displayed in the second display area 112 has been described. However, the second display area 112 may display an image type specified by the operator or an image type corresponding to the current operation (operation) content. The current operation (operation) content refers to a swing operation, winch operation, boom hoisting, extension operation, and other operations of the crane apparatus 12. The types of images displayed in the second display area 112 may include not only suspended load images and bird's-eye images, but also images captured by other cameras installed on the crane apparatus 12, for example. Even if the type of image displayed in the second display area 112 changes, the specific image is always displayed on the display 84 (third display area 113). Because the specific image is always displayed on the display 84, the operator can always grasp the status of the load pick-up and unloading positions regardless of the operation (task) being performed.

[0198] In the third modification, an example has been described in which the specific image is displayed superimposed on the image (suspended load image) displayed in the second display area 112. However, the specific image may be displayed superimposed on another image such as a bird's-eye view image or a forward extended image, or may be displayed on the display 84 alongside the suspended load image, bird's-eye view image, forward extended image, or other image, separately from these images.

[0199] In the third modification, an example has been described in which the overhead image and the specific image are always displayed on the display 84. However, the operator may be able to select whether or not to display at least one of the overhead image and the specific image on the display 84. In other words, the overhead image, the specific image, or both the overhead image and the specific image may be displayed on the display 84 only when designated by the operator.

[0200] In the third modification, an example has been described in which the specific image is enlarged and displayed. However, the specific image may be displayed at the same magnification as the overhead image without being enlarged.

[0201] In the third modification, an example has been described in which the specific image is rotated by an angle corresponding to the turning angle. However, the specific image may be displayed on the display 84 without being rotated by an angle corresponding to the turning angle.

[0202] In the third modification, an example has been described in which the specific image is rotated at a rotation angle of the boom 22 based on the case where the boom 22 faces a specific position in step S63. However, the specific image may also be rotated at a rotation angle of the boom 22 based on the case where the boom 22 faces forward of the traveling body 11 (rotation reference position).

[0203] In the third modification, an example has been described in which the display 84 is installed in the cockpit 13. However, the display 84 may be installed in a location other than the cockpit 13. For example, the display 84 may be installed at a work site or other location in order to check, monitor, supervise, etc. the work.

[0204] In the third modification, an example has been described in which the screen of the display 84 is divided into a plurality of display areas and a plurality of images are displayed on the display 84. However, a plurality of displays may be used and a plurality of images may be allocated and displayed on the plurality of displays. In this case, the plurality of displays corresponds to the "display" recited in the claims.

[0205] [Other Modifications]

[0206] In the above embodiment, an example has been described in which the first wide-angle camera 17 and the second wide-angle camera 18 are attached to the base boom 42. However, the first wide-angle camera 17 and the second wide-angle camera 18 may also be attached to the intermediate boom 43 or the top boom 44. In this case, when correcting the captured image data using the correction function, the controller 70 passes the boom length detected by the boom length sensor 62 as an argument to the correction function in addition to the captured image data and the boom hoisting angle θ. In other words, distortion of the captured image data is corrected based on the boom length as well.

[0207] In the above embodiment, an example has been described in which specific coordinates are stored in the memory 72 based on a change in the suspended load (S12: Yes). However, if only specific coordinates designated by the operator are stored in the memory 72 (S14: Yes, S13), the processing of step S12 does not need to be executed. In this case, the suspended load sensor 65 does not need to be provided.

[0208] In the above embodiment, an example has been described in which the controller 70 executes the processes of steps S12, S13, and S14 for storing specific coordinates. However, if the highlight objects 86, 87, and 88 are not displayed on the overhead image, the processes of steps S12, S13, and S14 do not need to be executed. In that case, the rotation angle sensor 61, the boom length sensor 62, the drum sensor 64, and the suspended load sensor 65 do not need to be provided.

[0209] In the above embodiment, an example has been described in which the captured image data is corrected using the hoisting angle θ detected by the hoisting angle sensor 63 as a variable parameter to generate overhead image data. However, the captured image data may be corrected based on a predetermined correction standard (see FIG. 6 ) without using the hoisting angle θ as a variable parameter to generate overhead image data. In this case, the hoisting angle sensor 63 may not be provided.

[0210] In the above embodiment, an example has been described in which a hook image is displayed on the display 84 in response to an instruction from the operator. However, the hook image does not have to be displayed on the display 84. In that case, the processes from step S19 to step S21 are not executed, and the hook image data is not input to the display 84 in step S25. In that case, the drum sensor 64 (wire sensor) does not have to be provided.

[0211] In the above embodiment, an example has been described in which the shapes of the highlight object 86 displayed at the load pickup position, the highlight object 87 displayed at the load unload position, and the highlight object 88 displayed at the position designated by the operator in the overhead image (see FIG. 7 ) are the same. However, the shapes of the highlight object 86, the highlight object 87, and the highlight object 88 may be different. In this case, in step S12 of the overhead image display process (see FIG. 5 ), the controller 70 individually determines whether the suspended load has increased or decreased. If the controller 70 determines that the suspended load has increased, it calculates specific coordinates indicating the load pickup position from the sensor detection values ​​and stores them in the memory 72. If the controller 70 determines that the suspended load has decreased, it calculates specific coordinates indicating the load unload position from the sensor detection values ​​and stores them in the memory 72 (S13).

[0212] In the above-described embodiment, an example was described in which the position (hook position) of the hook block 32 in the first captured image is identified based on the sensor detection value, and the first captured image data is corrected based on the hook position to generate hook image data. However, the hook position may also be identified based on the first captured image. For example, the memory 72 pre-stores sample image data representing a sample image of the hook block 32. The controller 70 identifies a boundary line in the first captured image based on brightness differences and color differences, and determines that the object represented by the shape is the hook block 32 when the shape defined by the boundary line matches or is similar to the shape represented in the sample image. The controller 70 determines the position where the shape determined to be the hook block 32 exists as the hook position. In this case, sensor detection values ​​are not necessary in generating the hook image. That is, the controller 70 may generate the hook image from the first captured image data without using sensor detection values.

[0213] In the above embodiment, an example has been described in which the hook image data is generated only when an instruction is given by the operator. However, the hook image data may be generated at all times. In other words, the hook image may be displayed at all times on the display 84 of the crane apparatus 12.

[0214] In the above embodiment, an example has been described in which overhead image data is not generated when the undulation angle is less than the threshold angle (S15: No). However, overhead image data may be generated even when the undulation angle is less than the threshold angle. In this case, the processing of step S15 is not executed.

[0215] In the above embodiment, an example has been described in which overhead image data is not generated when the hoisting angle is less than the threshold angle. However, when the hoisting angle is less than the threshold angle, overhead image data may be generated based only on the first captured image data generated by the first wide-angle camera 17. More specifically, when the hoisting angle is less than the threshold angle, the first wide-angle camera 17 captures an image of the area around the crane truck 10. The second wide-angle camera 18 captures an image of the sky above the crane truck 10. The first captured image data captured by the first wide-angle camera 17, which captured an image of the area around the crane truck 10, is corrected to generate overhead image data. In this case, the center of the correction is approximately the center of the first captured image. In this way, the threshold angle may be used as a threshold for switching the method of generating overhead image data.

[0216] In the above-described embodiment, the first and second correction functions may be functions that perform projective transformation to change the viewpoint in addition to correcting distortion. For example, the first and second correction functions may perform viewpoint transformation to convert the viewpoint from directly above the base end of the boom 22 or directly above the rotation center of the swivel base 21 to a viewpoint looking directly downward. In the viewpoint transformation, the hoisting angle θ detected by the hoisting angle sensor 63 is also used as a variable parameter. After the captured image data is corrected by the first or second correction function, the corrected image data may be further transformed by a transformation function (affine transformation matrix) that performs projective transformation.

[0217] In the above embodiment, an example has been described in which the crane apparatus 12 is operated by the control device 29 installed in the control room 13. However, the crane apparatus 12 may also be operated by a remote control device. In this case, the overhead image data and the hook image data are transmitted to the remote control device via the Internet. The overhead image and the hook image are displayed on a display provided in the remote control device.

[0218] DESCRIPTION OF SYMBOLS 10: Crane vehicle 11: Traveling body 12: Crane device 13: Control room 17: First wide-angle camera 18: Second wide-angle camera 19, 20: Cable 21: Swivel base 22: Boom 23: Winch 26: Sensor group 29: Control device 32: Hook block 34: Hook 35: Suspended load 41: Wire rope 42: Base boom (cylindrical body) 43: Intermediate boom (cylindrical body) 44: Top boom (cylindrical body) 45: Side (first side) 46: Side (second side) 47: Side (first side) 48: Side (second side) 51: Swing motor 52: Drawer cylinder 53: Telescopic cylinder 54: Hydraulic motor 56: Drum 61: Swing angle sensor 62: Boom length sensor 63: Drawer angle sensor 64: Drum sensor 65: Suspended load sensor 70: Controller 71: CPU 72: Memory 73: Communication interface 75: Control program 80: Control monitor device 84: Display 101: Boom top camera

Claims

1. A crane apparatus comprising: a swivel base; a boom that is extendable and foldable on the swivel base; a first wide-angle camera attached to a first side of the boom; a second wide-angle camera attached to a second side opposite the first side; and a controller that executes a correction process for correcting first captured image data generated by the first wide-angle camera and second captured image data generated by the second wide-angle camera, and an overhead image generation process for generating overhead image data showing an overhead image based on the corrected first captured image data and second captured image data.

2. A crane apparatus as described in claim 1, wherein the boom has a plurality of cylindrical bodies including a base boom, and the first wide-angle camera and the second wide-angle camera are provided on the base boom.

3. A crane apparatus as set forth in claim 1, wherein the first side surface is the underside of the boom in a lying-down position in which the boom extends along a horizontal direction, and the second side surface is the upper surface of the boom in the lying-down position.

4. The crane apparatus according to claim 1, further comprising: a control room in which a control device is installed; and a display installed in said control room, for displaying said overhead image.

5. The crane apparatus according to claim 1, further comprising: a control room in which a control device is installed; a display installed in the control room and displaying the overhead image; a boom hoisting angle sensor outputting a first detection value indicating the boom hoisting angle; a boom length sensor outputting a second detection value indicating the extended length of the boom; a rotation angle sensor outputting a third detection value indicating the rotation position of the rotating base; and a memory, wherein the controller further executes the following steps: a specific coordinate storage process for acquiring specific coordinates corresponding to the position of the tip of the boom in an actual coordinate system based on the crane apparatus, based on the first detection value, the second detection value, and the third detection value, and storing the specific coordinates in the memory; a specific display position generation process for generating position information on the overhead image corresponding to the specific coordinates, based on the first detection value, the second detection value, the third detection value, and the specific coordinates; and a display process for displaying the overhead image with an emphasis object arranged at the position indicated by the position information on the display.

6. A crane apparatus as described in claim 5, further comprising: a load hook suspended by a wire rope from the tip of the boom; and a load sensor which outputs a fourth detection value according to the load applied to the wire rope, wherein the controller executes the specific coordinate storage process based on a change in the load indicated by the fourth detection value.

7. A load hook suspended from the tip of the boom by a wire rope, a boom hoisting angle sensor outputting a first detection value indicating the boom hoisting angle, a boom length sensor outputting a second detection value indicating the extended length of the boom, and a wire sensor outputting a fifth detection value corresponding to the distance from the tip of the boom to the load hook, wherein the controller further executes a hook height calculation process for calculating a hook height indicating a height position of the load hook based on the first detection value, the second detection value, and the fifth detection value, and a hook image generation process for generating hook image data by correcting at least one of the first captured image data or the second captured image data based on the hook height, the correction process comprising: a process of correcting the first captured image data based on a predetermined position corresponding to the base end of the boom shown in the first captured image shown by the first captured image data; The crane apparatus according to claim 1, further comprising a process of correcting the second captured image data based on a predetermined position corresponding to the base end of the boom shown in the second captured image indicated by the second captured image data.

8. A crane apparatus as described in claim 1, further comprising: a load hook suspended from the tip of the boom by a wire rope; a boom hoisting angle sensor outputting a first detection value indicative of the boom hoisting angle; a boom length sensor outputting a second detection value indicative of the extended length of the boom; a load load sensor outputting a fourth detection value corresponding to the load applied to the wire rope; and a wire sensor outputting a fifth detection value corresponding to the distance from the tip of the boom to the load hook; wherein the controller further executes an elevation difference calculation process based on the first detection value, the second detection value, the fourth detection value, and the fifth detection value to calculate an elevation difference between the height position of a loading surface on which the load before transportation is placed or the height position of a loading surface on which the transported load is lowered, and a crane installation surface; and the correction process is further executed based on the elevation difference.

9. The crane apparatus according to claim 1, wherein the sum of the angle of view of the first wide-angle camera and the angle of view of the second wide-angle camera is 360 degrees or more.

10. A crane apparatus as described in claim 1, further comprising: a boom hoisting angle sensor which outputs a first detection value indicating the boom hoisting angle; and a memory which stores a threshold angle, wherein the controller executes the overhead image generation process based on the fact that the boom hoisting angle indicated by the first detection value is equal to or greater than the threshold angle.

11. The crane apparatus according to claim 1, further comprising a display for displaying the overhead image, a boom hoisting angle sensor for outputting a first detection value indicating the boom hoisting angle, a boom length sensor for outputting a second detection value indicating the extended length of the boom, a rotation angle sensor for outputting a third detection value indicating the rotation position of the rotating base, and a memory, wherein the controller executes the following steps: a specific coordinate storage process for acquiring specific coordinates corresponding to the position of the tip of the boom in an actual coordinate system based on the crane apparatus, based on the first detection value, the second detection value, and the third detection value, and storing the specific coordinates in the memory; a specific display position generation process for generating position information on the overhead image corresponding to the specific coordinates, based on the first detection value, the second detection value, the third detection value, and the specific coordinates; a specific image generation process for generating specific image data indicating a specific image which is an image of a portion of the overhead image indicated by the position information; and a display process for displaying the overhead image and the specific image on the display.

12. The crane apparatus of claim 11, wherein the controller further executes a generation process for generating forward extended image data showing a forward extended image including the area from the tip of the boom to the load hook based on the corrected first captured image data and the second captured image data, and the display process includes a process for displaying the forward extended image on a display.

Images