Load placement assistance device, loaded truck crane and program

Abstract

To support cargo placement operation in loading operation of cargo-loading type truck cranes.SOLUTION: A cargo placement support device 1 includes an image acquisition unit 10 that acquires an overhead image taken from the tip of a boom of a cargo-loading truck crane that has a cab and a loading platform, an input unit 13 that accepts an input that specifies a suspended cargo to be loaded in the overhead image, a depth acquisition unit 11 that acquires the distance from the tip of the boom to the suspended cargo, an object recognition unit 14 that recognizes the shape of the suspended cargo from the overhead image and distance, an area recognition unit 12 that recognizes an empty area of the loading platform, a cargo placement generating unit 15 that determines the recommended cargo placement in which, in the empty area of the loading platform, a value obtained by dividing the largest remaining area by the outer circumference of the remaining area in one shape after placing the suspended cargo among the placements in which the center of gravity of the suspended cargo is closest to the cab is the maximum, and a display unit 16 that displays the recommended cargo placement.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technique for generating an arrangement of loads to be loaded onto the loading platform of a loading truck crane. [Background technology]

[0002] Loading a load onto a truck crane requires a lot of thought to ensure a good load ratio and weight distribution. To reduce the burden of loading work, technologies to assist the worker have been proposed.

[0003] For example, Patent Document 1 describes a product loading planning device that selects multiple products from a group of products to be loaded and creates a plan to arrange the multiple products on a predetermined equipment to be loaded. The product loading planning device in Patent Document 1 calculates evaluation values ​​based on the deviation of the total weight of the products to be loaded on the equipment from a target weight, the deviation in the longitudinal direction of the center of gravity of the equipment to be loaded when the products are loaded on the equipment from a target center of gravity, and the deviation in the width direction of the center of gravity of the equipment to be loaded when the products are loaded on the equipment from a target center of gravity, and applies a genetic algorithm.

[0004] The product loading planning device of Patent Document 2 is equipped with a storage means for storing vehicle specifications, product specifications, and product loading status on the vehicle, and comprises a computer that processes input information according to a predetermined program, an input terminal device for inputting product loading instruction information on the vehicle to the computer, a terminal device for displaying operation messages output from the computer and the product loading status on the vehicle, and a terminal device for documenting the final product loading specifications output from the computer, and the product loading status on the vehicle according to the input information is cumulatively displayed on the display terminal device.

[0005] The shipping instruction device of Patent Document 3 comprises an input device that inputs a parts shipping plan and outputs actual part shipping records; storage means that stores attribute data for each part and attribute data for the pallet on which the parts are loaded; shipping instruction calculation means that uses the attribute data for each part and pallet stored in the storage means based on the shipping plan to determine the pallet to be used for transportation and the type and quantity of parts to be loaded on it; shipping instruction means that instructs the pallet and part type and quantity determined thereby; storage means that stores attribute data for the shipping location and attribute data for the transportation means; and shipping instruction optimization means that optimizes shipping instructions for each transportation.

[0006] The stacking device in Patent Document 4 is equipped with a stacking specification input device for inputting the degree of importance to be attached to each evaluation element, such as the number of items loaded on a pallet, packing density of the workpieces, overall stability, and slippage between the workpieces, as specifications for the packing style; a packing style specification conformance sorting list that stores candidate packing styles in the order that conforms to the specifications; and a conformance order display device that displays this list.It also identifies packing styles that cannot be handled within the order based on the workability of the machine in charge of the work. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Publication No. 9-190483 [Patent Document 2] Japanese Patent Application Publication No. 58-6841 [Patent Document 3] Japanese Patent Application Publication No. 4-233003 [Patent Document 4] Japanese Patent Application Publication No. 6-305567 Summary of the Invention [Problem to be solved by the invention]

[0008] When presented with a load-loading operation, the crane operator must repeatedly execute similar, complex motions for each load that needs to be loaded. In most cases, these loads are placed close together on the ground and require nearly identical movements to be placed in the truck bed. However, the shapes of the loads and the free space in the bed vary widely, requiring the crane operator to visualize the appropriate load placement for each load.

[0009] The technology disclosed in the patent document assists workers who load cargo, but there is room for reducing the cognitive load and fatigue of crane workers.

[0010] The present invention aims to assist the work of placing a load on a loading platform during loading work of a loading truck crane. [Means for solving the problem]

[0011] A cargo placement assistance device according to a first aspect of the present invention comprises: an image acquisition unit that acquires an overhead image taken from the tip of a boom of a loading truck crane having a cab and a loading platform; an input unit that receives an input specifying a suspended load to be loaded in the overhead image; a depth acquisition unit that acquires the distance from the boom tip to the suspended load; an object recognition unit that recognizes the shape of the suspended load from the overhead image and the distance; an area recognition unit that recognizes an empty area of ​​the loading platform; a cargo placement generation unit that determines, as a recommended cargo placement, a placement in which, among placements in the empty area of ​​the loading platform where the center of gravity of the suspended load is closest to the cab, a value obtained by dividing the area of ​​the largest remaining area in one figure after placing the suspended load by the perimeter of the remaining area is the largest; a display unit that displays the recommended cargo placement; Equipped with.

[0012] Preferably, When there is already loaded load on the loading platform, the cargo placement generation unit determines the recommended cargo placement to be the one of the possible cargo placements of the suspended load that has the greatest lateral distance between the center of gravity of the already loaded load and the center of gravity of the suspended load on the loading platform.

[0013] Preferably, When there are multiple suspended loads to be loaded, the cargo placement generation unit determines the recommended cargo placement to be the placement in which the center of gravity of the group of suspended loads is closest to the cab and the distance between the centers of gravity of the suspended loads is the greatest.

[0014] A loading truck crane according to a second aspect of the present invention comprises: a vehicle having a cab and a bed; a crane mounted on the vehicle; A cargo placement assistance device according to a first aspect; Equipped with.

[0015] A program according to a third aspect of the present invention includes: an image acquisition unit that acquires an overhead image taken from the tip of a boom of a loading truck crane having a cab and a loading platform; an input unit that receives an input specifying a suspended load to be loaded in the overhead image; a depth acquisition unit that acquires the distance from the boom tip to the suspended load; an object recognition unit that recognizes the shape of the suspended load from the overhead image and the distance; an area recognition unit that recognizes an empty area of ​​the loading platform; a cargo placement generation unit that determines, as a recommended cargo placement, a placement in which, among placements in the empty area of ​​the loading platform where the center of gravity of the suspended load is closest to the cab, a value obtained by dividing the area of ​​the largest remaining area in one figure after placing the suspended load by the perimeter of the remaining area is the largest; and a display unit that displays the recommended cargo placement; Function as. [Effects of the Invention]

[0016] According to the present invention, the loading arrangement of the loading platform can be generated simply by specifying the suspended load to be loaded onto the loading truck crane, thereby assisting the work of arranging the load on the loading platform during loading operations of the loading truck crane. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a block diagram showing a functional configuration of a cargo placement assistance device according to an embodiment of the present invention. [Figure 2] FIG. 1 is an external perspective view of a loading truck crane according to an embodiment. [Figure 3] Plan view of loading platform and suspended load for loading arrangement example 1 [Figure 4] Plan views showing the process of generating a recommended cargo layout for cargo layout example 1. (A) shows the initial layout of the suspended load, (B) shows the layout with the suspended load moved away from the existing load, and (C) shows the layout after adjustment. [Figure 5] Plan view showing recommended loading arrangement for loading arrangement example 1 [Figure 6] Plan view of loading platform and suspended load in loading arrangement example 2 [Figure 7] Plan view showing the oriented bounding rectangle of the suspended load in Load Arrangement Example 2 [Figure 8] Plan view showing an example of the initial arrangement of two suspended loads in cargo arrangement example 2 [Figure 9] Plan view showing an example of local optimization of two suspended loads in cargo arrangement example 2 [Figure 10] Plan view showing recommended loading arrangement for loading arrangement example 2 [Figure 11] 1 is a flowchart showing an example of an operation for generating a recommended cargo placement according to an embodiment. [Figure 12] FIG. 1 is a block diagram showing an example of a hardware configuration of a cargo placement assistance device according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] The present disclosure relates to a load placement assistance device that generates a recommended load placement for placing a suspended load on the loading platform of a truck crane. Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the drawings, the same or equivalent parts are designated by the same reference numerals.

[0019] Embodiment Figure 1 is a block diagram showing the functional configuration of a load placement assistance device according to an embodiment of the present invention. The load placement assistance device 1 acquires image data from an imaging device 2 installed at the tip of a crane's boom, recognizes features and the load to be loaded from the image data, and generates a recommended load placement for placing the load in an available space on the loading platform. Hereinafter, the load placement assistance device 1 may be abbreviated as placement assistance device 1. A crane operator or a crane control device can place the suspended load in accordance with the recommended load placement generated by the placement assistance device 1.

[0020] 2 is an external perspective view of a loaded truck crane according to an embodiment. The positioning assistance device 1 is provided on the loaded truck crane. The loaded truck crane has a crane 40 mounted near the cab 31 of a vehicle 30 having a loading platform 32. The crane 40 has outriggers 41, an extendable boom 42, a boom hoisting cylinder 43 that raises and lowers the boom 42, and a hook 44 suspended by a wire rope hung on a pulley provided at the tip 45.

[0021] When the crane 40 is lifting a load, the outriggers 41 are extended to the left and right, and are extended downward and placed on the ground to support the crane. When the vehicle 30 is traveling, the outriggers 41 are retracted and stored near the vehicle body. The boom 42 and the hoisting cylinder 43 are supported so that they can rotate about a vertical axis while maintaining their relative positions. The hook 44 can be hoisted and lowered by winding up and letting out the wire rope suspending the hook 44 using a hoisting device (not shown).

[0022] An imaging device 2 that captures images of the vertically downward direction is installed at the tip 45 of the boom 42. The imaging device 2 is supported, for example, by a gimbal so that it always faces vertically downward. The boom 42 of the crane 40 is raised and extended, and an overhead image is obtained by capturing an image of the downward direction with the imaging device 2 at the tip 45. The placement assistance device 1 recognizes the features and the load to be loaded from the overhead image captured by the imaging device 2. The overhead image used to recognize the features and the load to be loaded is not limited to one. In order to cover the entire area where the load loading work will be performed, the imaging device 2 may be positioned differently to capture multiple overhead images.

[0023] 1, the placement assistance device 1 includes an image acquisition unit 10, a depth acquisition unit 11, an area recognition unit 12, an input unit 13, an object recognition unit 14, a cargo placement generation unit 15, and a display unit 16. Connected to the placement assistance device 1 are an operation unit 3 that allows the worker to input specifications for the suspended load, and a display device 4 that displays an overhead image captured by the imaging device 2, recommended cargo placement, and the like.

[0024] The image acquisition unit 10 acquires images captured by the imaging device 2. The imaging device 2 is, for example, a stereo camera. Since the imaging device 2 is installed at the tip 45 of the boom 42 of the crane 40, the image acquisition unit 10 can calculate the imaging position based on the ground contact position of the outriggers 41 from the rotation angle, tilt angle, and length of the boom 42 and the extended length of the outriggers 41. The image acquisition unit 10 sends the acquired image data and imaging position to the depth acquisition unit 11.

[0025] The depth acquisition unit 11 measures the depth of each point on the feature from the imaging position based on the parallax of each point from image data, such as a stereo image. The imaging device 2 is not limited to a stereo camera. For example, it may be a lidar, laser scanner, or monocular camera. In the case of a monocular camera, the actual distance to each point in the image can be measured based on the position of the lens when the focus is set on that point. With a monocular camera, the distance to the subject can be measured based on the accuracy of the camera's depth of field and focusing error. Alternatively, the boom 42 can be rotated to measure the depth based on the parallax of corresponding points using images captured from two viewpoints at different positions.

[0026] The area recognition unit 12 recognizes features from the overhead image and the depth of each point in the image. The area recognition unit 12 calculates the coordinates of each point relative to the loaded truck crane from the captured position and the direction and depth of each point in the image, and generates a 3D map of the work site. The area recognition unit 12 converts the three-dimensional information expressed in the camera coordinate system into three-dimensional information expressed in the reference coordinate system of the loaded truck crane using the captured image position.

[0027] The area recognition unit 12 recognizes features, for example, as follows: First, the area recognition unit 12 acquires point cloud data for one frame. The point cloud data is the coordinates of each point in an area including the load L, the feature C, and the ground surface F, which are the measurement targets, from above the load L and the feature C.

[0028] The area recognition unit 12 divides the area of ​​the overhead image into a plurality of small areas S in a grid pattern. In each small area S, the area recognition unit 12 extracts point data p with the largest depth (distance from the imaging position) h. The point data p with the maximum depth hmax is estimated to be at the lowest position in that small area S. The area recognition unit 12 then calculates the distance D of the depth h of other point data from the point data p with the maximum depth hmax. Using the maximum depth hmax as a reference, the area recognition unit 12 extracts point data p whose distance D of the depth h is within a predetermined threshold r1, for example, within 7 cm, as point data constituting the ground surface F.

[0029] Next, the area recognition unit 12 estimates the reference height HS of the ground surface F in each small area S based on the depth h of the extracted point data p in each small area S. In this embodiment, the area recognition unit 12 sets the average value of the depth h of the extracted point data p as the reference height HS of the ground surface F in the small area S. With this configuration, the area recognition unit 12 can estimate the reference height HS of the ground surface F in any small area S.

[0030] The area recognition unit 12 estimates the reference height H0 of the ground surface F of the entire area based on the reference height HS of the ground surface F in each small area S. In this embodiment, the area recognition unit 12 further averages the reference height HS (average value of depth h) of the ground surface F in each small area S over all small areas S, and sets the average value as the reference height H0 of the ground surface F of the entire area.

[0031] With this configuration, the area recognition unit 12 can estimate the reference height H0 of the ground surface F of the entire area of ​​the overhead image. Then, the area recognition unit 12 calculates the altitude H of the point data p from the depth h and the reference height H0. The altitude H is the height of the point data p from the reference height H0.

[0032] When the difference between the reference height HS of the ground surface F in one small area S and the reference height H0 of the ground surface F in the entire area is greater than a predetermined threshold, the area recognition unit 12 determines that point data p that does not constitute the ground surface F has been extracted, and may correct the reference height H0 of the ground surface F in the entire area by using, instead of the reference height HS of the ground surface F in the one small area S, the reference height HS of the ground surface F of a small area S adjacent to the one small area S, whose difference is less than the predetermined threshold.

[0033] With this configuration, when the area recognition unit 12 estimates that point data p that does not constitute the ground surface F has been extracted, it can more accurately estimate the reference height H0 of the ground surface F of the entire area by using, instead of one small area S, the reference height H0 of the ground surface F of a small area S adjacent to the one small area S, whose difference is less than a predetermined threshold.

[0034] When estimating the reference height H0 of the ground surface F over the entire region, it is possible to exclude, from all small regions S, small regions S that are estimated to have extracted point data p that do not constitute the ground surface F. For example, of the reference heights H0 of the ground surface F calculated in each small region S, the reference height H0 of the ground surface F in the lowest small region S may be used as a reference, and the average value may be calculated using only the reference heights H0 of the ground surface F of small regions S that are within a predetermined threshold. As described above, when estimating the reference height H0 of the ground surface F over the entire region, it is not necessary to use the reference heights H0 of all small regions S, and it is also possible to use only the reference height H0 of a specific small region S.

[0035] With this configuration, the area recognition unit 12 can exclude small areas S from which it is estimated that point data constituting the ground surface F has not been extracted, thereby enabling accurate estimation of the reference height H0 for the entire area.

[0036] The area recognition unit 12 may estimate the ground surface F using a specific position in the overhead image as a reference. In the placement assistance device 1, the reference ground surface F can be determined by specifying the position of the ground surface on the operation unit 3 in the overhead image displayed on the display device 4. The area recognition unit 12 may be configured to automatically determine and specify a specific position in the overhead image, for example, the position around the outriggers 41 or the position around the loading platform 32.

[0037] In the manual case, the operator specifies a position that is clearly the ground surface in the overhead image displayed on the display device 4 using the operation unit 3. Then, the area recognition unit 12 generates a reference circle of a predetermined radius centered on the specified position (point). Then, the area recognition unit 12 selects a plurality of point data p included in the reference circle.

[0038] The area recognition unit 12 first extracts the point data p with the largest depth h (maximum depth hmax) from the selected multiple point data p. Then, the area recognition unit 12 calculates the distance D of the depth h of the other point data from the point data p with the maximum depth hmax. Using the maximum depth hmax as a reference, the area recognition unit 12 extracts point data p whose distance D of the depth h is within a predetermined threshold r1, for example, 7 cm, as point data constituting the ground surface F. The area recognition unit 12 estimates the reference height H0 of the ground surface F based on the depth h of the extracted point data p. The area recognition unit 12 adopts the average value of the depth h of the extracted point data p as the reference height H0 of the ground surface F.

[0039] The next step in data processing by the area recognition unit 12 is plane estimation processing. The area recognition unit 12 estimates the top surfaces of the suspended load L and feature C, which are measurement objects present throughout the entire area, using the top surface estimation method described below.

[0040] The area recognition unit 12 divides the point cloud data P acquired from the overhead image into layers with a predetermined thickness d in the height direction, and allocates the point cloud data P to multiple layers. At this time, the area recognition unit 12 assigns an individual layer ID to each divided layer, and associates each point data p with the layer ID.

[0041] The area recognition unit 12 estimates a plane for each layer using the multiple point data p included in that layer. The "plane" here refers to a plane that faces upward in the load L and the feature C, i.e., the upper surface of the load L and the feature C.

[0042] The area recognition unit 12 first selects two pieces of point data pi and pj from the plurality of pieces of point data p1, p2, ... included in the same layer. The area recognition unit 12 calculates the distance L1 between the selected two pieces of point data pi and pj.

[0043] Next, if the distance L1 is equal to or less than a predetermined threshold value r2, the area recognition unit 12 determines that the two points pi and pj are on the same plane. The threshold value r2 is equal to or less than twice the resolution of the overhead image of the point cloud data P. In other words, if the two points pi and pj are at a distance that allows them to be considered adjacent based on the resolution, they are determined to be on the same plane. If the distance L1 is greater than the threshold value r2, new two points are selected and the distance L1 is calculated.

[0044] The area recognition unit 12 searches for points included in the same layer whose distance from either of the points pi and pj determined to be on the same plane is equal to or less than a threshold value r2. If the area recognition unit 12 finds point data pk that is a neighboring point, it considers the neighboring point data pk to be on the same plane as the two previously selected point data pi and pj. The area recognition unit 12 searches for points included in the same layer whose distance from either of the points included in the set {pi} determined to be on the same plane is equal to or less than a threshold value r2, and successively adds the neighboring points as points on the same plane.

[0045] When there are no points in the same layer that are closer to any of the points in the set of points determined to be in the same plane than the threshold r2, the area recognition unit 12 searches for two points that were not in the set of points determined to be in the same plane but whose distance is less than the threshold r2, and extracts a set of points that are in the same plane from different points. This operation is repeated to extract a group of clusters, which are sets of points that can be considered to be in the same plane but different from each other, from the points in the same layer. The area recognition unit 12 regards one cluster as one plane.

[0046] The area recognition unit 12 divides the point cloud data P into point data p that are considered to be on the same plane, and sets up planar clusters. The upper surfaces of the load L and the feature C can be defined by each point data p that belongs to a planar cluster. A layer assigned the same layer ID may have multiple planar clusters. For each cluster, the area recognition unit 12 calculates the center of gravity G of the set of points included in the cluster.

[0047] One cluster can be estimated as the top surface of a feature, but the top surface of one feature is not necessarily one cluster. For example, the top surface of an object with a spherical top surface is made up of multiple annular clusters that each belong to a different layer. A group of clusters whose centers of gravity, projected onto a plane, are approximately the same may be determined to be the top surface of a single feature. Also, the top surface of a cylindrical object placed with its central axis horizontal is made up of parallel band-like clusters on each layer. In this case, a group of clusters belonging to different layers whose centers, projected onto a plane, are approximately the same may be determined to be the top surface of a single feature.

[0048] Next, the area recognition unit 12 combines the estimated planar clusters (upper surfaces). The area recognition unit 12 selects two of the estimated planar clusters that have different floor IDs assigned, and calculates the difference dH in the altitude H of each planar cluster. The altitude H of a planar cluster is the average value of the height from the reference altitude H0 of each point data p belonging to the planar cluster.

[0049] The area recognition unit 12 searches for a combination of plane clusters where the difference dH is within a threshold r3. When the area recognition unit 12 detects a combination of plane clusters where the difference dH in altitude H is within the threshold r3, it detects the overlap dW in the horizontal direction for those plane clusters. "Overlap" here refers to the degree of overlap and separation in the horizontal direction of the planes defined by the plane clusters, and "overlap" is detected when an overlap amount dW1 is detected in either of two orthogonal directions on the horizontal plane (dW1>0), or when the separation amount dW2 is equal to or less than a predetermined threshold r4 (0≦dW2≦r4).

[0050] If an overlap is detected, the area recognition unit 12 considers that the point data p belonging to the planar clusters exist on the same plane, combines the two planar clusters, and updates them as a new planar cluster. At this time, the area recognition unit 12 also calculates a new altitude H from each point data p belonging to the new planar cluster.

[0051] The area recognition unit 12 repeats the above process until there are no more combinations of plane clusters that satisfy the conditions, and estimates planes that exist across multiple layers. The area recognition unit 12 outputs the planes (i.e., plane clusters) combined by the above combining process. The planes defined by the plane clusters are planes that face upward in the load L and the feature C, i.e., the top surfaces of the load L and the feature C.

[0052] The above-described plane estimation method can estimate a plane without using the normal vector of the point cloud data P. This has the advantage of requiring less calculations than when estimating a plane using the normal vector of the point cloud data P. In this type of plane estimation method, by estimating the top surface of the load L or the feature C, the three-dimensional shape of the load L or the feature C can be grasped without acquiring point data p of the side surface of the load L or the feature C.

[0053] Next, the area recognition unit 12 classifies the point cloud data P into clusters of the same area. The same area clustering process is a process of clustering the generated planar clusters from a different perspective of whether or not they exist in the same area.

[0054] The area recognition unit 12 extracts a plane cluster including point data p whose altitude H is a maximum value Hh, and plane clusters that are not connected to this plane cluster. If the difference ΔH in altitude H between the two extracted plane clusters is equal to or less than a predetermined threshold, the area recognition unit 12 checks whether the two plane clusters overlap in the height direction.

[0055] When two plane clusters overlap in the height direction, the area recognition unit 12 regards these plane clusters as being in the "same area" and forms an same-area cluster from these plane clusters.

[0056] The area recognition unit 12 further searches for a planar cluster containing point data p having a maximum value Hh of altitude H and planar clusters that are not connected to this planar cluster, and if an unconnected planar cluster is extracted, it makes a judgment based on the difference ΔH and checks for overlap in the height direction, and if there is a planar cluster that meets the above conditions, it adds it to the same area cluster.

[0057] The area recognition unit 12 repeats this process until no unconnected planar clusters are found for the planar clusters including the point data p having the maximum value Hh of the altitude H. The area recognition unit 12 forms the same area clusters by the above process.

[0058] The area recognition unit 12 treats the point data p belonging to the same area cluster formed in this way as a single unit in shape, and sets a guide frame so as to surround the same area cluster.

[0059] It is preferable that such clustering processing of the same area is hierarchical clustering using a tree structure based on altitude. The area recognition unit 12 creates a tree structure for each feature C using altitude H through the clustering processing of the same area.

[0060] In hierarchical clustering using a tree structure based on altitude, the area recognition unit 12 sets the plane cluster with the smallest average value of altitude H as the root. If there is a plane cluster that overlaps the plane cluster that constitutes the root in the height direction, the area recognition unit 12 extends a branch from the root and adds the overlapping plane cluster to the end of the branch. The area recognition unit 12 sets the plane cluster with the largest average value of altitude H as a child.

[0061] The area recognition unit 12 also recognizes the load loaded in the area of ​​the loading platform 32. The area of ​​the loading platform 32 in the reference coordinate system is stored in advance. Since the range of the loading platform 32 is known, it is possible to define the area of ​​the loading platform 32 in the overhead image. Since the position of the floor of the loading platform 32 is known, it is possible to extract the floor of the loading platform from the area of ​​the loading platform 32 in the overhead image. The area recognition unit 12 can recognize the tree structure in the area of ​​the loading platform from the generated tree structure as the loaded load. The area excluding the part of the loaded load is the free area.

[0062] The area recognition unit 12 generates an outline that encloses all the planar clusters included in the hierarchical cluster of one tree structure as a guide frame. After classifying all the planar clusters in the entire area into one of the tree structures, the area recognition unit 12 sends these tree structures to the object recognition unit 14 and the cargo placement generation unit 15.

[0063] The input unit 13 accepts an input specifying a suspended load that is a loading target in the overhead image. The input unit 13 displays the overhead image on the display device 4 and sends the position of the image input from the operation unit 3 to the target recognition unit 14. The input unit 13 may recognize a gesture made by a worker in the overhead image to indicate a suspended load, and send the position in the overhead image indicated by the worker to the target recognition unit 14. Alternatively, when the worker operates the crane 40, stops the hook 44 directly above a certain feature C, and inputs a load specification into the operation unit 3, the target recognition unit 14 may recognize the feature C directly below the hook 44 as a suspended load that is a loading target.

[0064] The object recognition unit 14 identifies the suspended load to be loaded from the tree structure corresponding to the specified position. That is, the feature C represented by the planar cluster corresponding to the specified position in the bird's-eye image is set as the suspended load L to be loaded. The object recognition unit 14 may identify the three-dimensional information (point cloud data) of the suspended load by object recognition from the three-dimensional information (point cloud data). The object recognition method for the suspended load may be appearance-based object recognition or model-based object recognition.

[0065] The cargo placement generation unit 15 generates a recommended cargo placement for loading the suspended load L within the free space of the loading platform 32 in the bird's-eye view image using the footprint of the suspended load L, i.e., the shape obtained by projecting the outermost shape of the planar cluster that makes up the suspended load onto a horizontal surface. When the suspended load is placed in the free space, the cargo placement generation unit 15 determines, among the placements in which the center of gravity of the suspended load is closest to the cab 31, the placement in which the value obtained by dividing the area of ​​the largest remaining area in a single figure after the suspended load is placed by the perimeter of the remaining area is the largest, as the recommended cargo placement. For convenience, the cargo placement generation unit 15 determines the geometric center of gravity of the footprint of the suspended load as the center of gravity of the suspended load. If the position of the center of gravity of the suspended load is given, the cargo placement generation unit 15 determines that position as the center of gravity.

[0066] Loading arrangement example 1. 3 is a plan view of the loading platform and the suspended load to be loaded in Load Arrangement Example 1. In Load Arrangement Example 1, a suspended load L is placed on the loading platform 32 on which an already loaded load EL is loaded. The positions of the centers of gravity of the already loaded load EL and the suspended load L are indicated by small circles. The load arrangement generation unit 15 overlays the footprint of the suspended load L on the empty area of ​​the loading platform 32 in the bird's-eye image, rotates and translates the footprint, and searches for an arrangement in which the center of gravity of the suspended load L is closest to the cab 31 and the value obtained by dividing the area of ​​the largest remaining area in a single figure after placing the suspended load L by the perimeter of the remaining area is the largest.

[0067] 4A and 4B are plan views showing the process of generating a recommended cargo layout for cargo layout example 1. Fig. 4A shows the initial layout of the suspended load. The cargo layout generation unit 15 first places the suspended load L so that the center of gravity of the suspended load L is close to the cab 31 and close to the existing load EL.

[0068] 4(B) shows an arrangement in which the suspended load is moved away from the already loaded load. The load arrangement generation unit 15 assumes that a virtual repulsive force that is proportional to the mass of each and inversely proportional to the square of the distance acts between the already loaded load EL and the suspended load L in the lateral direction of the loading platform 32, and moves the suspended load L away from the already loaded load EL.

[0069] Next, the cargo layout generation unit 15 rotates and translates the load L, and determines as the recommended cargo layout the layout in which the value obtained by dividing the area of ​​the largest remaining region in a single figure after placing the load L by the perimeter of the remaining region is the largest among the layouts in which the center of gravity of the load L is closest to the cab 31. Figure 4(C) shows the layout after adjustment.

[0070] FIG. 5 is a plan view showing the recommended cargo layout of cargo layout example 1. In the recommended cargo layout, the value obtained by dividing the area of ​​the largest remaining area RA in one figure by the perimeter of the remaining area RA is maximized. For example, the recommended cargo layout has a greater distance between the center of gravity of the suspended load L and the center of gravity of the loaded load EL than an arrangement in which the suspended load L is rotated clockwise, with the narrow part of the suspended load L moved closer to the side edge of the loading platform and the wide part moved closer to the loaded load EL. The cargo layout generation unit 15 determines the recommended cargo layout to be the one in which the center of gravity of the loaded load EL and the center of gravity of the suspended load L are farther apart in the lateral direction of the loading platform 32.

[0071] Loading arrangement example 2. FIG. 6 is a plan view of the loading platform and suspended loads in cargo placement example 2. In cargo placement example 2, two suspended loads L1 and L2 are placed on the loading platform 32 on which the loaded load EL is already loaded. In the example of FIG. 6, the footprints of the suspended loads L1 and L2 are the same. The cargo placement generation unit 15 assumes an arrangement combining the two suspended loads L1 and L2, and fits the arrangement of the two suspended loads L1 and L2 into the free space on the loading platform 32 to search for a recommended cargo placement. To generate the arrangement of the two suspended loads L1 and L2, an oriented bounding rectangle (OBB) is introduced for each of the suspended loads L1 and L2.

[0072] Figure 7 is a plan view showing the directed bounding rectangles of the loads in Load Arrangement Example 2. Consider the smallest rectangle that circumscribes the footprints of the loads L1 and L2. The perpendicular line drawn to the side of the rectangle farthest from the center of gravity COG of the loads L1 and L2 is defined as the tail TL, and the farthest side is defined as the end TE. The rectangle containing the center of gravity COG, tail TL, and end TE is defined as the directed bounding rectangle OBB. The directed bounding rectangles OBB of the loads L1 and L2 can be used to generate an arrangement of the two loads L1 and L2.

[0073] FIG. 8 is a plan view showing an example of the initial arrangement of two loads in cargo arrangement example 2. In FIG. 8, load L1 is represented by a solid line and load L2 by a dotted line, and their respective oriented bounding rectangles OBB are also shown by dotted lines. The cargo arrangement generation unit 15 generates an arrangement in which any side of the oriented bounding rectangles OBB of the loads L1 and L2 is parallel. FIG. 8 shows 24 examples of initial arrangements. The cargo arrangement generation unit 15 translates and rotates load L2 relative to load L1 from the initial arrangement, minimizes the bounding rectangle circumscribing the arrangement of the two loads, and generates a locally optimized arrangement for each initial arrangement.

[0074] Figure 9 is a plan view showing an example of a locally optimized arrangement of two suspended loads in Load Arrangement Example 2. In Figure 9, a pseudo-minimum bounding rectangle, which is the minimized bounding rectangle circumscribing the arrangement of the two suspended loads L1 and L2, is shown by a solid line. The locally optimized arrangements in Figure 9 correspond to the initial arrangements in Figure 8, and the correspondence is indicated by the alphabets attached to the arrangements.

[0075] The cargo placement generation unit 15 applies the local optimization array to the empty space of the loading platform 32 and selects a local optimization array that fits the empty space. There may be two or more local optimization arrays selected. The cargo placement generation unit 15 selects the local optimization array that has the largest value obtained by dividing the area of ​​the largest remaining area in a single figure after placing the loads L1 and L2 by the perimeter of the remaining area, from among the arrays in which the center of gravity of the set of loads L1 and L2 is closest to the cab 31. In the example of Figure 9, the local optimization array G is selected. The cargo placement generation unit 15 adjusts the positions of the loads L1 and L2 in the local optimization array G that has been applied to the empty space and determines the array with the largest distance between the centers of gravity COG as the recommended cargo placement.

[0076] FIG. 10 is a plan view showing the recommended cargo layout of cargo layout example 2. The recommended cargo layout after the cargo layout generation unit 15 adjusts the positions of the loads L1 and L2 is the one in which the area of ​​the largest remaining area RA in a single figure after the loads L1 and L2 are positioned is divided by the perimeter of the remaining area RA among the layouts in which the center of gravity of the collective loads L1 and L2 is closest to the cab, and this value is the largest. Furthermore, the distance between the centers of gravity COG of the loads L1 and L2 is the largest. The cargo layout generation unit 15 sends the generated recommended cargo layout to the display unit 16. The display unit 16 displays the recommended cargo layout on the display device 4 by superimposing it on an image of the loading platform.

[0077] The above cargo layout examples show cases with one or two suspended loads. The number of suspended loads to be loaded is not limited to two or less. Even when there are three or more suspended loads to be loaded, as in cargo layout example 2, the cargo layout generation unit 15 generates a locally optimized array by minimizing each of the initial load layout examples, and applies these to the free space. From among the arrays in which the center of gravity of the group of suspended loads is closest to the cab 31, the cargo layout generation unit 15 selects the locally optimized array in which the value obtained by dividing the area of ​​the largest remaining region in a single figure after the suspended loads are placed by the perimeter of the remaining region is the largest. The positions of the suspended loads in the applied locally optimized array are then adjusted, and the array with the greatest distance between the centers of gravity COG is determined as the recommended cargo layout.

[0078] 11 is a flowchart showing an example of the operation of generating a recommended cargo placement according to the embodiment. The process of generating a recommended cargo placement is initiated by an operation by a crane operator. When the cargo placement assistance device 1 is initiated, the image acquisition unit 10 acquires an overhead image captured from the tip 45 of the boom 42 (step S10). The depth acquisition unit 11 acquires the depth of each point on the feature from the imaging position (step S11). The area recognition unit 12 recognizes features, including suspended loads, from the overhead image and the depth of each point in the image (step S12), and recognizes empty areas from the loads loaded in the area of ​​the loading platform 32 (step S13).

[0079] The input unit 13 receives an input specifying a load L (or L1, L2) that is the loading target in the overhead image (step S14). When the load L (or L1, L2) is specified, the object recognition unit 14 recognizes the position and shape of the load L (or L1, L2) (step S15). When multiple loads L1, L2 are specified, the load arrangement generation unit 15 generates an initial arrangement of the multiple loads L1, L2 (step S16) and minimizes the initial arrangement to generate a locally optimized arrangement (step S17).

[0080] The cargo placement generation unit 15 selects a locally optimized arrangement that fits the available space (step S18), and selects from among these locally optimized arrangements the one that has the largest area / perimeter of the remaining area RA in one figure after the loads L1 and L2 are placed (step S19).Furthermore, the cargo placement generation unit 15 adjusts the positions of the loads L1 and L2, and determines as the recommended cargo placement the arrangement in which the center of gravity of the set of loads L1 and L2 is closest to the cab and the distance between the centers of gravity of the loads L1 and L2 is the largest (step S20).

[0081] If there is one suspended load L, in steps S16 to S20, the recommended cargo placement is determined to be the placement with the largest area / perimeter of the largest remaining area RA in one figure after placing the suspended load L. The cargo placement assistance device 1 displays the generated recommended cargo placement on the display device 4 (step S21), and the placement assistance device 1 ends the process of generating the recommended cargo placement.

[0082] As described above, according to the cargo placement assistance device 1 of the embodiment, if a crane operator specifies a suspended load L or multiple suspended loads L1, L2, a recommended cargo placement is generated for placing the suspended load L (or L1, L2) in an available area on the loading platform 32, thereby assisting in the task of placing cargo during loading operations of a loaded truck crane.

[0083] Fig. 12 is a block diagram showing an example of the hardware configuration of a cargo placement assistance device according to an embodiment. As shown in Fig. 12, the placement assistance device 1 includes a control unit 51, a main memory unit 52, an external memory unit 53, an operation unit 54, a display unit 55, an input / output unit 56, and a transmission / reception unit 57. The main memory unit 52, the external memory unit 53, the operation unit 54, the display unit 55, the input / output unit 56, and the transmission / reception unit 57 are all connected to the control unit 51 via an internal bus 50.

[0084] The control unit 51 is composed of a CPU (Central Processing Unit) and the like, and executes each process of the image acquisition unit 10, depth acquisition unit 11, area recognition unit 12, input unit 13, target recognition unit 14, cargo placement generation unit 15, and display unit 16 of the placement assistance device 1 in accordance with a control program 58 stored in the external memory unit 53.

[0085] The main memory unit 52 is composed of RAM (Random-Access Memory) and the like, and is used as a work area for the control unit 51 by loading a control program 58 stored in the external memory unit 53 .

[0086] The external memory unit 53 is composed of non-volatile memory such as flash memory, a hard disk, DVD-RAM (Digital Versatile Disc Random-Access Memory), DVD-RW (Digital Versatile Disc ReWritable), etc., and stores a program for causing the control unit 51 to perform processing of the placement assistance device 1, as well as various data such as the position of the loading platform 32 relative to the center of rotation of the boom 42.In addition, in accordance with instructions from the control unit 51, the external memory unit 53 supplies the data stored by this program to the control unit 51, and stores data such as the shape of the load L, the locally optimized arrangement of the loads L1 and L2, and recommended cargo placement supplied from the control unit 51.

[0087] The operation unit 54 is composed of a keyboard, a pointing device such as a mouse, and an interface device that connects the keyboard and pointing device to the internal bus 50. Specification of the load L, etc. is input via the operation unit 54 and supplied to the control unit 51. The operation unit 54 includes the operation unit 3 in FIG. 1.

[0088] The display unit 55 is composed of a display device such as an LCD (Liquid Crystal Display) or an organic EL display and an interface device that connects the display device to the internal bus 50, and displays an overhead image, the specified suspended load L, recommended cargo placement, etc. The display unit 55 includes the display device 4 of FIG.

[0089] The input / output unit 56 is configured with a serial interface or a parallel interface. The imaging device 2 and the crane control device are connected to the input / output unit 56, and the control unit 51 acquires the overhead image via the input / output unit 56.

[0090] The transmitter / receiver 57 is composed of a wireless communication device and a serial interface or a LAN (Local Area Network) interface. The transmitter / receiver 57 communicates via wireless communication with a mobile terminal into which, for example, the selection of the load L is input. The transmitter / receiver 57 also updates various data and downloads control programs.

[0091] The processing of the image acquisition unit 10, depth acquisition unit 11, area recognition unit 12, input unit 13, target recognition unit 14, cargo placement generation unit 15, and display unit 16 of the placement assistance device 1 shown in Figure 1 is performed by the control program 58 using the control unit 51, main memory unit 52, external memory unit 53, operation unit 54, display unit 55, input / output unit 56, and transmission / reception unit 57, etc. as resources.

[0092] The configuration of the placement assistance device 1 described in each embodiment is an example, and can be changed and modified as desired. The configuration of the placement assistance device 1 is not limited to or all of the configurations shown in the embodiments. For example, a tablet terminal may be used as the placement assistance device 1. Furthermore, the placement assistance device 1 may be installed on a network, and the functions of the placement assistance device 1 may be provided via the network.

[0093] Furthermore, the above hardware configuration and flowchart are merely examples and can be changed and modified as desired.

[0094] The core part of the placement assistance device 1 that performs the recommended cargo placement generation process, which is composed of the image acquisition unit 10, depth acquisition unit 11, area recognition unit 12, input unit 13, target recognition unit 14, cargo placement generation unit 15, and display unit 16, etc., can be realized using an ordinary computer system rather than a dedicated system. For example, the placement assistance device 1 that performs the above process may be configured by storing and distributing a computer program for performing the above operations on a computer-readable recording medium (USB memory, CD-ROM, DVD-ROM, etc.), and installing the computer program on a computer. Alternatively, the placement assistance device 1 may be configured by storing the computer program in a storage device of a server device on a communication network such as the Internet, and downloading it to an ordinary computer system.

[0095] Furthermore, when the placement assistance device 1 is realized by sharing the work between an OS (operating system) and an application program, or by cooperation between an OS and an application program, only the application program portion may be stored in a recording medium or storage device.

[0096] It is also possible to superimpose a computer program on a carrier wave and distribute it over a communication network. For example, the computer program may be posted on a bulletin board system (BBS) on the communication network and distributed over the network. The computer program may then be started and executed under the control of an OS in the same way as other application programs, thereby enabling the above-mentioned processing to be performed. [Explanation of symbols]

[0097] 1. Load placement assistance device 2. Imaging device 3 Control section 4 Display device 10 Image acquisition unit 11 Depth acquisition section 12 Area recognition part 13 Input section 14 Object Recognition Unit 15 Load placement generation unit 16 Display 30 vehicles 31 Cab 32 Cargo bed 40 Crane 41 Outrigger 42 Boom 43 Elevating Cylinder 44 Hook 45 Tip COG center of gravity EL Existing load L, L1, L2 Hanging load OBB directed bounding rectangle RA Remaining area TE end TL tail

Claims

1. an image acquisition unit that acquires an overhead image taken from the tip of a boom of a loading truck crane having a cab and a loading platform; an input unit that receives an input specifying a suspended load to be loaded in the overhead image; a depth acquisition unit that acquires the distance from the boom tip to the suspended load; an object recognition unit that recognizes the shape of the suspended load from the overhead image and the distance; an area recognition unit that recognizes an empty area of ​​the loading platform; a cargo placement generation unit that determines, as a recommended cargo placement, a placement in which, among placements in the empty area of ​​the loading platform where the center of gravity of the suspended load is closest to the cab, a value obtained by dividing the area of ​​the largest remaining area in one figure after placing the suspended load by the perimeter of the remaining area is the largest; a display unit that displays the recommended cargo placement; A cargo placement assistance device comprising:

2. 2. The cargo placement support device according to claim 1, wherein, when there is an existing load on the loading platform, the cargo placement generation unit determines, as the recommended cargo placement, the cargo placement that is the furthest lateral distance between the center of gravity of the existing load and the center of gravity of the suspended load from among the possible cargo placements of the suspended load.

3. 3. The cargo placement assistance device according to claim 1, wherein, when there are multiple suspended loads to be loaded, the cargo placement generation unit determines, as the recommended cargo placement, the placement in which the centers of gravity of the group of suspended loads are closest to the cab and the distance between the centers of gravity of the suspended loads is the greatest.

4. a vehicle having a cab and a bed; a crane mounted on the vehicle; The cargo placement assistance device according to claim 1 or 2; A loaded truck crane equipped with:

5. a vehicle having a cab and a bed; a crane mounted on the vehicle; The cargo placement assistance device according to claim 3; A loaded truck crane equipped with:

6. Computer, an image acquisition unit that acquires an overhead image taken from the tip of a boom of a loading truck crane having a cab and a loading platform; an input unit that receives an input specifying a suspended load to be loaded in the overhead image; a depth acquisition unit that acquires the distance from the boom tip to the suspended load; an object recognition unit that recognizes the shape of the suspended load from the overhead image and the distance; an area recognition unit that recognizes an empty area of ​​the loading platform; a cargo placement generation unit that determines, as a recommended cargo placement, a placement in which, among placements in the empty area of ​​the loading platform where the center of gravity of the suspended load is closest to the cab, a value obtained by dividing the area of ​​the largest remaining area in one figure after placing the suspended load by the perimeter of the remaining area is the largest; and a display unit that displays the recommended cargo placement; A program that functions as a

Images