Apparatus and method for calculating object attitude

Abstract

To enable a forklift transport system in a physical distribution site to smoothly carry out transport work with a cargo-handling appliance, such as a forklift, by making correct the accuracy of calculating an attitude of a pallet to load a burden.SOLUTION: An apparatus for calculating an object attitude comprises an image-apex recognizing section that allows a two-dimensional camera-based two-dimensional image and a three-dimensional camera-based three-dimensional image to be inputted to recognize a plurality of apexes of an object concerned from the two-dimensional image, a three-dimensional reference point determining section that determines a three-dimensional coordinate of a three-dimensional reference point based on coordinates of the apexes recognized by the image-apex recognizing section, and an object-attitude calculating section that calculates an angle of one side face of the object concerned with respect to a predetermined axis relative to a predetermined coordinate system. The object-attitude calculating section calculates an angle of the object concerned based on the coordinates of the apexes in the three-dimensional coordinate, a restriction condition of the three-dimensional reference point in the three-dimensional coordinate, and a restriction condition of the three-dimensional coordinate corresponding to a two-dimensional image nature-based two-dimensional coordinate of the apexes.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an object posture calculation device and an object posture calculation method, and particularly relates to an object posture calculation device and an object posture calculation method suitable for accurately calculating the posture of a pallet on which a load is mounted and smoothly performing a conveyance operation by a cargo handling device such as a forklift in a logistics operation in a warehouse or the like.

Background Art

[0002] In recent logistics styles, it is common to perform a cargo transportation operation using a cargo handling platform called a pallet on which a load is placed and a forklift. The pallet is provided with insertion ports (fork pockets) on both sides of the surface on which the cargo is placed. By inserting the teeth (forks) of the forklift into these ports, vertical movement and conveyance can be easily performed, enabling efficient work.

[0003] In view of such a logistics work style, technologies have been proposed for recognizing a pallet by image recognition, grasping its position, and performing transportation by an unmanned automatic forklift or assisting the driver of a manned forklift.

[0004] For recognizing the position and posture of a pallet by using a 2D image and a 3D point cloud from a depth camera or the like for image recognition of the pallet. In such recognition of the position and posture of an object, generally, recognition is performed using two or more depths (for example, the left vertex and the right vertex) on a distant object. However, in the actual practice of logistics work, there is a problem that the measurement quality of the 3D point cloud is low due to the on-site environment and the state of the detection object, and it is necessary to recognize the position and posture of the target object using a small number of point clouds with high measurement quality.

[0005] Regarding an algorithm for obtaining the position and posture of an object using 3D reference points, for example, it is described in Non-Patent Document 1. In this Non-Patent Document 1, a mathematical formula representing the positional relationship between 3D reference points and their images in a 2D plane is described (§2 THE CAMERA POSE FROM THREE POINTS REVISITED, Fig 1).

[0006] Also, in relation to this, Patent Document 1 discloses a three-dimensional measurement and display device that calculates the position and orientation of an object based on a depth image acquired by a depth camera and a planar region extracted from the depth image.

Prior Art Documents

Patent Documents

[0007]

Patent Document 1

Non-Patent Documents

[0008]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0009] Hereinafter, the problems to be solved by the invention will be described with reference to FIG. 13.

[0010] FIG. 13 is a diagram for explaining the perspective view and specifications of a pallet. As described above, due to the problems in the measurement of current depth cameras, the measurement quality of the 3D point cloud is low. Therefore, it is considered to recognize the position and orientation of the target pallet using a small number of point clouds with high measurement quality.

[0011] Therefore, it is decided to reduce the necessary 3D point cloud and detect the position and orientation of the pallet using the 3D 1 point + 2D 1 point on the side surface of the detectable pallet and the positional relationship between these two points.

[0012] Here, the detection of the position and orientation of the pallet specifically means obtaining the inclination of the camera and the pallet related to the measurement (the specific image in space will be described later).

[0013] As shown in FIG. 13, the pallet 10 has a pallet side surface 12 with a height h and a width s, and fork pockets 11 are provided for inserting the forks of two forklifts and transporting. A rectangular area 13 between the left and right fork pockets 11l and 11r is formed. There are various standards for pallets. For example, the T11 type pallet defined by JIS has s = 1100 mm and h = 144 mm.

[0014] When detecting the position and orientation of the pallet, a bounding box is cut out by image recognition, and the four 2D recognized vertices of its side surface are a 11 , a 12 , a 21 , a 22 as shown in FIG. 13. The coordinate system is assumed to be a three-dimensional orthogonal coordinate system such as (X, Y, Z) as shown in the figure.

[0015] Here, the position and orientation of the pallet are to be detected by the 3D points corresponding to the a 22 of the upper right 2D point and the a 21 of the upper left. However, actually, for example, as shown in FIG. 13, an error in image recognition occurs, and the coordinates of a 11 are A 11 , the coordinates of a 21 are A 21 , and a problem occurs in that the error in the position and orientation of the pallet becomes large due to the 3D point corresponding to the coordinate A 21 of the 2D point including the error in image recognition.

[0016] An object of the present invention is to provide an object posture calculation device and an object posture calculation method that can accurately improve the accuracy of calculating the posture of a pallet on which a load is mounted and smoothly perform a transportation operation by a cargo handling device such as a forklift.

Means for Solving the Problem

[0017] The configuration of the object posture calculation device of the present invention is preferably an object posture calculation device that inputs a two-dimensional image captured by a two-dimensional camera and a three-dimensional image captured by a three-dimensional camera, and calculates the angle of a target object with respect to a predetermined coordinate system, including an image vertex recognition unit that recognizes a plurality of vertices of the target object from the two-dimensional image, a three-dimensional reference point determination unit that determines the three-dimensional coordinates of the three-dimensional reference point based on the coordinates of the vertices recognized by the image vertex recognition unit, and an object posture calculation unit that calculates the angle of one side surface of the target object with respect to a predetermined axis with respect to a predetermined coordinate system. The object posture calculation unit calculates the angle based on the coordinates of the vertices in the three-dimensional coordinates, the constraint conditions of the three-dimensional reference point in the three-dimensional coordinates, and the constraint conditions of the three-dimensional coordinates corresponding to the two-dimensional coordinates of the vertices based on the properties of the two-dimensional image.

Effect of the Invention

[0018] According to the present invention, it is possible to provide an object posture calculation device and an object posture calculation method that can accurately improve the accuracy of calculating the posture of a pallet on which a load is mounted and smoothly perform a conveying operation by a cargo handling device such as a forklift.

Brief Description of the Drawings

[0019]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Mode for Carrying Out the Invention

[0020] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The examples are for illustrative purposes to explain the present invention, and for the sake of clarity of explanation, appropriate omissions and simplifications have been made. The present invention can also be implemented in various other forms. Unless otherwise limited, each component may be in a single or plural number.

[0021] The positions, sizes, shapes, ranges, etc. of the respective components shown in the drawings may not represent the actual positions, sizes, shapes, ranges, etc. for the purpose of facilitating the understanding of the invention. For this reason, the present invention is not necessarily limited to the positions, sizes, shapes, ranges, etc. disclosed in the drawings.

[0022] As examples of various types of information, it may be described using expressions such as "table", "list", "queue", etc., but the various types of information may be represented by data structures other than these. For example, various types of information such as "XX table", "XX list", "XX queue" may be referred to as "XX information". When explaining identification information, expressions such as "identification information", "identifier", "name", "ID", "number" are used, but these are interchangeable with each other.

[0023] When there are a plurality of components having the same or similar functions, they may be described by attaching different subscripts to the same reference numeral. Also, when it is not necessary to distinguish these plurality of components, the subscripts may be omitted in the description.

[0024] In the embodiments, the processing performed by executing a program may be described. Here, the computer executes the program by a processor (for example, a CPU, a GPU), and performs the processing defined by the program while using a storage resource (for example, a memory) and an interface device (for example, a communication port), etc. Therefore, the subject of the processing performed by executing the program may be the processor. Similarly, the subject of the processing performed by executing the program may be a controller, a device, a system, a computer, or a node having a processor. The subject of the processing performed by executing the program may be an arithmetic unit, and may include a dedicated circuit for performing a specific processing. Here, the dedicated circuit is, for example, an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), a CPLD (Complex Programmable Logic Device), etc.

[0025] The program may be installed in a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server includes a processor and a storage resource for storing the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. Also, in the embodiment, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0026] Hereinafter, each embodiment according to the present invention will be described with reference to FIGS. 1 to 12.

[0027] 〔Embodiment 1〕 Hereinafter, Embodiment 1 according to the present invention will be described with reference to FIGS. 1 to 7.

[0028] First, the configuration of the forklift transportation system at the logistics site will be described with reference to FIGS. 1 and 2.

[0029] The forklift 20 is a device that inserts the fork 25 into the fork pocket 11 of the pallet 10 for transportation. The forklift 20 has an operation command unit 30 that gives instructions to the drive wheels 28 and the fork operation unit 26 to operate according to commands and operation programs from the outside.

[0030] Also, a 2D (Dimension) camera (RGB camera) 21 captures a two-dimensional image (2D point image), and a 3D (Dimension) camera (Depth camera) 22 captures a three-dimensional image (3D point image), which is then wirelessly transmitted to the access point 5 via the external interface unit 27. The 3D camera 22 is a camera that can refer to information regarding the depth of the captured image. The 3D camera 22 may be implemented by a stereo method that measures distance using two cameras, or a ToF (Time of Flight) method that measures the reflection time of light, or a structured illumination method that measures depth from a state where special light is irradiated.

[0031] Then, the images captured from the access point 5 are transmitted to the object pose calculation device 100.

[0032] Also, a radar sensor 40 measures surrounding objects and transmits the measured data to the operation command unit 30.

[0033] The object pose calculation device 100 is a device that obtains the pose (angle formed with the coordinate system) of the pallet 10 based on the captured images. The object pose calculation device 100 includes functional units such as an image vertex recognition unit 101, a 3D (Dimension) reference point determination unit 102, an object angle calculation unit 103, an image reception unit 104, an object angle transmission unit 105, and a storage unit 110.

[0034] The image vertex recognition unit 101 is a functional unit that performs image recognition from the 2D point image and recognizes the vertices of the side surface of the pallet 10. The 3D reference point determination unit 102 is a functional unit that determines the 3D reference points (details will be described later) used for object angle calculation from the points in the 3D point image. The object angle calculation unit 103 is a functional unit that calculates the angle of the pallet 10 from the 2D points and 3D points. The image reception unit 104 is a functional unit that receives the two-dimensional image (2D point image) captured by the 2D camera 21 and the three-dimensional image captured by the 3D camera 22. The object angle transmission unit 105 is a functional unit that transmits the calculated angle of the pallet 10.

[0035] The operation command unit 30 of the forklift 20 receives the angle of the pallet 10 via the access point 5 and the external interface unit 27, and generates appropriate pallet operation data.

[0036] The storage unit 110 is a functional unit that holds the data necessary for the object posture calculation device 100, and holds an image data DB 111 that stores 2D image data and 3D image data, a measurement data DB 112 that stores measurement data necessary for operation, and an object posture calculation data DB 113 that stores data for calculating the posture of an object.

[0037] Next, the hardware and software configuration of the object posture calculation device 100 will be described with reference to FIG. 2. The hardware configuration of the object posture calculation device 100 is realized by a general information processing device such as a personal computer shown in FIG. 2, for example.

[0038] The object posture calculation device 100 has a form in which a CPU (Central Processing Unit) 202, a main storage device 204, a network I / F (InterFace) 206, a display I / F 208, an input / output I / F 210, and an auxiliary storage I / F 212 are connected by a bus.

[0039] The CPU 202 controls each part of the object posture calculation device 100, and loads and executes a necessary program in the main storage device 204.

[0040] The main storage device 204 is usually composed of a volatile memory such as a RAM, and stores a program executed by the CPU 202 and data to be referenced.

[0041] The network I / F 206 is an interface for connecting to a network.

[0042] The display I / F 208 is an interface for connecting a display device 220 such as an LCD (Liquid Crystal Display).

[0043] The input / output I / F 210 is an interface for connecting an input / output device. In the example of FIG. 2, a keyboard 230 and a mouse 232 as a pointing device are connected.

[0044] The auxiliary storage I / F 212 is an interface for connecting an auxiliary storage device such as an HDD (Hard Disk Drive) 250 or an SSD (Solid State Drive).

[0045] The HDD 250 has a large storage capacity and stores programs for executing this embodiment. In the object posture calculation device 100, an image vertex recognition program 261, a 3D point determination program 262, an object angle calculation program 263, an image reception program 264, and an object angle transmission program 265 are installed.

[0046] The image vertex recognition program 261, the 3D point determination program 262, the object angle calculation program 263, the image reception program 264, and the object angle transmission program 265 are programs that execute the functions of the image vertex recognition unit 101, the 3D reference point determination unit 102, the object angle calculation unit 103, the image reception unit 104, and the object angle transmission unit 105, respectively.

[0047] Also, the HDD 250 holds an image data DB 111, a measurement data DB 112 for measurement data necessary for operation, and an object posture calculation data DB 113 used for calculating the posture of an object (pallet).

[0048] Next, with reference to FIGS. 3 and 4, processing related to object posture calculation in the forklift transportation system will be described.

[0049] First, the object posture calculation device 100 receives the two-dimensional image captured by the 2D camera 21 via the access point 5 (S01).

[0050] Next, the object posture calculation device 100 receives the three-dimensional image captured by the 3D camera 22 via the access point 5 (S02).

[0051] Next, the object pose calculation device 100 calculates the pose of an object (pallet) based on the two-dimensional image captured by the 2D camera 21 and the three-dimensional image captured by the 3D camera 22 (S03). This process will be described in detail below with reference to FIG. 4.

[0052] Next, the object pose calculation device 100 transmits information (pallet angle) regarding the calculated pose of the object (pallet) to the forklift via the access point 5 (S04).

[0053] Next, the details of the object pose calculation process will be described using FIGS. 13 and 4 which have already been described.

[0054] This process corresponds to the process of S03 in FIG. 3.

[0055] First, the image vertex recognition unit 101 of the object pose calculation device 100 performs image recognition processing by AI learning according to the pallet model, cuts out a bounding box from the two-dimensional image, and determines four vertices (a 11 , a 12 , a 21 , a 22 ) as seen from the forklift 20 side of the pallet (S10). However, in reality, due to errors, the four vertices are respectively A 11 , A 12 , A 21 , A 22 corresponding to a 11 , A 12 , A 21 , A 22 are obtained.

[0056] Next, the 3D reference point determination unit 102 of the object pose calculation device 100 calculates the coordinates of a point serving as a reference for 3D points (hereinafter referred to as "3D (Dimension: dimension) reference point") from the three-dimensional image captured by the 3D camera 22 (S20). The method for obtaining the coordinates of the 3D reference point will be described in detail later.

[0057] Calculate the 2D coordinates of one of the four vertices as seen from the side of the forklift 20, and the posture of the object (pallet) from the 3D reference point (S30). The detailed algorithm for calculating the posture of the object (pallet) will be described in detail later.

[0058] Next, the method for obtaining the coordinates of the 3D reference point will be described with reference to FIG. 5.

[0059] This is the process corresponding to S20 in FIG. 4.

[0060] First, in the process of obtaining the 3D reference point in S20, the straight line passing through the two opposite vertices of A 11 , A 12 , A 21 , A 22 is intersected, and this is referred to as the "apparent center point of the front surface of the pallet". In this embodiment, the apparent center point 14 of the front surface of the pallet is used as the 3D reference point.

[0061] A 11 , A 12 , A 21 , A 22 The 2D coordinates of are respectively A 11 =(x 11 , y 11 ), A 12 =(x 12 , y 12 ), A 21 =(x 21 , y 21 ), A 22 =(x 22 , y 22 ). Then, by solving the system of simultaneous equations in the following (Equation 1), the 2D coordinates (x0, y0) of the apparent center point 14 of the front surface of the pallet can be obtained.

[0062]

Equation

[0063] Then, by using 3D software, the 3D coordinates (x0, y0, z0) of the apparent center point 14 on the front surface of the palette corresponding to the 2D coordinates (x0, y0) of the apparent center point 14 on the front surface of the palette are obtained.

[0064] This process can be obtained, for example, by using the rs2_deproject_pixel_to_point function provided by Intel's RealSense (registered trademark) technology. The rs2_deproject_pixel_to_point function takes, as arguments, an object of the internal parameters of the camera, the coordinates (X, Y) in the image, and the depth, and converts them into the space with the camera as the origin.

[0065] Next, with reference to FIGS. 6 and 7, the process of calculating the 2D coordinates of one of the four vertices and the pose of the object (palette) from the 3D reference point will be described.

[0066] This is the process corresponding to S30 in FIG. 4.

[0067] As the coordinate system, the left-right direction of the 3D camera 22 is taken as the X-axis, the right side is the + direction of X, the front-back direction is taken as the Z-axis, the front side is the + direction of Z, the up-down direction is taken as the Y-axis, and the lower side is taken as the + direction of Y.

[0068] Here, it is assumed that the palette 10 is rotated by θ y minutes in the Y-axis direction.

[0069] The positional relationship between the 3D camera and the palette as viewed from the rear of the 3D camera is as shown in FIG. 6. Also, the positional relationship between the 3D camera and the palette as viewed from above the 3D camera is as shown in FIG. 7.

[0070] At this time, if the coordinates of the upper left vertex a 21 of the palette 10 are (x t , y t , z t ), there is the relationship shown in the following (Equation 2).

[0071] [Mathematics]

[0072] Here, as described with reference to FIG. 13, h is the height of the pallet 10, and s is the width of the pallet 10.

[0073] Next, consider the relationship between the coordinates of the upper left vertex a 21 of the pallet 10, which are (x t , y t , z t ), and the upper left vertex A 21 =(x 21 , y 21 ) for which image recognition processing is required.

[0074] There is the following relationship (Equation 3) between these two.

[0075] [Mathematics]

[0076] Here, r v is the horizontal resolution of the 2D image (e.g., 1920 pixels), r h is the vertical resolution of the 2D image (e.g., 1080 pixels), α is the vertical field of view of the 2D image (e.g., 59 degrees), and β is the horizontal field of view of the 2D image (e.g., 90 degrees).

[0077] This equation describes the relationship between 2D coordinates and 3D coordinates based on the properties of the 2D image.

[0078] From (Equation 2) and (Equation 3), eliminating (x t , y t , z t ) and using the following (Equation 4), which is a self-evident trigonometric relational expression, the range of the angle is limited to 0 ≦ θ y < π, and the rotation angle θ y of the pallet 10 in the Y-axis direction is obtained.

[0079] [Mathematics]

[0080] Note that although the rotation of the pallet 10 in the three-dimensional axis direction also exists in the X-axis and Z-axis directions, in the operation of the pallet 10, since the pallet 10 usually exists on a non-inclined ground or shelf, in order for the forklift 20 to recognize the position of the pallet 10, it is considered sufficient to only consider the rotation angle of the axis in the vertical direction (the pallet sways left and right).

[0081] As described above, according to the present embodiment, by using the 2D image obtained from the 2D camera and the 3D data obtained from the 3D camera, the posture of the pallet on which the luggage is loaded can be accurately grasped with simple calculation, and the conveying operation by a cargo handling device such as a forklift can be smoothly performed.

[0082] [Embodiment 2] Hereinafter, Embodiment 2 according to the present invention will be described with reference to FIGS. 8 to 11.

[0083] In Embodiment 1, an object posture calculation device that calculates the angle from the coordinate system of the pallet by using the 2D image obtained from the 2D camera and the 3D data obtained from the 3D camera was described.

[0084] This embodiment corrects the 3D reference points used in the calculation of Embodiment 1 so as to calculate them more accurately.

[0085] In this embodiment, the description will be centered on the differences from Embodiment 1.

[0086] The configuration of the forklift transportation system at the logistics site is almost the same as that of Embodiment 1. However, a 3D reference point correction unit 106 is added to the object posture calculation device 100. The 3D reference point correction unit 106 is a functional unit that obtains the corrected center point of the front surface of the pallet from the approximate center point 14 of the front surface of the pallet in Embodiment 1 (details will be described later).

[0087] Next, the processing of the object pose calculation device in Embodiment 2 will be described with reference to FIGS. 9 to 11.

[0088] The object pose calculation process of Embodiment 2 is substantially the same as the object pose calculation process of Embodiment 1 shown in FIG. 4, but 3D reference point coordinate correction processing (S21) is added between S20 and S30.

[0089] The 3D reference point coordinate correction process is a process of obtaining a corrected pallet front center point from the apparent pallet front center point 14.

[0090] Hereinafter, the 3D reference point coordinate correction process will be described in detail with reference to FIGS. 10 and 11.

[0091] In the 3D reference point coordinate correction process of the present embodiment, as shown in Embodiment 1, A recognized in 2D coordinates 11 , A 12 , A 21 , A 22 It is premised that the apparent pallet front center point 14, which is the intersection of the straight lines passing through the two opposite vertices of, has been obtained.

[0092] Next, the length of the line segment between A 11 and the apparent pallet front center point 14 is defined as L1, and the length of the line segment between the apparent pallet front center point 14 and A 22 is defined as L2.

[0093] L1 and L2 are obtained by the following (Equation 5).

[0094]

Equation

[0095] Then, L1 / L2 is obtained as the ratio of these line segments, and the corrected pallet front center point 15 is obtained from the apparent pallet front center point 14 according to this line segment ratio L1 / L2.

[0096] The coordinates of the corrected pallet front center point 15 are (xa , y a , z a ), the correction amount at the three-dimensional coordinates of the apparent pallet front center point 14 is (Δ x , Δ y , Δ z ). Then, the following (Equation 6) holds.

[0097]

Equation

[0098] From the line segment ratio L1 / L2, to obtain the correction amount (Δ x , Δ y , Δ z ), for example, for each line segment ratio L1 / L2, data such as the center point correction data table shown in FIG. 11 may be held and obtained.

[0099] As shown in FIG. 11, the center point correction data table 1131 is a table in which an X correction amount Δ x 1131b, a Y correction amount Δ y 1131c, and a Z correction amount Δ z 1131d are defined for each line segment ratio L1 / L21131a.

[0100] The appropriate values of the X correction amount Δ x 1131b, the Y correction amount Δ y 1131c, and the Z correction amount Δ z 1131d may be obtained, for example, by statistical data processing. In the above description, an example of obtaining the correction amount from the data of the table in which the correction amount for each line segment ratio L1 / L2 is defined has been described. However, a library function that takes the line segment ratio L1 / L2 as an argument and returns the correction amount as a value may be prepared, and the correction amount for the line segment ratio L1 / L2 may be obtained thereby.

[0101] The process of calculating the posture of the object (pallet) from the 2D coordinates of one of the four vertices as seen from the forklift 20 side in S30 hereinafter and the 3D reference point (corrected corrected pallet front center point) is the same as in Embodiment 1.

[0102] However, instead of (Equation 2) in Embodiment 1, the following (Equation 7) with the coordinates replaced is used.

[0103]

Equation

[0104] In this embodiment, in order to correct the apparent pallet front center point 14 obtained from the four vertices A 11 , A 12 , A 21 , A 22 of the pallet and adopt a more accurate 3D reference point, as a result, a more accurate posture of the pallet can be obtained.

[0105] 〔Embodiment 3〕 Hereinafter, Embodiment 3 according to the present invention will be described with reference to FIG. 12. This embodiment, similar to Embodiment 2, corrects the 3D reference point used in the calculation of Embodiment 1 so as to calculate it more accurately.

[0106] In this embodiment as well, the description will be centered on the differences from Embodiment 1.

[0107] The configuration of the forklift transportation system and the processing of the object posture calculation device at the logistics site are the same as those in Embodiment 2.

[0108] In this embodiment, compared with Embodiment 2, the content of the 3D reference point coordinate correction process (S21) is different.

[0109] Hereinafter, the 3D reference point coordinate correction process will be described in detail with reference to FIG. 12. In the 3D reference point coordinate correction process of this embodiment as well, it is premised that the apparent pallet front center point 14, which is the intersection of the straight lines passing through the two opposite vertices of A 11 , A 12 , A 21 , A 22 recognized in 2D coordinates, has been obtained.

[0110] Here, a point cloud extraction region 16 (i.e., height 2t, width 2p) with a length of t in the vertical direction and a length of p in the horizontal direction is formed from the center point 14 of the front surface of the apparent pallet.

[0111] And, assuming that there are m 3D coordinate points in the point cloud extraction region 16 for the image obtained in S02 of FIG. 3, the arithmetic mean in each (X, Y, Z) coordinate is taken as follows in (Equation 8), and the coordinates (x a , y a , z a ) of the corrected center point 15 of the front surface of the pallet are obtained.

[0112]

Equation

[0113] Here, (x i , y i , z i )(i = 1, …, m) are the coordinates of the points of the 3D image in the point cloud extraction region 16.

[0114] Also, the point cloud extraction region 16 takes an area of an appropriate size as viewed from the shape of the pallet. For example, the length t in the vertical direction and the length p in the horizontal direction are assumed to take values as in the following (Equation 9).

[0115]

Equation

[0116] Here, H h is the height of the fork pocket 11, H s is the width of the fork pocket 11, and R s is the width of the rectangular region between the fork pockets.

[0117] The process of calculating the 2D coordinates of one of the four vertices as seen from the forklift 20 side in S30 and the posture of the object (pallet) from the 3D reference point (corrected center point of the front surface of the corrected pallet) is the same as in Embodiment 1 and Embodiment 2.

[0118] Also, similar to Embodiment 2, instead of (Equation 2) in Embodiment 1, (Equation 7) with the coordinates replaced can be used.

[0119] In this embodiment as well, similar to Embodiment 2, for the four vertices A 11 , A 12 , A 21 , A 22 of the pallet, the apparent center point 14 of the front surface of the pallet obtained is corrected, and in order to adopt a more accurate 3D reference point, as a result, a more accurate posture of the pallet can be obtained.

Explanation of Signs

[0120] 5... Access point, 10... Pallet, 11... Fork pocket, 12... Side surface of the pallet, 13... Rectangular area between fork pockets, 14... Apparent center point of the front surface of the pallet, 15... Corrected center point of the front surface of the pallet, 16... Point cloud extraction area, 20... Forklift, 21... 2D camera (RGB camera), 22... 3D camera (Depth camera), 25... Fork, 27... External interface section, 28... Driving wheel, 26... Fork operating section, 30... Operation command section, 40... Radar sensor, 100... Object posture calculation device, 101... Image vertex recognition section, 102... 3D reference point determination section, 103... Object angle calculation section, 104... Image reception section, 105... Object angle transmission section, 106... 3D reference point correction section, 110... Storage section, 111... Image data DB, 112... Measurement data DB, 113... Object posture calculation data DB

Claims

1. An object pose calculation device that inputs a two-dimensional image captured by a two-dimensional camera and a three-dimensional image captured by a three-dimensional camera, and calculates the angle of a target object with respect to a predetermined coordinate system, comprising: an image vertex recognition unit that recognizes a plurality of vertices of the target object from the two-dimensional image; a three-dimensional reference point determination unit that determines the three-dimensional coordinates of the three-dimensional reference point based on the coordinates of the vertices recognized by the image vertex recognition unit; an object pose calculation unit that calculates the angle of one side surface of the target object with respect to a predetermined axis with respect to the predetermined coordinate system; The object pose calculation unit calculates the angle based on the coordinates of the vertices in the three-dimensional coordinates, the constraint conditions of the three-dimensional reference points in the three-dimensional coordinates, and the constraint conditions of the three-dimensional coordinates corresponding to the two-dimensional coordinates of the vertices based on the properties of the two-dimensional image. An object pose calculation device characterized by the above.

2. The object pose calculation device according to claim 1, wherein the target object has a rectangular parallelepiped shape, and the vertices recognized by the image vertex recognition unit include four vertices of a certain side surface of the rectangular parallelepiped shape.

3. The object pose calculation device according to claim 2, wherein the three-dimensional reference point is arranged at the intersection of line segments connecting two opposing points of the four vertices of the side surface.

4. Based on the ratio of the length of the first line segment connecting two opposing points and the length of the second line segment, a point corrected with respect to the intersection of the line segments connecting two opposing points of the four vertices of the side surface is used as the three-dimensional reference point. The object pose calculation device according to claim 2, characterized by the above.

5. Image data is acquired for a certain region centered on the intersection of the line segments connecting two opposing points of the four vertices of the side surface, and a point having as its coordinates the value obtained by taking the arithmetic mean of the respective coordinates in the three-dimensional coordinates of the image data is used as the three-dimensional reference point. The object pose calculation device according to claim 2, characterized by the above.

6. The object pose calculation device according to claim 1, wherein the target object is a pallet.

7. The object pose calculation device according to claim 5, wherein the target object is a pallet, and the certain region includes a rectangular region between the fork pockets of the pallet.

8. A two-dimensional camera that captures a two-dimensional image; A three-dimensional camera that captures a three-dimensional image; A forklift; A forklift transportation system comprising an object posture calculation device that inputs a two-dimensional image captured by a two-dimensional camera and a three-dimensional image captured by a three-dimensional camera, and calculates the angle of a target object with respect to a predetermined coordinate system. The step of the object posture calculation device acquiring a two-dimensional image captured by a two-dimensional camera. The step of the object posture calculation device acquiring a three-dimensional image captured by a three-dimensional camera. An image vertex recognition step in which the object posture calculation device recognizes a plurality of vertices of a pallet from the two-dimensional image. A three-dimensional reference point determination step in which the object posture calculation device determines the three-dimensional coordinates of a three-dimensional reference point based on the coordinates of the vertices recognized by the image vertex recognition unit. An object posture calculation step in which the object posture calculation device calculates the angle of one side surface of the target object with respect to a predetermined axis with respect to the predetermined coordinate system. Having an object posture calculation unit that calculates the angle of one side surface of the target object with respect to a predetermined axis with respect to the predetermined coordinate system. The step of the object posture calculation device transmitting the calculated angle to the forklift. The forklift has a step of operating the pallet based on the transmitted angle. In the object posture calculation step, the angle is calculated based on the coordinates of the vertices in the three-dimensional coordinates, the constraint conditions of the three-dimensional reference points in the three-dimensional coordinates, the two-dimensional coordinates of the vertices based on the properties of the two-dimensional image, and the constraint conditions of the corresponding three-dimensional coordinates. An object posture calculation method characterized by this.

Images