Identification system, calibration method, and work vehicle

Abstract

An identification system 1 for identifying a target from among objects existing around a work vehicle comprises: a three-dimensional sensor 11 attached to the work vehicle for detecting objects existing around the work vehicle; a camera 12 attached to the work vehicle for capturing an image of the surroundings of the work vehicle; and a controller 20. The controller 20 acquires point cloud data that is object detection data from the three-dimensional sensor 11 and image data of the captured image captured by the camera 12, identifies a feature point on the target on the basis of the point cloud data, identifies a feature point on the target on the basis of the captured image, and performs calibration processing for adjusting the relative position between the three-dimensional sensor 11 and the camera 12 on the basis of the position of the feature point identified from the point cloud data and the position of the feature point identified from the captured image.

Description

Identification system, calibration method and work vehicle

[0001] The present disclosure relates to an identification system, a calibration method, and a work vehicle.

[0002] For example, in order to perform accurate loading operations using work vehicles such as wheel loaders, shovels, and forklifts, it is necessary to improve the work vehicle's ability to identify work objects. Patent Document 1 discloses a known technology relating to an automatic excavator that automatically performs a series of processes, including excavating an excavation object and loading the excavated material into a loading object. The technology described in Patent Document 1 calculates the relative position of the loading object relative to the excavator and the relative attitude of the excavator relative to the loading object using images captured by a stereo camera. Patent Document 2 discloses a known technology relating to a loading vehicle that extracts an observation point cloud from three-dimensional distance data, and, if the extracted observation point cloud includes a feature representing the front of the loading object, extracts the feature and identifies the position, direction, and width of the loading object.

[0003] JP 10-088625 A JP 2016-210586 A

[0004] For example, it is desirable to estimate the position and orientation of a work target with high accuracy using a work vehicle such as a wheel loader, excavator, or forklift. Therefore, it is effective to identify the target using a 3D sensor such as LiDAR, which has high ranging accuracy, and images captured by a camera. To achieve this, a calibration process is required to adjust the relative positions of the 3D sensor and the camera.

[0005] An aspect of the present disclosure aims to provide an identification system, a calibration method, and a work vehicle that perform a calibration process to adjust the relative positions of a three-dimensional sensor and a camera.

[0006] According to an aspect of the present disclosure, there is provided an identification system for identifying an object from objects present in the vicinity of a work vehicle, the identification system including: an object detection sensor attached to the work vehicle for detecting objects present in the vicinity of the work vehicle, a camera attached to the work vehicle for photographing the vicinity of the work vehicle, and a controller. The controller acquires point cloud data, which is object detection data, from the object detection sensor and image data of images photographed by the camera, identifies feature points of the object based on the point cloud data, identifies the feature points of the object based on the photographed images, and performs a calibration process to adjust the relative positions of the object detection sensor and the camera based on the positions of the feature points identified from the point cloud data and the positions of the feature points identified from the photographed images.

[0007] According to an aspect of the present disclosure, there is provided a calibration method for adjusting the relative position of a camera and an object detection sensor for identifying a target from objects present in the vicinity of a work vehicle. The calibration method includes acquiring point cloud data, which is object detection data, from an object detection sensor attached to the work vehicle for detecting objects present in the vicinity of the work vehicle, acquiring image data of captured images from a camera attached to the work vehicle for capturing images of the vicinity of the work vehicle, identifying feature points of the target based on the point cloud data, identifying the feature points of the target based on the captured images, and adjusting the relative position of the object detection sensor and the camera based on the positions of the feature points identified from the point cloud data and the positions of the feature points identified from the captured images.

[0008] According to an aspect of the present disclosure, there is provided a work vehicle including a vehicle body, an object detection sensor attached to the work vehicle to detect objects present in the vicinity of the work vehicle, a camera attached to the work vehicle to photograph the vicinity of the work vehicle, and a controller. The controller acquires point cloud data, which is object detection data, from the object detection sensor and image data of images captured by the camera, identifies feature points of the object based on the point cloud data, identifies the feature points of the object based on the photographed images, and performs a calibration process to adjust the relative positions of the object detection sensor and the camera based on the positions of the feature points identified from the point cloud data and the positions of the feature points identified from the photographed images.

[0009] According to aspects of the present disclosure, there are provided an identification system, a calibration method, and a work vehicle that perform a calibration process to adjust the relative positions of a three-dimensional sensor and a camera.

[0010] FIG. 1 is a perspective view showing a work vehicle according to an embodiment. FIG. 2 is a functional block diagram showing an identification system according to an embodiment. FIG. 3 is a schematic diagram showing the positional relationship between a three-dimensional sensor and a camera. FIG. 4 is a schematic diagram showing the relative positions of the three-dimensional sensor and a camera. FIG. 5 is a block diagram showing a computer system according to an embodiment. FIG. 6 is a schematic diagram illustrating an example of an object identification process. FIG. 7 is a schematic perspective view illustrating another example of an object identification process. FIG. 8 is a schematic plan view illustrating another example of an object identification process. FIG. 9 is a diagram illustrating another example of an object identification process. FIG. 10 is a diagram showing another example of a binary image of an object. FIG. 11 is a diagram showing an example of a feature pattern of an object. FIG. 12 is a diagram showing the positions of feature points of an object identified from a point cloud. FIG. 13 is a diagram showing the positions of feature points of an object identified from a captured image. FIG. 14 is a schematic diagram showing the attachment position and detection range of a three-dimensional sensor according to an embodiment. FIG. 15 is a diagram showing an example of measurement points. FIG. 16 is a diagram showing another example of measurement points. FIG. 17 is a diagram showing another example of measurement points. Fig. 18 is a schematic diagram showing the positions of feature points in a camera coordinate system. Fig. 19 is a schematic diagram in which the positions of feature points in the camera coordinate system are projected onto a captured image. Fig. 20 is a diagram showing the error between the positions of feature points of an object identified from a captured image of the object and the positions of feature points identified from a point cloud projected onto the captured image. Fig. 21 is a diagram showing an example of a guide image that guides measurement points when a calibration process is performed. Fig. 22 is a flowchart showing an example of processing of a method for identifying an object using a point cloud. Fig. 23 is a flowchart showing an example of calibration processing according to an embodiment.

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings, but the present disclosure is not limited thereto. The components of each embodiment described below can be combined as appropriate. In addition, some components may not be used.

[0012] [Embodiment] <Work Vehicle> A work vehicle identifies a target from objects present around the work vehicle at a work site. In the embodiment, the work vehicle is, for example, a wheel loader or a forklift. In the following description, the work vehicle is described as a forklift, which is a type of cargo handling vehicle.

[0013] FIG. 1 is a perspective view showing a work vehicle according to an embodiment. A forklift 100 performs cargo handling operations. The cargo handling operations include retrieving transported goods stored in a predetermined storage location. In the embodiment, the object identified by the work vehicle is a transported goods. The transported goods are loading platforms or containers for carrying cargo. The transported goods are, for example, pallets, skids, or containers. The transported goods have a pair of fork insertion holes. In the embodiment, the object identified by the work vehicle is, for example, a box-shaped pallet.

[0014] The forklift 100 includes a vehicle body 102, a work implement 104 disposed in front of the vehicle body 102, and wheels 105 that support the vehicle body 102.

[0015] The vehicle body 102 includes a counterweight 120 and a fender 108. The counterweight 120 is disposed at the rear of the vehicle body 102. The counterweight 120 is attached to the rear of the vehicle body 102 to balance the weight of the forklift 100 in the front-to-rear direction when the forklift 100 picks up an object. The fender 108 is disposed at the front of the vehicle body 102. The fenders 108 are disposed on both the left and right sides of the vehicle body 102.

[0016] The work implement 104 performs at least a part of the loading and unloading work. The work implement 104 is disposed in front of the vehicle body 102. The work implement 104 is supported by the vehicle body 102. The work implement 104 has a mast 141, a bracket 142, and a fork 143.

[0017] The mast 141 is supported at the front of the vehicle body 102 so as to be tiltable. The mast 141 is long in the vertical direction. The bracket 142 supports the fork 143. The bracket 142 is supported by the mast 141. The bracket 142 is movable in the vertical direction along the mast 141. The fork 143 supports an object to be transported. The fork 143 is supported by the mast 141 via the bracket 142.

[0018] A pair of forks 143 are provided. The forks 143 include a first fork 143A and a second fork 143B disposed to the right of the first fork 143A. The bracket 142 supports the first fork 143A and the second fork 143B.

[0019] The wheels 105 support the vehicle body 102. At least a portion of the wheels 105 is disposed below the vehicle body 102. The wheels 105 include front wheels 105F and rear wheels 105R. The front wheels 105F are disposed forward of the rear wheels 105R. The front wheels 105F are disposed on the left and right sides of the vehicle body 102. The rear wheels 105R are disposed on the left and right sides of the vehicle body 102. Each of the front wheels 105F and the rear wheels 105R is rotatable around a rotation axis. One or both of the front wheels 105F and the rear wheels 105R are steered wheels. In this embodiment, the rear wheels 105R are steered wheels. Note that the front wheels 105F may be steered wheels, or both the front wheels 105F and the rear wheels 105R may be steered wheels.

[0020] In the embodiment, the left-right direction is a direction parallel to the rotation axes of the front wheels 105F and the rear wheels 105R when the forklift 100 travels straight ahead. The up-down direction is a direction perpendicular to the contact surfaces of the front wheels 105F and the rear wheels 105R. The front-rear direction is a direction perpendicular to both the left-right direction and the up-down direction.

[0021] The fenders 108 are positioned so as to cover at least a portion of the front wheels 105F. At least a portion of the fenders 108 are positioned higher than the front wheels 105F. At least a portion of the fenders 108 are positioned rearward of the front wheels 105F. The fenders 108 are positioned on both the left and right sides of the vehicle body 102. The left fender 108 is positioned so as to cover at least a portion of the left front wheel 105F. The right fender 108 is positioned so as to cover at least a portion of the right front wheel 105F.

[0022] In the following description, the terms "up-down direction," "vehicle width direction," and "front-rear direction" are used to describe the positional relationship of each part. The "up-down direction" refers to the direction perpendicular to the contact surface of the wheel 105 that contacts the ground. The "vehicle width direction" refers to the direction parallel to the rotation axis of the wheel 105. The "front-rear direction" refers to the direction perpendicular to the "up-down direction" and the "vehicle width direction."

[0023] "Up" refers to one direction in the up-down direction, and refers to the direction in which the cab 103 is located relative to the contact surface of the wheels 105. "Down" refers to the opposite direction of "up" in the up-down direction. "Left" refers to one direction in the vehicle width direction. "Right" refers to the opposite direction of "left" in the vehicle width direction. "Front" refers to one direction in the front-to-rear direction, and refers to the direction in which the work implement 110 is located relative to the cab 103. "Rear" refers to the opposite direction of "front" in the front-to-rear direction.

[0024] <Work vehicle identification system> Figure 2 is a functional block diagram showing an identification system according to an embodiment. The identification system 1 for the forklift 100 includes a three-dimensional sensor (object detection sensor) 11, a camera 12, an output unit 19, and a controller 20. The three-dimensional sensor 11, the camera 12, the output unit 19, and the controller 20 are connected to each other wirelessly or via a wire so as to be able to communicate data.

[0025] 3 is a schematic diagram showing the positional relationship between the three-dimensional sensors and the cameras. In this embodiment, the three-dimensional sensor 11 and the camera 12 are attached as a set to the forklift 100. The number of the three-dimensional sensors 11 and the cameras 12 is not particularly limited, and for example, a plurality of the three-dimensional sensors 11 and the cameras 12 may be arranged on the forklift 100.

[0026] The three-dimensional sensor 11 is attached to the forklift 100. In the embodiment, the three-dimensional sensor 11 is attached to the front of the forklift 100. The number of three-dimensional sensors 11 is not particularly limited, and for example, a plurality of three-dimensional sensors 11 may be arranged on the forklift 100. The three-dimensional sensor 11 is an object detection sensor that detects objects present around the forklift 100. In the embodiment, the three-dimensional sensor 11 is an object detection sensor that detects objects present in front of the forklift 100. The three-dimensional sensor 11 is capable of detecting a range of, for example, several tens of meters. The three-dimensional sensor 11 transmits detected object detection data to the controller 20.

[0027] The three-dimensional sensor 11 acquires three-dimensional data of an object. The three-dimensional data of the object includes a point cloud consisting of multiple detection points defined on the surface of the object. The point cloud indicates the relative distance and relative position between the three-dimensional sensor 11 and each of the multiple detection points defined on the surface of the object. An example of the three-dimensional sensor 11 is a laser sensor (LiDAR: Light Detection and Ranging) that detects objects by emitting laser light. However, the three-dimensional sensor 11 may also be another non-contact sensor, such as an infrared sensor that detects objects by emitting infrared light or a radar sensor (RADAR: Radio Detection and Ranging) that detects objects by emitting radio waves.

[0028] The camera 12 is attached to the forklift 100. The camera 12 is capable of capturing images of the periphery of the forklift 100. In this embodiment, the camera 12 is a camera that captures images of objects present in front of the forklift 100. The camera 12 is capable of capturing images of a range of, for example, several tens of meters. The camera 12 is disposed in a position where it can capture an image of the object. The camera 12 transmits image data of the captured image to the controller 20.

[0029] 4 is a schematic diagram showing the relative positions of the three-dimensional sensor and the camera. The position of the camera 12 shown by the dashed line indicates the design position (design value) relative to the three-dimensional sensor 11. However, when actually attached to the forklift 100, the position of the camera 12 shown by the solid line may differ from the design value due to installation errors, etc. In other words, the actual relative position of the three-dimensional sensor 11 and the camera 12 may differ from the design value.

[0030] The output unit 19 outputs the identification result. The output unit 19 is, for example, a monitor, a lamp, a buzzer, or the like. In the embodiment, the output unit 19 is a monitor. In the embodiment, the output unit 19 outputs a guide image for performing the calibration process under the control of the output control unit 29.

[0031] <Controller> The controller 20 identifies objects around the forklift 100. The controller 20 can acquire point cloud data, which is object detection data, from the three-dimensional sensor 11 serving as an object detection sensor, and image data of images captured by the camera 12. The controller 20 includes a numerical calculation device (processor) such as a CPU (Central Processing Unit).

[0032] FIG. 5 is a block diagram showing a computer system according to an embodiment. The controller 20 includes a computer system 1000. The computer system 1000 includes a processor 1001 such as a CPU, a main memory 1002 including a nonvolatile memory such as a read-only memory (ROM) and a volatile memory such as a random access memory (RAM), a storage 1003, and an interface 1004 including an input / output circuit. The functions of the controller 20 described above are stored as a program in the storage 1003. The processor 1001 reads the program from the storage 1003, loads it into the main memory 1002, and executes the above-described processing in accordance with the program. The program may be distributed to the computer system 1000 via a network.

[0033] In the embodiment, the controller 20 includes a point cloud identification unit 21 , an image processing unit 22 , a calibration processing unit 24 , and an output control unit 29 .

[0034] The point cloud identification unit 21 acquires object detection data from the three-dimensional sensor 11. The point cloud identification unit 21 identifies an object based on the object detection data. The point cloud identification unit 21 identifies the position and orientation of the object from the object detection data. Note that the method of identification in the point cloud identification unit 21 is not limited.

[0035] An example of an identification method in the point cloud identification unit 21 will be described. The point cloud identification unit 21 identifies an object based on point cloud data, which is object detection data from the three-dimensional sensor 11. More specifically, the point cloud identification unit 21 estimates a plane that constitutes at least a part of the object's outer shape from the acquired point cloud data. The point cloud identification unit 21 projects the point cloud data onto a projection plane and reconstructs a portion of the projection plane with high point cloud density into a three-dimensional representation to estimate the object's plane. The point cloud identification unit 21 then extracts a point cloud located on the estimated plane to generate a binary image. The point cloud identification unit 21 then detects, from the generated binary image, a shape that resembles a binary image of a feature pattern that indicates the shape of the plane portion of the object's outer shape, which has been stored in advance, to identify the object.

[0036] A more detailed description will be given of an example of the configuration of the point cloud identifying unit 21. The point cloud identifying unit 21 includes a sensor data acquiring unit 211, a storage unit 212, and an identifying unit 213.

[0037] The sensor data acquisition unit 211 acquires object detection data from the three-dimensional sensor 11. The object detection data acquired by the sensor data acquisition unit 211 is, for example, a point cloud made up of a plurality of detection points. Hereinafter, the object detection data will be referred to as point cloud data.

[0038] The storage unit 212 includes at least one of a RAM, a ROM, a flash memory, and a hard disk drive. The storage unit 212 stores data used in the processing of the controller 20.

[0039] The storage unit 212 stores a search range, which is a range in which the object is presumed to exist, and stores a feature pattern of the object for identifying the object.

[0040] The search range is a range for extracting a plane extraction point cloud used in processing to identify an object from point cloud data, which is object detection data. The point cloud data includes a point cloud of measurement distances and field of view angles of the three-dimensional sensor 11.

[0041] The feature pattern of the object is an image showing the shape of a flat portion of the object's outer shape. In this embodiment, the feature pattern is a binary image of fork insertion holes on a pallet.

[0042] The identification unit 213 identifies an object from the object detection data of the three-dimensional sensor 11. More specifically, the identification unit 213 estimates a plane that constitutes at least a part of the outline of the object from the point cloud data acquired by the sensor data acquisition unit 211, extracts a point cloud located on the estimated plane to generate a binary image, and identifies the object from the binary image.

[0043] The identification unit 213 projects the point cloud data onto a projection plane, and reconstructs the part of the projection plane where the point cloud density is high into three dimensions, thereby estimating the plane of the object.

[0044] The identification unit 213 detects from the generated binary image a shape that resembles a binary image of a feature pattern that indicates the shape of a plane portion of the pre-stored outline of the object, and identifies the object.

[0045] The classification processing in the classification unit 213 will be described in detail with reference to Figs. 6 to 11. Fig. 6 is a schematic diagram illustrating an example of the object classification processing. Fig. 7 is a schematic perspective view illustrating another example of the object classification processing. Fig. 8 is a schematic plan view illustrating another example of the object classification processing. Fig. 9 is a diagram illustrating another example of the object classification processing. Fig. 10 is a diagram illustrating another example of a binary image of the object. Fig. 11 is a diagram illustrating an example of a feature pattern of the object.

[0046] The point cloud is made up of a plurality of detection points defined on the surface of the object. First, the identification unit 213 estimates a plane that constitutes the object from the point cloud data.

[0047] As shown in FIG. 6(a1), the point cloud within the search range is extracted from the point cloud data to obtain the target point cloud. Then, as shown in FIG. 6(a2), the identification unit 213 projects the plane extraction point cloud onto the xy plane of a coordinate system in which the z-axis is perpendicular to the ground plane. FIG. 6(a3) shows the point cloud projected onto the xy plane. The point cloud surrounded by the dashed line on the xy plane corresponds to the point cloud on the side of the pallet perpendicular to the xy plane, surrounded by the dashed line. The point cloud on the plane perpendicular to the xy plane of the pallet is aggregated onto a single line segment to achieve high density. Then, as shown in FIG. 6(a4), the identification unit 213 converts the points back to their original three-dimensional positions, leaving only the area surrounded by the dashed line where the point cloud density is high. In this way, the identification unit 213 estimates the plane that constitutes the target by extracting the plane point cloud, which is the point cloud on the plane that constitutes the target. In this example, the plane point cloud is the point cloud on the front surface of the pallet.

[0048] The identification unit 213 moves the passing point cloud that has passed through the front surface of the pallet to the intersection of the line connecting each point with the 3D sensor 11 and the front surface of the pallet, and generates a binary image of the passing point cloud from the moved passing point cloud. As shown in Figures 7 and 8, the point cloud that passes through the estimated front surface of the pallet and is located behind the estimated front surface of the pallet is called a planar passing point cloud. The identification unit 213 calculates the intersection of the line connecting each point of the planar passing point cloud that has passed through the estimated front surface of the pallet to the position of the 3D sensor 11 and the front surface of the pallet. In Figure 8, the estimated front surface of the pallet is indicated by a dashed line. In Figure 8, the point cloud appears above the estimated front surface of the pallet. The point cloud surrounded by an ellipse indicates the point cloud that has passed through the estimated front surface of the pallet. In other words, the point cloud surrounded by an ellipse is the point cloud that has passed through the fork insertion holes of the transported item. The identification unit 213 moves the planar passing point cloud to the position of the calculated intersection, as shown in Figure 9. That is, the identification unit 213 moves the plane passing point cloud onto the front surface of the pallet. Fig. 9(a1) is a diagram showing the intersection of the estimated pallet front surface with a line connecting the point cloud that passes through the pallet front surface and the coordinate system reference of the 3D sensor 11. Fig. 9(a2) is a diagram in which the point cloud is interpolated at the position of the intersection found in (a1). In this way, the identification unit 213 calculates the position where each point of the plane passing point cloud passes through the plane.

[0049] The identification unit 213 generates a binary image from the plane passing point cloud, as shown in FIG. 10 . The identification unit 213 filters the generated binary image with the feature pattern of the fork insertion hole shown in FIG. 11 to calculate the similarity. The identification unit 213 searches the binary image for a pattern similar to the fork insertion hole pattern. In the example shown in FIG. 11 , region P11 is a region where the probability of a fork insertion hole being located is high. Region P12 is a region where the probability of a fork insertion hole being located is low. Region P13 is between regions P11 and P12 in terms of the probability of a fork insertion hole being located. C indicates the midpoint of the fork insertion hole.

[0050] 12 is a diagram showing the positions of the feature points of the object identified from the point cloud. The identification unit 213 identifies the feature points (X1s, Y1s, Z1s), (X2s, Y2s, Z2s), (X3s, Y3s, Z3s), and (X4s, Y4s, Z4s) of the pallet 300 from the point cloud in the manner described above.

[0051] In this embodiment, the characteristic points of the pallet 300 are the four corner points on the front of the pallet 300 .

[0052] The image processing unit 22 identifies an object from an image captured by the camera 12. The image processing unit 22 includes a feature extraction unit 221, an image classification unit 222, and a consistency checking unit 223.

[0053] The feature extraction unit 221 extracts features from an image and outputs feature data of the features extracted by the feature extraction unit 221 to the image classification unit 222.

[0054] The feature amount is, for example, a numerical representation of an edge (color boundary) in an image.

[0055] The image classification unit 222 classifies images by machine learning using the feature amount data to determine whether the image is a target image. The image classification unit 222 classifies images based on the feature amounts extracted by the feature amount extraction unit 221 and learned parameters of a machine learning model stored in a storage unit (not shown).

[0056] The method for identifying an object based on an image is not limited to the above, and the object may be identified by, for example, pattern matching.

[0057] The consistency checking unit 223 checks the consistency of the object identification results. The consistency checking unit 223 checks the consistency of the objects identified based on point cloud data. The consistency checking unit 223 checks the consistency between the identification result by the point cloud identification unit 21 and the identification result, which is the classification result by the image classification unit 222. The consistency checking unit 223 checks the consistency between the objects identified based on point cloud data and the objects identified based on images. If the objects identified by the point cloud identification unit 21 are classified as the same object by the image classification unit 222, the consistency checking unit 223 determines that the object identification results are consistent. If the objects identified by the point cloud identification unit 21 are not classified as the same object by the image classification unit 222, the consistency checking unit 223 determines that the object identification results are inconsistent. Note that if the identification result by the point cloud identification unit 21 and the identification result, which is the classification result by the image classification unit 222, differ, the consistency checking unit 223 may prioritize the identification result by the point cloud identification unit 21.

[0058] 13 is a diagram showing the positions of feature points of an object identified from a photographed image. As described above, the image processing unit 22 identifies the feature points (X1c, Y1c, Z1c), (X2c, Y2c, Z2c), (X3c, Y3c, Z3c), and (X4c, Y4c, Z4c) of the pallet 300 from the photographed image.

[0059] The calibration processing unit 24 performs a calibration process to adjust the relative positions of the three-dimensional sensor 11 and the camera 12. The calibration processing unit 24 adjusts the relative positions of the three-dimensional sensor 11 and the camera 12 based on the feature points identified by the point cloud identification unit 21 and the feature points identified by the image processing unit 22. In the embodiment, the calibration processing unit 24 adjusts the relative positions of the three-dimensional sensor 11 and the camera 12 based on the identification results of the three-dimensional sensor 11 and the camera 12 at multiple measurement points around the object.

[0060] 14 to 17, a description will be given of measurement points for performing calibration processing to adjust the relative positions of the three-dimensional sensor 11 and the camera 12. In this embodiment, nine measurement points are set around the periphery of the pallet.

[0061] 14 is a schematic diagram showing the mounting position and detection range of a three-dimensional sensor according to an embodiment. Reference numeral 101 schematically shows an example of the orientation direction and detection range of the three-dimensional sensor 11. Reference numeral 102 schematically shows an example of the orientation direction and detection range of the three-dimensional sensor 11. Reference numeral 103 schematically shows an example of the orientation direction and detection range of the three-dimensional sensor 11. The forklift 100 is capable of detecting in all directions ahead of the forklift 100 using multiple three-dimensional sensors 11.

[0062] Fig. 15 is a diagram showing an example of measurement points. In Fig. 14, the measurement points for calibrating the three-dimensional sensor 11 and camera 12 having the pointing direction and detection range of reference numeral 101 are measurement points P1 to P9. Adjacent measurement points in the X-axis direction are spaced 1.25 m apart. Adjacent measurement points in the Y-axis direction are spaced 1.5 m apart.

[0063] Fig. 16 is a diagram showing another example of measurement points. In Fig. 14, the measurement points for calibrating the three-dimensional sensor 11 and camera 12 having the pointing direction and detection range of reference numeral 102 are measurement points P1 to P9. Adjacent measurement points in the X-axis direction are spaced 2 m apart. Adjacent measurement points in the Y-axis direction are spaced 1 m apart.

[0064] Fig. 17 is a diagram showing another example of measurement points. In Fig. 14, the measurement points for calibrating the three-dimensional sensor 11 and camera 12 having the pointing direction and detection range of reference numeral 103 are measurement points P1 to P9. Adjacent measurement points in the X-axis direction are spaced 2 m apart. Adjacent measurement points in the Y-axis direction are spaced 1 m apart.

[0065] The positions of measurement points P1 to P9 relative to the pallet for each of the three-dimensional sensor 11 and camera 12, as shown in FIGS. 15 to 17, are stored in a storage unit (not shown).

[0066] The measurement points P1 to P9 may be marked on the ground, which makes it easier to position the forklift 100 at each measurement point.

[0067] The calibration processing unit 24 identifies the target at each measurement point while positioning the reference coordinate system of the vehicle body at the center of the front axle of the forklift 100.

[0068] FIG. 18 is a schematic diagram showing the positions of feature points in the camera coordinate system. In FIG. 18, the point cloud identification unit 21 identifies three feature points (X1s, Y1s, Z1s), (X2s, Y2s, Z2s), and (X3s, Y3s, Z3s), not shown, from the point cloud of the 3D sensor 11. The calibration processing unit 24 converts the identified three feature points (X1s, Y1s, Z1s), (X2s, Y2s, Z2s), and (X3s, Y3s, Z3s) into the camera coordinate system. The points converted into the camera coordinate system are designated as (X1sc, Y1sc, Z1sc), (X2sc, Y2sc, Z2sc), and (X3sc, Y3sc, Z3sc). During the initial optimization calculation, it is assumed that the relative positions of the 3D sensor 11 and the camera 12 are the same as the design positions (design values).

[0069] 19 is a schematic diagram showing the positions of feature points in the camera coordinate system projected onto a captured image. As shown in FIG. 19, the calibration processing unit 24 projects the points (X1sc, Y1sc, Z1sc), (X2sc, Y2sc, Z2sc), and (X3sc, Y3sc, Z3sc) converted into the camera coordinate system onto the captured image 160. The points projected onto the captured image 160 are represented by (X1sc / Z1sc, Y1sc / Z1sc,1), (X2sc / Z2sc, Y2sc / Z2sc,1), and (X3sc / Z3sc, Y3sc / Z3sc,1).

[0070] The calibration processing unit 24 optimizes the relative positions so as to minimize the positional error between the projected feature points and the feature points estimated in the captured image. For the optimization, a known method such as the Newton method or the Nelder-Mead method may be used.

[0071] In the embodiment, the calibration processing unit 24 calculates the positional error between the projected feature points and the feature points estimated in the captured image at all measurement points, and optimizes the relative position between the three-dimensional sensor 11 and the camera 12 so as to minimize the total error. The calibration processing unit 24 optimizes the relative position between the three-dimensional sensor 11 and the camera 12 through optimization calculations.

[0072] 20 is a diagram showing the error between the positions of feature points of an object identified from a photographed image of the object and the positions of feature points identified from a point cloud projected onto the photographed image. Black circles indicate feature points of the object identified from the photographed image by the image processing unit 22, and black diamonds indicate feature points identified from a point cloud of the 3D sensor 11 projected onto the photographed image. The errors, which are the deviations between the black circles and black diamonds at each feature point, are Δ1 to Δ4.

[0073] The output control unit 29 controls the output of the identification result by the point cloud identification unit 21 via the output unit 19. For example, when an object is identified, the output control unit 29 outputs a video signal that causes an image showing the identified object to be displayed on a monitor.

[0074] The output control unit 29 outputs a video signal for displaying the video of the guidance screen on the monitor when the calibration processing unit 24 is performing the calibration processing.

[0075] FIG. 21 is a diagram showing an example of a guide image that guides measurement points when the calibration process is performed.

[0076] The guide image includes a photographed image 160, a message image 170, a "Next" button image 175, and a "Cancel" button image 176. A guide frame image 161 and a counter image 162 are superimposed on the photographed image 160. The guide frame image 161 indicates the appropriate size of the pallet in the photographed image 160. When the pallet is small compared to the guide frame image 161, the forklift 100 is farther away from the pallet than the measurement point. When the pallet is large compared to the guide frame image 161, the forklift 100 is too close to the pallet than the measurement point. The counter image 162 indicates the measurement point currently being measured. In the example shown in FIG. 21 , the counter image 162 indicates that the first of nine measurement points is currently being measured.

[0077] <Processing by Controller> Fig. 22 is a flowchart showing an example of processing of a method for identifying an object using a point cloud. The processing of the flowchart shown in Fig. 22 is processing for identifying an object using a point cloud. The processing for identifying an object is executed during calibration processing by the forklift 100 and during loading and unloading operations. Note that the processing from step ST29 to step ST21 is not essential.

[0078] The controller 20 acquires a point cloud (step ST21). More specifically, the controller 20 receives the point cloud data from the three-dimensional sensor 11 via the sensor data acquisition unit 211. The controller 20 proceeds to step ST22.

[0079] The controller 20 crops the plane extraction point cloud (step ST22). More specifically, the controller 20 extracts, from the point cloud data, a point cloud included in a search range where it is estimated that the target exists, as the plane extraction point cloud, using the identification unit 213. The controller 20 proceeds to step ST23.

[0080] The controller 20 downsamples the plane extraction point cloud (step ST23). More specifically, the controller 20 downsamples the plane extraction point cloud using voxel downsampling, which is an example of a point cloud data downsampling method, by the identification unit 213. The controller 20 proceeds to step ST24.

[0081] The controller 20 extracts a point cloud on a plane and estimates a plane (step ST24). More specifically, the controller 20 extracts a point cloud on a plane, which is a point cloud on the front surface of the pallet, from the plane extraction point cloud downsampled by the identification unit 213, and estimates the side surface of the pallet, which is a plane. The controller 20 proceeds to step ST25.

[0082] The controller 20 crops the point cloud for generating a binary image (step ST25). More specifically, the controller 20 causes the identification unit 213 to crop the plane peripheral point cloud, which is the point cloud around the estimated plane, as a point cloud for generating a binary image. The controller 20 proceeds to step ST26.

[0083] The controller 20 generates a binary image of the plane peripheral point cloud (step ST26). More specifically, the controller 20 uses the identification unit 213 to generate a binary image on the yz plane of a coordinate system in which the normal vector of the estimated plane coincides with the direction of the x-axis, based on the cropped plane peripheral point cloud. The yz plane is a plane that includes the estimated plane. The controller 20 proceeds to step ST27.

[0084] The controller 20 generates a binary image of the plane passing points (step ST27). More specifically, the controller 20 generates a binary image of the plane passing points that have passed through the estimated front surface of the pallet using the identification unit 213. The controller 20 proceeds to step ST28.

[0085] The controller 20 detects features of the target from the binary image (step ST28). More specifically, the controller 20 detects a shape similar to the feature pattern in the binary image using the recognition unit 213. The controller 20 proceeds to step ST29.

[0086] The controller 20 restores the three-dimensional position (step ST29). More specifically, the controller 20 uses the recognition unit 213 to recognize the overall position, orientation, and shape of the pallet based on the detected positions in the binary image and the shapes of the fork insertion holes. The controller 20 proceeds to step ST30.

[0087] The controller 20 checks whether the pallet is within the effective field of view of the sensor (step ST30). More specifically, the controller 20 checks, using the recognition unit 213, whether the overall position of the pallet is within the effective field of view of the three-dimensional sensor 11. This is to confirm that the recognition performance is guaranteed, since recognition performance decreases around the boundary of the field of view of the three-dimensional sensor 11.

[0088] The controller 20 checks the consistency with the predicted value (step ST31). More specifically, the controller 20 uses the identification unit 213 to compare the predicted position of the target predicted from the previous estimation result with the currently estimated position of the target, and checks whether the difference is equal to or less than a threshold.

[0089] The controller 20 calculates the variance and likelihood of the point cloud identification (step ST32).

[0090] In this way, the controller 20 identifies the target pallet from the point cloud data.

[0091] FIG. 23 is a flowchart illustrating an example of a calibration process according to the embodiment.

[0092] The operator moves the forklift 100 to the first measurement point (step ST71).

[0093] The operator adjusts the vehicle position, work equipment, and attitude so that the display frame displayed on the monitor serving as the output unit 19 coincides with the front of the object, and presses the "Next" button (step ST72).

[0094] The controller 20 determines whether the identification of the object was successful (step ST73). The controller 20 executes the processing of the flowchart shown in FIG. 22 using the point cloud identification unit 21 to identify the feature points of the pallet from the object detection data of the 3D sensor 11. The controller 20 then identifies the feature points of the pallet from the captured image using the image processing unit 22. The controller 20 stores the feature points of the object identified by the point cloud identification unit 21 and the feature points of the object identified by the image processing unit 22 in the memory unit, in association with the measurement points. If the controller 20 determines that the identification of the object was successful (Yes in step ST73), the controller proceeds to step ST74. If the controller 20 does not determine that the identification of the object was successful (No in step ST73), the controller 20 executes the processing of step ST72 again.

[0095] The controller 20 determines whether or not measurement at all measurement points has been completed (step ST74). If the controller 20 determines that measurement at all measurement points has been completed (Yes in step ST74), the controller 20 proceeds to step ST76. If the controller 20 does not determine that measurement at all measurement points has been completed (No in step ST74), the controller 20 proceeds to step ST75.

[0096] If it is not determined that measurements at all measurement points have been completed (No in step ST74), the operator moves the forklift 100 to the next measurement point (step ST75).

[0097] If it is determined that measurements at all measurement points have been completed (Yes in step ST74), the operator presses the "Next" button to start the calibration process (step ST76).

[0098] The controller 20 causes the calibration processing unit 24 to project the positions of the feature points estimated in the point cloud onto the captured image (step ST77).

[0099] The controller 20 optimizes the relative positions of the 3D sensor 11 and the camera 12 by using the calibration processing unit 24 so as to minimize the positional error between the projected feature points and the feature points estimated in the captured image (step ST78). More specifically, the controller 20 calculates the positional error between the projected feature points and the feature points estimated in the captured image at all measurement points by using the calibration processing unit 24, and optimizes the relative positions of the 3D sensor 11 and the camera 12 so as to minimize the total error.

[0100] The controller 20 determines whether the difference between the obtained value and the design value is within a threshold value using the calibration processing unit 24 (step ST79). If the controller 20 determines that the difference between the obtained value and the design value is within the threshold value using the calibration processing unit 24 (Yes in step ST79), the controller 20 proceeds to step ST80. If the controller 20 does not determine that the difference between the obtained value and the design value is within the threshold value using the calibration processing unit 24 (No in step ST79), the controller 20 proceeds to step ST81.

[0101] When it is determined that the difference between the obtained value and the design value is within the threshold value (Yes in step ST79), the controller 20 causes the calibration processing unit 24 to determine that the calibration was successful and terminate the calibration process (step ST80).

[0102] If it is not determined that the difference between the obtained value and the design value is within the threshold value (No in step ST79), the controller 20 causes the calibration processing unit 24 to determine that the calibration has failed and terminate the calibration process (step ST81).

[0103] In this way, a calibration process is performed to adjust the relative positions of the three-dimensional sensor 11 and the camera 12.

[0104] <Effects> As described above, the embodiment can perform a calibration process to adjust the relative positions of the 3D sensor 11 and the camera 12 based on the positions of feature points identified from the point cloud data and the positions of feature points identified from the captured image.

[0105] In the embodiment, the positions of feature points identified from the point cloud data are projected onto the captured image, and a calibration process is performed to adjust the relative positions of the three-dimensional sensor 11 and the camera 12 based on the error between the positions of the projected feature points and the positions of the feature points identified from the captured image.

[0106] In the embodiment, a calibration process is performed to minimize the error by adjusting the relative positions of the three-dimensional sensor 11 and the camera 12. According to the embodiment, a calibration process can be performed to more appropriately adjust the relative positions of the three-dimensional sensor 11 and the camera 12.

[0107] In the embodiment, a calibration process is performed to adjust the relative positions of the three-dimensional sensor 11 and the camera 12 based on the error at each of a plurality of measurement points around the target. According to the embodiment, it is possible to perform a calibration process that more appropriately adjusts the relative positions of the three-dimensional sensor 11 and the camera 12.

[0108] In the embodiment, when the forklift 100 is positioned at a measurement point, a guide frame image 161 of a size that allows the target to fit within the frame is superimposed and displayed on the captured image. According to the embodiment, it is possible to assist the operator in appropriately moving the forklift 100 to the measurement point without placing markers around the target.

[0109] 1...identification system, 11...3D sensor (object detection sensor), 12...camera, 19...output unit, 20...controller, 21...point cloud identification unit, 211...sensor data acquisition unit, 212...memory unit, 213...identification unit, 22...image processing unit, 221...feature extraction unit, 222...image classification unit, 223...consistency verification unit, 24...calibration processing unit, 29...output control unit, 100...forklift (work vehicle), 102...body, 103...driver's cab, 104...work equipment, 105...wheel, 105F...front wheel, 105R...rear wheel, 108...fender, 141...mast, 142...bracket, 143...fork, 120...counterweight, 1000...computer system, 1001...processor, 1002...main memory, 1003...storage, 1004...interface.

Claims

1. An identification system for identifying an object from objects present in the vicinity of a work vehicle, comprising: an object detection sensor attached to the work vehicle for detecting objects present in the vicinity of the work vehicle; a camera attached to the work vehicle for photographing the vicinity of the work vehicle; and a controller, wherein the controller acquires point cloud data, which is object detection data, from the object detection sensor and image data of an image captured by the camera, identifies feature points of the object based on the point cloud data, identifies feature points of the object based on the captured image, and performs a calibration process to adjust the relative positions of the object detection sensor and the camera based on the positions of the feature points identified from the point cloud data and the positions of the feature points identified from the captured image.

2. The identification system of claim 1, wherein the controller performs a calibration process to project the positions of the feature points identified from the point cloud data onto the captured image, and adjust the relative positions of the object detection sensor and the camera based on the error between the projected positions of the feature points and the positions of the feature points identified from the captured image.

3. The identification system according to claim 2, wherein the controller performs a calibration process to adjust the relative positions of the object detection sensor and the camera so as to minimize the error.

4. The identification system described in claim 2 or 3, wherein the controller acquires the point cloud data and the image data at a plurality of measurement points around the object, calculates the error at each of the plurality of measurement points, and performs a calibration process to adjust the relative positions of the object detection sensor and the camera so that the sum of the errors is minimized.

5. The identification system according to claim 4, wherein the controller superimposes and displays a guide frame image on the captured image, the guide frame image being sized so that the target fits within the frame when the work vehicle is positioned at the measurement point.

6. A calibration method for adjusting the relative position between an object detection sensor and a camera for identifying an object from objects present in the vicinity of a work vehicle, comprising the steps of: acquiring point cloud data, which is object detection data, from an object detection sensor attached to the work vehicle for detecting objects present in the vicinity of the work vehicle; acquiring image data of a captured image from a camera attached to the work vehicle for capturing images of the vicinity of the work vehicle; identifying feature points of the object based on the point cloud data; identifying feature points of the object based on the captured image; and adjusting the relative position between the object detection sensor and the camera based on the positions of the feature points identified from the point cloud data and the positions of the feature points identified from the captured image.

7. A work vehicle comprising: a vehicle body; an object detection sensor attached to the work vehicle for detecting objects present in the vicinity of the work vehicle; a camera attached to the work vehicle for photographing the vicinity of the work vehicle; and a controller, wherein the controller acquires point cloud data, which is object detection data, from the object detection sensor and image data of an image captured by the camera, identifies feature points of an object based on the point cloud data, identifies feature points of the object based on the captured image, and performs a calibration process to adjust the relative positions of the object detection sensor and the camera based on the positions of the feature points identified from the point cloud data and the positions of the feature points identified from the captured image.

Images