Identification system, calibration method, and work vehicle

Abstract

To provide an identification system, calibration method and work vehicle that perform calibration processing of adjusting a relative position between a three-dimensional sensor and a camera.SOLUTION: An identification system 1 for identifying a target from objects existing around a work vehicle comprises: a three-dimensional sensor 11 that is attached to the work vehicle, and for detecting the object existing around the work vehicle; a camera 12 that is attached to the work vehicle, and for photographing surroundings of the work vehicle; and a controller 20. The controller 20 is configured to: acquire point group data serving as object detection data, and image data on a photographing image the camera 12 photographs from the three-dimensional sensor 11; identify a characteristic of the target on the basis of the point group data; identify a characteristic of the target on the basis of the photographing data; and perform calibration processing of adjusting a relative position between the three-dimensional sensor 11 and the camera 12 on the basis of a position of the characteristic identified from the point group data, and a position of the characteristic identified from the photographing image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to an identification system, a calibration method, and a work vehicle. [Background technology]

[0002] For example, in order to perform accurate loading and unloading operations using a work vehicle such as a wheel loader, a shovel, or a forklift, it is necessary to improve the identification performance of the work object by the work vehicle. As disclosed in Patent Document 1, a technology is known regarding an automatic excavator that automatically performs a series of processes of excavating an excavation object and loading the excavated material into a loading object. In the technology described in Patent Document 1, the relative position of the loading object to the excavator and the relative attitude of the excavator to the loading object are calculated using images captured by a stereo camera. As disclosed in Patent Document 2, a technology is known regarding a loading vehicle that extracts an observation point group from three-dimensional distance data, and when the extracted observation point group includes a feature representing the front of the loading object, extracts the feature and specifies the position, direction, and width of the loading object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 10-088625 [Patent Document 2] JP 2016-210586 A Summary of the Invention [Problem to be solved by the invention]

[0004] For example, it is desirable to estimate the position and orientation of a work target with high accuracy using a work vehicle such as a wheel loader, excavator, or forklift. Therefore, it is effective to identify the target using a 3D sensor such as LiDAR with high distance measurement accuracy and an image captured by a camera. For this purpose, it is necessary to perform a calibration process to adjust the relative position between the 3D sensor and the camera.

[0005] An aspect of the present disclosure aims to provide an identification system, a calibration method, and a work vehicle that perform a calibration process to adjust the relative position of a three-dimensional sensor and a camera. [Means for solving the problem]

[0006] According to an aspect of the present disclosure, there is provided an identification system for identifying an object from objects present in the vicinity of a work vehicle, the identification system including an object detection sensor attached to the work vehicle for detecting objects present in the vicinity of the work vehicle, a camera attached to the work vehicle for photographing the vicinity of the work vehicle, and a controller. The controller acquires point cloud data, which is object detection data, from the object detection sensor and image data of a photographed image captured by the camera, identifies feature points of the object based on the point cloud data, identifies feature points of the object based on the photographed image, and performs a calibration process to adjust the relative positions of the object detection sensor and the camera based on the positions of the feature points identified from the point cloud data and the positions of the feature points identified from the photographed image.

[0007] According to an aspect of the present disclosure, there is provided a calibration method for adjusting the relative position of an object detection sensor and a camera for identifying an object from objects present in the periphery of a work vehicle. The calibration method includes acquiring point cloud data, which is object detection data, from an object detection sensor that is attached to the work vehicle and for detecting objects present in the periphery of the work vehicle, acquiring image data of a captured image from a camera that is attached to the work vehicle and for capturing images of the periphery of the work vehicle, identifying feature points of the object based on the point cloud data, identifying the feature points of the object based on the captured image, and adjusting the relative position of the object detection sensor and the camera based on the positions of the feature points identified from the point cloud data and the positions of the feature points identified from the captured image.

[0008] According to an aspect of the present disclosure, there is provided a work vehicle including a vehicle body, an object detection sensor attached to the work vehicle for detecting objects present around the work vehicle, a camera attached to the work vehicle for photographing the periphery of the work vehicle, and a controller. The controller acquires point cloud data, which is object detection data, from the object detection sensor and image data of a captured image captured by the camera, identifies feature points of the target based on the point cloud data, identifies feature points of the target based on the captured image, and performs a calibration process to adjust the relative positions of the object detection sensor and the camera based on the positions of the feature points identified from the point cloud data and the positions of the feature points identified from the captured image. Effect of the Invention

[0009] According to aspects of the present disclosure, there are provided an identification system, a calibration method, and a work vehicle that perform a calibration process to adjust the relative positions of a three-dimensional sensor and a camera. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a perspective view showing a work vehicle according to an embodiment. [Diagram 2] FIG. 2 is a functional block diagram showing an identification system according to an embodiment. [Diagram 3] FIG. 3 is a schematic diagram showing the positional relationship between the three-dimensional sensor and the camera. [Figure 4] FIG. 4 is a schematic diagram showing the relative positions of the three-dimensional sensor and the camera. [Diagram 5] FIG. 5 is a block diagram illustrating a computer system according to an embodiment. [Figure 6] FIG. 6 is a schematic diagram illustrating an example of the target identification process. [Figure 7] FIG. 7 is a schematic perspective view illustrating another example of the object identification process. [Figure 8] FIG. 8 is a schematic plan view illustrating another example of the object identification process. [Figure 9] FIG. 9 is a diagram illustrating another example of the target identification process. [Figure 10] FIG. 10 is a diagram showing another example of a binary image of a target. [Figure 11] FIG. 11 is a diagram showing an example of a target feature pattern. [Figure 12] FIG. 12 is a diagram showing the positions of feature points of an object identified from a point group. [Figure 13] FIG. 13 is a diagram showing the positions of feature points of an object identified from a captured image. [Figure 14] FIG. 14 is a schematic diagram showing the attachment position and detection range of the three-dimensional sensor according to the embodiment. [Figure 15] FIG. 15 is a diagram showing an example of measurement points. [Figure 16] FIG. 16 is a diagram showing another example of the measurement points. [Figure 17] FIG. 17 is a diagram showing another example of the measurement points. [Figure 18] FIG. 18 is a schematic diagram showing the positions of feature points in the camera coordinate system. [Figure 19] FIG. 19 is a schematic diagram in which the positions of feature points in the camera coordinate system are projected onto a captured image. [Figure 20] FIG. 20 is a diagram showing the error between the positions of feature points of an object identified from a captured image of the object and the positions of feature points identified from a point cloud projected onto the captured image. [Figure 21] FIG. 21 is a diagram showing an example of a guide image that guides measurement points when the calibration process is performed. [Figure 22] FIG. 22 is a flowchart showing an example of a process of a method for identifying an object using a point cloud. [Figure 23] FIG. 23 is a flowchart illustrating an example of the calibration process according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings, but the present disclosure is not limited thereto. The components of each embodiment described below can be appropriately combined. In addition, some components may not be used.

[0012] [Embodiment] <Work vehicle> The work vehicle identifies an object from objects present around the work vehicle at a work site. In the embodiment, the work vehicle is, for example, a wheel loader or a forklift. In the following description, the work vehicle is a forklift, which is a type of loading vehicle.

[0013] FIG. 1 is a perspective view showing a work vehicle according to an embodiment. A forklift 100 performs loading and unloading operations. The loading and unloading operations include a loading and unloading operation of picking up transported objects placed at a predetermined storage position. In the embodiment, the object identified by the work vehicle is a transported object. The transported object is a loading platform or a container on which cargo is loaded. The transported object is, for example, a pallet, a skid, or a container. The transported object has a pair of fork insertion holes. In the embodiment, the object identified by the work vehicle is, for example, a box-shaped pallet.

[0014] The forklift 100 includes a vehicle body 102, a work implement 104 disposed in front of the vehicle body 102, and wheels 105 that support the vehicle body 102.

[0015] The vehicle body 102 includes a counterweight 120 and a fender 108. The counterweight 120 is disposed at the rear of the vehicle body 102. The counterweight 120 is attached to the rear of the vehicle body 102 in order to balance the weight of the forklift 100 in the front-to-rear direction when the forklift 100 picks up an object to be transported. The fender 108 is disposed at the front of the vehicle body 102. The fenders 108 are disposed on both the left and right sides of the vehicle body 102.

[0016] The work machine 104 performs at least a part of the loading and unloading work. The work machine 104 is disposed in front of the vehicle body 102. The work machine 104 is supported by the vehicle body 102. The work machine 104 has a mast 141, a bracket 142, and a fork 143.

[0017] The mast 141 is supported at the front of the vehicle body 102 so as to be able to tilt. The mast 141 is long in the vertical direction. The bracket 142 supports the fork 143. The bracket 142 is supported by the mast 141. The bracket 142 is able to move in the vertical direction along the mast 141. The fork 143 supports an object to be transported. The fork 143 is supported by the mast 141 via the bracket 142.

[0018] A pair of forks 143 are provided. The forks 143 include a first fork 143A and a second fork 143B disposed to the right of the first fork 143A. The bracket 142 supports the first fork 143A and the second fork 143B.

[0019] The wheels 105 support the vehicle body 102. At least a portion of the wheels 105 is disposed below the vehicle body 102. The wheels 105 have a front wheel 105F and a rear wheel 105R. The front wheel 105F is disposed forward of the rear wheel 105R. The front wheels 105F are disposed on the left and right sides of the vehicle body 102. The rear wheels 105R are disposed on the left and right sides of the vehicle body 102. Each of the front wheel 105F and the rear wheel 105R can rotate around a rotation axis. One or both of the front wheel 105F and the rear wheel 105R are steered wheels. In the embodiment, the rear wheel 5R is the steered wheel. Note that the front wheel 105F may be the steered wheel, or both the front wheel 105F and the rear wheel 105R may be steered wheels.

[0020] In the embodiment, the left-right direction is a direction parallel to the rotation axes of the front wheels 105F and the rear wheels 105R when the forklift 100 travels in a straight line. The up-down direction is a direction perpendicular to the ground contact surfaces of the front wheels 105F and the rear wheels 105R. The front-rear direction is a direction perpendicular to both the left-right direction and the up-down direction.

[0021] The fender 108 is positioned so as to cover at least a portion of the front wheel 105F. At least a portion of the fender 108 is positioned higher than the front wheel 105F. At least a portion of the fender 108 is positioned rearward of the front wheel 105F. The fenders 108 are positioned on both the left and right sides of the vehicle body 102. The left fender 108 is positioned so as to cover at least a portion of the left front wheel 105F. The right fender 108 is positioned so as to cover at least a portion of the right front wheel 105F.

[0022] In the following description, the positional relationship of each part will be described using the terms up-down direction, vehicle width direction, and front-rear direction. The up-down direction refers to the direction perpendicular to the ground contact surface of the wheel 105 that contacts the ground. The vehicle width direction refers to the direction parallel to the rotation axis of the wheel 105. The front-rear direction refers to the direction perpendicular to the up-down direction and the vehicle width direction.

[0023] "Up" refers to one direction in the up-down direction, and refers to the direction in which the cab 103 is located relative to the contact surface of the wheels 105. "Down" refers to the opposite direction of "up" in the up-down direction. "Left" refers to one direction in the vehicle width direction. "Right" refers to the opposite direction of "left" in the vehicle width direction. "Front" refers to one direction in the front-to-rear direction, and refers to the direction in which the work equipment 110 is located relative to the cab 103. "Rear" refers to the opposite direction of "front" in the front-to-rear direction.

[0024] <Work vehicle identification system> 2 is a functional block diagram showing an identification system according to an embodiment. The identification system 1 of the forklift 100 includes a three-dimensional sensor (object detection sensor) 11, a camera 12, an output unit 19, and a controller 20. The three-dimensional sensor 11, the camera 12, the output unit 19, and the controller 20 are connected to each other wirelessly or via a wire so as to be able to communicate data.

[0025] 3 is a schematic diagram showing the positional relationship between the three-dimensional sensor and the camera. In the embodiment, the three-dimensional sensor 11 and the camera 12 are attached as a pair to the forklift 100. The number of the three-dimensional sensors 11 and the cameras 12 is not particularly limited, and for example, a plurality of the three-dimensional sensors 11 and the cameras 12 may be arranged on the forklift 100.

[0026] The three-dimensional sensor 11 is attached to the forklift 100. In the embodiment, the three-dimensional sensor 11 is attached to the front of the forklift 100. The number of the three-dimensional sensors 11 is not particularly limited, and for example, a plurality of the three-dimensional sensors 11 may be arranged on the forklift 100. The three-dimensional sensor 11 is an object detection sensor that detects an object present around the forklift 100. In the embodiment, the three-dimensional sensor 11 is an object detection sensor that detects an object present in front of the forklift 100. The three-dimensional sensor 11 can detect a range of, for example, about several tens of meters. The three-dimensional sensor 11 transmits the detected object detection data to the controller 20.

[0027] The three-dimensional sensor 11 acquires three-dimensional data of an object. The three-dimensional data of the object includes a point cloud consisting of a plurality of detection points defined on the surface of the object. The point cloud indicates the relative distance and relative position between the three-dimensional sensor 11 and each of the plurality of detection points defined on the surface of the object. An example of the three-dimensional sensor 11 is a laser sensor (LiDAR: Light Detection and Ranging) that detects an object by emitting laser light. Note that the three-dimensional sensor 11 may be another non-contact sensor such as an infrared sensor that detects an object by emitting infrared light or a radar sensor (RADAR: Radio Detection and Ranging) that detects an object by emitting radio waves.

[0028] The camera 12 is attached to the forklift 100. The camera 12 is capable of capturing images of the periphery of the forklift 100. In the embodiment, the camera 12 is a camera that captures images of objects present in front of the forklift 100. The camera 12 is capable of capturing images of an area of, for example, several tens of meters. The camera 12 is disposed in a position where it can capture an image of the object. The camera 12 transmits image data of the captured image to the controller 20.

[0029] 4 is a schematic diagram showing the relative positions of the three-dimensional sensor and the camera. The position of camera 12 shown by the dashed line indicates the design position (design value) with respect to three-dimensional sensor 11. In contrast, when it is actually attached to forklift 100, the position of camera 12 may differ from the design value, as shown by the solid line, due to attachment errors, etc. In other words, the actual relative position of three-dimensional sensor 11 and camera 12 may differ from the design value.

[0030] The output unit 19 outputs the identification result. The output unit 19 is, for example, a monitor, a lamp, a buzzer, or the like. In the embodiment, the output unit 19 is a monitor. In the embodiment, the output unit 19 outputs a guide image for performing a calibration process under the control of the output control unit 29.

[0031] <Controller> The controller 20 identifies objects around the forklift 100. The controller 20 can acquire point cloud data, which is object detection data, from the three-dimensional sensor 11 serving as an object detection sensor, and image data of an image captured by the camera 12. The controller 20 includes a numerical calculation device (processor) such as a CPU (Central Processing Unit).

[0032] 5 is a block diagram showing a computer system according to an embodiment. The controller 20 includes a computer system 1000. The computer system 1000 includes a processor 1001 such as a CPU, a main memory 1002 including a non-volatile memory such as a ROM (Read Only Memory) and a volatile memory such as a RAM (Random Access Memory), a storage 1003, and an interface 1004 including an input / output circuit. The functions of the controller 20 described above are stored in the storage 1003 as a program. The processor 1001 reads the program from the storage 1003, loads it in the main memory 1002, and executes the above-mentioned processing according to the program. The program may be distributed to the computer system 1000 via a network.

[0033] In the embodiment, the controller 20 includes a point cloud identifier 21 , an image processor 22 , a calibration processor 24 , and an output controller 29 .

[0034] The point cloud identification unit 21 acquires object detection data from the three-dimensional sensor 11. The point cloud identification unit 21 identifies an object based on the object detection data. The point cloud identification unit 21 identifies the position and orientation of the object from the object detection data. Note that the method of identification in the point cloud identification unit 21 is not limited.

[0035] An example of an identification method in the point cloud identification unit 21 will be described. The point cloud identification unit 21 identifies an object based on point cloud data, which is object detection data from the three-dimensional sensor 11. More specifically, the point cloud identification unit 21 estimates a plane that constitutes at least a part of the object's outer shape from the acquired point cloud data. The point cloud identification unit 21 projects the point cloud data onto a projection plane, and estimates the object's plane by restoring a part with high point cloud density on the projection plane to three dimensions. Then, the point cloud identification unit 21 extracts a point cloud located on the estimated plane to generate a binary image. Then, the point cloud identification unit 21 detects a shape similar to a binary image of a feature pattern that indicates the shape of a plane part in the pre-stored outer shape of the object from the generated binary image, and identifies the object.

[0036] A more detailed example of the configuration of the point cloud identifying unit 21 will be described. The point cloud identifying unit 21 includes a sensor data acquiring unit 211, a storage unit 212, and an identifying unit 213.

[0037] The sensor data acquisition unit 211 acquires object detection data from the three-dimensional sensor 11. The object detection data acquired by the sensor data acquisition unit 211 is, for example, a point cloud made up of a plurality of detection points. Hereinafter, the object detection data will be referred to as point cloud data.

[0038] The storage unit 212 includes at least one of a RAM, a ROM, a flash memory, and a hard disk drive. The storage unit 212 stores data and the like used in the processing of the controller 20.

[0039] The storage unit 212 stores a search range, which is a range in which the target is assumed to exist, and stores a feature pattern of the target for identifying the target.

[0040] The search range is a range for extracting a plane extraction point cloud used in processing to identify an object from point cloud data, which is object detection data. The point cloud data includes a point cloud of the measurement distance and viewing angle of the three-dimensional sensor 11.

[0041] The feature pattern of an object is an image showing the shape of a flat portion of the outline of the object. In the embodiment, the feature pattern is a binary image of fork insertion holes of a pallet.

[0042] The identification unit 213 identifies an object from the object detection data of the three-dimensional sensor 11. More specifically, the identification unit 213 estimates a plane that constitutes at least a part of the outline of the object from the point cloud data acquired by the sensor data acquisition unit 211, extracts a point cloud located on the estimated plane to generate a binary image, and identifies the object from the binary image.

[0043] The identification unit 213 projects the point cloud data onto a projection plane, and restores a portion of the projection plane with high point cloud density into three dimensions to estimate the plane of the object.

[0044] The identification unit 213 detects, from the generated binary image, a shape that resembles a binary image of a characteristic pattern that indicates the shape of a planar portion of the pre-stored outline of the object, and identifies the object.

[0045] The classification processing in classification unit 213 will be described in detail with reference to Figs. 6 to 11. Fig. 6 is a schematic diagram for explaining an example of the classification processing of the object. Fig. 7 is a schematic diagram, a perspective view, for explaining another example of the classification processing of the object. Fig. 8 is a schematic diagram, a plan view, for explaining another example of the classification processing of the object. Fig. 9 is a diagram for explaining another example of the classification processing of the object. Fig. 10 is a diagram showing another example of a binary image of the object. Fig. 11 is a diagram showing an example of a feature pattern of the object.

[0046] The point cloud is made up of a plurality of detection points defined on the surface of the object. First, the identification unit 213 estimates a plane constituting the object from the point cloud data.

[0047] As shown in FIG. 6(a1), the point cloud within the search range is extracted from the point cloud data to obtain the target point cloud. Then, as shown in FIG. 6(a2), the identification unit 213 projects the plane extraction point cloud onto the xy plane of the coordinate system in which the z-axis is perpendicular to the ground plane. FIG. 6(a3) shows the point cloud projected onto the xy plane. The point cloud surrounded by the dashed line on the xy plane corresponds to the point cloud on the side of the pallet perpendicular to the xy plane surrounded by the dashed line. The point cloud on the plane perpendicular to the xy plane of the pallet is aggregated on a single line segment to become dense. Then, as shown in FIG. 6(a4), the identification unit 213 converts the points to the original three-dimensional position, leaving the part surrounded by the dashed line where the point cloud density is high on the xy plane. In this way, the identification unit 213 estimates the plane constituting the target by extracting the plane point cloud, which is the point cloud on the plane constituting the target. In this example, the plane point cloud is the point cloud on the front surface of the pallet (the front surface of the pallet).

[0048] The identification unit 213 moves the passing point group that has passed through the front surface of the pallet to the intersection point with the front surface of the pallet on a straight line connecting each point with the three-dimensional sensor 11, and generates a binary image of the passing point group from the moved passing point group. As shown in Figs. 7 and 8, a point group that passes through the estimated pallet front surface and is located behind the estimated pallet front surface is called a plane passing point group. The identification unit 213 calculates the intersection point between the straight line that connects each point of the plane passing point group that has passed through the estimated pallet front surface and the position of the three-dimensional sensor 11 and the pallet front surface. In Fig. 8, the estimated pallet front surface is shown by a dashed line. In Fig. 8, the point group appears above the estimated pallet front surface. The point group surrounded by an ellipse indicates the point group that has passed through the estimated pallet front surface. In other words, the point group surrounded by an ellipse is the point group that has passed through the fork insertion hole of the transported item. As shown in Fig. 9, the identification unit 213 moves the plane passing point group to the position of the calculated intersection point. That is, the identification unit 213 moves the plane passing point cloud onto the front surface of the pallet. Fig. 9(a1) is a diagram showing the intersection between the estimated pallet front surface and a straight line connecting the point cloud that passed through the pallet front surface and the coordinate system reference of the three-dimensional sensor 11. Fig. 9(a2) is a diagram showing the point cloud supplemented at the position of the intersection found in (a1). In this way, the identification unit 213 calculates the position where each point of the plane passing point cloud passed through the plane.

[0049] The identification unit 213 generates a binary image from the plane passing point group as shown in Fig. 10. The identification unit 213 filters the generated binary image with the feature pattern of the fork insertion hole shown in Fig. 11 to calculate the similarity. The identification unit 213 searches for a pattern similar to the fork insertion hole pattern in the binary image. In the example shown in Fig. 11, area P11 is an area where the probability of the fork insertion hole being located is high. Area P12 is an area where the probability of the fork insertion hole being located is low. Area P13 is between areas P11 and P12 in terms of the probability of the fork insertion hole being located. C indicates the midpoint of the fork insertion hole.

[0050] 12 is a diagram showing the positions of feature points of an object identified from the point cloud. The identification unit 213 identifies the feature points (X1s, Y1s, Z1s), (X2s, Y2s, Z2s), (X3s, Y3s, Z3s), and (X4s, Y4s, Z4s) of the palette 300 from the point cloud in the above manner.

[0051] In the embodiment, the feature points of the pallet 300 are the four corner points on the front of the pallet 300 .

[0052] The image processing unit 22 identifies an object from an image captured by the camera 12. The image processing unit 22 includes a feature extraction unit 221, an image classification unit 222, and a consistency confirmation unit 223.

[0053] The feature extraction unit 221 extracts features from an image and outputs feature data of the features extracted by the feature extraction unit 221 to the image classification unit 222.

[0054] The feature amount is, for example, a numerical representation of an edge (color boundary) in an image.

[0055] The image classification unit 222 classifies images by machine learning using the feature amount data to determine whether the image is a target image or not. The image classification unit 222 classifies images based on the feature amount extracted by the feature amount extraction unit 221 and learned parameters of a machine learning model stored in a storage unit (not shown).

[0056] The method of identifying an object based on an image is not limited to the above, and the object may be identified by, for example, pattern matching.

[0057] The consistency checking unit 223 checks the consistency of the identification result of the object. The consistency checking unit 223 checks the consistency of the object identified based on the point cloud data. The consistency checking unit 223 checks the consistency between the identification result by the point cloud identification unit 21 and the identification result which is the classification result by the image classification unit 222. The consistency checking unit 223 checks the consistency between the object identified based on the point cloud data and the object identified based on the image. If the object identified by the point cloud identification unit 21 is classified as the same object by the image classification unit 222, the consistency checking unit 223 determines that the identification result of the object is consistent. If the object identified by the point cloud identification unit 21 is not classified as the same object by the image classification unit 222, the consistency checking unit 223 determines that the identification result of the object is not consistent. Note that the consistency checking unit 223 may prioritize the identification result by the point cloud identification unit 21 when the identification result by the point cloud identification unit 21 and the identification result which is the classification result by the image classification unit 222 are different.

[0058] 13 is a diagram showing the positions of feature points of an object identified from a photographed image. The image processing unit 22 identifies the feature points (X1c, Y1c, Z1c), (X2c, Y2c, Z2c), (X3c, Y3c, Z3c), and (X4c, Y4c, Z4c) of the palette 300 from the photographed image as described above.

[0059] Calibration processing unit 24 performs a calibration process to adjust the relative position between 3D sensor 11 and camera 12. Calibration processing unit 24 adjusts the relative position between 3D sensor 11 and camera 12 based on the feature points identified by point cloud identification unit 21 and the feature points identified by image processing unit 22. In the embodiment, calibration processing unit 24 adjusts the relative position between 3D sensor 11 and camera 12 based on the respective identification results of 3D sensor 11 and camera 12 at multiple measurement points around the target object.

[0060] 14 to 17, a description will be given of measurement points for performing a calibration process to adjust the relative positions of three-dimensional sensor 11 and camera 12. In the embodiment, nine measurement points are set around the pallet.

[0061] 14 is a schematic diagram showing the mounting position and detection range of a three-dimensional sensor according to an embodiment. Reference numeral 101 typically shows an example of the orientation direction and detection range of the three-dimensional sensor 11. Reference numeral 102 typically shows an example of the orientation direction and detection range of the three-dimensional sensor 11. Reference numeral 103 typically shows an example of the orientation direction and detection range of the three-dimensional sensor 11. The forklift 100 is capable of detecting all directions ahead of the forklift 100 using multiple three-dimensional sensors 11.

[0062] Fig. 15 is a diagram showing an example of measurement points. In Fig. 14, the measurement points for calibrating the 3D sensor 11 and camera 12 having the pointing direction and detection range of the reference symbol 101 are measurement points P1 to P9. Adjacent measurement points in the X-axis direction are spaced 1.25 [m] apart. Adjacent measurement points in the Y-axis direction are spaced 1.5 [m] apart.

[0063] Fig. 16 is a diagram showing another example of measurement points. In Fig. 14, the measurement points for calibrating the 3D sensor 11 and camera 12 having the pointing direction and detection range of the reference symbol 102 are measurement points P1 to P9. Adjacent measurement points in the X-axis direction are spaced 2 [m] apart. Adjacent measurement points in the Y-axis direction are spaced 1 [m] apart.

[0064] Fig. 17 is a diagram showing another example of measurement points. In Fig. 14, the measurement points for calibrating the 3D sensor 11 and camera 12 having the pointing direction and detection range of the reference symbol 103 are measurement points P1 to P9. Adjacent measurement points in the X-axis direction are spaced 2 [m] apart. Adjacent measurement points in the Y-axis direction are spaced 1 [m] apart.

[0065] The positions of measurement points P1 to P9 with respect to the pallet for each of the three-dimensional sensor 11 and camera 12 as shown in FIGS. 15 to 17 are stored in a storage unit (not shown).

[0066] The measurement points P1 to P9 may be marked on the ground, which makes it easier to position the forklift 100 at each measurement point.

[0067] The calibration processing unit 24 identifies the target at each measurement point while positioning the reference coordinate system of the vehicle body at the center of the front axle of the forklift 100.

[0068] FIG. 18 is a schematic diagram showing the positions of feature points in the camera coordinate system. In FIG. 18, three feature points (X1s, Y1s, Z1s), (X2s, Y2s, Z2s), and (X3s, Y3s, Z3s) (not shown) are identified from the point cloud of the three-dimensional sensor 11 by the point cloud identification unit 21. The calibration processing unit 24 converts the three identified feature points (X1s, Y1s, Z1s), (X2s, Y2s, Z2s), and (X3s, Y3s, Z3s) into the camera coordinate system. The points converted into the camera coordinate system are (X1sc, Y1sc, Z1sc), (X2sc, Y2sc, Z2sc), and (X3sc, Y3sc, Z3sc). At the time of the first optimization calculation, it is assumed that the relative positions of the three-dimensional sensor 11 and the camera 12 are as per the design positions (design values).

[0069] Fig. 19 is a schematic diagram showing the positions of feature points in the camera coordinate system projected onto a captured image. As shown in Fig. 19, the calibration processing unit 24 projects the points (X1sc, Y1sc, Z1sc), (X2sc, Y2sc, Z2sc), and (X3sc, Y3sc, Z3sc) converted into the camera coordinate system onto the captured image 160. The points projected onto the captured image 160 are indicated by (X1sc / Z1sc, Y1sc / Z1sc,1), (X2sc / Z2sc, Y2sc / Z2sc,1), and (X3sc / Z3sc, Y3sc / Z3sc,1).

[0070] The calibration processing unit 24 optimizes the relative positions so as to minimize the positional error between the projected feature points and the feature points estimated in the captured image. For the optimization, a known method such as the Newton method or the Nelder-Mead method may be used.

[0071] In the embodiment, calibration processing unit 24 calculates the position error between the projected feature points and the feature points estimated in the captured image at all measurement points, and optimizes the relative position between 3D sensor 11 and camera 12 so as to minimize the sum of the errors. Calibration processing unit 24 optimizes the relative position between 3D sensor 11 and camera 12 by optimization calculation.

[0072] 20 is a diagram showing the error between the positions of feature points of an object identified from a captured image of the object and the positions of feature points identified from the point cloud projected onto the captured image. Black circles indicate feature points of the object identified from the captured image by image processing unit 22, and black diamonds indicate feature points identified from the point cloud of 3D sensor 11 projected onto the captured image. The errors, which are the deviations between the black circles and black diamonds at each feature point, are Δ1 to Δ4.

[0073] The output control unit 29 controls the point cloud identification unit 21 to output the identification result via the output unit 19. For example, when an object is identified, the output control unit 29 outputs a video signal for displaying an image showing the identified object on a monitor.

[0074] The output control unit 29 outputs a video signal for displaying an image of the guide screen on the monitor when the calibration processing unit 24 is executing the calibration processing.

[0075] FIG. 21 is a diagram showing an example of a guide image that guides measurement points when the calibration process is performed.

[0076] The guide image includes a photographed image 160, a message image 170, a "Next" button image 175, and a "Cancel" button image 176. A guide frame image 161 and a counter image 162 are superimposed on the photographed image 160. The guide frame image 161 indicates the appropriate size of the pallet in the photographed image 160. When the pallet is small relative to the guide frame image 161, the forklift 100 is farther away from the pallet than the measurement point is. When the pallet is large relative to the guide frame image 161, the forklift 100 is too close to the pallet than the measurement point is. The counter image 162 indicates the measurement point currently being measured. In the example shown in FIG. 21, the counter image 162 indicates that the first of nine measurement points is currently being measured.

[0077] <Processing by the controller> Fig. 22 is a flowchart showing an example of a process of identifying an object using a point cloud. The process of the flowchart shown in Fig. 22 is a process of identifying an object using a point cloud. The process of identifying an object is executed during a calibration process by the forklift 100 and during a loading and unloading operation. Note that the processes from step ST29 to step ST21 are not essential processes.

[0078] The controller 20 acquires a point cloud (step ST21). More specifically, the controller 20 receives the point cloud data from the three-dimensional sensor 11 via the sensor data acquisition unit 211. The controller 20 proceeds to step ST22.

[0079] The controller 20 crops the plane extraction point cloud (step ST22). More specifically, the controller 20 extracts, from the point cloud data, a point cloud included in a search range in which it is estimated that a target exists, as a plane extraction point cloud, using the identification unit 213. The controller 20 proceeds to step ST23.

[0080] The controller 20 downsamples the plane extraction point cloud (step ST23). More specifically, the controller 20 downsamples the plane extraction point cloud using voxel downsampling, which is an example of a point cloud data downsampling technique, by the discrimination unit 213. The controller 20 proceeds to step ST24.

[0081] The controller 20 extracts a point cloud on a plane and estimates a plane (step ST24). More specifically, the controller 20 extracts a point cloud on a plane, which is a point cloud on the front surface of the pallet, from the plane extraction point cloud downsampled by the identification unit 213, and estimates the side surface of the pallet, which is a plane. The controller 20 proceeds to step ST25.

[0082] The controller 20 crops the point group for generating a binary image (step ST25). More specifically, the controller 20 crops the plane surrounding point group, which is the point group around the plane estimated by the discrimination unit 213, as a point group for generating a binary image. The controller 20 proceeds to step ST26.

[0083] The controller 20 generates a binary image of the plane peripheral point group (step ST26). More specifically, the controller 20 uses the discrimination unit 213 to convert the cropped plane peripheral point group into a binary image on the yz plane of a coordinate system in which the normal vector of the estimated plane coincides with the direction of the x-axis. The yz plane is a plane that includes the estimated plane. The controller 20 proceeds to step ST27.

[0084] The controller 20 generates a binary image of the plane passing points (step ST27). More specifically, the controller 20 generates a binary image of the plane passing points that pass through the estimated pallet front surface by the discrimination unit 213. The controller 20 proceeds to step ST28.

[0085] The controller 20 detects the target feature from the binary image (step ST28). More specifically, the controller 20 detects a shape similar to the feature pattern in the binary image by the discrimination unit 213. The controller 20 proceeds to step ST29.

[0086] The controller 20 restores the three-dimensional position (step ST29). More specifically, the controller 20 uses the recognition unit 213 to recognize the overall position, attitude, and shape of the pallet based on the detected positions in the binary image and the shapes of the fork insertion holes. The controller 20 proceeds to step ST30.

[0087] Controller 20 checks whether it is within the effective field of view of the sensor (step ST30). More specifically, controller 20 checks, by recognition unit 213, whether the overall position of the pallet is within the effective field of view of three-dimensional sensor 11. This is to confirm that recognition performance is guaranteed, since recognition performance drops around the boundaries of the field of view of three-dimensional sensor 11.

[0088] The controller 20 checks the consistency with the predicted value (step ST31). More specifically, the controller 20 uses the discrimination unit 213 to compare the predicted position of the target predicted from the previous estimation result with the currently estimated position of the target, and checks whether the difference is equal to or smaller than a threshold value.

[0089] The controller 20 calculates the variance and likelihood of the point group classification (step ST32).

[0090] In this manner, the controller 20 identifies the target pallet from the point cloud data.

[0091] FIG. 23 is a flowchart illustrating an example of the calibration process according to the embodiment.

[0092] The operator moves the forklift 100 to the first measurement point (step ST71).

[0093] The operator adjusts the vehicle position, working equipment, and attitude so that the display frame displayed on the monitor serving as the output unit 19 coincides with the front of the object, and presses the "Next" button (step ST72).

[0094] The controller 20 determines whether the identification of the object has been successful (step ST73). The controller 20 executes the processing of the flowchart shown in FIG. 22 by the point cloud identification unit 21 to identify the feature points of the pallet from the object detection data of the three-dimensional sensor 11. The controller 20 identifies the feature points of the pallet from the captured image by the image processing unit 22. The controller 20 stores the feature points of the object identified by the point cloud identification unit 21 and the feature points of the object identified by the image processing unit 22 in association with the measurement points in the memory unit. If the controller 20 determines that the identification of the object has been successful (Yes in step ST73), the controller 20 proceeds to step ST74. If the controller 20 does not determine that the identification of the object has been successful (No in step ST73), the controller 20 executes the processing of step ST72 again.

[0095] The controller 20 judges whether or not the measurement of all the measurement points is completed (step ST74). If the controller 20 judges that the measurement of all the measurement points is completed (Yes in step ST74), the controller 20 proceeds to step ST76. If the controller 20 does not judge that the measurement of all the measurement points is completed (No in step ST74), the controller 20 proceeds to step ST75.

[0096] When it is not determined that the measurements at all the measurement points have been completed (No in step ST74), the operator moves the forklift 100 to the next measurement point (step ST75).

[0097] When it is determined that the measurements at all the measurement points have been completed (Yes in step ST74), the operator presses the "Next" button to start the calibration process (step ST76).

[0098] The controller 20 causes the calibration processing unit 24 to project the positions of the feature points estimated in the point cloud onto the captured image (step ST77).

[0099] Controller 20 optimizes the relative position between 3D sensor 11 and camera 12 so that the position error between the projected feature points and the feature points estimated in the captured image is minimized by calibration processing unit 24 (step ST78). More specifically, controller 20 calculates the position error between the projected feature points and the feature points estimated in the captured image at all measurement points by calibration processing unit 24, and optimizes the relative position between 3D sensor 11 and camera 12 so that the sum of the errors is minimized.

[0100] The controller 20 determines whether the difference between the obtained value and the design value is within the threshold value by the calibration processing unit 24 (step ST79). If the controller 20 determines that the difference between the obtained value and the design value is within the threshold value by the calibration processing unit 24 (Yes in step ST79), the controller 20 proceeds to step ST80. If the controller 20 does not determine that the difference between the obtained value and the design value is within the threshold value by the calibration processing unit 24 (No in step ST79), the controller 20 proceeds to step ST81.

[0101] When it is determined that the difference between the obtained value and the design value is within the threshold value (Yes in step ST79), the controller 20 causes the calibration processing unit 24 to end the calibration process as a success (step ST80).

[0102] When it is not determined that the difference between the obtained value and the design value is within the threshold value (No in step ST79), the controller 20 causes the calibration processing unit 24 to end the calibration process as a calibration failure (step ST81).

[0103] In this manner, a calibration process is performed to adjust the relative positions of the three-dimensional sensor 11 and the camera 12.

[0104] <Effects> As described above, the embodiment can perform a calibration process to adjust the relative positions of the 3D sensor 11 and the camera 12 based on the positions of feature points identified from the point cloud data and the positions of feature points identified from the captured image.

[0105] In the embodiment, the positions of feature points identified from point cloud data are projected onto the captured image, and a calibration process can be performed to adjust the relative positions of the 3D sensor 11 and the camera 12 based on the error between the positions of the projected feature points and the positions of the feature points identified from the captured image.

[0106] In the embodiment, a calibration process is performed to adjust the relative positions of the three-dimensional sensor 11 and the camera 12 so as to minimize the error. According to the embodiment, a calibration process can be performed to more appropriately adjust the relative positions of the three-dimensional sensor 11 and the camera 12.

[0107] In the embodiment, a calibration process is performed to adjust the relative positions between the three-dimensional sensor 11 and the camera 12 based on the error at each of a plurality of measurement points around the target. According to the embodiment, it is possible to perform a calibration process that more appropriately adjusts the relative positions between the three-dimensional sensor 11 and the camera 12.

[0108] In the embodiment, when the forklift 100 is located at the measurement point, a guide frame image 161 of a size that allows the target to fit within the frame is displayed superimposed on the captured image. According to the embodiment, it is possible to assist the operator in appropriately moving the forklift 100 to the measurement point without placing markers around the target. [Explanation of symbols]

[0109] 1...identification system, 11...3D sensor (object detection sensor), 12...camera, 19...output section, 20...controller, 21...point cloud identification section, 211...sensor data acquisition section, 212...memory section, 213...identification section, 22...image processing section, 221...feature extraction section, 221...image classification section, 222...consistency verification section, 24...calibration processing section, 29...output control section, 100...forklift (work vehicle), 102...body, 103...operator's cab, 104...work equipment, 105...wheel, 105F...front wheel, 105R...rear wheel, 108...fender, 141...mast, 142...bracket, 143...fork, 120...counterweight, 1000...computer system, 1001...processor, 1002...main memory, 1003...storage, 1004...interface.

Claims

1. An identification system for identifying an object from objects present in the vicinity of a work vehicle, comprising: an object detection sensor attached to the work vehicle for detecting objects present around the work vehicle; A camera attached to the work vehicle for photographing the surroundings of the work vehicle; A controller; Equipped with The controller: Obtaining point cloud data, which is object detection data, from the object detection sensor and image data of a captured image taken by the camera; Identifying feature points of an object based on the point cloud data; Identifying feature points of the object based on the captured image; performing a calibration process for adjusting a relative position between the object detection sensor and the camera based on the positions of the feature points identified from the point cloud data and the positions of the feature points identified from the captured image; Identification system.

2. The controller: projecting the positions of the feature points identified from the point cloud data onto the captured image, and performing a calibration process to adjust the relative positions of the object detection sensor and the camera based on an error between the projected positions of the feature points and the positions of the feature points identified from the captured image; The identification system of claim 1 .

3. The controller: performing a calibration process to adjust the relative positions of the object detection sensor and the camera so as to minimize the error; The identification system of claim 2 .

4. The controller: Acquire the point cloud data and the image data at a plurality of measurement points around the object; calculating the error at each of the plurality of measurement points, and performing a calibration process to adjust the relative positions of the object detection sensor and the camera so that the sum of the errors is minimized; An identification system according to claim 2 or 3.

5. The controller: a guide frame image having a size such that the target fits within the frame when the work vehicle is located at the measurement point is superimposed on the captured image. The identification system of claim 4.

6. A calibration method for adjusting a relative position between an object detection sensor and a camera for identifying a target from objects present around a work vehicle, comprising: Acquiring point cloud data, which is object detection data, from an object detection sensor attached to the work vehicle and configured to detect objects present around the work vehicle; Acquiring image data of a captured image from a camera attached to the work vehicle for capturing an image of the periphery of the work vehicle; Identifying feature points of an object based on the point cloud data; Identifying feature points of the object based on the captured image; adjusting a relative position between the object detection sensor and the camera based on the positions of the feature points identified from the point cloud data and the positions of the feature points identified from the captured image; Calibration methods.

7. A work vehicle, The car body and an object detection sensor attached to the work vehicle for detecting objects present around the work vehicle; A camera attached to the work vehicle for photographing the surroundings of the work vehicle; A controller; Equipped with The controller: Obtaining point cloud data, which is object detection data, from the object detection sensor and image data of a captured image taken by the camera; Identifying feature points of an object based on the point cloud data; Identifying feature points of the object based on the captured image; performing a calibration process for adjusting a relative position between the object detection sensor and the camera based on the positions of the feature points identified from the point cloud data and the positions of the feature points identified from the captured image; Work vehicle.

Images