Identification system, identification method, and work vehicle

Abstract

An identification system 1 comprises: a three-dimensional sensor 11 that is attached to a work vehicle and that detects an object which exists in the periphery of the work vehicle; a camera 12 that is attached to the work vehicle and that captures an image of the periphery of the work vehicle; and a controller 20. The controller 20: acquires, from the three-dimensional sensor 11, point cloud data that is object detection data and image data of a captured image captured by the camera 12; identifies a subject on the basis of the point cloud data; converts the captured image by using the identification result based on the point cloud data; and identifies the subject on the basis of the converted image.

Description

Identification system, identification method, and work vehicle

[0001] The present disclosure relates to an identification system, an identification method, and a work vehicle.

[0002] For example, in order to perform accurate work using work vehicles such as dump trucks, wheel loaders, excavators, and forklifts, it is necessary to improve the work vehicle's ability to identify work objects.As disclosed in Patent Document 1, a technology is known that includes a laser sensor that detects the distance to the target pallet, a plane equation calculation unit that calculates a plane equation for the front surface of the pallet based on detection data from the laser sensor, and an estimation calculation unit that estimates the position and attitude of the pallet using the plane equation for the front surface of the pallet calculated by the plane equation calculation unit, and the estimation calculation unit calculates the yaw angle, pitch angle, and roll angle of the pallet as the attitude of the pallet.

[0003] Japanese Patent Application Laid-Open No. 2021-42070

[0004] For example, in order to accurately estimate the position and orientation of a work target by a work vehicle such as a dump truck, wheel loader, excavator, or forklift, it is effective to identify the target using a three-dimensional sensor such as LiDAR, which has high ranging accuracy. However, a point cloud, which is measurement data from a three-dimensional sensor, may incorrectly identify the target due to, for example, a lack of color information of the target or low resolution. To improve identification performance, it is effective to use images captured by a camera, but this may result in a high computational load.

[0005] An aspect of the present disclosure aims to provide an identification system, an identification method, and a work vehicle that can identify an object from an image captured by a camera while reducing the calculation load.

[0006] According to an aspect of the present disclosure, there is provided an identification system for identifying an object from objects present in the vicinity of a work vehicle, the system including an object detection sensor attached to a work vehicle for detecting objects present in the vicinity of the work vehicle, a camera attached to the work vehicle for photographing the vicinity of the work vehicle, and a controller. The controller acquires point cloud data, which is object detection data, from the object detection sensor and image data of an image captured by the camera, identifies the object based on the point cloud data, transforms the photographed image using the identification result based on the point cloud data, and identifies the object based on the transformed image.

[0007] According to an aspect of the present disclosure, there is provided an identification method for identifying an object from objects present in the vicinity of a work vehicle, comprising the steps of: acquiring point cloud data, which is object detection data, from an object detection sensor attached to a work vehicle for detecting objects present in the vicinity of the work vehicle; acquiring image data of a captured image from a camera attached to the work vehicle for capturing images of the vicinity of the work vehicle; identifying the object based on the point cloud data; transforming the captured image using an identification result based on the point cloud data; and identifying the object based on the transformed image.

[0008] According to an aspect of the present disclosure, there is provided a work vehicle including a vehicle body, an object detection sensor attached to the work vehicle to detect objects present in the vicinity of the work vehicle, a camera attached to the work vehicle to capture images of the vicinity of the work vehicle, and a controller. The controller acquires point cloud data, which is object detection data, from the object detection sensor and image data of images captured by the camera, identifies objects based on the point cloud data, transforms the captured images using the identification results based on the point cloud data, and identifies the objects based on the transformed images.

[0009] According to aspects of the present disclosure, an identification system, an identification method, and a work vehicle are provided that can identify an object from an image captured by a camera while reducing the computational load.

[0010] FIG. 1 is a side view showing a work vehicle according to an embodiment. FIG. 2 is a functional block diagram showing an identification system according to an embodiment. FIG. 3 is a block diagram showing a computer system according to an embodiment. FIG. 4 is a schematic diagram showing the operation of a work vehicle according to an embodiment. FIG. 5 is a diagram showing an example of a point cloud representing an object. FIG. 6 is a schematic diagram explaining an example of an object identification process. FIG. 7 is a diagram showing an example of a binary image of an object. FIG. 8 is a diagram showing an example of a feature pattern of an object. FIG. 9 is a schematic diagram showing the positional relationship between a 3D sensor, a camera, and an object. FIG. 10 is a diagram explaining an example of a captured image of an object. FIG. 11 is a diagram explaining an example of a converted image obtained by performing viewpoint conversion on the captured image shown in FIG. 10. FIG. 12 is a flowchart showing an example of processing of an object identification method using a point cloud. FIG. 13 is a flowchart showing an example of processing of an object identification method according to an embodiment.

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings, but the present disclosure is not limited thereto. The components of each embodiment described below can be combined as appropriate. In addition, some components may not be used.

[0012] [Embodiment] <Work Vehicle> A work vehicle identifies an object from objects present around the work vehicle at a work site. In this embodiment, the work vehicle is, for example, a wheel loader or a forklift.

[0013] 1 is a side view showing a work vehicle according to an embodiment. In this embodiment, the work vehicle 100 is a wheel loader. The work vehicle 100 may be a manned vehicle operated by a driver, or may be an unmanned vehicle operated without a driver. The wheel loader 100 performs loading work, loading transported objects picked up by a work implement 110.

[0014] In the embodiment, the object identified by the wheel loader 100 is a dump truck. The object is not limited to a dump truck and may be a transported object. The transported object may be, for example, a pallet, a skid, or a container. The transported object has a pair of fork insertion holes.

[0015] In the following description, the terms "up-down direction," "vehicle width direction," and "front-rear direction" are used to describe the positional relationship of each part. The "up-down direction" refers to the direction perpendicular to the contact surface of the tire 106 that contacts the ground RS. The "vehicle width direction" refers to the direction parallel to the rotation axis of the wheel 105. The "front-rear direction" refers to the direction perpendicular to the "up-down direction" and the "vehicle width direction."

[0016] "Up" refers to one direction in the up-down direction, and refers to the direction in which the cab 103R is located relative to the contact surface of the tire 106. "Down" refers to the opposite direction of "up" in the up-down direction. "Left" refers to one direction in the vehicle width direction. "Right" refers to the opposite direction of "left" in the vehicle width direction. "Front" refers to one direction in the front-to-rear direction, and refers to the direction in which the work implement 110 is located relative to the cab 103R. "Rear" refers to the opposite direction of "front" in the front-to-rear direction.

[0017] As shown in FIG. 1 , the wheel loader 100 includes a vehicle body 102 , a traveling device 104 , and a work implement 110 .

[0018] The vehicle body 102 includes a front vehicle body portion 102F and a rear vehicle body portion 102R disposed rearward of the front vehicle body portion 102F. The front vehicle body portion 102F and the rear vehicle body portion 102R are flexibly connected via a joint mechanism 109.

[0019] The traveling device 104 supports the vehicle body 102 and travels on the ground surface RS. The traveling device 104 has a power source 104A, a steering device 104B, wheels 106, and a power transmission device 108.

[0020] The power source 104A generates power for driving the wheel loader 100. The power source 104A is, for example, an electric motor.

[0021] The steering device 104B is provided to the joint mechanism 109 in order to steer the wheel loader 100. When the steering device 104B is actuated, the front vehicle body 102F bends relative to the rear vehicle body 102R. When the front vehicle body 102F bends relative to the rear vehicle body 102R, the traveling direction of the wheel loader 100 is adjusted. The steering device 104B is, for example, a hydraulic cylinder.

[0022] The wheels 106 include two front wheels 106F rotatably supported on the vehicle body front portion 102F, and two rear wheels 106R rotatably supported on the vehicle body rear portion 102R.

[0023] In the embodiment, the left-right direction is a direction parallel to the rotation axes of the front wheels 106F and rear wheels 106R when the wheel loader 100 travels in a straight line. The up-down direction is a direction perpendicular to the contact surfaces of the front wheels 106F and rear wheels 106R. The front-rear direction is a direction perpendicular to both the left-right direction and the up-down direction.

[0024] The power transmission device 108 transmits the driving force of the power source 104A to the wheels 106. The power transmission device 108 changes the output from the power source 104A, thereby accelerating or decelerating the wheel loader 100. The power transmission device 108 switches the rotation direction of the wheels 106, thereby causing the wheel loader 100 to move forward or backward.

[0025] The work implement 110 is supported by the vehicle body front section 102F. At least a portion of the work implement 110 is disposed forward of the vehicle body front section 102F. The work implement 110 has a boom 111, a bucket 112, a bell crank 113, a link 114, a boom cylinder 115, and a bucket cylinder 116.

[0026] The boom 111 is rotatably connected to the front vehicle body 102F. The bell crank 113 is connected to the bucket 112 via a link 114. The bucket 112 is rotatably connected to the tip of the boom 111. The bucket 112 is a working member having a tip 112B including a cutting edge. The bucket 112 is positioned forward of the front tires 106F.

[0027] The boom cylinder 115 is a hydraulic cylinder that generates power to operate the boom 111. The boom cylinder 115 extends and retracts, causing the boom 111 to move up and down.

[0028] The bucket cylinder 116 is a hydraulic cylinder that generates power to operate the bucket 112. The bucket 112 performs a dumping operation or a tilting operation when the bucket cylinder 116 extends and retracts. When the bucket 112 performs a tilting operation, the bucket 112 scoops up the excavated material. When the bucket 112 performs a dumping operation, the excavated material held in the bucket 112 is ejected from the bucket 112.

[0029] <Work vehicle identification system> Figure 2 is a functional block diagram showing the identification system according to the first embodiment. The identification system 1 for the wheel loader 100 comprises a three-dimensional sensor (object detection sensor) 11, a camera 12, an output unit 19, and a controller 20. The three-dimensional sensor 11, the camera 12, the output unit 19, and the controller 20 are connected to each other wirelessly or via a wire so as to be able to communicate data.

[0030] In the embodiment, the three-dimensional sensor 11 and the camera 12 are attached as a set to the wheel loader 100. The number of three-dimensional sensors 51 and cameras 52 is not particularly limited, and for example, a plurality of three-dimensional sensors 11 and cameras 12 may be arranged on the wheel loader 100.

[0031] The three-dimensional sensor 11 is attached to the wheel loader 100. In the embodiment, the three-dimensional sensor 11 is attached to the front of the wheel loader 100. The number of three-dimensional sensors 11 is not particularly limited, and, for example, a plurality of three-dimensional sensors 11 may be arranged on the wheel loader 100. The three-dimensional sensor 11 is an object detection sensor that detects objects present in the vicinity of the wheel loader 100. In the embodiment, the three-dimensional sensor 11 is an object detection sensor that detects objects present in front of the wheel loader 100. The three-dimensional sensor 11 is capable of detecting a range of, for example, several tens of meters. The three-dimensional sensor 11 transmits detected object detection data to the controller 20.

[0032] The three-dimensional sensor 11 acquires three-dimensional data of an object. The three-dimensional data of the object includes a point cloud consisting of multiple detection points defined on the surface of the object. The point cloud indicates the relative distance and relative position between the three-dimensional sensor 11 and each of the multiple detection points defined on the surface of the object. An example of the three-dimensional sensor 11 is a laser sensor (LiDAR: Light Detection and Ranging) that detects objects by emitting laser light. However, the three-dimensional sensor 11 may also be another non-contact sensor, such as an infrared sensor that detects objects by emitting infrared light or a radar sensor (RADAR: Radio Detection and Ranging) that detects objects by emitting radio waves.

[0033] The camera 12 is attached to the wheel loader 100. The camera 12 is capable of capturing images of the periphery of the wheel loader 100. In this embodiment, the camera 12 is a camera that captures images of objects present in front of the wheel loader 100. The camera 12 is capable of capturing images of a range of, for example, several tens of meters. The camera 12 is disposed in a position where it can capture an image of the object. The camera 12 transmits image data of the captured image to the controller 20.

[0034] The output unit 19 outputs the identification result. The output unit 19 is, for example, a monitor, a lamp, a buzzer, etc. The output unit 19 outputs the identification result under the control of the output control unit 29.

[0035] <Controller> The controller 20 identifies objects in the vicinity of the wheel loader 100. The controller 20 is capable of acquiring point cloud data, which is object detection data, from the three-dimensional sensor 11, which serves as an object detection sensor, and image data of images captured by the camera 12. The controller 20 includes a numerical calculation device (processor) such as a CPU (Central Processing Unit).

[0036] FIG. 3 is a block diagram showing a computer system according to the first embodiment. The controller 20 includes a computer system 1000. The computer system 1000 includes a processor 1001 such as a CPU, a main memory 1002 including a nonvolatile memory such as a read-only memory (ROM) and a volatile memory such as a random access memory (RAM), a storage 1003, and an interface 1004 including an input / output circuit. The functions of the controller 20 described above are stored as a program in the storage 1003. The processor 1001 reads the program from the storage 1003, loads it into the main memory 1002, and executes the above-described processing in accordance with the program. The program may be distributed to the computer system 1000 via a network.

[0037] In the embodiment, the controller 20 includes a point cloud identification unit 21 , an image processing unit 22 , and an output control unit 29 .

[0038] The point cloud identification unit 21 acquires object detection data from the three-dimensional sensor 11. The point cloud identification unit 21 identifies an object based on the object detection data. The point cloud identification unit 21 identifies the position and orientation of the object from the object detection data. Note that the method of identification in the point cloud identification unit 21 is not limited.

[0039] An example of an identification method in the point cloud identification unit 21 will be described. The point cloud identification unit 21 identifies an object based on point cloud data, which is object detection data from the three-dimensional sensor 11. More specifically, the point cloud identification unit 21 estimates a plane that constitutes at least a part of the object's outer shape from the acquired point cloud data. The point cloud identification unit 21 projects the point cloud data onto a projection plane and reconstructs a portion of the projection plane with high point cloud density into a three-dimensional representation to estimate the object's plane. The point cloud identification unit 21 then extracts a point cloud located on the estimated plane to generate a binary image. The point cloud identification unit 21 then detects, from the generated binary image, a shape that resembles a binary image of a feature pattern that indicates the shape of the plane portion of the object's outer shape, which has been stored in advance, to identify the object.

[0040] A more detailed description will be given of an example of the configuration of the point cloud identifying unit 21. The point cloud identifying unit 21 includes a sensor data acquiring unit 211, a storage unit 212, and an identifying unit 223.

[0041] The sensor data acquisition unit 211 acquires object detection data from the three-dimensional sensor 11. The object detection data acquired by the sensor data acquisition unit 211 is, for example, a point cloud made up of a plurality of detection points. Hereinafter, the object detection data will be referred to as point cloud data.

[0042] The storage unit 212 includes at least one of a RAM, a ROM, a flash memory, and a hard disk drive. The storage unit 212 stores data used in the processing of the controller 20.

[0043] The storage unit 212 stores a search range, which is a range in which the object is presumed to exist, and stores a feature pattern of the object for identifying the object.

[0044] The search range is a range for extracting a plane extraction point cloud used in processing to identify an object from point cloud data, which is object detection data. The point cloud data includes a point cloud of measurement distances and field of view angles of the three-dimensional sensor 11.

[0045] The feature pattern of the object is an image that shows the shape of a flat portion of the object's outer shape. In this embodiment, the feature pattern is a binary image of the side surface of the vessel of a dump truck.

[0046] The identification unit 223 identifies an object from the object detection data of the three-dimensional sensor 11. More specifically, the identification unit 223 estimates a plane that constitutes at least a part of the outline of the object from the point cloud data acquired by the sensor data acquisition unit 211, extracts a point cloud located on the estimated plane to generate a binary image, and identifies the object from the binary image.

[0047] The identification unit 223 projects the point cloud data onto a projection plane, and reconstructs the part of the projection plane where the point cloud density is high into three dimensions, thereby estimating the plane of the object.

[0048] The identification unit 223 detects from the generated binary image a shape that resembles a binary image of a feature pattern that indicates the shape of a plane portion of the pre-stored outline of the object, and identifies the object.

[0049] 4 is a schematic diagram showing the operation of the work vehicle according to the first embodiment. The wheel loader 100 identifies the dump truck 200 to be loaded in order to load the transported goods picked up by the work implement 110. The wheel loader 100 identifies the vessel BE of the dump truck 200 based on the object detection data acquired from the three-dimensional sensor 11.

[0050] The object identification process performed by the identification unit 223 will be described with reference to FIGS. 5 to 8. FIG. 5 is a diagram illustrating an example of a point cloud representing an object. FIG. 6 is a schematic diagram illustrating an example of the object identification process. FIG. 7 is a diagram illustrating an example of a binary image of the object. FIG. 8 is a diagram illustrating an example of a feature pattern of the object. As shown in FIG. 5, the point cloud is composed of a plurality of detection points defined on the surface of the object. First, as shown in FIG. 6, the identification unit 223 estimates a plane constituting the object from the point cloud data. As shown in FIG. 6(a1), a point cloud within a search range is extracted from the point cloud data to obtain a point cloud of the object. Then, as shown in FIG. 6(a2), the identification unit 223 projects the plane extraction point cloud onto the xy plane of a coordinate system in which the z-axis is perpendicular to the ground plane. FIG. 6(a3) shows the point cloud projected onto the xy plane. The point cloud surrounded by a dashed line on the xy plane corresponds to the point cloud on the side of the dump truck, which is surrounded by a dashed line and perpendicular to the xy plane. The point cloud on the plane perpendicular to the xy plane of the dump truck is aggregated onto a single line segment to form a high density. Then, as shown in FIG. 6( a4 ), the identification unit 223 converts the points back to their original three-dimensional positions, leaving only the area surrounded by the dashed line where the point cloud density is high on the xy plane. In this way, the identification unit 223 estimates the plane that constitutes the object by extracting a planar point cloud, which is a point cloud on the plane that constitutes the object. In this example, the planar point cloud is a point cloud on the side surface of the vessel.

[0051] As shown in Fig. 7 , the identification unit 223 generates a binary image based on the extracted plane point cloud. The identification unit 223 generates a binary image by cropping the plane peripheral point cloud, which is a point cloud around the estimated plane, as a point cloud for generating a binary image. The identification unit 223 detects a shape similar to the feature pattern indicating the vessel shape shown in Fig. 8 in the generated binary image. If a similar shape is found, the identification unit 223 identifies the overall shape of the dump truck based on the detected position and the vessel shape of the feature pattern.

[0052] The image processing unit 22 identifies an object from an image captured by the camera 12. More specifically, the image processing unit 22 identifies the object based on a converted image obtained by converting the viewpoint of the captured image so that it is viewed from a predetermined direction using an identification result based on the point cloud data. The image processing unit 22 includes a viewpoint conversion unit 23, a feature extraction unit 24, an image classification unit 25, and a consistency confirmation unit 26.

[0053] In this embodiment, before the viewpoint conversion unit 23 executes the viewpoint conversion process, the relative positional relationship between the three-dimensional sensor 11 and the camera 12 is calibrated.

[0054] The viewpoint conversion unit 23 performs viewpoint conversion on the captured image captured by the camera 12 to create a converted image viewed from a predetermined direction. More specifically, the viewpoint conversion unit 23 performs viewpoint conversion on the captured image based on the position and orientation of the object identified based on the point cloud data and the relative positional relationship between the 3D sensor 11 and the camera 12.

[0055] The predetermined direction is, for example, a direction that makes it easy to classify the object by image classification. The predetermined direction is, for example, a direction viewed from the front when the wheel loader 100 performs work on the object. The predetermined direction is, for example, a direction in which the wheel loader 100 directly faces the object. If the object is a dump truck, the predetermined direction is a direction in which the side of the dump truck is viewed from the front. If the object is an object to be transported, the predetermined direction is a direction in which the wheel loader 100 directly faces the fork insertion holes.

[0056] 9 is a schematic diagram showing the positional relationship between the three-dimensional sensor, the camera, and the target. In FIG. 9, the target is a dump truck 200. Because the three-dimensional sensor 11 and the camera 12 are attached at different positions, a misalignment occurs between the coordinate system of the three-dimensional sensor 11 and the coordinate system of the camera 12. Points P1, P2, P3, and P4 are defined as the four corners of the vessel side of the dump truck 200. The positions of points P1, P2, P3, and P4 differ when expressed in the coordinate system of the three-dimensional sensor 11 and when expressed in the coordinate system of the camera 12.

[0057] The viewpoint conversion method will be described in detail. First, the viewpoint conversion unit 23 projects the positions of four or more points located on the plane estimated by the point cloud identification unit 21 onto the captured image. In this embodiment, the point cloud identification unit 21 projects the positions of the four corners of the estimated plane onto the captured image. When the object is a box-shaped pallet, the point cloud identification unit 21 projects the estimated positions of the four corners of one surface of the pallet onto the captured image. When the object is a dump truck, the point cloud identification unit 21 projects the estimated positions of the four corners of the vessel of the dump truck onto the captured image.

[0058] The viewpoint conversion unit 23 then performs viewpoint conversion based on the multiple points projected onto the captured image. In this embodiment, the viewpoint conversion unit 23 converts a shape whose outer shape is defined by the multiple points projected onto the captured image into a captured image viewed from a predetermined direction. The viewpoint conversion unit 23 performs viewpoint conversion, for example, so that the outer shape defined by the multiple points becomes a shape in a front view that is stored in advance in a storage unit. The viewpoint conversion unit 23 outputs converted image data of the converted image after viewpoint conversion to the feature extraction unit 24.

[0059] Viewpoint conversion will be described using an image of the dump truck 200 with reference to Fig. 10 and Fig. 11. Fig. 10 is a diagram illustrating an example of a captured image of an object. Fig. 11 is a diagram illustrating an example of a converted image obtained by converting the captured image shown in Fig. 10 into a viewpoint. Fig. 10 is a diagram in which points P1, P2, P3, and P4 at the four corners of the vessel side surface of the dump truck 200 estimated by the point cloud identification unit 21 are projected onto the captured image. Fig. 11 is a converted image obtained by converting the viewpoint so that points P1, P2, P3, and P4 at the four corners of the vessel side surface are viewed from the side of the dump truck, which is in a direction directly facing the vessel.

[0060] The feature extraction unit 24 extracts features from the transformed image after viewpoint transformation, and outputs feature data of the extracted features to the image classification unit 25.

[0061] The feature amount is, for example, a numerical representation of an edge (color boundary) in an image.

[0062] The image classification unit 25 classifies the converted images by machine learning using the feature data to determine whether the converted images are target images. The image classification unit 25 classifies the converted images based on the feature extracted by the feature extraction unit 24 and learned parameters of the machine learning model stored in a storage unit (not shown).

[0063] The method for identifying an object based on a converted image is not limited to the above, and the object may be identified by, for example, pattern matching.

[0064] The consistency checking unit 26 checks the consistency of the object identification results. The consistency checking unit 26 checks the consistency of the objects identified based on the point cloud data. The consistency checking unit 26 checks the consistency between the identification result by the point cloud identification unit 21 and the identification result, which is the classification result by the image classification unit 25. The consistency checking unit 26 checks the consistency between the objects identified based on the point cloud data and the objects identified based on the converted image. If the objects identified by the point cloud identification unit 21 are classified as the same object by the image classification unit 25, the consistency checking unit 26 determines that the object identification results are consistent. If the objects identified by the point cloud identification unit 21 are not classified as the same object by the image classification unit 25, the consistency checking unit 26 determines that the object identification results are inconsistent. Note that if the identification result by the point cloud identification unit 21 and the identification result, which is the classification result by the image classification unit 25, differ, the consistency checking unit 26 may prioritize the identification result by the point cloud identification unit 21.

[0065] The output control unit 29 controls the output of the identification result by the point cloud identification unit 21 via the output unit 19. For example, when an object is identified, the output control unit 29 outputs a video signal that causes an image showing the identified object to be displayed on a monitor. For example, when an object is identified, the output control unit 29 outputs a control signal that causes a lamp to light up. For example, when an object is identified, the output control unit 29 outputs a control signal that causes a buzzer to sound.

[0066] <Processing by Controller> When the object identification process is started while the wheel loader 100 is running, the processes shown in Fig. 12 and Fig. 13 are executed. Fig. 12 is a flowchart showing an example of the process of the object identification method using point clouds. The process of the flowchart shown in Fig. 12 is the object identification process using point clouds. Note that the processes from step ST19 to step ST21 are not essential processes.

[0067] The controller 20 acquires the point cloud (step ST11). More specifically, the controller 20 receives the point cloud data from the three-dimensional sensor 11 via the sensor data acquisition unit 211. The controller 20 proceeds to step ST12.

[0068] The controller 20 crops the plane extraction point cloud (step ST12). More specifically, the controller 20 extracts, from the point cloud data, a point cloud included in a search range where it is estimated that the target exists, as the plane extraction point cloud using the identification unit 223. The controller 20 proceeds to step ST13.

[0069] The controller 20 downsamples the plane extraction point cloud (step ST13). More specifically, the controller 20 downsamples the plane extraction point cloud using voxel downsampling, which is an example of a point cloud data downsampling method, by the identification unit 223. The controller 20 proceeds to step ST14.

[0070] The controller 20 extracts a point cloud on a plane and estimates a plane (step ST14). More specifically, the controller 20 extracts a point cloud on a plane, which is a point cloud on the vessel side surface, from the plane extraction point cloud downsampled by the identification unit 223, and estimates the vessel side surface, which is a plane. The controller 20 proceeds to step ST15.

[0071] The controller 20 crops the point cloud for generating a binary image (step ST15). More specifically, the controller 20 causes the identification unit 223 to crop the plane peripheral point cloud, which is the point cloud around the estimated plane, as a point cloud for generating a binary image. The controller 20 proceeds to step ST16.

[0072] The controller 20 generates a binary image of the plane peripheral point cloud (step ST16). More specifically, the controller 20 uses the identification unit 223 to generate a binary image on the yz plane of a coordinate system in which the normal vector of the estimated plane coincides with the x-axis direction, based on the cropped plane peripheral point cloud. The yz plane is a plane that includes the estimated plane. The controller 20 proceeds to step ST17.

[0073] The controller 20 detects features of the target from the binary image (step ST17). More specifically, the controller 20 detects a shape similar to the feature pattern in the binary image using the recognition unit 223. The controller 20 proceeds to step ST18.

[0074] The controller 20 restores the three-dimensional position (step ST18). More specifically, the controller 20 uses the identification unit 223 to identify the overall position, posture, and shape of the dump truck based on the detected position in the binary image and the vessel shape of the feature pattern. The controller 20 proceeds to step ST19.

[0075] The controller 20 checks whether the dump truck is within the effective field of view of the sensor (step ST19). More specifically, the controller 20 checks, using the identification unit 223, whether the entire position of the dump truck is within the effective field of view of the three-dimensional sensor 11. This is to confirm that the identification performance is guaranteed, since the identification performance decreases around the boundary of the field of view of the three-dimensional sensor 11.

[0076] The controller 20 checks the consistency with the predicted value (step ST20). More specifically, the controller 20 uses the identification unit 223 to compare the predicted position of the target predicted from the previous estimation result with the currently estimated position of the target, and checks whether the difference is equal to or less than a threshold.

[0077] The controller 20 calculates the variance and likelihood of the point cloud classification (step ST21).

[0078] In this way, the controller 20 identifies the vessel of the target dump truck from the point cloud data.

[0079] Following the object identification process based on the point cloud data described above, the process shown in Fig. 13 is executed. Fig. 13 is a flowchart showing an example of the process of the object identification method according to the embodiment. The process of the flowchart shown in Fig. 13 is object identification process by image classification using learned parameters. Note that the process of step ST55 is not essential.

[0080] The controller 20 acquires an identification result based on the point cloud (step ST51). More specifically, the controller 20 acquires point cloud data consisting of a plurality of detection points, which is object detection data, from the 3D sensor 11 using the point cloud identification unit 21, and identifies the object from the point cloud data. The process in step ST51 is specifically shown in the flowchart illustrated in FIG. 11 above. The controller 20 proceeds to step ST52.

[0081] The controller 20 performs viewpoint conversion on the image (step ST52). More specifically, the controller 20 performs viewpoint conversion on the image captured by the camera 12 to convert it into an image viewed from a predetermined direction using the viewpoint conversion unit 23. The controller 20 proceeds to step ST53.

[0082] The controller 20 extracts features (step ST53). More specifically, the controller 20 extracts features from the converted image after viewpoint conversion using the feature extraction unit 24. The feature extraction unit 24 outputs feature data of the extracted features to the image classification unit 25. The controller 20 proceeds to step ST54.

[0083] The controller 20 classifies the image (step ST54). More specifically, the controller 20 classifies the converted image using the image classifying unit 25 based on the feature extracted by the feature extracting unit 24 and the trained parameters of the machine learning model stored in a storage unit (not shown). The controller 20 proceeds to step ST55.

[0084] The controller 20 checks the consistency (step ST55). More specifically, the controller 20 checks the consistency between the classification result by the point cloud classification unit 21 and the classification result, which is the classification result by the image classification unit 25, using the consistency checking unit 26.

[0085] In this way, the calculation load is reduced and the object is identified from the image captured by the camera 12.

[0086] As described above, in the embodiment, the captured image is subjected to viewpoint conversion using the identification result based on point cloud data, and the object is identified based on the converted image obtained by the viewpoint conversion. According to the embodiment, by not performing a process with a high computational load such as deep learning, the computational load can be reduced, and the object can be identified from the captured image captured by the camera 12.

[0087] In the embodiment, the object is identified based on the transformed image that has undergone viewpoint transformation, and therefore can be identified with high accuracy.

[0088] In the embodiment, the captured image is subjected to viewpoint conversion based on the position and orientation of the object identified based on the point cloud data and the relative positional relationship between the 3D sensor 11 and the camera 12. In the embodiment, by using the identification result based on the point cloud data, the captured image can be subjected to viewpoint conversion more appropriately.

[0089] In the embodiment, the viewpoint is converted so that the object is viewed from a direction seen from the front when the wheel loader 100 is performing work on the object. According to the embodiment, the object can be identified based on a converted image in which the viewpoint is converted so that the object is viewed from a direction that makes it easy to identify the object. The embodiment enables identification with high accuracy.

[0090] In the embodiment, a plurality of points located on a plane estimated from point cloud data are projected onto a captured image, and viewpoint conversion is performed based on the plurality of points. According to the embodiment, viewpoint conversion can be performed with reduced calculation load.

[0091] 1...Identification system, 11...3D sensor (object detection sensor), 12...Camera, 19...Output unit, 20...Controller, 21...Point cloud identification unit, 211...Sensor data acquisition unit, 212...Memory unit, 213...Identification unit, 22...Image processing unit, 221...Viewpoint conversion unit, 222...Feature extraction unit, 223...Image classification unit, 224...Consistency verification unit, 29...Output control unit, 100...Wheel loader (work vehicle), 102...Vehicle body, 102F...Front vehicle body, 102R...Rear vehicle body, 103...Driver's cab, 103R...Driver's compartment, 104...Traveling device, 104A...Engine, 104B...Brake , 104C...steering cylinder, 105...wheel, 105F...front wheel, 105R...rear wheel, 106...tire, 106F...front tire, 106R...rear tire, 107...transmission, 108...parking brake, 109...joint mechanism, 110...work implement, 111...boom, 112...bucket, 112B...tip, 113...bell crank, 114...link, 115...boom cylinder, 116...bucket cylinder, 1000...computer system, 1001...processor, 1002...main memory, 1003...storage, 1004...interface.

Claims

1. An identification system for identifying an object from objects present in the vicinity of a work vehicle, comprising: an object detection sensor attached to the work vehicle for detecting objects present in the vicinity of the work vehicle; a camera attached to the work vehicle for photographing the vicinity of the work vehicle; and a controller, wherein the controller acquires point cloud data, which is object detection data, from the object detection sensor and image data of an image captured by the camera, identifies an object based on the point cloud data, transforms the captured image using an identification result based on the point cloud data, and identifies the object based on the transformed image.

2. The identification system according to claim 1, wherein the converted image is an image obtained by performing viewpoint conversion on the captured image so that the captured image is viewed from a predetermined direction using the identification result based on the point cloud data.

3. The identification system of claim 2, wherein the controller performs viewpoint conversion of the captured image based on the position and orientation of the object identified based on the point cloud data and the relative positional relationship between the object detection sensor and the camera.

4. The identification system according to claim 2 or 3, wherein the predetermined direction is a direction seen from the front when the work vehicle is performing work on the target.

5. The identification system of claim 2 or 3, wherein the controller estimates a plane that constitutes at least a part of the object's outline from the point cloud data, extracts a point cloud located on the estimated plane to generate a binary image, identifies the object from the binary image, projects a plurality of points located on the estimated plane onto the captured image, and performs viewpoint transformation based on the plurality of points projected onto the captured image.

6. The identification system according to claim 1, wherein the controller verifies consistency between the object identified based on the point cloud data and the object identified based on the converted image.

7. An identification method for identifying an object from objects present in the vicinity of a work vehicle, comprising the steps of: acquiring point cloud data, which is object detection data, from an object detection sensor attached to the work vehicle for detecting objects present in the vicinity of the work vehicle; acquiring image data of a captured image from a camera attached to the work vehicle for capturing images of the vicinity of the work vehicle; identifying the object based on the point cloud data; transforming the captured image using an identification result based on the point cloud data; and identifying the object based on the transformed image.

8. A work vehicle comprising: a vehicle body; an object detection sensor attached to the work vehicle for detecting objects present around the work vehicle; a camera attached to the work vehicle for photographing the area around the work vehicle; and a controller, wherein the controller acquires point cloud data, which is object detection data, from the object detection sensor and image data of an image captured by the camera, identifies an object based on the point cloud data, transforms the captured image using the identification result based on the point cloud data, and identifies the object based on the transformed image.

Images