System and program

Abstract

According to the invention, the processing time required for detecting the loading platform area is reduced. The system includes a correction unit, an approach section detection unit, and a receptacle detection unit. The correction unit corrects the distance image including a receiving portion on which an object is loaded by the work device in accordance with a movement of the work device in the distance image. A proximity section detection unit detects a proximity section of the accommodating portion based on a distance histogram generated from the corrected distance image. The housing portion detection unit detects the housing portion based on the detected proximity section and the distance image.

Description

Technical Field

[0001] The present disclosure relates to a system and a program. Background Art

[0002] The work of loading soil, sand, etc. on a cargo bed of a dump truck or the like using an excavator is performed at a construction site or the like. At this time, since the boom of the excavator is adjacent to the cargo bed, the boom and the cargo bed may collide. In order to prevent such a collision accident, a system is proposed that acquires a captured image showing a target to be loaded and unloaded of a transported object of a working machine such as an excavator, and specifies at least one surface of the target to be loaded and unloaded (for example, see Patent Document 1).

[0003] Reference List

[0004] Patent Literature

[0005] Patent Document 1: JP 2020-126363 A Summary of the invention

[0006] Technical issues

[0007] However, in the above-mentioned related art, since the point cloud data of the cargo bed area is generated using the result of semantic segmentation, there is a problem of long processing time.

[0008] Therefore, the present disclosure proposes a system and process that reduces the processing time for detecting a cargo bed area.

[0009] Solution to the problem

[0010] The system according to the present disclosure includes a correction unit, a proximity detection unit, and a receiving portion detection unit. The correction unit corrects the distance image according to the movement of the working device in the distance image including the receiving portion, on which the object is placed by the working device. The proximity detection unit detects the proximity of the receiving portion based on a distance histogram generated according to the corrected distance image. The receiving portion detection unit detects the receiving portion based on the detected proximity and the distance image.

[0011] In addition, the program according to the present disclosure includes: a process of correcting a distance image according to a movement of a working device in a distance image including a housing portion, on which an object is placed by the working device; a process of detecting a proximity portion of the housing portion based on a distance histogram generated based on the corrected distance image; and a process of detecting the housing portion based on the detected proximity portion and the distance image. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 : is a diagram showing a configuration example of a working device according to an embodiment of the present disclosure.

[0013] Figure 2 is a diagram showing a configuration example of a detection device according to an embodiment of the present disclosure.

[0014] Figure 3 is a diagram showing a configuration example of a cargo bed detection processing unit according to an embodiment of the present disclosure.

[0015] Figure 4 is a diagram illustrating an example of a target area image according to an embodiment of the present disclosure.

[0016] Figure 5A 2 is a diagram illustrating an example of correction according to an embodiment of the present disclosure.

[0017] Figure 5B 2 is a diagram illustrating an example of correction according to an embodiment of the present disclosure.

[0018] Figure 5C 2 is a diagram illustrating an example of correction according to an embodiment of the present disclosure.

[0019] Fig. 6A is a diagram illustrating an example of detection of a cargo bed area according to an embodiment of the present disclosure.

[0020] Figure 6B is a diagram showing an example of detection of a cargo bed area according to an embodiment of the present disclosure.

[0021] Figure 6C is a diagram showing an example of detection of a cargo bed area according to an embodiment of the present disclosure.

[0022] Figure 7 is a diagram illustrating an example of a processing procedure of a cargo bed area detection process according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0024] [Configuration of working equipment]

[0025] Figure 1 1 is a diagram showing a configuration example of a working device according to an embodiment of the present disclosure. The figure assumes a work of loading an object 30 such as soil and sand onto a cargo bed 21 of a dump truck 20 using an excavator 10. The working device 11 shown in the figure is provided on a rotating body 15 of the excavator 10. The rotating body 15 is supported by a mobile body 16.

[0026] The working device 11 includes a boom 14, an arm 12, and a bucket 13. The boom 14 is attached to a rotating body 15. The arm 12 is attached to one end of the boom 14. The bucket 13 is attached to one end of the arm 12. The bucket 13 is a container that holds an object 30 such as soil and sand.

[0027] The process of the operation is described. First, the bucket 13 scoops up and holds the object 30. Then, the excavator 10 approaches the dump truck 20 while the object 30 is held by the bucket 13. At this time, the excavator 10 moves to the vicinity of the dump truck 20 using the mobile body 16, and moves the working device 11 to the upper part of the cargo bed 21 according to the rotation of the rotating body 15. Thereafter, the excavator 10 operates the bucket 13 to place the object 30 on the cargo bed 21.

[0028] At the time of this operation, since the arm 12 and the bucket 13 are adjacent to the cargo bed 21, the arm 12 and the bucket 13 may collide with the cargo bed 21. Specifically, when the excavator 10 is remotely controlled, the possibility of collision increases. Therefore, a detection device is provided to detect the position of the cargo bed 21 from the side of the excavator 10. With this detection device, the position of the cargo bed 21 can be grasped, and the operator can be warned. Note that the cargo bed 21 is an embodiment of the accommodation portion described in the claims.

[0029] [Configuration of the detection device]

[0030] Figure 2 1 is a diagram showing a configuration example of a detection device according to an embodiment of the present disclosure. The figure is a block diagram showing a configuration example of a detection device 1. The detection device 1 includes a camera 110, a distance measuring sensor 120, a target area extraction unit 130, and a cargo bed area detection unit 200. Note that the detection device 1 is an embodiment of the system described in the claims.

[0031] The camera 110 is provided at the front of the rotating body 15 of the excavator 10, and generates an image near the working device 11. The camera 110 outputs the generated image to the target area extraction unit 130.

[0032] The target area extraction unit 130 extracts an image of the target area from the image output by the camera 110. The image of the excavator 10 corresponds to the target area. The target area extraction unit 130 retrieves the area imaged by the excavator 10 from the image, processes the area into data of a bounding box, and outputs the data as an image of the target area. For example, the area imaged by the excavator 10 can be retrieved by AI (artificial intelligence). The target area extraction unit 130 outputs the target area image to the point cloud data generation unit 210 of the cargo bed area detection unit 200.

[0033] The distance measuring sensor 120 is provided at the front of the rotating body 15 of the excavator 10, and generates a distance image near the working device 11. The distance image is also called a depth map, and is an image reflecting distance information for each pixel. The distance measuring sensor 120 outputs the generated distance image to the point cloud data generating unit 210 of the cargo bed area detecting unit 200.

[0034] The cargo bed area detection unit 200 detects the cargo bed area based on the target area image and the distance image. The cargo bed area detection unit 200 includes a point cloud data generation unit 210, a cargo bed detection processing unit 220, and a data conversion unit 230.

[0035] The point cloud data generating unit 210 generates point cloud data of the dump truck 20 including the cargo bed 21 based on the target area image and the depth map. Here, the point cloud data is data configured by representing an image of an object using a plurality of points. The point cloud data generating unit 210 extracts an area of ​​the distance image included in the target area image to generate point cloud data. A known method can be used to generate the point cloud data. The point cloud data generating unit 210 outputs the generated point cloud data to the cargo bed detection processing unit 220.

[0036] The cargo bed detection processing unit 220 detects the cargo bed area based on the point cloud data. The cargo bed detection processing unit 220 outputs the detected cargo bed area to the data conversion unit 230. The detailed configuration of the cargo bed detection processing unit 220 is described below.

[0037] The data conversion unit 230 converts the cargo bed area into point cloud data. The data conversion unit 230 may also correct the point cloud data. The data conversion unit 230 outputs the point cloud data of the cargo bed area to an external device.

[0038] [Configuration of cargo bed detection processing unit]

[0039] Figure 3 2 is a block diagram showing a configuration example of a cargo bed detection processing unit 220. The cargo bed detection processing unit 220 includes a correction unit 221, a proximity detection unit 222, and a cargo bed detection unit 223.

[0040] The correction unit 221 corrects the position of the cargo bed 21 in the distance image. The correction unit 221 corrects the point cloud data according to the movement of the working device 11. For example, when the rotating body 15 of the excavator 10 rotates, the angle of the cargo bed 21 relative to the working device 11 changes, and the distance from the cargo bed 21 changes, resulting in an error. Therefore, the change in the angle of the cargo bed 21 relative to the working device 11 is corrected to reduce the error. The correction unit 221 outputs the corrected point cloud data to the approaching portion detection unit 222.

[0041] The approaching portion detection unit 222 detects an area where the cargo bed 21 approaches the working device 11 based on the point cloud data. The approaching portion detection unit 222 generates a distance image based on the point cloud data. Subsequently, the approaching portion detection unit 222 generates a distance histogram representing distances as frequencies based on the generated distance image. The mode value of the distance histogram is assumed to be the distance to the approaching portion, which is an area where the cargo bed 21 is adjacent to the working device 11, and an area of ​​the distance image in a distance range close to the mode value is detected as a distance image of the approaching portion. The approaching portion detection unit 222 outputs the detected distance image to the cargo bed detection unit 223.

[0042] The cargo bed detection unit 223 detects an image of the cargo bed 21 based on the distance image output from the approach portion detection unit 222. The cargo bed detection unit 223 detects the cargo bed by generating an image of the cargo bed 21 based on the distance image. Note that the cargo bed detection unit 223 is an embodiment of an accommodation portion detection unit.

[0043] The image of the cargo bed 21 can be generated as follows. First, the cargo bed detection unit 223 normalizes the distance with respect to the distance image. For example, this can be performed by converting the distance data into grayscale data having a predetermined bit width. Specifically, the conversion to 8-bit grayscale data can be performed by dividing the pixel value of the distance image by the maximum value of the distance, and then multiplying the pixel value by the value "255". Subsequently, the cargo bed detection unit 223 performs edge detection processing on the normalized image to generate an image of the edge of the approaching portion. For example, the Canny method can be applied to the detection of the edge. Subsequently, the cargo bed detection unit 223 performs contour extraction processing on the edge image to generate the contour of the approaching portion. Known methods can be applied to the contour extraction processing. Subsequently, the cargo bed detection unit 223 detects the area of ​​the distance image inside the contour using the detected contour as a mask. The cargo bed detection unit 223 outputs the detected area of ​​the distance image as an image of the area of ​​the cargo bed 21.

[0044] [Configuration of cargo bed detection processing unit]

[0045] Figure 4 301 is a diagram showing an example of a target area image according to an embodiment of the present disclosure. The figure is a diagram showing an example of a target area image output from the target area extraction unit 130. The dotted rectangular area in the image of the dump truck 20 in the figure represents a bounding box of the target area image 301.

[0046] [Correction Processing]

[0047] FIG. 5A to FIG. 5C 21 are diagrams showing an example of correction according to an embodiment of the present disclosure. These diagrams are diagrams showing an example of correction in the correction unit 221. Figure 5A1 is a diagram showing the excavator 10 and the dump truck 20 viewed from above. The excavator 10 in the figure shows an embodiment in which the working device 11 is arranged to be offset by an angle θ relative to the normal direction of the side surface of the cargo bed 21 of the dump truck 20 by the swing of the rotating body 15. Figure 5B 3 is a diagram showing the correction process. The point cloud data 302 of the cargo bed 21 is corrected by rotating by θ.

[0048] The detection of the angle θ can be performed as follows. First, a normal vector is generated for each point cloud of the point cloud data. The inner product of the normal vector and the unit vector in the direction parallel to the working device 11 is calculated to generate an angle map. The angle map represents the relative angle between each pixel point and the working device 11. An angle histogram is generated based on the angle map. The mode of the angle histogram can be detected as the angle θ. Figure 5C An embodiment of an angle histogram is shown. The bar 303 in the figure represents the mode of the angle histogram.

[0049] [Cargo bed area inspection]

[0050] FIG. 6A to FIG. 6C is a schematic diagram illustrating an example of detection of a cargo bed area according to an embodiment of the present disclosure. Fig. 6A is shown based on Figure 4 304 is an image of point cloud data of a target area image 301 of an image of an image. An image 304 in the figure represents point cloud data of a dump truck 20 including a cargo bed 21. A white area in the figure represents point cloud data of the cargo bed 21. Figure 6B 305 is a diagram showing an image of a proximity portion. The image 305 is an image (distance image) of the proximity portion of the cargo bed 21 generated by the proximity portion detection unit 222. The area indicated by the dotted line in the figure represents an area excluded from the image 305. Figure 6C 3 is a diagram showing an embodiment of a distance histogram generated by the approaching portion detection unit 222. The bar 306 shown in the figure represents the mode of the distance histogram. The mode can be determined as the distance to the approaching portion of the cargo bed 21. This is because the approaching surface of the cargo bed 21 occupies the largest area in the target area image 301. The approaching portion of the cargo bed 21 can be detected by extracting a distance image using an area having a predetermined width relative to the distance mode as a cropping range.

[0051] [Cargo bed area detection and processing]

[0052] Figure 7is a diagram showing an example of a processing procedure of a cargo bed area detection process according to an embodiment of the present disclosure. The figure is a flowchart showing an example of a processing procedure in the cargo bed area detection unit 200. First, the point cloud data generation unit 210 generates point cloud data (step S101). Subsequently, the correction unit 221 generates an angle map (step S102). Subsequently, the correction unit 221 detects the relative angle (step S103). Subsequently, the correction unit 221 performs angle correction on the point cloud data (step S104). Subsequently, the approaching portion detection unit 222 generates a distance image (step S105). Subsequently, the approaching portion detection unit 222 generates a distance histogram (step S106). Subsequently, the approaching portion detection unit 222 extracts a distance image around the mode of the histogram (step S107). Subsequently, the cargo bed detection unit 223 extracts the contour of the approaching portion (step S108). Subsequently, the data conversion unit 230 converts the distance image into point cloud data (step S109). With the processing described above, the area of ​​the cargo bed 21 can be detected.

[0053] As described above, the cargo bed area detection unit 200 in the embodiment of the present disclosure detects the distance of the cargo bed 21 based on the point cloud data, and detects the area of ​​the cargo bed 21. Therefore, it is possible to simplify the image extraction process for the area of ​​the cargo bed 21.

[0054] Note that the configuration of the cargo bed area detection unit 200 is not limited to this embodiment. For example, the cargo bed area detection unit 200 can be applied to working devices other than the excavator 10. For example, the cargo bed area detection unit 200 can also be applied to a working device that transports wood used in forestry to the cargo bed of a truck. For example, the cargo bed area detection unit 200 can also be applied, for example, when an object is transported to a storage portion by a robot arm. The technology disclosed in the present invention can also be applied to the detection of the gripping point of a gripping target object when the object is gripped by a robot arm. Specifically, the cargo bed detection processing unit 220 can detect the gripping point of the gripping target object instead of the cargo bed (storage portion).

[0055] (Other variations)

[0056] The detection device 1 of this embodiment may be implemented by a dedicated computer system or a general-purpose computer system.

[0057] For example, the program for executing the above-described operation is stored in a computer-readable recording medium such as an optical disk, a semiconductor memory, a magnetic tape, or a flexible disk and is distributed. Then, for example, the control device is configured by installing the program in a computer and executing the above-described processing.

[0058] The communication program described above can be stored in a disk device included in a server device on a network (such as the Internet) so that it is feasible to download the communication program to a computer. The functions described above can be implemented by the cooperation of an OS (operating system) and application software. In this case, the part other than the OS can be stored in a medium and distributed, or the part other than the OS can be stored in a server device so that it is feasible to download the part to a computer.

[0059] In the various processes described in the embodiments, all or part of the processes described as being automatically performed may be manually performed, or all or part of the processes described as being manually performed may be automatically performed by a known method. In addition, unless otherwise specifically stated, the processing procedures, specific names, and information including various data and parameters described in the document and shown in the drawings may be optionally changed. For example, the various information shown in the drawings is not limited to the information shown.

[0060] The components of the illustrated device are conceptual in function and do not always need to be physically configured as shown in the figure. That is, the specific form of distribution and integration of the device is not limited to the form shown, and all or part of the device can be functionally or physically distributed and integrated into any unit according to various loads, usage conditions, etc. Note that such configuration by distribution and integration can be performed dynamically.

[0061] The embodiments described above can be combined as appropriate within the range where the processing contents do not contradict each other. The order of the steps shown in the flowcharts in the embodiments described above can be changed as appropriate.

[0062] For example, the present embodiment may be implemented as any component configuring a device or system, such as a processor used as a system LSI (large scale integration) or the like, a module using multiple processors, a unit using multiple modules, or a collection obtained by further adding other functions to the unit (i.e., a component as part of the device).

[0063] Note that in this embodiment, a system means a collection of multiple components (devices, modules (components), etc.). It is not important whether all components exist in the same housing. Therefore, multiple devices housed in separate housings and connected via a network and a device in which multiple modules are housed in one housing are both systems.

[0064] For example, the present embodiment may adopt a configuration of cloud computing in which one function is collaboratively shared and processed by a plurality of devices via a network.

[0065] Although the embodiments of the present disclosure are described above, the technical scope of the present disclosure is not limited to the embodiments themselves. Various changes can be made without departing from the gist of the present disclosure. Components in different embodiments and modifications can be appropriately combined.

[0066] The processing procedures described in the embodiments may be regarded as methods including these series of processes, and may be regarded as programs for causing a computer to execute these series of processes or recording media storing the programs. As the recording medium, for example, a floppy disk, a CD-ROM (compact disk read-only memory), an MO (magneto-optical) disk, a DVD (digital versatile disk), a Blu-ray (registered trademark) disk, a magnetic disk, a semiconductor memory, a memory card, etc. may be used.

[0067] Note that the effects described in this specification are only illustrative and not limiting. Other effects may exist.

[0068] Note that the present technology can also adopt the following configurations.

[0069] (1) A system comprising:

[0070] a correction unit that corrects the distance image according to a movement of the working device in the distance image including a receiving portion on which the object is placed by the working device;

[0071] a proximity detection unit that detects a proximity of the accommodation portion based on a distance histogram generated from the corrected distance image; and

[0072] The receiving portion detecting unit detects the receiving portion based on the detected proximity portion and the distance image.

[0073] (2) The system according to (1) above, further comprising:

[0074] The point cloud data generating unit generates point cloud data of the accommodation portion, wherein:

[0075] The correction unit corrects the point cloud data.

[0076] (3) The system according to (1) above further includes: a data conversion unit that converts the detected receiving portion into point cloud data.

[0077] (4) The system according to any one of (1) to (3) above, wherein the close portion detection unit detects the close portion based on the range image of a region close to a mode of the range histogram.

[0078] (5) A procedure comprising:

[0079] a process of correcting the range image based on movement of the working device in the range image including a receiving portion on which the object is placed by the working device;

[0080] A process of detecting a proximity portion of a receiving portion based on a distance histogram generated from the corrected distance image; and

[0081] A process of detecting a receiving portion based on the detected proximity portion and the distance image.

[0082] Reference Numbers List

[0083] 1. Detection device

[0084] 10 Excavator

[0085] 11 Operating equipment

[0086] 12 Stick

[0087] 20 Dump Truck

[0088] 21 Cargo bed

[0089] 110 Camera

[0090] 120 Distance Sensor

[0091] 130 Target area extraction unit

[0092] 200 Cargo bed area detection unit

[0093] 210 Point cloud data generation unit

[0094] 220 Cargo bed detection and processing unit

[0095] 221 Calibration Unit

[0096] 222 Proximity detection unit

[0097] 223 Cargo bed detection unit

[0098] 230 Data conversion unit.

Claims

1. A system comprising: a correction unit that corrects the distance image according to a movement of the working device in the distance image including a receiving portion on which the object is placed by the working device; a proximity detection unit that detects a proximity portion of the accommodation portion based on a distance histogram generated from the corrected distance image; as well as The receiving portion detecting unit detects the receiving portion based on the detected approaching portion and the distance image.

2. The system according to claim 1, further comprising: A point cloud data generating unit generates point cloud data of the receiving portion, wherein: The correction unit corrects the point cloud data.

3. The system according to claim 1, further comprising: The data conversion unit converts the detected receiving portion into point cloud data.

4. The system according to claim 1, wherein: The close portion detection unit detects the close portion based on the range image of a region close to a mode of the range histogram.

5. A program comprising: a process of correcting the range image according to a movement of the working device in the range image including a receiving portion on which the object is placed by the working device; a process of detecting a proximity portion of the receiving portion based on a distance histogram generated from the corrected distance image; and A process of detecting the receiving portion based on the detected approaching portion and the distance image.

Images