Plane detection device and plane detection method

Abstract

A plane detecting device according to the present disclosure is provided with an information acquiring unit, a likelihood acquiring unit, and a plane detecting unit. The information acquiring unit acquires visible image information of a target object having a predetermined plane, and 3D coordinate information corresponding to the visible image information. The likelihood acquiring unit acquires, from the visible image information, a likelihood indicating a planarity of the predetermined plane in the target object. The plane detecting unit uses the 3D coordinate information and the likelihood to detect the predetermined plane in the target object by means of a robust estimating method.

Description

[Technical Field]

[0001] The present disclosure relates to a plane detection device and a plane detection method. [Background technology]

[0002] For example, Patent Document 1 discloses a technology for detecting planes. The technology disclosed in Patent Document 1 detects planes from an image captured by a TOF (Time Of Flight) camera. Patent Document 1 describes that plane information is detected from image data obtained by TOF using the RANSAC method. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2010 / 018009 Summary of the Invention

[0004] However, there is a demand for a technique that can detect the plane of an object with higher accuracy than the technique described in Patent Document 1, for example.

[0005] Therefore, the present disclosure provides a plane detection device and a plane detection method that can detect the plane of an object with higher accuracy.

[0006] A plane detection device according to one aspect of the present disclosure includes: an information acquisition unit that acquires visible image information of an object having a predetermined plane and 3D coordinate information corresponding to the visible image information; a likelihood acquisition unit that acquires, from the visible image information, a likelihood indicating a flatness of the predetermined plane of the object; The apparatus further includes a plane detection unit that detects the predetermined plane of the object by a robust estimation method using the 3D coordinate information and the likelihood.

[0007] A plane detection method according to another aspect of the present disclosure includes: Obtaining visible image information of an object having a predetermined plane and 3D coordinate information corresponding to the visible image information; obtaining a likelihood indicating a flatness of the predetermined plane of the object from the visible image information; The method further comprises detecting the predetermined plane of the object by a robust estimation method using the 3D coordinate information and the likelihood.

[0008] These general and specific aspects may be realized by a system, a method, a computer program, a computer-readable recording medium, and a combination thereof.

[0009] According to the present disclosure, it is possible to provide a plane detection device and a plane detection method that can detect the plane of an object with higher accuracy. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram showing a schematic block configuration of a plane detection device according to a first embodiment of the present disclosure. [Figure 2] 1 is a schematic diagram illustrating an example palette contained in visual image information. [Figure 3] FIG. 10 illustrates 3D coordinate information for an example pallet. [Figure 4] FIG. 10 is a diagram illustrating an example of likelihood for each pixel. [Figure 5] 10 is a flowchart showing the flow of a plane detection method. [Figure 6] FIG. 1 is a schematic diagram illustrating the operation of a plane detection method. [Figure 7] FIG. 1 is a schematic diagram illustrating the operation of a plane detection method. [Figure 8] FIG. 1 is a schematic diagram illustrating the operation of a plane detection method. [Figure 9] FIG. 1 is a schematic diagram illustrating the operation of a plane detection method. [Figure 10] FIG. 1 is a schematic diagram illustrating the operation of a plane detection method. [Figure 11]10 is a flowchart showing a specific flow of plane detection. [Figure 12] FIG. 1 is a schematic diagram illustrating an example pallet. [Figure 13] FIG. 1 is a schematic diagram illustrating the operation of a plane detection method. [Figure 14] FIG. 1 is a schematic diagram illustrating the operation of a plane detection method. [Figure 15] FIG. 1 is a schematic diagram illustrating the operation of a plane detection method. [Figure 16] FIG. 1 is a schematic diagram illustrating the operation of a plane detection method. [Figure 17] FIG. 1 is a schematic diagram illustrating the operation of a plane detection method. DETAILED DESCRIPTION OF THE INVENTION

[0011] The present disclosure relates to a plane detection device for detecting a predetermined plane of an object. Hereinafter, a specific description will be given based on the drawings showing embodiments.

[0012] <First Embodiment> Fig. 1 is a block diagram showing a schematic configuration of a plane detection device 10 according to this embodiment. As shown in Fig. 1, the plane detection device 10 includes a control unit 20, an output unit 30, and an imaging unit 40. The plane detection device 10 also includes a storage unit 201 that stores various data including machine learning models. As shown in Fig. 1, the control unit 20 is communicably connected to the output unit 30, the imaging unit 40, and the storage unit 201.

[0013] (composition) The control unit 20 can be configured with a microcomputer, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit). The functions of the control unit 20 may be configured with hardware alone, or may be realized by combining hardware and software.

[0014] The control unit 20 realizes predetermined functions by reading out data and programs stored in the storage unit 201 and performing various arithmetic processing. The control unit 20 also includes an information acquisition unit 101, a likelihood acquisition unit 102, and a plane detection unit 103 as functional blocks.

[0015] First, the information acquisition unit 101 will be described. The information acquisition unit 101 acquires visible image information and 3D coordinate information of an object. Here, the object has a predetermined plane. Note that the plane detection device 10 according to this embodiment detects the predetermined plane using each piece of information acquired by the information acquisition unit 101. The information acquisition unit 101 acquires each piece of information from image data of the object captured by the imaging unit 40. The imaging unit 40 is, for example, a depth camera. When the imaging unit 40 captures an image of the object, the information acquisition unit 101 acquires visible image information (RGB image data) of the object shown in FIG. 2 and 3D coordinate information corresponding to the visible image information shown in FIG. 3.

[0016] As shown in FIG. 2, in this embodiment, the object is a pallet 1 on which an object P can be loaded. The pallet 1 includes a flat plate portion 1a and a first support portion 1b. The object P is loaded on the flat plate portion 1a in the vertical direction (loading direction) of the drawing. The first support portion 1b extends from the flat plate portion 1a in the loading direction (vertical direction in FIG. 2). As shown in FIG. 2, the first support portion 1b has a first surface 1br on the opposite side of the space in which the object P is loaded. In other words, the first surface 1br faces the outside of the pallet 1. As shown in FIG. 2, the first surface 1br faces the direction A. The predetermined plane includes the first surface 1br. The first support portion 1b may be a rectangular parallelepiped, but is not limited to this. The first support portion 1b does not have to be a rectangular parallelepiped as long as it has a planar area.

[0017] The 3D coordinate information shown in FIG. 3 illustrates an image. The 3D coordinate information includes 3D (three-dimensional) coordinate values ​​corresponding to each pixel of the visible image information (color image). In other words, information including the 3D coordinate values ​​of each pixel of the visible image information is the 3D coordinate information. Note that the position of the imaging unit 40 (depth camera) is set as the origin. Various methods can be used to acquire the 3D coordinates, such as Stereo and LiDAR. The 3D coordinates may also be acquired by converting, for example, depth values.

[0018] Next, the likelihood acquisition unit 102 will be described. The likelihood acquisition unit 102 acquires a likelihood indicating the likelihood of an object being a predetermined plane from the visible image information shown in FIG. 2. Here, the "predetermined plane likelihood" refers to the likelihood of a specific plane region of a specific object. The likelihood acquisition unit 102 acquires (calculates) a likelihood for each pixel of the visible image information using the visible image information, which is input information, and a machine learning model. Here, the visible image information is acquired by the information acquisition unit 101, and the machine learning model is stored in the storage unit 201. The likelihood acquisition unit 102 calculates the likelihood of the predetermined plane by inference using Mask RCNN or the like, using the visible image information and the machine learning model. In this embodiment, the likelihood acquisition unit 102 calculates the likelihood of the first surface 1br of the first support portion 1b from the visible image information (the "likelihood" here represents the likelihood of the first surface 1br of the first support portion 1b being a planar region, as described above).

[0019] FIG. 4 is a schematic diagram illustrating an example of the likelihood calculated for the first surface 1br of the first support portion 1b. The likelihood is calculated for each pixel of the visible image information. The likelihood is calculated as a value between 0 and 1. In FIG. 4, the likelihood for each pixel of the first surface 1br is visually illustrated. Here, in FIG. 4, the darker the shade of black and white, the higher the likelihood, and the lighter the shade of black and white, the lower the likelihood. For example, when the likelihood of a pixel is 0, this indicates that the pixel is least likely to be a predetermined plane (first surface 1br) (white). On the other hand, when the likelihood of a pixel is 1, this indicates that the pixel is most likely to be a predetermined plane (first surface 1br) (black).

[0020] In this embodiment, likelihood is automatically acquired (calculated) using a machine learning model. However, likelihood may be determined for each pixel of visible image information by other methods. For example, the likelihood acquisition unit 102 may be connected to a unit (operation unit) that accepts user operations, and may acquire the likelihood based on input information from the operation unit. For example, the visible image information may be displayed on a display or the like (output unit 30), and the user may manually select and determine the likelihood of each pixel through the operation unit. The likelihood acquisition unit 102 may determine (acquire) the likelihood for each pixel based on the operation. In this case, the storage unit 201 that stores the machine learning model may be omitted.

[0021] Next, the plane detection unit 103 will be described. The plane detection unit 103 detects a predetermined plane of the object by a robust estimation method using the 3D coordinate information shown in FIG. 3 and the likelihood acquired by the likelihood acquisition unit 102. For example, the plane detection unit 103 detects the predetermined plane of the object by RANSAC or the like using a plurality of sample points and the likelihoods corresponding to each of the plurality of sample points. Here, the plurality of sample points (for example, at least three points) are randomly selected from the 3D coordinate information corresponding to the predetermined plane (for example, the first surface 1br). Note that in this embodiment, detecting a plane means estimating a plane equation of the predetermined plane of the object. In addition to the above-mentioned RANSAC, a least squares method or the like can also be adopted as a method for estimating the plane equation.

[0022] In this embodiment, the plane detection unit 103 detects a plane in consideration of (using) the likelihood acquired by the likelihood acquisition unit 102 (i.e., the likelihood is used as a "weight" during robust estimation). Note that the specific method of detecting a plane will be described later in the explanation of the operation.

[0023] Next, we will explain the storage unit 201. The storage unit 201 is a storage medium that stores programs and data necessary to realize the functions of the plane detection device 10. For example, the storage unit 201 can be realized by a hard disk (HDD), a solid state drive (SSD), a random access memory (RAM), a dynamic RAM (DRAM), a ferroelectric memory, a flash memory, a magnetic disk, or a combination of these.

[0024] The storage unit 201 also stores a machine learning model constructed by machine learning. The machine learning model is used by the likelihood acquisition unit 102 when acquiring (calculating) the likelihood. In this embodiment, machine learning is performed in advance to generate a machine learning model so that the likelihood of the first surface 1br of the first support portion 1b of the pallet 1 can be calculated. In other words, machine learning is performed so as to calculate the likelihood of only a specific planar region (the first surface 1br) rather than the entire pallet 1.

[0025] The likelihood indicates the degree to which an object is flat. In this embodiment, the likelihood indicates the degree to which the first surface 1br of the first support portion 1b of the pallet 1 is flat.

[0026] A machine learning model can be generated, for example, as follows: First, visible image information (image data) of palette 1 is acquired. Then, for each pixel of the visible image information, the flatness of the first surface 1br (predetermined plane) is labeled. This process is performed on multiple pieces of visible image information, and by using the labeling results, a machine learning model can be generated through machine learning.

[0027] The output unit 30 has a display unit that displays the results of the arithmetic processing of the control unit 20. For example, the display unit may be configured as a liquid crystal display or an organic EL display. The output unit 30 may also include a speaker that emits sound.

[0028] The imaging unit 40 captures an image of an object as a subject. Based on the imaging information from the imaging unit 40, the information acquisition unit 101 acquires visible image data of the object and 3D coordinate information associated with the visible image data. The visible image data is color image data. The 3D coordinate information associated with the visible image data is information on 3D coordinates corresponding to each pixel of the image data.

[0029] The imaging unit 40 is, for example, a depth camera. The depth camera measures the distance to an object and generates depth information indicating the measured distance as a depth value for each pixel. For example, the depth camera may be an infrared active stereo camera, a LiDAR depth camera, or the like. Note that the imaging unit 40 is not limited to these depth cameras.

[0030] (operation) Fig. 5 is a diagram showing the flow of the plane detection method according to this embodiment. Specific plane detection operations will be described below using the schematic configuration diagram shown in Fig. 1 and the flow shown in Fig. 5. This embodiment will describe in detail the case of detecting a predetermined plane including a first surface 1br of a pallet 1, which is an object.

[0031] Before step S3 in Figure 5 described later is started for the object of plane detection, the plane detection device 10 sets 0 (zero) as the initial value, the provisional addition value L', and nothing is set as the initial value, the provisional plane equation θ' (hereinafter referred to as provisional plane θ') (for example, a value such as 0 or null is set).

[0032] The imaging unit 40 captures an image of the pallet 1. In response to the imaging result, the information acquisition unit 101 acquires visible image information of the pallet 1 (see FIG. 2) and 3D coordinate information corresponding to the visible image information (see FIG. 3) (step S1 in FIG. 5).

[0033] Next, the likelihood acquisition unit 102 acquires (calculates) the likelihood for the first support member 1b of the pallet 1 from the visible image information (step S2 in FIG. 5). Here, the likelihood is acquired using a machine learning model. The likelihood is calculated for each pixel representing the first surface 1br of the first support member 1b. FIG. 4 shows a visual example of the likelihood of each pixel on the first surface 1br. As described above, the first surface 1br faces the opposite side of the space on the pallet 1 where the object P is loaded (facing the direction A as shown in FIG. 4).

[0034] Next, the plane detection unit 103 uses the 3D coordinate information acquired by the information acquisition unit 101 and the likelihood acquired by the likelihood acquisition unit 102 to detect a predetermined plane including the first surface 1br of the pallet 1 using a robust estimation method (step S3 in Figure 5).

[0035] Specifically, first, the plane detection unit 103 determines target sample points belonging to the first plane 1br in the 3D coordinate information acquired by the information acquisition unit 101. For example, sample points belonging to an area where likelihood > 0 can be determined as target sample points. FIG. 6 illustrates an example of how target sample points belonging to the first plane 1br are determined. FIG. 6 is a diagram of the first support column 1b in FIG. 4 viewed from above. Each white circle indicates a determined target sample point. Furthermore, as illustrated in FIG. 6, each target sample point is associated with the likelihood acquired in step S2.

[0036] Next, the plane detection unit 103 randomly extracts at least three target sample points from the plurality of target sample points in the 3D coordinate information (see FIG. 7). For example, the extraction is performed by RANSAC. In the example of FIG. 7, the black circles are randomly extracted target sample points. In the example of FIG. 7, the number of randomly extracted target sample points is three. The process of extracting the target sample points is referred to as a random extraction process.

[0037] Next, the plane detection unit 103 obtains a plane equation θ (hereinafter simply referred to as plane θ) based on the three target sample points extracted above, for example, by RANSAC or the like. For example, it is assumed here that plane θ1 is obtained as plane θ. FIG. 8 illustrates the obtained plane θ1. The process of obtaining the plane θ will be referred to as a plane equation obtaining process.

[0038] Next, the plane detection unit 103 calculates the distance from the plane θ1 to each target sample point belonging to the first plane 1br. FIG. 9 illustrates the calculated distances d1, d2, d3, and dn. For simplicity, FIG. 9 illustrates only the distances d1, d2, d3, and dn, but the plane detection unit 103 performs the process of calculating the distances for all target sample points belonging to the first plane 1br. The process of calculating each distance is referred to as the distance acquisition process.

[0039] Next, the plane detection unit 103 extracts target sample points from all target sample points (for example, all target sample points with likelihoods greater than 0) whose calculated distances d1, d2, d3, and dn are less than a threshold value t (see FIG. 10). As shown in FIG. 10, the distance to the threshold value t is indicated by a dotted line. The threshold value t is a preset value, and as can be seen from the example shown in FIG. 10, a dotted line is drawn at the position where the distance from the plane θ1, which is the current plane θ, is the threshold value t. Therefore, in this process, the plane detection unit 103 extracts target sample points that are located within the area surrounded by the dotted line in FIG. 10. In this embodiment, target sample points that are "less than the threshold value t" are not extracted because they are on the dotted line.

[0040] Here, likelihoods are determined for all target sample points through the likelihood acquisition process described above. The plane detection unit 103 then calculates the sum L of likelihoods for all target sample points that are located at a distance from the plane θ1 that is less than the threshold value t. For example, in the example of FIG. 10 , a total of six target sample points exist in the area surrounded by the dotted line. Therefore, the plane detection unit 103 extracts these six target sample points. In FIG. 10 , likelihood values ​​are assigned to these six target sample points. Therefore, in the example of FIG. 10 , the plane detection unit 103 calculates the sum L of likelihoods for these six target sample points. In the example of FIG. 10 , likelihoods of 0.2, 0.2, 0.7, 0.7, 1.0, and 1.0 are assigned to the target sample points. Therefore, the plane detection unit 103 calculates the sum L of likelihoods as 3.8 (=0.2 + 0.2 + 0.7 + 0.7 + 1.0 + 1.0). The process of extracting target sample points from all target sample points whose distances d1, d2, d3, and dn calculated above are less than the threshold value t, and calculating the added value L of the likelihood, will be referred to as the added value L acquisition process.

[0041] Next, the plane detection unit 103 compares the currently set provisional sum L' with the likelihood sum L calculated this time. Then, the larger value is set as the new provisional sum L'. Here, as explained above, the provisional sum L' currently set as the initial value is 0. Therefore, the likelihood sum L calculated this time is always larger than the initial provisional sum L'. Therefore, the plane detection unit 103 sets the likelihood sum L calculated this time as the new provisional sum L'. In the above example, the plane detection device 10 sets 3.8 as the new provisional sum L'.

[0042] Furthermore, the plane detection unit 103 newly sets the plane θ corresponding to the newly set provisional added value L' as a provisional plane θ' in the plane detection device 10. In this case, the added value L (= 3.8) is calculated for the plane θ1, and this added value L (= 3.8) is set as the new provisional added value L'. Therefore, the plane detection unit 103 newly sets the plane θ1 corresponding to the calculated likelihood added value L (= 3.8) in the plane detection device 10 as a provisional plane θ'.

[0043] The process of determining the magnitude relationship between the provisional sum L' and the calculated likelihood sum L, and the process of newly setting the provisional sum L' and the provisional plane θ' will be referred to as provisional value setting process.

[0044] The above series of steps, including the random extraction process, the plane equation acquisition process, the distance acquisition process, the additional value L acquisition process, and the provisional value setting process, is referred to as a plane setting loop. In step S3 of FIG. 5, the plane setting loop is performed a predetermined number of times. The predetermined number of times may be set in advance in the plane detection device 10, for example. In this case, higher accuracy can be achieved. Alternatively, the predetermined number of times can be determined within the algorithm (see, for example, http: / / people.inf.ethz.ch / pomarc / pubs / RaguramPAMI13.pdf). In this case, higher speed processing is possible. As an example, it is assumed here that the predetermined number of times of the plane setting loop is set to k in the plane detection device 10.

[0045] Figure 11 is a diagram showing the flow of the plane setting loop in step S3 of Figure 5. As illustrated in Figure 11, the plane setting loop includes a random extraction process S11, a plane equation acquisition process S12, a distance acquisition process S13, an additional value L acquisition process S14, and a temporary value setting process S15. Figure 11 also shows that if the plane setting loop has been completed less than k times (i.e., k-1 times), the process returns from the temporary value setting process S15 to the random extraction process S11, and the next plane setting loop is performed. Figure 11 also shows that if the plane setting loop has been completed k times, the plane setting loop (in other words, the plane detection process of step S3 of Figure 5) is completed.

[0046] In the above example, when the first plane setting loop is completed, the provisional added value L' is set to 3.8 and the provisional plane θ' is set to θ1. If the predetermined number of times is 2 or more, the process returns to the random extraction process S11 in FIG. 11 and the second plane setting loop is performed.

[0047] For example, the plane detection unit 103 randomly extracts three target sample points from the plurality of target sample points based on the 3D coordinate information (see random extraction process S11 in FIG. 11).

[0048] Next, the plane detection unit 103 obtains a plane θ2 as an equation of a plane based on the three target sample points extracted above, for example, by RANSAC or the like (see plane equation acquisition process S12 in FIG. 11).

[0049] Next, the plane detection unit 103 obtains the distance from the plane θ2 to each of the target sample points belonging to the first plane 1br (see distance acquisition processing S13 in FIG. 11).

[0050] Next, the plane detection unit 103 extracts target sample points whose distance calculated above is less than the threshold value t from among all target sample points (for example, all target sample points whose likelihood is greater than 0). The plane detection unit 103 then calculates the likelihood sum L of all target sample points whose distance from the plane θ2 is less than the threshold value t (see the sum L acquisition process S14 in FIG. 11). For example, suppose that the likelihood sum L is 3.0 in the second plane setting loop.

[0051] Next, the plane detection unit 103 compares the currently set provisional sum L' (=3.8) with the likelihood sum L calculated this time (here, the second time). The larger value is then set as the new provisional sum L' (see provisional value setting process S15 in FIG. 11). As explained above, the currently set provisional sum L' is 3.8, and the likelihood sum L calculated in the second plane setting loop is 3.0. Therefore, the plane detection unit 103 sets the currently set value as the new provisional sum L'. In other words, the value of 3.8 is continued to be set as the provisional sum L'.

[0052] Furthermore, the plane detection unit 103 newly sets the plane θ corresponding to the newly set provisional added value L' as a provisional plane θ' in the plane detection device 10 (see provisional value setting process S15 in FIG. 11). In this case, the added value L (= 3.8) is calculated for the plane θ1, and this added value L (= 3.8) is set as the new provisional added value L'. Therefore, the plane detection unit 103 newly sets the plane θ1 corresponding to the calculated likelihood added value L (= 3.8) as a provisional plane θ' in the plane detection device 10 (continuously setting the plane θ1).

[0053] As a result, the processing of the second plane setting loop is completed, and if the number of plane setting loops completed up to this point is less than k, the process returns to step S11 and starts the next plane setting loop. As described above, when the plane setting loop illustrated in FIG. 11 has been completed k times, the plane setting loop (in other words, the plane detection process of step S3 in FIG. 5) ends. Then, when the processing of step S3 ends, the plane detection unit 103 outputs the provisional added value L' and provisional plane θ' set at the time of completion as the plane detection result. In other words, the output provisional plane θ' represents an equation related to the plane detected from the object.

[0054] (Effects explanation) In a first aspect of this embodiment, the plane detection device 10 includes an information acquisition unit 101, a likelihood acquisition unit 102, and a plane detection unit 103. The information acquisition unit 101 acquires visible image information of an object having a predetermined plane and 3D coordinate information corresponding to the visible image information. The likelihood acquisition unit 102 acquires, from the visible image information, a likelihood indicating the flatness of the predetermined plane of the object. Then, the plane detection unit 103 uses the 3D coordinate information and the likelihood to detect the predetermined plane of the object by a robust estimation method.

[0055] In another aspect of the present embodiment, visible image information of an object having a predetermined plane and 3D coordinate information corresponding to the visible image information are acquired. Then, a likelihood indicating the planarity of the predetermined plane of the object is acquired from the visible image information. Then, the predetermined plane of the object is detected by a robust estimation method using the 3D coordinate information and the likelihood.

[0056] In other words, the plane detection device 10 detects planes using likelihood. Therefore, it is possible to detect a predetermined plane of an object with higher accuracy than conventional plane detection. For example, even if a specific object plane is occluded by an object such as paper, resulting in multiple sample points that are distant from the specific object plane, it is possible to robustly detect the predetermined plane of the object with high accuracy.

[0057] For example, referring to Fig. 10, a method of detecting a predetermined plane can be considered based on the proportion of target sample points (target sample points within a range) that are within a distance from the plane θ1 that is less than a threshold value t, i.e., the proportion of target sample points within the range to all target sample points (for example, the plane with the largest proportion is detected as the predetermined plane of the object). In contrast, in this embodiment, the detection process for the predetermined plane is performed using likelihood information as well. Therefore, for example, if the number of loops of the RANSAC algorithm is set, high accuracy can be achieved, while if the predetermined number is determined within the algorithm, high speed can be achieved.

[0058] In a second aspect of this embodiment, the plane detection device 10 further includes a storage unit 201 that stores a machine learning model constructed by machine learning. The likelihood acquisition unit 102 then acquires a likelihood for each pixel of the visible image information using the visible image information, which is input information, and the machine learning model. This allows the likelihood acquisition unit 102 to acquire a likelihood for each pixel of the visible image information more quickly and with higher accuracy.

[0059] In addition, in a third aspect of this embodiment, the target object includes a pallet 1 on which an object P can be loaded. Therefore, it becomes possible to detect the surface of a pallet 1 used in a factory or the like. This makes it possible to control various automatic operations using the surface detection results, for example.

[0060] In a fourth aspect of this embodiment, the pallet 1 includes a flat plate portion 1a and a first support portion 1b. An object P is loaded on the flat plate portion 1a. The first support portion 1b extends from the flat plate portion 1a in the direction in which the object P is loaded. The predetermined plane includes a first surface 1br of the first support portion 1b. Therefore, when the pallet 1 has a first support portion 1b extending in the vertical direction (the direction in which the object P is loaded), the first surface 1br of the first support portion 1b can be detected.

[0061] In addition, in a fifth aspect of the present embodiment, the plane detection unit 103 detects a predetermined plane of the object by RANSAC using a plurality of sample points randomly selected from the 3D coordinate information and likelihoods corresponding to each of the plurality of sample points. Therefore, it is possible to automatically, highly accurately, and practically detect the first surface 1br of the object.

[0062] <Embodiment 2> In the first embodiment, as an example, a case where a predetermined plane including a first surface 1br of a pallet 1 is detected is described. In the present embodiment, a predetermined plane detection process is described in a case where the target object, pallet 1, has, for example, two supports (a first support portion and a second support portion). That is, in the present embodiment, a case where a first support portion has a first surface and a second support portion has a second surface, and a predetermined plane including the first surface and the second surface is detected is described in detail.

[0063] As shown in Figure 12, in this embodiment as well, the object is a pallet 1 on which objects can be loaded. Figure 12 is a schematic diagram of the pallet 1 when viewed from the side (from direction A in Figure 4, etc.). In this embodiment, the pallet 1 includes a flat plate portion 1a, a first support portion 1b, and a second support portion 1c. Note that direction A is the direction from the front to the back of the paper in Figure 12.

[0064] Objects are loaded on the flat plate portion 1a in the vertical direction in FIG. 12. The first support column 1b extends from the flat plate portion 1a in the loading direction (vertical direction in FIG. 12). The first support column 1b has a first surface 1br on the opposite side of the space in which the object P is loaded. In other words, the first surface 1br faces the outside of the pallet 1 (facing direction A). The second support column 1c extends from the flat plate portion 1a in the loading direction (vertical direction in FIG. 12). The second support column 1c is disposed on the flat plate portion 1a separately from the first support column 1b. In other words, as illustrated in FIG. 12, the second support column 1c is disposed at a position separated from the first support column 1b. The second support column 1c has a second surface 1cr on the opposite side of the space in which the object P is loaded. That is, the second surface 1cr faces the outside of the pallet 1 (facing the direction A).

[0065] In this embodiment, the first surface 1br and the second surface 1cr are located on the same plane. That is, in this embodiment, a predetermined plane having the first surface 1br and the second surface 1cr is detected. As described for the first support column 1b in the first embodiment, the second support column 1c may be a rectangular parallelepiped, but is not limited to this. The second support column 1c does not have to be a rectangular parallelepiped as long as it has a planar area.

[0066] In this embodiment, the physical configuration of the plane detection device 10 is the same as that illustrated in the schematic block diagram of Fig. 1. In this embodiment, the plane detection device 10 also performs a series of processes in the same flow as in Fig. 5, and the plane detection unit 103 performs a series of processes in the same flow as in Fig. 11. Below, the plane detection operation will be described, focusing on the differences.

[0067] (operation) In this embodiment, the target object, pallet 1, has a first surface 1br and a second surface 1cr. The following describes in detail the case where a predetermined plane including the first surface 1br and the second surface 1cr is detected.

[0068] As in embodiment 1, before step S3 in FIG. 5 is started for the object to be detected as a plane, the plane detection device 10 sets 0 (zero) as the initial value, the provisional addition value L', and nothing is set as the initial value, the provisional plane equation θ' (hereinafter referred to as provisional plane θ') (for example, a value such as 0 or null is set).

[0069] The imaging unit 40 captures an image of the pallet 1. Through this imaging, the information acquisition unit 101 acquires visible image information of the pallet 1 (see FIG. 2) and 3D coordinate information corresponding to the visible image information (see FIG. 3) (step S1 in FIG. 5).

[0070] Next, the likelihood acquisition unit 102 acquires (calculates) likelihoods for the first support member 1b and the second support member 1c from the visible image information (step S2 in FIG. 5). The method of acquiring the likelihoods is the same as in the first embodiment. The likelihoods are acquired for each pixel representing the first surface 1br of the first support member 1b and each pixel representing the second surface 1cr of the second support member 1c. In this embodiment, the likelihood indicates the planarity of the first surface 1br of the first support member 1b and / or the planarity of the second surface 1cr of the second support member 1c.

[0071] Next, the plane detection unit 103 detects the above-mentioned specified plane using the 3D coordinate information acquired by the information acquisition unit 101 and the likelihood acquired by the likelihood acquisition unit 102 by a robust estimation method (RANSAC) (step S3 in Figure 5).

[0072] Specifically, first, the plane detection unit 103 determines target sample points belonging to the first plane 1br in the 3D coordinate information acquired by the information acquisition unit 101. Furthermore, the plane detection unit 103 determines target sample points belonging to the second plane 1cr in the 3D coordinate information acquired by the information acquisition unit 101. The method of obtaining the target sample points is the same as in the first embodiment. FIG. 13 illustrates an example of how the target sample points belonging to the first plane 1br and the target sample points belonging to the second plane 1cr are obtained. FIG. 13 is a view of the first and second support pillars 1b and 1c viewed from above. Each white circle indicates the obtained target sample point. Each target sample point is associated with the likelihood obtained in step S2 (FIG. 5).

[0073] Next, the plane detection unit 103 randomly extracts at least three target sample points from the plurality of target sample points in the 3D coordinate information (see random extraction process S11 in FIG. 11 and FIG. 14). For example, the extraction is performed by RANSAC. In the example of FIG. 14, the black circles are randomly extracted target sample points. In the example of FIG. 14, the number of randomly extracted target sample points is three.

[0074] In this embodiment, in the random sampling process S11, at least one target sample point belonging to the first surface 1br is selected, and at least one target sample point belonging to the second surface 1cr is selected. In the example of Fig. 14, in the random sampling process S11, one target sample point belonging to the first surface 1br is selected, and two target sample points belonging to the second surface 1cr are selected.

[0075] Next, the plane detection unit 103 obtains the equation θ of the plane (plane θ) based on the three target sample points extracted above, for example, by RANSAC or the like (see plane equation acquisition process S12 in FIG. 11). For example, it is assumed here that plane θi is obtained as plane θ. FIG. 15 illustrates an example of the obtained plane θi.

[0076] Next, the plane detection unit 103 calculates the distance from the plane θi to each of the target sample points (see distance acquisition process S13 in FIG. 11). The calculated distance d is illustrated in FIG. 16. For simplicity, FIG. 16 illustrates the distance d for two target sample points, but the plane detection unit 103 performs the process of calculating the distance for all of the target sample points.

[0077] Next, the plane detection unit 103 extracts target sample points for which the distance d calculated above is less than the threshold value t from among all target sample points (for example, all target sample points for which the likelihood is greater than 0) (see FIG. 17). As shown in FIG. 17, the distance to the threshold value t is indicated by a dotted line. The plane detection unit 103 extracts target sample points that are located within the area surrounded by the dotted line in FIG. 17, for example. In this embodiment, target sample points that are "less than the threshold value t" are not extracted, as they are located on the dotted line.

[0078] The likelihoods are determined for all target sample points through the likelihood acquisition process described above. The plane detection unit 103 then calculates the sum L of the likelihoods of all target sample points that are located at a distance from the plane θi that is less than the threshold value t (see the sum L acquisition process S14 in FIG. 11). For example, in the example of FIG. 17, a total of six target sample points exist in the area surrounded by the dotted line. Therefore, the plane detection unit 103 extracts these six target sample points. Assume that likelihoods of 0.3, 0.4, 0.7, 1.0, 1.0, and 1.0 are assigned to the target sample points, respectively. In this case, the plane detection unit 103 calculates 4.4 as the sum L of the likelihoods.

[0079] Next, the plane detection unit 103 compares the currently set provisional sum L' with the likelihood sum L calculated this time. Then, the larger value is set as a new provisional sum L' (see provisional value setting process S15 in FIG. 11). Here, as explained above, the provisional sum L' currently set as the initial value is 0. Therefore, the plane detection unit 103 sets the likelihood sum L calculated this time as the new provisional sum L'. In the above example, the plane detection device 10 sets 4.4 as the new provisional sum L'.

[0080] Furthermore, the plane detection unit 103 newly sets the plane θ corresponding to the newly set provisional added value L' as a provisional plane θ' in the plane detection device 10 (see provisional value setting process S15 in FIG. 11). In this case, the added value L (= 4.4) is calculated for the plane θi, and this added value L (= 4.4) is set as a new provisional added value L'. Therefore, the plane detection unit 103 newly sets the plane θi corresponding to the calculated likelihood added value L (= 4.4) in the plane detection device 10 as a provisional plane θ'.

[0081] In step S3 of FIG. 5, a plane setting loop (see FIG. 11) is performed a predetermined number of times (k times). In FIG. 11, if the plane setting loop has been completed less than k times (i.e., k-1 times), the process returns from the provisional value setting process S15 to the random extraction process S11, and the next plane setting loop is performed. In contrast, in FIG. 11, if the plane setting loop has been completed k times, the plane setting loop (in other words, the plane detection process of step S3 of FIG. 5) ends. Then, when the process of step S3 ends, the plane detection unit 103 outputs the provisional added value L' and provisional plane θ' set at the time of the end as the plane detection result. In other words, the output provisional plane θ' represents an equation related to the plane detected from the object.

[0082] (Effects explanation) In a sixth aspect of the present embodiment, the pallet 1 includes a second support column 1c in addition to the first support column 1b. The second support column 1c extends from the flat plate portion 1a in the direction in which the objects are loaded (the up-down direction in FIG. 12). The second support column 1c also includes a second surface 1cr. The predetermined plane has a first surface 1br and a second surface 1cr.

[0083] Therefore, when the pallet 1 has first and second support pillars 1b, 1c extending vertically (in the direction in which the objects are loaded) and the first surface 1br and the second surface 1cr are in the same plane, a predetermined plane (the plane including the first surface 1br and the second surface 1cr) can be detected with respect to the object, the pallet 1.

[0084] In addition, in a seventh aspect of the present embodiment, the plane detection unit 103 uses a plurality of sample points randomly selected from the 3D coordinate information and likelihoods corresponding to each of the plurality of sample points to detect a predetermined plane of the pallet 1 by RANSAC. At least one of the plurality of sample points is selected for the first plane 1br, and at least one of the plurality of sample points is selected for the second plane 1cr.

[0085] Therefore, it becomes possible to automatically, highly accurately, and practically detect the first and second surfaces 1br and 1cr of the pallet 1. Furthermore, since the first surface 1br and the second surface 1cr are disposed apart from each other and exist in the same plane, it is possible to detect the predetermined plane more quickly and accurately than when detecting the predetermined plane by targeting only the first surface 1br.

[0086] Although the present invention has been fully described in connection with the preferred embodiments with reference to the accompanying drawings, various changes and modifications will be apparent to those skilled in the art, and it is to be understood that such changes and modifications are included within the scope of the present invention as defined by the appended claims unless they depart therefrom. [Industrial Applicability]

[0087] The present disclosure can be used in the field of transportation, such as loading cargo onto trucks or warehouses, because it can easily measure a specific surface of a pallet with high accuracy, for example. [Explanation of symbols]

[0088] 1 palette 10 Plane detection device 20 Control Unit 30 Output section 101 Information acquisition department 102 Likelihood acquisition unit 103 Plane detection unit 201 Storage section 1a Flat plate part 1b First support part 1br First Side 1c Second support part 1cr Second Side P thing

Claims

1. an information acquisition unit that acquires visible image information of an object having a predetermined plane and 3D coordinate information corresponding to the visible image information; a likelihood acquisition unit that acquires, from the visible image information, a likelihood indicating a flatness of the predetermined plane of the object; a plane detection unit that detects the predetermined plane of the object by a robust estimation method using the 3D coordinate information and the likelihood; a storage unit that stores a machine learning model constructed by machine learning, the likelihood acquisition unit acquires the likelihood for each pixel of the visible image information by using the visible image information as input information and the machine learning model; the target object is a pallet capable of loading the object, the pallet including a flat plate portion on which the object is loaded and a first support portion extending from the flat plate portion in a direction in which the object is loaded, The predetermined plane includes a first surface of the first support portion. Plane detection device.

2. The plane detection unit a plurality of sample points randomly selected from the 3D coordinate information; 2. The plane detection device according to claim 1, wherein the predetermined plane of the object is detected by RANSAC using the likelihood corresponding to each of the plurality of sample points.

3. The pallet is A second support portion is provided separately from the first support portion and extending from the flat plate portion in the direction in which the object is loaded, The second support portion is a second surface; The predetermined plane is having the first surface and the second surface; The plane detection device according to claim 1 .

4. The plane detection unit a plurality of sample points randomly selected from the 3D coordinate information; detecting the predetermined plane of the pallet by RANSAC using the likelihood corresponding to each of the plurality of sample points; The plurality of sample points are At least one point is selected for the first surface and at least one point is selected for the second surface; 4. The plane detection device according to claim 3.

5. Obtaining visible image information of an object having a predetermined plane and 3D coordinate information corresponding to the visible image information; obtaining a likelihood indicating a flatness of the predetermined plane of the object from the visible image information; Detecting the predetermined plane of the object by a robust estimation method using the 3D coordinate information and the likelihood; the likelihood is acquired for each pixel of the visible image information using the visible image information and a machine learning model constructed by machine learning; the target object is a pallet capable of loading the object, the pallet including a flat plate portion on which the object is loaded and a first support portion extending from the flat plate portion in a direction in which the object is loaded, The predetermined plane includes a first surface of the first support portion. Plane detection method.

6. A computer-readable recording medium having recorded thereon a program for causing a computer to execute the method according to claim 5.

7. A program for causing a computer to execute the method according to claim 5.

Images