Item recognition device and item recognition method

Abstract

To properly recognize each of a plurality of articles arranged in a close-contact manner without preparing information on a pattern and size of a recognition object article in advance.SOLUTION: An article recognition device 10 according to an embodiment of the present invention comprises: an article region reception unit 11 which receives input of information of an article region obtained by performing polygonal approximation on a region where pixels continuously exist; a template determination unit 12d which generates a template image using information about a region where a pattern amount of a pattern region is equal to or greater than a prescribed first threshold and a distance to the closest angle of the article region is minimum; and an article surface information acquisition unit 12g which acquires and outputs article surface information in the article region by using information on one or more similar regions whose similarity to the template image is equal to or greater than a prescribed second threshold.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an article recognition device and an article recognition method. [Background technology]

[0002] Conventionally, for example, in logistics warehouses, robots such as picking robots have been used as an alternative to human labor in processes such as sorting products, palletizing cases, and depalletizing. Picking robots have robot arms with grippers or the like attached to the tips, and grasp and manipulate items based on information acquired by sensors.

[0003] One of the functions that influences the picking performance of a picking robot is the item recognition function for analyzing information acquired by sensors. Generally, the item recognition function can estimate the position and orientation of an item within a work scene when given 2D images and 3D point clouds of the work scene acquired by sensors. Therefore, the item recognition function plays an important role in the subsequent accurate grasping and manipulation of the item by the picking robot.

[0004] The difficulty of recognizing an item using an item recognition function varies depending on the shape, size, surface pattern, arrangement, lighting, and other environmental factors of the item to be recognized, but the difficulty increases when multiple items are lined up closely together. A situation in which multiple items are lined up closely together occurs, for example, when a container is filled with products (items) immediately after arrival in a logistics warehouse, or when cardboard boxes are densely stacked.

[0005] When objects are in close contact with each other, the gap between them becomes very small, making it difficult for a depth sensor to detect the gap. Although it may be possible to detect very small gaps by using a high-precision depth sensor, high-precision depth sensors are very expensive, which increases production costs. Furthermore, even when a high-precision depth sensor is used, stable measurement may not be possible depending on the size of the gap.

[0006] Furthermore, because gaps between objects appear as straight lines with similar pixel values ​​in a two-dimensional image, one approach to improving object recognition performance would be to identify object boundaries by detecting straight lines in the two-dimensional image. However, if an object has a pattern on its surface and that pattern is linear, an object recognition device with an object recognition function may mistakenly determine that pattern as an object boundary, which could result in incorrect object separation.

[0007] In recognition technology for closely aligned objects (hereinafter also referred to as "densely aligned objects"), there is also an approach that utilizes information about the object to be recognized. Information that can be utilized for object recognition includes the shape, size, and surface pattern of the object. If this information can be obtained in advance, it becomes possible to detect the object more accurately by matching the data acquired by the sensor with the previously obtained information about the object to be recognized. However, when preparing information about the object to be recognized in advance, the cost of obtaining (preparing) the information about the object to be recognized becomes enormous if there are many types of objects to be handled.

[0008] One possible method for reducing the time required to acquire information about the object to be recognized is to extract part of the information about the object to be recognized from data acquired by a sensor. For example, Patent Document 1 discloses a work position recognition device that detects the position and orientation of each work from contour data acquired from an image, and outputs the position and orientation of each work when pattern matching between a master image pattern, which is an image of the work part in the image, and the image of each work part is successful. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-32258 Summary of the Invention [Problem to be solved by the invention]

[0010] The workpiece position recognition device described in Patent Document 1 performs pattern matching on each mark or pattern on the top surface of the workpiece. This allows the workpiece position recognition device to verify whether the detected position and posture of each workpiece contain any erroneous detection results. However, to extract pattern information of the recognition target item from given recognition target data, it is first necessary to identify candidate areas of the recognition target item.

[0011] In this case, it is conceivable that, even if information about the shape of an item is provided as information for identifying a candidate region, information about the size of the item is not provided. Under such circumstances, when an item recognition device performs item recognition on densely aligned items, it is conceivable that the number of item arrangement patterns realizing the alignment surface of the densely aligned items will be enormous. In this case, it becomes difficult for the item recognition device to properly detect the boundaries of each item in the densely aligned items and correctly extract the items. If item size information is acquired in advance to avoid such a situation, this advance acquisition is costly.

[0012] The present invention has been made in consideration of the above circumstances, and an object of the present invention is to enable appropriate recognition of each of a plurality of closely arranged objects without having to prepare in advance information such as the pattern or size of the objects to be recognized. [Means for solving the problem]

[0013] An item recognition device according to one embodiment of the present invention comprises an input unit that receives input of information about an item area obtained by polygonal approximation of an area where pixels exist contiguously in a two-dimensional image including an image corresponding to the alignment surface of a plurality of items arranged closely together; a template image generation unit that generates a template image using information about an area where a pattern amount of a pattern area where pixels are densely packed at a predetermined value or more is equal to or greater than a predetermined first threshold and where the distance to the closest angle to the item area is the smallest among the corners that make up the item area; and an item surface information acquisition unit that acquires and outputs item surface information including the position and size of the surface of the item in the item area using information about one or more similar areas whose similarity to the template image is equal to or greater than a predetermined second threshold.

[0014] Furthermore, an item recognition method according to one aspect of the present invention includes the steps of: inputting, into an input unit, information about an item area obtained by polygonal approximation of an area where pixels exist contiguously in a two-dimensional image including an image corresponding to the alignment surface of a plurality of items arranged closely together; a template image generation unit generating a template image using information about an area where a pattern amount of a pattern area where pixels are densely packed equal to or greater than a predetermined value is equal to or greater than a predetermined first threshold and where, among the corners that make up the item area, the distance to the closest angle to the item area is the smallest; and an item surface information acquisition unit acquiring and outputting item surface information including the position and size of the surface of the item in the item area using information about one or more similar areas whose similarity to the template image is equal to or greater than a predetermined second threshold. [Effects of the Invention]

[0015] According to at least one aspect of the present invention, it becomes possible to properly recognize each of a plurality of closely arranged articles without preparing information such as the pattern or size of the article to be recognized in advance. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0016] [Figure 1]1 is a block diagram showing a schematic configuration of an item recognition device according to an embodiment of the present invention. [Figure 2] FIG. 10 is a diagram illustrating an example of an article region according to an embodiment of the present invention. [Figure 3] 1 is a block diagram showing an example of the configuration of hardware that constitutes an item recognition device according to an embodiment of the present invention. [Figure 4] FIG. 4 is a diagram showing an overview of a template candidate extraction process performed by a template candidate extraction unit according to an embodiment of the present invention. [Figure 5] FIG. 10 is a diagram showing an overview of processing by a template candidate evaluation unit and a template determination unit according to an embodiment of the present invention. [Figure 6] FIG. 10 is a diagram showing an overview of processing by a template candidate evaluation unit and a template determination unit according to an embodiment of the present invention. [Figure 7] FIG. 10 is a diagram showing an overview of processing by a template candidate evaluation unit and a template determination unit according to an embodiment of the present invention. [Figure 8] 10 is a diagram showing an article area in which three articles each having two star patterns and two line patterns are closely aligned in accordance with one embodiment of the present invention. FIG. [Figure 9] FIG. 10 is a diagram showing an example of the configuration of a recommended countermeasure information table according to an embodiment of the present invention. [Figure 10] FIG. 10 is a diagram showing an overview of processing by an article surface information acquisition unit according to an embodiment of the present invention. [Figure 11] FIG. 10 is a diagram showing an overview of processing by an article surface information acquisition unit according to an embodiment of the present invention. [Figure 12] FIG. 10 is a diagram showing an overview of processing by an article surface information acquisition unit according to an embodiment of the present invention. [Figure 13] FIG. 10 is a diagram showing an overview of processing by an article surface information acquisition unit according to an embodiment of the present invention. [Figure 14] 10 is a flowchart illustrating an example of a procedure for an item recognition process performed by an item recognition device according to an embodiment of the present invention. [Figure 15] 1 is a diagram showing an example of information on an article surface output from an article recognition device according to an embodiment of the present invention; [Figure 16] 1 is a diagram showing an example of a work scene to which an item recognition device according to an embodiment of the present invention is applied; [Figure 17] 10A and 10B are diagrams illustrating an example of a work scene in which an article has a cylindrical shape according to a modified example of the present invention. [Figure 18] FIG. 10 is a block diagram showing a schematic configuration of an article recognition device according to a modified example of the present invention. [Figure 19] 10 is a flowchart showing an example of the procedure of an item recognition process performed by an item recognition device according to a modified example of the present invention. [Figure 20] 1 is a diagram showing an example of a work scene when an item recognition device according to an embodiment of the present invention is applied to an apparatus that performs recognition processing of items to be picked by a picking robot. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, examples of modes for carrying out the present invention (hereinafter referred to as "embodiments") will be described with reference to the accompanying drawings. The present invention is not limited to the embodiments, and various numerical values ​​in the embodiments are merely examples. Furthermore, in this specification and drawings, identical components or components having substantially the same functions will be designated by the same reference numerals, and redundant explanations will be omitted.

[0018] [Configuration of the item recognition device] First, the configuration of an item recognition device according to one embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a block diagram showing a schematic configuration of an item recognition device 10 according to one embodiment of the present invention.

[0019] As shown in FIG. 1, the item recognition device 10 of this embodiment includes an item region receiving unit 11, a recognition processing unit 12, and an item surface information output unit 13.

[0020] In this embodiment, information about an item region corresponding to the alignment surface of a plurality of items aligned in close contact with one another is input to the item recognition device 10. Then, when executing the item recognition process, the item recognition device 10 extracts a template image for template matching from the item region, which is used when acquiring information about the surface of each item (hereinafter also referred to as "item surface") from the item region. Then, the item recognition device 10 extracts similar regions from the item region whose similarity to the extracted template image is equal to or greater than a predetermined threshold, and acquires information about the item surface (position, size, etc.) of each item that constitutes the alignment surface based on the information about the similar regions.

[0021] In other words, the item recognition device 10 of this embodiment uses a template image extracted when executing the item recognition process to determine information about each item surface within the item area, so there is no need to provide information about the pattern of the item that is the target of the item recognition process or information about the size of the item in advance.

[0022] The item recognition device 10 according to this embodiment achieves the highest item recognition performance when the item area is composed of surfaces (item surfaces) of the same type of item, but is also applicable when the item area is composed of two or more types of item surfaces.

[0023] The item area receiving unit 11 (an example of an input unit) receives information about the item area. The item area is information obtained by extracting a portion of the alignment surface from an image captured by a sensor of the alignment surface of multiple items aligned in close contact with each other. The sensor is configured, for example, with an RGB-D (Red Green Blue-Depth) camera or the like, and outputs a two-dimensional image corresponding to the three-dimensional point cloud as the captured image.

[0024] The contour of an image captured by a sensor is generally expressed as a polygon, such as a rectangle. When the captured image contains an area formed by one or more surfaces of the same type of object, information on the object area corresponding to the alignment surface on which multiple objects are aligned in close contact can be obtained by cutting out the area and approximating the contour of the area with a polygon.

[0025] Then, the item region receiving unit 11 outputs the acquired information about the item region to the recognition processing unit 12. The pixel values ​​in the item region are expressed as scalar quantities, multidimensional vector quantities, etc. For example, if the item region is represented by an RGB image, the pixel values ​​are expressed as three-dimensional vectors.

[0026] Note that the information about the item region input to the item recognition device 10 according to this embodiment is required to include a distinctive pattern that does not overlap with other types of patterns within the item region. This is because the item recognition device 10 according to this embodiment generates a template image using information about a partial region that includes a distinctive pattern included in an item and the distance between the partial region and the closest angle to the item region. The closest angle to the item region is the corner that is closest to the partial region of interest among several corners that make up the item region.

[0027] In other words, the object area in this embodiment can be said to be constructed by laying out one or more types of polygons on a two-dimensional plane, each of which contains a distinctive pattern that does not overlap with other types of patterns within the object area, and in which the pixel values ​​of areas other than the pattern area are predetermined constant values.

[0028] An example of an item area will now be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of an item area. Fig. 2A shows a rectangular item area 21 in which six rectangular items 211 are closely aligned. The items 211 include one black star pattern and two straight line patterns.

[0029] Figure 2B shows a decagonal article area 22 in which five articles 211 shown in Figure 2A are closely aligned. In the article area 22, the sides of each article 211 are arranged perpendicular or horizontal to one another. Figure 2C shows an octagonal article area 23 in which five parallelogram-shaped articles 231 are closely aligned. The article 231 includes one black star pattern and one line pattern.

[0030] Returning to Figure 1, the explanation will continue. The recognition processing unit 12 calculates the position of one or more object surfaces in the object region, and outputs information on the calculated object surface positions to the object surface information output unit 13. The object surface position is information necessary to identify the area occupied by the object surface within the object region, and includes, for example, the vertex coordinates of the object surface.

[0031] The recognition processing unit 12 includes a pattern amount calculation unit 12a, a template candidate extraction unit 12b, a template candidate evaluation unit 12c, a template determination unit 12d, a similar region detection unit 12e, a similar region evaluation unit 12f, an article surface information acquisition unit 12g, and a template candidate information storage unit 12h.

[0032] The pattern amount calculation unit 12a calculates the distribution of pattern amounts in the object area input from the object area receiving unit 11. The pattern amount is an index that indicates the degree of pattern in the image contained in a specified area, and is calculated from the pixel values ​​of the two-dimensional image that constitutes the object area. The pattern degree indicates the probability that the area constitutes some kind of pattern, and the pattern is, for example, a star pattern or the like shown in Figures 2A and 2B.

[0033] The amount of pattern is indicated by, for example, a luminance gradient that indicates the difference in pixel value between adjacent pixels, saliency that quantifies how easily an image is noticed by humans, a local image feature represented by SIFT (Scale Invariant Feature Transform), an image feature calculated using a neural network, etc. Alternatively, the amount of pattern may be indicated by a combination of one or more of these.

[0034] The template candidate extraction unit 12b extracts one or more partial regions within the article region where the pattern amount (or density of the pattern amount) is equal to or greater than a predetermined threshold (an example of a first threshold) as candidates for template images (hereinafter referred to as "template candidates"). The threshold applied to the pattern amount can be set to any value based on information obtained from actual photographed images of an article that includes a pattern. The density of the pattern amount indicates the degree of density of multiple images detected as a pattern.

[0035] The template candidate evaluation unit 12c calculates the closest angle, that is, the distance between each of the template candidates extracted by the template candidate extraction unit 12b and the closest corner among the corners that make up the article region.

[0036] The template candidate information holding unit 12h holds correspondence information between one or more template candidates and one or more distances (distances to the closest angles) corresponding to the template candidates. The template candidate information holding unit 12h also holds a template image generated by the next template determination unit 12d or an image generated based on the article surface information acquired by the article surface information acquisition unit 12g. The template image held in the template candidate information holding unit 12h or the image generated based on the article surface information is also used in the second and subsequent processes performed by the similar region detection unit 12e, the similar region evaluation unit 12f, and the article surface information acquisition unit 12g.

[0037] The template determination unit 12d (an example of a template image generation unit) refers to the template candidate information storage unit 12h, determines the template candidate with the smallest distance from the closest angle of the object region as the template for generating the template image, and then generates a template image from the selected template candidate.

[0038] The similar region detection unit 12e detects one or more similar regions within the article region using the template image extracted by the template candidate information storage unit 12h. A similar region is a region that includes an image whose similarity to the template image is equal to or greater than a predetermined threshold (an example of a second threshold). The threshold for measuring the level of similarity can be appropriately set to an expected similarity to the template image.

[0039] The similar region evaluation unit 12f selects the similar region that has the smallest distance from the closest angle to the object region among the similar regions detected by the similar region detection unit 12e and calculates the relative positional relationship (an example of relative positional relationship information) between the object region and the closest angle. The relative positional relationship is information for specifying the position of the similar region in the object plane that includes the similar region of interest, and is indicated, for example, by a two-dimensional vector that represents the displacement between a predetermined point in the similar region of interest and the closest angle to the object region.

[0040] The object surface information acquisition unit 12g calculates information (position and size) of one or more object surfaces in the object region based on the relative positional relationship calculated by the similar region evaluation unit 12f. Then, the object surface information acquisition unit 12g outputs the obtained information of one or more object surfaces (hereinafter also referred to as "object surface information") to the object surface information output unit 13.

[0041] The article surface information output unit 13 outputs information on one or more article surfaces input from the recognition processing unit 12 to an external robot (not shown) or the like.

[0042] [Example of computer hardware configuration] Next, the hardware configuration of the item recognition device 10 shown in Fig. 1 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of the hardware configuration of the item recognition device 10.

[0043] 3 is hardware used as a computer in the item recognition device 10. The calculator 200 includes a processor 201, a memory 202, an auxiliary storage device 203, an input device 204, an output device 205, and a communication IF (Interface) 206. The elements constituting the calculator 200 are connected to each other via a bus 37 so as to be able to communicate with each other.

[0044] The processor 201 is composed of a CPU (Central Processing Unit) or an MPU (Multi-Functional Peripherals), and performs the item recognition processing according to this embodiment (each process executed by each part of the item recognition device 10) by reading and executing a program stored in the memory 202.

[0045] The memory 202 is composed of, for example, a nonvolatile storage element and a volatile storage element. The nonvolatile storage element is composed of, for example, a ROM (Read Only Memory), and the ROM stores programs that do not need to be changed. The volatile storage element is composed of, for example, a RAM (Random Access Memory), and the RAM temporarily stores programs executed by the processor 201 and data used when the programs are executed.

[0046] In this embodiment, the program stored in the memory 202 is assumed to be a program for executing an item recognition process, but various other programs may also be stored. The memory 202 also stores input data and output data for executing the program.

[0047] The auxiliary storage device 203 includes a nonvolatile large-capacity storage device such as a magnetic storage device (HDD: Hard Disk Drive), and stores programs executed by the processor 201 and data (including databases, etc.) used when the programs are executed. The program for the item recognition processing in this embodiment is, for example, stored in advance in the auxiliary storage device 203, read out from there, loaded into the memory 202, and executed by the processor 201. In other words, the memory 202 or the auxiliary storage device 203 is used as an example of a computer-readable non-transitory storage medium that stores a program executed by the computer 200.

[0048] The input device 204 is an input unit configured with a keyboard, a mouse, etc., and receives input from an operator (not shown). The computer 200 receives instructions such as program execution and input data specification via the input device 204.

[0049] The output device 205 is composed of a display device that displays information or a printer that prints information, and visibly outputs the execution results of the program. For example, the output device 205 displays or outputs the article surface position, etc., output from the article surface information output unit 13 (see FIG. 1).

[0050] The communication IF 206 is a network interface device that connects the item recognition device 10 to other devices and controls communication between them.

[0051] The hardware configuration of this computer 200 may be configured as a single computer, or one or more components of the computer 200, namely, the processor 201, memory 202, auxiliary storage device 203, input device 204, output device 205, and communication IF 206, may be configured as one or more computers connected by a network not shown.

[0052] [Processing by template candidate extraction unit] Next, the template candidate extraction process performed by template candidate extraction unit 12b (see FIG. 1) will be described with reference to Fig. 4. Fig. 4 is a diagram showing an overview of the template candidate extraction process performed by template candidate extraction unit 12b.

[0053] Based on the pattern amount distribution calculated by the pattern amount calculation unit 12a, the template candidate extraction unit 12b extracts, as template candidates, one or more partial regions within the article region where the pattern amount or density of the pattern amount is equal to or greater than a predetermined threshold value. The example shown on the left side of Fig. 4 shows an example in which template candidates Tc1 to Tc5 including star-shaped patterns are extracted from the article region 21, and the template candidates Tc1 to Tc5 are each indicated by a dashed rectangular frame. The template candidates extracted by the template candidate extraction unit 12b may differ from one another in shape or size, and may overlap one another.

[0054] It is preferable that an image extracted as a template candidate belongs to a single item. If an image extracted as a template candidate is an image formed across multiple items, the similar regions extracted using the template image will reflect information about the multiple items. Then, the information about the object surfaces extracted based on the information about such similar regions will no longer correspond to each individual item.

[0055] In particular, when there is a distinctive pattern near the boundary between objects, an image of an area spanning multiple objects may be extracted as a template candidate. In the example shown on the left side of Figure 4, template candidates Tc2 and Tc5 are examples of this.

[0056] Template candidates that span multiple items are more likely to have line patterns resulting from the boundaries between items than those that belong to a single item. Furthermore, such line patterns are located at positions that divide the area of ​​the template candidate. Therefore, by excluding template candidates that have line patterns at positions that divide the area of ​​the template candidate from the template candidates, it is possible to prevent images of areas that span multiple items from being extracted as template candidates.

[0057] The right side of Fig. 4 shows an example of an object region 21A after template candidates having a linear pattern at a position dividing the region of the template candidate have been excluded from the template candidates. In the object region 21A shown on the right side of Fig. 4, template candidates Tc2 and Tc5 shown on the left side of Fig. 4 have been excluded from the template candidates. The template candidate extraction unit 12b outputs information on template candidates Tc1, Tc3, and Tc4 to the template candidate evaluation unit 12c.

[0058] [Processing by the template candidate evaluation unit and template determination unit] Next, the processing by the template candidate evaluation unit 12c and the template determination unit 12d will be described with reference to Figures 5 to 7. Figures 5 to 7 are diagrams showing an overview of the processing by the template candidate evaluation unit 12c and the template determination unit 12d.

[0059] For each template candidate extracted by the template candidate extraction unit 12b, the template candidate evaluation unit 12c calculates the distance between the closest corner (closest angle) among the corners of the object region. As this distance, the template candidate evaluation unit 12c calculates, for example, the minimum value of the two-dimensional Euclidean distance between a corner of interest in the object region and a pixel of interest in the template candidate.

[0060] Here, the template candidate evaluation unit 12c selects only corners of the item region whose difference from the smallest angle of the corners of the item region is less than a predetermined threshold (an example of a third threshold) as corners of the item region for which the distance to the template candidate is to be calculated. This processing is performed so that template candidates belonging to a single item can be extracted from multiple template candidates in the subsequent template determination processing performed by the template determination unit 12d.

[0061] To achieve the above objective, the corner in the item region that is focused on when calculating the distance to the closest angle must be a corner occupied by a single item, not a corner occupied by multiple items. To exclude corners occupied by multiple items from the candidates for selection (focus), the template candidate evaluation unit 12c identifies the minimum value of the angles of the corners in the item region and excludes from the candidates for selection any corner whose difference from the minimum value is equal to or greater than a predetermined threshold. The threshold value associated with the difference between the minimum angle of the corner in the item region and the angle of the corner of focus can be set to any value that allows appropriate determination of template candidates.

[0062] Since the angle of an angle occupied by multiple items is smaller than the angle of an angle occupied by a single item, by performing this processing, it is possible to exclude angles occupied by multiple items from the angles considered when calculating the distance to the closest angle. However, for this processing to be valid, the minimum angle of the corner of the item surface must match the minimum angle of the item area angle.

[0063] In both the object region 22 shown on the left side of Fig. 5 and the object region 23 shown on the right side, the corners selected (focused) as targets for calculating the distance to the closest angle are indicated by black circles. In the object region 22 shown on the left side of Fig. 7, only angles with an interior angle of 90 degrees are selected, and angles with an angle of 270 degrees, which is greater than 90 degrees, are not selected.

[0064] 5, only corners with interior angles of 60 degrees are selected, and angles of 120 degrees and 240 degrees, which are greater than 60 degrees, are excluded from the selection of corners. By performing such processing by the template candidate evaluation unit 12c, it becomes possible to exclude corners occupied by multiple items from targets for selection (attention) when calculating the distance to the closest angle.

[0065] Fig. 6 shows an object region 21 in which the distance to the closest angle is indicated by an arrow. In Fig. 6, template candidates Tc1 to Tc5 are indicated by rectangular frames drawn with dashed or solid lines, and two-dimensional vectors corresponding to the distance from the closest angle of the object region 21 are indicated by solid arrows. Template candidate Tc1, indicated by a rectangular frame drawn with solid lines, is the region in which the distance D1 from the closest angle Cc1 is the smallest. On the other hand, template candidates Tc2 to Tc5, indicated by rectangular frames drawn with dashed lines, are regions in which the distances D2 to D5 from the respective closest angles Cc2 to Cc4 are not the smallest.

[0066] Among these template candidates Tc2 to Tc5 for which the distances D2 to D5 from the closest angles Cc2 to Cc4 are not the smallest, there is a possibility that a template candidate that spans multiple articles is included. In fact, template candidate Tc2 and template candidate Tc5 span multiple articles. If such a template candidate is determined as a template image by the template determination unit 12d and the position of the article surface is calculated using this template image, the article boundary may be erroneously detected, and the article may be erroneously separated.

[0067] As described above, by selecting the template candidate with the smallest distance from the closest angle as the template image, it is possible to prevent erroneous recognition of the boundary of an article. However, by having the template candidate evaluation unit 12c transform the region of the template candidate, it is possible to further increase the probability that only template candidates that belong to a single article can be selected.

[0068] A template candidate that spans multiple items may appear by including patterns near the boundaries of adjacent items. Therefore, if the template candidate evaluation unit 12c transforms (shrinks) the template candidate area so that it is limited to a smaller subarea, it becomes possible to narrow down the template candidates to only those that belong to a single item.

[0069] The template can be deformed, for example, by shrinking the template candidate by a predetermined percentage in the direction of the closest angle of the article region. The amount of reduction may be set to a fixed size, such as 1 / 2, or may be set to an arbitrary size based on the size of the detected pattern. Alternatively, the region of the template candidate may be deformed by shrinking it to a size that no longer includes the linear pattern that divides the region of the template candidate.

[0070] An overview of the transformation process for the template candidate region is shown in Fig. 7. The item region 24 shown on the left side of Fig. 7 and the item region 24A shown on the right side are regions formed by four closely-spaced items 241 each including one star-shaped pattern and two straight line patterns.

[0071] On the left side of Fig. 7, three template candidates Tc11 to Tc13 are indicated by solid rectangular frames. Of these template candidates Tc11 to Tc13, template candidate Tc11 has the shortest distance to the closest angle. However, template candidate Tc11 spans multiple articles.

[0072] The right side of Fig. 7 shows an example of an article region 24A when the template candidate evaluation unit 12c reduces the region of each of the template candidates Tc11 to Tc13 in the direction of the closest adjacent angle. In the article region 24 shown on the right side of Fig. 7, the template candidates Tc11 to Tc13 before reduction are indicated by rectangular frames drawn with dashed lines, and the template candidates Tc11' to Tc13' after reduction are indicated by rectangular frames drawn with solid lines. It can be seen that among the template candidates Tc11' to Tc13' whose regions have been reduced in the direction of the closest adjacent angle, there are no candidates that span multiple articles.

[0073] The template determination unit 12d determines the template candidate having the smallest distance from the closest angle as the template to be used to generate the template image.

[0074] Here, we will explain the effect obtained by the template determination unit 12d selecting template candidates that are close to the closest corner of the item region. The area near the corner of the item region is more likely to be occupied by a single item than the area inside the item region. Therefore, template candidates located near the corner are more likely to belong to a single item than template candidates located inside the item region.

[0075] On the other hand, a template image generated based on a template candidate that exists at a position that spans multiple items is likely to span multiple items. Therefore, the position of the object surface calculated based on that template image is likely to differ from the actual position of the object surface. In other words, it becomes impossible to correctly separate individual items that are closely aligned with each other. To prevent such erroneous separation, the template determination unit 12d of this embodiment selects a template candidate that is close to the closest adjacent angle of the object region and generates a template image.

[0076] The template image does not necessarily have to match the template candidate selected by the template determination unit 12d. For example, the shape may be a bounding box surrounding the selected template candidate. Furthermore, the components may not necessarily be the pixel values ​​of the corresponding regions, but may be some image feature calculated from the pixel values, as long as it enables subsequent similarity calculations.

[0077] [Processing by the similar region detection unit] Next, the processing by the similar region detection unit 12e (see FIG. 1) will be described. The similar region detection unit 12e detects one or more regions (similar regions) within the object region that are similar to the template image determined by the template determination unit 12d. The similar region detection unit 12e extracts (detects) as similar regions a region whose pixel value or a similarity to the template image calculated from image features calculated based on the pixel value is equal to or greater than a predetermined threshold (an example of a second threshold).

[0078] The similar region detection unit 12e acquires a group of images obtained by rotating the template image and the object region including the template image by an arbitrary angle in two-dimensional space within the object region, by scanning the object region. The rotation angle here can be selected arbitrarily depending on the shape of the object surface.

[0079] For example, in the object region 21 shown in Fig. 2A or the object region 22 shown in Fig. 2B, if the template image is rotated at three angles, 90 degrees, 180 degrees, and 270 degrees, it is possible to detect an area similar to the template image within the object region 21 or 22. Furthermore, in the object region 23 shown in Fig. 2C, if the template image is rotated at five angles, 60 degrees, 120 degrees, 180 degrees, 240 degrees, and 300 degrees, it is possible to detect an area similar to the template image within the object region 23.

[0080] Note that the similar regions extracted by the similar region detection unit 12e may include regions that span multiple object surfaces. Therefore, in this embodiment, the similar region detection unit 12e re-evaluates the similarity using area information obtained by enlarging each similar region, in order to further improve the accuracy of detecting similar regions. The threshold used in this similarity evaluation may be different from the threshold used to measure the similarity when detecting similar regions.

[0081] Fig. 8 shows an article region 25 in which three articles 251, each having two star patterns and two straight line patterns, are closely aligned. On the left side of Fig. 8, the template image Tp11 is indicated by a solid rectangular frame, and the template candidates Tc21 to Tc25 are indicated by dashed-dotted rectangular frames.

[0082] The template image Tp11 has the shortest distance D21 to the closest angle Cc21 of the article region 25 and is a region that does not straddle multiple article surfaces, and therefore was determined as the template image by the template determination unit 12d.

[0083] In the center of Fig. 8, similar regions Fs11 to Fs15 extracted using template image Tp11 are indicated by solid-line rectangular frames. However, in the example shown in the center of Fig. 8, two similar regions are included in each of the three object surfaces (object surfaces of object 251) included in object area 25A. If the positions of the object surfaces are calculated based on such similar regions, the boundaries of the objects will be erroneously recognized.

[0084] The right side of Fig. 8 shows an example of the object region 25B after the similar region detection unit 12e has newly evaluated the similarity using information on a portion of each similar region that corresponds to that region. In the example shown on the right side of Fig. 8, the similar region detection unit 12e first focuses on similar region Fs11, of the similar regions Fs11 to Fs16, which has the shortest distance D22 (center of Fig. 8) from the closest angle Cc12 of the object region 25, and expands the similar region Fs11 in the direction of the closest angle Cc12 to obtain a similar region Fs11' (an example of an expanded similar region).

[0085] The similar region detection unit 12e also performs similar enlargement on the other similar regions Fs12 to Fs16 to obtain similar regions Fs12' to Fs16' (examples of enlarged similar regions). The enlargement of the other similar regions Fs12 to Fs16 must be performed taking into account the angle of rotation applied to obtain the template image. The similar region detection unit 12e then calculates the similarity between the enlarged region Fs11' (an example of a first enlarged region) of the similar region Fs11 and the similar regions Fs12' to Fs16'.

[0086] Because similar regions Fs12', Fs14', and Fs16' do not contain a linear pattern within the object surface, their similarity to similar region Fs11', which does contain a linear pattern within the object surface, is low. The similar region detection unit 12e excludes, i.e., discards, such similar regions whose similarity to similar region Fs11' is less than a predetermined threshold (an example of the fourth threshold). On the right side of Figure 8, the discarded similar regions Fs12', Fs14', and Fs16' are indicated by dashed rectangular frames, and the remaining similar regions Fs13' and Fs15' are indicated by solid rectangular frames.

[0087] By the similar region detection unit 12e re-evaluating the similar regions in this way, only the similar regions Fs13' and Fs15' that belong to one article surface are treated as similar regions.

[0088] [Processing by the similarity region evaluation unit] Next, the processing by the similar region evaluation unit 12f (see FIG. 1) will be described with reference to Fig. 9. Fig. 9 is a diagram showing an outline of the processing by the similar region evaluation unit 12f.

[0089] The similar region evaluation unit 12f selects the similar region that has the smallest distance from the closest angle of the article region among the similar regions extracted by the similar region evaluation unit 12f and calculates the relative positional relationship between the similar region and the closest angle.

[0090] The relative positional relationship here is indicated by, for example, a two-dimensional vector that represents the displacement between a specific point in the similar region of interest and the closest angle to the object region. By using this two-dimensional vector, it is possible to identify where the similar region of interest is located within the object surface that includes the similar region of interest.

[0091] Furthermore, the similar area evaluation unit 12f can limit the selected corners to those occupied by a single item by selecting the corner of the item area when determining the closest angle of the item area as an angle whose difference from the smallest angle of the corner of the item area is less than a predetermined threshold.

[0092] As described above, the similar region evaluation unit 12f selects the similar region that has the smallest distance from the closest angle to the object region, rather than the similar region that corresponds to the template image, as the target for calculating the relative positional relationship with the closest angle. This is to enable the appropriate calculation of the object surface position even when the template image is generated from a template candidate that is not included in the object surface that indicates the corner of the object region due to factors such as the addition of template candidate selection conditions.

[0093] In the object region 21 shown in FIG. 9, the similar region Fs1, which has the smallest distance from the closest angle of the object region, is indicated by a solid rectangular frame, and the other similar regions Fs2 to Fs6 are indicated by dashed-dotted rectangular frames.

[0094] 9 also shows, with a solid arrow, a two-dimensional displacement vector R1 that indicates the relative positional relationship between the similar region Fs1, which is the closest to the closest angle of the object region, and its closest angle Cc1. By applying this two-dimensional displacement vector R1 to each similar region, the similar region evaluation unit 12f can calculate the position of the corner of the object surface corresponding to each similar region. In other words, the corner located at a distance from the edge of each similar region indicated by the two-dimensional displacement vector can be identified as the corner of the object surface corresponding to that similar region (hereinafter also referred to as the "article corner"). The position of the article corner is indicated by a black circle in FIG. 9.

[0095] [Processing by the item surface information acquisition unit] Next, the processing by the article surface information acquisition unit 12g will be described with reference to Fig. 10 to Fig. 13. Fig. 10 to Fig. 13 are diagrams showing an overview of the processing by the article surface information acquisition unit 12g.

[0096] The object surface information acquisition unit 12g calculates the position and size of one or more object surfaces within the object area based on the relative positional relationship with the object angle in each similar area calculated by the similar area evaluation unit 12f and information about each similar area, and acquires this as object surface information.

[0097] Specifically, the object surface information acquisition unit 12g calculates the position and size of the object surface based on the information about the object corners acquired by the similar area evaluation unit 12f. The position and size of the object surface can be determined by performing similarity scaling on the polygonal shape based on information about the polygonal shape of the object area and information about which corners of the object area correspond to the corners identified as corners of the object surface. The shape of the object surface is defined by the length (ratio) of the two sides extending from the vertices of the object corners.

[0098] Specifically, the item surface information acquisition unit 12g first determines candidate lengths for the two sides extending from the vertices of the item corners based on the distance between the item corners and the distance from the vertex of each item corner to the corner of the item area, and then adjusts the lengths of the two sides to lengths that satisfy the following conditions (1) to (3), etc., using the determined candidate lengths as initial values. (1) A length that does not cause overlapping of items, or a length that keeps the area of ​​overlapping parts of items below a predetermined threshold. (2) A length that does not extend outside the item area, or a length that does not extend outside the item area but is equal to or less than a predetermined threshold. (3) The length that can maximize the area of ​​each object surface, specified by the length of the two sides extending from the vertex of the object corner, in the object area.

[0099] The article surface information acquisition unit 12g may determine the lengths of the two sides extending from the vertex of the article corner using one of these conditions, or may combine one or more of these conditions. Furthermore, if information about the shape of the article surface is given in advance, the time required to search for the size of the article surface (the lengths of the two sides extending from the vertex of the article corner) can be reduced by using that information.

[0100] Here, an example of specifying the position and size of each object surface within the object region (hereinafter also referred to as "specifying the object surface") will be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of specifying each object surface within the object region by the object surface information acquisition unit 12g.

[0101] Note that, in the example shown in FIG. 10, the shape of the object surface is rectangular, but similar processing can be applied when the shape of the object surface is a parallelogram or the like. The object region 26 shown in the top row of FIG. 10 is composed of three objects 261. The object 261 has one star-shaped pattern and two straight line patterns. Furthermore, in the object region 26, the position of the object corner of each object surface is identified, and each object corner is indicated by a black circle in FIG. 10. Furthermore, in FIG. 10, the shape and size of the object surface, which are determined by the lengths of the two sides extending from the vertex of the object corner, are indicated by the shape and size of the area with a diagonal line pattern sloping downward to the right.

[0102] In the item area 26A shown on the left side of the middle row of Figure 10, the longer side of the two sides extending from the vertex of the item corner is too long, resulting in overlapping of items and protrusion of the item surface from the item area 26A.

[0103] In the item area 26B shown on the right side of the middle row of Figure 10, the lengths of the two sides extending from the vertex of the item corner are too short, so there is no overlap between the items, but the area occupied by the item surface in the item area 26B is small.

[0104] On the other hand, in the item area 26C shown in the bottom row of Figure 10, the lengths of the two sides extending from the vertex of the item corner are appropriately set, so that there is no overlap between the item surfaces or the item surfaces protruding from the item area 26, and the proportion of the item surfaces in the item area 26C is also maximized.

[0105] The object surface information acquiring unit 12g may also search for the object surface by using information on the similarity between object surfaces calculated based on pixel values. Fig. 11 is a diagram showing an example of identifying object surfaces in an object region including different types of objects.

[0106] 11 includes two items 271 each including a star pattern and two straight line patterns, and one item 272 each including a straight line pattern and two straight line patterns perpendicular to the star pattern. In addition, two item corners indicated by black circles are identified in the item area 27.

[0107] When estimating the lengths of the two sides extending from each of two object corners, if only the above conditions (1) to (3) are used, the conditions will be met even if the object surface of object 271 overlaps object 272, as in object area 27A shown in the upper right of Figure 11.

[0108] Here, let us add a condition (condition (4)) that the length is such that the similarity between the identified object surfaces is equal to or greater than a predetermined threshold. In this case, the pixel values ​​of the parts of the object surface 2711 including the object angle Ca11 and the object surface 2712 including the object angle Ca12 that overlap the object 272 are different, so the similarity determined based on the pixel values ​​is low. Therefore, the object surface information acquisition unit 12g determines that the lengths of the two sides that make up the object surfaces 2711 and 2712 are inappropriate.

[0109] On the other hand, when the lengths of the two sides that satisfy all of the above conditions (1) to (4) are applied, the object areas 2713 and 2714 will each appropriately reflect the position and size of the object 271, as shown in the object area 27B in the lower right of Figure 11.

[0110] It is also possible to calculate the position and size of an object surface without using information about the relative positional relationship between the closest angle of an object region and a similar region, as in the object recognition method of the above-described embodiment. However, in a situation where there is no information about the relative positional relationship between the closest angle of an object region and a similar region and only the position of the similar region is known, it becomes necessary to search for information about the position of the similar region in the object region in addition to the lengths of the two sides. In other words, the object search dimension becomes "4." In contrast, in this embodiment, it is only necessary to search for the lengths of the two sides extending from the vertices of the object corners, so the search dimension becomes "2," enabling search time to be reduced.

[0111] From the perspective of reducing search time, it is also possible to determine candidate values ​​for the lengths of the two sides based on information such as the relative positional relationship between article corners of the same type of article or between similar regions, and the distance from the vertex of the article corner to a point on the outline of the article region. For example, in the article region 26 shown in FIG. 10, the distance between article corner Ca1 and the lower left corner of the article region 26 matches the length of the long side of the article 261. The distance between article corner Ca2 and article corner Ca3 matches the length of the short side of the article 261 and the distance between article corner Ca3 and the lower right corner of the article region 26. In other words, by using information such as the relative positional relationship between article corners of the same type of article or between similar regions, and the distance from the vertex of the article corner to a point on the outline of the article region, it is possible to further improve the accuracy of searching for the article surface.

[0112] Furthermore, if the object corners of two adjacent object faces are located at diagonal positions in the object area, there may be two possible directions in which the two sides starting from the vertices of the object corners extend. In such cases, even if all of the above conditions (1) to (4) are met, the object faces may not be correctly identified.

[0113] Fig. 12 is a diagram showing an example of identifying an item surface in an item area where the item corners of two adjacent item surfaces are located at diagonal positions in the item area. The item area 28 shown in Fig. 12 includes two items 281, each including one straight line pattern and one star pattern. As shown in Fig. 12, the item corner Ca21 of the item 281 included in the item area 28 and the item corner Ca22 of the item 281 are located at diagonal positions to each other.

[0114] In this case, there are two options for the direction of the two sides extending from each of the item angles Ca21 and Ca22: a direction in which the longer side of the two sides divides the vertical direction of item area 28A, as in item area 28A shown in the center of Figure 12, and a direction in which the longer side divides the horizontal direction of item area 28B, as in item area 28B shown on the right side of Figure 12.

[0115] In such a case, by setting the direction in which the straight line pattern dividing the item area 28 exists to the direction in which the longer side of the two sides is arranged, it is possible to separate the item surface 2811 and the item surface 2822 at the same position as the boundary between the two actual items 281. The item area 28A shown in the center of Fig. 12 corresponds to this example.

[0116] Fig. 13 is a diagram showing an example of an appropriate specification of an object surface in an object region. In the object region 29 shown in Fig. 13, the lengths of the two sides extending from the vertices of the object corners are determined to satisfy any one or a combination of the conditions (1) to (4) described above, or all of them. Therefore, in the object region 29, there is no overlap between the object surfaces, and no object surfaces extend beyond the object region 29. Furthermore, in the object region 29, the patterns within each object surface match, and the object surfaces can cover almost the entire surface of the object region 29.

[0117] The article surface information acquisition unit 12g (see FIG. 1) outputs to the article surface information output unit 13 the information on the position and size of each article surface within the article region acquired in this manner.

[0118] [Item recognition method using an item recognition device] Next, an article recognition method performed by the article recognition device 10 (see FIG. 1) according to this embodiment will be described with reference to Fig. 14. Fig. 14 is a flowchart showing an example of the procedure for article recognition processing performed by the article recognition device 10.

[0119] First, upon receiving information about an object region from the object region receiving unit 11, the pattern amount calculation unit 12a of the object recognition device 10 calculates the distribution of pattern amounts in the object region (step S1). As described above, the pattern amount is an index that represents the degree of pattern in a given region, and is calculated from the two-dimensional pixel values ​​that make up the object region.

[0120] Next, based on the distribution of the pattern amount calculated in step S1, the template candidate extraction unit 12b extracts one or more partial areas within the article area where the pattern amount (or density of the pattern amount) is equal to or greater than a predetermined threshold as template candidates (step S2).

[0121] Next, the template candidate evaluation unit 12c calculates the distance from the closest angle for each of the one or more template candidates extracted in step S2 (step S3). Then, correspondence information between the template candidate and information on the distance (distance from the closest angle) corresponding to the template candidate is stored in the template candidate information holding unit 12h.

[0122] Next, the template determination unit 12d refers to the template candidate information storage unit 12h and determines whether there is one or more unselected template candidates (step S4). If it is determined in step S4 that there are no unselected template candidates (if the determination in step S4 is NO), the item recognition process by the item recognition device 10 ends. Note that the template determination unit 12d may also end the process here if it determines that subsequent processing cannot be performed due to reasons such as computation time constraints, regardless of whether there are unselected template candidates.

[0123] On the other hand, if it is determined in step S4 that there is an unselected template candidate (if the determination in step S4 is YES), the template determination unit 12d selects the template candidate having the smallest distance from the closest angle calculated in step S3, and generates a template image based on that template candidate (step S5).

[0124] Next, the template determination unit 12d determines whether the template image generated in step S5 satisfies a predetermined condition (step S6). The predetermined condition may be, for example, that the area of ​​the template image is equal to or greater than the area of ​​the pickup / holding portion when a pickup robot (not shown) picks up or holds a part. If it is determined in step S6 that the template image does not satisfy the predetermined condition, the template determination unit 12d returns to step S4 and performs the process.

[0125] On the other hand, if it is determined in step S6 that the template image satisfies the predetermined condition (if step S6 is determined as YES), the similar region detection unit 12e detects one or more regions within the item region that are similar to the template image as similar regions (step S7). As described above, the similar region detection unit 12e detects, as similar regions, regions whose similarity to the template image calculated from pixel values ​​or image features calculated based on pixel values ​​is equal to or greater than a predetermined threshold. At this time, the similar region detection unit 12e searches for regions that are highly similar to the template image, also targeting a group of images obtained by rotating the template image and the item region by any angle in two-dimensional space.

[0126] Furthermore, the similar regions detected by the similar region detection unit 12e include at least a region corresponding to the template candidate selected in step S5 for generating the template image.

[0127] As described above, after detecting similar regions, the similar region detection unit 12e may reevaluate the similarity with other similar regions by using information on a region obtained by expanding the similar region that is the shortest distance from the closest angle to the object region in the direction of the closest angle, and may then exclude regions with low similarity from the similar regions.

[0128] For example, if noise is included in the information about the object region received by the object region receiving unit 11 (see FIG. 1), a situation may arise in which a corner of the object region that is not close to the template candidate is erroneously recognized as the closest angle. In such a case, although the template candidate with the smallest distance to the closest angle should be selected, a template candidate that does not satisfy this condition may be erroneously selected.

[0129] By performing the re-evaluation by the similar region detection unit 12e, even if a template candidate different from the expected template candidate is selected, the similar region that has been erroneously detected using information on the similarity between the erroneously selected template candidate and the template image can be discarded.

[0130] Next, the similar region evaluation unit 12f selects the similar region that has the smallest distance from the closest angle of the item region among the one or more similar regions obtained in step S7, and calculates the relative positional relationship between the closest angle of that item region (step S8). As described above, the relative positional relationship is information used to identify where the similar region of interest is located within the item region that includes the similar region of interest, and is indicated, for example, by a two-dimensional vector that represents the displacement between a predetermined point in the similar region of interest and the closest angle of that similar region.

[0131] In step S8, the similar region evaluation unit 12f focuses on similar regions that are closer to the closest corner of the item region than similar regions that correspond to the template image because this process can be used to deal with cases where the template candidate determined by the template determination unit 12d is inappropriate. By performing this process, even if, for example, the template determination unit 12d erroneously selects a template candidate that includes a corner that is different from the corners that constitute the item region, the calculated relative positional relationship can be used as information regarding the closest angle that belongs to the corner that constitutes the item region.

[0132] The similar region evaluation unit 12f may also perform processing to exclude angles whose difference from the smallest angle of the angles constituting the item region is equal to or greater than a predetermined threshold from the selection candidates for the closest angle to be used in calculating the relative positional relationship with the similar region. By performing such processing, it is possible to prevent a corner occupied by multiple items from being erroneously selected as the closest angle.

[0133] Next, the article surface information acquisition unit 12g acquires information (position, size, etc.) of one or more article surfaces within the article region based on the relative positional relationship acquired in step S8 and the information of one or more similar regions acquired in step S7 (step S9). By using the relative positional relationship, the article surface information acquisition unit 12g can identify the position of a corner of the article surface including each of the one or more similar regions. Then, the article surface information acquisition unit 12g calculates (searches for) the size of the article surface by performing homothetic scaling of the polygonal shape of the article surface including the similar region.

[0134] Next, the item surface information acquisition unit 12g determines whether the item surface information calculated in step S9 satisfies predetermined conditions (step S10). The predetermined conditions are the above-mentioned conditions (1) to (4). The item surface information acquisition unit 12g determines whether the item surface information (the lengths (or ratio) of the two sides extending from the vertex of the item corner) calculated in step S9 satisfies any one, a combination, or all of the conditions (1) to (4).

[0135] The article surface information acquisition unit 12g may make a determination according to the constraints on the calculation time instead of the determination in step S10. That is, the article surface information acquisition unit 12g may end the processing here even if it determines that the subsequent processing cannot be performed due to the constraints on the calculation time, regardless of whether the article surface information satisfies the predetermined condition or not.

[0136] If it is determined in step S10 that the information on the article surface does not satisfy the predetermined condition (if the determination in step S10 is NO), the article surface information acquisition unit 12g removes the template candidate selected in step S5 from the template candidate information storage unit 12h, and then returns to step S4 to perform the process. In other words, it determines whether there is a next unselected template candidate.

[0137] On the other hand, if it is determined in step S10 that the information on the item surface satisfies the predetermined condition (if the determination in step S10 is YES), the item surface information acquisition unit 12g outputs the information on the item surface that satisfies the condition in step S10 to an external device such as a picking robot (not shown) via the item surface information output unit 13 (step S11). After the processing of step S11, the item recognition processing by the item recognition device 10 ends.

[0138] The above-described item recognition device 10 includes an item region receiving unit 11, a template determination unit 12d, and an item surface information acquisition unit 12g. Information about an item region is input to the item region receiving unit 11. The item region is a region obtained by polygonal approximation of a region in which contiguous pixels exist in a two-dimensional image that includes an image corresponding to the alignment surface of a plurality of items arranged closely together.

[0139] The template determination unit 12d generates a template image using information on a pattern region where the pattern amount of a pattern region where pixels equal to or greater than a predetermined value are densely concentrated is equal to or greater than a predetermined threshold (first threshold) and where, among the corners that make up the object region, the distance to the closest corner to the object region is the smallest. The object surface information acquisition unit 12g acquires and outputs object surface information including the position and size of the object surface in the object region using information on one or more similar regions whose similarity to the template image is equal to or greater than a predetermined threshold (second threshold).

[0140] Therefore, the item recognition device 10 of this embodiment can properly recognize each item of multiple items that are closely aligned based on the template image generated based on information obtained from the item area and the information on the similar area, even if information such as the pattern or size of the item to be recognized is not prepared in advance.

[0141] Furthermore, the template candidate extraction unit 12b of the item recognition device 10 of this embodiment extracts template candidates for generating a template image from the item area using information on an area where the pattern amount of the pattern area is equal to or greater than a predetermined threshold and, among the corners that make up the item area, the distance to the closest angle to the area itself is the smallest.

[0142] An area that satisfies these conditions is likely to be an area that belongs to the surface of a single item. In other words, in this embodiment, the object surface information of each object that constitutes the alignment surface is calculated based on template candidates that are assumed to correspond to areas that belong to the surface of a single item, so this embodiment makes it possible to appropriately obtain information about the object surfaces that constitute the alignment surface.

[0143] Here, we will explain the characteristics of the output content when information that satisfies predetermined conditions is input to the item recognition device 10 according to this embodiment. Items that can be recognized by the item recognition device 10 according to this embodiment are items that have a distinctive pattern that does not overlap with other patterns on the surface of the same item, and in which the pixel values ​​of areas other than the area with the pattern and areas where a different type of pattern exists are constant.

[0144] Furthermore, the shape of the object surface must be a shape that can be identified when one of the corners constituting the polygon and the ratio of the lengths of the two sides extending from that corner are given. In this embodiment, a captured image of an alignment surface in which such objects are aligned in close contact with one another is input to the object recognition device 10. However, what is input to the object recognition device 10 may not be a captured image of the actual alignment surface, but rather a pseudo-generated image.

[0145] The information on the object surface output from (the object surface information output unit 13 of) the object recognition device 10 according to this embodiment includes at least information on the position and size of the object surface, including the corners of the object region. The object recognition device 10 according to this embodiment performs the object recognition process on the assumption that, among the regions within the object region that include a distinctive pattern, the region that is the shortest distance from the closest corner of the object region includes one of the corners that make up the object region.

[0146] Therefore, information about the position (and size) of the object surface, including the corners of the object area, is always included in the information about the object surface output from the object recognition device 10. Therefore, by checking whether information output from another object recognition device includes information about the position (and size) of the object surface, including the corners of the object area, it is possible to determine whether that device is performing object recognition processing similar to that of the object recognition device 10 according to this embodiment.

[0147] FIG. 15 is a diagram showing an example of item surface information output from the item recognition device 10. The item area 30 shown in FIG. 15A is configured by four items 301 each having a star-shaped pattern arranged side by side. In the item area 30, the pixel values ​​in areas other than the star-shaped pattern area are assumed to be constant. In other words, the pixel values ​​of the boundary portions of the items are assumed to be the same as the pixel values ​​in areas other than the boundary. An item area with such characteristics may appear when a depalletizing process is performed on a single pallet.

[0148] When such information about the item region 30 is input, the information output from the item recognition device 10 according to this embodiment is information about the item faces 3011A to 3014A included in the item region 30A shown in the upper right of Fig. 15, or information about the item faces 3011B to 3014B included in the item region 30B shown in the lower right of Fig. 15. In the item region 30A, each of the item faces 3011A to 3014A includes one of the corners of the item region 30, and in the item region 30B, the item face 3012B includes a corner of the item region 30B.

[0149] In the article region 30B shown in the lower right of FIG. 15A, even when the proportion of the article surfaces 3011B to 3014B in the article region 30B decreases, the corner of one of the article surfaces (in this case, article surface 3012B) still includes the corner of the article region 30B.

[0150] Next, a process for identifying an article surface in an article region configured by an article in which a distinctive pattern is repeatedly present within the surface of a single article will be described with reference to FIGS. 15B and 15C.

[0151] 15B is composed of three items 311 with star patterns, two items 312 with lightning bolt patterns, and three items 313 with black circle patterns. Item area 32 shown in FIG. 15C is also composed of three items 321 with star patterns, two items 322 with lightning bolt patterns, and three items 323 with black circle patterns. Item areas 31 and 32 like these may appear, for example, when a depalletizing process is performed on a mixed pallet. Here, assume that the item recognition device 10 selects an area containing only star patterns as a template candidate because the star patterns have the greatest contrast.

[0152] In this case, the item recognition device 10 outputs information shown in the item region 31 on the right side of Fig. 15B. Specifically, information on an item surface 3111 including a corner of the item region 31, and item surfaces 3112 and 3113 similar to the item surface 3111, is output. Then, near the corner of the item region 31, a star-shaped pattern used to extract template candidates appears.

[0153] On the other hand, if the pattern with the highest contrast is not a star pattern but a thunderbolt pattern, the item recognition device 10 outputs information shown in the item region 32 on the right side of Fig. 15C. Specifically, information on an item surface 3211 including a corner of the item region 32 and an item surface 3212 similar to the item surface 3211 is output. Then, the thunderbolt pattern used to extract the template candidate appears near the corner of the item region 32.

[0154] However, in both the item region 31 shown in Fig. 15B and the item region 32 shown in Fig. 15C, the identified item surface differs from the surface of the actual item. In other words, when there are multiple characteristic patterns included on the surface of the item in the information input to the item recognition device 10 according to this embodiment, it is not possible to obtain appropriate item surface information that corresponds to the actual item.

[0155] [Application example of item recognition device] Next, an application example of the item recognition device 10 according to this embodiment will be described with reference to FIG. 16. FIG. 16 is a diagram showing an example of a work scene to which the item recognition device 10 is applied. In FIG. 16, four items 40 to be recognized are closely aligned on a workbench 41. A sensor 42 is configured, for example, with an RGB-D sensor or the like, and is installed above the workbench 41. The sensor 42 captures an image of the items 40 aligned on the workbench 41, and transmits a three-dimensional point cloud 43 of the alignment surface of the items 40 and a two-dimensional image 44 corresponding to the three-dimensional point cloud 43 to the item recognition device 10 (not shown in FIG. 16).

[0156] An alignment surface where items 40 are closely aligned may appear as a continuous area in the 3D point cloud 43, with no gaps between the items detected. On the other hand, in the 2D image 44, it may be difficult to distinguish between the straight lines formed by the boundaries of the items and the straight lines formed by the linear pattern. However, according to the item recognition device 10 of this embodiment, even when a 2D image corresponding to such a 3D point cloud is input, the position of each item that constitutes the alignment surface can be properly recognized.

[0157] Note that the object to be recognized by the object recognition device 10 according to this embodiment is not limited to an object having a flat surface facing the sensor 42. For example, any object may be used as long as it has a shape that allows the alignment surface in the two-dimensional image to be approximated by a polygon.

[0158] Fig. 17 is a diagram showing an example of a work scene in which an article according to a modified example of the present invention is cylindrical. In the example shown on the left side of Fig. 17, three cylindrical articles 50 are lined up closely together, and their top surfaces are photographed by the sensor 42. In this case, the image photographed by the sensor 42 is an image of an article area in which three rectangles are lined up, as shown on the right side of Fig. 17. In other words, the photographed image is one that can be recognized by the article recognition device 10 according to this embodiment.

[0159] Furthermore, when targeting such a three-dimensional article, the article recognition device 10 may output three-dimensional article surface information. Fig. 18 is a block diagram showing a schematic configuration example of an article recognition device 10A according to a modified example of the present invention.

[0160] As shown in FIG. 18, the item recognition device 10A includes a data receiving unit 15, a three-dimensional item area identifying unit 16, a two-dimensional item area identifying unit 17, an item area receiving unit 11, a recognition processing unit 12A, a template candidate information holding unit 12h, a three-dimensional item surface information acquisition unit 18, and a three-dimensional item information output unit 19.

[0161] The data receiving unit 15 receives a two-dimensional image corresponding to a three-dimensional point cloud obtained by photographing the alignment surface of the parts aligned in close contact with each other, and outputs the two-dimensional image to the three-dimensional article region identifying unit 16. The two-dimensional image is an image photographed by a sensor (not shown).

[0162] The three-dimensional item region identification unit 16 identifies an area (contiguous area) where pixels corresponding to the three-dimensional point cloud are continuous in the two-dimensional image input from the data receiving unit 15, and outputs this to the two-dimensional item region identification unit 17. The two-dimensional item region identification unit 17 approximates, by a polygon, the two-dimensional area in the two-dimensional image that corresponds to the continuous area identified by the three-dimensional item region identification unit 16, and identifies a two-dimensional pixel set surrounded by a polygonal frame. The two-dimensional item region identification unit 17 then outputs the identified two-dimensional pixel set to the item region receiving unit 11 as an item region.

[0163] The item region receiving unit 11 outputs the item region information input from the two-dimensional item region identifying unit 17 to the recognition processing unit 12. The recognition processing unit 12 is the same as the recognition processing unit 12 according to the above-described embodiment, and therefore a detailed description thereof will be omitted here.

[0164] The recognition processing unit 12 generates article surface information such as the position and size of the article surface in two-dimensional space, and outputs the article surface information to the three-dimensional article surface information acquisition unit 18. The three-dimensional article surface information acquisition unit 18 acquires information on the article surface in three-dimensional space corresponding to the two-dimensional article surface information, based on the two-dimensional article surface information input from the recognition processing unit 12 and a two-dimensional image corresponding to the three-dimensional point cloud received by the data receiving unit 15 from a sensor (not shown).

[0165] Specifically, the three-dimensional item surface information acquisition unit 18 calculates and acquires the position, size, and orientation of the item surface in three-dimensional space, which correspond to the position and size of the two-dimensional item surface. Then, the three-dimensional item surface information acquisition unit 18 outputs the acquired three-dimensional item surface information to an external device such as a picking robot via the three-dimensional item surface information output unit 19.

[0166] The template candidate information holding unit 12h is the same as the template candidate information holding unit 12h according to the embodiment described above, and therefore a description thereof will be omitted here.

[0167] FIG. 19 is a flowchart showing an example of the procedure of the item recognition process performed by the item recognition device 10A according to this modified example.

[0168] First, the three-dimensional article region identifying unit 16 (see FIG. 18) extracts a continuous region of pixels in the three-dimensional point cloud from the two-dimensional image input from the data receiving unit 15, and identifies it as a three-dimensional article region (step S21).

[0169] Next, the two-dimensional item region identification unit 17 determines whether or not there are any unselected three-dimensional item regions among the three-dimensional item regions identified in step S21 (step S22). If it is determined in step S22 that there are no unselected three-dimensional item regions (if step S22 returns NO), the two-dimensional item region identification unit 17 determines that information about the item to be recognized is not included in the data, and ends the process here.

[0170] On the other hand, if it is determined in step S22 that there is an unselected three-dimensional item region (if step S22 is determined to be YES), the two-dimensional item region identification unit 17 calculates a two-dimensional item region corresponding to the three-dimensional item region identified in step S16 (step S23). Specifically, the two-dimensional item region identification unit 17 extracts a region corresponding to the three-dimensional item region identified in step S16 from the two-dimensional image and approximates the contour of the region with a polygon, thereby identifying the region as a two-dimensional item region. Here, if the three-dimensional item region has a tilt, the two-dimensional item region identification unit 17 may perform processing to correct the tilt.

[0171] The process of step S23 can also be executed in parallel for two or more three-dimensional article regions, in which case a reduction in calculation time can be expected.

[0172] Next, the recognition processing unit 12 acquires article surface information such as the position or size of one or more article surfaces included in the two-dimensional article region identified in step S23 (step S24). Next, the three-dimensional article surface information acquisition unit 18 acquires three-dimensional article surface information corresponding to the two-dimensional article surface information acquired in step S24 (step S25). Specifically, the three-dimensional article surface information acquisition unit 18 acquires three-dimensional article surface information (position, size, orientation, etc. of the article surface) corresponding to the two-dimensional article surface information based on information on the position of the article surface indicated in the two-dimensional article surface information and information on the two-dimensional image corresponding to the three-dimensional point cloud input to the data receiving unit 15.

[0173] Information such as the position or orientation of the three-dimensional object surface obtained by the three-dimensional object surface information acquisition unit 18 is information necessary to identify the area of ​​the object surface in the three-dimensional point cloud. For example, if the object surface is configured on a plane, the position or orientation of the three-dimensional object surface corresponds to the three-dimensional position or orientation of the polygon corners that configure the object surface.

[0174] Next, the three-dimensional article surface information acquisition unit 18 determines whether the three-dimensional article surface information acquired in step S25 satisfies predetermined conditions (step S26). The predetermined conditions include conditions (1) to (4) described in the above-described embodiment. If it is determined in step S26 that the three-dimensional article surface information does not satisfy the predetermined conditions (if step S26 is judged as NO), the information on the article area that was the processing target is discarded, and the process returns to step S21.

[0175] On the other hand, if it is determined in step S26 that the three-dimensional item surface information satisfies the predetermined conditions (if step S26 is determined as YES), the three-dimensional item surface information acquisition unit 18 determines whether there is an unselected continuous area within the three-dimensional point cloud received by the data receiving unit 15 (see Figure 18) (step S27).

[0176] If it is determined in step S27 that there is an unselected continuous area (if step S27 is judged as YES), the three-dimensional article surface information acquisition unit 18 returns to step S21 to perform the process. On the other hand, if it is determined in step S27 that there is no unselected continuous area (if step S27 is judged as NO), the three-dimensional article surface information acquisition unit 18 outputs the three-dimensional article surface information that satisfies the conditions of step S26 to the outside via the three-dimensional article surface information output unit 19 (step S28).

[0177] Even if there are unselected continuous regions remaining in the three-dimensional point cloud, if there are constraints on the calculation time, the process may proceed to step S28 without returning to step S21.

[0178] If step S26 returns NO, or if step S27 returns YES after the processing of step S21, the image information stored in the template candidate information holding unit 12h can be reused in the processing performed by the two-dimensional object region identification unit 17 and subsequent units. For example, even if a collection of densely arranged objects exists in multiple captured images, the pattern information of the object surfaces obtained from one captured image can be used in the other captured images. By performing such processing, it is possible to improve the object recognition accuracy and reduce the calculation time.

[0179] Next, an example in which the item recognition device 10 according to this embodiment is applied to an apparatus that performs recognition processing on items to be picked by a picking robot will be described with reference to Fig. 20. Fig. 20 is a diagram showing an example of a work scene in which the item recognition device 10 is applied to an apparatus that performs recognition processing on items to be picked by a picking robot.

[0180] 20, the item recognition device 10 according to this embodiment is realized by a computer 200. The computer 200 has a mouse 204 as an input device 204 (see FIG. 2) and a display device 205 as an output device 205. A picking robot 300 and a sensor 42 are connected to a communication IF 206 (see FIG. 2) of the computer 200 via a communication line 2061.

[0181] The picking robot 300 receives the result of the item recognition by the computer 200 and performs a predetermined operation. The sensor 42 is placed on the work table 41 and captures an image of a work scene including the item 40 that is to be picked by the picking robot 300. Note that the sensor 42, the computer 200, etc. may be formed integrally with the picking robot 300.

[0182] The flow of operations of the picking robot 300 shown in FIG. 20 is, for example, as follows: First, the item recognition device 10 performs item recognition processing on a two-dimensional image corresponding to a three-dimensional point cloud acquired by the sensor 42, and obtains a recognition result. Next, the item recognition device 10 analyzes the recognition result using an appropriate program, and obtains information for inducing appropriate robot operation. Then, the item recognition device 10 transmits the information to the picking robot 300, causing the picking robot 300 to perform the operation.

[0183] The item recognition device 10 causes the picking robot 300 to perform operations such as grasping an item 40 and moving the grasped item 40 to a predetermined position. If the picking robot 300 performs operations based on the results of the item recognition process by the item recognition device 10 according to this embodiment, at least one of the items 40 to be grasped by the picking robot 300 will include a part having an item surface that includes a corner of the item area.

[0184] Therefore, by examining the characteristics of the item grasped by the picking robot 300, it is possible to determine whether the item recognition device that issues instructions to the picking robot is performing processing similar to the item recognition processing performed by the item recognition device 10 of this embodiment.

[0185] 20 shows an example in which the work scene is fixed, but the present invention is not limited to this. The work scene may be changed as appropriate, and the item recognition process by the item recognition device 10 according to this embodiment can be applied to a work scene that changes from moment to moment, such as an actual production line.

[0186] Furthermore, for example, the above-described embodiments and variant examples provide detailed and specific descriptions of the configuration of the device (item recognition device) in order to clearly explain the present invention, and are not necessarily limited to devices that include all of the configurations described.

[0187] In addition, the control lines or information lines indicated by solid arrows in Figure 1 are those considered necessary for explanation, and do not necessarily show all control lines or information lines in the product. In reality, it can be considered that almost all components are interconnected.

[0188] Furthermore, in this specification, processing steps describing chronological processing include not only processing that is performed chronologically in the order described, but also processing that is not necessarily performed chronologically but is performed in parallel or individually (for example, parallel processing or processing by objects).

[0189] Furthermore, the components of the item recognition device according to the embodiment of the present disclosure described above may be implemented in any hardware as long as the respective hardware can transmit and receive information to and from each other via a network. Furthermore, the processing performed by a certain processing unit may be realized by a single piece of hardware, or may be realized by distributed processing using multiple pieces of hardware. [Explanation of symbols]

[0190] 10...Item recognition device, 11...Item region receiving unit, 12...Recognition processing unit, 12a...Pattern amount calculation unit, 12b...Template candidate extraction unit, 12c...Template candidate evaluation unit, 12d...Template determination unit, 12e...Similar region detection unit, 12f...Similar region evaluation unit, 12g...Item surface information acquisition unit, 12h...Template candidate information storage unit, 13...Item surface information output unit

Claims

1. an input unit to which information on an article area obtained by polygonal approximation of an area in which pixels exist consecutively in a two-dimensional image including an image corresponding to an alignment surface of a plurality of articles arranged in close contact with each other is input; a template image generating unit that generates a template image using information on a pattern region in which a predetermined number of pixels or more are densely packed, the pattern amount of which is a predetermined first threshold value or more, and the region having the smallest distance from the closest angle to itself among the corners that make up the item region; an article surface information acquisition unit that acquires and outputs article surface information including the position and size of the surface of the article in the article region using information on one or more similar regions whose similarity to the template image is equal to or greater than a predetermined second threshold value. Article recognition device.

2. the object region includes a pattern that does not overlap with other types of patterns in the object region, and is configured by one or more types of polygons, each having a pixel value of a predetermined constant value in a region other than the pattern region, tiled on a two-dimensional plane; The size of the polygon in the article area is defined by the ratio of the lengths of two sides extending from a vertex of one of the corners of the polygon, The article surface information output from the article surface information acquisition unit includes information on the positions of corners of the one or more types of polygons that correspond to corners of the article region. The article recognition device according to claim 1 .

3. a similar region evaluation unit that calculates relative positional relationship information between the one or more similar regions and the closest angle to the region itself, The article surface information acquisition unit acquires the article surface information using the relative positional relationship information calculated by the similar area evaluation unit and information on one or more of the similar areas. The article recognition device according to claim 1 or 2.

4. The closest angle to be calculated as the distance to the object's own area or the relative positional relationship information is the angle whose difference from the smallest angle of the angles constituting the object area is less than a predetermined third threshold. The article recognition device according to claim 3 .

5. The template image generating unit excludes an area in which a linear pattern dividing the area exists from candidates for a template that is the basis of the template image. The article recognition device according to claim 1 or 2.

6. The template image generation unit reduces the region of the template candidate in a direction that forms the closest angle with the region of the template candidate, for each of the one or more template candidates. The article recognition device according to claim 5 .

7. The similar region evaluation unit expands each of the one or more similar regions in the direction of the closest angle to obtain expanded similar regions, and excludes a first expanded similar region that is the shortest distance from the closest angle to the expanded similar region itself and an expanded similar region whose similarity is less than a predetermined fourth threshold value from the similar regions. The article recognition device according to claim 3 .

8. an input unit receiving information on an object area obtained by polygonal approximation of an area in which pixels exist consecutively in a two-dimensional image including an image corresponding to an alignment surface of a plurality of objects arranged in close contact with each other, acquired by a sensor; a step in which a template image generating unit generates a template image using information on a pattern region in which a predetermined number of pixels or more are densely packed, the pattern amount of which is a predetermined first threshold or more, and the region having the shortest distance from the closest angle to the object region among the corners constituting the object region; an article surface information acquisition unit that acquires article surface information including the position and size of the surface of the article in the article region using information on one or more similar regions whose similarity to the template image is equal to or greater than a predetermined second threshold, and outputs the acquired article surface information to an output unit. Article recognition method.

Images