OBJECT RECOGNITION DEVICE AND OBJECT RECOGNITION METHOD

Abstract

To provide an object recognition device which can accurately recognize an object even when no reference object model exists.SOLUTION: An object recognition device comprises: an input unit to be used for inputting scene information indicating a state of a predetermined scene in a container that stores an object; a processing unit that, on the basis of the scene information input to the input unit, sorts objects in the container by size, and on the basis of the number of objects sorted by size, detects objects of sizes that are not allowed to be stored in the container; and an output unit that outputs processing results of the processing unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates generally to the recognition of objects present in a container. [Background technology]

[0002] Conventionally, in the field of logistics and the like, a robot that takes out an item from a storage box based on an order is provided in a warehouse. Since the same type of object is stored in the storage box in various packing ways, the robot needs to estimate the attitude and size of the item. If the attitude of the item is unknown, the robot cannot, for example, perform an operation of moving its hand close to the item to be grasped without colliding with it. Furthermore, if the size of the item is unknown, the robot cannot, for example, perform an operation of grasping the item and then taking out the item while avoiding obstacles such as other items.

[0003] In recent years, a method for estimating the posture and size of a product using a product model has been known. In such a method, since a product model is available, the size of the product is known and there is no need to estimate the size of the product. However, since a product model must be created for each type of product, the number of man-hours required for creating the product models increases as the number of types of products increases.

[0004] In this regard, a position and orientation recognition device has been disclosed that can recognize the position and orientation of randomly stacked workpieces with a small amount of calculation and can use a SCARA robot with little control information (see Patent Document 1). With this position and orientation recognition device, the workpiece to be grasped is limited to an approximately cylindrical shape, point cloud data is generated from a three-dimensional image of the workpieces in a randomly stacked state, and the size and position and orientation of the workpiece are calculated based on the point cloud data, so there is no need to prepare a model or the like in advance. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2020-34526 A Summary of the Invention [Problem to be solved by the invention]

[0006] In the technology described in Patent Document 1, based on information on the prototype shape, such as a roughly cylindrical workpiece, the prototype shape is fitted to input data to estimate the object's posture and size. However, even if part of the object is hidden, there are many cases where the prototype shape can be fitted with high accuracy, and since there is no reference object model, there is a problem in that it is not possible to determine whether the estimated object size is correct or not.

[0007] The present invention has been made in consideration of the above points, and aims to propose an object recognition device etc. that can recognize an object with high accuracy even if a reference model of the object does not exist. [Means for solving the problem]

[0008] In order to solve such problems, the present invention provides an input unit that inputs scene information indicating a specified scene state in a container for storing objects, a processing unit that sorts objects present in the container by size based on the scene information input by the input unit, and detects objects of a size that is not permissible for storage in the container based on the number of sorted objects of each size, and an output unit that outputs the results of processing by the processing unit.

[0009] In the above configuration, objects of a size that is not permissible for storage in a container are detected based on the number of objects of each size. Therefore, even if there is no reference model of the object, objects of a size that is not permissible for storage in the container can be excluded from the container. Effect of the Invention

[0010] According to the present invention, an object can be recognized with high accuracy even if a reference object model does not exist. [Brief description of the drawings]

[0011] [Figure 1] FIG. 2 is a diagram illustrating an example of a process performed by the object recognition system according to the first embodiment. [Diagram 2] FIG. 1 is a diagram illustrating an example of a configuration of an object recognition system according to a first embodiment. [Diagram 3] FIG. 1 is a diagram illustrating an example of a hardware configuration of an object recognition device according to a first embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of object recognition processing according to the first embodiment. [Diagram 5] FIG. 13 is a diagram showing a result image according to the first embodiment. [Figure 6] FIG. 11 is a diagram illustrating an example of a configuration of an object recognition device according to a second embodiment. [Figure 7] FIG. 11 is a diagram illustrating an example of object recognition processing according to a second embodiment. [Figure 8] FIG. 13 is a diagram showing an image of a result according to the second embodiment. [Figure 9] 13 is a diagram illustrating an example of a configuration of an object recognition device according to a third embodiment. [Figure 10] FIG. 13 is a diagram illustrating an example of object recognition processing according to a third embodiment. [Figure 11] FIG. 13 is a diagram illustrating an example of a process performed by an object recognition device according to a fourth embodiment. [Figure 12] FIG. 13 is a diagram illustrating an example of a process performed by an object recognition device according to a fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] (I) First embodiment An embodiment of the present invention will be described in detail below, however, the present invention is not limited to the embodiment.

[0013] The object recognition device according to the present embodiment receives scene information about a scene in which an object is present in a container. The scene information includes an estimation result of the size of the object. The object recognition device identifies the size of the object present in the scene based on the number of objects of each size.

[0014] According to the above configuration, it is possible to exclude estimation results of sizes that are not permissible for the scene from estimation results of the sizes of objects included in the scene information, and to select an estimation result of the correct size of the object.

[0015] The designations "first," "second," "third," and the like in this specification are given to identify components and do not necessarily limit the number or order. Furthermore, numbers for identifying components are used in each context, and a number used in one context does not necessarily indicate the same configuration in another context. Furthermore, a component identified by a certain number is not prevented from also having the function of a component identified by another number.

[0016] Next, an embodiment of the present invention will be described based on the drawings. The following description and drawings are examples for explaining the present invention, and are omitted and simplified as appropriate for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

[0017] In the following description, the same elements in the drawings are given the same numbers and descriptions thereof will be omitted as appropriate.

[0018] 1, reference numeral 100 generally denotes an object recognition system according to a first embodiment.

[0019] The object recognition system 100 performs an input process 110, a selection process 120, and an output process 130.

[0020] In the input process 110, the object recognition system 100 inputs data related to a scene (target scene) for which object recognition is to be performed as scene information. The scene information includes an estimation result 111. The estimation result 111 includes information indicating the result of estimating the size of an object included in the target scene. Note that, although the estimation of the size of an object included in the target scene will be described using an example in which the estimation is performed in a robot 210 described later, the present invention is not limited to this. For example, the estimation of the size of an object included in the target scene may be performed in the object recognition system 100, or may be performed in a system, computer, etc. different from the object recognition system 100.

[0021] In the selection process 120, the object recognition system 100 uses the estimation result 111 for the target scene (entire scene) to select likely objects in the target scene (objects of a size acceptable in the target scene) to generate a selection result 121. The selection result 121 includes information 122 indicating objects of a size acceptable in the target scene. For example, the object recognition system 100 selects objects of a size acceptable in the target scene by voting for the sizes of a plurality of objects.

[0022] In the output process 130, the object recognition system 100 generates and outputs the recognition result 131 based on the selection result 121. The recognition result 131 includes information indicating an object of a size acceptable for the target scene, information indicating an object of a size unacceptable for the target scene, information indicating the orientation of an object of a size unacceptable for the target scene, information indicating an evaluation of the estimation result 111 (for example, information indicating whether or not an object included in the target scene is an object of a size acceptable for the target scene), and other information.

[0023] FIG. 2 is a diagram illustrating an example of the configuration of the object recognition system 100. As shown in FIG.

[0024] The object recognition system 100 includes a robot 210 and an object recognition device 220. The robot 210 and the object recognition device 220 are communicatively connected. The robot 210 and the object recognition device 220 may be connected via a network or may be connected without a network.

[0025] The robot 210 estimates the size and orientation of an object in the storage box 211 in response to an order (order information), includes the estimation result 111 in scene information of a target scene, and transmits the estimation result 111 to the object recognition device 220. Upon receiving the scene information, the object recognition device 220 generates a selection result 121 using the received scene information, generates a recognition result 131 based on the selection result 121, and transmits the generated recognition result 131 to the robot 210. Based on the recognition result 131, the robot 210 takes out an object of a size allowable in the target scene from the storage box 211, and moves the object to a specified location 212.

[0026] In this embodiment, the storage box 211 basically stores objects of one size (first size). However, when storing objects in the storage box 211, a situation may occur in which an object of an incorrect size (a second size different from the first size) is mixed in due to human error or the like. Even if such a situation occurs, the object recognition device 220 selects objects of a size that is not permitted in the target scene, so that the robot 210 can take out (pick) an object of an ordered size (an object of a size permitted in the target scene) from the storage box 211.

[0027] More specifically, the object recognition device 220 includes an input unit 221 , a processing unit 222 , and an output unit 223 , and recognizes objects stored in the storage box 211 .

[0028] The input unit 221 receives scene information transmitted from the robot 210 and inputs the received scene information. The processing unit 222 performs processing such as selecting an object based on the scene information input by the input unit 221. The output unit 223 outputs the result of the object selection by the processing unit 222.

[0029] FIG. 3 is a diagram showing an example of a hardware configuration of the object recognition device 220. As shown in FIG.

[0030] The object recognition device 220 includes a processor 301 , a main memory device 302 , an auxiliary memory device 303 , an input device 304 , an output device 305 , and a communication device 306 .

[0031] The processor 301 is a device that performs arithmetic processing. The processor 301 is, for example, a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), an artificial intelligence (AI) chip, or the like.

[0032] The main memory device 302 is a device that stores programs, data, etc. The main memory device 302 is, for example, a read only memory (ROM), a random access memory (RAM), etc. The ROM is, for example, a static random access memory (SRAM), a non-volatile RAM (NVRAM), a mask read only memory (mask ROM), a programmable ROM (PROM), etc. The RAM is, for example, a dynamic random access memory (DRAM), etc.

[0033] The auxiliary storage device 303 is a hard disk drive, a flash memory, a solid state drive (SSD), an optical storage device, etc. The optical storage device is a compact disc (CD), a digital versatile disc (DVD), etc. The programs, data, etc. stored in the auxiliary storage device 303 are loaded into the main storage device 302 as needed.

[0034] The input device 304 is a user interface that accepts information from a user, and is, for example, a keyboard, a mouse, a card reader, a touch panel, or the like.

[0035] The output device 305 is a user interface that outputs various types of information (display output, audio output, print output, etc.). The output device 305 is, for example, a display device that visualizes various types of information, an audio output device (speaker), a printer, etc. The display device is an LCD (Liquid Crystal Display), a graphics card, etc.

[0036] The communication device 306 is a communication interface that communicates with other devices via a communication medium. The communication device 306 is, for example, a network interface card (NIC), a wireless communication module, a universal serial interface (USB) module, a serial communication module, etc. The communication medium is, for example, a communication medium conforming to various communication standards such as a universal serial bus (USB) or RS-232C, a local area network (LAN), a wide area network (WAN), the Internet, a dedicated line, etc. The configuration of the communication medium is not necessarily limited. The communication device 306 can also function as an input device that receives information from other devices that are communicatively connected. The communication device 306 can also function as an output device that transmits information to other devices that are communicatively connected.

[0037] The functions of the object recognition device 220 (the input unit 221, the processing unit 222, the output unit 223, etc.) may be realized, for example, by the processor 301 reading a program stored in the auxiliary storage device 303 into the main storage device 302 and executing it (software), or may be realized by hardware such as a dedicated circuit, or may be realized by a combination of software and hardware. Note that one function of the object recognition device 220 may be divided into multiple functions, or multiple functions may be integrated into one function. Also, a part of the function of the object recognition device 220 may be provided as a separate function, or may be included in another function. Also, a part of the function of the object recognition device 220 may be realized by another computer capable of communicating with the object recognition device 220. For example, a part or all of the function of the object recognition device 220 may be realized in the robot 210, or in cloud computing.

[0038] FIG. 4 is a diagram showing an example of the object recognition process (the input process 110, the selection process 120, and the output process 130) performed by the object recognition device 220. As shown in FIG.

[0039] In step S401, scene information is input to the object recognition device 220. The scene information includes the estimation result 111.

[0040] In step S402, the object recognition device 220 creates a histogram based on the scene information. In the histogram, the horizontal axis represents classes and the vertical axis represents frequency, with the frequency of each class being represented by a bin (e.g., a rectangular column). A label value indicating the class is provided. The classes may be represented by a unique value or may be represented by a range (predetermined interval). For example, in the case of a predetermined class in which the vertical range of the class is "10.00 cm to 10.99 cm" and the horizontal range of the class is "8.00 cm to 8.99 cm", "10 cm x 8 cm" is provided as the label value.

[0041] Here, for example, if the scene information includes data on a specific object that is estimated to have a vertical length of "10.01 cm" and a horizontal length of "8.16 cm", the specific object belongs to the above-mentioned predetermined class, so the object recognition device 220 adds "1" to the frequency of the predetermined class.

[0042] The object recognition device 220 may include an interface that allows a user to set a predetermined interval. Such an interface allows the user to set, for example, a hyperparameter (allowable error).

[0043] In step S403, the object recognition device 220 selects an object based on the histogram created in step S402. For example, the object recognition device 220 takes a majority vote of the classes and estimates the label value of the most frequent class as the size of the object permitted in the target scene. For example, if the most frequent class is the above-mentioned predetermined class, the object recognition device 220 estimates "10 cm x 8 cm", which is assigned as the label value of the predetermined class, as the size of the object.

[0044] In step S404, the object recognition device 220 generates and outputs the recognition result 131 based on the result selected in step S403 (selection result 121). Note that the object recognition device 220 may transmit the recognition result 131 to the robot 210, may display the recognition result 131 on the output device 305, or may transmit the recognition result 131 to another computer.

[0045] FIG. 5 is a diagram showing an example (image of result) of the processing performed in the object recognition processing.

[0046] Result image 510 shows an example of the estimation result 111. As shown in result image 510, estimation result 111 includes, for each object included in the target scene, an ID for identifying the object and a value indicating the size of the object. Note that result image 510 shows an example in which the object is box-shaped, and the parameters indicating the size of the object are length and width. However, the size of the object may include a combination other than length and width, for example, a parameter of height.

[0047] Result image 520 shows an example of sorting result 121. As shown in result image 520, sorting result 121 includes information showing the result of voting by exact numerical match, the result of voting by histogram in predetermined increments (e.g., 1 cm increments), etc.

[0048] Result image 530 shows an example of the recognition result 131. As shown in result image 530, the recognition result 131 includes information that can be considered as the size of an object selected as an object of an allowable size in the target scene (e.g., an object ID).

[0049] According to this embodiment, it is possible to exclude the estimation results of objects whose sizes are unacceptable in the target scene from the estimation results included in the scene information, and to correctly select the estimation results of objects whose sizes are acceptable in the target scene.

[0050] (II) Second embodiment The present embodiment differs from the first embodiment mainly in that an estimation result 111 is generated by an object recognition device 600. In the present embodiment, the same components as those in the first embodiment are denoted by the same reference numerals and will not be described.

[0051] FIG. 6 is a diagram showing an example of the configuration of an object recognition device 600 according to this embodiment.

[0052] The object recognition device 600 is connected to one or more sensors 601 that acquire the state (physical phenomenon, chemical phenomenon, etc.) of the storage box 211. The object recognition device 600 includes an input unit 611, a processing unit 222, and an output unit 223. The processing unit 222 includes a size estimation unit 612 and a posture estimation unit 613.

[0053] The input unit 611 inputs sensor information acquired by the sensor 601. The sensor information is an example of scene information indicating a target scene, and is a color image, a point cloud, a depth image, a grayscale image, a combination of these, etc. The size estimation unit 612 estimates the size of an object in the target scene. The pose estimation unit 613 estimates the pose of an object in the target scene.

[0054] In the following description, it is assumed that the shape of an object is a prototype shape, and that objects are stored in storage box 211 without being aligned. The prototype shape is a combination of parameters such as length, width, and depth that indicate the shape of an object. In other words, when the shape of an object is the prototype shape, it can be said that the size of the object is a combination of parameter values ​​that express the prototype shape.

[0055] Here, the object recognition device 600 has registered therein prototype shape information 614 including information indicating parameters expressing the prototype shape of the object. The parameters expressing the prototype shape of the object are length, width, and height, or length and width, if the prototype shape is box-shaped, radius and height, radius, or height, if the prototype shape is cylindrical, and radius, if the prototype shape is spherical. For example, if the prototype shape is box-shaped, the object recognition device 600 can estimate box-shaped objects of various sizes. Note that the prototype shape is not limited to the above-mentioned example, and may be other prototype shapes such as a triangular prism shape or a combination of the above-mentioned prototype shapes.

[0056] Furthermore, the configuration of the object recognition device 600 is not limited to the above-mentioned configuration. For example, the object recognition device 600 does not need to be directly connected to the sensor 601, and may be indirectly connected to a device connected to the sensor 601. For example, the object recognition device 600 does not need to include the attitude estimation unit 613.

[0057] FIG. 7 is a diagram showing an example of the object recognition process performed by the object recognition device 600. As shown in FIG.

[0058] In step S701, the object recognition device 600 inputs scene information from the sensor 601. The scene information may be transmitted from the sensor 601 at a first timing, or may be requested and acquired by the object recognition device 600 at a second timing. The first timing may be always, may be the timing when an order is accepted, may be periodically, may be a time designated in advance, or may be a timing designated by a user. The second timing may be the same timing as the first timing, or may be a timing different from the first timing.

[0059] In step S702, the object recognition device 600 performs area extraction based on the scene information input in step S701, and cuts out an object point group estimated to constitute an object. For example, the object recognition device 600 divides the area for each object by instant segmentation, and recognizes the type of the object. For example, the object recognition device 600 classifies the pixel of the image as to which of a plurality of instances the pixel belongs to (cuts out an object point group having (x, y, z) coordinates distributed in a three-dimensional space). The object recognition device 600 may perform segmentation using other methods such as watershed. In addition, the area extraction is not limited to instant segmentation, and an object detection method such as BING (Binarized Normed Gradients) may be used.

[0060] In step S703, the object recognition device 600 performs object surface estimation. For example, the object recognition device 600 reads the original shape information 614, and uses the original shape of the object to find at least one surface that best fits the object point group extracted in step S702 using a method such as RANSAC (Random Sample Consensus), checks whether the found surface satisfies a rule that should be satisfied for each original shape of the object, and extracts the point group that constitutes the object. The rule that should be satisfied for each original shape of the object is, for example, a rule that can be satisfied if the original shape is such that the found surfaces are mutually orthogonal if the original shape of the object is a box shape.

[0061] In step S704, the object recognition device 600 performs size estimation. The object recognition device 600 estimates the size of the object from the point cloud constituting the object estimated in step S703. For example, when the original shape of the object is box-shaped, if the point cloud constituting the object has one surface, the minor axis and major axis of the point cloud are the size of the object. Also, if the point cloud constituting the object has two or three surfaces, the length of each axis of the three-dimensional rectangle with the smallest volume among the three-dimensional rectangles covering the point cloud is the size of the object.

[0062] In step S705, the object recognition device 600 performs pose estimation. For example, the object recognition device 600 creates a point cloud (prototype model) of the size of the object estimated in step S703, and estimates the pose of the object (for example, estimates a rotation matrix and a translation matrix) using an algorithm (for example, ICP (Iterative Closest Point)) that minimizes the distance between the point cloud constituting the object cut out in step S702 and the model of the prototype shape.

[0063] Steps S402 to S404 have been described in the first embodiment, so the description thereof will be omitted.

[0064] FIG. 8 is a diagram showing an example (image of result) of the processing result performed in the object recognition processing.

[0065] A result image 810 shows an example of the result (scene information) of inputting the sensor information acquired by the sensor 601 in step S701. Here, it is shown that a color image and a point cloud are used as the sensor information included in the scene information.

[0066] Result image 820 shows an example of the result (estimation result) of estimating the size of an object in step S704. As shown in result image 820, the estimation result includes, for each object included in the target scene, an ID for identifying the object and a value indicating the size of the object. Note that result image 820 shows an example in which the object is box-shaped, and the parameters indicating the size of the object are length and width. However, the size of the object may include a combination other than length and width, for example, a parameter of height. The estimation result may also include the result of estimating the orientation of the object in step S705.

[0067] According to this embodiment, it is possible to estimate the size of an object based on sensor information of a target scene, and to detect an object of an unacceptable size in the target scene.

[0068] (III) Third embodiment The present embodiment differs from the second embodiment mainly in that the size and orientation of an object are simultaneously estimated. In the present embodiment, the same components as those in the second embodiment are denoted by the same reference numerals and will not be described.

[0069] FIG. 9 shows an example of the configuration of an object recognition device 900 according to this embodiment.

[0070] The object recognition device 900 includes an input unit 611, a processing unit 222, and an output unit 223. The processing unit 222 includes a size and orientation estimation unit 911.

[0071] The size and orientation estimation unit 911 estimates the size and orientation of an object based on the scene information input at the input unit 611 .

[0072] FIG. 10 is a diagram showing an example of the object recognition process performed by the object recognition device 900. As shown in FIG.

[0073] In step S1001, the object recognition device 900 performs fitting. More specifically, the object recognition device 900 calculates the error between the point cloud constituting the object cut out in step S702 and an arbitrarily created model of the original shape (for example, a box point cloud). For example, the object recognition device 900 estimates the posture of the object using an algorithm (for example, ICP) that minimizes the distance between point clouds. In this case, the object recognition device 900 calculates the distance between a certain point of the original shape model and the closest point of the object point cloud by decomposing it into the x-axis direction, y-axis direction, and z-axis direction, and sets the average of the distances for each axis calculated for all points as the error for each axis.

[0074] In step S1002, the object recognition device 900 determines whether or not a termination condition is satisfied. If the object recognition device 900 determines that the termination condition is satisfied, the process proceeds to step S402, and if the object recognition device 900 determines that the termination condition is not satisfied, the process proceeds to step S1003. The termination condition is, for example, that all errors for each axis calculated in step S1002 are equal to or smaller than a threshold value, that the process of step S1001 has been performed a predetermined number of times, etc.

[0075] In step S1003, the object recognition device 900 adjusts the parameters and moves the process to step S1001. For example, the object recognition device 900 changes (for example, halves or doubles) the parameter value of the model of the original shape in the direction with the largest error among the x-axis direction, the y-axis direction, and the z-axis direction.

[0076] In this way, the object recognition device 900 estimates the pose and size of an object by repeating fitting and parameter adjustment.

[0077] According to this embodiment, it is possible to simultaneously estimate the size and orientation of an object, and detect an object of an unacceptable size in a target scene.

[0078] (IV) Fourth embodiment The present embodiment differs from the second embodiment mainly in that the size of an object is estimated and selected, and if an object (referred to as a "foreign object") with a size that is not permitted in the target scene is detected, the foreign object is presented to the user. In the present embodiment, the same components as those in the second embodiment are designated by the same reference numerals and their description is omitted.

[0079] FIG. 11 is a diagram showing an example of processing performed by the object recognition device 1100 of this embodiment.

[0080] The object recognition device 1100 performs an input process 1110 , an estimation process 1120 , a selection process 1130 , a foreign object detection process 1140 , and an output process 1150 .

[0081] The object recognition device 1100 inputs scene information in an input process 1110. The scene information includes sensor information 1111. The sensor information 1111 includes a color image, a point cloud, and the like acquired by the sensor 601.

[0082] In estimation processing 1120, the object recognition device 1100 estimates the size of an object included in a target scene and generates an estimation result 1121. At this time, the object recognition device 1100 may estimate the orientation of the object.

[0083] In the selection process 1130, the object recognition device 1100 uses the estimation result 1121 for the target scene (entire scene) to select objects of a size acceptable for the target scene and generate a selection result 1131. The selection result 1131 includes information 1132 indicating objects of a size acceptable for the target scene and information 1133 indicating foreign objects. For example, the object recognition device 1100 selects objects acceptable for the target scene and foreign objects by voting for the sizes of a plurality of objects.

[0084] In the foreign object detection process 1140, when a foreign object is detected in the selection process 1130, the object recognition device 1100 identifies the foreign object from among the objects included in the sensor information 1111, and generates foreign object information 1141 as information indicating the identified foreign object. Note that the foreign object information 1141 may be an ID or the like capable of identifying an object of an unacceptable size in the target scene.

[0085] In the output process 1150, the object recognition device 1100 outputs a recognition result 1151 including foreign object information 1141. The recognition result 1151 is displayed on the output device 305. The recognition result 1151 may be displayed on a tablet terminal carried by the user, a display provided in the warehouse, or the like. According to such a display, the user can recognize the foreign object in the storage box 211 and remove the foreign object from the storage box 211.

[0086] In addition to or instead of presenting the recognition result 1151 to the user, the object recognition device 1100 may present the recognition result 1151 to the robot 210, and the robot 210 may grasp and remove the foreign object to maintain the consistency of the objects in the storage box 211.

[0087] In this embodiment, the presence or absence of an object of a size that does not actually exist in the storage box is determined, and the determination result is output. According to this embodiment, for example, even if a wrong object is placed in the storage box due to human error or the like when storing an object in the storage box, the object can be removed from the storage box.

[0088] (V) Fifth embodiment The present embodiment differs from the second embodiment mainly in that the types of objects stored in the storage box are multiple. In the present embodiment, the same components as those in the second embodiment are designated by the same reference numerals and will not be described.

[0089] FIG. 12 is a diagram showing an example of processing performed by the object recognition device 1200 of this embodiment.

[0090] The object recognition device 1200 performs an input process 1210 , an estimation process 1220 , a selection process 1230 , and an output process 1240 .

[0091] In input processing 1210, the object recognition device 1200 inputs scene information. The scene information includes sensor information 1211 and presence information 1212. The sensor information 1211 includes a color image, a point cloud, etc. acquired by the sensor 601. The presence information 1212 includes information indicating the number of objects of the same type present in the target scene (in this example, "2").

[0092] In estimation processing 1220, the object recognition device 1200 estimates the size of an object included in a target scene and generates an estimation result 1221. At this time, the object recognition device 1200 may estimate the orientation of the object.

[0093] In a selection process 1230, the object recognition device 1200 uses an estimation result 1221 for a target scene (entire scene) and presence information 1212 to select objects of an allowable size in the target scene, and generates a selection result 1231.

[0094] For example, when the presence information 1212 is "2", the object recognition device 1200 votes for the sizes of a plurality of objects, and determines the top two objects in size as objects of sizes permitted in the target scene (object of a first size and object of a second size), and classifies the remaining objects as foreign objects. Note that the object recognition device 1200 may classify an object of one of the top two sizes as a foreign object if the number of objects of that size is less than a threshold value (e.g., 3).

[0095] In this example, the selection result 1231 includes information 1232 and 1233 indicating objects that are permissible in the target scene. Additionally, the selection result 1231 may also include information indicating a foreign object.

[0096] In the output process 1240, the object recognition device 1200 generates and outputs a recognition result 1241 based on the selection result 1231. The recognition result 1241 is displayed on the output device 305. The recognition result 1241 may be displayed on a tablet terminal carried by the user, a display provided in the warehouse, or the like. According to such a display, the user can recognize the foreign object in the storage box 211 and remove the foreign object from the storage box 211.

[0097] In this embodiment, multiple sizes are allowed by additionally inputting the number of sizes of objects in the storage box and an estimation result is selected, so that even if two or more types of objects are stored in the storage box, objects of sizes that are not allowed in the storage box can be detected.

[0098] (VI) Notes The above-described embodiment includes, for example, the following contents.

[0099] In the above-described embodiment, the present invention is described as being applied to an object recognition device, but the present invention is not limited to this and can be widely applied to various other systems, devices, robots, methods, and programs.

[0100] In the above-described embodiment, a part or all of the programs may be installed from a program source into a device such as a computer that realizes the object recognition device. The program source may be, for example, a program distribution server connected via a network or a computer-readable recording medium (for example, a non-transitory recording medium). In the above description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0101] Furthermore, the screens shown and described in the above embodiment are merely examples, and any design may be used as long as the information received is the same.

[0102] Furthermore, the screens shown and described in the above embodiment are merely examples, and any design may be used as long as the information presented is the same.

[0103] In addition, in the above-mentioned embodiment, the average value is used as the statistical value. However, the statistical value is not limited to the average value, and may be other statistical values ​​such as the maximum value, the minimum value, the difference between the maximum value and the minimum value, the mode, the median, the standard deviation, etc.

[0104] In the above-described embodiment, the output of information is not limited to display on a display. The output of information may be audio output from a speaker, output to a file, printing on a paper medium or the like by a printing device, projection on a screen or the like by a projector, or other form.

[0105] In addition, in the above description, information such as programs, tables, files, etc. that realize each function can be placed in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC card, an SD card, or a DVD.

[0106] The above-described embodiment has the following characteristic configurations, for example.

[0107] (1) An object recognition device (e.g., object recognition device 220, object recognition device 600, object recognition device 900, object recognition device 1100, object recognition device 1200) includes an input unit (e.g., input unit 221, input unit 611, circuit, object recognition device, computer) that inputs scene information (object estimation result 111, sensor information 1111, sensor information 1211, color image, point cloud, etc.) indicating a predetermined scene state in a container (e.g., storage box 211) for storing objects, a processing unit (e.g., processing unit 222, circuit, object recognition device, computer) that performs a process of sorting objects present in the container by size based on the scene information input by the input unit and detecting objects of a size that is not permitted for storage in the container based on the number of sorted objects by size, and an output unit (e.g., output unit 223, circuit, object recognition device, computer) that outputs a result of processing by the processing unit.

[0108] In the above configuration, size information indicating the size of an object included in a specified scene may be size information estimated by another computer included in the scene information, or may be estimated from sensor information included in the scene information.

[0109] In the above configuration, since objects of a size that is not permitted for storage in a container are detected based on the number of objects of each size, for example, even if there is no reference object model, objects of a size that is not permitted for storage in the container can be removed from the container. According to the above configuration, objects of a size that is not permitted for storage in a container can be detected without creating an object model in which the size, color, texture, etc. of the object are set for each object.

[0110] (2) The scene information input by the input unit includes at least one of a color image, a point cloud, a depth image, and a grayscale image, and the processing unit estimates the size of objects present in the container based on the scene information input by the input unit, and sorts the objects present in the container by size based on the estimated object sizes (see Figures 7, 10, etc.).

[0111] In the above configuration, the size of an object is estimated from at least one of information from a color image, a point cloud, a depth image, and a grayscale image, and objects present in the container are sorted by size. Therefore, for example, by inputting the information into an object recognition device, objects present in the container can be sorted by size.

[0112] (3) The size of an object present in the container is a combination of parameter values ​​that represent the original shape of the object, and the processing unit selects the objects present in the container for each combination of parameter values ​​that represent the original shape based on the scene information input by the input unit (see Figures 5, 8, etc.).

[0113] In the above configuration, objects are selected using parameter values ​​that represent the prototype shape, making it easier to compare the sizes of objects and to select objects more easily than, for example, when objects are selected using the object's existence area (the area that the object occupies in space).

[0114] (4) The parameters representing the prototype shape are length, width, and height, or length and width, if the prototype shape is box-shaped, and are radius and height, or radius, or height, if the prototype shape is cylindrical.

[0115] (5) The processing unit creates a histogram showing the frequency distribution of objects present in the container according to a predetermined range that is set for each parameter expressing the prototype shape and that is determined to be the same parameter value, and detects objects of a size that is not permissible for storage in the container based on the number of elements in each bin of the created histogram (see Figures 4, 7, 10, etc.).

[0116] In the above configuration, objects having a combination of parameter values ​​that fall within a predetermined range are determined to be objects of the same size. Therefore, for example, by providing an interface for setting the predetermined range, the user can set hyperparameters (tolerable errors).

[0117] (6) The scene information input by the input unit includes sensor information (color image, point cloud, depth image, grayscale image, combinations of these, etc.) of the state of the container acquired by a sensor, and the processing unit estimates the size of objects present in the container based on the sensor information included in the scene information input by the input unit, and sorts the objects present in the container by size based on the estimated object sizes (see Figures 7, 10, etc.).

[0118] In the above configuration, the size of the object is estimated from the sensor information and the objects present in the container are sorted by size. Therefore, for example, by connecting the sensor directly or indirectly to an object recognition device, the objects present in the container can be sorted by size.

[0119] (7) The size of an object present in the container is a combination of parameter values ​​representing the original shape of the object, and the processing unit estimates the parameter values ​​representing the original shape as the size of the object present in the container based on sensor information included in the scene information input by the input unit (see Figures 7, 10, etc.).

[0120] In the above configuration, the size of an object is estimated using the values ​​of parameters that represent the prototype shape, making it easier to compare the sizes of objects and to easily select objects compared to, for example, when the size of an object is estimated using the area in which the object exists.

[0121] (8) The processing unit estimates the orientation of the object using the estimated size of the object, and the output unit outputs the size and orientation of the object estimated by the processing unit (see FIG. 7, etc.).

[0122] In the above configuration, the size and posture of an object are output, so that, for example, a robot can approach an object whose size does not allow for storage in a container without colliding with its own hand, grasp the object, and then remove the object from the container while avoiding obstacles such as other objects.

[0123] (9) The processing unit estimates the orientation of the object by matching the estimated size of the object with the model of the original shape (see FIG. 7, etc.).

[0124] In the above configuration, the posture of an object is estimated by aligning the estimated size of the object with a model of the original shape. Therefore, even if there is no reference model of the object, it is possible to estimate the posture of the object by creating a model of the original shape.

[0125] (10) The size of an object present in the container is a combination of parameter values ​​expressing the original shape of the object, and the scene information input by the input unit includes sensor information in which the state of the container is acquired by a sensor. The processing unit repeatedly fits an object point cloud extracted from the sensor information included in the scene information input by the input unit to a model of the original shape, and adjusts the parameter values ​​of the model of the original shape, to estimate the size and orientation of the object (see Figure 10, etc.).

[0126] According to the above configuration, for example, even if there is no reference object model, it is possible to estimate the size and orientation of an object by creating a model of the original shape.

[0127] (11) The output unit outputs information indicating the object detected by the processing unit that is of a size that is not permissible for storage in the container (see FIG. 11, etc.).

[0128] In the above configuration, information indicating objects of a size that is not permissible for storage in the container is output, so that, for example, a user, a robot, etc. can remove objects of a size that is not permissible for storage in the container from the container.

[0129] (12) The input unit inputs presence information (e.g., presence information 1212) indicating the number of types of objects present in the container, and the processing unit detects objects of a size that is not permissible for storage in the container based on the presence information (see Figure 12, etc.).

[0130] In the above configuration, objects of a size that is not permitted for storage in a container are detected based on the presence information, so that even if objects of multiple sizes are stored in a container, objects of a size that is not permitted for storage in the container can be detected.

[0131] Furthermore, the above-described configurations may be modified, rearranged, combined, or omitted as appropriate without departing from the spirit and scope of the present invention.

[0132] It should be understood that items listed in the format "at least one of A, B, and C" can mean (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). Similarly, items listed in the format "at least one of A, B, or C" can mean (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). [Explanation of symbols]

[0133] 220: object recognition device, 221: input section, 222: processing section, 223: output section.

Claims

1. An input unit for inputting scene information indicating a predetermined scene state in a container for storing an object; a processing unit that performs a process of sorting objects present in the container by size based on the scene information input by the input unit, and detecting objects of a size that cannot be stored in the container based on the number of sorted objects by size; an output unit that outputs a result of the processing by the processing unit; Equipped with the size of an object present in the container is a combination of parameter values ​​expressing a prototype shape of the object; The processing unit selects objects present in the container for each combination of parameter values ​​expressing the prototype shape based on the scene information input by the input unit. Object recognition device.

2. The scene information input by the input unit includes at least one of a color image, a point cloud, a depth image, and a grayscale image; The processing unit estimates a size of an object present in the container based on the scene information input by the input unit, and sorts the objects present in the container by size based on the estimated object sizes. The object recognition device according to claim 1 .

3. The parameters expressing the prototype shape are length, width, and height, or length and width, when the prototype shape is a box shape, and are radius and height, or radius, or height, when the prototype shape is a cylinder shape. The object recognition device according to claim 1 .

4. the processing unit creates a histogram showing a frequency distribution of objects present in the container according to a predetermined range that is set for each parameter expressing the prototype shape and that is determined to be the same parameter value, and detects objects of a size that is not permissible for storage in the container based on the number of elements in each bin of the created histogram. The object recognition device according to claim 1 .

5. The output unit outputs information indicating an object detected by the processing unit that is a size that is not permitted for storage in the container. The object recognition device according to claim 1 .

6. An input unit for inputting scene information indicating a state of a predetermined scene in a container for storing an object; a processing unit that performs a process of sorting objects present in the container by size based on the scene information input by the input unit, and detecting objects of a size that cannot be stored in the container based on the number of sorted objects by size; an output unit that outputs a result of the processing by the processing unit; Equipped with The scene information input by the input unit includes sensor information obtained by a sensor regarding a state of the container, The processing unit estimates a size of an object present in the container based on sensor information included in the scene information input by the input unit, and sorts the objects present in the container by size based on the estimated object sizes; the size of an object present in the container is a combination of parameter values ​​expressing an original shape of the object; The processing unit estimates a value of a parameter expressing the prototype shape as a size of the object present in the container based on sensor information included in the scene information input by the input unit. Object recognition device.

7. The processing unit estimates a posture of the object using the estimated size of the object; The output unit outputs the size and orientation of the object estimated by the processing unit. The object recognition device according to claim 6.

8. The processing unit estimates a posture of the object by aligning the estimated size of the object with a model of the original shape. The object recognition device according to claim 7.

9. An input unit for inputting scene information indicating a state of a predetermined scene in a container for storing an object; a processing unit that performs a process of sorting objects present in the container by size based on the scene information input by the input unit, and detecting objects of a size that cannot be stored in the container based on the number of sorted objects by size; an output unit that outputs a result of the processing by the processing unit; Equipped with the size of an object present in the container is a combination of parameter values ​​expressing a prototype shape of the object; The scene information input by the input unit includes sensor information obtained by a sensor regarding a state of the container, the processing unit repeats fitting an object point cloud obtained by extracting an object from sensor information included in scene information input by the input unit to a model of the original shape and adjusting parameter values ​​of the model of the original shape to estimate a size and a posture of the object. Object recognition device.

10. An input unit for inputting scene information indicating a predetermined scene state in a container for storing an object; a processing unit that performs a process of sorting objects present in the container by size based on the scene information input by the input unit, and detecting objects of a size that cannot be stored in the container based on the number of sorted objects by size; an output unit that outputs a result of the processing by the processing unit; Equipped with the input unit inputs presence information indicating the number of types of objects present in the container; The processing unit detects an object having a size that is not permitted for storage in the container based on the presence information. Object recognition device.

11. An input unit inputs scene information indicating a predetermined scene state in a container that stores an object; a processing unit, based on the scene information input by the input unit, sorting objects present in the container by size, and based on the number of sorted objects by size, detecting objects of a size that cannot be stored in the container; an output unit outputs a result of the processing by the processing unit; Including, the size of an object present in the container is a combination of parameter values ​​expressing a prototype shape of the object; the processing unit sorts out the objects present in the container for each combination of parameter values ​​expressing the prototype shape based on the scene information input by the input unit; Object recognition method.

12. An input unit inputs scene information indicating a state of a predetermined scene in a container for storing an object; a processing unit, based on the scene information input by the input unit, sorting objects present in the container by size, and based on the number of sorted objects by size, detecting objects of a size that cannot be stored in the container; an output unit outputs a result of the processing by the processing unit; Including, The scene information input by the input unit includes sensor information obtained by a sensor regarding a state of the container, The processing unit estimates a size of an object present in the container based on sensor information included in the scene information input by the input unit, and sorts the objects present in the container by size based on the estimated object sizes; the size of an object present in the container is a combination of parameter values ​​expressing a prototype shape of the object; The processing unit estimates a value of a parameter expressing the prototype shape as a size of an object present in the container based on sensor information included in the scene information input by the input unit. Object recognition method.

13. An input unit inputs scene information indicating a state of a predetermined scene in a container for storing an object; a processing unit, based on the scene information input by the input unit, sorting objects present in the container by size, and based on the number of sorted objects by size, detecting objects of a size that cannot be stored in the container; an output unit outputs a result of the processing by the processing unit; Including, the size of an object present in the container is a combination of parameter values ​​expressing a prototype shape of the object; The scene information input by the input unit includes sensor information obtained by a sensor regarding a state of the container, the processing unit repeats fitting an object point cloud obtained by extracting an object from sensor information included in scene information input by the input unit to a model of the original shape and adjusting parameter values ​​of the model of the original shape to estimate a size and a posture of the object. Object recognition method.

14. An input unit inputs scene information indicating a state of a predetermined scene in a container for storing an object; a processing unit, based on the scene information input by the input unit, sorting objects present in the container by size, and based on the number of sorted objects by size, detecting objects of a size that cannot be stored in the container; an output unit outputs a result of the processing by the processing unit; Including, the input unit inputs presence information indicating the number of types of objects present in the container; the processing unit detects an object having a size that is not permitted for storage in the container based on the presence information. Object recognition method.

Images