Image processing method for amorphous object

Abstract

An image processing unit (401) individually extracts objects from among objects to be screened including a plurality of amorphous objects (1a, 1b), whereby image processing can be individually performed on each object, and gripping positions, types, etc. can be derived.

Description

Technical Field

[0001] The present invention relates to an image processing method of an amorphous object for identifying a single object from among screening objects including a plurality of amorphous objects by deep learning. Background Art

[0002] In recent years, in various industries in society, picking devices that use robots to pick up specific parts from screening objects composed of multiple component groups for screening have been widely used. For example, in the field of home appliance recycling, specific parts are screened and recovered by identifying and picking up disassembled products or crushed products that have become useless or have been roughly crushed according to each component, so as to be reused. In addition, with respect to the screening objects, products with different numbers of components are randomly disassembled and put into a conveyor. Therefore, it is difficult to arrange the disassembled products neatly for transportation, and it is necessary to determine the holding position of the randomly placed screening objects and judge whether they can be picked up.

[0003] Patent Document 1 proposes a method of identifying the gripping positions of randomly placed components by comparing a master model consisting of simple shapes of the components with a screening object.

[0004] exist Fig.12 , the appearance structure of the robot 501 described in Patent Document 1 is shown. Fig.12 In the figure, 501 is a robot, 502 is a workbench, 503 is a robot body, 504 is a robot controller, 505 is a hand, 506 is a tray, 507 is a component, 508 is a component supply unit, 509 is a holding position detection device, 510 is a camera, and 511 is an image recognition device.

[0005] Next, for the processing steps based on the above structure, refer to Fig.13 as well as Fig.14 To explain.

[0006] Fig.13 The flowchart of FIG. 1 shows the processing steps executed by the holding position detection device 509 and the image recognition device 511 .

[0007] exist Fig.14 In FIG. 1 , a schematic diagram is shown for explaining the relationship between the degree of coincidence and the overlapping state of the components. Fig.14 , P1 to P3 are images of candidate parts that can be grasped.

[0008] In step S1 , the component supply unit 508 is photographed by the camera 510 , and the two-dimensional image is input to the image recognition device 511 .

[0009] Next, in step S2 , the entire image data is compared with the first main model to obtain the degree of consistency under the first main model, thereby extracting the graspable candidate component.

[0010] Next, in step S3, it is determined whether a candidate component that can be held is extracted. If so, in step S4, the candidate component that can be held is compared with the second main model to obtain the consistency under the second main model. Here, the candidate component that can be held extracted in the first comparison stage of step S2 includes a part that overlaps with other components 507. That is, among the candidate components that can be held, Fig.14 There are parts that overlap in parts other than the holding part, such as parts shown in P1, which overlap slightly to the extent that the holding part is in contact with other parts 507, such as parts shown in P2, and parts that overlap with other parts 507 in the holding part, such as parts shown in P3. In this case, in the second comparison stage of the above-mentioned step S4, i.e., the comparison of the holding part, in the state of P3, the consistency under the second main model becomes smaller. In the state of P2, although the consistency under the second main model is relatively large, it is less than the consistency under the first main model. In the state of P1, the consistency under the second main model is relatively large and greater than the consistency under the first main model. Therefore, regarding Fig.14 In the region indicated by the diagonal lines, since the degree of consistency under the second main model is greater than the degree of consistency under the first main model, it can be determined that there is no overlap with other components 507 at the gripping position and the component can be gripped.

[0011] Prior Art Literature

[0012] Patent Literature

[0013] Patent Document 1: Japanese Patent Application Publication No. 2002-200588 Summary of the invention

[0014] An image processing method for an amorphous object according to one embodiment of the present invention includes:

[0015] The image processing unit acquires a plurality of learning image data captured from above a conveying unit that conveys a plurality of irregular screening objects;

[0016] An image obtained by cutting out only the screening object from an area smaller than the area of ​​the learning image data from the acquired multiple learning image data is used as the learning image data of the object, an image obtained by cutting out only the background from an area smaller than the area of ​​the learning image data from the acquired multiple learning image data is used as the learning image data of the background, and an image obtained by cutting out the boundary portions where the multiple screening objects are connected to each other from the acquired multiple learning image data is used as the learning image data of the boundary portion; based on a learned model generated by learning the learning image data of the object, the learning image data of the background, and the learning image data of the boundary portion through deep learning, new image data captured from above the conveying portion that conveys the multiple screening objects is classified into an object, a background, and a boundary portion for each pixel of the new image data by the image processing portion;

[0017] The image processing unit derives the position coordinates, angle, gripping position or type of the screening object from the classified image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1A It is a schematic structural diagram of the pickup device in Embodiment 1 of the present invention.

[0019] Figure 1B It is a schematic configuration diagram of an image processing unit and the like of a pickup device in Embodiment 1 of the present invention.

[0020] Figure 1C This is a schematic configuration diagram of another example of the image processing unit of the pickup device in the first embodiment of the present invention.

[0021] Figure 2A This is a schematic diagram of captured image data in Embodiment 1 of the present invention.

[0022] Figure 2B This is a schematic diagram of the inferred image data in the first embodiment of the present invention.

[0023] Figure 2C This is a schematic diagram of binary image data in the first embodiment of the present invention.

[0024] Figure 3A This is a schematic diagram of the procedure for generating a learned model in the first embodiment of the present invention.

[0025] Figure 3B This is a schematic diagram of the inference processing method of the learned model in the first embodiment of the present invention.

[0026] Figure 4This is a flowchart showing the image processing procedure in Embodiment 1 of the present invention.

[0027] Figure 5A This is a schematic diagram of binary image data in the first embodiment of the present invention.

[0028] Figure 5B This is a schematic diagram of captured image data in Embodiment 1 of the present invention.

[0029] Figure 5C This is a schematic diagram of transmission image data in Embodiment 1 of the present invention.

[0030] Fig. 6A This is a schematic diagram of captured image data in Embodiment 1 of the present invention.

[0031] Figure 6B This is a schematic diagram of classified image data in Embodiment 1 of the present invention.

[0032] Fig. 7A This is a schematic diagram showing captured image data and a rectangular area in Embodiment 1 of the present invention.

[0033] Figure 7B This is a schematic diagram showing a learning image database in Embodiment 1 of the present invention.

[0034] Figure 8 It is a schematic structural diagram of a pickup device in Embodiment 2 of the present invention.

[0035] Fig. 9 This is a flowchart showing the image processing procedure in Embodiment 2 of the present invention.

[0036] Fig. 10A This is a schematic diagram of captured image data in Embodiment 2 of the present invention.

[0037] Fig. 10B This is a schematic diagram of inferred image data in Embodiment 2 of the present invention.

[0038] Fig. 10C This is a schematic diagram of binary image data in Embodiment 2 of the present invention.

[0039] Fig.11A This is a schematic diagram showing captured image data and a rectangular area in Embodiment 2 of the present invention.

[0040] Fig. 11B This is a schematic diagram showing a learning image database in Embodiment 2 of the present invention.

[0041] Fig.12 This is an external structural diagram of the robot described in Patent Document 1.

[0042] Fig.13 This is a flowchart showing the processing procedure executed by the gripping position detection device described in Patent Document 1.

[0043] Fig.14 This is a diagram showing the relationship between the degree of coincidence described in Patent Document 1 and the overlapping state of other components. DETAILED DESCRIPTION

[0044] The method of Patent Document 1 can determine the holding position for a fixed-shaped product for which a master model can be prepared in advance, but has a problem that the holding position cannot be determined for an indefinite-shaped product for which a master model is difficult to prepare in advance.

[0045] For example, the dismantled products of used home appliances in home appliance recycling factories have various manufacturers, models, and years, and they are also dirty or deformed, with large deviations. In addition, these dismantled products are sometimes transported on belt conveyors in a state of contact or overlapping with each other. When robots pick up these dismantled products transported on belt conveyors, in order to determine the gripping position, it is necessary to understand whether the dismantled products overlap with each other.

[0046] However, it is very difficult to compare the various external shapes of dismantled products of used home appliances with a master model as in Patent Document 1.

[0047] The present invention solves the above-mentioned conventional problems and aims to provide an image processing method of an indefinite object capable of deriving position coordinates, angles, gripping positions or types of various indefinite objects of disassembled home appliances.

[0048] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0049] [Implementation Method 1]

[0050] First, a pickup device capable of implementing the image processing method of an indefinite object in the first embodiment of the present invention will be described.

[0051] Figure 1A , Figure 1B , Figure 1C It is a structural diagram of the pickup device in Embodiment 1 of the present invention.

[0052] exist Figure 1A , Figure 1B , Figure 1C301 is a conveying unit such as a conveyor belt, 302 is an object group, 303 is a camera such as a camera, 304 is a field of view of the camera, 305 is a robot, 306 is a recycling box, 307 is a specific object, 400 is a control unit, 401 is an image processing unit, and 200 is a learned model. The control unit 400 includes an image processing unit 401, and controls the action of the robot 305 based on the processing in the image processing unit 401.

[0053] like Figure 1B As shown, the image processing unit 401 includes an image data acquisition unit 401a, an inference processing unit 401b, a binarization processing unit 401c, a circumscribed rectangle generation unit 401d, an image data cutting unit 401e, an information extraction unit 401f and a transmission unit 401h, but the details are described later. The image data acquisition unit 401a acquires the image data to be processed. The inference processing unit 401b performs inference processing on the acquired data. The binarization processing unit 401c performs binarization processing on the inference result. The circumscribed rectangle generation unit 401d generates a circumscribed rectangle for the result of the binarization processing. The image data cutting unit 401e cuts data from the circumscribed rectangle. The information extraction unit 401f extracts information from the cut data. The transmission unit 401h transmits the extracted information to the control unit 400. The control unit 400 includes a grasping possibility determining unit 401g, which determines whether the robot 305 is graspable based on the extracted information and controls the operation of the robot 305 based on the result.

[0054] exist Figure 1A In the embodiment, the conveying unit 301 conveys a group of various objects 302, including disassembled products of a product, placed on a conveyor 301 as an example of the conveying unit, in one direction, for example, Figure 1A Here, the object group 302 is sometimes not arranged neatly at equal intervals on the conveyor belt, but is randomly supplied to the conveying unit 301 for conveyance. A camera device 303 is installed above the upstream side of the conveying unit 301A toward the conveying unit 301. The camera device 303 can be controlled by detecting the presence or absence of an image of the object group 302 by the image data acquisition unit 401a of the image processing unit 401, so that the camera device 303 starts to capture images when the object group 302 enters its field of view.

[0055] exist Figure 2A , a schematic diagram of captured image data 100 obtained by the image data acquisition unit 401 a from the imaging device 303 is shown.

[0056] exist Figure 2AIn the above, object 1a and object 1b are amorphous objects such as recycled home appliance dismantled products with different appearance shapes. Background 2 is the background part of the image data other than object 1a and object 1b. Boundary 3 is the boundary part where object 1a and object 1b overlap each other, or the boundary part where object 1a and object 1b touch each other.

[0057] The inference processing unit 401 b of the image processing unit 401 performs processing based on image recognition on the captured image data 100 obtained from the imaging device 303 described above.

[0058] Next, the learned model 200 stored in the inference processing unit 401b of the image processing unit 401 is described. The learned model used here refers to a learning model based on machine learning that is pre-constructed using image data as learning image data. In machine learning, there are deep learning, supervised learning, or reinforcement learning. In this embodiment 1, learning based on deep learning is adopted. There are two stages in the learned model. One is the stage of generating the learned model, and the other is the stage of performing inference processing using the generated learned model.

[0059] exist Figure 3A as well as Figure 3B , an outline of the generation steps and inference processing method of the learned model 200 is shown.

[0060] exist Figure 3A In the following, the steps for generating the learned model 200 are described.

[0061] right Figure 3A Step S11 is described below. In step S11, a plurality of captured image data 100 in which the object 1a and the object 1b are captured are collected. In this case, in each of the plurality of captured image data 100, not only the object 1a and the object 1b are captured, but also a plurality of objects may be captured.

[0062] Next, Figure 3A In step S12, image processing is performed on the plurality of captured image data 100 to create a database of learning images.

[0063] Next, Figure 3A In step S13, the learning model 200 is generated by executing the deep learning on the database of learning images, and is stored in the inference processing unit 401b.

[0064] exist Figure 3B In the following, the inference processing method of the learned model 200 is described.

[0065] right Figure 3BIn step S14 , the captured image data 100 is input to the image data acquisition unit 401 a of the image processing unit 401 .

[0066] Next, Figure 3B In step S15, the learned model 200 of the inference processing unit 401b performs inference processing on the captured image data 100 of the image data acquisition unit 401a.

[0067] Next, Figure 3B In step S16, an inference process based on the learned model 200 is executed, and image information of the screening object required for picking up is generated by the inference processing unit 401b as an inference result, and output to the binarization processing unit 401c.

[0068] Next, for the image processing method in the image processing unit 401, refer to Figure 2A to Figure 2C as well as Figure 4 To explain.

[0069] Figure 2A 2 is a schematic diagram of the captured image data 100. The captured image data 100 is image data in which an object 1a and an object 1b overlap each other.

[0070] against Figure 2A The camera image data 100 that has been subjected to inference processing based on the learned model 200 is: Figure 2B A schematic diagram of inferred image data 101.

[0071] against Figure 2B The image data 101 after inference is converted into binary form. Figure 2C Schematic diagram of binary image data 102.

[0072] Figure 4 The flowchart shows the processing steps in the image processing unit 401.

[0073] First of all, Figure 4 In step S21 , the captured image data 100 obtained by capturing the group of objects 302 conveyed on the conveying unit 301 by the camera 303 is input to the image data acquisition unit 401 a of the image processing unit 401 .

[0074] Next, Figure 4 In step S22, the image data input to the image data acquisition unit 401a is processed. Figure 2A The captured image data 100 is subjected to inference processing based on the learned model 200 of the inference processing unit 401b to generate Figure 2BThe inferred image data 101 is output to the binary processing unit 401c.

[0075] exist Figure 2B In FIG. 4 , object data 41 is image data showing the inference result of object 1a and object 1b. Background data 42 is image data showing the inference result of background 2. Boundary data 43 is image data showing the inference result of boundary 3.

[0076] Next, Figure 4 In step S23, the binarization processing unit 401c of the image processing unit 401 performs binarization processing on the inferred image data 101 to generate Figure 2C The binary image data 102 is output to the circumscribed rectangle generating unit 401d.

[0077] exist Figure 2C In the figure, the binarized object data 51a and the binarized object data 51b are image data obtained by the binarization processing unit 401c performing binarization processing on the object data 41. The binarized background data 51c is image data obtained by the binarization processing unit 401c performing binarization processing on an image data region obtained by combining the background data 42 and the boundary data 43.

[0078] In step S23, the binarization processing unit 401c generates binarized object data 51a and binarized object data 51b as two independent object image data. That is, although the object 1a and the object 1b partially overlap each other in the captured image data 100, the overlapping objects can be displayed as separate image data by generating the binarized image data 102.

[0079] Next, Figure 4 Step S24 is described below. Figure 5A Schematic diagram showing the circumscribed rectangle 61c and the circumscribed rectangle 61d on the binary image data 102. In step S24, the circumscribed rectangle generating unit 401d of the image processing unit 401 surrounds the binary object data 51a and the binary object data 51b with circumscribed rectangles, thereby generating the circumscribed rectangle 61c and the circumscribed rectangle 61d, and outputting them to the image data cutting unit 401e. Figure 5A In FIG. 5 , the circumscribed rectangle 61 c is a circumscribed rectangle that encloses the binary object data 51 a , and the circumscribed rectangle 61 d is a circumscribed rectangle that encloses the binary object data 51 b .

[0080] Next, Figure 4 Step S25 is described below.

[0081] Figure 5BIt is a schematic diagram showing the circumscribed rectangle 61 c and the circumscribed rectangle 61 d displayed on the captured image data 100 .

[0082] Figure 5C 61e and 61f are schematic diagrams of transmission image data. Figure 5C In step S25, the circumscribed rectangle 61c and the circumscribed rectangle 61d are displayed on the captured image data 100 at the same coordinates as the binary image data 102. In step S25, the image data cutout section 401e cuts out the image data within the circumscribed rectangle 61c and the circumscribed rectangle 61d in the captured image data 100, respectively, to create the transmission image data 61e and the transmission image data 61f, and outputs them to the information extraction section 401f.

[0083] exist Figure 5B In the embodiment, the circumscribed rectangle 61c and the circumscribed rectangle 61d are displayed in the captured image data 100, so that the objects 1a and 1b are surrounded by the circumscribed rectangle 61c and the circumscribed rectangle 61d. Thus, the multiple different objects 1a and 1b conveyed in an overlapping state can be recognized as separate objects.

[0084] Next, Figure 4 In step S26, the position coordinates of the object, the angle of the object, the holding position of the object, and / or the type data of the object, which are the information required for picking up, are obtained from the transmission image data 61e and the transmission image data 61f respectively through image processing based on the information extraction unit 401f.

[0085] In addition, in this step S26, instead of using the transmission image data 61e and the transmission image data 61f extracted from the camera image data 100 in step S25, an image extracted by cutting out the binary image data 102 using the circumscribed rectangle 61c and the circumscribed rectangle 61d may be used. In addition, in the case where the object group 302 circulating in the pickup device is composed of multiple types of objects, that is, in the case of multiple types of screening with multiple recycling boxes 306, in this step S26, both the type discrimination and the type information can be obtained. In addition, generally, the appearance information of the object is required for type discrimination, so the image of the object cut out from the camera image data 100 is more suitable, but in the case where the type discrimination can be performed based on the shape of the object, the image of the object cut out from the binary image data 102 can also be used.

[0086] Next, Figure 4In step S27, the information acquired in step S26 can also be transmitted by the transmission unit 401h to the control unit 400 of the picking device, and the gripping judgment unit 403 of the control unit 400 judges whether the object can be picked up, i.e., gripped, based on the transmitted information, thereby performing the desired picking action, and the robot 305 selects and recycles the specific object 307 into the recycling box 306.

[0087] Instead of such processing, the following processing can be performed. That is, instead of arranging the gripping ability determination unit in the control unit 400, Figure 1C As shown in FIG. 2 , the image processing unit 401 is provided with the image processing unit 401, so that as step S27A, based on the information obtained in step S26, the grasping determination unit 401g of the image processing unit 401 determines whether it is possible to pick up, i.e., grasp. Then, in step S27, the transmission unit 401h can also transmit the determination result to the control unit 400 of the picking device to perform the desired picking action, and the robot 305 selects and collects the specific object 307 into the collection box 306.

[0088] Next, the method of creating the learned model 200 will be described in detail.

[0089] exist Fig. 6A , an example of captured image data 100 used for creating learning image data is shown.

[0090] As the captured image data 100, image data of various appearance shapes and various materials as the captured object can be used. The captured image data 100 is an example of two-dimensional image data captured with the recycled dismantled home appliance as the object. Fig. 6A In the figure, object 1a and object 1b are indefinite objects such as recycled home appliance dismantled products with different appearance shapes. Background 2 is the background part of the image data excluding object 1a and object 1b. Boundary 3 is the boundary part where object 1a and object 1b are in contact with each other.

[0091] During learning, a plurality of captured image data 100 obtained by capturing the object 1a and the object 1b in advance are prepared. In this case, each of the plurality of captured image data 100 may capture not only the object 1a and the object 1b but also a plurality of objects.

[0092] exist Figure 6B , a schematic diagram of inferred image data 101 is shown. The inferred image data 101 refers to image data obtained by classifying the captured image data 100 into object data 41, background data 42, and boundary data 43. Figure 6BIn FIG. 4 , object data 41 is image data of object 1 a and object 1 b , background data 42 is image data of background 2 , and boundary data 43 is image data of boundary 3 .

[0093] Fig. 7A 2 is a schematic diagram showing a rectangular area 20 displayed on the captured image data 100. Fig. 7A In FIG. 1 , a rectangular area 20 is image data showing an area cut out from the captured image data 100 in order to create a learning image database 201 .

[0094] exist Figure 7B , a schematic diagram of the learning image database 201 is shown. Figure 7B In FIG. 2 , the learning image database 201 refers to a learning image data set storing an object data set 31 , a background data set 32 ​​, and a boundary data set 33 .

[0095] exist Figure 7B In the figure, the object data set 31 is an image data set storing a plurality of object learning image data 21. The background data set 32 ​​is an image data set storing a plurality of background learning image data 22. The boundary data set 33 is an image data set storing a plurality of boundary learning image data 23.

[0096] exist Figure 7B , the object learning image data 21 is image data obtained by cutting out the rectangular region 20 from the object 1a and the object 1b. The background learning image data 22 is image data obtained by cutting out the rectangular region 20 from the background 2. The boundary learning image data 23 is image data obtained by cutting out the rectangular region 20 from the boundary 3.

[0097] In addition, Figure 7B In the figure, the object 1a and the object 1b are divided into two parts in the rectangle in the boundary learning image data 23. However, the boundary surface may be in the horizontal direction, in the vertical direction, or in the oblique direction as in the boundary learning image data 23.

[0098] In the first embodiment, the rectangular regions 20 are equal in size. However, the rectangular regions 20 may be cut out in regions of different sizes.

[0099] The created learning image database 201 is learned through deep learning, thereby creating a learned model 200 and storing it in the inference processing unit 401 b .

[0100] As a result, as described in step S22, the inference processing unit 401b performs inference processing based on the learned model 200 thus created, thereby generating inferred image data 101 from the captured image data 100. Furthermore, by generating the inferred image data 101 by the inference processing unit 401b, the inference processing unit 401b can obtain image data classified into the object data 41, the background data 42, and the boundary data 43 for each pixel.

[0101] As described above, according to the image processing method of amorphous objects involved in the above-mentioned embodiment of the present invention, the inference processing unit 401b, the binarization processing unit 401c, the circumscribed rectangle generating unit 401d and the image data cutting unit 401e of the image processing unit 401 can extract objects 1a and 1b separately from the screening target objects including a plurality of amorphous objects, thereby making it possible to perform image processing on each object 1a and 1b individually, and derive the position coordinates, angles, holding positions or types of various amorphous objects such as dismantled home appliances of the objects 1a and 1b by the information extraction unit 401f, thereby making it possible to perform screening operations.

[0102] [Implementation Method 2]

[0103] According to the first embodiment, by learning the boundary portion 3 which serves as the dividing line between the objects 1a and 1b, objects 1a and 1b can be individually extracted from the screening objects including a plurality of amorphous objects, thereby making it possible to perform image processing on each object 1a and 1b individually, and perform screening operations by deriving the holding positions of the objects 1a and 1b and distinguishing their types.

[0104] However, the image data of the outer edge of the object 1a, which is the boundary between the object 1a and the background 2, and the image data of the boundary 3, which is the boundary between the objects 1a and 1b, are image data with very similar features. Therefore, in the image processing method of the first embodiment, when the outer edge of the object 1a is mistakenly recognized as the boundary 3, the binary object data 51a is sometimes extracted smaller than the actual image data of the object 1a by the amount corresponding to the image data of the boundary 3. Therefore, a learned model is created in which the outer edge, which is the boundary between the object and the background, is additionally learned in addition to the object, background, and boundary learned by the learned model 200. By using the image processing method using the learned model that has learned the outer edge, the object 1a can be extracted as a region of the same size as the actual size, and the image processing of the object can be performed more accurately.

[0105] First, a pickup device according to a second embodiment of the present invention will be described.

[0106] Figure 8 2 is a structural diagram of a pickup device in Embodiment 2 of the present invention. Figure 8 In Figure 1A The same components as those of the pickup device are denoted by the same reference numerals and their descriptions are omitted.

[0107] exist Figure 8 In the figure, 210 is a learned model and 402 is a control unit. The control unit 402 is provided separately from the image processing unit 401, but similarly to the first embodiment, the gripping determination unit may be provided in the control unit 402 or in the image processing unit 401.

[0108] The difference from the first embodiment is that the learned model 210 is stored in advance in the inference processing unit 401b of the image processing unit 401. Therefore, the description of the pickup device is omitted.

[0109] The learned model 210 is a learned model obtained by making the learned model 200 learn the image data anew. Therefore, the step of generating the learned model 210 can be performed separately from the following image processing method (for example, before the image processing method is implemented) in order to Figure 3A The learning completed model 200 is generated in the same steps as the generation steps, so the description is omitted.

[0110] Next, for the image processing method in the image processing unit 401, refer to Fig. 9 To explain.

[0111] Fig. 9 The flowchart shows the processing steps in the image processing unit 401.

[0112] exist Fig. 9 In, with Figure 4 The same processing in the flowchart is assigned the same reference numerals, and description thereof is omitted.

[0113] Fig. 10A is a schematic diagram of the captured image data 100. Fig. 10A The camera image data 100 that has been subjected to inference processing based on the learned model 210 is: Fig. 10B A schematic diagram of inferred image data 111.

[0114] exist Fig. 9 In, with Figure 4 The processing method of the flowchart is different in step S22' and step S23'.

[0115] right Fig. 9The step S22' of FIG. 2 is described below. In step S22', the inference processing unit 401b performs inference processing based on the learned model 210 on the camera image data 100 input to the image data acquisition unit 401a to generate inferred image data 111, and outputs it to the binary processing unit 401c. That is, the inference processing unit 401b uses an image obtained by cutting out the boundary portion of the screening object and the background from the acquired multiple learning image data with an area smaller than the area of ​​the learning image data as the learning image data for the outer edge to make the learned model 210 learn, so that each pixel of the new image data can be classified into an object, a background, a boundary portion, and an outer edge.

[0116] exist Fig. 10B , object data 71 is image data showing the inference result of object 1a and object 1b. Background data 72 is image data showing the inference result of background 2. Boundary data 73 is image data showing the inference result of boundary 3. Moreover, as a point different from embodiment 1, outer edge data 74 is image data showing the inference result of outer edge 4.

[0117] Next, Fig. 9 In step S23', the binarization processing unit 401c of the image processing unit 401 performs binarization processing on the inferred image data 111, thereby generating binarized image data 112, and outputting it to the circumscribed rectangle generating unit 401d.

[0118] exist Fig. 10C , the binarized object data 81a and the binarized object data 81b are image data obtained by binarizing the image data region obtained by the binarization processing unit 401c by adding the object data 71 and the outer edge data 74. The binarized background data 81c is image data obtained by binarizing the image data region obtained by adding the background data 72 and the boundary data 73.

[0119] In step S23 ′, the binarized object data 81 a and the binarized object data 81 b are generated as two independent object image data, and are output to the circumscribed rectangle generating unit 401 d .

[0120] That is, although the object 1 a and the object 1 b overlap each other in the captured image data 100 , the overlapping objects can be displayed separately as image data by generating the binary image data 112 .

[0121] Thus, after classifying the image data into objects, backgrounds, boundaries, and outer edges, the image data obtained by summing up the pixels of the classified objects and outer edges can be used as the classified image data during export, thereby exporting the position coordinates, angle, holding position, or type in step S26.

[0122] In addition, the area region of the binarized object data 81a and the binarized object data 81b is the total area of ​​the object and the outer edge. Therefore, the area region of the binarized object data 81a and the binarized object data 81b is extracted with a larger area than the area region of the object 1a and the object 1b of the captured image data 100. Therefore, by shrinking the area region of the extracted binarized object data 81a and the binarized object data 81b within the range of 5.5% to 12.0%, it is possible to set it to the same area region as the object 1a and the object 1b.

[0123] Regarding the processing after step S24, Figure 4 The processing after step S24 of the flowchart is the same and thus omitted.

[0124] Therefore, the image data obtained by shrinking the sum of the pixels of the object and the outer edge within the range of 5.5% to 12.0% can be used as classified image data for export, and the position coordinates, angle, gripping position or type can be exported in step S26.

[0125] Next, the method of creating the learned model 210 will be described in detail.

[0126] During learning, a plurality of captured image data 100 in which the object 1 a and the object 1 b are captured in advance are prepared.

[0127] exist Fig.11A , which is a schematic diagram showing a rectangular area 20 displayed on the captured image data 100.

[0128] exist Fig.11A In FIG. 2 , the rectangular area 20 is image data showing an area cut out from the captured image data 100 in order to create the learning image database 211 .

[0129] exist Fig. 11B , a schematic diagram of the learning image database 211 is shown.

[0130] exist Fig. 11B In, with Figure 7B The difference is that the outer edge data group 34 is added, so the detailed description of other parts is omitted.

[0131] exist Fig. 11BIn the example, the learning image database 211 refers to a learning image data set storing an object data set 31 , a background data set 32 ​​, a boundary data set 33 , and an outer edge data set 34 .

[0132] exist Fig. 11B In FIG. 1 , the outer edge data group 34 is an image data group storing a plurality of outer edge learning image data 24 .

[0133] exist Fig. 11B In FIG. 4 , the outer edge learning image data 24 is image data obtained by cutting out a rectangular region 20 from the outer edge 4 .

[0134] In addition, Fig. 11B In the image data 24 for learning the outer edge, the object 1a or object 1b and the background 2 are divided into two parts in the rectangle, but as in the image data 24 for learning the outer edge, the boundary surface can be in the horizontal direction, in the vertical direction, or in the oblique direction.

[0135] The created learning image database 211 is learned through deep learning, thereby creating a learned model 210 and storing it in the inference processing unit 401 b .

[0136] As described above, according to the second embodiment, the captured image data 100 can be classified into the object data 71, the background data 72, the boundary data 73, and the outer edge data 74 by creating a learning model 210 that has learned the outer edge learning image data 24 in addition to the learning image data of the object learning image data 21, the background learning image data 22, and the boundary learning image data 23 described in the first embodiment. Thus, by using the image processing method of the second embodiment, the object 1a can be extracted as a region having the same size as the actual size, and the image processing of the object can be performed more accurately compared with the first embodiment.

[0137] Embodiments 1 and 2 of the present invention have been described above, but the above-mentioned embodiments 1 and 2 are merely examples for implementing the present invention. Therefore, the present invention is not limited to the above-mentioned embodiments 1 and 2, and can be implemented by appropriately modifying the above-mentioned embodiments 1 and 2 without departing from the main purpose thereof. That is, by appropriately combining any of the various embodiments or modified examples, the effects possessed by each can be brought into play. In addition, the embodiments can be combined with each other, the embodiments can be combined with each other, or the embodiments can be combined with the embodiments, and the features in different embodiments or embodiments can also be combined with each other.

[0138] As described above, according to the image processing method of the amorphous object according to the above-mentioned aspect of the present invention, the image processing unit extracts an object individually from the screening target objects including a plurality of amorphous objects, thereby performing image processing on each object individually, and deriving the position coordinates, angle, holding position or type of various amorphous objects such as dismantled home appliances, thereby enabling screening operation. Therefore, it is useful for holding amorphous objects such as dismantled home appliances, for example.

[0139] Industrial Applicability

[0140] By using the image processing method of amorphous objects involved in the above-mentioned embodiment of the present invention, objects can be individually extracted from screening objects including multiple amorphous objects, thereby making it possible to perform image processing on each object individually and perform screening operations by deriving the holding position and distinguishing the type.

[0141] Description of Reference Numerals

[0142] 301 Transportation Department

[0143] 302 Object Groups

[0144] 303 Camera

[0145] 304 Camera Field of View

[0146] 305 Robot

[0147] 306 Recycling Bin

[0148] 307 Specific Objects

[0149] 400 Control Department

[0150] 401 Image Processing Department

[0151] 401a Image data acquisition unit

[0152] 401b Inference Processing Unit

[0153] 401c Binary Processing Department

[0154] 401d Circumscribed rectangle generation unit

[0155] 401e Image data extraction unit

[0156] 401f Information Extraction Department

[0157] 401g Control whether the judgment department

[0158] 401h Transmission Department

[0159] 402 Control Department

[0160] 403 Control whether the judgment department

[0161] 100 Camera image data

[0162] 1a Object

[0163] 1b Object

[0164] 2 Background

[0165] 3 Boundary Department

[0166] 200 Learning completed model

[0167] 101 Inferred Image Data

[0168] 41 Object Data

[0169] 42 Background Data

[0170] 43 Boundary data

[0171] 102 Binary image data

[0172] 51a Binarized object data

[0173] 51b Binary object data

[0174] 51c Background data after binary conversion

[0175] 61c Circumscribed rectangle

[0176] 61d Circumscribed rectangle

[0177] 61e Transmission image data

[0178] 61f Transmission image data

[0179] 31 Object Data Group

[0180] 32 Background data set

[0181] 33 Boundary data set

[0182] 21 Image Data for Object Learning

[0183] 22 Image Data for Background Learning

[0184] 23 Image data for boundary learning

[0185] 20 Rectangular area

[0186] 201 Image Database for Learning

[0187] 210 Learning Complete Model

[0188] 111 Inferred image data

[0189] 71 Object Data

[0190] 72 Background Data

[0191] 73 Boundary data

[0192] 74 External data

[0193] 112 Binary image data

[0194] 81a Binarized object data

[0195] 81b Binarized object data

[0196] 81c Background data after binary conversion

[0197] 211 Learning Image Database

[0198] 34 External data group

[0199] 24 Image data for external edge learning

[0200] 501 Robot

[0201] 502 Workbench

[0202] 503 Robot Body

[0203] 504 Robot Controller

[0204] 505 Hands

[0205] 506 Pallet

[0206] 507 Parts

[0207] 508 Parts Supply Department

[0208] 509 Grip Position Detection Device

[0209] 510 Camera

[0210] 511 Image recognition device

[0211] P1 Candidate components that can be held

[0212] P2 Candidate components that can be held

[0213] Candidate components that P3 can hold.

Claims

1. A method for processing an image of an amorphous object, comprising: The image processing unit acquires a plurality of learning image data captured from above a conveying unit that conveys a plurality of irregular screening objects; An image obtained by cutting out only the screening object from an area smaller than the area of ​​the learning image data from the acquired multiple learning image data is used as the learning image data of the object, an image obtained by cutting out only the background from an area smaller than the area of ​​the learning image data from the acquired multiple learning image data is used as the learning image data of the background, and an image obtained by cutting out the boundary portions where the multiple screening objects are connected to each other from the acquired multiple learning image data is used as the learning image data of the boundary portion; based on a learned model generated by learning the learning image data of the object, the learning image data of the background, and the learning image data of the boundary portion through deep learning, new image data captured from above the conveying portion that conveys the multiple screening objects is classified into an object, a background, and a boundary portion for each pixel of the new image data by the image processing portion; The image processing unit derives the position coordinates, angle, gripping position or type of the screening object from the classified image data.

2. The image processing method of an amorphous object according to claim 1, wherein: The image processing unit determines whether the screening object can be held based on the derived position coordinates, angle, holding position or type of the screening object.

3. The image processing method of an amorphous object according to claim 1, wherein: Through the image processing unit, an image obtained by cutting out the boundary portion of the screening object and the background from an area smaller than the area of ​​the learning image data from the multiple learning image data acquired is used as the learning image data of the outer edge so that the learned model is learned, and each pixel of the new image data is classified into an object, background, boundary and outer edge.

4. The image processing method of an amorphous object according to claim 3, wherein: After the image data is classified into the object, background, boundary and outer edge through the image processing unit, the image data obtained by adding up the pixels of the classified object and outer edge is used as the classified image data during the export to export the position coordinates, angle, holding position or type.

5. The image processing method of an amorphous object according to claim 4, wherein: The image processing unit uses the image data obtained by summing up the pixels of the object and the outer edge to shrink the image data within a range of 5.5% to 12.0% as the classified image data during the export to export the position coordinates, angle, gripping position or type.

Images