Annotation method, device, system, method and computer program product using the same

Abstract

A labeling method, an apparatus, a system, a method, and a computer program product using the same. A method for automatically generating a pickable area in an image and a labeled image, comprising the following steps. First, under a generated background condition, a three-dimensional image is generated, and the three-dimensional image includes at least one three-dimensional object image. Then, a two-dimensional image of the three-dimensional image is captured, and the two-dimensional image includes a two-dimensional object image of the three-dimensional object image. Then, the object area of the two-dimensional object image is identified. Then, the exposure ratio of the exposure area of the two-dimensional object image to the object area of the object area is obtained. Then, it is determined whether the exposure ratio is greater than a preset ratio. Then, when the exposure ratio is greater than the preset ratio, the exposure area is defined as the pickable area.

Description

Technical Field

[0001] The present disclosure relates to a method for marking pickable areas in an image, an apparatus applying the same, a pick-and-place system, a pick-and-place method, and a computer program product, and particularly relates to a method for automatically generating an image and marking pickable areas in the image, an apparatus applying the same, a pick-and-place system, a pick-and-place method, and a computer program product. Background Art

[0002] The known marking method is to manually capture a two-dimensional object image of a physical object, and then manually mark the pickable areas in the two-dimensional object image, and then use machine learning technology to learn these marked information. However, machine learning usually requires a large number of two-dimensional object images. Therefore, the method of manually capturing two-dimensional object images of physical objects is quite time-consuming and inefficient. Therefore, how to improve the foregoing known problems is one of the goals that those skilled in the art strive for. Summary of the Invention

[0003] An embodiment of the present disclosure provides a method for automatically generating an image and marking pickable areas in the image. The method for automatically generating an image and marking pickable areas in the image includes the following steps: generating a three-dimensional image under a generated background condition, where the three-dimensional image includes at least one three-dimensional object image; capturing a two-dimensional image of the three-dimensional image, where the two-dimensional image includes a two-dimensional object image of the three-dimensional object image; identifying an object area of the two-dimensional object image; obtaining an exposure ratio of an exposure area of the two-dimensional object image to an object area of the object area; determining whether the exposure ratio is greater than a preset ratio; and when the exposure ratio is greater than the preset ratio, defining the exposure area as a pickable area.

[0004] Another embodiment of the present disclosure provides an apparatus for automatically generating an image and marking pickable areas in the image. The apparatus includes a generator, an apparatus camera, and a marker. The generator is configured to generate a three-dimensional image under a generated background condition, where the three-dimensional image includes at least one three-dimensional object image. The apparatus camera is configured to capture a two-dimensional image of the three-dimensional image, where the two-dimensional image includes a two-dimensional object image of the three-dimensional object image. The marker is configured to: identify an object area of the two-dimensional object image; obtain an exposure ratio of an exposure area of the object area to an object area of the object area; determine whether the exposure ratio is greater than a preset ratio; and when the exposure ratio is greater than the preset ratio, define the exposure area as a pickable area.

[0005] Another embodiment of the present disclosure provides a pick-and-place system. The pick-and-place system includes a device for automatically generating an image and annotating a pick-and-place area in the image, a system camera, a robotic arm, and a controller. The device includes a generator, a device camera, and an annotator. The generator is configured to generate a three-dimensional image under a generated background condition, where the three-dimensional image includes at least one three-dimensional object image. The device camera is configured to capture a two-dimensional image of the three-dimensional image, where the two-dimensional image includes a two-dimensional object image of the three-dimensional object image. The annotator is configured to: identify an object area of the two-dimensional object image; obtain an exposure ratio of an exposed area of the object area to an object area of the object area; determine whether the exposure ratio is greater than a preset ratio; and, when the exposure ratio is greater than the preset ratio, define the exposed area as a first pick-and-place area. The system camera is configured to capture a two-dimensional image of a physical object, and the two-dimensional image includes the two-dimensional object image. The controller is electrically connected to the device and is configured to: analyze the two-dimensional object image and obtain a second pick-and-place area of the two-dimensional object image based on the information of the first pick-and-place area obtained by the device; and control the robotic arm to pick and place the physical object at a pick-and-place part corresponding to the second pick-and-place area.

[0006] Another embodiment of the present disclosure provides a pick-and-place method. The pick-and-place method includes the following steps. Under a generated background condition, generate a three-dimensional image, where the three-dimensional image includes at least one three-dimensional object image; capture a two-dimensional image of the three-dimensional image, where the two-dimensional image includes a two-dimensional object image of the three-dimensional object image; identify an object area of the two-dimensional object image; obtain an exposure ratio of an exposed area of the object area to an object area of the object area; determine whether the exposure ratio is greater than a preset ratio; when the exposure ratio is greater than the preset ratio, define the exposed area as a first pick-and-place area; capture a two-dimensional image of a physical object, where the two-dimensional image includes the two-dimensional object image; analyze the two-dimensional object image and obtain a second pick-and-place area of the two-dimensional object image based on the information of the first pick-and-place area; and control the robotic arm to pick and place the physical object at a pick-and-place part corresponding to the second pick-and-place area.

[0007] Another embodiment of the present disclosure provides a computer program product. The computer program product is loaded into a device for automatically generating an image and annotating a pick-and-place area in the image to execute a method for automatically generating an image and annotating a pick-and-place area in the image. The method includes: under a generated background condition, generate a three-dimensional image, where the three-dimensional image includes at least one three-dimensional object image; capture a two-dimensional image of the three-dimensional image, where the two-dimensional image includes a two-dimensional object image of the three-dimensional object image; identify an object area of the two-dimensional object image; obtain an exposure ratio of an exposed area of the object area to an object area of the object area; determine whether the exposure ratio is greater than a preset ratio; and, when the exposure ratio is greater than the preset ratio, define the exposed area as a pick-and-place area.

[0008] To have a better understanding of the above and other aspects of the present disclosure, specific embodiments are given below and are described in detail in conjunction with the accompanying drawings as follows: Description of the Drawings

[0009] Figure 1 A functional block diagram of a device for automatically generating an image and an annotated image of a first pick-and-place area according to an embodiment of the present disclosure is shown.

[0010] Figure 2 Shown Figure 1 is a schematic diagram of a three-dimensional image generated by a generator of the device.

[0011] Figure 3 Shown Figure 2 is a schematic diagram of a two-dimensional image captured by a device camera of the device.

[0012] Figure 4 A depth perspective schematic diagram of a three-dimensional image according to another embodiment of the present disclosure is shown.

[0013] Figure 5 Shown Figure 3 is a schematic diagram of a plurality of two-dimensional object images.

[0014] Figure 6 A schematic diagram of a pick-and-place system according to an embodiment of the present disclosure is shown.

[0015] Figure 7 Shown Figure 6 is a flowchart of an automatic generation image and an annotated image of a first pick-and-place area of the device.

[0016] Figure 8 Shown Figure 6 is a pick-and-place flowchart of picking and placing a physical object of the pick-and-place system.

[0017] Figure 9 Shown Figure 6 is a schematic diagram of a two-dimensional object image captured by the pick-and-place system.

[0018] Description of Reference Numerals

[0019] 10: Pick-and-place system

[0020] 11: Pick-and-placer

[0021] 11a: Pick-and-place opening

[0022] 12: Robot arm

[0023] 13: System camera

[0024] 14: Controller

[0025] 100: Device

[0026] 110: Generator

[0027] 120: Device camera

[0028] 130: Marker

[0029] AER: Exposed area

[0030] AOR: Object area

[0031] A11: Pick-and-place opening area

[0032] C1: Second pick-and-place area

[0033] D: Information

[0034] H1: Preset depth

[0035] O1: Physical object

[0036] O11: Pick-and-place part

[0037] M2D, M2D, 1, M2D, 2, MO, 2D, MO, 2D, 1, MO, 2D, 2: Two-dimensional object image

[0038] M3D: Three-dimensional object image

[0039] MOR, MOR, 1, MOR, 2, MOR1, MOR2, MOR3, MOR4: Object region

[0040] MER, MER1, MER2: Exposed region

[0041] MSR: Occluded part region

[0042] MV: Container image

[0043] MV1: Bottom

[0044] P2D, PO, 2D: Two-dimensional image

[0045] P3D: Three-dimensional image

[0046] R: Exposed ratio

[0047] S110~S160, S210~S250: Steps Detailed implementation manners

[0048] Please refer to Figures 1 to 6 , Figure 1 , which shows a functional block diagram of the device 100 for automatically generating an image and a labeled image of a pick-and-place area according to an embodiment of the present disclosure, Figure 2 shows Figure 1 a schematic diagram of the three-dimensional image P3D generated by the generator 110 of the device 100, Figure 3 shows Figure 2 a schematic diagram of the two-dimensional image P2D captured by the device camera 120 of the device 100,Figure 4 A schematic diagram of a depth perspective of a three-dimensional image P3D according to another embodiment of the present disclosure is shown. Figure 5 Shown Figure 3 is a schematic diagram of a plurality of two-dimensional object images M2D of Figure 6 A schematic diagram of a pick-and-place system 10 according to an embodiment of the present disclosure is shown.

[0049] As Figure 1 shown, the device 100 includes a generator 110, a device camera 120, and a labeler 130. At least two of the generator 110, the device camera 120, and the labeler 130 can be integrated into a single component. Alternatively, at least one of the generator 110, the device camera 120, and the labeler (or marker) 130 can be integrated into a processor (not shown) or a controller (not shown) of the device 100. At least one of the generator 110, the device camera 120, and the labeler 130 can adopt a physical circuit structure (circuit) formed by, for example, a semiconductor process, such as a semiconductor chip, a semiconductor package, or other types of circuit structures.

[0050] The generator 110 is used to generate a three-dimensional image P3D, where the three-dimensional image P3D includes at least one three-dimensional object image M3D. The device camera 120 is used to capture a two-dimensional image P2D of the three-dimensional image P3D, where the two-dimensional image P2D includes a two-dimensional object image M2D of the three-dimensional object image M3D. The labeler 130 is used to: (1). Identify the object region MOR of the two-dimensional object image M2D; (2). Obtain the exposure ratio R of the exposure area AER of the exposed region MER of the object region MOR to the object area AOR of the object region MOR; (3). Determine whether the exposure ratio R is greater than a preset ratio; and, (4). When the exposure ratio R is greater than the preset ratio, define the exposed region MER as the first pick-and-place area. The object area AOR of the aforementioned object region MOR is, for example, the area of the region surrounded by the outer boundary of the image of the object region MOR. Compared with manual labeling (or marking), the embodiment of the present disclosure quickly labels the first pick-and-place area of the two-dimensional object image by the device 100, which is relatively time-saving, fast, and more efficient.

[0051] As Figure 5As shown, taking two-dimensional object images M2D,1 and M2D,2 as examples, the two-dimensional object image M2D,1 overlaps the two-dimensional object image M2D,2. Therefore, a part of the object region MOR,2 of the two-dimensional object image M2D,2 is covered by the object region MOR,1 of the two-dimensional object image M2D,1. Specifically, the object region MOR,2 of the two-dimensional object image M2D,2 includes an occluded region MSR, an exposed region MER1, and an exposed region MER2. The occluded region MSR is covered by the two-dimensional object image M2D,1, and the exposed regions MER1 and MER2 are exposed from the two-dimensional object image M2D,1.

[0052] The annotator 130 identifies the range of the object region MOR,1, the range of the object region MOR,2, the range of the occluded region MSR of the object region MOR,2, the range of the exposed region MER1, and the range of the exposed region MER2 through image analysis technology, and calculates the area of the object region MOR,1, the area of the object region MOR,2, the area of the occluded region MSR of the object region MOR,2, the area of the exposed region MER1, and the area of the exposed region MER2 based on this. The aforementioned "identifying the range of the region" is, for example, "obtaining the coordinates of each of the multiple pixel points of the image of the region".

[0053] After obtaining the areas, the annotator 130 can calculate the exposure ratio R of the exposure area AER of the exposed region MER to the object area AOR of the object region MOR, and define the exposed region MER with an exposure ratio R greater than the preset ratio as the first pick-and-place area. For example, Figure 5 the exposure ratio R of the exposed region MER1 to the object area AOR is greater than the preset ratio. Therefore, the annotator 130 defines the exposed region MER1 as the first pick-and-place area, while Figure 5 the exposure ratio R of the exposed region MER2 to the object area AOR is not greater than the preset ratio. Therefore, the annotator 130 does not define the exposed region MER1 as the first pick-and-place area.

[0054] The embodiments of the present disclosure do not limit the value of the aforementioned preset ratio. It can be any real number between 20% and 80%, or less than 20%, such as 0%, or more than 80%, such as 100%. When the preset ratio is set to 0%, in the actual pick-and-place process, as long as it is a physical object with an exposed region, it can be picked and placed. When the preset ratio is set to 100%, in the actual pick-and-place process, only the completely exposed physical object will be picked and placed. The preset ratio can be determined according to the type of the object and / or the environment, and the embodiments of the present disclosure do not limit it.

[0055] In one embodiment, as Figure 5 shown, the annotator 130 is further configured to: (1). Determine the pick-and-place device 11 (shown in Figure 6) the pick-and-place opening 11a (shown in Figure 6 ) whether the area A11 of the pick-and-place opening completely falls within the exposed area MER, where the area A11 of the pick-and-place opening can be pre-set information; (2). When the area A11 of the pick-and-place opening completely falls within the exposed area MER, define the exposed area MER as the first pick-and-place area. For example, as Figure 5 shown, the labeler 130 uses image processing technology to determine that the area A11 of the pick-and-place opening completely falls within the exposed area MER1, so this exposed area MER1 is defined as the first pick-and-place area.

[0056] In an embodiment, the labeler 130 is further configured to: (1). Determine whether the depth where the three-dimensional object image M3D is located is greater than a preset depth; (2). When the depth where the three-dimensional object image M3D is located is greater than the preset depth, for the object area MOR above the preset depth, perform the steps of identifying the object area MOR of the two-dimensional object image M2D, obtaining the exposure ratio R, determining whether the exposure ratio R is greater than a preset ratio, and defining the exposed area MER as the first pick-and-place area.

[0057] For example, as Figure 4 shown, the three-dimensional image P3D further includes a container image MV, and all the three-dimensional object images M3D are located within the container image MV. Since the probability or the covered area of the three-dimensional object image M3D close to the bottom MV1 of the container image MV being covered by the upper three-dimensional object image M3D is large, it can be considered negligible. Based on this, the labeler 130 can only analyze the object areas above the preset depth H1, such as the object areas MOR1, MOR2, MOR3, and MOR4. In this way, the number of three-dimensional object images M3D analyzed by the device 100 can be reduced, the analysis speed can be increased, and the time required for analysis can be reduced. The labeler 130 analyzes the aforementioned object areas MOR1, MOR2, MOR3, and MOR4 in a manner similar to or the same as analyzing Figure 5 the object areas MOR, 1 and MOR, 2, so it will not be elaborated here.

[0058] In addition, Figure 4 merely for the purpose of indicating the preset depth H1, the device 100 actually still obtains the object areas above the preset depth H1 by analyzing Figure 3 the two-dimensional image P2D. In addition, the labeler 130 can use different preset depths H1 to analyze the object areas for different two-dimensional images P2D. For example, when the number of objects in the two-dimensional image P2D is stacked higher, the preset depth H1 can be higher.

[0059] Regarding the application of the information D of the first pick-and-place area, as Figure 6As shown, the tagger 130 can output the information D of the first pick-and-place area to an electronic file (not shown), or output it to the robotic arm 12 or the controller 14 of the pick-and-place system 10 (shown in Figure 6 ). In one embodiment, the information D of the first pick-and-place area includes the object name of the object region MOR and the coordinate values of each of the multiple pixels of the first pick-and-place area. For output to the pick-and-place system 10, during the actual process of picking and placing at least one physical object O1, the pick-and-place system 10 can, according to the information D of the first pick-and-place area, obtain the pick-and-place part O11 of this at least one physical object O1 corresponding to the first pick-and-place area, and through this pick-and-place part, facilitate / quickly pick and place the physical object O1.

[0060] As Figure 2 shown, the three-dimensional object image M3D is, for example, a three-dimensional object image of an item, and this item is, for example, any object that can be picked and placed by the pick-and-place system 10, such as a container, a tool, stationery, a doll (such as a cloth doll, etc.). The aforementioned container is, for example, a container in various fields such as a plastic bottle, a glass bottle, a Tetra Pak, a kettle, a bag, etc., the tool is, for example, a processing tool used in various fields such as a wrench, a hammer, etc., and the stationery is, for example, various document processing stationery such as a pen, a correction tape, a stapler, etc. The embodiments of the present disclosure do not limit the type of the item, and it can be any object that can be picked and placed by the pick-and-place system 10. In addition, the object can have hardness, softness, or a combination thereof, where softness means that the object has a large deformability or flexibility, and such objects are, for example, made of materials such as paper, cloth, rubber, plastic (possibly thin in thickness), or a combination thereof. Hardness means that the object has a small deformability or small flexibility, and such objects are, for example, made of materials such as metal, glass, plastic (possibly thick in thickness), wood, or a combination thereof.

[0061] In this embodiment, the device camera 120 is, for example, a virtual camera. Specifically, the device camera 120 is not a physical camera. The three-dimensional image P3D generated by the generator 110 includes at least one three-dimensional object image M3D. The device 100 can capture the two-dimensional object image M2D of the three-dimensional object image M3D through the device camera 120 to facilitate subsequent analysis of the first pick-and-place area of the two-dimensional object image M2D.

[0062] In addition, the device 100 can analyze the first pick-and-place area under generated background conditions. The generated background conditions include various environmental parameters that simulate (or are similar to) the actual environment of the pick-and-place system 10, such as light source type, number of light sources, light source attitude, light source irradiation angle, object type, number of objects, object surface texture, object attitude, background environment, camera viewing angle of the device camera 120, and / or the distance between the device camera 120 and the object. The annotator can execute a random algorithm based on any combination of the foregoing environmental parameters, so that one or more 3D models generate multiple virtual objects with different object attitude parameters and instantaneously generate light and shadow changes in a simulated scene containing light source objects based on randomly generated parameters.

[0063] Regarding the light source parameters, the light source parameters are, for example, one of a directional light, a point light, a spot light, and a sky light. In addition, different light source attitudes can cause different light and shadow changes in the virtual object (3D object image M3D) due to different lighting positions. Regarding the object attitude parameters, the object attitude parameters can be, for example, a combination of position information (Location), rotation information (Rotation), and scale information (Scale) represented by X, Y, and Z axis values. The foregoing position information can be represented, for example, as (x, y, z) or (x, y, z, rx, ry, rz), where x, y, and z are the coordinate values of the X, Y, and Z axes respectively, and rx, ry, and rz are physical quantities of rotation around the x, y, and z axes (r means rotation), such as angle values.

[0064] When randomly generating the foregoing object pose parameters, taking the annotator (simulator) as the Unreal Engine as an example, random algorithms such as Random Rotator, Random Rotator from Stream, Random Float in Range, RandomFloat in Range from Stream, Random Integer, Random Integer From Stream, RandomPoint in Bounding Box and other functions can be applied to randomly generate the object pose parameters of each virtual object. Taking the annotator as the Unreal Engine as an example, the random algorithm can be, for example, functions such as Random Integer, RandomInteger From Stream, Random Integer in Range, Random Integer In Range FromStream provided by the annotator, but not limited to this. As long as the function can generate random output values, it can be adopted according to requirements.

[0065] Regarding the environmental object parameters, the environmental object parameters are, for example, background objects located in the field of view of the device camera, such as a material basket or a trolley. Among them, the material basket itself also has defined object pose parameters, object type parameters and / or material parameters, so that the color, texture and / or size of the material basket can be defined in the annotator, and the type and / or size of the material basket can also be used as part of the marking information assigned to the material basket.

[0066] Since the generated background conditions are as consistent or close as possible to the actual environment where the pick-and-place system 10 is located, the first pick-and-place area information D obtained by analysis can increase the pick-and-place success rate of the pick-and-place system 10 when actually picking and placing objects (the higher the recognition accuracy rate of the first pick-and-place area, the higher the actual pick-and-place success rate).

[0067] Figure 2The three-dimensional image P3D is a three-dimensional image generated under specific generation background conditions. After analyzing a three-dimensional image P3D, the generator 110 can change at least one of the aforementioned generation background conditions, then generate a new three-dimensional image P3D, and use the same analysis method to obtain information on at least one first pick-and-place area of the new three-dimensional image P3D. The way the generator 110 changes the generation background conditions is, for example, randomly or according to set conditions, where the set conditions are, for example, pre-input by the user. In one embodiment, the number of three-dimensional images P3D analyzed by the device 100 is equal to or greater than 1. The more three-dimensional images P3D the device 100 analyzes under different generation background conditions, the more samples there are, and the higher the pick-and-place success rate of the pick-and-place system 10 when actually picking and placing objects. The embodiments of the present disclosure do not limit the number of three-dimensional images P3D analyzed by the device 100, which can be any number equal to or greater than 1. In one embodiment, the device 100 can continuously analyze three-dimensional images P3D under different generation background conditions.

[0068] Please refer to Figure 7 , which shows Figure 6 The flowchart of the automatic generation image and the first pick-and-place area in the labeled image of the device 100.

[0069] In step S110, as Figure 1 and Figure 2 shown, the generator 110 generates a three-dimensional image P3D, where the three-dimensional image P3D includes at least one three-dimensional object image M3D.

[0070] In step S120, as Figure 1 and Figure 3 shown, the device camera 120 captures a two-dimensional image P2D of the three-dimensional image P3D, where the two-dimensional image P2D includes a two-dimensional object image M2D of the three-dimensional object image M3D.

[0071] In step S130, the labeler 130 identifies the object region MOR of the two-dimensional object image M2D. Taking Figure 5 two two-dimensional object images M2D, 1 and M2D, 2 as an example, the labeler 130 can use image analysis techniques to identify the range of the object region MOR, 1, the range of the object region MOR, 2, and the range of the masked part region MSR of the object region MOR, 2, the range of the exposed region MER1, and the range of the exposed region MER2.

[0072] In step S140, the labeler 130 obtains the exposure ratio R of the exposure area AER of the exposed area MER of the object region MOR to the object area AOR of the object region MOR. Taking Figure 5The two-dimensional object image M2D. For example, the tagger 130 can adopt image analysis technology to calculate the object area MOR, and the exposure ratio R of the exposure area AER of the exposed area MER1 of the object area MOR, 2 to the object area AOR of the object area MOR, 2.

[0073] In step S150, the tagger 130 determines whether the exposure ratio R is greater than a preset ratio. If so, the process proceeds to step S160; if not, the generator 110 changes at least one of the foregoing generated background conditions, and then the process returns to step S110. In one embodiment, the generator 110 can randomly change at least one of the foregoing generated background conditions, or change at least one of the foregoing generated background conditions according to the foregoing set conditions, and then return to step S110. In one embodiment, after all two-dimensional object images M2D in the two-dimensional image P2D are analyzed, or after all two-dimensional object images M2D with a depth higher than the preset depth H1 are analyzed, the process returns to step S110.

[0074] In step S160, the tagger 130 defines the exposed area MER as the first pick-and-place area. Figure 5 For the two-dimensional object image M2D, 2, for example, since the exposure ratio R of the exposed area MER1 to the object area AOR is greater than the preset ratio, the tagger 130 defines the exposed area MER1 as the first pick-and-place area. In another embodiment, the tagger 130 can adopt image processing technology to determine that the pick-and-place port area A11 completely falls within the exposed area MER1, and thus defines the exposed area MER1 as the first pick-and-place area.

[0075] Then, the generator 110 can randomly change at least one of the foregoing generated background conditions, or change at least one of the foregoing generated background conditions according to the foregoing set conditions, and then the process returns to step S110. In another embodiment, the tagger 130 can output the object name of the first pick-and-place area and the coordinates of each of multiple pixel points to the robotic arm 12 (illustrated in Figure 6 ), the controller 14 (illustrated in Figure 6 ) or an electronic file (not illustrated).

[0076] As can be seen from the above, the device 100 continuously analyzes multiple three-dimensional images P3D under different generated background conditions. The more three-dimensional images P3D analyzed (the larger the number of samples), the higher the pick-and-place success rate of the pick-and-place system 10 when actually picking and placing objects. The embodiments of the present disclosure do not limit the number of three-dimensional images P3D analyzed by the device 100, which can be any positive integer equal to or greater than 1.

[0077] In an embodiment, steps S110 to S160 are automatically and / or actively completed by device 100, and manual processing is not required during the process. Therefore, the first pick-and-place area information of the object area can be obtained quickly, efficiently, and in a time-saving manner.

[0078] Please refer to Figure 8 and Figure 9 , Figure 8 illustrating Figure 6 the pick-and-place flow chart of pick-and-place entity object O1 of pick-and-place system 10, and Figure 9 illustrating Figure 6 the two-dimensional object image MO, 2D captured by pick-and-place system 10, a schematic diagram of 2D.

[0079] First, pick-and-place system 10 is provided. As Figure 6 shown, pick-and-place system 10 includes device 100, pick-and-placer 11, robotic arm 12, system camera 13, and controller 14. Controller 14 can be configured outside robotic arm 12, but can also be integrated into robotic arm 12 or configured inside device 100.

[0080] As Figure 6 shown, pick-and-placer 11 can be configured on robotic arm 12. In this embodiment, pick-and-placer 11 is, for example, a suction nozzle, which is connected to a vacuum source (not shown). The vacuum source can provide vacuum suction to pick-and-placer 11, so that pick-and-placer 11 can suck the pick-and-place part of entity object O1. When the vacuum source does not provide vacuum suction to pick-and-placer 11, pick-and-placer 11 can release the pick-and-place part of entity object O1. In addition, pick-and-placer 11 has a pick-and-place port 11a, and the pick-and-place port area A11 of pick-and-place port 11a is smaller than the pick-and-place part area of entity object O1, so that the maximum suction can be used to suck entity object O1. In another embodiment, pick-and-placer 11 is, for example, a magnetic gripper, and pick-and-placer 11 can selectively provide magnetic force or cut off the supply of magnetic force to pick-and-place entity object O1. In another embodiment, pick-and-placer 11 can also be a gripper to pick-and-place entity object O1 in a clamping manner.

[0081] As Figure 6 shown, system camera 13 is, for example, a solid-state camera, which can capture a two-dimensional image PO, 2D of at least one entity object O1, where the two-dimensional image PO, 2D includes the two-dimensional object image MO, 2D of each entity object O1. Controller 14 is used for: (1). Analyzing the two-dimensional object image MO, 2D, and obtaining the second pick-and-place area of each two-dimensional object image MO, 2D according to the information of the first pick-and-place area provided by device 100; (2). Controlling robotic arm 12 to move above or around the pick-and-place part corresponding to the second pick-and-place area in entity object O1; (3). Controlling pick-and-placer 11 to pick and place the pick-and-place part of entity object O1.

[0082] In step S210, the controller 14 receives the information D from the pick-and-place area of the device 100.

[0083] In step S220, the system camera 13 captures a two-dimensional image PO,2D of the physical object O1. The two-dimensional image PO,2D includes at least one two-dimensional object image MO,2D, such as Figure 9 the two-dimensional object images MO,2D,1 and MO,2D,2 shown.

[0084] In step S230, the controller 14 analyzes the two-dimensional object image MO,2D and, based on the information D of the first pick-and-place area provided by the device 100, obtains the second pick-and-place area of each two-dimensional object image MO,2D. Taking the Figure 9 two-dimensional object image MO,2D,2 as an example, the controller 14 analyzes the two-dimensional object image MO,2D,2 and, based on the information D of the first pick-and-place area provided by the device 100, obtains the second pick-and-place area C1 of the two-dimensional object image MO,2D,2. Since the device 100 has provided the controller 14 with the information D of at least one first pick-and-place area, the controller 14 can quickly obtain the information of the second pick-and-place area C1 of the two-dimensional object image MO,2D, such as size and / or position, etc., without the need or omission of complex image analysis of the two-dimensional object image MO,2D.

[0085] In step S240, the controller 14 controls the robotic arm 12 to move above or around the pick-and-place part O11 (the pick-and-place part O11 is shown in Figure 6 ) of the physical object O1 corresponding to the second pick-and-place area C1.

[0086] In step S250, the controller 14 controls the pick-and-place device 11 to suck the pick-and-place part O11 of the physical object O1 to suck the physical object O1. Specifically, the pick-and-place device 11 picks and places the physical object O1 through the pick-and-place part O11.

[0087] In one embodiment, Figure 8 before step S240 of

[0088] In addition, in one embodiment, Figure 7 and Figure 8 the process shown in

[0089] can be implemented by a computer program product (not shown).In summary, the apparatus for automatically generating an image and a labeled image of a pick-and-place area according to an embodiment of the present disclosure can automatically generate at least one image under different background conditions and label the first pick-and-place area in the labeled image. The information of this first pick-and-place area can be output as data in the form of a data table or provided to a pick-and-place system for use by the pick-and-place system. For example, the pick-and-place system captures a two-dimensional object image of a physical object and obtains a second pick-and-place area of the two-dimensional object image according to the information of the first pick-and-place area. Therefore, the physical object can be picked and placed through the pick-and-place part corresponding to the second pick-and-place area in the physical object. By the method for automatically generating an image and a labeled image of a pick-and-place area of the present application, the visual (or image) recognition success rate can be higher than 80%, for example, more than 92%.

[0090] In summary, although the present disclosure has been disclosed above in embodiments, it is not intended to limit the present disclosure. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the scope defined by the scope of the patent application of the claims.

Claims

1. A method for automatically generating pickable areas in an image and a labeled image, comprising: Generating a three-dimensional image under a generated background condition, the three-dimensional image including at least one three-dimensional object image; Capturing a two-dimensional image of the three-dimensional image, the two-dimensional image including a two-dimensional object image of the three-dimensional object image; Identifying the object area of the two-dimensional object image; Obtaining an exposure ratio of an exposure area of the exposed area of the two-dimensional object image to an object area of the object area; Determining whether the exposure ratio is greater than a preset ratio; And When the exposure ratio is greater than the preset ratio, defining the exposed area as a pickable area; Wherein the steps of generating the three-dimensional image, capturing the two-dimensional image of the three-dimensional image, identifying the object area of the two-dimensional object image, obtaining the exposure ratio, determining whether the exposure ratio is greater than the preset ratio, and defining the exposed area as the pickable area are automatically completed by a device.

2. The method for automatically generating pickable areas in an image and a labeled image according to claim 1, further comprising: Determining whether a pick-and-place opening area of a pick-and-place device completely falls within the exposed area; Wherein, the step of defining the exposed area as the pickable area further comprises: When the pick-and-place opening area completely falls within the exposed area, defining the exposed area as the pickable area.

3. The method for automatically generating pickable areas in an image and a labeled image according to claim 1, further comprising: Outputting the object name of the pickable area and the coordinate values of each of a plurality of pixel points to a robotic arm.

4. The method for automatically generating pickable areas in an image and a labeled image according to claim 1, further comprising: Determining whether a depth at which the three-dimensional object image is located is greater than a preset depth; When the depth at which the three-dimensional object image is located is greater than the preset depth, for the object area higher than the preset depth, performing the steps of identifying the object area of the two-dimensional object image, obtaining the exposure ratio, determining whether the exposure ratio is greater than the preset ratio, and defining the exposed area as the pickable area.

5. The method for automatically generating pickable areas in an image and a labeled image according to claim 1, further comprising: After defining the exposed area as the pickable area, changing the generated background condition, and then returning to the step of generating the three-dimensional image.

6. The method for automatically generating pickable areas in an image and a labeled image according to claim 1, further comprising: When the exposure ratio is less than or equal to the preset ratio, changing the generated background condition, and then returning to the step of generating the three-dimensional image.

7. A device for automatically generating pickable areas in an image and a labeled image, comprising: A generator for generating a three-dimensional image under a generated background condition, wherein the three-dimensional image includes at least one three-dimensional object image; A device camera for capturing a two-dimensional image of the three-dimensional image, wherein the two-dimensional image includes a two-dimensional object image of the three-dimensional object image; and A labeler for: Identifying the object area of the two-dimensional object image; Obtaining an exposure ratio of an exposure area of the exposed area of the object area to an object area of the object area; Determining whether the exposure ratio is greater than a preset ratio; And When the exposure ratio is greater than the preset ratio, define the exposed area as the pick-and-place area; Among them, the steps of generating the three-dimensional image, the step of capturing the two-dimensional image of the three-dimensional image, the step of identifying the object area of the two-dimensional object image, the step of obtaining the exposure ratio, the step of determining whether the exposure ratio is greater than the preset ratio, and the step of defining the exposed area as the pick-and-place area are automatically completed by the device.

8. The device for automatically generating an image and labeling a pick-and-place area in the image according to claim 7, wherein the labeler is further configured to: Determine whether the pick-and-place area of the pick-and-place opening of the pick-and-place device completely falls within the exposed area; and When the pick-and-place area completely falls within the exposed area, define the exposed area as the pick-and-place area.

9. The device for automatically generating an image and labeling a pick-and-place area in the image according to claim 7, wherein the labeler is further configured to: Output the object name of the pick-and-place area and the coordinate values of each of a plurality of pixels to the robotic arm.

10. The device for automatically generating an image and labeling a pick-and-place area in the image according to claim 7, wherein the labeler is further configured to: Determine whether the depth at which the three-dimensional object image is located is greater than a preset depth; When the depth at which the three-dimensional object image is located is greater than the preset depth, for the object area higher than the preset depth, perform the steps of identifying the object area of the two-dimensional object image, obtaining the exposure ratio, determining whether the exposure ratio is greater than the preset ratio, and defining the exposed area as the pick-and-place area.

11. The device for automatically generating an image and labeling a pick-and-place area in the image according to claim 7, wherein the generator is further configured to: After defining the exposed area as the pick-and-place area, change the generation background condition and then generate another three-dimensional image.

12. The device for automatically generating an image and labeling a pick-and-place area in the image according to claim 7, wherein the generator is further configured to: When the exposure ratio is less than or equal to the preset ratio, change the generation background condition and then generate another three-dimensional image.

13. A pick-and-place system, comprising: A device for automatically generating an image and labeling a pick-and-place area in the image, comprising: A generator for generating a three-dimensional image under a generation background condition, wherein the three-dimensional image includes at least one three-dimensional object image; A device camera for capturing a two-dimensional image of the three-dimensional image, wherein the two-dimensional image includes a two-dimensional object image of the three-dimensional object image; and A labeler for: Identifying the object area of the two-dimensional object image; Obtaining the exposure ratio of the exposed area of the object area to the object area of the object area; Determining whether the exposure ratio is greater than a preset ratio; and When the exposure ratio is greater than the preset ratio, defining the exposed area as the first pick-and-place area; A system camera for capturing a two-dimensional image of a physical object, the two-dimensional image including a two-dimensional object image; A robotic arm; and A controller electrically connected to the device and configured to: Analyze the two-dimensional object image and obtain a second pick-and-place area of the two-dimensional object image based on the information of the first pick-and-place area obtained by the device; and Control the pick-and-place part of the robotic arm for picking and placing the physical object corresponding to the second pick-and-place area.

14. A pick-and-place method, comprising: Generating a three-dimensional image under a generation background condition, the three-dimensional image including at least one three-dimensional object image; Capturing a two-dimensional image of the three-dimensional image, the two-dimensional image including a two-dimensional object image of the three-dimensional object image; Identifying the object area of the two-dimensional object image; Obtaining an exposure ratio of an exposure area of the two-dimensional object image to an object area of the object area; Determining whether the exposure ratio is greater than a preset ratio; When the exposure ratio is greater than the preset ratio, defining the exposure area as a first pick-and-place area; Capturing a two-dimensional image of a physical object, the two-dimensional image including a two-dimensional object image; Analyzing the two-dimensional object image and obtaining a second pick-and-place area of the two-dimensional object image according to the information of the first pick-and-place area; And Controlling the robotic arm to pick and place the physical object corresponding to the pick-and-place part of the second pick-and-place area.

15. The pick-and-place method according to claim 14, further comprising configuring a pick-and-place system before generating the three-dimensional image; the pick-and-place system includes a device for automatically generating an image and annotating a pick-and-place area in the image, a system camera, a robotic arm, and a controller; the device includes a generator, a device camera, and an annotator; wherein, The generator is used to generate the three-dimensional image under the generation background condition; the device camera is used to capture the two-dimensional image of the three-dimensional image; the annotator is used to identify the object area of the two-dimensional object image; the system camera is used to capture the two-dimensional image of the physical object; the controller is electrically connected to the device and is used to analyze the two-dimensional object image and obtain the second pick-and-place area of the two-dimensional object image according to the information of the first pick-and-place area obtained by the device; and control the robotic arm to pick and place the physical object corresponding to the pick-and-place part of the second pick-and-place area.

16. A computer program product, loaded in a device for automatically generating a pick-and-place area in an image and annotating the image, to execute a method for automatically generating a pick-and-place area in an image and annotating the image, the method comprising: Generating a three-dimensional image under a generation background condition, the three-dimensional image including at least one three-dimensional object image; Capturing a two-dimensional image of the three-dimensional image, the two-dimensional image including a two-dimensional object image of the three-dimensional object image; Identifying the object area of the two-dimensional object image; Obtaining an exposure ratio of an exposure area of the two-dimensional object image to an object area of the object area; Determining whether the exposure ratio is greater than a preset ratio; And When the exposure ratio is greater than the preset ratio, defining the exposure area as a pick-and-place area; Wherein the steps of generating the three-dimensional image, capturing the two-dimensional image of the three-dimensional image, identifying the object area of the two-dimensional object image, obtaining the exposure ratio, determining whether the exposure ratio is greater than the preset ratio, and defining the exposure area as the pick-and-place area are automatically completed by the device.

Images