Image processing device

Abstract

The present invention provides technology with which it is possible to accurately detect an object from an image by instance segmentation processing in which an appropriate division number corresponding to a state such as the shape of the object and the use of detection processing is set. An image processing device 10 includes: a detector 11 that cuts out a region 31 including a workpiece W as an object from an image 30, and performs determination processing for identifying an object W to be detected for each divided region obtained by dividing the region 31 into a predetermined number of divisions so as to execute instance segmentation processing for identifying the workpiece W; and a division-number-setting unit 14 that can change the number of divisions set to the detector 11 on the basis of a preset criteria.

Description

Image Processing Device

[0001] The present disclosure relates to an image processing device.

[0002] In the past, when detecting an object from an image, a technique has been known in which an image including the object is divided into multiple parts to detect and display the object. Techniques of this type are described in, for example, Patent Document 1 and Patent Document 2.

[0003] JP 2021-528757 A JP 2009-207677 A

[0004] In the instance segmentation process, which detects an image containing an object by distinguishing it into instances on a pixel-by-pixel basis, a two-stage detection process is sometimes employed. In the first stage, an area where an object is likely to exist is inferred as a rectangle or other shape, and the rectangular area is cut out from the image. In the second stage, the cut-out area is divided, and the object is identified by determining whether each area corresponds to an instance to be detected.

[0005] Because the number of divisions into the rectangular image is preset, depending on the shape of the object to be detected, the set number of divisions may not be sufficient, resulting in low detection accuracy. For example, a 14 x 14 division number can be used to detect a workpiece with a simple shape relatively accurately, but it may not be possible to detect a workpiece with a relatively complex shape accurately. Increasing the number of divisions is one way to solve this problem. However, increasing the number of divisions increases the detection time and memory usage, thereby increasing the processing load. Also, in applications such as detecting objects from an image in order to remove the detected workpiece with a magnetic hand, it may be sufficient to be able to detect the approximate position of the workpiece.

[0006] The present disclosure has been made in consideration of the above-mentioned problems, and aims to provide a technology that can accurately detect objects from images using instance segmentation processing in which an appropriate number of divisions is set according to circumstances such as the shape of the object and the purpose of the detection processing.

[0007] The present disclosure relates to an image processing device that includes a detector that performs an instance segmentation process to identify an object by cutting out an area containing an object from an image and dividing the area into a predetermined number of divisions, thereby performing a determination process to identify the object to be detected for each divided area, and a division number setting unit that can change the division number set in the detector based on a predetermined criterion.

[0008] According to the present disclosure, it is possible to provide a technology that can accurately detect an object from an image by using an instance segmentation process in which an appropriate number of divisions is set according to circumstances such as the shape of the object and the purpose of the detection process.

[0009] FIG. 1 is a schematic diagram showing a configuration of a robot control system to which an image processing device according to an embodiment of the present disclosure is applied; FIG. 2 is a functional block diagram of an image processing device according to an embodiment of the present disclosure; FIG. 3 is a schematic diagram illustrating a process of cutting out a region including a detection target in a first detection process; FIG. 4 is a schematic diagram illustrating an example of division of a region including an object to be detected when the number of divisions is set to be small; FIG. 5 is a schematic diagram illustrating an example of division of a region including an object to be detected when the number of divisions is set to be large; FIG. 6 is a diagram illustrating an example of an image of a workpiece having a simple shape captured at various angles; FIG. 7 is a diagram illustrating an example of an image of a workpiece having a complex shape captured at various angles; FIG. 8 is a schematic diagram illustrating an example of annotation data of a workpiece having a simple shape; FIG. 9 is a schematic diagram illustrating a learning process for detecting an object; FIG. 10 is a flowchart illustrating an example of the flow of object detection processing by an image processing device according to an embodiment of the present disclosure;

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0011] First, a description will be given of the overall configuration of a robot control system 1 to which an image processing device 10 is applied. Fig. 1 is a schematic diagram showing the configuration of a robot control system 1 to which an image processing device 10 according to an embodiment of the present disclosure is applied.

[0012] As shown in FIG. 1, the robot control system 1 includes a robot 2, an input device 3, a display device 4, a control device 5, an imaging device 8, and an image processing device 10 as main components.

[0013] The robot 2 is, for example, a multi-joint robot such as a six-axis vertical multi-joint robot or a four-axis vertical multi-joint robot. The robot 2 is electrically connected to a control device 5 and operates based on information input from the control device 5. Note that the robot 2 is not limited to a multi-joint robot, and may be a Cartesian coordinate robot, a SCARA robot, a parallel link robot, or the like.

[0014] The robot 2 performs an operation of, for example, gripping a workpiece W placed on a table 6 with a gripping device 7 such as a hand or a chuck device, and moving it to a predetermined location under the control of the control device 5.

[0015] The input device 3 is an interface that receives information about the work performed by the robot 2 and the control of the robot 2. The input device 3 is configured with, for example, a keyboard, a touch panel, etc., and outputs input information based on an input operation by a user to the control device 5.

[0016] The display device 4 is an output device that outputs information related to the work performed by the robot 2 and information related to the control of the robot 2. The display device 4 is configured, for example, by a display, a display with a speaker function, etc., and displays an image based on information input from the control device 5. The display device 4 may be configured by a touch panel display that is configured integrally with the input device 3.

[0017] The control device 5 is configured using, for example, a computer including memories such as ROM (read only memory) and RAM (random access memory), a CPU (control processing unit), and a communication control unit, all connected to one another via a bus. The functions and operations of each functional unit of the control device 5 described below are achieved by the cooperation of the CPU and memory installed in the computer and the control program stored in the memory. The control device 5 may also be configured to control the robot 2 based on information input from an external computer.

[0018] The imaging device 8 is a camera that captures an image including a workpiece W as an object on which the robot 2 performs a machining operation or the like.

[0019] The image processing device 10 acquires the image captured by the imaging device 8 and executes a detection process to identify an object from the image. This detection process identifies the state of the object, including its position, orientation, etc. The image processing device 10 also controls the imaging of the imaging device 8. The result of the object detection process by the image processing device 10 is transmitted to the control device 5. The control device 5 controls the robot 2 based on the detection result input from the image processing device 10.

[0020] Next, the detection process of the image processing device 10 will be described with reference to Fig. 2. Fig. 2 is a functional block diagram of the image processing device 10 according to an embodiment of the present disclosure.

[0021] As shown in FIG. 2, the image processing device 10 includes a detector 11, a division number setting unit 14, a display processing unit 15, and an input processing unit 16 as functional units executed on a processor.

[0022] The detector 11 identifies a region in the image, and performs instance segmentation processing to identify the object by dividing the identified region, thereby identifying the workpiece W in the image. The detector 11 detects the workpiece W as an object by using a neural network such as Mask R-CNN (Mask Region-based Convolutional Neural Network), for example.

[0023] The detector 11 of this embodiment has a first detection processing unit 12 and a second detection processing unit 13. Identification of the workpiece W as an object is performed in stages by the first detection processing unit 12 and the second detection processing unit 13.

[0024] The first detection process performed by the first detection processing unit 12 will be described with reference to Fig. 3. Fig. 3 is a schematic diagram illustrating the process of cutting out a region 31 including the detection target in the first detection process. As shown in Fig. 3, the first detection processing unit 12 performs a process of cutting out a rectangular region 31 including the workpiece W1 to be detected from an image 30 captured by the imaging device 8.

[0025] The first detection processing unit 12 performs a process of cutting out a region 31 from an image 30 using, for example, a Region Proposal Network (RPN) that predicts a bounding box of an object proposal based on an anchor box, a Selective Search algorithm, or the like.

[0026] The second detection process of the second detection processing unit 13 will be described. In the second detection process, the region 31 cut out by the first detection processing unit 12 is divided based on the division number set by the division number setting unit 14 described later, and a determination process is performed for each divided area to identify the object. In the determination process performed for each area, for example, if the output of an activation function obtained by a prior learning process using a neural network is equal to or greater than a threshold, the area is determined to be a specific part of the workpiece W to be determined. The second detection process can be realized using known techniques such as RoI Align or RoI Align.

[0027] The division number setting unit 14 sets the number of divisions to segment (divide) the area 31 cut out by the first detection processing unit 12 according to the workpiece W to be detected. The division number according to the workpiece W to be detected is set based on the workpiece W to be detected and the reference information stored in the storage unit 20.

[0028] The reference information is information indicating a criterion for determining the number of divisions set for each workpiece W. The reference information is, for example, the complexity indicating the complexity of the shape of the workpiece W, the size of the region 31 cut out by the first detection processing unit 12, user setting information, the operation of the robot 2, or a combination thereof.

[0029] When the number of divisions is set based on the complexity, the division number setting unit 14 changes the number of divisions according to the level of complexity. For example, if the complexity is high, the number of divisions is set to be large, and if the complexity is low, the number of divisions is set to be small. The complexity is set for each work W to be detected (work W1 or work W2). The method for setting the complexity will be described later.

[0030] When the number of divisions is set based on the size of the region 31, the division number setting unit 14 changes the number of divisions according to the size of the region 31. The region 31 is the area of ​​the portion cut out by the first detection processing unit 12. The division number setting unit 14 sets the number of divisions according to the area of ​​the region 31. For example, if the size of the region 31 is small, the number of divisions is set to be small because jaggies are not noticeable, and if the size of the region 31 is large, the number of divisions is set to be large because jaggies are noticeable. Note that the number of divisions according to the area is determined based on region determination information such as a preset threshold value, a table, a calculation formula, etc.

[0031] When the number of divisions is set based on the user's setting information, the division number setting unit 14 sets the number of divisions based on division number specifying information for specifying the number of divisions included in the user's setting information. The division number specifying information may be the number of divisions itself specified by the user via the input device 3 or the like, or may be information such as detection accuracy that indirectly indicates the degree of the number of divisions. The user can perform detection processing with an appropriate number of divisions by setting the user's setting information depending on the shape of the workpiece W, the purpose of the detection processing, etc. The user's specified information can set the number of divisions corresponding to each type of workpiece W.

[0032] When the number of divisions is set based on the operation of the robot 2, the division number setting unit 14 acquires operation specification information for specifying the operation of the robot 2 from the control device 5, and sets the number of divisions based on the acquired operation specification information. For example, when the operation of the robot 2 does not require high detection accuracy, such as fixing the workpiece W with a magnet or suction operation, the division number setting unit 14 sets the number of divisions to be low. On the other hand, when the operation of the robot 2 requires high detection accuracy, such as gripping a predetermined part of the workpiece W with a claw or the like, the division number setting unit 14 sets the number of divisions to be high.

[0033] Note that the setting of the number of divisions for the operation of the robot 2 here is merely an example. For example, if high-precision detection is required even for magnet or suction operation, the number of divisions can be set to a larger number. In this way, the number of divisions set according to the operation of the robot 2 can be set arbitrarily.

[0034] The division number setting unit 14 may be configured to allow the user to select each piece of reference information as a rule for determining the division number, or may be configured to automatically set the division number based on preset reference information.Furthermore, the division number setting unit 14 may be configured to determine the division number by combining each piece of reference information.

[0035] 4 and 5, an example will be described in which different division numbers are set by the division number setting unit 14 even in the same image processing device 10. FIG. 4 is a schematic diagram showing an example of division of an area 31 including an object to be detected when the division number is set to a small number. In FIG. 4, a workpiece W1 having a relatively simple shape is shown as the object to be detected. The number of divisions is set to a small number in accordance with the simple shape of the workpiece W1.

[0036] On the other hand, Fig. 5 is a schematic diagram showing an example of division of the region 31 including the object to be detected when the number of divisions is set to be large. Fig. 5 shows a workpiece W2 as the object to be detected, which has a more complex shape than the workpiece W1 in Fig. 4. The number of divisions is set to be large in accordance with the complex shape of the workpiece W2.

[0037] The display processing unit 15 executes processing for displaying information related to the number of divisions. The information related to the number of divisions is, for example, information on the detection accuracy corresponding to the number of divisions, the number of divisions, etc. The detection accuracy may be indicated by a level such as a numerical value, by text such as high accuracy, normal accuracy, low accuracy, etc., or by a symbol, etc.

[0038] The display processing unit 15 transmits to the display device 4 information for displaying on the display device 4 the detection accuracy corresponding to the number of divisions set by the division number setting unit 14. The display processing unit 15 may cause the display device 4 to display the detection accuracy as information relating to the number of divisions together with the detection result of the workpiece W. The information to be displayed on the display device 4 may be transmitted to the display device 4 via the control device 5 or may be transmitted directly to the display device 4.

[0039] The input processing unit 16 receives user setting information input by the user. The user setting information received by the input processing unit 16 is stored in the storage unit 20 as the above-mentioned reference information.

[0040] As described above, the detector 11 of this embodiment uses a neural network to execute a process for identifying the workpiece W as an object from the image 30. Next, a learning process that is performed in advance when complexity is used will be described.

[0041] In the learning process, first, an image of the workpiece W to be detected is taken using the imaging device 8 or another imaging device. FIG. 6 is a diagram showing an example of images of a workpiece W1 having a simple shape taken at various angles. FIG. 7 is a diagram showing an example of images of a workpiece W2 having a complex shape taken at various angles. As shown in FIGS. 6 and 7, images of the workpiece W1 and workpiece W2 to be detected are taken at various angles. Furthermore, imaging may be performed not only at different angles but also to include images of the same workpiece W1 and workpiece W2 that are different in size.

[0042] Next, annotation data instructing the position of an object to be detected is created for the captured image 30. Fig. 8 is a schematic diagram showing an example of annotation data for a workpiece W1 having a simple shape. Fig. 9 is a schematic diagram showing an example of annotation data for a workpiece W2 having a complex shape. As shown in Figs. 8 and 9, the annotation data is, for example, a group of vertices of a polygon that surrounds the workpiece W (workpiece W1 or workpiece W2) as an object.

[0043] 10 is a schematic diagram illustrating a learning process for detecting an object. As shown in FIG. 10, an image 30 and annotation data corresponding to this image 30 are input to the detector 11 as a data set, and supervised learning is performed on the workpiece W.

[0044] Furthermore, the image processing device 10 sets a complexity level indicating the degree of complexity of the object shape based on the input annotation data. The complexity level is automatically determined using the number of vertices in the annotation data, the area ratio between the convex hull created from the annotation data and the polygons in the annotation data, etc. The number of vertices is 9 for the work W1 in FIG. 8 and 52 for the work W2 in FIG. 9. The convex hull is the smallest convex polygon (convex polyhedron) that contains all the vertices acquired from the image.

[0045] 6 and 7, the complexity of the shape of the workpiece W1 or workpiece W2 varies depending on the imaging angle. Therefore, a representative value such as the average value of the workpiece W (workpiece W1 or workpiece W2) included in the learning data is used as the complexity of the shape of the workpiece W1 or workpiece W2. Note that the complexity is not limited to the average value. The complexity may be calculated using a calculation method other than the average value, or may be calculated using the area ratio between the convex hull and the polygon of the annotation data, as described above.

[0046] A learning process is performed for each work W to be segmented, dividing the region 31 according to the degree of complexity of the object shape. This constructs a learning model that predicts the object (work W1) in the image. A similar learning process is performed for work W2. Then, the complexity level indicating the degree of complexity for each work W is recorded in the storage unit 20.

[0047] Next, a detection process using a learning process will be described with reference to Fig. 11. Fig. 11 is a flowchart showing an example of the flow of a detection process for an object (workpiece W) performed by the image processing device 10 according to an embodiment of the present disclosure.

[0048] In step S10 , an image 30 including the workpiece W captured by the imaging device 8 is input to the image processing device 10 .

[0049] In step S11 , the first detection processing unit 12 executes a process of cutting out an area 31 including the workpiece W from the image 30 .

[0050] In step S12, the division number setting unit 14 predicts the shape of an object included in the cut-out region 31. For example, the division number setting unit 14 predicts that the object in region 31 is workpiece W1 or workpiece W2 using a learning model that has been learned in advance.

[0051] In step S13, the division number setting unit 14 sets the division number to be used in the detection process of the object to be detected based on preset reference information. If the reference information is complexity, the division number setting unit 14 obtains the complexity of the workpiece W stored in the storage unit 20 and sets the division number according to the complexity.

[0052] In step S14 , the second detection processing unit 13 executes a segmentation process to divide the region 31 according to the division number set by the division number setting unit 14 .

[0053] In step S15, the second detection processing unit 13 performs an object determination process in each region divided by the segmentation process, and identifies the object based on the result of the determination process.

[0054] In step S16, the display processing unit 15 executes a process of displaying information indicating the detection accuracy corresponding to the division number set by the division number setting unit 14 on the display device 4 together with the detection result of the workpiece W.

[0055] As described above, the image processing device 10 of this embodiment comprises a detector 11 that performs an instance segmentation process to identify the work W by cutting out an area 31 containing a work W as an object from an image 30 and performing a judgment process to identify the work W to be detected for each divided area obtained by dividing the area 31 into a predetermined number of divisions, and a division number setting unit 14 that can change the division number set in the detector 11 based on a predetermined criterion.

[0056] This allows accurate detection of objects such as the workpiece W from the image 30 through instance segmentation processing in which an appropriate number of divisions is set according to the object's shape, the purpose of the detection processing, and other circumstances. Therefore, detection accuracy can be improved only when necessary, avoiding unnecessary increases in detection time. In conventional technology, the number of divisions is static and cannot be changed during the detection processing of the workpiece W, but the configuration of this embodiment makes it possible to dynamically change the number of divisions according to the type and size of the detected workpiece W.

[0057] Furthermore, in this embodiment, the division number setting unit 14 sets the division number using, as a criterion, information indicating the degree of complexity of the shape of the workpiece W. As a result, the number of divisions can be increased for a workpiece W2 with a complex shape to perform accurate detection processing, and the number of divisions can be reduced for a workpiece W1 with a simple shape to shorten processing time and reduce processing load. Furthermore, because an appropriate number of divisions according to the degree of complexity of the workpiece W to be detected is automatically set, the user does not need to perform tasks such as changing the number of divisions.

[0058] Furthermore, in this embodiment, the division number setting unit 14 sets the division number using the size of the region 31 cut out by the first detection processing unit 12 of the detector 11 as a reference. As a result, an appropriate division number is automatically set taking into account the influence of jaggies, which varies depending on the size of the region 31. For example, if the region 31 is small, the jaggies are not noticeable so the division number is set to be small, which can reduce the processing time and the processing load. On the other hand, if the region 31 is large, the jaggies are noticeable so the division number is set to be large, which can suppress the influence of jaggies and enable accurate detection of the workpiece W.

[0059] Furthermore, in this embodiment, the division number setting unit 14 sets the division number based on information set by the user. As a result, when high accuracy is not required for the operation of the robot 2 or machine tool, the number of divisions can be set to a small number even if the shape of the workpiece W2 is complex, thereby shortening the processing time and reducing the processing load. Furthermore, when high-accuracy detection processing is required even if the shape of the workpiece W1 is simple, the number of divisions can be set to a large number, allowing high-accuracy detection processing of the workpiece W2.

[0060] Furthermore, in this embodiment, the division number setting unit 14 sets the division number based on the movement of the robot 2. As a result, an appropriate division number is automatically set in accordance with various movements of the robot 2 that require different detection accuracies.

[0061] The image processing device 10 of this embodiment further includes a display processing unit 15 that performs processing to display information about the detection accuracy corresponding to the number of divisions set by the division number setting unit 14. This allows the user to manage the work of the robot 2, the work of the machine tool, etc. while understanding the detection accuracy that changes depending on the number of divisions.

[0062] In the above embodiment, the image processing device 10 is applied to the robot control system 1, but the present invention is not limited to this configuration. For example, the image processing of the image processing device 10 can be applied to the machining of a machine tool. In this case, a numerical control device can be used as the control device for controlling the machine tool.

[0063] Furthermore, in the above embodiment, an example in which a neural network is used has been described as an embodiment, but the present disclosure can also be applied to image processing that does not involve a neural network or machine learning.

[0064] The above-described series of processes can be executed by hardware or software. In other words, the above-described functional configuration is merely exemplary and is not particularly limited. That is, it is sufficient for a computer to have the functionality to execute the above-described series of processes as a whole, and the functional blocks used to realize these functions are not particularly limited to the above-described example. Furthermore, the locations of the functional blocks are also not particularly limited and may be arbitrary. For example, one functional block may be configured as a single piece of hardware, a single piece of software, or a combination thereof. When the series of processes are executed by software, the program that constitutes the software is installed into a computer or the like from a network or a recording medium. The computer may be a computer built into dedicated hardware. Furthermore, the computer may be a computer capable of executing various functions by installing various programs, such as a server, a general-purpose smartphone, or a personal computer.

[0065] A recording medium containing such a program may be constituted not only by a removable medium (not shown) that is distributed separately from the device main body in order to provide the program to users, but also by a recording medium that is provided to users in a state where it is pre-installed in the device main body. Since the program can be distributed via a network, the recording medium may be installed in or accessible from a computer that is connected to or connectable to the network. Furthermore, the steps that describe the program recorded on the recording medium include not only processes that are performed chronologically in accordance with the order in which they are written, but also processes that are not necessarily processed chronologically but are executed in parallel or individually.

[0066] Although the present disclosure has been described in detail, the present disclosure is not limited to the individual embodiments described above. Various additions, substitutions, modifications, partial deletions, etc. are possible in these embodiments without departing from the gist of the present disclosure or the spirit of the present disclosure derived from the content of the claims and their equivalents. These embodiments can also be implemented in combination. For example, in the above-described embodiments, the order of each operation and the order of each process are shown as examples and are not limited to these. The same applies when numerical values ​​or mathematical expressions are used in the description of the above-described embodiments.

[0067] The following supplementary notes are further disclosed regarding the above-described embodiment and modified examples: (Supplementary Note 1) An image processing device comprising: a detector (11) that performs instance segmentation processing to identify objects (W, W1, W2) by cutting out an area (31) including objects (W, W1, W2) from an image (30) and dividing the area (31) into a predetermined number of divisions, and performing a determination processing to identify the object (W, W1, W2) to be detected for each divided area; and a division number setting unit (14) that can change the division number set in the detector (11) based on a preset criterion.

[0068] (Supplementary Note 2) In the image processing device (10), the division number setting unit (14) sets the division number using information indicating the degree of complexity of the shape of the object (W, W1, W2) as the criterion.

[0069] (Supplementary Note 3) In the image processing device (10), the division number setting unit (14) sets the division number using the size of the region cut out by the detector as the criterion.

[0070] (Supplementary Note 4) In the image processing device (10), the division number setting unit (14) sets the division number based on information set by a user.

[0071] (Supplementary Note 5) In the image processing device (10), the division number setting unit (14) sets the division number based on the movement of the robot (2).

[0072] (Supplementary Note 6) The image processing device (10) described above further comprises a display processing unit (15) that performs processing to display information related to the detection accuracy corresponding to the division number set by the division number setting unit (14).

[0073] REFERENCE SIGNS LIST 1 Robot control system 2 Robot 10 Image processing device 11 Detector 12 First detection processing unit 13 Second detection processing unit 14 Division number setting unit 15 Display processing unit 16 Input processing unit

Claims

1. An image processing device comprising: a detector that performs instance segmentation processing to identify an object by cutting out an area including an object from an image and dividing the area into a predetermined number of divisions, thereby performing a determination process to identify the object to be detected for each divided area; and a division number setting unit that can change the division number set in the detector based on a predetermined criterion.

2. The image processing device according to claim 1, wherein the division number setting unit sets the division number using information indicating the degree of complexity of the shape of the object as the criterion.

3. The image processing device according to claim 1, wherein the division number setting unit sets the division number using the size of the region cut out by the detector as the criterion.

4. The image processing device according to claim 1, wherein the division number setting section sets the division number based on information set by a user.

5. The image processing device according to claim 1, wherein the division number setting unit sets the division number based on the movement of a robot.

6. The image processing device according to any one of claims 1 to 5, further comprising a display processing unit that performs processing to display information relating to the detection accuracy corresponding to the division number set by the division number setting unit.

Images