Workpiece detection device, workpiece detection method, workpiece detection system, and workpiece detection program

Abstract

To provide a workpiece detection device and a method that correctly and accurately detect a position of workpiece even if a part of the workpiece is hidden in a loading area.SOLUTION: A workpiece detection device detects an auxiliary device captured in a picked-up image such that an actual region of the auxiliary device in a loading area can be removed from the picked-up image, with the picked-up image from which the actual region is removed as a search range, executes first matching between an image of workpiece captured in the search range and a plurality of workpiece models consisting of shape data representing a two-dimensional shape of workpiece stored in advance to detect an existence position of the workpiece in the loading area, for the shape data of the work models matched with the image of the workpiece whose existence position becomes clear in the first matching, edits a line segment representing a contour component overlapping a region covering the workpiece in the actual region to generate new shape data, and executes second matching between a workpiece model indicated by the new shape data and the image of the workpiece to specify an actual loading position of the workpiece on the loading area.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a workpiece detection device, a workpiece detection method, a workpiece detection system, and a workpiece detection program. [Background technology]

[0002] Conventionally, a loading device such as a supply robot supplies a plurality of workpieces such as plate-shaped members loaded in a loading area to a processing machine (bending machine) such as a press brake that performs bending processing, etc. During loading, a workpiece flipping device (one-piece workpiece pick-up device) such as a magnet floater that uses magnetic force to float a workpiece and separate it from other workpieces, or an air separator that uses air injection pressure to separate a workpiece from other workpieces, is used.

[0003] Usually, a work turning device such as an air separator is fixed to a mounting table installed in a position adjacent to the loading area. Therefore, a plurality of works loaded on the loading area are placed in one place, and one work turning device is arranged to correspond to these works.

[0004] On the other hand, for example, when the loading area is large, it is assumed that multiple different types of workpieces will be placed in multiple locations within the loading area. In such a case, if multiple workpiece turning devices corresponding to each workpiece are placed within the loading area, multiple workpieces and workpiece turning devices will be mixed within the loading area. Note that a workpiece detection device (see, for example, Patent Document 1) that detects the position of a workpiece placed within the loading area captures an image of the workpiece with a monocular camera and detects its position. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2018-120388 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, in a loading area where multiple workpieces and work-turning devices are mixed as described above, part of the work-turning device may be placed on the workpiece. In such cases, part of the workpiece will be hidden in the imaging range of the monocular camera, making it difficult to detect the position of the work accurately and with high precision.

[0007] The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide a work detection device, a work detection method, a work detection system, and a work detection program that can accurately and with high precision identify the position of the work and detect the work even if part of the work is hidden in the loading area. [Means for solving the problem]

[0008] The workpiece detection device according to the present invention is a workpiece detection device that detects a workpiece in a loading area on the basis of an image captured by an imaging device, the loading area being a loading area on which workpieces are loaded and in which a part of an auxiliary device having portability and freedom of placement for assisting in operations on the workpieces is placed on the workpieces, the workpiece detection device comprising: an apparatus detection unit that detects the auxiliary device shown in the captured image such that an actual area of ​​the auxiliary device in the loading area can be excluded from the captured image; a captured image from which the actual area has been excluded is set as a search range, and an image of the workpiece shown in this search range and a previously stored two-dimensional shape of the workpiece are detected; The system includes a first detection unit that detects the location of the workpiece within the loading area by performing a first match with a plurality of workpiece models consisting of shape data, a data creation unit that creates new shape data for the shape data of the workpiece model that matches the image of the workpiece whose location has been determined in the first match by editing lines that represent the contour components of the portion of the actual area that overlaps the area that covers the workpiece, and a second detection unit that identifies the actual loading position of the workpiece on the loading area by performing a second match between the workpiece model indicated by the new shape data and the image of the workpiece.

[0009] In one embodiment of the present invention, the data creation section performs editing to invalidate line segments representing the contour components of the shape data by dividing or erasing them.

[0010] In another embodiment of the present invention, the second detection unit performs the second matching by narrowing a search range more than the first matching performed by the first detection unit.

[0011] In still another embodiment of the present invention, the data creation section and the second detection section repeatedly create the new shape data and perform the second matching multiple times while gradually narrowing down the search range.

[0012] In yet another embodiment of the present invention, the second detection unit identifies a three-dimensional position including a height, a size, and a rotation angle of the workpiece on the loading area as the actual loading position.

[0013] In yet another embodiment of the present invention, the device detection unit detects an indicator that is provided at least on a top surface portion of a device body of the auxiliary device and is capable of identifying the position and orientation of the auxiliary device, matches the detected indicator with an indicator indicated by a pre-stored indicator model image, detects the placement position and placement orientation of the auxiliary device within the loading area, and calculates the actual area.

[0014] A workpiece detection method according to the present invention is a method for detecting a workpiece in a loading area, on the basis of an image captured by an imaging device, in which a part of an auxiliary device having a freely positionable portability for assisting in work on the workpiece is placed on the workpiece, the method including a step of detecting the auxiliary device shown in the captured image such that an actual area of ​​the auxiliary device in the loading area can be excluded from the captured image, a step of detecting the auxiliary device shown in the captured image such that an actual area of ​​the auxiliary device in the loading area can be excluded from the captured image, and a step of detecting an image of the workpiece shown in this search area and a two-dimensional shape of the workpiece stored in advance. The method includes a step of detecting the location of the work within the loading area by performing a first matching with a plurality of work models consisting of shape data representing the shape of the work; a step of creating new shape data for the shape data of the work models that matched in the first matching to the image of the work whose location has been determined by editing the lines representing the contour components of the portion of the actual area that overlaps the area that covers the work; and a step of performing a second matching between the work model indicated by the new shape data and the image of the work to identify the actual loading position of the work on the loading area.

[0015] In one embodiment of the present invention, in the step of creating the shape data, editing is performed to invalidate by dividing or erasing line segments that represent the contour components of the shape data.

[0016] In another embodiment of the present invention, in the step of specifying the actual loading position, the second matching is performed by narrowing a search range more than the first matching in the step of detecting the presence position.

[0017] In yet another embodiment of the present invention, in the process of creating the shape data and the process of identifying the actual loading position, creating the new two-dimensional shape data and performing the second matching are repeated multiple times in succession while gradually narrowing down the search range.

[0018] In yet another embodiment of the present invention, in the step of identifying the actual loading position, a three-dimensional position including the height, size, and rotation angle of the workpiece on the loading area is identified as the actual loading position.

[0019] In yet another embodiment of the present invention, in the process of detecting the auxiliary device, an indicator capable of identifying the position and orientation of the auxiliary device is detected, which is provided at least on the top surface portion of the main body of the auxiliary device, and the detected indicator is matched with an indicator indicated by a pre-stored indicator model image to detect the placement position and orientation of the auxiliary device within the loading area, and the actual area is calculated.

[0020] The workpiece detection system according to the present invention comprises an auxiliary device having portability and can be freely arranged to assist in work on a workpiece loaded in a loading area, an imaging device capable of imaging the loading area, and a workpiece detection device that detects the workpiece in the loading area, with a part of the auxiliary device arranged on the workpiece, based on an image captured by the imaging device. The workpiece detection device comprises a device detection unit that detects the auxiliary device shown in the image captured such that an actual area of ​​the auxiliary device in the loading area can be excluded from the image captured, and a search range is set for the image captured from which the actual area has been excluded, and a search range is set for an image of the workpiece shown in this search range. The system has a first detection unit that detects the location of the workpiece within the loading area by performing a first match with a plurality of workpiece models consisting of shape data representing the stored two-dimensional shape of the workpiece, a data creation unit that creates new shape data for the shape data of the workpiece model that matches the image of the workpiece whose location has been determined in the first match by editing lines that represent the contour components of the portion of the actual area that overlaps with the area that covers the workpiece, and a second detection unit that identifies the actual loading position of the workpiece on the loading area by performing a second match between the workpiece model indicated by the new shape data and the image of the workpiece.

[0021] In one embodiment of the present invention, the apparatus further includes a control device that controls the operation of the auxiliary device with respect to the detected workpiece based on a detection result from the workpiece detection device.

[0022] The present invention relates to a workpiece detection program for detecting a workpiece in a loading area, on the basis of an image captured by an imaging device, in which a workpiece is loaded and a part of an auxiliary device having portability and freedom of placement for assisting in work on the workpiece is placed on the workpiece. The workpiece detection program includes a step of causing a computer to detect the auxiliary device shown in the captured image so that an actual area of ​​the auxiliary device in the loading area can be excluded from the captured image, and a step of detecting the auxiliary device in the loading area so that an actual area of ​​the auxiliary device ... The method executes the steps of: detecting the location of the work within the loading area by performing a first matching with a plurality of work models consisting of shape data representing a two-dimensional shape; creating new shape data for the shape data of the work models that match the image of the work whose location has been determined in the first matching by editing the lines representing the contour components of the portion of the actual area that overlaps the area that covers the work; and identifying the actual loading position of the work on the loading area by performing a second matching between the work model indicated by the new shape data and the image of the work. Effect of the Invention

[0023] According to the present invention, the position of the workpiece can be identified accurately and with high precision and the workpiece can be detected even if part of the workpiece is hidden in the loading area. [Brief description of the drawings]

[0024] [Figure 1] 1 is a perspective view showing an outline of a configuration example of a workpiece detection system according to an embodiment of the present invention; [Diagram 2] 2 is a block diagram illustrating a schematic functional configuration of a workpiece detection device in the workpiece detection system. FIG. [Diagram 3] FIG. 2 is a configuration diagram illustrating a hardware configuration of the workpiece detection device. [Figure 4] 4 is a flowchart showing an example of a workpiece detection process in the workpiece detection system. [Diagram 5] 13 is a flowchart showing an example of a device detection process in the work detection process. [Figure 6] 3 is an image diagram conceptually showing an image captured by an imaging device of the workpiece detection system. FIG. [Figure 7] FIG. 2 is a perspective view showing an auxiliary device in the workpiece detection system. [Figure 8] 13 is a diagram showing an example of an indicator provided on the assisting device. FIG. [Figure 9] FIG. 13 is a diagram showing another example of the index. [Figure 10] 13 is an image diagram conceptually showing an actual area of ​​the auxiliary device in the captured image. FIG. [Figure 11] FIG. 13 is an image diagram for conceptually explaining a first matching using the captured image. [Figure 12] 13 is an image diagram for conceptually explaining an image of a workpiece in the captured image and a workpiece model matched within a search range. FIG. [Figure 13] FIG. 13 is an image diagram for conceptually explaining data creation and second matching using the captured image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] Hereinafter, a workpiece detection device, a workpiece detection method, a workpiece detection system, and a workpiece detection program according to the embodiments of the present invention will be described in detail with reference to the attached drawings. However, the following embodiments do not limit the invention according to each claim, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention. In addition, in the present embodiment, the scale and dimensions of each component may be exaggerated, and some components may be omitted.

[0026] Fig. 1 is a perspective view showing a schematic configuration example of a workpiece detection system 100 according to an embodiment of the present invention. As shown in Fig. 1, the workpiece detection system 100 according to this embodiment includes a workpiece turning device 10 as an auxiliary device that assists in operations (e.g., loading into a bending machine, etc.) on a workpiece W loaded in a loading area TA on a pallet P, and a camera 20 as an imaging device that can image the loading area TA.

[0027] In addition, the work detection system 100 has functions such as an NC device that controls the entire work detection system 100, and also has functions as an image processing device that performs general image processing such as image analysis and image correction based on the image captured by the camera 20, and is equipped with a work detection device 30 that identifies the actual loading position (actual loading position) of the work W on the loading area TA and detects the work.

[0028] In this embodiment, the work detection device 30 is configured to include a display 69 as a display device and an input unit 35a consisting of input devices such as a keyboard and a mouse, but is not limited to this as long as it can be replaced by something that has equivalent functions in place of the display 69 and the input unit 35a (for example, a display means or input means that can be used remotely).

[0029] In addition, in this embodiment, for example, processing machines such as bending machines and loading devices such as workpiece holding robots are not shown in the illustration, but these processing machines and loading devices, etc. may be included in the workpiece detection system 100 in a state where various operations can be controlled by the workpiece detection device 30.

[0030] In addition, in this embodiment, the work turning device 10 is taken as an example of the auxiliary device to be described, but the auxiliary device is not limited to this as long as it can assist in various operations on the work W. Furthermore, although not shown in the drawings, each of the work turning devices 10, the camera 20, and the work detection device 30 is connected to each other by wire or wirelessly so as to be able to transmit and receive data and signals.

[0031] The work turning device 10 is portable and can be freely placed in a plurality of locations inside and outside the loading area TA so as to correspond to each of the plurality of workpieces W placed in the loading area TA on the pallet P. In this embodiment, a plurality of work turning devices 10 can be placed in the loading area TA so as to correspond, for example, one-to-one with each workpiece W so that a portion of the workpiece is placed on the workpiece. The detailed configuration of this work turning device 10 will be described later.

[0032] The camera 20 is configured, for example, with an inexpensive and highly versatile monocular camera (i.e., one camera), and is disposed directly above the center of the loading area TA of the pallet P via a support member such as a camera stand 21 so that the entire loading area TA can be captured as an image capturing range. Note that with regard to capturing an image of the loading area TA on the pallet P by the camera 20, various calibrations of the captured image to be obtained are assumed to have been performed in advance.

[0033] Although not shown, lighting equipment having a plurality of light-emitting diodes (LEDs) or the like is provided at predetermined locations around the pallet P, which irradiates light onto the workpieces W loaded in the loading area TA. When the loading area TA is imaged by the camera 20, the brightness of the captured image can be appropriately adjusted with proper exposure by irradiating the workpieces W with illumination light from this lighting equipment.

[0034] On the other hand, the workpiece detection device 30 is functionally configured as follows. FIG. 2 is a block diagram showing generally the functional configuration of the workpiece detection device 30 in the workpiece detection system 100, and FIG.

[0035] 2, image data of an image captured by the camera 20 is input to a workpiece detection device 30. Functionally, the workpiece detection device 30 includes an image acquisition unit 31, an image processing unit 32, a calculation unit 33, a storage unit 34, and an operation unit 35. Detection of the workpiece W is mainly performed by the image processing unit 32 having a device detection unit 36, a first detection unit 37, a data creation unit 38, and a second detection unit 39.

[0036] Here, the image acquisition unit 31 acquires image data of a captured image output from the camera 20. The image processing unit 32 performs various image conversion and image analysis processes (hereinafter, such processes are collectively referred to as "image processing") on the image data acquired by the image acquisition unit 31, including, for example, binarization processing, morphological processing, approximation processing using the Newton method, matching processing such as template matching, and the like.

[0037] In addition, the image data (raster data) can be converted into numerical data (vector data) that can be expressed by linearly connected line segments through image processing by the image processing unit 32, in which the contour components of each object captured in the loading area TA, such as the work turner 10 and the work W, are extracted as feature values ​​(geometric shape data), and the matching described below can be performed between vector data unless otherwise specified, in order to reduce the processing load.

[0038] The device detection unit 36 ​​of the image processing unit 32 detects the work turner 10 appearing in the captured image indicated by the image data by calculating the actual area of ​​the work turner 10 within the loading area TA and detecting the work turner 10 so that it can be excluded from the captured image. The detection of the work turner 10 in the work detection system 100 will be described in detail later.

[0039] The first detection unit 37 uses the captured image from which the actual area of ​​the work turning device 10 is excluded by the device detection unit 36 ​​as a search range, and performs matching (first matching) between the image of the work W captured in this search range and a work model consisting of shape data representing the two-dimensional shapes of various workpieces W pre-stored in the storage unit 34. Then, the first detection unit 37 detects the presence position representing the approximate position of the workpiece W within the loading area TA.

[0040] The data creation unit 38 creates new shape data by editing the lines representing the contour components of the portion of the actual area of ​​the work turning device 10 that overlaps with the area (see FIG. 6) that covers the workpiece W, of the shape data of the workpiece model that matches the image of the workpiece W whose position has been determined by the first detection unit 37. Specifically, the data creation unit 38 performs editing so as to invalidate the lines representing the contour components of the shape data by dividing or erasing them.

[0041] Then, the second detection unit 39 performs matching (second matching) between the work model indicated by the new shape data created by the data creation unit 38 and the image of the work W, and detects the work W by identifying the actual loading position, which is the actual loading position of the work W on the loading area TA. Specifically, the second detection unit 39 identifies a three-dimensional position including the height of the work W on the loading area TA, the size of the work W, and the rotation angle of the work W as the actual loading position of the work W.

[0042] For example, the matching by the second detection unit 39 may be performed with a narrower search range than the matching by the first detection unit 37, in order to improve the detection accuracy (i.e., the search range is not the entire loading area TA, but an enlarged image of the workpiece W and the workpiece model). Also, the creation of shape data by the data creation unit 38 and the matching by the second detection unit 39 may be performed multiple times in succession while gradually narrowing the search range in order to further improve the detection accuracy.

[0043] The calculation unit 33 performs various calculation processes, such as calculations for controlling the operation of the work turning device 10 corresponding to the detected work W and calculations for controlling the entire work detection system 100, based on the detection results from the image processing unit 32, and outputs control to each unit. The storage unit 34 readably and writably stores various data used in the work detection device 30, such as CAD data (shape data) representing the work W and the work turning device 10 in two or three dimensions, various image data including model (work model) images serving as references for work W of various shapes, and various program data.

[0044] Note that the workpiece model image may be stored in a plurality of types for each workpiece W, such as images that appear different when captured by the camera 20 depending on the placement location, loading height, rotation, etc. of the workpiece W actually loaded on the loading area TA (e.g., images that appear tilted or distorted, etc.), and may be linked to the above-mentioned shape data (shape data whose dimensions are changed according to each image). Also, the operation unit 35 accepts operation input by the user of the workpiece detection device 30 via the input unit 35a.

[0045] 3, the workpiece detection device 30 includes, as a hardware configuration, for example, a CPU 61, a RAM 62, a ROM 63, an HDD (hard disk drive) 64, and an SSD (solid state drive) 65. The workpiece detection device 30 also includes an input I / F (interface) 66, an output I / F (interface) 67, and a communication I / F (interface) 68. The components 61 to 68 are connected to each other via a bus 60.

[0046] The CPU 61 controls the entire work detection system 100, including the work detection device 30, by executing various programs stored in the RAM 62, ROM 63, HDD 64, SSD 65, etc., and realizes the functions of the image processing unit 32 and the calculation unit 33 by executing the work detection program.

[0047] The RAM 62 can be used as a working area for the CPU 61. The ROM 63 stores at least the various programs in a readable manner. The HDD 64 and the SSD 65 store the various data described above in a readable and writable manner, and together with the RAM 62 and the ROM 64, realize the function of the storage unit 34.

[0048] The camera 20 is connected to the input I / F 66 to acquire a captured image. Thus, the input I / F 66 realizes the function of the image acquisition unit 31. At the same time, a touch panel 69a functioning as the input unit 35a of the operation unit 35 is connected to the input I / F 66 to receive information associated with an operation input from the user. Note that input means such as a keyboard and a mouse (not shown) may also be connected to the input I / F 66.

[0049] A display 69 of the work detection device 30 having a built-in touch panel 69a is connected to the output I / F 67, and various information displayed on the monitor is output. The work detection device 30 can be connected to a network such as the Internet (not shown) or an external device via the communication I / F 68.

[0050] The workpiece detection process in the workpiece detection system 100 including the workpiece detection device 30 configured as above is performed, for example, as follows. Fig. 4 is a flowchart showing an example of work detection processing in the work detection system 100, and Fig. 5 is a flowchart showing an example of device detection processing of the work turning device 10 in the work detection processing. Fig. 6 is an image diagram conceptually showing a captured image captured by the camera 20 of the work detection system 100, and Fig. 7 is a perspective view generally showing the work turning device 10 in the work detection system 100.

[0051] 4, when detecting the workpiece W, first, the camera 20 captures an image of the loading area TA where the workpiece W and the workpiece turning device 10 are loaded and arranged (step S100), and outputs image data of the captured image to the workpiece detection device 30. Next, the workpiece detection device 30 acquires image data of the captured image via the image acquisition unit 31, and performs image analysis on the image data as described above (step S101).

[0052] Here, as shown in FIG. 6, an image 22 captured by a camera 20 installed above a pallet P shows the work W loaded on a loading area TA, as well as a work turning device 10 having an apparatus main body 10A and a turning unit 10B (which will be described later), with, for example, a part of it (part of the turning unit 10B) overlapping a part of the work W.

[0053] Generally, even if one attempts to detect the work W using the captured image 22 in which the work W and the work turning device 10 are captured in this state, the part of the turning unit 10B that covers the work W obscures the contours and shape of the work W underneath, making it impossible to recognize the relevant part, and detection of the work W by matching frequently fails (misdetection).

[0054] Therefore, the workpiece detection device 30 of this embodiment executes a device detection process (step S102) for detecting the workpiece turning device 10 in the loading area TA in advance based on the captured image 22 after the image analysis in the above step S101. The device detection process in step S102 is performed in the device detection unit 36, for example, as follows.

[0055] As shown in Fig. 5, in the device detection process, first, for example, markers 14, 15 (see Fig. 6) provided on the work turning device 10 are detected by edge detection process or the like (step S110). Here, the markers 14, 15, etc. will be described. The markers 14, 15, etc. indicate the position and direction of the work turning device 10 in an identifiable manner.

[0056] As a premise, as shown in FIG. 7, the work turning device 10 comprises an apparatus main body 10A having, for example, a rectangular outer shape when viewed from above, and a turning unit 10B that is arranged in a cantilevered manner relative to the apparatus main body 10A so as to be movable relative to the apparatus main body 10A and to be in contact with and able to hold the work W.

[0057] The turning unit 10B has a function of turning over and separating the topmost work W of a plurality of plate-shaped works W loaded in the loading area TA on the pallet P from the other works W, for example, by air injection pressure and suction force. In this way, the work turning device 10 can be used as a one-piece taking-up device for taking up one work W by the turning unit 10B, but since the detailed configuration of the one-piece taking-up device is publicly known, a description thereof will be omitted here. In the work turning device 10 of this embodiment, the side where the turning unit 10B is arranged is defined as the front side of the device.

[0058] The device body 10A has a top surface 11 provided on the upper side of the device, a sloped surface 12 that slopes from the top surface 11 at two locations on the left and right sides of the front side of the device, and a pair of side surfaces 13 that connect the top surface 11 (and the sloped surface 12) to the left and right sides of the device. Markers 14, 15, and 16 are provided at predetermined locations on the top surface 11, the sloped surface 12, and the pair of side surfaces 13 of the device body 10A as indicators that can identify the position and direction of the work turning device 10.

[0059] Specifically, the markers 14 and 16 are provided in multiple locations on the top surface 11 and the pair of side surface portions 13 of the device body 10A, and the markers 15 are provided in multiple locations on the sloped surface portion 12 of the device body 10A. More specifically, the markers 14 are provided in two locations, left and right, near the rear side of the device on the top surface portion 11 of the device body 10A, and the markers 16 are provided in locations near the upper side of the device on the pair of side surface portions 13 of the device body 10A. In other words, the markers 16 are provided to the side and directly below the marker 14. The markers 15 are also provided in locations near both ends in the inclination direction on the two sloped surface portions 12 of the device body 10A.

[0060] Fig. 8 is a diagram showing an example of the markers 14 to 16 provided on the work turning device 10, and Fig. 9 is a diagram showing another example of the markers 14 to 16. The markers 14, 15, and 16 provided on the device body 10A of the work turning device 10 can be configured with AR (augmented reality) markers including, for example, the well-known ArUco marker and ChArUco marker.

[0061] As shown in Fig. 8, the marker 14 (15, 16) is made of a film-like or plate-like member having at least an entire area 18 that represents the entire marker, and an identification area 19 for identifying the position and direction of the marker. Note that the identification areas 19 of the markers 14 to 16 are not shown in Figs. 1, 6, 7, etc.

[0062] The shape of the identification area 19 of the markers 14 to 16 is, for example, a shape formed by combining simple rectangles in the example shown in Figure 8, but as shown in Figures 9(a) to 9(g), the shape is not limited to what is shown in the figures, and various shapes, such as more complex shapes or simpler shapes, can be adopted, such as combining multiple rectangles or squares, or changing the position of the identification area 19 within the overall area 18, as long as the shape can at least identify the position and direction of the work turning device 10 by itself.

[0063] The dimensions (size and shape of the entire area 18 and the identification area 19, etc.) of the markers 14 to 16 can be set according to the imaging performance of the camera 20, such as the number of pixels and imaging resolution, but as long as the markers can be detected so as to enable sufficient determination of the placement position and orientation of the work turning device 10 from the captured image 22 captured from above the loading area TA, it is sufficient to provide at least one marker on the top surface 11 of the device main body 10A. In other words, when the work turning device 10 is in the normal placement state, the top surface 11 is most likely to be captured in the captured image.

[0064] In addition, the markers 16 provided on a pair of side portions 13 of the device main body 10A are provided so that the position and orientation of the work turning device 10 (its position and orientation when it falls within the loading area TA) can be identified even if the work turning device 10 falls over due to some kind of accident (for example, a collision with a robot arm of a loading device not shown).

[0065] 5 again, once the markers 14, 15 are detected in step S110, position information indicating the pixel (picture element) positions of the four corners of the entire area 18 of each of the markers 14, 15 is calculated and stored in the storage unit 34 (step S111). Note that a pixel (picture element) refers to the smallest unit or smallest element having color information (color tone, gradation, etc.) when the captured image 22 is handled by the workpiece detection device 30.

[0066] Next, for each marker 14, 15, position information of the model position indicating the position of the markers 14, 15 actually placed on the work turning device 10 is obtained based on various data such as dimensional information of each part of the work turning device 10 indicated by the CAD data stored in the memory unit 34 and image data representing an image of the marker model (step S112).

[0067] Then, the device detection unit 36 ​​repeats the following processes of steps S113 to S116 multiple times (for example, five times). That is, the acquired model positions and the stored pixel positions are subjected to approximation processing such as Newton's method to calculate the physical three-dimensional positions (x, y, z positions) and rotations (rz) of the markers 14 and 15 within the loading area TA (step S113).

[0068] Next, after the three-dimensional position and rotation are calculated, a marker image is generated that represents the markers 14, 15 in a state that reflects the physical three-dimensional position (x, y, z position) and rotation (rz) based on the calculation result (step S114). Then, the markers 14, 15 represented by the generated marker image are searched for by, for example, performing template matching processing (step S115) with various marker model images that differ according to the three-dimensional position and rotation state, which are stored in advance in, for example, the storage unit 34. Based on the matching information (x, y, z position and rotation rz) of the markers 14, 15 represented by the searched marker model images, the position information representing the pixel positions of the markers 14, 15 stored in the storage unit 34 is updated (step S116).

[0069] After repeating steps S113 to S116 multiple times, it is determined whether the matching information has sufficiently converged (step S117), and if it is determined that the matching information has not sufficiently converged (No in step S117), the process proceeds to step S113 and the subsequent processes are repeated.

[0070] On the other hand, if it is determined that sufficient convergence has occurred (Yes in step S117), an actual area R (see FIG. 10) of the work turner 10 in the loading area TA is calculated based on various data stored in the storage unit 34 so that the work turner 10 can be excluded from the captured image 22, and the work turner 10 is detected (step S118). Note that FIG. 10 conceptually illustrates the actual area R of the work turner 10 in the captured image 22. Once the work turner 10 has been detected in this way, the device detection process according to this flowchart is terminated, and the process proceeds to step S103 in FIG. 4.

[0071] That is, in step S103, in order to avoid erroneous detection of the work W, the pixel range of the actual area R of the work turning device 10 is extracted in the image processing unit 32 from the captured image 22 showing the loading area TA, which is the matching search range, as shown in FIG. 10, and the extracted actual area R is excluded from the processing target (step S103).

[0072] 11, the first detection unit 37 sets the entire area (loading area TA) of the captured image 22 excluding the real area R as a search range SR, and performs a first matching (step S104) between the images of each workpiece W captured in this search range SR and a plurality of workpiece models Wm consisting of shape data representing the two-dimensional shape of the workpiece W pre-stored in, for example, the storage unit 34. The shape data of the workpiece models Wm is vector data defined by, for example, points (x, y coordinate points), line segments (lines connecting x, y coordinate points), and surfaces (line segments closed by lines connecting x, y coordinate points), and specifies the dimensions and shape of the workpiece W, such as length, width, length, and thickness.

[0073] 12, this first matching makes it possible to determine where in the search range SR the workpiece W corresponding to the workpiece model Wm is located (whether the workpiece W matches the workpiece model Wm) by performing calculations with reference to shape data previously linked to the image of the workpiece W and the workpiece model Wm matched within the search range SR in a database (DB) in the storage unit 34, for example. This makes it possible to grasp the approximate location of the workpiece W within the loading area TA. Note that in the following explanation including FIG. 12, the explanation focuses on one workpiece W and work turning device 10 among the multiple workpieces W and work turning devices 10 within the loading area TA, but it goes without saying that the same applies to the other workpieces W and work turning devices 10.

[0074] After the first matching, the following processing of steps S105 to S107 is repeated multiple times (for example, four times). That is, when the position of the workpiece W in the loading area TA is determined, the data creation unit 38 creates a workpiece model Wmr of new shape data by editing the line segments representing the contour components of the portion of the actual area R of the work turning device 10 that overlaps with the area 10BR (see FIG. 13) that covers the workpiece W for the shape data of the workpiece model Wm that matched the image of the workpiece W in the first matching (step S105).

[0075] 13, the data creation unit 38 reads out the shape data of the matched work model Wm from the DB, and edits the line segment L and line segments S1 to S4 that represent the contour components of the portion overlapping with the covered area 10BR of the work turning device 10 by, for example, dividing the line segment L into line segments L1 and L2 and erasing the line segments S1 to S4, thereby creating a work model Wmr of new shape data (deformed data) in which the line segments Ld and Sd are invalidated from the work model Wm. Then, the second detection unit 39 performs a second matching (step S106) between the work model Wmr indicated by the deformation data and the image of the work W, and narrows down the search range SR (for example, sets the search range SSR, which is more microscopic than the search range SR, as the search range) (step S107).

[0076] After repeating steps S105 to S107 multiple times, it is determined whether matching is complete, for example, by the workpiece W and the workpiece model Wmr being sufficiently matched down to the smallest detail (step S108). If it is determined that matching is not complete (No in step S108), the process proceeds to step S105 and the subsequent processes are repeated.

[0077] On the other hand, if it is determined that matching is completed (Yes in step S108), the second detection unit 39 can obtain accurate matching information (various information such as the loading position coordinates of the work W, loading height, loading dimensions, loading angle, etc.) in the search range SR through calculations by the image processing unit 32 based on the captured image 22 and various data in the memory unit 34, so it identifies the actual loading position, which is the actual loading position of the work W on the loading area TA (step S109), detects the work W, and terminates the processing according to this flowchart.

[0078] In this way, according to the workpiece detection system 100 of this embodiment, even if a part of the workpiece W is hidden from the imaging range of the camera 20 by the workpiece turning device 10 arranged at a separate arbitrary location in the loading area TA, the approximate position of the workpiece W in the loading area TA is grasped by matching. Furthermore, the shape data of the matched workpiece model Wm is edited taking into account the hidden part and then matched with the image of the workpiece, for example, multiple times while narrowing the search range. Then, the actual loading position of the workpiece W on the loading area TA is specified by performing calculations based on the result and various data such as CAD data stored in advance. This makes it possible to accurately and highly precisely specify the loading position of the workpiece W in the loading area TA and detect the workpiece W.

[0079] Although the embodiment of the present invention has been described above, this embodiment is presented as an example and is not intended to limit the scope of the invention. This new embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0080] 10 Work turning device (auxiliary device) 10A Device body 10B Turning unit (movable part) 11 Top section 12 Slope 13 Side part 14,15,16 Markers 18 whole area 19 Identification Area 20 Camera (imaging device) 21 Camera Stand 22 Captured images 30 Workpiece detection device 100 Workpiece Inspection System P Palette R Reality domain SR, SSR search range TA Loading Area Double work Wm Work Model

Claims

1. A work detection device detects a work in a loading area based on an image captured by an imaging device, the loading area being a loading area on which a work is loaded and a part of an auxiliary device having a freely positionable portability for assisting a task on the work is placed on the work, the work being detected based on an image captured by an imaging device, a device detection unit that detects the auxiliary device shown in the captured image such that an actual area of ​​the auxiliary device within the loading area can be excluded from the captured image; a first detection unit that detects the position of the workpiece within the loading area by performing a first matching between a captured image excluding the actual area as a search range and a plurality of workpiece models consisting of shape data representing a two-dimensional shape of the workpiece that is stored in advance; a data creation unit that creates new shape data by editing line segments that represent contour components of a portion of the actual area that overlaps with an area that covers the workpiece, for shape data of the workpiece model that has been matched in the first matching to the image of the workpiece whose position has been determined; a second detection unit that performs a second matching between a workpiece model indicated by the new shape data and an image of the workpiece to identify an actual loading position of the workpiece on the loading area. Workpiece detection device.

2. The data creation unit performs editing to divide or erase the line segments representing the contour components of the shape data to invalidate them. The workpiece detection device according to claim 1.

3. The second detection unit performs the second matching by narrowing a search range more than the first matching performed by the first detection unit.

3. The workpiece detection device according to claim 1 or 2.

4. The data creation unit and the second detection unit repeatedly create the new shape data and perform the second matching a plurality of times while gradually narrowing down the search range.

3. The workpiece detection device according to claim 1 or 2.

5. The second detection unit identifies a three-dimensional position including a height, a size, and a rotation angle of the workpiece on the loading area as the actual loading position. The workpiece detection device according to any one of claims 1 to 4.

6. The device detection unit detects an index that is provided at least on a top surface of a device body of the auxiliary device and that is capable of identifying a position and orientation of the auxiliary device, matches the detected index with an index indicated by a pre-stored index model image, detects a placement position and a placement orientation of the auxiliary device within the loading area, and calculates the actual area. The workpiece detection device according to any one of claims 1 to 5.

7. A method for detecting a workpiece in a loading area, in which a workpiece is loaded and a part of an auxiliary device having a freely positionable portability for assisting a task on the workpiece is placed on the workpiece, based on an image captured by an imaging device, the method comprising: detecting the auxiliary device shown in the captured image such that an actual area of ​​the auxiliary device within the loading area can be excluded from the captured image; A process of detecting the position of the workpiece within the loading area by performing a first matching between the captured image excluding the actual area as a search range and a plurality of workpiece models consisting of shape data representing a two-dimensional shape of the workpiece that is stored in advance, and A process of creating new shape data by editing line segments representing contour components of a portion of the actual area that overlaps with an area that covers the workpiece, for the shape data of the workpiece model that has been matched in the first matching to the image of the workpiece whose position has been determined; and performing a second matching between the workpiece model indicated by the new shape data and the image of the workpiece to identify the actual loading position of the workpiece on the loading area. Workpiece detection method.

8. In the step of creating the shape data, editing is performed to invalidate the line segments representing the contour components of the shape data by dividing or erasing them. The workpiece detection method according to claim 7.

9. In the step of specifying the actual loading position, the second matching is performed by narrowing a search range more than the first matching in the step of detecting the location. The workpiece detection method according to claim 7 or 8.

10. In the step of creating the shape data and the step of identifying the actual loading position, the step of creating the new shape data and performing the second matching is repeated multiple times while narrowing down the search range step by step. The workpiece detection method according to claim 7 or 8.

11. In the step of identifying the actual loading position, a three-dimensional position including the height, size, and rotation angle of the workpiece on the loading area is identified as the actual loading position. The workpiece detection method according to any one of claims 7 to 10.

12. In the step of detecting the auxiliary device, an index capable of identifying a position and a direction of the auxiliary device, which is provided at least on a top surface of a main body of the auxiliary device, is detected, and the detected index is matched with an index indicated by a pre-stored index model image to detect a placement position and a placement direction of the auxiliary device in the loading area, and the actual area is calculated. The workpiece detection method according to any one of claims 7 to 11.

13. a portable auxiliary device that can be freely positioned to assist in work on the workpieces loaded in the loading area; An imaging device capable of imaging the loading area; A work detection device that detects the work in the loading area in which a part of the auxiliary device is placed on the work based on an image captured by the imaging device, The workpiece detection device is a device detection unit that detects the auxiliary device shown in the captured image such that an actual area of ​​the auxiliary device within the loading area can be excluded from the captured image; a first detection unit that detects the position of the workpiece within the loading area by performing a first matching between a captured image excluding the actual area as a search range and a plurality of workpiece models consisting of shape data representing a two-dimensional shape of the workpiece that is stored in advance; a data creation unit that creates new shape data by editing line segments that represent contour components of a portion of the actual area that overlaps with an area that covers the workpiece, for shape data of the workpiece model that has been matched in the first matching to the image of the workpiece whose position has been determined; a second detection unit that performs a second matching between a workpiece model indicated by the new shape data and an image of the workpiece to identify an actual loading position of the workpiece on the loading area. Workpiece detection system.

14. The apparatus further includes a control device that controls the operation of the auxiliary device with respect to the detected workpiece based on the detection result from the workpiece detection device. The workpiece detection system according to claim 13.

15. A workpiece detection program for detecting a workpiece in a loading area on the basis of an image captured by an imaging device, the loading area being a loading area on which a workpiece is loaded and a part of an auxiliary device having a freely positionable portability for assisting in a task performed on the workpiece is placed on the workpiece, the workpiece being detected based on an image captured by an imaging device, On the computer, detecting the auxiliary device shown in the captured image such that an actual area of ​​the auxiliary device within the loading area can be excluded from the captured image; A process of detecting the position of the workpiece within the loading area by performing a first matching between the captured image excluding the actual area as a search range and a plurality of workpiece models consisting of shape data representing a two-dimensional shape of the workpiece that is stored in advance, and A process of creating new shape data by editing line segments representing contour components of a portion of the actual area that overlaps with an area that covers the workpiece, for the shape data of the workpiece model that has been matched in the first matching to the image of the workpiece whose position has been determined; and performing a second matching between the workpiece model indicated by the new shape data and the image of the workpiece to specify an actual loading position of the workpiece on the loading area. Workpiece detection program.

Images