Parts counting device and robotic system

Abstract

A component count device (100) comprises: an oscillation unit (20) which oscillates a housing unit (2) which houses a plurality of components (1); a photographing unit (30) which continuously photographs the components in the housing unit being oscillated by the oscillation unit; and an information processing unit (41) which, on the basis of photographic images (50) sequentially photographed by the photographing unit, tracks the components being moved by oscillation and which counts the number of the components in the housing unit.

Description

Technical Field

[0001] This invention relates to a component counting device and a robot system, and particularly to a component counting device and a robot system for counting components.

Background Art

[0002] Conventionally, a component counting device for counting components has been known (see, for example, Patent Document 1).

[0003] In the above Patent Document 1, a component counting device for counting components is disclosed. This component counting device is configured to count components based on a captured image obtained by capturing components arranged on a sheet by a capturing device. Further, this component counting device is configured to display an image obtained by processing an image in which a portion where component overlap is detected is made distinguishable when components on the sheet overlap. When components overlap, it is impossible to accurately count the components. For this reason, the user visually recognizes the component overlap on the displayed image, manually eliminates the component overlap on the sheet, and causes the component counting device to execute the counting process again.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the component counting device described in Patent Document 1, since the user manually eliminates the component overlap and causes the counting process to be executed again, there is a disadvantage in that it takes time and effort for the user. For this reason, there is a problem that it is difficult to accurately count components while saving the user's time and effort.

[0006] This invention was made to solve the above-mentioned problems, and one of its objectives is to provide a parts counting device and robot system that can count parts accurately while saving the user time and effort. [Means for solving the problem]

[0007] A parts counting device according to the first aspect of this invention comprises a vibrating unit that vibrates a housing unit that houses a plurality of parts, an imaging unit that continuously photographs the parts inside the housing unit being vibrated by the vibrating unit, and an information processing unit that tracks the parts moving due to vibration and counts the parts inside the housing unit based on the images sequentially captured by the imaging unit, wherein the information processing unit tracks the parts moving due to vibration and counts the parts inside the housing unit based on the images sequentially captured by the imaging unit. A likelihood map is created for the components included in the captured image, and based on that likelihood map, The system is configured to estimate the position and orientation of a component and to track the component based on the estimated position and orientation. In this specification, vibration is a broad concept that includes shaking and rocking motion. A parts counting device according to a second aspect of this invention comprises a vibrating unit that vibrates a housing unit that houses a plurality of parts, an imaging unit that continuously photographs the parts in the housing unit that are vibrated by the vibrating unit, and an information processing unit that tracks the parts moving due to vibration and counts the parts in the housing unit based on the images sequentially taken by the imaging unit. The information processing unit is configured to create a likelihood map of the parts included in the images based on the images sequentially taken by the imaging unit, and to track the parts based on the likelihood map. The information processing unit is configured to estimate the position and orientation of the parts based on the created likelihood map, extract feature points of the parts based on the estimated position and orientation of the parts, and track the extracted feature points to track the parts.

[0008] The first part of this invention and the secondIn the parts counting device using this method, as described above, a vibrating unit is provided to vibrate the storage section that houses multiple parts. This allows parts to be moved by vibrating them with the vibrating unit, eliminating the need for the user to manually resolve overlapping parts. As a result, user effort is reduced. Furthermore, by tracking the parts that move due to vibration based on images sequentially captured by the imaging unit, it is possible to count parts while creating a situation with high counting accuracy (a situation where parts are not placed too close together), thus enabling accurate part counting. As a result, parts can be counted accurately while reducing user effort.

[0009] In the parts counting device according to the first aspect described above, preferably, the information processing unit is configured to create a likelihood map of parts included in the captured images based on the captured images sequentially captured by the imaging unit, and to track the parts based on the likelihood map. With this configuration, parts can be recognized with high accuracy based on the likelihood map, and therefore parts can be tracked with high accuracy.

[0010] In the parts counting device according to the second aspect described above, The information processing unit is configured to track a part by estimating the position and orientation of the part based on the created likelihood map, extracting feature points of the part based on the estimated position and orientation, and tracking the extracted feature points. With this configuration, the position and orientation of the part can be accurately estimated based on the likelihood map, and therefore the feature points of the part can be accurately extracted. As a result, the part can be accurately tracked by tracking the accurately extracted feature points.

[0011] In the above configuration for extracting feature points, preferably, the information processing unit is configured to set an extraction region for extracting feature points based on the estimated position and orientation of the part, and to extract feature points from within the extraction region. With this configuration, feature points can be extracted from within the extraction region where there is a high probability that feature points exist, thus enabling more accurate extraction of the part's feature points. As a result, by tracking the more accurately extracted feature points, the part can be tracked with greater accuracy.

[0012] In this case, preferably, the information processing unit is configured to obtain the degree of agreement between feature points extracted from the extraction region and feature points in the reference image, and to determine the feature points extracted from the extraction region whose degree of agreement is equal to or greater than a threshold as the feature points to be tracked. With this configuration, by tracking feature points with a high degree of agreement, the parts can be tracked with even greater accuracy.

[0013] In the above configuration for extracting feature points, preferably, the information processing unit is configured to count the components in the storage unit by counting the components that track the feature points. With this configuration, by counting the components that track the feature points, it is possible to avoid counting the same component twice, and thus the components can be counted with greater accuracy.

[0014] In this case, preferably, the information processing unit is configured to count parts by extracting characteristic points for each type of part and counting the parts for which characteristic points are tracked for each type. With this configuration, even if there are multiple types of parts in the storage unit, the parts can be counted accurately for each type of part.

[0015] In the above configuration for extracting feature points, preferably, the information processing unit compares the likelihood of the currently created likelihood map with the likelihood of likelihood maps created a predetermined number of times prior for each component region. For component regions where the likelihood of the currently created likelihood map is the highest among the compared component regions, the currently created likelihood map is used for extracting feature points. For component regions where the likelihood of the currently created likelihood map is not the highest among the compared component regions, a likelihood map created a predetermined number of times prior to the current likelihood map with a higher likelihood is used for extracting feature points. With this configuration, it is possible to track feature points that have been extracted with a high likelihood at all times.

[0016] The above 1 and the second In a parts counting device based on the plane of a machine, preferably, the information processing unit is configured to track parts using a machine learning model generated by machine learning. With this configuration, parts can be easily and accurately tracked using the machine learning model.

[0017] The above 1 and the second In a parts counting device based on the following conditions, preferably, the information processing unit is configured to notify the user if the number of counted parts does not meet a predetermined number. With this configuration, if the number of counted parts does not meet a predetermined number (i.e., the number of parts is greater than or less than the predetermined number), the user can easily find out that the number of parts does not meet the predetermined number and can easily take action to ensure that the number of parts meets the predetermined number.

[0018] The above 1 and the second In a parts counting device using a curved surface, preferably, the vibrating section includes a vibrator that vibrates the housing section in the horizontal direction. With this configuration, parts can be easily moved by vibrating them in the horizontal direction, making it easy to create a situation with high counting accuracy (a situation where parts are not placed close together).

[0019] The robot system according to the third aspect of this invention includes a robot that transfers parts within a storage unit or transfers parts into the storage unit, and a part counting device that counts the parts within the storage unit. The part counting device includes a vibration unit that vibrates the storage unit, an imaging unit that continuously images the parts within the storage unit vibrated by the vibration unit, and an information processing unit that tracks the parts that move due to vibration and counts the parts within the storage unit based on the captured images sequentially captured by the imaging unit. The information processing unit is configured to A likelihood map is created for the components included in the captured image, and based on that likelihood map, estimate the position and orientation of the parts, and track the parts based on the estimated position and orientation of the parts.

[0020] In the robot system according to the 3 aspect of this invention, as described above, a vibration unit that vibrates a storage unit that houses a plurality of parts is provided. As a result, the parts can be moved by vibrating the parts with the vibration unit, so there is no need for the user to manually eliminate the overlapping of the parts. As a result, the user's labor can be saved. Also, based on the captured images sequentially captured by the imaging unit, by tracking the parts that move due to vibration, it is possible to create a situation with high counting accuracy (a situation where the parts are not arranged close to each other) while counting the parts, so the parts can be accurately counted. As a result, the parts can be accurately counted while saving the user's labor.

Effects of the Invention

[0021] According to the present invention, as described above, the parts can be accurately counted while saving the user's labor.

Brief Description of the Drawings

[0022] [Figure 1] It is a schematic diagram showing a part counting device according to an embodiment. [Figure 2] It is a diagram for explaining the part counting process of the part counting device according to an embodiment. [Figure 3]This figure illustrates the feature point extraction process for parts counting according to one embodiment. [Figure 4] This figure illustrates a machine learning model used in feature point extraction processing according to one embodiment. [Figure 5] This is a flowchart illustrating the control process related to parts counting according to one embodiment. [Figure 6] This is a flowchart illustrating the image processing of captured images according to one embodiment. [Figure 7] This is a schematic diagram showing an example configuration of a robot system equipped with a parts counting device according to one embodiment. [Figure 8] This is a flowchart illustrating the control process on the imaging unit side related to parts counting, based on a modified example of one embodiment. [Figure 9] This is a flowchart illustrating the control processing on the information processing side for parts counting according to a modified example of one embodiment. [Modes for carrying out the invention]

[0023] The following describes embodiments of the present invention based on the drawings.

[0024] First, with reference to Figure 1, the configuration of a parts counting device 100 according to one embodiment will be described.

[0025] (Configuration of a parts counting device) As shown in Figure 1, the parts counting device 100 is a device that counts multiple parts 1 housed in a storage unit 2. The storage unit 2 is, for example, a tray. The parts 1 are, for example, screws or washers. In an industrial product assembly line, a storage unit 2 containing parts 1 may be supplied for the assembly of industrial products. The parts counting device 100 is used, for example, to check whether the correct number of parts 1 are housed in the storage unit 2 by counting the parts 1 housed in the storage unit 2 before it is supplied to the industrial product assembly line. The parts counting device 100 can also be described as an inspection device.

[0026] As shown in Figure 1, the parts counting device 100 comprises a mounting unit 10, a vibration unit 20, an imaging unit 30, and a control unit 40.

[0027] The mounting section 10 is configured to accommodate the housing section 2. The mounting section 10 is a platform on which the housing section 2 can be placed. The mounting section 10 is also provided with a vibration section 20.

[0028] The vibrating unit 20 is configured to vibrate the housing unit 2. Specifically, the vibrating unit 20 includes a first vibrator 21, a second vibrator 22, and a diaphragm 23. The first vibrator 21 is configured to vibrate the housing unit 2 mounted on the mounting unit 10 in the horizontal direction. Specifically, the first vibrator 21 is configured to vibrate the housing unit 2 mounted on the mounting unit 10 in the front-rear direction and the left-right direction (two directions that are substantially orthogonal to each other in the horizontal plane). The second vibrator 22 is configured to vibrate the housing unit 2 mounted on the mounting unit 10 in the up-down direction. The diaphragm 23 is configured to vibrate as each of the first vibrator 21 and the second vibrator 22 vibrates. As the diaphragm 23 vibrates, the components 1 inside the housing unit 2 are vibrated. Note that the first vibrator 21 is an example of a "vibrator" as defined in the claims.

[0029] The imaging unit 30 is a camera that continuously captures images of the component 1 inside the housing unit 2, which is vibrated by the vibration unit 20. The imaging unit 30 is configured to capture video or continuous still images that are close to video by shooting at a predetermined frame rate. The imaging unit 30 is positioned above the mounting unit 10 (i.e., above the component 1, which is the subject). The imaging unit 30 is also equipped with an illumination unit 31 that shines illumination light towards the component 1, which is the subject, when shooting.

[0030] The control unit 40 is configured to control the operation of the parts counting device 100. Specifically, the control unit 40 includes an information processing unit 41, a device control unit 42, and a storage unit 43. The information processing unit 41 includes a processor such as a CPU (Central Processing Unit) and processes various types of information. In this embodiment, the information processing unit 41 is configured to acquire an image 50 (see Figure 2) of the parts 1 taken by the imaging unit 30, and to count the parts 1 in the storage unit 2 based on the acquired image 50. The device control unit 42 includes a processor such as a CPU (Central Processing Unit) and controls the operation of the vibration unit 20 and the imaging unit 30. When counting the parts 1 in the storage unit 2, the vibration operation of the vibration unit 20 is performed and the imaging operation of the imaging unit 30 is performed under the control of the device control unit 42. The storage unit 43 includes a rewritable non-volatile memory such as flash memory and is configured to store various types of information.

[0031] (Configuration of parts counting process) As shown in Figures 2a and 2b, the device control unit 42 is configured to vibrate the housing unit 2 with the vibration unit 20 to move the components 1 inside the housing unit 2, while the imaging unit 30 continuously photographs the components 1 inside the housing unit 2. The information processing unit 41 is configured to acquire the images 50 captured by the imaging unit 30 and to count the components 1 inside the housing unit 2 based on the acquired images 50.

[0032] In this embodiment, as shown in Figures 2b to 2e, the information processing unit 41 is configured to track the parts 1 that move due to vibration and to count the parts 1 in the storage unit 2, based on the captured images 50 sequentially captured by the imaging unit 30. Specifically, the information processing unit 41 is configured to create a likelihood map 60 of the parts 1 included in the captured images 50 based on the captured images 50 sequentially captured by the imaging unit 30. The likelihood map 60 is a map that represents the distribution of the likelihood of the parts 1 in the captured images 50. The information processing unit 41 is configured to create the likelihood map 60 by calculating the likelihood that the region containing each pixel of the captured image 50 corresponds to the part that captures the part 1, using a method that uses template matching or a method that extracts features. Normally, if the parts 1 are adjacent to each other, the likelihood is calculated to be low, and if the parts 1 are not adjacent to each other, the likelihood is calculated to be high. The information processing unit 41 is also configured to track the parts 1 based on the created likelihood map 60.

[0033] Furthermore, in this embodiment, as shown in Figures 2 and 3, the information processing unit 41 is configured to track part 1 by estimating the position and orientation (direction) of part 1 based on the created likelihood map 60, extracting feature points 1a of part 1 based on the estimated position and orientation of part 1, and tracking the extracted feature points 1a. Specifically, as shown in Figures 2c, 2d, and 3a, the information processing unit 41 is configured to acquire a high-likelihood region 61 where the likelihood is above a threshold (for example, 90% or more) based on the created likelihood map 60, and to estimate the center position 62 of the acquired high-likelihood region 61 as the center position of part 1. Also, as shown in Figure 3a, the information processing unit 41 is configured to estimate the orientation of part 1 based on the shape of the acquired high-likelihood region 61. For example, if part 1 is an elongated shape such as a screw, the information processing unit 41 is configured to estimate the orientation of part 1 by assuming that the longitudinal direction of the high-likelihood region 61 is the longitudinal direction of part 1.

[0034] Furthermore, in this embodiment, as shown in Figures 3b and 3c, the information processing unit 41 is configured to set an extraction region 70 from which feature points 1a are extracted based on the estimated position and orientation of the part 1, and to extract feature points 1a from within the extraction region 70. The extraction region 70 is an area where there is a high probability that feature points 1a of the part 1 exist. Specifically, the extraction region 70 is an area representing the outer shape (contour) of the part 1. Information representing the outer shape of the part 1 is pre-stored in the storage unit 43. The information processing unit 41 is configured to set the extraction region 70 based on the information representing the outer shape of the part 1 stored in the storage unit 43 and the estimated position and orientation of the part 1, assuming that the part 1 is placed at the estimated position and with the estimated orientation.

[0035] Furthermore, in this embodiment, as shown in Figure 3d, the information processing unit 41 is configured to obtain the degree of agreement between the feature points 1a extracted from within the extraction region 70 and the feature points 1a in the reference image 80, and to determine the feature points 1a extracted from within the extraction region 70 whose degree of agreement is equal to or greater than a threshold as the feature points 1a to be tracked (final feature points 1a). The reference image 80 is an image representing part 1, and feature points 1a are pre-set in the reference image 80. For example, feature points 1a near the outer shape, such as corners and edges, are set as the feature points 1a of the reference image 80. This makes it possible to set feature points 1a that hold information such as thickness and length, which are highly relevant to the type of part 1. One or more reference images 80 are pre-stored in the storage unit 43.

[0036] Furthermore, in this embodiment, as shown in Figure 2e, the information processing unit 41 is configured to count the parts 1 in the storage unit 2 by counting the parts 1 that track the feature point 1a. In the example shown in Figure 2e, there is one part 1 that tracks the feature point 1a (i.e., one detected part 1), so the number of parts 1 counted is one. By vibrating the storage unit 2 with the vibration unit 20 to move the parts 1 in the storage unit 2, and continuously photographing the parts 1 in the storage unit 2 with the photography unit 30, and performing image processing on the sequentially acquired photographed images 50, the number of parts 1 that track the feature point 1a gradually increases. The information processing unit 41 is configured to continue the above-described part counting process until the number of counted parts 1 reaches a predetermined number (the correct number that should be stored in the storage unit 2).

[0037] Furthermore, in this embodiment, the information processing unit 41 is configured to notify the user if the number of counted parts 1 does not meet a predetermined number. Specifically, the information processing unit 41 is configured to notify the user if the number of counted parts 1 remains below the predetermined number even after a predetermined time has elapsed, or if the number of counted parts 1 exceeds the predetermined number. The information processing unit 41 is configured to notify the user that the number of counted parts 1 does not meet the predetermined number by displaying it on the display unit. For example, if the parts counting device 100 has a display unit, the information processing unit 41 displays the notification on the display unit of the parts counting device 100, and if the parts counting device 100 does not have a display unit, the information processing unit 41 displays the notification on an external display unit.

[0038] Furthermore, in this embodiment, the information processing unit 41 compares the likelihood (accuracy) of the currently created likelihood map 60 (the latest likelihood map 60) with the likelihood (accuracy) of the likelihood maps 60 created up to a predetermined number of times prior, for each component region (regions with high likelihood, such as high-likelihood regions 61). For component regions where the likelihood of the currently created likelihood map 60 is the maximum among the compared component regions, the currently created likelihood map 60 is used to extract feature points 1a. For component regions where the likelihood of the currently created likelihood map 60 is not the maximum among the compared component regions, the likelihood map 60 created up to a predetermined number of times with a higher likelihood than the currently created likelihood map 60 is used to extract feature points 1a. For example, the information processing unit 41 is configured to use the likelihood map 60 with the highest likelihood among the likelihood maps 60 created up to a predetermined number of times prior to extract feature points 1a for component regions where the likelihood of the currently created likelihood map 60 is not the maximum among the compared component regions.

[0039] Here, for convenience, Figures 2 and 3 illustrate an example in which one type of component 1 is housed in the housing 2, but there are also cases in which multiple types of components 1 are housed in the housing 2. Therefore, in this embodiment, when multiple types of components 1 exist in the housing 2, the information processing unit 41 is configured to count components 1 for each type by extracting feature points 1a of component 1 for each type of component 1 and counting the components 1 that track the feature points 1a for each type of component 1. In this case, the information processing unit 41 is configured to either create separate likelihood maps 60 for each type of component 1, or to create one likelihood map 60 that includes likelihood information for each type of component 1. And, although a detailed explanation is omitted, the information processing unit 41 is configured to extract feature points 1a of component 1 for each type of component 1 based on the created likelihood maps 60, in the same manner as described above.

[0040] (Machine learning model configuration) Furthermore, in this embodiment, as shown in Figure 4, the information processing unit 41 is configured to track the component 1 using a machine learning model 90 generated by machine learning. Specifically, as shown in Figure 4a, the information processing unit 41 is configured to extract feature points 1a using the trained machine learning model 90. More specifically, the machine learning model 90 is configured to take an image within the extraction region 70 of the captured image 50 as input and output feature point data 91, which is two-dimensional data representing the feature points 1a within the extraction region 70. The machine learning model 90 is composed of, for example, a convolution network. When the extraction region 70 is set, the information processing unit 41 extracts an image within the extraction region 70 of the captured image 50 and inputs it to the machine learning model 90. The information processing unit 41 also extracts the feature points 1a within the extraction region 70 by obtaining feature point data 91 from the machine learning model 90 that has received the image as input.

[0041] Figure 4b shows the training procedure for the machine learning model 90. As shown in Figure 4b, first, the training image 92 is input to the machine learning model 90, and the machine learning model 90 outputs feature point data 91. Then, feature points are extracted based on the feature point data 91, and tracking is performed using the extracted feature points. Then, a loss value representing the tracking accuracy is calculated based on the tracking results. Then, the parameters of the machine learning model 90 are updated by inversely assigning the loss value to the parameters of the machine learning model 90. By repeating the update of the parameters of the machine learning model 90 using the above procedure, a machine learning model 90 capable of accurately extracting feature points 1a can be generated. In addition, the trained machine learning model 90 is pre-stored in the memory unit 43 and used.

[0042] (Control processing related to parts counting) Referring to Figures 5 and 6, the control process for parts counting by the parts counting device 100 of this embodiment will be explained based on a flowchart.

[0043] As shown in Figure 5, first, in step S1, the first vibrator 21 and the second vibrator 22 of the vibrating unit 20 are activated.

[0044] Then, in step S2, the housing unit 2 is vibrated by the vibrating unit 20, and the component 1 inside the housing unit 2 is photographed by the imaging unit 30. As a result, an image 50 of the component 1 inside the housing unit 2 is obtained.

[0045] Then, in step S3, image processing is performed on the captured image 50. Details of the image processing of the captured image 50 will be described later.

[0046] Then, in step S4, it is determined whether or not a predetermined number of parts have been detected (counted). If it is determined that a predetermined number of parts have been detected, the process proceeds to step S5.

[0047] Then, in step S5, the first vibrator 21 and the second vibrator 22 of the vibrating unit 20 are stopped. The control process is then terminated.

[0048] Furthermore, if it is determined in step S4 that a predetermined number of parts have not been detected, the process proceeds to step S6.

[0049] Then, in step S6, it is determined whether or not an error condition has been met. The error conditions include the condition that the number of counted parts 1 is less than a predetermined number even after a predetermined time has elapsed, and the condition that the number of counted parts 1 has become greater than a predetermined number. If it is determined that an error condition has been met, the process proceeds to step S7.

[0050] Then, in step S7, the user is notified of an error indicating that the number of counted parts 1 does not meet the predetermined number. Subsequently, the first vibrator 21 and the second vibrator 22 of the vibrating unit 20 are stopped, and the control process is terminated.

[0051] Furthermore, if it is determined in step S6 that the error condition is not met, the process proceeds to step S2. Then, steps S2 to S6 are repeated until a predetermined number of parts are detected or the error condition is met.

[0052] Referring to Figure 6, the details of the image processing of the captured image 50 in step S3 will be explained. In addition, in the image processing of the captured image 50, processes to enhance part 1, such as binarization and edge enhancement, may be performed as appropriate to improve the recognition accuracy and tracking accuracy of part 1.

[0053] As shown in Figure 6, first, in step S11, a likelihood map 60 of part 1 is created based on the captured image 50.

[0054] Then, in step S12, the position and orientation of part 1 are estimated based on the likelihood map 60. Furthermore, if multiple types of parts 1 are housed in the housing 2, the type of part 1 is estimated by either creating a separate likelihood map 60 for each type of part 1, or by creating a single likelihood map 60 that includes likelihood information for each type of part 1.

[0055] Then, in step S13, the likelihood of the likelihood map 60 created this time is compared with the likelihood of the likelihood maps 60 created a predetermined number of times earlier for each component region, and it is determined for each component region whether the likelihood of the likelihood map 60 created this time is the highest. For component regions that are determined to have the highest likelihood in the likelihood map 60 created this time, the process in step S14 is performed.

[0056] Then, in step S14, the feature point 1a of part 1 is extracted using the likelihood map 60 created in this step. The extracted feature point 1a of part 1 is updated and stored in the memory unit 43.

[0057] Then, in step S15, the extracted feature point 1a of part 1 is tracked, thereby tracking part 1. After that, the process proceeds to step S4.

[0058] Furthermore, in step S13, for component regions where it is determined that the likelihood is not maximized in the likelihood map 60 created in this step, the processing in step S16 is performed.

[0059] Then, in step S16, feature point 1a is extracted using a likelihood map 60 created a predetermined number of times prior to the current likelihood map 60, which has a higher likelihood. Furthermore, the extracted feature point 1a of part 1 is tracked, thereby tracking part 1. The process then proceeds to step S4.

[0060] (Configuration of a robot system equipped with a parts counting device) Next, with reference to Figure 7, a robot system 200 equipped with the parts counting device 100 will be described as an example of the application of the parts counting device 100 of this embodiment.

[0061] The robot system 200 comprises a parts counting device 100, a robot 110, and a robot control unit 120. The robot 110 is a picking robot that transfers parts 1 within the storage unit 2 or transfers parts 1 into the storage unit 2. The robot 110 is, for example, a vertical articulated robot. The robot 110 includes an articulated arm 111 and a hand 112 attached to the end of the arm 111 to hold the parts 1. The robot control unit 120 is configured to control the operation of the robot 110. In the robot system 200, the parts counting device 100 counts the parts 1 within the storage unit 2 to verify, for example, whether the number of parts 1 transferred into the storage unit 2 by the robot 110 is correct. Alternatively, the parts counting device 100 counts the parts 1 within the storage unit 2 to verify whether the number of parts 1 in the storage unit 2 before they were transferred by the robot 110 is correct. The parts counting process by the parts counting device 100 is as described above, so a detailed explanation will not be repeated.

[0062] Furthermore, in order to check whether the number of parts 1 in the storage unit 2 before being transferred by the robot 110 is correct, the parts counting device 100 counts the parts 1 in the storage unit 2. If the correct number of parts 1 is counted, the imaging unit 30 and information processing unit 41 of the parts counting device 100 operate as follows: The imaging unit 30 is configured to continue continuously imaging the parts 1 even after the vibration unit 20 stops vibrating the storage unit 2. The information processing unit 41 is configured to recognize the posture of the stationary parts 1 in the storage unit 2 based on the images captured by the imaging unit 30 after the vibration unit 20 stops vibrating the storage unit 2. The robot control unit 120 is configured to determine the part of the parts 1 that the robot 110's hand 112 will hold based on the posture of the parts 1 recognized by the information processing unit 41. This makes it possible to effectively utilize the imaging unit 30 and information processing unit 41 of the parts counting device 100 to determine the part of the parts 1 that the robot 110's hand 112 will hold.

[0063] Furthermore, if multiple types of parts 1 are housed in the housing section 2, the information processing unit 41 is configured to recognize the orientation of each type of part 1 stationary within the housing section 2 based on the image captured by the imaging unit 30 after the vibration unit 20 has stopped vibrating the housing section 2. The robot control unit 120 is then configured to determine the portion of the robot 110's hand 112 that will hold the part 1, based on the orientation of each type of part 1 recognized by the information processing unit 41. This makes it possible to effectively utilize the imaging unit 30 and the information processing unit 41 of the parts counting device 100 to appropriately determine the portion of the robot 110's hand 112 that will hold the part 1, according to the type of part 1.

[0064] (Effects of this embodiment) In this embodiment, the following effects can be obtained.

[0065] In this embodiment, as described above, a vibration unit 20 is provided to vibrate the housing unit 2 that houses multiple parts 1. This allows the parts 1 to be moved by vibrating them with the vibration unit 20, eliminating the need for the user to manually resolve overlaps of the parts 1. As a result, user effort is reduced. Furthermore, by tracking the parts 1 that move due to vibration based on the captured images 50 sequentially captured by the imaging unit 30, it is possible to count the parts 1 while creating a situation with high counting accuracy (a situation where the parts 1 are not placed close together), thus enabling accurate counting of the parts 1. As a result, the parts 1 can be counted accurately while reducing user effort.

[0066] Furthermore, in this embodiment, as described above, the information processing unit 41 is configured to create a likelihood map 60 of the component 1 included in the captured image 50 based on the captured image 50, and to track the component 1 based on the likelihood map 60. As a result, the component 1 can be accurately recognized based on the likelihood map 60, and therefore the component 1 can be accurately tracked.

[0067] Furthermore, in this embodiment, as described above, the information processing unit 41 is configured to track part 1 by estimating the position and orientation of part 1 based on the created likelihood map 60, extracting feature points 1a of part 1 based on the estimated position and orientation of part 1, and tracking the extracted feature points 1a. As a result, the position and orientation of part 1 can be accurately estimated based on the likelihood map 60, and the feature points 1a of part 1 can be accurately extracted. Consequently, part 1 can be accurately tracked by tracking the accurately extracted feature points 1a.

[0068] Furthermore, in this embodiment, as described above, the information processing unit 41 is configured to set an extraction region 70 for extracting feature points 1a based on the estimated position and orientation of the part 1, and to extract feature points 1a from within the extraction region 70. This allows for the extraction of feature points 1a from within the extraction region 70, where there is a high probability that feature points 1a exist, thereby enabling more accurate extraction of feature points 1a from the part 1. As a result, by tracking the more accurately extracted feature points 1a, the part 1 can be tracked with greater accuracy.

[0069] Furthermore, in this embodiment, as described above, the information processing unit 41 is configured to obtain the degree of agreement between the feature points 1a extracted from within the extraction region 70 and the feature points 1a in the reference image 80, and to determine the feature points 1a extracted from within the extraction region 70 whose degree of agreement is equal to or greater than a threshold as the feature points 1a to be tracked. As a result, by tracking the feature points 1a with a high degree of agreement, the component 1 can be tracked with even greater accuracy.

[0070] Furthermore, in this embodiment, as described above, the information processing unit 41 is configured to count the components 1 in the housing unit 2 by counting the components 1 that track the feature points 1a. This avoids double counting of the same component 1 by counting the components 1 that track the feature points 1a, thus enabling more accurate counting of the components 1.

[0071] Furthermore, in this embodiment, as described above, the information processing unit 41 is configured to count parts 1 for each type of part 1 by extracting feature points 1a of part 1 for each type of part 1 and counting the parts 1 that track the feature points 1a for each type of part 1. This makes it possible to accurately count parts 1 for each type of part 1 even when multiple types of parts 1 exist in the storage unit 2.

[0072] Furthermore, in this embodiment, as described above, the information processing unit 41 compares the likelihood of the currently created likelihood map 60 with the likelihood of the likelihood maps 60 created a predetermined number of times prior, for each component region. For component regions where the likelihood of the currently created likelihood map 60 is the highest among the compared component regions, the currently created likelihood map 60 is used to extract feature points 1a. For component regions where the likelihood of the currently created likelihood map 60 is not the highest among the compared component regions, the likelihood map 60 created a predetermined number of times prior to the current likelihood map 60, which has a higher likelihood than the currently created likelihood map 60, is used to extract feature points 1a. This makes it possible to track feature points 1a that have been extracted with a high likelihood at all times.

[0073] Furthermore, in this embodiment, as described above, the information processing unit 41 is configured to track the component 1 using a machine learning model 90 generated by machine learning. This makes it possible to easily and accurately track the component 1 using the machine learning model 90.

[0074] Furthermore, in this embodiment, as described above, the information processing unit 41 is configured to notify the user if the number of counted parts 1 does not meet a predetermined number. As a result, if the number of counted parts 1 does not meet a predetermined number (i.e., the number of parts 1 is greater than or less than the predetermined number), the user can easily find out that the number of parts 1 does not meet the predetermined number and can easily take action to ensure that the number of parts 1 meets the predetermined number.

[0075] Furthermore, in this embodiment, as described above, the vibrating unit 20 includes a first vibrator 21 that vibrates the housing unit 2 in the horizontal direction. This makes it possible to easily move the component 1 by vibrating it in the horizontal direction, thus easily creating a situation with high counting accuracy (a situation where the component 1 are not placed in close proximity to each other).

[0076] [Differentiation] It should be noted that the embodiments disclosed herein are illustrative and not restrictive in all respects. The scope of the present invention is indicated by the claims rather than by the description of the embodiments above, and further includes all modifications (exceptions) within the meaning and scope of the claims.

[0077] For example, the above embodiment shows an example of applying the parts counting device to a robot system, but the present invention is not limited to this. In the present invention, the parts counting device may be applied to systems other than robot systems.

[0078] Furthermore, while the above embodiment shows an example where the parts counting device is an inspection device that checks the number of parts by counting them, the present invention is not limited to this. In the present invention, the parts counting device may be an inspection device that further has inspection functions other than the function of checking the number of parts.

[0079] Furthermore, while the above embodiment shows an example of tracking feature points extracted based on a likelihood map, the present invention is not limited to this. In the present invention, feature points may be extracted by image processing other than a likelihood map, and the extracted feature points may be tracked.

[0080] Furthermore, although the above embodiment shows an example of setting an extraction region for extracting feature points, the present invention is not limited thereto. The present invention does not require setting an extraction region for extracting feature points.

[0081] Furthermore, while the above embodiment shows an example of obtaining the degree of agreement between feature points extracted from the extraction region and feature points in the reference image, the present invention is not limited to this. In the present invention, it is not necessary to obtain the degree of agreement between feature points extracted from the extraction region and feature points in the reference image. That is, feature points extracted from the extraction region may be determined as the feature points to be tracked.

[0082] Furthermore, while the above embodiment shows an example in which the likelihood of the likelihood map created this time is compared with the likelihood of the likelihood maps created a predetermined number of times earlier for each component region, the present invention is not limited to this. In the present invention, it is not necessary to compare the likelihood of the likelihood map created this time with the likelihood of the likelihood maps created a predetermined number of times earlier for each component region. That is, the likelihood map created this time may always be used for feature point extraction.

[0083] Furthermore, while the above embodiment demonstrates an example of extracting feature points using a machine learning model, the present invention is not limited thereto. In the present invention, feature points may be extracted using rule-based image processing or other methods besides machine learning models.

[0084] Furthermore, while the above embodiment demonstrates an example of tracking parts by extracting feature points using a machine learning model trained to extract feature points, the present invention is not limited to this. In the present invention, a likelihood map may be created using a machine learning model trained to create a likelihood map, and parts may be tracked.

[0085] Furthermore, although the above embodiment shows an example in which the vibrating section includes both a first vibrator that vibrates the housing section horizontally and a second vibrator that vibrates it vertically, the present invention is not limited to this. In the present invention, the vibrating section may include only one of the first vibrator that vibrates the housing section horizontally and the second vibrator that vibrates it vertically.

[0086] Furthermore, in the above embodiment, for the sake of explanation, the processing operations of the control unit were described using a flow-driven flowchart that processes sequentially according to the processing flow, but the present invention is not limited thereto. In the present invention, the processing operations of the control unit may be performed by event-driven processing, which executes processing on an event-by-event basis. In this case, it may be performed as a completely event-driven system, or a combination of event-driven and flow-driven systems may be used.

[0087] Furthermore, in this invention, if the shooting speed is greater than the image processing speed, the shooting process and image processing may be executed in parallel in separate flows, as shown in the modified examples in Figures 8 and 9. In Figures 8 and 9, the same reference numerals are used for processes identical to those in the flowchart of Figure 5, and their detailed descriptions are omitted.

[0088] As shown in Figure 8, in step S101, the imaging unit 30 determines whether all of the captured images 50 stored in the storage unit 43 have been processed. If it is determined that not all of the captured images 50 stored in the storage unit 43 have been processed, the process in step S101 is repeated. If it is determined that all of the captured images 50 stored in the storage unit 43 have been processed, the process proceeds to step S102. In step S102, the imaging unit 30 photographs the component 1 inside the storage unit 2. The captured images 50 are also stored in the storage unit 43. In step S103, it is determined whether a predetermined number of shots have been taken. If it is determined that a predetermined number of shots have not been taken, the process in step S102 is repeated. If it is determined that a predetermined number of shots have been taken, the process proceeds to step S4.

[0089] Furthermore, as shown in Figure 9, in step S111, the information processing unit 41 determines whether or not there are any captured images 50 stored in the storage unit 43 that have not yet undergone image processing. If it is determined that there are no captured images 50 stored in the storage unit 43 that have not yet undergone image processing, the process in step S111 is repeated. If it is determined that there are captured images 50 stored in the storage unit 43 that have not yet undergone image processing, the process proceeds to step S3, where image processing is performed on the captured images 50 that have not yet undergone image processing. Then, the process proceeds to step S111. By executing the flowcharts shown in Figures 8 and 9 in parallel, the camera 30 can take pictures while waiting for the information processing unit 41 to complete the image processing. [Explanation of Symbols]

[0090] 1 part 1a Feature points 2. Storage area 20 Vibration section 21. First oscillator (oscillator) 30 Photography Department 41 Information Processing Section 50 photographed images 60 Likelihood Maps 70 Extraction area 80 Reference Images 90 Machine Learning Models 100 parts counting device 110 Robots 200 Robot Systems

Claims

1. A vibrating unit that vibrates a housing that contains multiple components, The vibrating unit is equipped with a shooting unit that continuously photographs the components inside the housing that are being vibrated by the vibrating unit, The system includes an information processing unit that tracks the components moving due to vibration based on the images sequentially captured by the aforementioned imaging unit, and counts the components within the housing unit, The component counting device is configured such that the information processing unit creates a likelihood map of the components included in the captured images based on the captured images sequentially captured by the imaging unit, estimates the position and orientation of the components based on the likelihood map, and tracks the components based on the estimated position and orientation of the components.

2. A vibrating unit that vibrates a housing that contains multiple components, The vibrating unit is equipped with a shooting unit that continuously photographs the components inside the housing that are being vibrated by the vibrating unit, The system includes an information processing unit that tracks the components moving due to vibration based on the images sequentially captured by the aforementioned imaging unit, and counts the components within the housing unit, The information processing unit is configured to create a likelihood map of the components included in the captured images based on the captured images sequentially captured by the imaging unit, and to track the components based on the likelihood map. The information processing unit is configured to track the parts by estimating the position and orientation of the parts based on the likelihood map created, extracting feature points of the parts based on the estimated position and orientation of the parts, and tracking the extracted feature points.

3. The parts counting device according to claim 2, wherein the information processing unit is configured to set an extraction region for extracting feature points based on the estimated position and orientation of the parts, and to extract the feature points from within the extraction region.

4. The parts counting device according to claim 3, wherein the information processing unit is configured to obtain the degree of agreement between the feature points extracted from the extraction region and the feature points in the reference image, and to determine the feature points extracted from the extraction region whose degree of agreement is equal to or greater than a threshold as the feature points to be tracked.

5. The component counting device according to claim 2, wherein the information processing unit is configured to count the components in the housing by counting the components that track the feature points.

6. The component counting device according to claim 5, wherein the information processing unit is configured to count the components for each type of component by extracting characteristic points of the component for each type of component and counting the components for which the characteristic points are tracked for each type of component.

7. The component counting device according to claim 2, wherein the information processing unit is configured to compare the likelihood of the likelihood map created this time with the likelihood of the likelihood maps created a predetermined number of times prior for each component region, and for the component regions among the compared component regions where the likelihood of the likelihood map created this time is the maximum, the likelihood map created this time is used for extracting the feature points, and for the component regions among the compared component regions where the likelihood of the likelihood map created this time is not the maximum, the likelihood map created a predetermined number of times prior to the component regions where the likelihood of the likelihood map created this time is greater is used for extracting the feature points.

8. The parts counting device according to claim 1 or 2, wherein the information processing unit is configured to track the parts using a machine learning model generated by machine learning.

9. The parts counting device according to claim 1 or 2, wherein the information processing unit is configured to notify the user if the number of parts counted does not meet a predetermined number.

10. The parts counting device according to claim 1 or 2, wherein the vibrating section includes a vibrator that vibrates the housing section in the horizontal direction.

11. A robot that transfers parts within a storage compartment, or transfers parts into the storage compartment, The system includes a parts counting device for counting the parts within the storage compartment, The aforementioned parts counting device is A vibrating unit that vibrates the aforementioned housing unit, The vibrating unit is equipped with a shooting unit that continuously photographs the components inside the housing that are being vibrated by the vibrating unit, Based on the images sequentially captured by the aforementioned imaging unit, an information processing unit tracks the components that move due to vibration and counts the components within the housing unit. Includes, The robot system is configured such that the information processing unit creates a likelihood map of the parts included in the captured images based on the captured images sequentially captured by the imaging unit, estimates the position and orientation of the parts based on the likelihood map, and tracks the parts based on the estimated position and orientation of the parts.

Images