Anomaly detection device and anomaly detection method

Abstract

It detects parts overhanging or overlapping parts. [Solution] The anomaly detection device moves integrally with the actuator and is capable of communicating with a camera capable of imaging parts. It comprises: an acquisition unit that acquires the amount of movement of the actuator and a plurality of captured images of the parts; an estimation unit that estimates a first movement distance of the parts based on the amount of movement of the actuator; an extraction unit that extracts features of the parts that appear in the plurality of captured images and extracts a second movement distance of the parts based on the positional information of the extracted features; and a determination unit that performs an anomaly detection process to determine whether or not there is an anomaly in the arrangement of the parts based on the difference between the first movement distance and the second movement distance.

Description

Technical Field

[0001] The present disclosure relates to an abnormality detection apparatus and an abnormality detection method.

Background Art

[0002] Patent Document 1 discloses a component mounting apparatus that determines whether a component has an abnormal posture in which the component protrudes from a component storage portion in a direction intersecting the vertical direction. The component mounting apparatus calculates a feature amount related to the posture of a component based on the component region, the component shadow region and the storage region in a captured image including the component region representing the image of the component, the component shadow region representing the image of the shadow of the component, and the storage region representing the image of the component storage portion, and determines the abnormal posture of the component based on the feature amount.

[0003] Patent Document 2 discloses that when a specific event that allows inference that a component exists in an abnormal state in an area set in a predetermined range near the component pickup position of one component supply unit among a plurality of component supply units is detected, the component pickup position of the component supply unit adjacent to the one component supply unit is imaged, and based on the captured image, it is detected whether or not a component exists in an abnormal state in the area of the adjacent component supply unit.

Prior Art Literature

Patent Literature

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problem to be Solved by the Invention

[0005] The present disclosure has been devised in view of conventional circumstances, and an object thereof is to provide an abnormality detection apparatus and an abnormality detection method that detect protruding components or overlapping between components.

Means for Solving the Problem

[0006] This disclosure provides an anomaly detection device that moves integrally with an actuator and can communicate with a camera capable of imaging a component, comprising: an acquisition unit that acquires the amount of movement of the actuator and a plurality of captured images of the component; an estimation unit that estimates a first movement distance of the component based on the amount of movement of the actuator; an extraction unit that extracts features of the component captured in the plurality of captured images and extracts a second movement distance of the component based on the positional information of the extracted features; and a determination unit that performs an anomaly detection process to determine whether or not there is an anomaly in the arrangement of the component based on the difference between the first movement distance and the second movement distance.

[0007] Furthermore, this disclosure provides an anomaly detection method performed by a computer that moves integrally with an actuator and can communicate with a camera capable of imaging a component, the method comprising: acquiring the amount of movement of the actuator and a plurality of captured images of the component; estimating a first movement distance of the component based on the amount of movement of the actuator; extracting features of the component captured in the plurality of captured images; extracting a second movement distance of the component based on the positional information of the extracted features; and performing an anomaly detection process to determine whether or not there is an anomaly in the arrangement of the component based on the difference between the first movement distance and the second movement distance. [Effects of the Invention]

[0008] According to this disclosure, it is possible to detect parts overhanging or overlapping parts. [Brief explanation of the drawing]

[0009] [Figure 1] A diagram illustrating an example of a picking system. [Figure 2] Block diagram showing an example of the internal configuration of an image processing device. [Figure 3] A diagram showing an example of the time period for detecting an anomaly. [Figure 4] A diagram showing a comparative example of normal travel distance and characteristic travel distance. [Figure 5] A diagram showing a comparative example of normal travel distance and characteristic travel distance. [Figure 6] Flowchart showing an example of the operation procedure of the image processing apparatus in the embodiment. [Figure 7] Flowchart showing an example of the operation procedure of the image processing apparatus in the embodiment. [Figure 8] A diagram showing an example of a status display screen. [Modes for carrying out the invention]

[0010] (Background leading to this disclosure) In the component pick-up process using component mounting devices, there has been a problem in that if components overlap or are in an abnormal position, the correct component cannot be accurately picked up. Therefore, conventional methods for detecting abnormal component positions have been proposed, such as a method for detecting components that are in an upright position or protruding from the component storage area using image processing (see Patent Document 1), and a method for detecting whether a component is in an abnormal state in the area of ​​an adjacent component supply unit based on an image of the component removal position when a component falls, or a component jumps due to vibration or collision during feeder attachment / detachment is detected (see Patent Document 2).

[0011] However, Patent Document 1, mentioned above, aims to determine an abnormal orientation of a component based on whether or not the component is protruding from the component storage area, and Patent Document 2 aims to detect whether or not a component is in an abnormal state in the area of ​​a component supply unit adjacent to a component supply unit. For this reason, it has been difficult to detect overlapping of components as an abnormal orientation of a component using conventional methods for detecting abnormal orientations of components.

[0012] The following describes in detail each embodiment of the anomaly detection device and anomaly detection method specifically disclosed herein, with appropriate reference to the attached drawings. However, unnecessary details may be omitted. For example, detailed explanations of already well-known matters and redundant explanations of substantially identical configurations may be omitted. This is to avoid the following explanation becoming unnecessarily verbose and to facilitate understanding by those skilled in the art. The attached drawings and the following explanation are provided to enable those skilled in the art to fully understand this disclosure and are not intended to limit the subject matter described in the claims.

[0013] First, the picking system 100 in this disclosure will be described with reference to Figures 1 and 2, respectively. Figure 1 is a diagram illustrating an example configuration of the picking system 100. Figure 2 is a block diagram showing an example of the internal configuration of the image processing device P1.

[0014] The picking system 100 transports the component Tg1 stored in the tray feeder by a belt conveyor. The picking system 100 captures an image of the component Tg1 with a camera CM and analyzes the captured image of the component Tg1 with an image processing device P1 to determine whether or not there is an abnormality in the component Tg1 that would prevent it from being picked. If the picking system 100 determines that there is no abnormality in the component Tg1, it uses the end effector EF to pick up the component Tg1 and mount it on the substrate. If it determines that there is an abnormality, it omits picking the component Tg1.

[0015] The picking system 100 shown in Figure 1 is an example that includes a component mounting device that uses an end effector to pick up and mount components stored in a tray feeder onto a circuit board (not shown), but is not limited to this example. The picking system 100 may also be a robot that uses an end effector to pick up packages or components such as products stored in a tray.

[0016] Furthermore, the picking system 100 shown in FIG. 1 is an example, and the present invention is not limited thereto. For example, the image processing apparatus P1 may be connected to a plurality of actuators AC, a plurality of encoders EN, a plurality of cameras CM, a plurality of monitors MN, and a plurality of mice MS such that data communication can be performed therebetween, and may concurrently execute component abnormality detection processing executed by the plurality of actuators AC.

[0017] The actuator AC controls, via a plurality of axes, an end effector EF and a camera CM provided on the end effector EF such that each can move three-dimensionally. As shown in FIG. 1, the actuator AC controls the camera CM to be movable three-dimensionally, thereby controllably changing the positional relationship between a component Tg1 moving on a belt conveyor, the end effector EF that picks the component Tg1, and the camera CM fixedly installed on the end effector EF.

[0018] The encoder EN periodically detects the three-dimensional movement amount and movement direction of the actuator AC, and outputs the detection result to the image processing apparatus P1.

[0019] The end effector EF is, for example, a suction nozzle provided in a component mounting apparatus arranged corresponding to the picking system 100, and approaches the component Tg1 under the control of the actuator AC, and picks the component Tg1 by suctioning it. Note that any end effector may be used as the end effector EF depending on the picking target, the purpose of use of the actuator AC, or the like.

[0020] The camera CM is arranged in the vicinity of the end effector EF, moves integrally with the end effector EF under the control of the actuator AC to check and image the component Tg1. The camera CM transmits the captured image of the component Tg1 obtained each time imaging is performed to the image processing apparatus P1 each time.

[0021] The image processing device P1 performs an abnormality detection process to determine whether or not there is an abnormality in the part Tg1 to be picked, based on the image captured by the camera CM and the amount and direction of movement of the actuator AC from the encoder EN. Based on the abnormality detection result, the image processing device P1 controls the actuator AC to either perform picking of part Tg1 or skip picking of part Tg1.

[0022] The image processing device P1 is implemented by, for example, a personal computer (hereinafter referred to as "PC"), a notebook PC, etc. The image processing device P1 may also be implemented by an on-premise server or a cloud server. The image processing device P1 includes a communication unit (not shown), a processor 11, and a memory 12. The image processing device P1 is connected to the actuator AC, encoder EN, camera CM, feature database DB, mouse MS, and monitor MN via the communication unit (not shown), enabling data communication between them.

[0023] The processor 11 is configured using, for example, a Central Processing Unit (CPU) or a Field Programmable Gate Array (FPGA), and works in cooperation with the memory 12 to perform various processes and controls. Specifically, the processor 11 refers to the programs and data held in the memory 12 and executes those programs to realize functions such as the actuator control unit 111, the abnormality determination time zone estimation unit 112, the normal movement distance estimation unit 113, the feature extraction unit 114, the feature movement distance extraction unit 115, the abnormality degree determination unit 116, and the UI generation unit 117.

[0024] The actuator control unit 111 controls the actuator AC based on the abnormality detection result of the abnormality degree determination unit 116 for part Tg1. If the abnormality degree determination unit 116 does not detect any abnormality in part Tg1, the actuator control unit 111 controls the actuator AC to pick part Tg1. On the other hand, if an abnormality is detected in part Tg1, the actuator control unit 111 performs predetermined control such as omitting (skipping) the picking of part Tg1 or stopping the operation of the actuator AC.

[0025] The abnormality determination time zone estimation unit 112 determines whether the current time is within the abnormality determination time zone T (see Figure 3) based on the amount and direction of movement of actuator AC output from encoder EN, and outputs this to the normal movement distance estimation unit 113 and the feature extraction unit 114, respectively.

[0026] The normal travel distance estimation unit 113 estimates the travel distance of component Tg1 during the abnormality detection time period T (hereinafter referred to as the "normal travel distance") based on the amount of movement and direction of movement of actuator AC output from encoder EN.

[0027] The feature extraction unit 114 extracts features (e.g., edges) of component Tg1 from each of the multiple images captured by the camera CM during the anomaly detection time period T. The feature extraction unit 114 extracts the features of component Tg1 using a convolutional neural network in deep learning, a pre-trained model capable of extracting features of component Tg1, or a rule-based system.

[0028] The feature displacement distance extraction unit 115 extracts the displacement distance of component Tg1 during the abnormality determination time period T (hereinafter referred to as "feature displacement distance") based on the position of features extracted from each of the multiple captured images taken during the abnormality determination time period T, and the capture time of each captured image.

[0029] The abnormality determination unit 116 calculates the difference between the normal movement distance output from the normal movement distance estimation unit 113 and the feature movement distance output from the feature movement distance extraction unit 115. Based on the calculated difference, the abnormality determination unit 116 determines the degree to which the arrangement of the part Tg1, which is the picking target, is abnormal and outputs the determination result to the UI generation unit 117.

[0030] Based on the processing results of the normal movement distance estimation unit 113, the feature extraction unit 114, the feature movement distance extraction unit 115, and the abnormality degree determination unit 116, the UI generation unit 117 generates a status display screen SC1 (see Figure 8) that visualizes the status of the abnormality detection process of part Tg1 (for example, normal movement distance, feature movement distance, and abnormality degree, etc.) for the operator, and outputs it to the monitor MN.

[0031] Furthermore, the UI generation unit 117 receives operator input on the status display screen SC1 via the mouse MS. The UI generation unit 117 executes processing corresponding to the received operator input and outputs information corresponding to the processing result to the abnormality determination unit 116 or the actuator control unit 111.

[0032] Memory 12 includes, for example, Random Access Memory (RAM) used as work memory when executing each process of the processor 11, and Read Only Memory (ROM) which stores programs and data that define the operation of the processor 11. Data or information generated or acquired by the processor 11 is temporarily stored in RAM. Programs that define the operation of the processor 11 are written to ROM.

[0033] The feature database DB is, for example, flash memory, a hard disk drive (HDD), or a solid state drive (SSD). The feature database DB stores each of the multiple frames captured during the anomaly detection time period T, along with the positional information of the features extracted from each frame.

[0034] The monitor MN is a device that outputs (displays) the status display screen SC1 (see Figure 8) generated by the UI generation unit 117, and is composed of, for example, a Liquid Crystal Display (LCD) or an organic electroluminescence (EL) device.

[0035] The mouse MS is an interface that detects operator input. Note that the mouse MS is just one example of an interface and is not limited to it. The mouse MS may also be implemented as a touch panel integrated with the monitor MN. When the mouse MS receives operator input, it generates an electrical signal based on the operator input and outputs it to the UI generation unit 117.

[0036] Next, we will explain the abnormality detection time period T with reference to Figure 3. Figure 3 is a diagram showing an example of the abnormality detection time period T. Note that the frames FRMi,FRM(i+n) shown in Figure 3 are just examples and are not limited to them.

[0037] The abnormality determination time zone estimation unit 112 calculates the amount of movement of actuator AC in the horizontal direction based on the amount of movement and direction of movement of actuator AC output from encoder EN. Based on the calculated amount of movement of actuator AC in the horizontal direction, the abnormality determination time zone estimation unit 112 estimates the abnormality determination time zone T, which is the time period during which the part Tg1 to be picked is within the field of view of camera CM. Based on the estimated result of the abnormality determination time zone T, the abnormality determination time zone estimation unit 112 determines whether the current time is within the abnormality determination time zone T.

[0038] In the example shown in Figure 3, the time corresponding to the anomaly detection period T is between time ti and time t(i+n). The feature extraction unit 114 acquires the i-th frame FRMi (i: an integer greater than or equal to 1) captured from camera CM at time ti. During the anomaly detection period T, the feature extraction unit 114 repeatedly acquires frames captured by camera CM and acquires the (i+n)th frame FRM(i+n) captured from camera CM at time t(i+n). In other words, the feature extraction unit 114 acquires a total of n frames (n: an integer greater than or equal to 2) during the anomaly detection period T.

[0039] Next, with reference to Figure 4, we will explain the normal travel distance and the characteristic travel distance when there is overlap between parts. Figure 4 is a diagram showing a comparative example of the normal travel distance L11 and the characteristic travel distances L21 and L22. In Figure 4, as in the example shown in Figure 1, we will explain the characteristic travel distance when part Tg2 overlaps part Tg1.

[0040] The normal travel distance L11 is estimated by the normal travel distance estimation unit 113. The normal travel distance estimation unit 113 acquires information on the amount and direction of movement of actuator AC output from encoder EN during the abnormality detection time period T. Based on the conveying speed of part Tg1 by the belt conveyor and the acquired information on the amount and direction of movement of actuator AC, the normal travel distance estimation unit 113 estimates the position of part Tg1 as seen from actuator AC for each time period.

[0041] The normal movement distance estimation unit 113 estimates the normal movement distance L11 of part Tg1 in a predetermined direction during the abnormality determination time period T, based on the estimated position of part Tg1. In this disclosure, the predetermined direction is the direction in which part Tg1 is transported by the belt conveyor, but it may be any direction and is not limited thereto. For example, if part Tg1 is simply stored in tray TRY1 and does not move, the predetermined direction may be the direction in which part Tg1 is transported by the end effector EF, etc.

[0042] In the example shown in Figure 4, the normal movement distance estimation unit 113 estimates that the edges Pt11, Pt12, Pt13, and Pt14 of the component Tg1 stored in tray TRY1 have each moved by a normal movement distance L11 in a predetermined direction. Here, the normal movement distance L11 is estimated based on each edge of the component Tg1, but the normal movement distance L11 may also be estimated based on any position.

[0043] The feature movement distances L21 and L22 are extracted by the feature movement distance extraction unit 115. The feature movement distance extraction unit 115 extracts the positional information of the features of parts Tg1 and Tg2 that are reflected in each of the i-th frame FRMi to (i+n)-th frame FRM(i+n) which are extracted during the abnormality determination time period T and stored in the feature database DB. Based on the transport speed of parts Tg1 and Tg2 by the belt conveyor and the change in the acquired positional information of each part Tg1 and Tg2, the feature movement distances L21 and L22 of parts Tg1 and Tg2 in a predetermined direction are estimated by the feature movement distance extraction unit 115. In Figure 4, for the sake of clarity, an example is shown in which the respective edges of parts Tg1 and Tg2 are extracted as features of parts Tg1 and Tg2.

[0044] In the example shown in Figure 4, the feature displacement distance extraction unit 115 estimates that each of the edges Pt11, Pt12, Pt13, and Pt14 of component Tg1, which is stored in tray TRY1, has moved by a feature displacement distance L21 in a predetermined direction. The feature displacement distance extraction unit 115 also estimates that each of the edges Pt21, Pt22, Pt23, and Pt24 of component Tg2, which is stored in tray TRY1 and overlapping component Tg1, has moved by a feature displacement distance L22 in a predetermined direction.

[0045] Here, the normal movement distance L11, which indicates the distance the edge of component Tg1 moved during the abnormality detection time period T, and the feature movement distance L21 are approximately the same distance. On the other hand, the normal movement distance L11 and the feature movement distance L22 estimate the movement distance of edges of different components. Specifically, since component Tg2 is imaged overlapping component Tg1, the distance between camera CM and component Tg2 becomes smaller. Therefore, the feature movement distance L22, obtained by extracting features of components Tg1 and Tg2 that are captured in the frame, will be a different distance (length) from the feature movement distance L21. Consequently, the difference between the normal movement distance L11 and the feature movement distance L22 is larger than the difference between the normal movement distance L11 and the feature movement distance L21.

[0046] As a result, the image processing device P1 in this disclosure can detect whether or not another part Tg2 is overlapping the part Tg1 that is to be picked, based on a comparison (difference) between the normal movement distance L11 based on the amount of movement of actuator AC and the feature movement distances L21 and L22 obtained by image processing of the captured image.

[0047] Next, with reference to Figure 5, we will explain the normal travel distance and characteristic travel distance when a part is protruding. Figure 5 is a diagram showing a comparative example of the normal travel distance L11 and characteristic travel distances L21 and L23. In Figure 5, we will explain the characteristic travel distance when part Tg1 is protruding from tray TRY1. Note that the normal travel distance shown in Figure 5 is the same as the normal travel distance shown in Figure 4, so we will omit the explanation.

[0048] The feature movement distances L21 and L22 are extracted by the feature movement distance extraction unit 115. The feature movement distance extraction unit 115 extracts the positional information of the features of part Tg1 that are reflected in each of the i-th frame FRMi to the (i+n)th frame FRM(i+n) which are extracted during the abnormality determination time period T and stored in the feature database DB. Based on the transport speed of part Tg1 by the belt conveyor and the change in the acquired positional information of each feature of part Tg1, the feature movement distances L21 and L23 of part Tg1 in a predetermined direction are estimated by the feature movement distances L21 and L23 of part Tg1. In Figure 5, for the sake of clarity, an example is shown in which each edge of part Tg1 is extracted as a feature of part Tg1.

[0049] In the example shown in Figure 5, the feature displacement distance extraction unit 115 estimates that each of the edges Pt11 and Pt12 of the component Tg1 stored in tray TRY1 has moved by a feature displacement distance L21 in a predetermined direction. The feature displacement distance extraction unit 115 also estimates that each of the edges Pt13 and Pt14 of the component Tg1 that extends beyond tray TRY1 has moved by a feature displacement distance L23 in a predetermined direction.

[0050] Here, the normal movement distance L11 and feature movement distance L21, which indicate the distance the edge of part Tg1 moved during the abnormality detection time period T, and the normal movement distance L11 and feature movement distance L23 estimate the movement distance of the edge of different parts. Specifically, part Tg1 is imaged with its orientation tilted, and a part of it protrudes into the part of tray TRY1 that is not the storage area. Therefore, the distance between camera CM and part Tg1 is slightly greater for the part of part Tg1 that is stored in tray TRY1 compared to the normal orientation. Also, the distance between camera CM and part Tg1 is closer for the part of part Tg1 that protrudes from tray TRY1 compared to the normal orientation and for the part of part Tg1 that is stored in tray TRY1. Thus, the feature movement distance L23 obtained by extracting the features of part Tg1 that are captured in the frame will be different distances (lengths) from feature movement distance L21 (i.e., feature movement distance L21 < normal movement distance L11 < feature movement distance L23). In Figure 5, the change in distance from the camera CM is greater because a large portion of component Tg1 extends beyond the tray TRY1. Therefore, the difference between the normal movement distance L11 and the feature movement distance L23 is greater than the difference between the normal movement distance L11 and the feature movement distance L21.

[0051] As a result, the image processing device P1 in this disclosure can detect that the part Tg1 to be picked is protruding from the tray TRY1 based on a comparison (difference) between the normal movement distance L11 based on the amount of movement of actuator AC and the feature movement distances L21 and L23 obtained by image processing of the captured image.

[0052] Next, an example of the operation procedure of the image processing device P1 will be described with reference to Figures 6 and 7, respectively. Figure 6 is a flowchart of an example of the operation procedure of the image processing device P1 in the embodiment. Figure 7 is a flowchart of an example of the operation procedure of the image processing device P1 in the embodiment.

[0053] The processor 11 of the image processing device P1 resets the number of frames F for feature extraction (F=0 (zero)) and starts the initial setup of the anomaly detection time period T (St11). The actuator control unit 111 starts controlling the end effector EF to move toward part Tg1 (object), which is the current picking target, in order to pick it.

[0054] The abnormality detection time zone estimation unit 112 acquires information on the amount and direction of movement of actuator AC from encoder EN (St12). Based on the movement speed and direction of part Tg1 and the acquired information on the amount and direction of movement of actuator AC, the abnormality detection time zone estimation unit 112 estimates the time period during which part Tg1 (object) is imaged by camera CM and sets the abnormality detection time zone T (St13).

[0055] The abnormality detection time zone estimation unit 112 determines whether the current time is within the abnormality detection time zone T (St14).

[0056] If the anomaly detection time zone estimation unit 112 determines in step St14 that the current time is within the anomaly detection time zone T (St14, YES), it increments the number of frames F from which features are to be extracted (F = F + 1) (St15).

[0057] On the other hand, in step St14, if the anomaly detection time zone estimation unit 112 determines that the current time is not within the anomaly detection time zone T after the anomaly detection time zone T has ended (St14, NO), it increments the number of frames F from which features are to be extracted (F = F + 1) (St15). Note that if the anomaly detection time zone estimation unit 112 determines that the current time is not within the anomaly detection time zone T before the anomaly detection time zone T has ended, it repeatedly executes the process in step St14 until the anomaly detection time zone T has arrived.

[0058] The normal travel distance estimation unit 113 estimates the normal travel distance L11 based on the information of the amount of movement and direction of movement of the actuator AC output from the encoder EN (St16).

[0059] The feature extraction unit 114 acquires image data of the F-th frame captured by the camera CM (St17). The feature extraction unit 114 determines whether the current F-value is F=1 or not (St18).

[0060] If the feature extraction unit 114 determines in step St18 that the current F-value is F=1 (St18, YES), it extracts features of component Tg1 from the first frame and selects robust features from the extracted features (St19). The feature extraction unit 114 associates the feature quantity of the selected features from the first frame with the feature's position information and the frame number (i.e., F-value=1) and stores them in the feature database DB (St21).

[0061] On the other hand, if the feature extraction unit 114 determines in step St18 that the current F value is not F=1 (St18, NO), it reads out the feature quantities and position information of the features extracted (selected) from the (F-1)th frame stored in the feature database DB (St20). The feature extraction unit 114 searches for the feature quantities corresponding to the features read out from the (F-1)th frame in the Fth frame (St20).

[0062] The feature extraction unit 114 associates the feature quantities of the features selected or searched from the F-th frame with the feature location information and the frame number (i.e., the F-value) and stores them in the feature database DB (St21).

[0063] After the abnormality determination time period T ends, the feature movement distance extraction unit 115 reads out the feature quantity, feature location information, and frame number information stored in the feature database DB (St22). The feature movement distance extraction unit 115 applies a time domain filter to the read feature quantity to obtain the movement distance (feature movement distance) of component Tg1 in a predetermined direction during the abnormality determination time period T (St22).

[0064] The abnormality determination unit 116 compares the feature movement distance with the normal movement distance and calculates an abnormality level (St23) that indicates the degree of abnormality in the placement of part Tg1, such as part Tg1 overhanging the tray TRY1 or part Tg1 overlapping with another part (e.g., part Tg2). Specifically, the abnormality determination unit 116 calculates the abnormality level based on the sum of the differences between the movement distance corresponding to each feature obtained by image processing and the movement distance corresponding to each edge of part Tg1 obtained based on the amount of movement of actuator AC.

[0065] The abnormality determination unit 116 determines whether the calculated abnormality level is equal to or greater than the threshold Th (St24). The threshold Th is the abnormality level value indicating that component Tg1 is abnormal, that is, that another component is overlapping component Tg1.

[0066] If the abnormality determination unit 116 determines in step St24 that the abnormality level is equal to or greater than the threshold Th (St24, YES), it instructs the actuator control unit 111 to control the current part Tg1 (object) (St25). The actuator control unit 111 performs the following controls for the current object: skipping the picking of part Tg1, stopping the operation of actuator AC, or outputting an alarm to notify that an abnormality has been detected. The controls for the current object may be, but are not limited to, controls corresponding to the intended use of the technology of this disclosure.

[0067] On the other hand, if the abnormality determination unit 116 determines in step St24 that the abnormality level is not equal to or greater than the threshold Th (St24, NO), it instructs the actuator control unit 111 to pick the current part Tg1 (target object). The processor 11 also clears the various data (various information) stored in the feature database DB (St28).

[0068] The UI generation unit 117 generates a status display screen SC1 (see Figure 8) that visualizes the status of the characteristic movement distance of the component Tg1 (object) acquired (extracted) at the current Fth frame (movement distance status), the status of the normal movement distance estimated at the current Fth frame (normal movement distance status), and the status of the degree of abnormality of each component (object) that has been processed for abnormality detection so far (degree of abnormality status), and outputs it to the monitor MN (St26).

[0069] The UI generation unit 117 receives operator input on the status display screen SC1 output to the monitor MN via the mouse MS and executes processing corresponding to the operator input (St27). Note that the processing in step St27 is not mandatory, and some or all of it may be omitted.

[0070] For example, based on operator operations, the UI generation unit 117, as a feedback process for operator judgment results regarding the degree of abnormality, receives an operator operation to correct the abnormality detection result of a part (object), and feeds back the operator judgment result, which includes the abnormality detection result of the part corrected by the operator operation and information on the degree of abnormality of this part, to the abnormality degree determination unit 116 and stores it (St27).

[0071] Furthermore, for example, if the UI generation unit 117 receives a threshold recalculation request from an operator, it causes the abnormality determination unit 116 to recalculate and reset the threshold Th based on the accumulated operator judgment results (St27). The abnormality determination unit 116 calculates the variance, average, or maximum value of the abnormality degree based on the abnormality detection result or operator judgment result for each part, and sets it as the new threshold Th.

[0072] The abnormality determination unit 116 clears the various data (various information) stored in the feature database DB (St28), and then instructs the actuator control unit 111 to pick the current part Tg1 (target object). The processor 11 also clears the various data (various information) stored in the feature database DB (St28). The processor 11 then determines whether or not there is another part (target object) (St29).

[0073] If the processor 11 determines in step St29 that there is the next part (object) (St29, YES), it returns to the process in step St11.

[0074] On the other hand, if the processor 11 determines in step St29 that there is no next component (object) (St29, NO), it terminates its operation.

[0075] As described above, the image processing device P1 according to the embodiment can detect whether or not the part Tg1 to be picked is sticking out, or whether another part is overlapping part Tg1, based on the difference between the travel distance of the actuator AC output from the encoder EN (normal travel distance) and the amount of movement of the features of parts Tg1 and Tg2 based on the image data captured by the camera CM (feature travel distance).

[0076] Furthermore, the image processing device P1 can improve the detection accuracy of abnormalities related to the placement of component Tg1 by performing feedback processing of the operator's judgment result regarding the degree of abnormality and threshold recalculation processing based on the operator's operation. Specific examples of the feedback processing of the operator's judgment result regarding the degree of abnormality and the threshold recalculation processing will be described later.

[0077] Next, the status display screen SC1 will be described with reference to Figure 8. Figure 8 is a diagram showing an example of the status display screen SC1. Note that the status display screen SC1 shown in Figure 8 is just an example and is not limited to this.

[0078] The status display screen SC1 is generated by the UI generation unit 117 and output to the monitor MN. The status display screen SC1 includes a travel distance status display area AR11, a normal travel distance status display area AR12, an abnormality degree status display area AR13, operator judgment buttons BT11 and BT12, and a threshold recalculation button BT13.

[0079] The movement distance status display area AR11 is an area that displays the status of the feature movement distance of the parts extracted at the current Fth frame. The movement distance status display area AR11 includes the abnormal parts collection display button BT21, the current parts display button BT22, and the frame FRM31.

[0080] The abnormal parts display button BT21 accepts display operations to display parts that have been determined to be abnormal, or parts that have been determined to be abnormal by the operator. When the abnormal parts display button BT21 is pressed (selected) by the operator, the UI generation unit 117 displays captured images (frames) of parts that have been previously determined to be abnormal in the movement distance status display area AR11.

[0081] The current part display button BT22 accepts a display operation to display frame FRM31, which is an image of the part Tg1 (object) currently being picked. When the current part display button BT22 is pressed (selected) by the operator, the UI generation unit 117 displays the image (frame FRM31) of the part Tg1 (object) currently being picked in the movement distance status display area AR11. As a result, the UI generation unit 117 switches between the image of a part that has been determined to be abnormal and the image of the part Tg1 currently being picked, supporting the operator's visual confirmation work regarding whether or not part Tg1 is abnormal.

[0082] The UI generation unit 117 superimposes feature points (white dots (circles) shown in Figure 8) that visualize the positions of robust features selected from the first frame FRM1 onto the first frame FRM1. Note that the frame FRM31 shown in Figure 8 shows part Tg31 of "part No. 1024" and part Tg32 which is superimposed on part Tg31, and feature points that visualize the positions of robust features selected from part Tg31 and part Tg32 are superimposed.

[0083] The UI generation unit 117 generates a frame FRM31 by superimposing a straight line on each feature point (white dots (○) shown in Figure 8) superimposed on the frame FRM31, with the direction corresponding to the movement direction of the components Tg31 and Tg32 (objects) extracted at the current F-th frame, and the length corresponding to the feature movement distance. The UI generation unit 117 displays the generated frame FRM31 in the movement distance status display area AR11.

[0084] The normal travel distance status display area AR12 is an area that displays the status of the normal travel distance of actuator AC as estimated at the current time (i.e., when the Fth frame has been acquired). The UI generation unit 117 estimates the position of the feature points of component Tg31 as seen from actuator AC at the time corresponding to the first frame. The UI generation unit 117 superimposes the estimated feature points of component Tg31 (black dots (●) shown in Figure 8) onto the frame FRM41, which is modeled after the field of view of camera CM. The UI generation unit 117 generates a frame FRM41 by superimposing a straight line on each feature point (black dots (●) shown in Figure 8) superimposed on the frame FRM41, having a direction corresponding to the direction of movement of component Tg31 extracted at the current time and a length corresponding to the feature travel distance. The UI generation unit 117 displays the generated frame FRM41 in the normal travel distance status display area AR12.

[0085] The abnormality status display area AR13 is an area that displays the distribution of abnormality levels of multiple parts relative to a threshold Th, based on the abnormality level of each part (target object) for which abnormality detection processing has been completed. The UI generation unit 117 generates a graph plotting the abnormality levels of multiple parts Tg21, Tg22, Tg23, Tg24, Tg25, Tg26, Tg27, and Tg28, which have been calculated so far, in a time series, and displays it in the abnormality status display area AR13.

[0086] The abnormality distribution shown in Figure 8 indicates that the abnormality levels of parts Tg21-Tg24, Tg26, and Tg28 are below the threshold Th, meaning no abnormalities were detected in these parts. The abnormality distribution also indicates that the abnormality levels of parts Tg25 and Tg27 are above the threshold Th, meaning abnormalities were detected in these parts.

[0087] The operator judgment buttons BT11 and BT12 accept the operator judgment results regarding the degree of abnormality for each of the parts Tg21 to Tg28 for which the degree of abnormality has already been calculated, based on operator operation.

[0088] For example, if the operator selects a point (a black dot (●) shown in Figure 8) indicating either part Tg25 or Tg27 whose abnormality level is above the threshold Th, the UI generation unit 117 changes the abnormality detection result of the selected part to "normal" when the operator judgment button BT11 is pressed (selected). The UI generation unit 117 associates the changed abnormality detection result "normal" with the identification information of this part (e.g., "part no.") and the abnormality level (score), and feeds this back to the abnormality level determination unit 116.

[0089] Furthermore, for example, if the operator selects a point (white dot (○) shown in Figure 8) indicating any of the parts Tg21~Tg24, Tg26, or Tg28 whose degree of abnormality is less than the threshold Th, the UI generation unit 117 changes the abnormality detection result of the selected part to "abnormal" when the operator judgment button BT12 is pressed (selected). The UI generation unit 117 associates the changed abnormality detection result "abnormal" for the part with the identification information of this part (e.g., "part No.") and the degree of abnormality (score), and feeds this back to the abnormality degree determination unit 116.

[0090] Furthermore, in actual operation, it can be difficult to determine if a part has an abnormality level close to the threshold. In such cases, the abnormality level determination unit 116 instructs the actuator control unit 111 to stop the operation of actuator AC and outputs an alert to the operator. This alert may be displayed on the UI generation unit 117 or output as sound via a speaker (not shown). The operator determines whether there is an abnormality in the placement of part Tg1 based on the image of part Tg1 (frame FRM31) displayed on the monitor MN, and inputs the abnormality determination result for part Tg1 by selecting either the operator determination button BT11 or the operator determination button BT12. Based on this selection operation by the operator using either the operator determination button BT11 or the operator determination button BT12, the UI generation unit 117 recalculates the threshold Th or skips the picking operation.

[0091] The threshold recalculation button BT13 recalculates the threshold Th based on the degree of abnormality (score) of parts Tg21 to Tg28 and the abnormality detection results for parts Tg21 to Tg28. For example, if the abnormality detection result for part Tg27 is changed to "normal" by operator operation, the UI generation unit 117 recalculates the threshold Th at which the degree of abnormality (score) of parts Tg21 to Tg24 and Tg26 to Tg28 are determined to be "normal". The UI generation unit 117 feeds back the information of the recalculated threshold Th to the abnormality degree determination unit 116 and sets it as the new threshold Th.

[0092] Furthermore, the recalculation of the threshold Th may be performed by excluding the degree of abnormality of parts that have been determined to be abnormal by an operator, in order to exclude outliers that reduce the accuracy of the threshold from the recalculation, or it may be performed by targeting the degree of abnormality of all parts, including those that have been determined to be abnormal by an operator.

[0093] In this disclosure, the operator judgment buttons BT11 and BT12 and the threshold recalculation button BT13 are displayed as different buttons on the monitor MN. This allows the UI generation unit 117 to distinguish between the degree of abnormality determined by the operator and the degree of abnormality determined by the degree of abnormality determination unit 116, based on the selection operation of the operator judgment buttons BT11 and BT12. Therefore, when recalculating the threshold Th, the UI generation unit 117 can exclude the degree of abnormality of the part that the operator has determined to be abnormal from the degree of abnormality used in the recalculation.

[0094] (Note) Based on the descriptions of the embodiments described above, the following technologies are disclosed.

[0095] (Technology 1) An anomaly detection device (image processing device P1) that moves integrally with actuator AC and is capable of communicating with a camera CM that can image component Tg1, An acquisition unit (processor 11) acquires the amount of movement of the actuator AC and a plurality of captured images of the component Tg1, An estimation unit (normal travel distance estimation unit 113) estimates the first travel distance (normal travel distance) of the component Tg1 based on the amount of movement of the actuator AC, An extraction unit (feature extraction unit 114 and feature movement distance extraction unit 115) extracts features of the component Tg1 that appear in the plurality of captured images, and extracts a second movement distance (feature movement distance) of the component Tg1 based on the positional information of the extracted features, The system includes a determination unit (abnormality determination unit 116) that performs an abnormality detection process to determine whether or not there is an abnormality in the arrangement of the component Tg1 based on the difference between the first movement distance (normal movement distance) and the second movement distance (characteristic movement distance), Anomaly detection device (image processing device P1). As a result, the image processing device P1 can detect whether the part Tg1 to be picked is sticking out or whether another part Tg2 is overlapping part Tg1, based on a comparison (difference) between the normal movement distance L11, which is based on the amount of movement of the actuator AC that moves integrally with the camera CM, and the feature movement distance L21, which is obtained by image processing of the captured image taken by the camera CM.

[0096] (Technology 2) The estimation unit (abnormality determination time period estimation unit 112) estimates the time period (abnormality determination time period T) during which the component Tg1 is imaged by the camera CM based on the amount of movement of the actuator AC, and estimates the first travel distance (normal travel distance) based on the amount of movement of the actuator AC during the time period (abnormality determination time period T). The extraction unit (feature extraction unit 114 and feature movement distance extraction unit 115) extracts the second movement distance (feature movement distance) based on the positional information of the features of the component Tg1 captured in the plurality of captured images taken during the time period (anomaly determination time period T). An anomaly detection device (image processing device P1) as described in (Technology 1). As a result, the image processing device P1 defines a predetermined time period during which part Tg1 is imaged by camera CM as an abnormality detection time period T. By calculating a comparison (difference) between the normal movement distance L11 based on the amount of movement of actuator AC during this abnormality detection time period T and the feature movement distance L21 obtained by image processing of the captured image, the device can detect whether or not part Tg1, which is the target of picking, is protruding, or whether or not another part Tg2 is overlapping part Tg1.

[0097] (Technology 3) The determination unit (abnormality determination unit 116) calculates an abnormality degree based on the difference between the first movement distance (normal movement distance) and the second movement distance (characteristic movement distance), which evaluates the likelihood that the arrangement of the component Tg1 is abnormal. The system further includes a generation unit (UI generation unit 117) that generates and outputs a screen (status display screen SC1) that visualizes a threshold Th indicating an abnormality in the arrangement of the component Tg1, and the degree of abnormality of component Tg1 calculated in the past relative to the threshold Th. An anomaly detection device (image processing device P1) as described in (Technology 1) or (Technology 2). As a result, even if there are individual differences or unique characteristics in parts due to lot variations, the image processing device P1 can visualize the degree of abnormality and threshold Th of each of the multiple parts used in the abnormality detection process, allowing the operator to see whether the threshold Th used in the abnormality detection process is appropriate for the actual parts.

[0098] (Technology 4) The generation unit (UI generation unit 117) is capable of receiving operator input and modifies the result of the abnormality detection process for a selected part Tg1 from among the abnormality levels of the parts Tg1. An anomaly detection device (image processing device P1) as described in (Technical 3). As a result, if the image processing device P1 obtains an incorrect result from the anomaly detection process, it can accept a correction (change) of the anomaly detection result in order to improve the accuracy of subsequent anomaly detection processes.

[0099] (Technology 5) The generation unit (abnormality determination unit 116) recalculates the threshold Th based on the abnormality degree of the part Tg1 whose abnormality detection result has been changed, and the abnormality degree of the past abnormality detection process for part Tg1, and updates the threshold Th to the recalculated threshold. An anomaly detection device (image processing device P1) as described in (Technical 4). As a result, the image processing device P1 can update the threshold Th used in the anomaly detection process to a threshold Th that is more suitable for the actual part, based on the results of the anomaly detection process of the actual part. Therefore, the image processing device P1 can improve the judgment accuracy of the anomaly detection process.

[0100] (Technology 6) The generation unit (UI generation unit 117) generates and outputs the screen (status display screen SC1) which includes information on the first travel distance (normal travel distance) at the current time and information on the second travel distance (characteristic travel distance) at the current time. An anomaly detection device (image processing device P1) as described in (Technical 3). As a result, even if there are individual differences or unique characteristics in parts due to lot variations, etc., the image processing device P1 can compare the normal travel distance and the characteristic travel distance at the current time and visualize them in real time, thereby supporting the operator's judgment as to whether the abnormality detection processing result for part Tg1 is correct or incorrect.

[0101] (Technology 7) The system further includes a control unit (actuator control unit 111) that causes the actuator AC to execute control on the component Tg1 based on the result of the abnormality detection process performed by the determination unit (abnormality degree determination unit 116), An anomaly detection device (image processing device P1) described in any one of (Technology 1) to (Technology 6). This allows the image processing device P1 to cause the actuator AC to perform control based on the results of the abnormality detection process for component Tg1.

[0102] (Technology 8) The control unit (actuator control unit 111), when the determination unit (abnormality determination unit 116) determines that there is an abnormality in the component Tg1, executes control to skip control of the component Tg1 by the actuator AC, control to stop the actuator AC, or control to issue an alarm notifying that an abnormality has been determined in the arrangement of the component Tg1. An anomaly detection device (image processing device P1) as described in (Technical 7). This allows the image processing device P1 to cause the actuator AC to execute control on component Tg1, which has been found to be "abnormal" as a result of the abnormality detection process.

[0103] (Technology 9) An anomaly detection method performed by a computer (image processing device P1) that moves integrally with actuator AC and can communicate with a camera CM capable of imaging component Tg1, The amount of movement of the actuator AC and a plurality of captured images of the component Tg1 are acquired. Based on the amount of movement of the actuator AC, the first travel distance (normal travel distance) of the component Tg1 is estimated. Features of the component Tg1 captured in the plurality of captured images are extracted, and based on the positional information of the extracted features, a second movement distance (feature movement distance) of the component Tg1 is extracted. Based on the difference between the first travel distance (normal travel distance) and the second travel distance (feature travel distance), an anomaly detection process is performed to determine whether or not there is an anomaly in the arrangement of the component Tg1. Anomaly detection method. As a result, the image processing device P1 can detect whether the part Tg1 to be picked is sticking out or whether another part Tg2 is overlapping part Tg1, based on a comparison (difference) between the normal movement distance L11, which is based on the amount of movement of the actuator AC that moves integrally with the camera CM, and the feature movement distance L21, which is obtained by image processing of the captured image taken by the camera CM.

[0104] Although various embodiments have been described above with reference to the attached drawings, this disclosure is not limited to such examples. It will be clear to those skilled in the art that various modifications, alterations, substitutions, additions, deletions, and equivalents can be conceived within the scope of the claims, and these will also be understood to fall within the technical scope of this disclosure. Furthermore, the components of the various embodiments described above can be combined arbitrarily without departing from the spirit of the invention. [Industrial applicability]

[0105] This disclosure is useful as an anomaly detection device and anomaly detection method for detecting parts protruding or overlapping parts. [Explanation of symbols]

[0106] 11 processors 12 memory 100 Picking Systems 111 Actuator Control Unit 112 Anomaly detection time estimation unit 113 Normal movement distance estimator 114 Feature Extraction Unit 115 Feature Movement Distance Extraction Unit 116 Abnormality Degree Determination Unit 117 UI generation part AC Actuator AR11 Distance traveled display area AR12 Normal travel distance status display area AR13 Abnormality Status Display Area BT11, BT12 Operator detection button BT13 Threshold Recalculation Button CM Camera DB Feature Database EF End Effector EN encoder FRM, FRM1, FRM31, FRM41, FRMi, FRM(i+n) Frames L11 Normal travel distance L21, L22 Feature Movement Distance MN Monitor MS Mouse P1 Image Processing Device SC1 Status Display Screen T Abnormality detection time period Tg1, Tg2 parts Th threshold TRY1 Tray

Claims

1. An anomaly detection device that moves integrally with an actuator and can communicate with a camera capable of imaging parts, An acquisition unit that acquires the amount of movement of the actuator and a plurality of captured images of the component, An estimation unit that estimates the first movement distance of the component based on the amount of movement of the actuator, An extraction unit extracts features of the component captured in the plurality of captured images, and extracts a second movement distance of the component based on the positional information of the extracted features. The system includes a determination unit that performs an abnormality detection process to determine whether or not there is an abnormality in the arrangement of the component based on the difference between the first movement distance and the second movement distance. Anomaly detection device.

2. The estimation unit estimates the time period during which the component is imaged by the camera based on the amount of movement of the actuator, and estimates the first travel distance based on the amount of movement of the actuator during that time period. The extraction unit extracts the second travel distance based on the positional information of the features of the component captured in the plurality of captured images taken during the time period. An anomaly detection device according to claim 1.

3. The determination unit calculates a degree of abnormality, which evaluates the likelihood that the arrangement of the part is abnormal, based on the difference between the first movement distance and the second movement distance. The system further includes a generation unit that generates and outputs a screen visualizing a threshold value indicating an abnormality in the arrangement of the aforementioned components, and the degree of abnormality of the components calculated in the past in relation to the threshold value. An anomaly detection device according to claim 1.

4. The generation unit is capable of receiving operator input and modifies the result of the abnormality detection process for selected parts among the abnormality levels of the parts. An anomaly detection device according to claim 3.

5. The generation unit recalculates the threshold based on the degree of abnormality of the part whose abnormality detection result has been changed, and the results and degree of abnormality of past parts, and updates the threshold to the recalculated threshold. An anomaly detection device according to claim 4.

6. The generation unit generates and outputs the screen which includes information on the first travel distance at the current time and information on the second travel distance at the current time. An anomaly detection device according to claim 3.

7. The system further includes a control unit that causes the actuator to perform control on the component based on the result of the abnormality detection process by the determination unit, An anomaly detection device according to claim 1.

8. If the determination unit determines that there is an abnormality in the component, the control unit executes a control to skip control of the component by the actuator, a control to stop the actuator, or a control to issue an alarm notifying that an abnormality has been determined in the placement of the component. An anomaly detection device according to claim 7.

9. An anomaly detection method performed by a computer that moves integrally with an actuator and can communicate with a camera capable of imaging parts, The amount of movement of the actuator and a plurality of captured images of the component are acquired. Based on the amount of movement of the actuator, the first movement distance of the component is estimated. Features of the component captured in the plurality of captured images are extracted, and based on the positional information of the extracted features, a second movement distance of the component is extracted. Based on the difference between the first movement distance and the second movement distance, an abnormality detection process is performed to determine whether or not there is an abnormality in the arrangement of the component. Anomaly detection method.

Images