Image processing method, program, image processing system, image processing device, and detection device
The image processing method employs a dual-detector system to enhance object detection accuracy and efficiency by using a second detector with higher recognition accuracy and automatic learning on a separate computer, addressing the limitations of low-processing-power hardware in existing technologies.
Patent Information
- Application Number
- PCT/JP2023/017971
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-12
- Publication Date
- 2024-11-21
AI Technical Summary
Existing image processing technologies face challenges in detecting objects with high accuracy while maintaining processing efficiency, particularly when hardware with low processing power is used, and there is a need to prioritize processing speed over recognition accuracy.
An image processing method that employs a two-detector system, where a first detector performs initial object detection and a second detector with higher recognition accuracy is engaged when necessary, with a determination unit comparing output results to identify errors and trigger the second detector for improved accuracy, and learning processing is performed on a separate, higher-performance computer to enhance the first detector's model.
This approach allows for accurate object detection while maintaining processing efficiency, automatically improving the first detector's accuracy by using the second detector's results as learning data and reducing the need for manual operation, and enables efficient learning processing on more powerful computers.
Smart Images

Figure JP2023017971_21112024_PF_FP_ABST
Abstract
Description
Image processing method, program, image processing system, image processing device, and detection device
[0001] The present disclosure relates to an image processing method, a program, an image processing system, an image processing device, and a detection device.
[0002] There is known a technology for detecting an object using a vision sensor attached to a robot or a predetermined position and controlling the robot's operation. Object detection focuses on one feature, such as a contour, and therefore detection may not be possible if the feature becomes invisible due to changes in brightness, etc. By using deep learning or machine learning, it is possible to learn appropriate features depending on the object, thereby increasing the robustness of object detection. Examples of technologies related to this type of learning are described in Patent Documents 1 and 2.
[0003] JP 2020-21326 A JP 2022-119754 A
[0004] To detect objects using a model trained through deep learning, hardware with relatively high processing power is required. Processing technologies that can be executed on hardware with low processing power have been developed, but the recognition accuracy of the model is limited by the hardware performance. On the other hand, there are cases where processing speed and reduced computer load are prioritized over accuracy in object detection processing.
[0005] The present disclosure has been made in consideration of the above-mentioned problems, and aims to provide an image processing method, a program, an image processing system, an image processing device, and a detection device that can detect objects with high accuracy while maintaining processing efficiency.
[0006] One aspect of the present disclosure is an image processing method for extracting an object from an image, the image processing method including: a step of capturing an image of the object with a visual sensor; a first detection step of detecting or inspecting the object from the image captured by the visual sensor with a first detector; an execution judgment step of determining whether detection or inspection by a second detector is necessary; a second detection step of executing a detection process by the second detector if it is determined that detection or inspection by the second detector is necessary; and a judgment step of comparing the output result of the first detector with the output result of the second detector to determine whether there is an error in the output result of the first detector.
[0007] One aspect of the present disclosure is a program that causes a computer that executes a process of extracting an object from an image to execute an acquisition process function that captures an image of the object using a visual sensor, a first detection process function that uses a first detector to detect or inspect the object from the image captured by the visual sensor, an execution judgment function that determines whether detection or inspection by a second detector is necessary, a second detection process function that executes a detection process by the second detector when it is determined that detection or inspection by the second detector is necessary, and a judgment process function that compares the output result of the first detector with the output result of the second detector to determine whether there is an error in the output result of the first detector.
[0008] One aspect of the present disclosure is an image processing system that extracts an object from an image, comprising: an acquisition processing unit that captures an image of the object using a visual sensor; a first detector that detects or inspects the object from the image captured by the visual sensor; a second detector that detects or inspects the object from the image captured by the visual sensor; an execution judgment unit that determines whether detection or inspection by the second detector is necessary; and an appropriateness judgment unit that, when it is determined that detection or inspection by the second detector is necessary, compares the output result of the first detector with the output result of the second detector to determine whether there is an error in the output result of the first detector.
[0009] One aspect of the present disclosure is an image processing device that extracts an object from an image, comprising: an acquisition processing unit that captures an image of the object using a visual sensor; a first detector that detects or inspects the object from the image captured by the visual sensor; a second detector that detects or inspects the object from the image captured by the visual sensor; an execution judgment unit that determines whether detection or inspection by the second detector is necessary; and an appropriateness judgment unit that, when it is determined that detection or inspection by the second detector is necessary, compares the output result of the first detector with the output result of the second detector to determine whether there is an error in the output result of the first detector.
[0010] One aspect of the present disclosure is a detection device that detects or inspects an object, comprising: a first detector that detects or inspects the object; a second detector that detects or inspects the object; an execution judgment unit that determines whether detection or inspection by the second detector is necessary; and an appropriateness judgment unit that, when it is determined that detection or inspection by the second detector is necessary, compares the output result of the first detector with the output result of the second detector to determine whether there is an error in the output result of the first detector.
[0011] It is a diagram showing the configuration of an image processing system according to the present embodiment. It is a diagram showing the functional configuration of the image processing system. It is a flowchart showing the flow of image processing in the image processing system. It is a diagram showing the functional configuration of an image processing apparatus according to a modified example.
[0012] An example of an embodiment of the present invention will be described below. Fig. 1 is a diagram showing the configuration of an image processing system 100 according to this embodiment. The image processing system 100 shown in Fig. 1 is also a robot system that processes an object W using image processing technology.
[0013] The image processing system 100 of this embodiment includes a control device 1, an image processing device 2, a robot 3, an arm 4, a vision sensor 5, an input device 7, a display device 8, and an external analysis unit 10.
[0014] The control device 1 is a computer that controls the operation of the robot 3 based on a predetermined program for processing a workpiece, i.e., an object W. The control device 1 controls the operation of the robot 3 based on the results of image processing output from the image processing device 2.
[0015] The image processing device 2 is a computer that controls the vision sensor 5 and performs image processing on images captured by the vision sensor 5. The image processing device 2 outputs the results of the image processing to the control device 1. The image processing device 2 may be composed of multiple computers.
[0016] The image processing device 2 may have calibration data that defines the relative positional relationship between the vision sensor 5 and the robot 3, the internal parameters of the vision sensor 5, etc. This makes it possible to convert positions on an image captured by the vision sensor 5 into positions on a coordinate system (such as a robot coordinate system) fixed to the working space.
[0017] The robot 3 is, for example, an articulated robot, and a hand or tool is attached to the tip of an arm 4 of the robot 3. The robot 3 performs work such as handling or processing of an object W on a stand 6 under the control of the control device 1. In addition, a vision sensor 5 is attached to the tip of the arm 4 of the robot 3.
[0018] The type of robot 3 is not particularly limited, and may be another type of robot. Furthermore, the vision sensor 5 does not have to be attached to the robot 3, and may be fixedly installed at a predetermined position, for example.
[0019] The vision sensor 5 captures an image of the target object W under the control of the image processing device 2. The vision sensor 5 may be a camera that captures grayscale images or color images, or may be a stereo camera or a three-dimensional sensor that can acquire distance images or three-dimensional point clouds. In this embodiment, the vision sensor 5 is assumed to be calibrated.
[0020] The input device 7 is an interface for an operator using the image processing system 100 to perform various operations. The input device 7 may be configured with buttons, a keyboard, etc., a touch panel display, or a computer equipped with an operation unit. In this way, the configuration of the input device 7 is not particularly limited.
[0021] The display device 8 is an output device that notifies the operator of information relating to the use of the image processing system 100 by image or sound. The display device 8 is configured by a display, a speaker, etc. The configuration of the display device 8 is not particularly limited.
[0022] The external analysis unit 10 performs image processing on the image captured by the vision sensor 5. The external analysis unit 10 is connected to the image processing device 2 via a network such as a dedicated line or the Internet. The external analysis unit 10 is a second image processing device realized by a computer different from the image processing device 2 or by cloud computing. The external analysis unit 10 performs image processing on the same image as the image processed by the image processing device 2.
[0023] There is no particular limitation on the route by which the external analysis unit 10 acquires the image captured by the vision sensor 5. The external analysis unit 10 may acquire the image via the image processing device 2, or via the robot 3 or the control device 1.
[0024] The image processing function of this embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing the functional configuration of the image processing system 100.
[0025] First, we will explain the image processing device 2. As shown in Fig. 2, the image processing device 2 includes a control unit 11 and a storage unit 12. The control unit 11 is a processor such as a CPU (Central Processing Unit), and realizes various functions by executing programs stored in the storage unit 12.
[0026] The storage unit 12 is a storage device such as a ROM (Read Only Memory) or RAM (Random Access Memory) that stores an OS (Operating System), application programs, etc., a hard disk drive or SSD (Solid State Drive) that stores various other information. The storage unit 12 stores various information such as a vision program that controls the vision sensor 5, for example.
[0027] The image processing device 2 of this embodiment includes, as functional units executed by the control unit 11, an acquisition processing unit 13, a first detector 14, an execution judgment unit 15, a suitability judgment unit 16, and a notification processing unit 17.
[0028] The acquisition processing unit 13 is an acquisition processing function that acquires images captured by the vision sensor 5. The acquisition processing unit 13 transmits a vision execution command based on a preset vision program to the vision sensor 5, causing the vision sensor 5 to capture an image of the target object W. The vision program is a program stored in the memory unit 12 that describes processing details related to vision.
[0029] The first detector 14 has a first detection processing function that detects or inspects an object W from an image acquired by the vision sensor 5. The detection or inspection may include abnormality detection and class classification in addition to object extraction.
[0030] The first detector 14 executes a process of detecting the object W from the image based on a model generated by, for example, deep learning or machine learning. Note that the detection process by the first detector 14 may use pattern matching using a preset model pattern of the object W.
[0031] The execution decision unit 15 is an execution decision function that determines whether or not detection or inspection by the second detector 23, which has relatively higher recognition accuracy than the first detector 14, is necessary. The recognition accuracy can be determined based on an evaluation index of the model of the first detector 14. The evaluation index is not particularly limited. For example, the evaluation index can be the recall rate, which indicates the probability that the predicted value by the model in all data in which the correct answer (true value) is "positive" (positive example) is also "positive." In addition, examples of evaluation indexes that can be evaluated in a single execution include the match rate with the model pattern, whether the detected workpiece was successfully grasped, whether the number of detected objects matches a predetermined number, and the confidence level of the learner.
[0032] In this embodiment, the criteria for the determination by the execution determining unit 15 can be selected from a plurality of patterns. Examples of the criteria will be described below.
[0033] The execution decision unit 15 may cause the second detector 23 to execute the detection process based on the decision of an operator using the image processing system 100. For example, when the operator, having viewed the output result of the first detector 14, wants to improve the recognition accuracy of the first detector 14, the operator inputs an instruction to execute the detection process by the second detector 23 from the input device 7. Upon receiving an instruction from the operator via the input device 7, the execution decision unit 15 executes the detection process by the second detector 23. In this way, the result of the operator's visual decision can be used as a trigger to execute the detection process by the second detector 23.
[0034] Furthermore, the execution decision unit 15 may cause the second detector 23 to execute the detection process based on the number of times that the detection process is executed on the object W from the image. For example, the execution decision unit 15 counts the number of times that the detection process is executed by the first detector 14, and executes the detection process by the second detector 23 when the number of times reaches a predetermined number.
[0035] Furthermore, the execution determination unit 15 may set a condition for executing the detection process by the second detector 23 at a specific timing, and when the condition is satisfied, cause the second detector 23 to execute the detection process. An example of the specific timing will be described below.
[0036] For example, when the robot 3 is running on a production line, it may be difficult to execute the detection process of the second detector 23 because it needs to operate at high speed due to cycle time. Therefore, the execution determination unit 15 may set a condition for executing the necessity determination assuming that the robot 3 is being taught if it is not running on the production line. This makes it possible to determine the appropriateness of the output result of the first detector 14 by referring to the output result of the second detector 23 during teaching. Also, the image processing device 2 and the external analysis unit 10 may be configured to be connected via a network only during teaching.
[0037] The execution decision unit 15 may also set a condition that the detection process of the second detector 23 is executed on a specific date and time based on a preset schedule.
[0038] Furthermore, the execution decision unit 15 may determine the output result of the first detector 14 based on a preset evaluation index, and execute the detection process of the second detector 23. For example, the execution decision unit 15 monitors the output result of the first detector 14, and executes the detection process of the second detector 23 when the output result deviates from an evaluation reference value.
[0039] The evaluation index may be, for example, the degree of matching. The execution determination unit 15 may be configured to, for example, monitor the output result of the first detector 14 in chronological order and determine whether or not the execution is necessary when the degree of matching tends to decrease.
[0040] Furthermore, the execution decision unit 15 may make the necessity decision on the condition that a case has occurred in which the first detector 14 is unable to detect the object W.
[0041] The above has described examples of the judgment criteria of the execution judgment unit 15. Note that the judgment criteria may be one of the above-mentioned patterns set in advance by the manufacturer, and the operator may not be able to change (select) the judgment criteria of the execution judgment unit 15. Next, the suitability judgment unit 16 will be described.
[0042] The suitability determination unit 16 has a suitability determination function that compares the output result of the first detector 14 with the output result of the second detector 23 and determines whether the output result of the first detector 14 is appropriate. The suitability determination unit 16 can, for example, compare the XYZ coordinates of the object W extracted in the image and determine suitability based on the degree of deviation of the output result of the first detector 14 from the output result of the second detector 23. The suitability determination unit 16 may also determine suitability based on the occurrence of erroneous detection, such as when the object W is detected by the first detector 14 but not by the second detector 23.
[0043] Next, the notification processing unit 17 will be described. The notification processing unit 17 has a notification processing function that executes processing to notify the operator of the occurrence of an error when the suitability determination unit 16 determines that an error has occurred in the output result of the first detector 14. The notification processing unit 17, for example, displays information indicating the occurrence of an error on the display screen of the display device 8, or notifies the operator of the occurrence of an error using a speaker function of the display device 8.
[0044] Next, the external analysis unit 10 will be described. The external analysis unit 10 includes a control unit 21 and a storage unit 22. The control unit 21 is a processor such as a CPU, and realizes various functions by executing programs stored in the storage unit 22. The storage unit 22 is a storage device such as a ROM or RAM that stores an OS, application programs, etc., and a hard disk drive or SSD that stores various other information. The storage unit 22 stores, for example, various information such as a vision program that controls the vision sensor 5, as well as learning data for training the first detector 14.
[0045] The external analyzer 10 of this embodiment includes a second detector 23 and a learning processor 24 as functional units executed by the controller 21 .
[0046] The second detector 23 is a second detection processing function that detects or inspects the object W from the image acquired by the vision sensor 5. The second detector 23 is set to have higher recognition accuracy than the first detector 14. The second detector 23 executes processing to detect the object W from the image based on a model generated by deep learning, for example. The second detector 23 can execute the detection processing on the same image as the image on which the first detector 14 executed the detection processing.
[0047] The learning processing unit 24 is a learning processing function that executes learning processing of the first detector 14. In this embodiment, the learning processing unit 24 executes the learning processing when the suitability determining unit 16 determines that there is an error in the output result of the first detector 14. The output result of the first detector 14 may contain an error if the number of detections used to learn the model of the first detector 14 is insufficient, and this also includes cases where learning is insufficient.
[0048] In the learning process, the learning processing unit 24 stores, as learning data, an image that is the original data on which the second detector 23 performed the detection process, and the output result output by the second detector 23 for that image. The output result of the second detector 23 is a label associated with the image. The data stored as the label may include, in addition to the output result of the second detector 23, information such as the position, orientation, and size of the object included in the detection result. The learning processing unit 24 stores learning data including the image and the label for the image in the memory unit 22 of the external analysis unit 10. The learning data may be stored in the memory unit 12 of the image processing device 2.
[0049] The learning processing unit 24 re-learns the model of the first detector 14 based on a data set, which is a collection of learning data stored in the storage unit 22. The learning process is for the first detector 14 in the image processing device 2, but is executed by the external analysis unit 10, which has higher processing performance than the image processing device 2. However, this is not limited to this, and the learning process may be executed by the image processing device 2. The learning process may be machine learning using teaching data or deep learning.
[0050] The learning process may be automatically executed on the external analysis unit 10 or manually executed by an operator. In the case of automatic execution, the learning processing unit 24 may perform re-learning each time an error occurs in the output result of the first detector 14, or may execute the learning process in response to a trigger when a predetermined number of learning data sets have been accumulated. Even in the case of manual execution by an operator, the operator can use a data set, which is a collection of learning data stored in the memory unit 22, so that learning processes such as machine learning and deep learning by the learning processing unit 24 can be easily and quickly started.
[0051] Next, the flow of image processing will be described with reference to Fig. 3. Fig. 3 is a flowchart showing the flow of image processing in the image processing system 100. Note that the flowchart shown in Fig. 3 is merely an example, and the order and content of the flow can be changed as appropriate.
[0052] In step S11, the acquisition processing unit 13 executes a process of acquiring an image captured by the vision sensor 5. Next, in step S12, the first detector 14 executes a detection process of detecting or inspecting the object W from the image acquired by the acquisition processing unit 13.
[0053] After the detection process by the first detector 14, in step S13, the execution decision unit 15 decides whether or not it is necessary to execute the detection process by the second detector 23 based on a predetermined decision criterion.
[0054] If the execution decision unit 15 determines that it is not necessary to execute the detection process of the second detector 23, the process ends without going through the processes of step S14 and thereafter (step S13; No). If the execution decision unit 15 determines that it is necessary to execute the detection process of the second detector 23, the process proceeds to step S14.
[0055] In step S14 , the second detector 23 executes a detection process to detect or inspect the object W from the image acquired by the acquisition processing unit 13 .
[0056] In step S15, the propriety determining unit 16 compares the output result of the first detector 14 with the output result of the second detector 23, and determines whether or not the output result of the first detector 14 contains an error.
[0057] If the suitability determination unit 16 determines that there is no error in the output result of the first detector 14, the process ends without going through step S16 and subsequent steps (step S15; No). If the suitability determination unit 16 determines that there is no error in the output result of the first detector 14, the process proceeds to step S16.
[0058] In step S16, the notification processing unit 17 executes a process of notifying the operator that an error has occurred in the output result of the first detector 14.
[0059] In step S17, the learning processing unit 24 stores the output result of the second detector 23 executed on the image in which an error occurred in the output result of the first detector 14 as learning data in the memory unit 22 of the external analysis unit 10. Also in step S17, the learning processing unit 24 re-learns the model of the first detector 14 based on a data set which is a collection of learning data stored in the memory unit 22 or the memory unit 12. As described above, the trigger for executing the learning process may be the accumulation of a predetermined number of learning data, or may be based on an operation by the operator.
[0060] As described above, the image processing method of this embodiment includes a step of capturing an image of an object W using a vision sensor 5 as a visual sensor, a first detection step of detecting or inspecting the object W from the image captured by the vision sensor 5 using the first detector 14, an execution judgment step of determining whether detection or inspection by the second detector 23 is necessary, a second detection step of executing a detection process by the second detector 23 if it is determined that detection or inspection by the second detector 23 is necessary, and a judgment step of comparing the output result of the first detector 14 with the output result of the second detector 23 to determine whether there is an error in the output result of the first detector 14.
[0061] In addition, the program of this embodiment executes an acquisition processing function of capturing an image of the object W using the vision sensor 5, a first detection processing function of detecting or inspecting the object W from the image captured by the vision sensor 5 using the first detector 14, an execution judgment function of determining whether detection or inspection by the second detector 23 is necessary, a second detection processing function of executing detection processing by the second detector 23 when it is determined that detection or inspection by the second detector 23 is necessary, and a judgment processing function of comparing the output result of the first detector 14 with the output result of the second detector 23 to determine whether there is an error in the output result of the first detector 14.
[0062] In addition, the image processing system 100 of this embodiment is equipped with an acquisition processing unit 13 that captures an image of an object W using a vision sensor 5, a first detector 14 that detects or inspects the object W from the image captured by the vision sensor 5, a second detector 23 that detects or inspects the object W from the image captured by the vision sensor 5, an execution judgment unit 15 that determines whether detection or inspection by the second detector 23 is necessary, and an appropriateness judgment unit 16 that, when it is determined that detection or inspection by the second detector 23 is necessary, compares the output result of the first detector 14 with the output result of the second detector 23 to determine whether there is an error in the output result of the first detector 14.
[0063] By configuring the image processing method, program, and image processing system 100 as described above, the detection process of the first detector 14, which normally places a low load on the computer and consumes low power, can be executed, and if an error occurs in the output result of the first detector 14, the output result of the second detector 23, which has high recognition accuracy, can be used.
[0064] Furthermore, in this embodiment, if it is determined that there is an error in the determination step, the image and the detection result of the second detector 23 are stored as learning data. This allows the learning processing unit 24 to use a data set, which is a collection of learning data stored in the storage unit 22, so that the process of preparing a data set, such as setting labels, can be omitted, and the learning process can be performed efficiently.
[0065] Furthermore, in this embodiment, if it is determined that there is an error in the determination step, learning of the first detector 14 is performed again using the image and the detection results of the second detector 23 as learning data. As a result, learning processing based on appropriate learning data is automatically performed without any operation by the operator, and the recognition accuracy of the first detector 14 can be improved without any effort.
[0066] Furthermore, in this embodiment, the learning is performed by an external analysis unit 10 on a computer or cloud other than the computer that realizes the first detector 14. By configuring the external analysis unit 10 as a computer with higher performance than the image processing device 2, the learning process, which places a heavy load on the computer, can be performed without difficulty.
[0067] In this embodiment, if it is determined that an error exists in the determination step, a process of notifying the operator of the occurrence of the error is executed, thereby enabling the operator to quickly grasp that an error has occurred in the output result of the first detector 14.
[0068] In addition, in the present embodiment, the execution determination step may be configured so that the operator determines, through an operation, whether or not detection processing by the second detector 23 is necessary. When this method is used, the operator's visual judgment or judgment based on experience is reflected in the determination of whether or not detection processing by the second detector 23 is necessary.
[0069] Furthermore, in this embodiment, the execution determination step may be configured to determine that detection processing by the second detector 23 is necessary when the detection processing by the first detector 14 has been performed a predetermined number of times. When this method is used, when the detection processing has been performed a predetermined number of times, a determination as to whether detection processing by the second detector 23 is necessary is automatically performed, eliminating the need for an operator to perform an operation.
[0070] In addition, in this embodiment, the execution determination step may be configured to determine, when a preset condition is satisfied, that detection processing by the second detector 23 is necessary. When this method is used, the timing at which an error in the first detector 14 is highly likely to occur or the timing at which it is better to check for the occurrence of an error can be set as an operation condition for the second detector 23.
[0071] Furthermore, in this embodiment, the execution determination step may be configured such that, when the output result of the first detector 14 satisfies a predetermined evaluation index, it is determined that detection processing by the second detector 23 is necessary. When this method is used, the actual output result of the first detector 14 is monitored, and therefore, when the evaluation index indicates a situation in which an error is likely to occur, a determination as to whether detection processing is necessary or not is made quickly.
[0072] In this embodiment, the second detector 23 operates on an external analysis unit 10 that is a computer or cloud-based system separate from the computer that realizes the first detector 14. By configuring the external analysis unit 10 with a computer that has higher performance than the image processing device 2, the second detector 23, which has higher recognition accuracy and a greater computer load than the first detector 14, can be operated smoothly.
[0073] Next, a modified example of the above embodiment will be described. Fig. 4 is a diagram showing the functional configuration of an image processing device 2A of the modified example. In the modified example shown in Fig. 4, the image processing device 2 of the above embodiment and the external analysis unit 10 are integrated into an image processing device 2A configured as a single computer. In the following description, components common to the configuration of the above embodiment will be assigned the same reference numerals and detailed description will be omitted.
[0074] 4, the image processing device 2A includes an acquisition processing unit 13, a first detector 14, an execution determination unit 15, a suitability determination unit 16, a notification processing unit 17, a second detector 23, and a learning processing unit 24. That is, the first detector 14 and the second detector 23 exist in the same image processing device 2A, and the learning process for the first detector 14 is also executed in the image processing device 2A.
[0075] In this modification, the detection process of the first detector 14, which normally places a low load on the computer and consumes little power, is executed first, and the output result of the second detector 23, which has high recognition accuracy, can be used when there is a high probability that an error will occur in the output result of the first detector 14. Furthermore, when executed on the same hardware, the first detector 14 operates faster than the second detector 23 and uses less memory, so the operation time can be shortened compared to when the detection process is executed using only the second detector 23.
[0076] Although the embodiments of the present invention have been described above, the present invention is not limited to the configurations of the above embodiments and modified examples. For example, components such as the execution determination unit 15 and the suitability determination unit 16 may be arranged in the external analysis unit 10. Furthermore, the image processing system 100 may process the object W using a machine tool instead of the robot 3. In this case, the control device 1 becomes a numerical control device.
[0077] The image processing device 2 and the external analysis unit 10 or the image processing device 2A can be realized by hardware, software, or a combination thereof. The control method performed by the image processing device 2 and the external analysis unit 10 or the image processing device 2A can also be realized by hardware, software, or a combination thereof. Here, "realized by software" means that the method is realized by a computer reading and executing a program.
[0078] The program can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)).
[0079] Furthermore, although the above-described embodiments are preferred embodiments of the present invention, the scope of the present invention is not limited to only the above-described embodiments, and the present invention can be implemented in various modified forms within the scope that does not deviate from the gist of the present invention.
[0080] For example, in the above-described embodiments, detectors that detect or inspect an object from an image acquired by a vision sensor are used as the first and second detectors, and the present disclosure is applied to detection in image processing, but is not limited to this, and the present disclosure can be widely applied as a detection device in processes other than image processing. That is, another embodiment of the present disclosure is a detection device 100 that detects or inspects an object W, the detection device 100 including a first detector 14 that detects or inspects the object W, a second detector 23 that detects or inspects the object W, an execution determination unit 15 that determines whether detection or inspection by the second detector 23 is necessary, and an appropriateness determination unit 16 that, when it is determined that detection or inspection by the second detector 23 is necessary, compares the output result of the first detector 14 with the output result of the second detector 23 and determines whether there is an error in the output result of the first detector 14.
[0081] More specifically, in another embodiment of the present disclosure, a combination of a force sensor attached to the robot's hand and a torque sensor attached to each joint of the robot can be used as the combination of the first and second detectors. The state of the robot and the type of work being performed by the robot can be detected from the outputs of the force sensor and torque sensor, so this embodiment also achieves the same effects as the above embodiment.
[0082] In another embodiment of the present disclosure, a combination of a microphone and a vibration sensor can be used as the combination of the first detector and the second detector. The output of the microphone and the vibration sensor can be used to detect the situation around the robot and the state of the robot, and therefore, this embodiment also achieves the same effects as the above embodiment.
[0083] The following supplementary notes are further disclosed regarding the above-described embodiments and modifications: (Supplementary Note 1) An image processing method for extracting an object (W) from an image, comprising: a step of capturing an image of the object (W) with a visual sensor (5); a first detection step of detecting or inspecting the object (W) from the image captured by the visual sensor (5) with a first detector (14); an execution determination step of determining whether detection or inspection by a second detector (23) is necessary; a second detection step of executing detection processing by the second detector (23) when it is determined that detection or inspection by the second detector (23) is necessary; and a determination step of comparing an output result of the first detector (14) with an output result of the second detector (23) to determine whether there is an error in the output result of the first detector (14).
[0084] (Supplementary Note 2) In the image processing method, if it is determined that there is an error in the determining step, the image and the detection result of the second detector (23) are stored as learning data.
[0085] (Supplementary Note 3) In the above image processing method, if it is determined that there is an error in the determination step, the image and the detection result of the second detector (23) are used as learning data to perform learning again on the first detector (14).
[0086] (Supplementary Note 4) In the image processing method described above, the learning is performed on a computer (10) other than a computer that realizes the first detector (14) or on a cloud.
[0087] (Supplementary Note 5) In the image processing method described above, in the determining step, if it is determined that an error exists, a process of notifying that the error has occurred is executed.
[0088] (Supplementary Note 6) In the image processing method, in the execution determining step, it is determined by an operator's operation whether or not the detection process by the second detector (23) is necessary.
[0089] (Supplementary Note 7) In the above image processing method, in the execution determination step, when the detection process by the first detector (14) has been performed a predetermined number of times, it is determined that the detection process by the second detector (23) is necessary.
[0090] (Supplementary Note 8) In the image processing method, in the execution determining step, it is determined that the detection process by the second detector (23) is necessary when a preset condition is satisfied.
[0091] (Supplementary Note 9) In the above image processing method, in the execution determination step, if the output result of the first detector (14) satisfies a predetermined evaluation index, it is determined that the detection process by the second detector (23) is necessary.
[0092] (Supplementary Note 10) In the image processing method, the second detector (23) operates on a computer (10) or a cloud separate from a computer that realizes the first detector (14).
[0093] (Supplementary Note 11) A program that causes a computer (2) that executes processing to extract an object (W) from an image to execute: an acquisition processing function that captures an image of the object (W) using a visual sensor (5); a first detection processing function that uses a first detector (14) to detect or inspect the object (W) from the image captured by the visual sensor (5); an execution judgment function that judges whether detection or inspection by a second detector (23) is necessary; a second detection processing function that executes detection processing by the second detector (23) when it is judged that detection or inspection by the second detector (23) is necessary; and a judgment processing function that compares the output result of the first detector (14) with the output result of the second detector (23) to judge whether there is an error in the output result of the first detector (14).
[0094] (Supplementary Note 12) An image processing system (100) that extracts an object (W) from an image, comprising: an acquisition processing unit (13) that captures an image of the object (W) using a visual sensor (5); a first detector (14) that detects or inspects the object (W) from the image captured by the visual sensor (5); a second detector (23) that detects or inspects the object (W) from the image captured by the visual sensor (5); an execution judgment unit (15) that determines whether detection or inspection by the second detector (23) is necessary; and an appropriateness judgment unit (16) that, when it is determined that detection or inspection by the second detector (23) is necessary, compares the output result of the first detector (14) with the output result of the second detector (23) to determine whether there is an error in the output result of the first detector (14).
[0095] (Supplementary Note 13) An image processing device (2A) that extracts an object (W) from an image, comprising: an acquisition processing unit (13) that captures an image of the object (W) using a visual sensor (5); a first detector (14) that detects or inspects the object (W) from the image captured by the visual sensor (5); a second detector (23) that detects or inspects the object (W) from the image captured by the visual sensor (5); an execution judgment unit (15) that determines whether detection or inspection by the second detector (23) is necessary; and an appropriateness judgment unit (16) that, when it is determined that detection or inspection by the second detector (23) is necessary, compares the output result of the first detector (14) with the output result of the second detector (23) and determines whether there is an error in the output result of the first detector (14).
[0096] (Appendix 14) A detection device (100) for detecting or inspecting an object (W), comprising: a first detector (14) that detects or inspects the object (W); a second detector (23) that detects or inspects the object (W); an execution judgment unit (15) that judges whether detection or inspection by the second detector (23) is necessary; and an appropriateness judgment unit (16) that, when it is determined that detection or inspection by the second detector (23) is necessary, compares the output result of the first detector (14) with the output result of the second detector (23) and judges whether there is an error in the output result of the first detector (14).
[0097] REFERENCE SIGNS LIST 1 control device 2 image processing device 3 robot 4 arm 5 vision sensor 10 external analysis unit 13 acquisition processing unit 14 first detector 15 execution decision unit 16 suitability determination unit 17 notification processing unit 23 second detector 24 learning processing unit W object 100 image processing system
Claims
1. An image processing method for extracting an object from an image, comprising: a step of capturing an image of the object with a visual sensor; a first detection step of using a first detector to detect or inspect the object from the image captured by the visual sensor; an execution judgment step of determining whether detection or inspection by a second detector is necessary; a second detection step of executing a detection process by the second detector when it is determined that detection or inspection by the second detector is necessary; and a judgment step of comparing the output result of the first detector with the output result of the second detector to determine whether there is an error in the output result of the first detector.
2. The image processing method according to claim 1, further comprising storing the image and the detection result of the second detector as learning data when it is determined that an error exists in the determining step.
3. The image processing method according to claim 1, further comprising the steps of: if it is determined that there is an error in said determination step, re-learning of said first detector is performed using said image and the detection result of said second detector as learning data.
4. The image processing method according to claim 3, wherein the learning is performed on a computer or cloud other than a computer that realizes the first detector.
5. The image processing method according to claim 1, wherein, in said judging step, if it is judged that an error exists, a process of notifying that an error has occurred is executed.
6. An image processing method according to any one of claims 1 to 5, wherein in the execution decision step, whether or not the detection process by the second detector is required is decided by an operation of an operator.
7. An image processing method according to any one of claims 1 to 5, wherein in the execution decision step, it is decided that the detection process by the second detector is necessary when the detection process by the first detector has been executed a predetermined number of times.
8. An image processing method according to any one of claims 1 to 5, wherein in said execution decision step, it is decided that said detection process by said second detector is necessary if a preset condition is satisfied.
9. An image processing method according to any one of claims 1 to 5, wherein in the execution decision step, it is decided that the detection process by the second detector is necessary if the output result of the first detector satisfies a predetermined evaluation index.
10. An image processing method according to any one of claims 1 to 9, wherein the second detector operates on a computer or cloud separate from a computer that realizes the first detector.
11. A program that causes a computer that executes a process to extract an object from an image to execute: an acquisition process function that captures an image of the object using a visual sensor; a first detection process function that uses a first detector to detect or inspect the object from the image captured by the visual sensor; an execution judgment function that determines whether detection or inspection by a second detector is necessary; a second detection process function that executes detection processing by the second detector when it is determined that detection or inspection by the second detector is necessary; and a judgment process function that compares the output result of the first detector with the output result of the second detector to determine whether there is an error in the output result of the first detector.
12. An image processing system for extracting an object from an image, comprising: an acquisition processing unit that captures an image of the object using a visual sensor; a first detector that detects or inspects the object from the image captured by the visual sensor; a second detector that detects or inspects the object from the image captured by the visual sensor; an execution judgment unit that determines whether detection or inspection by the second detector is necessary; and a suitability judgment unit that, when it is determined that detection or inspection by the second detector is necessary, compares the output result of the first detector with the output result of the second detector and determines whether there is an error in the output result of the first detector.
13. An image processing device that extracts an object from an image, comprising: an acquisition processing unit that captures an image of the object using a visual sensor; a first detector that detects or inspects the object from the image captured by the visual sensor; a second detector that detects or inspects the object from the image captured by the visual sensor; an execution judgment unit that determines whether detection or inspection by the second detector is necessary; and a suitability judgment unit that, when it is determined that detection or inspection by the second detector is necessary, compares the output result of the first detector with the output result of the second detector and determines whether there is an error in the output result of the first detector.
14. A detection device for detecting or inspecting an object, comprising: a first detector for detecting or inspecting the object; a second detector for detecting or inspecting the object; an execution judgment unit for determining whether or not detection or inspection by the second detector is necessary; and a suitability judgment unit for, when it is determined that detection or inspection by the second detector is necessary, comparing the output result of the first detector with the output result of the second detector to determine whether or not there is an error in the output result of the first detector.
Citation Information
Patent Citations
Image recognition system
JP2011059810A
Information transfer device, leaning system, information transfer method, and program
JP2016191973A
Detection recognizing system
JP2018088157A
Detection system, information processing apparatus, evaluation method, and program
JP2019106119A
Self oil supply monitoring system and learning device
JP2023007794A
