Object detection device, control device, and computer program for object detection
By combining robust detection and precision detection methods, the problem of insufficient detection accuracy caused by changes in object position in automatic machinery is solved, and accurate tracking and operation control of objects in automatic machinery is achieved, and the operation accuracy and efficiency are improved.
Patent Information
- Application Number
- CN202010101889.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-02-22
- Filing Date
- 2020-02-19
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2040-02-19
AI Technical Summary
In automatic machinery, changes in the position and posture of the object lead to insufficient image detection accuracy and it is difficult to accurately track the object. Especially when the object part is hidden or the image area is unknown, full image matching is required, which makes it difficult for the control device to accurately move the movable part To perform operations on objects.
By adopting a combination of a robust detection unit and a precision detection unit, the robust detection unit detects the object position through a pre-learning recognizer when the object does not meet the predetermined position, and the precision detection unit performs high-precision detection through template comparison when the object is satisfied. And the movable part is controlled to track the object according to the detection results.
Even if the position relationship of the object relative to the photographing part changes, the position and posture of the object can still be accurately detected, which improves the tracking and working accuracy of the object by the automatic machinery, reduces the detection time, and ensures that the movable part can accurately move to the working position.
Smart Images

Figure CN111604942B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to, for example, an object detection device and a computer program for object detection that detect an object represented by an image, and a control device that controls an automatic machine having a movable part using a detection result of an object represented by an image. Background Art
[0002] In an automatic machine such as a robot or a machine tool having a movable part such as an arm and using a servo motor to drive the movable part, in order to control the automatic machine according to the relative positional relationship between the movable part and the object to be operated by the automatic machine, the following technology is proposed: using an image obtained by a camera representing the object to be operated. As one of the technologies, a technology for detecting an object from an image using graphic matching is proposed (for example, refer to Japanese Patent Publication No. 2018-158439). In addition, as another such technology, the following technology is proposed: inputting an image representing the object to be operated to a recognizer implemented by a machine learning method such as a neural network, thereby obtaining control information for causing the automatic machine to perform a predetermined action (for example, refer to Japanese Patent Publication No. 6376296, Japanese Patent Publication No. 2009-83095, Japanese Patent Publication No. 2018-126799, and Japanese Patent Publication No. 2018-205929).
[0003] When an object to be a work object is conveyed by a belt conveyor or the like, the object may move. In such a case, as the positional relationship between the camera and the object changes, the position, posture, and size of the object represented on the image change. Therefore, the features of the object used for pattern matching, such as a part of the surface or outline of the object when viewed from a specific direction, are hidden, and it is sometimes difficult to accurately detect the position and posture of the object in pattern matching. As a result, during the period when the control device of the automatic machine tracks the object based on the detection result of detecting the object from a series of images obtained in a time series, the object may not be tracked, and the movable part may not be moved to a position where the work can be performed on the object. In addition, since the position, posture, and size of the object to be a work object represented on the image change, when the area where the object on the image is located is unknown, in order to detect the object, it is sometimes necessary to perform pattern matching on the entire image, and it takes too much time until the object is detected. In such a case, it is difficult to make the movable part follow the position change of the object.
[0004] On the other hand, when the position and posture of an object to be operated shown on an image are detected using a recognizer based on a machine learning method, the control device of the automatic machine can detect the object from the image regardless of the position and posture of the object relative to the camera. However, in this case, the detection accuracy of the position and posture of the object may not be sufficient, and sometimes the control device cannot accurately move the movable part to a position where the movable part can perform the operation on the object based on the position of the object detected by the recognizer. Summary of the invention
[0005] In one aspect, an object is to provide an object detection device capable of detecting an object even when the relative positional relationship of the object with respect to an imaging unit for imaging the object to be detected changes.
[0006] According to one embodiment, an object detection device is provided. The object detection device comprises: a storage unit (33) storing a template representing features of the appearance of an object (10) when the object (10) is observed from a predetermined direction; a robust detection unit (41) inputting an image into a recognizer that has been pre-learned by detecting the object (10) from an image when a photographing unit (4) that photographs the object (10) and generates an image showing the object (10) does not satisfy a predetermined positional relationship with the object (10), thereby detecting the position of the object (10) on the image; and a precision detection unit (42) detecting the position of the object (10) on the image by comparing the image with the template when the photographing unit (4) and the object (10) satisfy a predetermined positional relationship.
[0007] According to another embodiment, a control device for an automatic machine (2) having at least one movable member (12-16) is provided. The control device comprises: a storage unit (33) storing a template representing features of the appearance of an object (10) to be a work object of an automatic machine (2) when the object (10) is observed from a predetermined direction; a robust detection unit (41) inputting the image (10) into a recognizer pre-learned by detecting the object from the image when a photographing unit (4) mounted on a movable unit (12-16) for photographing the object (10) to generate an image showing the object (10) and the object (10) do not satisfy a predetermined positional relationship with the object (10), thereby detecting the position of the object (10) on the image, and detecting the position of the object (10) in real space based on the position of the object (10) on the image; and a precision detection unit (42) detecting the position of the object (10) on the image by comparing the image with the template when the photographing unit (4) and the object (10) satisfy a predetermined positional relationship, and detecting the position of the object (10) on the image based on the position of the object (10) on the image. A device for detecting the position of an object (10) in real space, or detecting the position of an object (10) in real space based on position information obtained by a position information acquisition unit, the position information acquisition unit acquiring position information indicating a relative position to the object (10); a proximity control unit (44) which controls the movable units (12-16) so that the imaging unit (4) and the object (10) satisfy a predetermined positional relationship based on the position of the object (10) in real space detected by a robust detection unit (41) when the imaging unit (4) and the object (10) do not satisfy a predetermined positional relationship; and an operation control unit (45) which controls the movable units (12-16) so that the movable units (12-16) move to a position where the automatic machine (2) can perform a predetermined operation on the object (10) based on the position of the object (10) in real space detected by a precision detection unit (42) when the imaging unit (4) and the object (10) satisfy a predetermined positional relationship.
[0008] According to another embodiment, a computer program for object detection is provided. The computer program for object detection includes a command for causing a computer to execute the following processing: when a photographing unit (4) for photographing a target object (10) to generate an image showing the target object (10) does not satisfy a predetermined positional relationship with the target object (10), the image is input to a recognizer that has been pre-learned by detecting the target object (10) from the image, thereby detecting the position of the target object (10) on the image; when the photographing unit (4) and the target object (10) satisfy the predetermined positional relationship, a template representing features of the appearance of the target object (10) when the target object (10) is observed from a predetermined direction is compared with the image, thereby detecting the position of the target object (10) on the image.
[0009] According to one aspect, even when the relative positional relationship of the object to be detected with respect to the imaging unit that images the object changes, the object can be detected. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Figure 1 This is a schematic structural diagram of a robot system according to one embodiment.
[0011] Figure 2 This is a schematic diagram of the control device.
[0012] Figure 3 This is a functional block diagram of a processor included in a control device related to movable part control processing including object detection processing.
[0013] Figure 4 An overview of the movable part control process including the object detection process will be described.
[0014] Figure 5 This is an operation flow chart of the movable part control process including the object detection process.
[0015] Figure 6 It is a schematic structural diagram of a robot system according to a modified example.
[0016] Figure 7 This is a schematic diagram of the structure of an object detection device. DETAILED DESCRIPTION
[0017] The object detection device according to the embodiment of the present invention is described below with reference to the accompanying drawings. The object detection device detects the object to be operated by the automatic machine (hereinafter simply referred to as the object) from a series of images obtained by a photographing unit installed on a movable part of an automatic machine, for example, to detect the position of the object in the actual space. The automatic machine has a movable part such as an arm. At this time, before the photographing unit and the object meet a predetermined positional relationship, the object detection device inputs an image to a recognizer that has been pre-learned by detecting the object from an image, thereby robustly detecting the position of the object on the image, and detecting the position of the object in the actual space based on the detection result. On the other hand, when the photographing unit and the object meet a predetermined positional relationship, the object detection device compares a template that indicates the features of the appearance of the object when the object is observed from a predetermined direction obtained in advance with the image, thereby detecting the position and posture of the object on the image with high accuracy, and detecting the position and posture of the object in the actual space based on the detection result. Furthermore, the object detection device is incorporated into a control device of an automatic machine, and the movable part of the automatic machine is controlled based on the detected position of the target object or the detected position and posture of the target object so that the automatic machine can perform operations on the target object.
[0018] Figure 1 This is a schematic diagram of a robot system 1 equipped with an object detection device according to an embodiment. The robot system 1 includes a robot 2, a control device 3 for controlling the robot 2, and a camera 4 mounted on a movable part of the robot 2 for photographing a workpiece 10 as an example of a target object. The robot system 1 is an example of an automatic machine.
[0019] The robot 2 includes a base 11, a rotating table 12, a first arm 13, a second arm 14, a wrist 15, and a tool 16. The rotating table 12, the first arm 13, the second arm 14, the wrist 15, and the tool 16 are each an example of a movable part. The rotating table 12, the first arm 13, the second arm 14, and the wrist 15 are each supported by a shaft provided at a joint to which the rotating table 12, the first arm 13, the second arm 14, and the wrist 15 are respectively mounted, and are operated by driving the shaft by a servo motor. In addition, the workpiece 10 is conveyed by, for example, a belt conveyor, and the robot 2 performs a predetermined operation on the workpiece 10 while the workpiece 10 is within a predetermined range.
[0020] The base 11 is a member that serves as a platform when the robot 2 is placed on the ground. The rotating table 12 is mounted on the top surface of the base 11 via a joint 21 so as to be rotatable about an axis (not shown) provided to be orthogonal to one surface of the base 11.
[0021] One end of the first arm 13 is mounted on the rotating table 12 at a joint 22, and the joint 22 is provided on the rotating table 12. Figure 1 As shown, the first arm portion 13 is rotatable via a joint 22 about an axis (not shown) provided parallel to the surface of the base 11 on which the rotating table 12 is mounted.
[0022] One end of the second arm 14 is mounted on the first arm 13 via a joint 23, and the joint 23 is provided at the other end of the first arm 13 on the opposite side to the joint 22. Figure 1 As shown, the second arm portion 14 is rotatable, via a joint 23, around an axis (not shown) provided parallel to the surface of the base 11 on which the rotating table 12 is mounted.
[0023] The wrist 15 is attached to the front end of the second arm 14 on the opposite side of the joint 23 via the joint 24. The wrist 15 has the joint 25, and can be bent by the joint 25 with an axis (not shown) provided parallel to the axis of the joint 22 and the axis of the joint 23 as the rotation center. In addition, the wrist 15 can also rotate within a plane orthogonal to the long side direction of the second arm 14 with an axis (not shown) parallel to the long side direction of the second arm 14 as the rotation center.
[0024] The tool 16 is mounted on the front end of the wrist 15 on the opposite side from the joint 24. The tool 16 has a mechanism or device for operating the workpiece 10. For example, the tool 16 may have a laser for processing the workpiece 10, or a servo gun for welding the workpiece 10. Alternatively, the tool 16 may have a manipulator mechanism for grasping the workpiece 10 or a component to be assembled to the workpiece 10.
[0025] The camera 4 is an example of a photographing unit, and is installed on the tool 16, for example. In addition, the camera 4 may be installed on other movable parts such as the wrist 15 or the second arm 14. When the robot 2 performs an operation on the workpiece 10 or when the tool 16 is brought close to the workpiece 10, the camera 4 faces a direction including the workpiece 10 in the photographing range of the camera 4. And, the camera 4 photographs the photographing range including the workpiece 10 in each predetermined photographing cycle to generate an image showing the workpiece 10. Whenever the camera 4 generates an image, it outputs the generated image to the control device 3 via the communication line 5.
[0026] In addition, the automatic machine to be controlled by the control device 3 is not limited to Figure 1 The robot 2 shown may be a controlled object having at least one movable part.
[0027] The control device 3 is connected to the robot 2 via the communication line 5, and receives information indicating the motion status of the servo motors for driving the axes provided at the joints of the robot 2, images from the camera 4, etc., from the robot 2 via the communication line 5. The control device 3 controls the servo motors based on the received information, images, and the motion of the robot 2 received from the upper control device (not shown) or preset, thereby controlling the positions and postures of the movable parts of the robot 2.
[0028] Figure 2 3 is a schematic diagram of the control device 3. The control device 3 includes a communication interface 31, a drive circuit 32, a memory 33, and a processor 34. Furthermore, the control device 3 may include a user interface such as a touch panel (not shown).
[0029] The communication interface 31 includes, for example, a communication interface for connecting the control device 3 to the communication line 5 and a circuit for executing processing related to signal transmission and reception via the communication line 5. Furthermore, the communication interface 31 receives information indicating the operating status of the servo motor 35, such as a measured value of the rotation amount from an encoder, from the robot 2 via the communication line 5. The encoder is used to detect the rotation amount of the servo motor 35 as an example of a driving unit, and forwards the information to the processor 34. Figure 2 In the figure, one servo motor 35 is representatively shown, but the robot 2 may have a servo motor for each joint to drive the axis of the joint. In addition, the communication interface 31 receives an image from the camera 4 and forwards it to the processor 34.
[0030] The drive circuit 32 is connected to the servo motor 35 via a current supply cable, and supplies the servo motor 35 with power corresponding to a torque, a rotation direction, or a rotation speed to be generated by the servo motor 35 in accordance with the control of the processor 34 .
[0031] The memory 33 is an example of a storage unit, and includes, for example, a readable and writable semiconductor memory and a read-only semiconductor memory. The memory 33 may include a storage medium such as a semiconductor memory card, a hard disk, or an optical storage medium, and a device for accessing the storage medium.
[0032] The memory 33 stores various computer programs for controlling the robot 2 executed by the processor 34 of the control device 3. In addition, the memory 33 stores information for controlling the movement of the robot 2 when the robot 2 is moved. In addition, the memory 33 stores information indicating the movement status of the servo motor 35 obtained from the robot 2 during the movement of the robot 2. In addition, the memory 33 also stores various data used in the object detection process. Such data includes, for example, a parameter group for specifying an identifier, a template for detecting the workpiece 10 indicating the appearance features of the workpiece 10 when the workpiece 10 is observed from a predetermined direction (for example, vertically above), camera parameters indicating information related to the camera 4 such as the focal length, installation position, and orientation of the camera 4, and an image obtained from the camera 4.
[0033] The processor 34 is an example of a control unit, and includes, for example, a Central Processing Unit (CPU) and its peripheral circuits. The processor 34 may also include a numerical calculation processor. The processor 34 controls the entire robot system 1. The processor 34 also performs a movable part control process including an object detection process.
[0034] Figure 3 1 is a functional block diagram of the processor 34 related to the movable part control process including the object detection process. The processor 34 includes: a robust detection unit 41, a precision detection unit 42, a determination unit 43, an approach control unit 44, and an operation control unit 45. These units of the processor 34 are, for example, functional modules implemented by a computer program executed by the processor 34. Alternatively, these units may be implemented as a dedicated operation circuit installed in a part of the processor 34. In addition, the processing of the robust detection unit 41, the precision detection unit 42, and the determination unit 43 among these units of the processor 34 is included in the object detection process.
[0035] When the camera 4 and the workpiece 10 do not satisfy a predetermined positional relationship, the robust detection unit 41 inputs each image in a series of images in a time series acquired by the camera 4 into a recognizer, thereby detecting the position of the workpiece 10 on the image, that is, the position of the detection target object. Furthermore, the robust detection unit 41 detects the position of the workpiece 10 in the actual space based on the position of the workpiece 10 on the detected image. The robust detection unit 41 uses a recognizer to detect the position of the workpiece 10, and therefore can detect the position of the workpiece 10 more robustly than the precision detection unit 42. The robust detection unit 41 only needs to perform the same processing on each image, so the processing for one image is described below. In addition, the predetermined positional relationship is described in detail in association with the determination unit 43.
[0036] The recognizer is pre-learned by detecting the workpiece 10 shown in the image. For example, as the recognizer, the robust detection unit 41 can use a convolutional neural network (CNN) that is pre-learned by detecting the object shown in the image and representing the area where the detected object is located, such as VGG or Single Shot Multibox Detector (SSD). At this time, the robust detection unit 41 inputs the image of interest to the recognizer, thereby being able to obtain the circumscribed rectangle of the workpiece 10 shown in the image.
[0037] Alternatively, as a recognizer, the robust detection unit 41 may also use a recognizer that has been pre-learned according to other machine learning methods such as support vector machines or AdaBoost. In this case, the robust detection unit 41 sets a plurality of windows that are different from each other for the image of interest, and extracts feature quantities such as Histograms of Oriented Gradients (HOG: Histogram of Oriented Gradients) or Haar-Like features (Haar features) for each window. In addition, when the robust detection unit 41 detects the position of the workpiece 10 in an image acquired in the past, it may be limited to setting a plurality of windows near the detected position. Furthermore, the robust detection unit 41 inputs the feature quantity extracted from the window to the recognizer for each window. When the feature quantity is input, the recognizer outputs a determination result of whether the workpiece 10 is shown in the window. Therefore, the robust detection unit 41 may determine that the workpiece 10 is shown in the window for which the recognizer outputs a determination result that the workpiece 10 is shown.
[0038] As described above, when the region showing the workpiece 10 is detected on the image, the robust detection unit 41 sets a predetermined position within the region, for example, the center of gravity of the region, as the position of the workpiece 10 on the image.
[0039] When the position of the workpiece 10 on the image is obtained, the robust detection unit 41 detects the position of the workpiece 10 in the real space based on the obtained position.
[0040] Here, the position of each pixel on the image corresponds one-to-one to the direction observed from the camera 4. Therefore, the robust detection unit 41 can set the direction corresponding to the center of gravity of the area showing the workpiece 10 on the image as the direction from the camera 4 to the workpiece 10, for example. In addition, the robust detection unit 41 can calculate the estimated distance from the camera 4 to the workpiece 10 by multiplying the ratio of the area of the workpiece 10 on the image to the area of the area showing the workpiece 10 on the image when the distance from the camera 4 to the workpiece 10 is a predetermined reference distance by the reference distance. Therefore, the robust detection unit 41 can detect the position of the workpiece 10 in the camera coordinate system based on the position of the camera 4, based on the direction from the camera 4 to the workpiece 10 and the estimated distance. In addition, the robust detection unit 41 outputs the detected position of the workpiece 10 to the determination unit 43 and the proximity control unit 44.
[0041] According to a modified example, the robust detection unit 41 can detect not only the position of the workpiece 10 but also the posture of the workpiece 10. At this time, as a recognizer, the robust detection unit 41 can use a CNN that has been pre-learned by detecting the area where an object is shown on an image and the posture of the object (for example, parameter B Tekin and others, "Real-Time Seamless Single Shot 6D Object Pose Prediction", CVPR2018). By inputting an image to such a recognizer, the robust detection unit 41 can detect not only the position of the workpiece 10 on the image but also the posture of the workpiece 10 observed from the camera 4, that is, represented by the camera coordinate system. The detected posture of the workpiece 10 is represented, for example, by a combination of a roll angle, a pitch angle, and a yaw angle.
[0042] At this time, the robust detection unit 41 may output the detected position and posture of the workpiece 10 to the determination unit 43 and the proximity control unit 44 .
[0043] When the camera 4 and the workpiece 10 satisfy a predetermined positional relationship, the precision detection unit 42 compares each image in a series of images acquired by the camera 4 in a time series with the template, thereby detecting the position and posture of the workpiece 10 on the image with higher accuracy than the robust detection unit 41. In addition, the precision detection unit 42 detects the position and posture of the workpiece 10 in the real space based on the position and posture of the workpiece 10 on the image. The precision detection unit 42 may perform the same processing for each image, so the processing for one image is described below. For example, the precision detection unit 42 reads a template from the memory 33, compares the template with the image of interest, and thereby detects the area on the image where the workpiece 10 is shown. For example, the template is set to be a template that can show the image of the workpiece 10 when viewed from a predetermined direction. In this case, the precision detection unit 42 changes the relative position and orientation of the template with respect to the image within the comparison range on the image, and calculates a value indicating the similarity between the template and the image, such as a cross-correlation value between the template and the image, and determines that the workpiece 10 is shown in the area corresponding to the position of the template when the similarity is greater than a predetermined threshold. Furthermore, the precision detection unit 42 can obtain the actual rotation amount of the workpiece 10 relative to the orientation of the workpiece 10 shown by the template when viewed from a predetermined direction, based on the orientation of the template when the similarity is greater than a predetermined threshold value, and can therefore obtain the posture of the workpiece 10 based on the rotation amount. In addition, the precision detection unit 42 may set the comparison range to, for example, a predetermined range centered on the predicted position of the workpiece 10 on the image of interest, where the predicted position of the workpiece 10 on the image of interest is the position of the workpiece 10 obtained in each of a series of images obtained before the image of interest, for example, a predicted position obtained by applying a prediction filter such as a Kalman filter to the center of gravity of the region showing the workpiece 10.
[0044] In addition, the template may also represent multiple feature points of the workpiece 10 when viewed from a predetermined direction, such as multiple points on the contour of the workpiece 10. In this case, the precision detection unit 42 may detect multiple feature points by applying an edge detection filter such as a Sobel filter, an angle detection filter such as a Harris filter, or other feature point detection filters such as SIFT to the comparison range on the image. Furthermore, the precision detection unit 42 may change the relative position and orientation of the template relative to the image within the comparison range, and calculate the ratio of the number of feature points on the template that are consistent with the feature points on the image to the number of feature points on the template for each position of the template as the similarity. In this case, the precision detection unit 42 may determine that the workpiece 10 is shown in the area corresponding to the position of the template when the similarity is above a predetermined threshold.
[0045] Alternatively, when the difference between the brightness of the workpiece 10 and the brightness of the background around the workpiece 10 is large, the precision detection unit 42 can detect the area showing the workpiece 10 by binarizing each pixel of the image according to the brightness value of the pixel. For example, when the workpiece 10 is brighter than the surroundings, the precision detection unit 42 can detect pixels having a brightness value higher than a predetermined brightness threshold, and the area that is a collection of the detected pixels is set as the area showing the workpiece 10. At this time, the precision detection unit 42, for example, compares a template prepared for each posture of the workpiece 10 with the area showing the workpiece 10, and sets the posture corresponding to the template having the shape closest to the area showing the workpiece 10 as the actual posture of the workpiece 10. In addition, at this time, the template may be set as a binary image having different values from the area showing the workpiece and other areas.
[0046] When the position of the area showing the workpiece 10 on the image is known, the precise detection unit 42 can obtain the position of the workpiece 10 represented by the camera coordinate system, similarly to the robust detection unit 41. Furthermore, the precise detection unit 42 can obtain the posture of the workpiece 10 represented by the camera coordinate system by rotating the posture of the workpiece 10 on the image by the difference between the predetermined direction defined for the template and the orientation corresponding to the center of gravity of the area showing the workpiece 10 as viewed from the camera 4.
[0047] In addition, multiple templates may be prepared in advance. In this case, the combination of the direction of observing the workpiece 10 and the size of the workpiece 10 on the template may be different for each template. At this time, the precision detection unit 42 calculates the similarity for each position of the template while changing the relative position and orientation of the template with respect to the image within the comparison range in the same manner as described above for each of the multiple templates. And, the precision detection unit 42 determines that the workpiece 10 observed from the direction indicated by the template is shown in the area corresponding to the position of the template with the greatest similarity. And, the size of the workpiece 10 shown by the template corresponds to the distance from the camera 4 to the workpiece 10, so the precision detection unit 42 can set the distance corresponding to the size of the workpiece 10 shown by the template with the greatest similarity as the distance from the camera 4 to the workpiece 10.
[0048] The precision detection unit 42 may also detect the position of the workpiece 10 in the actual space by other methods. For example, the precision detection unit 42 may detect the absolute position of each movable part of the robot 2 (i.e., the position expressed by the world coordinate system) based on the rotation amount of the servo motor 35 from the position when the robot 2 is in the reference posture, and detect the position and orientation of the camera 4 in the world coordinate system based on the relationship between the detected absolute position and the installation position of the camera 4 and the orientation of the camera 4. Furthermore, the precision detection unit 42 can obtain the orientation from the camera 4 to the workpiece 10 in the world coordinate system based on the center of gravity of the area where the workpiece 10 is shown on the image. Furthermore, if the position of the belt conveyor for conveying the workpiece 10 in the world coordinate system and the thickness of the workpiece 10 are known (these values are pre-stored in the memory 33, for example), the precision detection unit 42 can calculate the distance from the camera 4 to the belt conveyor for conveying the workpiece 10 in the orientation from the camera 4 to the workpiece 10 because the position of the camera 4 in the world coordinate system is also known. Furthermore, the precision detection unit 42 can obtain a value obtained by subtracting the thickness of the workpiece 10 from the calculated distance as the distance from the camera 4 to the workpiece 10. Based on the direction from the camera 4 toward the workpiece 10 and the distance from the camera 4 to the workpiece 10 thus obtained, the precision detection unit 42 can detect the position of the workpiece 10 in the actual space.
[0049] The precise detection unit 42 outputs the detected position and posture of the workpiece 10 to the determination unit 43 and the operation control unit 45 .
[0050] The determination unit 43 determines whether the camera 4 and the workpiece 10 satisfy a predetermined positional relationship each time an image is acquired from the camera 4. The predetermined positional relationship may be, for example, that the distance between the camera 4 and the workpiece 10 is less than a predetermined distance. The predetermined distance may be, for example, the distance between the camera 4 and the workpiece 10 corresponding to the size of the workpiece 10 indicated by the template used by the precision detection unit 42. For example, for the latest image, when the robust detection unit 41 detects the position of the workpiece 10, the determination unit 43 compares the estimated distance from the camera 4 to the workpiece 10 obtained by the robust detection unit 41 with the predetermined distance. And, if the estimated distance is less than the predetermined distance, the determination unit 43 determines that the camera 4 and the workpiece 10 satisfy the predetermined positional relationship. And, for the next image acquired, the determination unit 43 instructs the precision detection unit 42 to detect the position and posture of the workpiece 10. On the other hand, if the estimated distance is greater than the predetermined distance, it is determined that the camera 4 and the workpiece 10 do not satisfy the predetermined positional relationship. And, for the next image acquired, the determination unit 43 instructs the robust detection unit 41 to detect the position of the workpiece 10.
[0051] Furthermore, when the coordinate system of the robot 2 is moved based on the movement amount of the workpiece 10 and the operation program of the robot 2 is written on the coordinate system, when the robot 2 reaches a predetermined position on the coordinate system, the determination unit 32 switches from the robust detection unit 41 to the precision detection unit 42 .
[0052] Similarly, for the latest image, when the precision detection unit 42 detects the position of the workpiece 10, the determination unit 43 compares the distance from the camera 4 to the workpiece 10 obtained by the precision detection unit 42 with the predetermined distance. And, if the distance is less than the predetermined distance, the determination unit 43 determines that the camera 4 and the workpiece 10 satisfy the predetermined positional relationship. And, for the next image obtained, the determination unit 43 instructs the precision detection unit 42 to detect the position and posture of the workpiece 10. On the other hand, if the distance is greater than the predetermined distance, it is determined that the camera 4 and the workpiece 10 do not satisfy the predetermined positional relationship. And, for the next image obtained, the determination unit 43 instructs the robust detection unit 41 to detect the position of the workpiece 10.
[0053] In addition, the predetermined positional relationship may be that the distance between the camera 4 and the workpiece 10 is less than a predetermined distance, and the workpiece 10 is in a predetermined posture relative to the camera 4. The predetermined posture may be, for example, a posture in which the camera 4 is located within a predetermined angle range relative to the normal line of a predetermined surface of the workpiece 10. The predetermined surface may be, for example, a surface of the workpiece 10 indicated by the template used by the precision detection unit 42. In this case, for the latest image, when the robust detection unit 41 detects the position and posture of the workpiece 10, the determination unit 43 may determine whether the detected posture is a predetermined posture. Furthermore, if the estimated distance from the camera 4 to the workpiece 10 obtained by the robust detection unit 41 is less than a predetermined distance, and the posture of the workpiece 10 is a predetermined posture, the determination unit 43 determines that the camera 4 and the workpiece 10 satisfy the predetermined positional relationship. Furthermore, the determination unit 43 instructs the precision detection unit 42 to detect the position and posture of the workpiece 10 for the next image to be obtained. On the other hand, if the estimated distance is greater than the predetermined distance, or the posture of the workpiece 10 is not the predetermined posture, it is determined that the camera 4 and the workpiece 10 do not satisfy the predetermined positional relationship. Then, the determination unit 43 instructs the robust detection unit 41 to detect the position of the workpiece 10 for the next acquired image.
[0054] Similarly, for the latest image, when the precision detection unit 42 detects the position and posture of the workpiece 10, if the distance from the camera 4 to the workpiece 10 obtained by the precision detection unit 42 is less than the predetermined distance, and the posture of the workpiece 10 is the predetermined posture, the determination unit 43 determines that the camera 4 and the workpiece 10 satisfy the predetermined positional relationship. In addition, the determination unit 43 instructs the precision detection unit 42 to detect the position and posture of the workpiece 10 for the next image to be obtained. On the other hand, if the distance is greater than the predetermined distance, or the posture of the workpiece 10 is not the predetermined posture, it is determined that the camera 4 and the workpiece 10 do not satisfy the predetermined positional relationship. In addition, the determination unit 43 instructs the robust detection unit 41 to detect the position of the workpiece 10 for the next image to be obtained.
[0055] When the camera 4 and the workpiece 10 do not satisfy the predetermined positional relationship, the proximity control unit 44 controls the movable part of the robot 2 so that the camera 4 and the workpiece 10 satisfy the predetermined positional relationship based on the position of the workpiece 10 detected by the robust detection unit 41. For example, the proximity control unit 44 applies a prediction filter such as a Kalman filter to the detection position of the workpiece 10 in the actual space respectively obtained in the most recent series of images, thereby being able to predict the position of the workpiece 10 in the world coordinate system when the next image is generated. In addition, the proximity control unit 44 sets a target position at which the predicted position of the camera 4 satisfies the predetermined positional relationship with respect to the workpiece 10. In order to move the camera 4 to the target position, the proximity control unit 4 sets the rotation amount and rotation direction of the servo motor 35 so that the movable part of the robot 2 on which the camera 4 is mounted moves.
[0056] At this time, the proximity control unit 44 calculates the transformation formula from the camera coordinate system to the world coordinate system when each image in the most recent series of images is generated, for example, based on the position and orientation of the camera 4 in the world coordinate system. For example, the proximity control unit 44 can detect the absolute position of each movable part of the robot 2 based on the rotation amount of the servo motor 35 from the position of each movable part when the robot 2 is in the reference posture when each image is generated. In addition, the proximity control unit 44 detects the position and orientation of the camera 4 in the world coordinate system based on the relationship between the detected absolute position and the installation position of the camera 4 and the orientation of the camera 4, and calculates the transformation formula from the camera coordinate system to the world coordinate system based on the detection result. In addition, the proximity control unit 44 detects the position of the workpiece 10 represented by the camera coordinate system when each image is generated by applying the transformation formula. In addition, as described above, the proximity control unit 44 can obtain the predicted position of the workpiece 10 in the world coordinate system when the next image is generated by applying the prediction filter to the detected position of the workpiece 10 in the world coordinate system when each image is generated.
[0057] When the predicted position of the workpiece 10 in the world coordinate system is obtained, with respect to the predetermined positional relationship, when the distance between the camera 4 and the workpiece 10 is less than the predetermined distance, the proximity control unit 44 may determine the position at the predetermined distance from the predicted position as the target position. Furthermore, the proximity control unit 44 determines the target rotation amount and target rotation direction of the servo motor 35 so that the camera 4 moves to the target position.
[0058] In addition, the predetermined positional relationship is set as that the distance between the camera 4 and the workpiece 10 is less than the predetermined distance, and the workpiece 10 is in a predetermined posture relative to the camera 4, and the position and posture of the workpiece 10 in the real space are detected by the robust detection unit 41. At this time, the proximity control unit 44 can predict the posture of the workpiece 10 when the next image is generated based on the posture of the workpiece 10 when the latest series of images are generated. At this time, the proximity control unit 44 also applies the transformation formula from the camera coordinate system to the world coordinate system to the posture of the workpiece 10 represented by the camera coordinate system when the latest series of images are generated to obtain the posture of the workpiece 10 in the world coordinate system, and applies the prediction filter to the posture of the workpiece 10 in the world coordinate system, thereby predicting the posture of the workpiece 10 in the world coordinate system when the next image is generated. In addition, the proximity control unit 44 can obtain the target position and target posture of the camera 4 when the distance between the camera 4 and the workpiece 10 is less than the predetermined distance and the workpiece 10 is in a predetermined posture relative to the camera 4 based on the predicted position and posture of the workpiece 10. The proximity control unit 44 may determine the target rotation amount and target rotation direction of the servo motor 35 so that the camera 4 moves to the target position and the camera 4 is in the target posture.
[0059] When the workpiece 10 is stationary, the proximity control unit 44 determines the target position and target posture by setting the detected position and posture of the workpiece 10 when the latest image is generated as the predicted position and predicted posture, thereby determining the target rotation amount and target rotation direction of the servo motor 35 .
[0060] When the target rotation amount and target rotation direction of the servo motor 35 are determined, the proximity control unit 44 controls the servo motor 35 as described in, for example, Japanese Patent Application Laid-Open No. 2006-172149. That is, when controlling the operation of the robot 2, the proximity control unit 44 calculates a speed command for specifying the rotation speed of the servo motor 35 based on the target rotation amount, the target rotation direction, and the actual rotation amount of the servo motor 35. The proximity control unit 44 performs speed loop control based on the speed command and speed feedback calculated by differentiating the actual rotation amount of the servo motor 35 to calculate a torque command, which indicates the amount of current supplied to the servo motor 35. Furthermore, the proximity control unit 44 performs loop control on the current supplied to the servo motor 35 based on the torque command and the current feedback fed back from the current detector (not shown) provided in the drive circuit 32 to drive the servo motor 35.
[0061] When the camera 4 and the workpiece 10 meet a predetermined positional relationship, the operation control unit 45 controls the movable parts of the robot 2 so that the tool 16 of the robot 2 moves to a position where the workpiece 10 can be operated, based on the position and posture of the workpiece 10 detected by the precision detection unit 42. At this time, the operation control unit 45, like the approach control unit 44, can apply a prediction filter such as a Kalman filter to the position and posture of the workpiece 10 in the world coordinate system respectively obtained in the most recent series of images, to obtain the predicted position and predicted posture of the workpiece 10 when the next image is generated. In addition, the operation control unit 45 can set the target position and the target orientation of the tool 16 based on the predicted position and predicted posture, thereby determining the rotation amount and rotation direction of the servo motor 35.
[0062] Alternatively, the operation control unit 45 may control the movable parts of the robot 2 so that the workpiece 10 is represented by a predetermined position and a predetermined size corresponding to the position where the tool 16 operates on the workpiece 10 in the image generated by the camera 4. In this case, the operation control unit 45 may control the movable parts of the robot 2 according to a method of controlling a robot based on an image of an object obtained by a camera, such as a position reference method or a feature reference method (for example, see Hashimoto, "Vision and Control", Conference Seminar of the Control Department of the Measurement and Automation Society, Kyoto, pp. 37-68, 2001).
[0063] In this way, the work control unit 45 controls the movable part of the robot 2 based on the relative position and posture of the workpiece 10 with respect to the camera 4 detected with high accuracy, and thus the robot 2 can be controlled so as to appropriately perform work on the workpiece 10 .
[0064] Figure 4The outline of the movable part control process including the object detection process is described. In this example, the predetermined positional relationship is set to be the distance between the camera 4 and the workpiece 10 is less than the predetermined distance Lth. Figure 4 As shown, when the camera 4 is at the position P1, the distance d1 between the camera 4 and the workpiece 10 is greater than the predetermined distance Lth, so the robust detection unit 41 uses the identifier to detect the position (or position and posture) of the workpiece 10. Then, based on the detection result, the approach control unit 44 controls the movable part of the robot 2 so that the camera 4 approaches the workpiece 10. On the other hand, when the camera 4 is at the position P2, the distance d2 between the camera 4 and the workpiece 10 is smaller than the predetermined distance Lth, so the precision detection unit 42 uses the template to detect the position and posture of the workpiece 10. Then, based on the detection result, the operation control unit 45 controls the movable part of the robot 2 so that the tool 16 on which the camera 4 is installed can move to a position where the workpiece 10 can be operated.
[0065] Figure 5 This is an operation flow chart of the movable part control process including the object detection process. The processor 34 executes the movable part control process according to the following operation flow every time an image is acquired from the camera 4. In addition, the processes of steps S101 to S102 and S104 in the following operation flow are included in the object detection process.
[0066] The determination unit 43 determines whether the camera 4 and the workpiece 10 satisfy a predetermined positional relationship based on the detection results of the position and posture of the workpiece 10 when the image was generated last time (step S101). When the camera 4 and the workpiece 10 do not satisfy the predetermined positional relationship (step S101 No), the determination unit 43 instructs the robust detection unit 41 to detect the position of the workpiece 10. Then, the robust detection unit 41 detects the position of the workpiece 10 on the image by inputting the latest image into the identifier, and detects the position of the workpiece 10 in the actual space based on the detection result (step S102). In addition, the proximity control unit 44 controls the movable part of the robot 2 based on the detected position of the workpiece 10 so that the camera 4 and the workpiece 10 satisfy the predetermined positional relationship (step S103).
[0067] On the other hand, when the camera 4 and the workpiece 10 satisfy the predetermined positional relationship (step S101 is), the determination unit 43 instructs the precision detection unit 42 to detect the position and posture of the workpiece 10. Furthermore, the precision detection unit 42 detects the position and posture of the workpiece 10 on the image by comparing the latest image with the template, and detects the position and posture of the workpiece 10 in the actual space based on the detection result (step S104). Furthermore, the operation control unit 45 controls the movable part of the robot 2 based on the detected position and posture of the workpiece 10, so that the tool 16 moves to a position where the workpiece 10 can be operated (step S105).
[0068] After step S103 or S105 , processor 34 ends the movable part control process.
[0069] As described above, the object detection device robustly detects the position of the object using the recognizer learned by the machine learning method when the imaging unit and the object do not satisfy the predetermined positional relationship. On the other hand, when the imaging unit and the object satisfy the predetermined positional relationship, the object detection device detects the position and posture of the object on the image with high accuracy by comparing the template obtained in advance and representing the features of the appearance of the object when observed from a predetermined direction with the image. Therefore, the object detection device can detect the object even when the relative positional relationship of the object with respect to the imaging unit changes. Moreover, the object detection device can obtain the position of the object in the actual space using only one imaging unit without using the stereo method that uses multiple cameras to obtain the position of the object in the actual space. In addition, the control device of the robot equipped with the object detection device controls the movable part of the robot according to the position of the object detected by the recognizer when the imaging unit and the object do not satisfy the predetermined positional relationship, so that the imaging unit and the object satisfy the predetermined positional relationship. On the other hand, when the imaging unit and the target object satisfy a predetermined positional relationship, the control device controls the movable part of the robot according to the position and posture of the target object detected as a result of the comparison, so that the operation can be performed on the target object. Therefore, even if the relative positional relationship between the imaging unit and the target object changes, the control device can continue to track the target object and can appropriately control the movable part of the robot according to the positional relationship.
[0070] Furthermore, when the workpiece 10 is cylindrical and its shape does not change even when rotated about its central axis, or when the workpiece 10 is arranged in a specific posture, the precision detection unit 42 may not detect the posture of the workpiece 10 on the image and in the real space.
[0071] According to a modified example, a position information acquisition unit for acquiring position information indicating a relative position relative to the workpiece 10 may be provided separately from the camera 4, wherein the position information indicating a relative position relative to the workpiece 10 is used by the precision detection unit 42 to detect the position and posture of the workpiece 10. For example, one or more pressure sensors may be provided at the front end of the tool 16. The pressure sensor is an example of a position information acquisition unit. At this time, the precision detection unit 42 may refer to a reference table indicating a relationship between the distance between a predetermined portion of the tool 16 and the workpiece 10 and the pressure detected by the pressure sensor to detect the distance between the predetermined portion of the tool 16 and the workpiece 10 corresponding to the detected pressure. In addition, such a reference table is pre-stored in the memory 33, for example. As in the above-mentioned embodiment, the operation control unit 45 controls the movable part of the robot 2 based on the image obtained by the camera 4, so that the tool 16 moves to a position where the pressure sensor and the workpiece 10 can contact each other. Thereafter, the operation control unit 45 controls the movable part of the robot 2 so that the pressure detected by the pressure sensor becomes a predetermined value. At this time, the pressure detected by the pressure sensor is an example of position information.
[0072] Alternatively, a camera separate from the camera 4 may be installed on the movable part of the robot 2, and the precision detection unit 42 performs the same processing as the above-mentioned embodiment on the image generated by the separate camera, thereby detecting the position and posture of the workpiece 10. Alternatively, a distance sensor such as a depth camera may be installed on the movable part of the robot 2. And, the precision detection unit 42 may detect the position and posture of the workpiece 10 based on the distance sensor based on the measured value of the distance to each part of the workpiece 10 detected by the distance sensor. In addition, the separate camera and distance sensor are another example of the position information acquisition unit, and the image generated by the separate camera and the distance measurement value of the distance sensor are another example of the position information.
[0073] According to another embodiment, the camera 4 may be fixedly installed separately from the movable part of the robot 2. For example, the camera 4 may be installed facing downward on the ceiling of the room where the robot 2 is installed. In this case, for each pixel of the image acquired by the camera 4, a distance table may be pre-stored in the memory 33, and the distance table indicates the distance between the camera 4 and the workpiece 10 when the workpiece 10 is located at the orientation corresponding to the pixel as viewed from the camera 4. When the robust detection unit 41 and the precision detection unit 42 detect the area representing the workpiece 10 on the image, they refer to the distance table to estimate the distance corresponding to the pixel where the center of gravity of the area is located as the distance from the camera 4 to the workpiece 10, thereby detecting the position of the workpiece 10 in the actual space. In addition, at this time, since the position of the camera 4 does not change, the robust detection unit 41 and the precision detection unit 42 apply the transformation formula from the camera coordinate system to the world coordinate system based on the position of the camera 4 to the position and posture of the workpiece 10 represented by the camera coordinate system, thereby being able to easily obtain the position and posture of the workpiece 10 in the world coordinate system. Furthermore, the approach control unit 44 and the work control unit 45 may control the movable part of the robot 2 in the same manner as in the above-described embodiment, based on the position and posture of the workpiece 10 represented by the world coordinate system.
[0074] In addition, according to another embodiment, the object detection device and the control device of the robot 2 may also be provided separately.
[0075] Figure 6 : is a schematic structural diagram of a robot system 100 of this modification. The robot system 100 includes: a robot 2, a control device 3 for controlling the robot 2, a camera 4 for photographing a workpiece 10 as an example of a target object, which is mounted on a movable part of the robot 2, and an object detection device 6. The robot 2 and the control device 3 are communicably connected via a communication line 5. Furthermore, the camera 4 and the object detection device 6 are communicably connected via a communication line 7, and the control device 3 and the object detection device 6 are communicably connected via a communication line 8. In addition, the camera 4, the control device 3, and the object detection device 6 can be communicably connected to each other via a communication network based on a predetermined communication standard.
[0076] Robot system 100 and Figure 1 The robot system 1 shown in FIG. 1 is different in that an object detection device 6 is provided separately from the control device 3. Therefore, the object detection device 6 and its related parts will be described below.
[0077] Figure 7 : is a schematic diagram of the structure of the object detection device 6. The object detection device 6 includes a communication interface 61, a memory 62, and a processor 63. Furthermore, the object detection device 6 may include a user interface such as a touch panel (not shown).
[0078] The communication interface 61 includes, for example, a communication interface for connecting the object detection device 6 to the communication line 7 and the communication line 8, and a circuit for executing a process of sending and receiving signals via the communication line 7 and the communication line 8. In addition, the communication interface 61 receives an image from the camera 4 via the communication line 7, for example, and forwards it to the processor 63. In addition, the communication interface 61 receives information indicating the position and direction of the camera 4 determined according to the absolute position of each movable part when each image is generated from the control device 3, and forwards it to the processor 63. In addition, the communication interface 61 receives information indicating the detected position and posture of the workpiece 10 from the processor 63, and outputs it to the control device 3 via the communication line 8.
[0079] The memory 62 is another example of a storage unit, and includes, for example, a readable and writable semiconductor memory and a read-only semiconductor memory. The memory 62 may include a storage medium such as a semiconductor memory card, a hard disk, or an optical storage medium, and a device for accessing the storage medium.
[0080] The memory 62 stores various information used in the object detection process executed by the processor 63 of the object detection device 6, for example, a parameter group that specifies the recognizer, a template for detecting the workpiece 10, camera parameters, and an image obtained from the camera 4. In addition, the memory 62 stores information indicating the position and posture of the workpiece 10 detected as a result of the object detection process.
[0081] The processor 63 is an example of a control unit, and includes, for example, a Central Processing Unit (CPU) and its peripheral circuits. In addition, the processor 63 may include a numerical calculation processor. In addition, the processor 63 performs object detection processing. That is, each time an image is received from the camera 4, the processor 63 executes Figure 3 The processing of the robust detection unit 41, the precision detection unit 42 and the determination unit 43 among the various units of the processor 34 of the control device 3 shown in the figure is sufficient. In addition, the processor 63 outputs the information indicating the position and posture of the detected workpiece 10 to the control device 3 via the communication interface 61 and the communication line 8. In addition, the processor 63 may also include information indicating the determination result of whether the camera 4 and the workpiece 10 satisfy a predetermined positional relationship in the information indicating the detected position and posture of the workpiece 10. The processor 34 of the control device 3 may execute the processing of the approach control unit 44 and the operation control unit 45 according to the received position and posture of the workpiece 10.
[0082] According to this modification, the object detection process and the control of the robot are executed by separate devices, so the calculation load on the processor of the robot control device is reduced.
[0083] The robust detection unit 41 of the processor 63 may detect only the position of the workpiece 10 or the position and posture of the workpiece 10 on each image received from the camera 4, without detecting the position and posture of the workpiece 10 in the actual space. Similarly, the precision detection unit 42 of the processor 63 may detect only the position of the workpiece 10 or the position and posture of the workpiece 10 on each image received from the camera 4, without detecting the position and posture of the workpiece 10 in the actual space. In this case, the processor 63 outputs the position and posture of the workpiece 10 on the image to the control device 3 via the communication interface 61 and the communication line 8. In addition, the processor 34 of the control device 3 may perform processing for detecting the position and posture of the workpiece 10 in the actual space based on the position and posture of the workpiece 10 on the image detected by the robust detection unit 41 and the precision detection unit 42.
[0084] Furthermore, according to another variation, a plurality of servers communicably connected to the camera 4 and the control device 3 may also perform object detection processing. In this case, for example, one of the plurality of servers performs the processing of the robust detection unit 41, another of the plurality of servers performs the processing of the precise detection unit 42, and another of the plurality of servers performs the processing of the determination unit 43.
[0085] In addition, the computer program for executing the processing of each part of the processor 34 of the control device 3 can be provided by storing the computer program in a computer-readable removable storage medium such as a semiconductor memory, a magnetic storage medium, or an optical storage medium.
[0086] All examples and specific terms cited herein are intended to have teaching purposes, to help readers understand the present invention and the concept that the inventors have contributed to the development of the technology, and it should be understood that the present invention is not limited to the specific examples and conditions such as the structure of any example in this specification that represent the advantages and disadvantages of the present invention. The embodiments of the present invention are described in detail, but it should be understood that various changes, substitutions and modifications can be made thereto without departing from the spirit and scope of the present invention.
Claims
1. An object detection device, characterized in that: have: a storage unit storing a template representing features of an appearance of a target object when the target object is observed from a predetermined direction; a robust detection unit that detects the position of the object on the image by inputting the image into a recognizer that has been pre-learned to detect the object from the image when a photographing unit that photographs the object to generate an image showing the object and the object do not satisfy a predetermined positional relationship; as well as A precise detection unit detects the position of the target object on the image by comparing the image with the template when the imaging unit and the target object satisfy the predetermined positional relationship.
2. The object detection device according to claim 1, characterized in that: The robust detection unit detects the position of the target object in the real space based on the detected position of the target object on the image, The precise detection unit detects the position of the target object in the real space based on the detected position of the target object on the image.
3. The object detection device according to claim 2, characterized in that: The precision detection unit detects the position and posture of the object on the image based on the relative position relationship of the template with respect to the image when the image is most consistent with the template, and detects the position and posture of the object in the actual space based on the position and posture of the object on the image.
4. The object detection device according to claim 2 or 3, characterized in that: The recognizer further learns by detecting the posture of the object shown in the image, and the robust detection unit detects the position and posture of the object on the image by inputting the image into the recognizer, and detects the position and posture of the object in the actual space based on the position and posture of the object on the image.
5. The object detection device according to claim 2 or 3, characterized in that: The object detection device also has a determination unit, which determines whether the imaging unit and the object object satisfy the predetermined positional relationship based on the position of the object in the actual space and the position of the imaging unit. When the imaging unit and the object object do not satisfy the predetermined positional relationship, the robust detection unit detects the position of the object object on the image. On the other hand, when the imaging unit and the object object satisfy the predetermined positional relationship, the precision detection unit detects the position of the object object on the image.
6. A control device for an automatic machine having at least one movable part, characterized in that: The control device has: a storage unit storing a template representing features of the appearance of the target object to be worked on by the automatic machine when the target object is observed from a predetermined direction; a robust detection unit that detects the position of the object on the image by inputting the image into a recognizer that has been pre-learned by detecting the object from the image, when a camera mounted on the movable unit that captures the object to generate an image showing the object and the object do not satisfy a predetermined positional relationship, and detects the position of the object in the actual space based on the position of the object on the image; a precision detection unit that detects the position of the object on the image by comparing the image with the template when the imaging unit and the object satisfy the predetermined positional relationship, and detects the position of the object in the actual space based on the position of the object on the image, or detects the position of the object in the actual space based on the position information acquired by a position information acquisition unit that acquires the position information indicating a relative position with respect to the object; a proximity control unit for controlling the movable unit so that the imaging unit and the target object satisfy the predetermined positional relationship according to the position of the target object in the actual space detected by the robust detection unit when the imaging unit and the target object do not satisfy the predetermined positional relationship; as well as An operation control unit controls the movable unit so that the movable unit moves to a position where the automatic machine can perform a predetermined operation on the object, based on the position of the object in the actual space detected by the precision detection unit, when the shooting unit and the object satisfy the predetermined positional relationship.
7. The control device according to claim 6, characterized in that: The predetermined positional relationship is that the distance between the imaging unit and the target object is less than a predetermined distance.
8. The control device according to claim 6, characterized in that: The predetermined positional relationship is that the distance between the imaging unit and the target object is less than a predetermined distance, and the target object is in a predetermined posture relative to the imaging unit. The recognizer is further learned in a manner of detecting the posture of the object shown in the image, the robust detection unit detects the position and posture of the object on the image by inputting the image into the recognizer, and detects the position and posture of the object in the actual space based on the position and posture of the object on the image. The proximity control unit controls the movable unit according to the position and posture of the target object in the actual space detected by the robust detection unit so that the distance between the imaging unit and the target object is less than the predetermined distance and the target object assumes the predetermined posture relative to the imaging unit.
9. A computer program product for object detection, characterized in that The computer program product for object detection causes a computer to execute the following processing: When a photographing unit that photographs a target object to generate an image showing the target object and the target object do not satisfy a predetermined positional relationship, the image is input to a recognizer that has been pre-learned by detecting the target object from the image, thereby detecting the position of the target object on the image, When the imaging unit and the target object satisfy the predetermined positional relationship, a template representing features of the appearance of the target object when the target object is observed from a predetermined direction is compared with the image, thereby detecting the position of the target object on the image.
Citation Information
Patent Citations
Controller at end point of machine
JP2006172149A
Control method of robot device, and the robot device
JP2009083095A
Control device, robot, and robot system
JP2018126799A
Object handling device, control device, and calibration method
JP2018158439A
Learning device, learning method, learning model, detection device and gripping system
JP2018205929A
