Training data generation method, training data generation system, and training data generation support program

Abstract

This generates training data for a model that estimates the coordinates of feature points of connectors in an image. [Solution] A method for generating training data, comprising the steps of: preparing three-dimensional position information of feature points of a connector; acquiring a reference connector image by imaging the connector with a three-dimensional vision sensor; calculating the three-dimensional position coordinates of selected feature points captured in the reference connector image; calculating the three-dimensional position coordinates of the feature points from the three-dimensional position coordinates of the selected feature points and the three-dimensional position information of the feature points; acquiring a connector image by imaging the connector from a different position; calculating the position of the feature points on the connector image based on the position and orientation of the three-dimensional vision sensor at the time of capturing the connector image and the three-dimensional position coordinates of the feature points; and labeling the connector image with the position of the feature points on the connector image.

Description

Technical Field

[0001] The present invention relates to a method for generating teacher data used for learning a model that estimates the positions of feature points of connectors included in an image.

Background Art

[0002] In the manufacturing process of electronic devices and the like, the automation of connector connection work using robots has been progressing. To handle a connector with a robot, it is first necessary to accurately recognize the position and orientation of the connector in real time. The recognition of the position and orientation of the connector has conventionally been performed by a visual sensor and image processing, but in recent years, attempts have been made to estimate the position and orientation of the connector using a machine learning model.

[0003] Patent Document 1 describes a method of cutting out an image portion related to a connector from image data including a minute connector and estimating the positions of feature points of the connector included in the image by using a learned feature point estimation learning model. Further, it is described that the positions of the feature points are estimated in two images of a stereo camera and the position and orientation of the connector in a three-dimensional space are calculated.

[0004] Also, methods for automatically generating teacher data for machine learning have been disclosed in several documents.

[0005] Patent Document 2 describes a method of three-dimensionally photographing the surroundings of an object with a three-dimensional camera to create three-dimensional solid data of the object for the purpose of improving the efficiency of deep learning annotation, and then adding annotation information to an image of the object photographed with the three-dimensional camera or another camera based on the three-dimensional solid data.

[0006] Patent Document 3 describes an annotation device that captures images a plurality of times while changing the position of an object or an imaging device, converts the position of the object into a position in the image coordinate system at the time of imaging, and stores the position information of the object together with the image.

[0007] Patent Document 4 describes a method for creating a CAD model representing the three-dimensional shape of an object by combining multiple distance images taken of the target object from multiple angles, generating an extracted image that identifies the gripping position, and using the combined distance image and extracted image as training data for learning. [Prior art documents] [Patent Documents]

[0008] [Patent Document 1] Japanese Patent Publication No. 2023-034203 [Patent Document 2] Japanese Patent Publication No. 2023-147385 [Patent Document 3] Japanese Patent Publication No. 2021-071878 [Patent Document 4] Japanese Patent Publication No. 2019-056966 [Overview of the project] [Problems that the invention aims to solve]

[0009] To train a single estimation model using machine learning, it is necessary to prepare a large amount of training data. To train the model described in Patent Document 1, which estimates the coordinates of feature points of a connector on an image of the connector, it is necessary to prepare training data in which the coordinates of the feature points on the connector image are labeled as the correct answer. Labeling a large number of connector images with the correct answers was a time-consuming task.

[0010] However, the methods described in Patent Documents 2-4 regarding the automatic generation of training data involve labeling bounding boxes and other elements surrounding objects in images, or extracting gripping regions. The estimation models used for machine learning were all aimed at object detection or object region segmentation. The methods described in Patent Documents 2-4 could not be applied to generating training data for the model described in Patent Document 1, which estimates the position of feature points of connectors on an image in order to grip tiny connectors.

[0011] The present invention has been made in consideration of the above, and aims to provide a method for efficiently generating training data for machine learning a model that estimates the coordinates of feature points of connectors on an image, as described in Patent Document 1. [Means for solving the problem]

[0012] The present invention provides a method for generating training data, comprising the steps of: preparing three-dimensional positional information of feature points of a connector; acquiring a reference connector image by imaging the connector with a three-dimensional vision sensor mounted on a robot; calculating the three-dimensional positional coordinates of at least three selected feature points selected from the feature points visible in the reference connector image based on the reference connector image; calculating the three-dimensional positional coordinates of the feature points from the three-dimensional positional coordinates of the selected feature points and the three-dimensional positional information of the feature points; moving the robot to move the three-dimensional vision sensor and acquiring a connector image by imaging the connector from a different position; calculating the position of the feature points on the connector image based on the position and orientation of the three-dimensional vision sensor at the time of image acquisition and the three-dimensional positional coordinates of the feature points; and labeling the connector image with the position of the feature points on the connector image. The order in which each step is performed is not limited to the order described herein, and any order that enables the generation of training data, which is the objective of the invention, is acceptable.

[0013] Here, the 3D position information of feature points refers to information that reveals the positional relationship between multiple feature points in 3D space. The 3D position coordinates of feature points refer to the coordinates of the feature points in 3D space, and the coordinate system is not particularly limited, but for example, it is the robot coordinate system based on the robot's base. The position and orientation of the 3D vision sensor refers to the position and orientation of the 3D vision sensor. The position of a feature point on the image refers to the coordinates of the feature point in the image coordinate system, which is a 2D coordinate system. In addition, labels used to label connector images may be saved in the same file as the connector image, but are usually saved as a separate file linked to the connector image.

[0014] This method allows us to label images of connectors with the positions of their feature points within the image, generating training data for a model that estimates the positions of feature points of connectors contained in an image.

[0015] Preferably, the above-described method for generating training data further includes the steps of: acquiring a reference marker image by imaging a correction marker with the 3D vision sensor from the same position as when the reference connector image was captured; calculating the 3D position coordinates of the correction marker based on the reference marker image; moving the 3D vision sensor by operating the robot to image the correction marker from the same position as when the connector image was captured to acquire a marker image; calculating the position of the correction marker on the marker image as a predicted marker position based on the position and orientation of the 3D vision sensor at the time the marker image was captured and the 3D position coordinates of the correction marker; and correcting the position of the feature point on the connector image acquired by imaging from the same position as the marker image using the difference between the position of the correction marker on the marker image and the predicted marker position as a correction value. The order in which each step is performed is not limited to the order described herein, and any order that enables the generation of training data, which is the objective of the invention, is acceptable.

[0016] Preferably, in any of the above methods for generating training data, the 3D vision sensor is a stereo camera.

[0017] The training data generation system of the present invention comprises a 3D vision sensor, a robot for moving the 3D vision sensor, and a control device. The control device stores 3D position information of feature points of a connector, instructs the robot to move the 3D vision sensor to a reference imaging position, instructs the 3D vision sensor to image the connector and acquire a reference connector image, calculates the 3D position coordinates of at least three selected feature points from the feature points captured in the reference connector image, calculates the 3D position coordinates of the feature points from the 3D position coordinates of the selected feature points and the 3D position information of the feature points, instructs the robot to move the 3D vision sensor to a different position, instructs the 3D vision sensor to image the connector and acquire a connector image, calculates the position of the feature points on the connector image based on the position and orientation of the 3D vision sensor at the time the connector image was captured and the 3D position coordinates of the feature points, and stores the position of the feature points on the connector image as a label on the connector image.

[0018] The present invention provides a training data generation support program, which supports the generation of training data, and includes the following functions: instructing a robot equipped with a 3D vision sensor to move the 3D vision sensor to a reference imaging position; instructing the 3D vision sensor to image a connector and acquire a reference connector image; calculating the 3D position coordinates of at least three selected feature points selected from the feature points captured in the reference connector image; calculating the 3D position coordinates of the feature points from the 3D position coordinates of the selected feature points and the 3D position information of the feature points; instructing the robot to move the 3D vision sensor to a different position; instructing the 3D vision sensor to image the connector and acquire a connector image; calculating the position of the feature points on the connector image based on the position and orientation of the 3D vision sensor at the time the connector image was captured and the 3D position coordinates of the feature points; and labeling and storing the position of the feature points on the connector image in the connector image. [Effects of the Invention]

[0019] According to the teacher data generation method of the present invention, teacher data for training a model for estimating the coordinates of feature points of a connector on an image of the connector can be efficiently generated.

Brief Description of the Drawings

[0020] [Figure 1] It is a functional block diagram of the teacher data generation system according to the embodiment. [Figure 2] It is a process flow diagram of the teacher data generation method according to the first embodiment. [Figure 3] A: It is a diagram for explaining the feature points of the connector. B: It is an example of the three-dimensional position information of the connector. [Figure 4] It is a diagram for explaining the imaging position of the connector by the three-dimensional vision sensor. [Figure 5] It is a diagram for explaining the selected feature points on the reference connector image. [Figure 6] A: Connector image, B: It is a diagram for explaining the positions of the feature points on the connector image. [Figure 7] It is a diagram for explaining the deviation between the position of the feature point on the connector image and the calculated value. [Figure 8] It is a flow diagram of the process following FIG. 2 in the teacher data generation method according to the second embodiment. [Figure 9] A: Diagram showing the arrangement of the correction marker, B: Enlarged view of the correction marker. [Figure 10] It is a diagram showing the deviation between the position of the correction marker on the marker image and the predicted marker position.

Modes for Carrying Out the Invention

[0021] The first embodiment of the teacher data generation system and the teacher data generation method of the present invention will be described based on FIGS. 1 to 6.

[0022] Referring to FIG. 1, the teacher data generation system 18 of the present embodiment includes a stereo camera 20, a robot 30 for moving the stereo camera, and a control device 40.

[0023] The stereo camera 20 is a 3D vision sensor that captures images of the connector with two cameras and measures the distance to each point on the image using the principle of triangulation from two images taken from different viewpoints. Note that a 3D vision sensor other than a stereo camera may be used, as long as it can acquire images of the connector along with distance information.

[0024] The stereo camera 20 has an imaging unit 21 and a control unit 22, the control unit 22 having a calculation unit 23, a storage unit 24 and a communication unit 25. The imaging unit 21 has two cameras and acquires two images by imaging the connector. Hereinafter, the two images will be referred to as the left image and the right image. The calculation unit 23 of the control unit 22 calculates the distance to the target point, performs various calculations necessary for 3D measurement, and controls the entire stereo camera 20. The storage unit 24 stores various parameters obtained by calibrating the stereo camera. The storage unit 24 stores the internal parameters of each camera in the imaging unit 21 and the baseline length between the cameras, so that the calculation unit 23 can calculate the distance to the target point. It stores the conversion parameters between the coordinate system based on the part of the robot to which the stereo camera is attached, for example, the robot hand, and the sensor coordinate system. These conversion parameters are obtained by performing calibration while changing the position and orientation with the stereo camera 20 attached to the robot 30. The communication unit 25 is the interface for communication between the stereo camera and the outside.

[0025] The stereo camera 20 is attached to the end of the arm of the robot 30, or to a hand attached to the end of the arm. The imaging unit 21 of the stereo camera needs to be attached to a movable part of the robot 30, but the control unit 22 may be fixedly installed in a location separate from the imaging unit 21 and electrically connected to the imaging unit 21 by a cable. Attaching only the imaging unit 21 to the robot reduces the weight to be attached, making it easier to mount to the robot.

[0026] Robot 30 is a vertical articulated robot that moves the stereo camera 20. Other types of robots may be used as long as they can move the stereo camera to the required position and in the required orientation.

[0027] The robot 30 has a joint control unit 31 and a control unit 32, and the control unit 32 has a calculation unit 33, a storage unit 34, and a communication unit 35. The joint control unit 31 controls the operation of the actuators of each joint of the robot 30. The calculation unit 33 of the control unit 32 performs various calculations, including forward kinematics calculations to calculate the position and orientation of the end-effector from joint variables, inverse kinematics calculations to calculate joint variables from the target position and orientation of the end-effector, and coordinate transformations between coordinate systems based on each part of the robot, as well as controlling the robot 30 as a whole. An example of a coordinate transformation is the transformation between a robot coordinate system based on the robot's base and a hand coordinate system based on the hand to which the stereo camera 20 is attached. The storage unit 34 stores various data necessary for the operation and coordinate transformations of the robot 30. The communication unit 35 is the interface for communication between the robot 30 and the outside.

[0028] The control device 40 controls the entire training data generation system 18. The control device 40 has a calculation unit 41, a storage unit 42, a communication unit 43, and an input / output unit 44. The calculation unit 41 executes a training data generation support program and performs various calculations. The storage unit 42 stores position information of connector feature points and generated training data, as well as temporarily stores images referenced during training data generation and intermediate calculation results. The communication unit 43 is the interface for communication between the control device 40 and the outside world. The input / output unit 44 displays images to the operator and accepts input from the operator.

[0029] The control device 40 can be a computer such as a personal computer. The control device 40 is defined by its function and may consist of one computer or multiple computers.

[0030] Alternatively, the control device 40 and the control unit 22 of the stereo camera 20 may be configured with a single computer. In that case, the control device 40 and the control unit 22 may be placed in the same housing together with the imaging unit 21 of the stereo camera, or only the imaging unit 21 may be attached to the robot, and the control device 40 and the control unit 22 may be placed separately from the imaging unit 21. By configuring the control device 40 and the control unit 22 with a single computer, the calculation unit (41, 23), storage unit (42, 24), and communication unit (43, 25) can be integrated, so communication between the control device 40 and the stereo camera 20 can be omitted, and the procedure for the training data generation method described later can be simplified.

[0031] The processing details of each part of the stereo camera 20, robot 30, and control device 40 will be explained in the following embodiment of the training data generation method.

[0032] Next, the first embodiment of the training data generation method of this embodiment will be described in accordance with the flow chart in Figure 2.

[0033] (S10: Preparation of 3D positional information for connector feature points) Referring to Figure 3A, feature points are determined on the connector 10. In Figure 3A, the eight corners indicated by circles are designated as feature points 11 (11a-h). The position and orientation of the connector can be determined if there are three or more feature points, or more precisely, three or more points that are not on a straight line. Preferably, the eight corners of a roughly rectangular prism-shaped connector are used as feature points. This is because knowing the coordinates of the eight corners makes it easier to grasp the connector 10 with the robot's hand after estimating its position and orientation following training of the estimation model.

[0034] Prepare the 3D position information for the determined feature points 11. 3D position information for feature points refers to information that reveals the relative positions of multiple feature points in 3D space. This 3D position information can be created by obtaining the coordinates of the feature points from the 3D shape data of the connector 10, such as CAD data.

[0035] Figure 3B shows an example of the 3D position information of feature point 11. The 3D position information shown in Figure 3B is a Cartesian coordinate system in which one feature point 11a is the origin (0,0,0), feature point 11a→11b is the X-axis, and feature point 11a→11d is the Y-axis. The position coordinates of the eight feature points 11a to 11h are expressed in millimeters in CSV (comma-separated values) format. The method of representing the 3D position information of feature point 11 is not limited to this; any format that shows the relative positional relationship of each feature point is acceptable. This CSV file is stored in the memory unit 42 of the control device 40, or in the memory unit 24 of the control unit 22 of the stereo camera 20 integrated with the control device.

[0036] (S11: Determination of the imaging position and orientation of the stereo camera) The position and orientation of the stereo camera 20 when it images the connector 10 for the purpose of generating training data are determined. In the following, the position and orientation of the stereo camera when imaging will be collectively referred to simply as the "imaging position". Multiple imaging positions are determined so that a sufficient number of training data can be created. In general deep learning, around 100 to several thousand images are required, but if the number of images is too large, image acquisition takes too long, so preferably around 250 imaging positions are determined so that around 500 images can be acquired. It is preferable that the imaging positions are taken from the positions that are expected to be used when the connector 10 is actually recognized. Referring to Figure 4, it is preferable to take the imaging positions on a sphere 15 centered on the location where the connector 10 is placed. This is because the connector 10 can be imaged from various directions while keeping the distance from the connector almost constant. For example, it can be determined by changing the central angle with respect to the reference direction with respect to the position of the connector as the center within a range of ±45 degrees in two orthogonal directions (X, Y). The determined series of imaging positions are stored in the storage unit 42 of the control device 40.

[0037] (S12: Connector placement) Referring to Figure 4, position the connector 10 to be imaged. Fix the connector 10 so that it does not move during the series of imaging. It is preferable to image the connector 10 while it is suspended in mid-air, and fix the cable portion so that it does not deform, for example by hardening the cable connected to the connector with adhesive.

[0038] (S13: Acquisition of reference connector image) One of the imaging positions is designated as the reference imaging position, and the connector 10 is imaged from the reference imaging position using the stereo camera 20 to obtain a reference connector image. Specifically, the control device 40 instructs the robot 30 to move the stereo camera 20 to the reference imaging position, the control device instructs the stereo camera to image the connector 10, and the control device receives the image from the stereo camera and uses it as the reference connector image. The position and orientation of the stereo camera 20 when it is at the reference imaging position can be calculated by the robot 30's hand position and orientation information and the stereo camera's calculation unit 23 using coordinate transformation parameters (transformation parameters between the hand coordinate system and the sensor coordinate system) stored in the storage unit 24. Alternatively, the position and orientation of the stereo camera when it is at the reference imaging position can also be calculated by the control device's calculation unit 41 if the control device 40 has received these coordinate transformation parameters from the stereo camera 20 in advance.

[0039] The reference connector image needs to contain at least three feature points so that at least three feature points can be selected in the next step (S14). Therefore, the reference imaging position is set to a position where at least three feature points are captured when the connector is imaged.

[0040] Since it is necessary to obtain the image of the connector 10 and distance information of the selected feature points described later from the reference connector image, when the image is captured by the stereo camera 20, the set of left and right images is used as the reference connector image. The reference connector image is stored in the storage unit 42 of the control device 40, along with the imaging position of the stereo camera at the time the image was captured.

[0041] (S14: Selection of feature points on the reference connector image) Figure 5 shows a reference connector image. The left side of Figure 5 is the left image, and the right side is the right image. Three or more feature points are selected from the feature points visible in the reference connector image. Specifically, the control device 40 displays the left and right images of the reference connector image on the input / output unit 44, and the operator selects feature points on the screen and obtains the specified positions from the input / output unit. The selected feature points will be referred to as selected feature points below. In Figure 5, three points marked with an "X" (11a, 11e, and 11f in Figure 3A) are selected as selected feature points 13 from among several feature points visible in the reference connector image. The control device 40 stores the positions of the selected feature points 13 along with the reference connector image in the storage unit 42.

[0042] (S15: Calculation of the 3D position coordinates of selected feature points) Since the positions of the corresponding selected feature points 13 are specified on the left and right images of the reference connector image, the 3D position coordinates of the selected feature points 13 are calculated based on the specified positions. Because the stereo camera 20 has a function to calculate the distance to a point specified on the left and right images, the control device 40 can transmit the positions of the selected feature points on the left and right images to the stereo camera, causing the calculation unit 23 of the stereo camera to calculate the 3D position coordinates of the selected feature points. The 3D position coordinates of the selected feature points 13 obtained in the sensor coordinate system can be converted to the robot coordinate system using coordinate transformation parameters stored in the stereo camera 20, the robot 30, or the control device 40. The calculated 3D position coordinates of the selected feature points are stored in the storage unit 42 of the control device 40.

[0043] (S16: Calculation of the 3D position coordinates of feature points) The 3D position coordinates of all feature points 11 are calculated by fitting the 3D position coordinates of the selected feature points with the 3D position information of the feature points shown in Figure 3B. The 3D position coordinates of the selected feature points are expressed in the robot coordinate system, while the 3D position information of the feature points is expressed in a coordinate system based on the connector. Through iterative calculation, a transformation matrix can be obtained that matches the 3D position information of the selected feature points with the 3D position coordinates of the selected feature points. In reality, both the 3D position information of the selected feature points and the 3D position coordinates of the selected feature points contain slight errors, so a transformation matrix that minimizes the discrepancy between the two is found using the least squares method or similar. By applying the obtained transformation matrix to the 3D position information of the other feature points, the 3D position coordinates of all feature points 11 can be calculated. The control device 40 calculates the 3D position coordinates of the feature points 11 in the calculation unit 41 and stores them in the storage unit 42.

[0044] (S17: Acquisition of connector image) The stereo camera 20 captures images of the connector 10 from different positions to acquire connector images. Specifically, the control device 40 refers to a series of imaging positions stored in the storage unit 42 and instructs the robot 30 to move the stereo camera 20 to the next imaging position. The control device then instructs the stereo camera to image the connector 10, and the control device receives the image from the stereo camera to obtain the connector image. Unlike the reference connector image, distance information does not need to be obtained from the connector image, so two connector images, a left image and a right image, can be obtained with a single image capture by the stereo camera 20. The connector image, along with the imaging position of the stereo camera at the time of image capture, is stored in the storage unit 42 of the control device 40.

[0045] (S18: Calculation of the position of feature points on the connector image) For each of the two connector images (left and right images) obtained in S17, the position of the feature points on the image is calculated. By performing a projection transformation using the position and orientation of the stereo camera 20 at the time of connector image acquisition and the 3D position coordinates of the feature points obtained in step S16, the position of the feature points 11 on the connector image can be calculated. Figure 6A shows the two connector images (left and right images). Figure 6B is a virtual image in which the outline of the connector is represented by a dotted line, and a frame 12 connecting the feature points 11 is superimposed on it. This is displayed on the screen, which is the input / output unit 44 of the control device 40, for the operator's convenience. In step S18, the positions of all feature points 11 on the connector image, including feature points not visible in the image, can be calculated. This calculation is performed by the calculation unit 41 of the control device 40.

[0046] (S19: Labeling) The connector image is labeled with the position of the feature point on the connector image as a label, and this is used as training data. The generated training data is stored in the storage unit 42 of the control device 40. Specifically, the connector image file and the label file are linked and stored in the storage unit 42.

[0047] In steps S17 to S19 described above, one training data point is generated. In the case of a stereo camera, two connector images are obtained in a single imaging session, so two training data points are generated. By repeating this process for all planned imaging positions, a large amount of training data can be efficiently generated.

[0048] Furthermore, the left and right images of the reference connector image can also be used as training data by calculating and labeling the positions of feature points on the images.

[0049] The training data generation method of this embodiment allows for various changes in the order of each step.

[0050] For example, the preparation steps S10-S12 can be executed independently of each other, so they can be carried out in any order.

[0051] Furthermore, in the flow shown in Figure 2, for each imaging position, the following steps were performed: acquisition of a connector image (S17), calculation of the position of feature points on the connector image (S18), and labeling (S19). However, for each imaging position, steps S17 and S18 only need to be performed before step S19, and the order of steps S17 and S18 can be reversed.

[0052] Furthermore, the order of steps S17 to S19 for all imaging locations can be changed as long as the above conditions are met. In other words, steps S17, S18, and S19 can be performed for multiple planned imaging locations, provided that steps S17 and S18 are performed before step S19 for each imaging location. For example, after acquiring connector images at all imaging locations (S17), the position of feature points on the image can be calculated for all connector images (S18), and all connector images can be labeled (S19).

[0053] Furthermore, after acquiring a reference connector image (S13) and all connector images (S17), feature points may be selected on the reference connector image (S14).

[0054] Furthermore, the calculation of the 3D position coordinates of the feature points (S16) only needs to be completed before the first calculation of the position of the feature points on the connector image (S18).

[0055] The order of each step is not limited to the above-mentioned examples of modifications; any order that enables the generation of training data, which is the objective of this embodiment, is acceptable.

[0056] Next, a second embodiment of the training data generation method of the present invention will be described.

[0057] The training data generation system that executes the training data generation method of this embodiment is the same as the training data generation system 18 of the first embodiment described above. In addition to all the steps of the first embodiment described above, the training data generation method of this embodiment corrects the position of feature points on the connector image using correction markers.

[0058] The training data generation method of the first embodiment made it possible to efficiently generate training data. However, depending on the type and arrangement of the robot 30 used, the position of the feature points on the connector image sometimes deviated from the calculated value based on the position and orientation of the stereo sensor at the time of imaging and the three-dimensional position coordinates of the feature points (Figure 7).

[0059] As a result of repeated experiments by the inventors, (1) no shift occurred when taking images by returning to the reference imaging position from another position, and (2) the direction and amount of the shift were determined by the imaging position of the stereo camera and were reproducible. From these results, it was found that the cause of the shift was the low absolute positioning accuracy when the robot moves its hand. While articulated robots have extremely high position repeatability and can repeat the same work with high precision, their absolute positioning accuracy is not so high. In the training data generation method of the first embodiment, the stereo camera is moved to a specified imaging position (S17), and the position of the feature point on the connector image is calculated based on the specified imaging position (S18). Therefore, if the distance between the stereo camera 20 attached to the hand and the connector 10 to be measured is large, it can be understood that even if the rotation angle of the robot's rotary joint is slightly different, the difference between the position of the feature point on the image and the calculated value is amplified. Therefore, in the training data generation method of this embodiment, the difference between the position on the image and the calculated value is found for each imaging position, and the calculated value is corrected.

[0060] The following describes each step newly executed in the training data generation method of this embodiment, following steps S10 to S19 of the first embodiment, in accordance with the flow chart in Figure 8.

[0061] (S20: Placement of correction markers) Referring to Figure 9, the correction marker 50 is placed near the connector 10. The correction marker 50 has a shape such as a donut-shaped concentric circle drawn on a plate-shaped substrate 51. The correction marker 50 used during the development of this embodiment was a red donut shape with a large diameter of 4 mm and a white hollow portion with a diameter of 1 mm. The shape of the correction marker is not limited to this, but a solid circle or donut shape is preferable because it is easier to recognize by image processing. The correction marker 50 is fixed so as not to move during a series of imaging.

[0062] (S21: Acquisition of reference marker image) A reference marker image is obtained by capturing a correction marker 50 from the reference imaging position when the reference connector image is captured using the stereo camera 20. The specific method is the same as for acquiring the reference connector image (S13).

[0063] (S22: Obtaining the position of the correction marker on the reference marker image) Correction markers 50 are recognized on the left and right images of the reference marker image, and the position of the correction markers on the respective images is determined. The correction markers can be recognized by image processing, and their centroid can be obtained as the position on the image. Alternatively, the operator may specify the approximate position of the marker on the reference image, and the exact centroid of the marker may be determined by image processing. The centroid position of the marker obtained by image processing may be displayed on the screen as a "+" so that the operator can visually confirm it. In this case, if the marker is donut-shaped, the "+" indicating the centroid position will be displayed in the white area in the center of the marker, making it easy for the operator to visually confirm whether the exact centroid position has been recognized. If a discrepancy in the centroid position on the image is detected, the process is restarted from acquiring the reference marker image (S21).

[0064] (S23: Calculation of the 3D position coordinates of the correction marker) Based on the positions of the correction markers 50 on the left and right images, the three-dimensional position coordinates of the correction markers are calculated. The specific calculation method is the same as that used to calculate the three-dimensional position coordinates of the selected feature points 13 (S15). The calculated three-dimensional position coordinates of the correction markers 50 are stored in the storage unit 42 of the control device 40.

[0065] (S24: Obtaining marker image) From the same imaging position as when the connector image was captured, the correction marker 50 is imaged with the stereo camera 20, and the marker image is obtained. The specific method is the same as for acquiring the connector image (S17).

[0066] (S25: Obtaining the position of the correction marker on the marker image) For each of the two marker images (left and right images), a correction marker 50 is recognized on the image, and the position of the correction marker on that image is determined. The correction marker can be recognized by image processing, and its centroid can be obtained as its position on the image.

[0067] (S26: Calculation of predicted marker position on marker image) For each of the two marker images (left and right images), the position of the correction marker 50 on the marker image is calculated from the position and orientation of the stereo camera 20 at the time of marker image acquisition and the 3D position coordinates of the correction marker calculated in step S23. This calculated value is the predicted marker position. The specific calculation method is the same as that used to calculate the position of the feature point on the connector image (S18).

[0068] (S27: Correction of the position of feature points on the connector image) Referring to Figure 10, the difference 53 between the actual position of the correction marker 50 on the marker image acquired in step S25 and the predicted position 52 of the correction marker on the marker image calculated in step S26 is the error associated with the calculation of the predicted position 52. As mentioned above, this difference 53 is determined by the imaging position of the stereo camera 20 and does not depend on whether the imaging target is the correction marker 50 or the connector 10. Therefore, this difference 53 can be used as a correction value to correct the position of the feature points on the connector image calculated in step S18. Specifically, for example, the position of each feature point on the connector image calculated in step S18 is shifted in the same direction and by the same distance as the difference 53. If the position of the feature point on the connector image is already labeled on the connector image, the label is corrected. This correction step (S27) is performed by the calculation unit 41 of the control device 40.

[0069] In steps S24 to S27 described above, one marker image of the feature point location on the connector image can be corrected. In the case of a stereo camera, two marker images are obtained in a single image capture, so two data points of the feature point locations on the connector image can be corrected. This process is repeated for all imaging positions.

[0070] According to this embodiment, even if the position of a feature point on the connector image differs from the position calculated using the stereo sensor's position and orientation and the feature point's 3D position information, this difference can be corrected, enabling more accurate labeling. This embodiment is particularly effective when the robot 30 is a vertical articulated robot, as errors in the joint rotation angle tend to cause discrepancies between the position of the feature point on the connector image and the calculated position.

[0071] The training data generation method of this embodiment can also be modified in various ways by changing the order of each step.

[0072] For example, in the flow shown in Figure 8, for each imaging position, the following steps were performed: acquisition of a marker image (S24), acquisition of the position of the correction marker on the marker image (S25), calculation of the predicted marker position on the marker image (S26), and correction of the position of the feature points on the connector image (S27). However, for each imaging position, steps S24 to S27 only need to be performed in this order, with steps S24 and S25 before step S27, and step S26 before step S27.

[0073] Furthermore, the order of steps S24 to S27 for all imaging positions can be changed as long as the above conditions are met. In other words, steps S24, S25, S26, and S27 can be performed for multiple planned imaging positions, provided that steps S24 and S25 are performed in this order before step S27 for each imaging position, and step S26 is performed before step S27. For example, after calculating the predicted marker positions on the marker images for all imaging positions (S26), marker images may be acquired for all imaging positions (S24), the positions of correction markers may be acquired on all marker images (S25), and the positions of feature points on all connector images may be corrected (S27).

[0074] Alternatively, processing of the correction markers may be performed before processing of the connectors. In this case, in step S20, with the connector 10 absent, correction markers 50 may be placed near the planned location of the connector (S20), and after acquiring reference marker images and marker images at all imaging positions (S21, S24), the correction markers may be removed and the connectors may be placed (S12). In this case, the calculation of the position of feature points on the connector image (S18) may be performed together with the correction of the position of feature points on the connector image (S27), and the corrected position of the feature points may be labeled on the connector image. This eliminates the need to label the position of the feature points before correction (S19).

[0075] The order of each step is not limited to the above-mentioned modification examples; it is sufficient as long as it enables the generation of training data in which the positions of feature points on the connector image are corrected using correction markers, which is the objective of this embodiment. Furthermore, since each coordinate system can be transformed using the various transformation parameters described above, any coordinate system may be used as the reference in various calculations.

[0076] The present invention is not limited to the embodiments or modifications described above, and various further modifications are possible within the scope of its technical concept. [Explanation of Symbols]

[0077] 10 connectors 11, 11a~h Feature points 12 Frames formed by connecting feature points 13 Selected Feature Points 15 Spherical 18. Training Data Generation System 20. Stereo camera (3D vision sensor) 21 Imaging Department 22 Control Unit 23 Arithmetic section 24 Memory section 25 Communications Department 30 robots 31 Joint control unit 32 Control Unit 33 Arithmetic section 34 Storage section 35 Communications Department 40 Control device 41 Arithmetic section 42 Storage section 43 Communications Department 44 Input / output section 50 Correction Markers 51 Base material 52 Marker predicted position 53. Discrepancy between the position of the correction marker on the marker image and the predicted position of the marker.

Claims

1. The process involves preparing three-dimensional positional information of the connector's characteristic points, A step of acquiring a reference connector image by imaging the connector with a 3D vision sensor attached to the robot, A step of calculating the three-dimensional position coordinates of at least three selected feature points selected from the feature points shown in the reference connector image, based on the reference connector image. A step of calculating the three-dimensional position coordinates of the feature point from the three-dimensional position coordinates of the selected feature point and the three-dimensional position information of the feature point, The process of moving the three-dimensional vision sensor by operating the robot to capture images of the connector from different positions and acquire a connector image, A step of calculating the position of the feature point on the connector image based on the position and orientation of the 3D vision sensor at the time the connector image was captured and the 3D position coordinates of the feature point, The process of labeling the connector image with the position of the feature point on the connector image, A method for generating training data that includes the following.

2. In addition to all the steps described in claim 1, The process of acquiring a reference marker image by capturing a correction marker with the 3D vision sensor from the same position as when the reference connector image was captured, A step of calculating the three-dimensional position coordinates of the correction marker based on the reference marker image, The process of moving the robot to move the three-dimensional vision sensor and taking an image of the correction marker from the same position as when the connector image was taken, thereby acquiring a marker image, A step of calculating the position of the correction marker on the marker image as the predicted marker position, based on the position and orientation of the three-dimensional visual sensor at the time the marker image was captured and the three-dimensional position coordinates of the correction marker, with respect to the marker image; The process of correcting the position of the feature point on the connector image acquired by imaging from the same position as the marker image, using the difference between the position of the correction marker on the marker image and the predicted position of the marker as a correction value, A method for generating training data that includes the following.

3. The aforementioned three-dimensional visual sensor is a stereo camera. The method for generating training data according to claim 1 or 2.

4. 3D vision sensor and, A robot that moves the aforementioned three-dimensional vision sensor, It has a control device, The control device is The three-dimensional positional information of the connector's characteristic points is stored. The robot is instructed to move the three-dimensional vision sensor to the reference imaging position, and the three-dimensional vision sensor is instructed to image the connector to acquire a reference connector image. The three-dimensional position coordinates of at least three selected feature points, chosen from the feature points shown in the reference connector image, are calculated. The three-dimensional position coordinates of the selected feature point are calculated from the three-dimensional position coordinates of the feature point and the three-dimensional position information of the feature point. The robot is instructed to move the 3D vision sensor to a different position, and the 3D vision sensor is instructed to image the connector and acquire a connector image. Based on the position and orientation of the 3D visual sensor at the time the connector image was captured and the 3D position coordinates of the feature points, the position of the feature points on the connector image is calculated. The connector image is stored with the position of the feature point on the connector image labeled. A system for generating training data.

5. A program that supports the generation of training data, A robot equipped with a 3D vision sensor is instructed to move the 3D vision sensor to a reference imaging position, and the 3D vision sensor is instructed to image the connector to acquire a reference connector image. The three-dimensional position coordinates of at least three selected feature points, chosen from the feature points shown in the reference connector image, are calculated. The three-dimensional position coordinates of the selected feature point are calculated from the three-dimensional position coordinates of the feature point and the three-dimensional position information of the feature point. The robot is instructed to move the 3D vision sensor to a different position, and the 3D vision sensor is instructed to image the connector and acquire a connector image. Based on the position and orientation of the 3D visual sensor at the time the connector image was captured and the 3D position coordinates of the feature points, the position of the feature points on the connector image is calculated. The connector image is stored with the position of the feature point on the connector image labeled. A training data generation support program.

Images