CONTROL DEVICE AND ROBOT CONTROL DEVICE FOR ESTIMATING CONNECTOR FEATURE POINTS, CONNECTOR FEATURE POINT ESTIMATION METHOD, AND SYSTEM

Abstract

To provide a robot control device which gives, to the robot, data for handling by recognizing an attitude and position of a connector in real time.SOLUTION: A control device in a robot control device includes: an image acquisition unit for acquiring an image containing a connector; and an estimation unit which inputs the acquired image into a feature point estimation learning model being machine learned for estimating a feature point of the connector in the image so as to acquire an estimation result obtained by estimating the feature point of the connector contained in the image from the feature point estimation learning model.SELECTED DRAWING: Figure 1A

Description

[Technical Field]

[0001] The present disclosure relates to an industrial robot, and in particular to a control device, a robot control device, a feature point estimation method, and a system that estimates feature points of a connector. [Background technology]

[0002] There is a demand for automation using robots for the insertion of connectors in the manufacturing process of electronic devices, etc. Note that connectors (terminals, plugs, adapters, etc.) here refer to components that are attached to the ends of cables to connect electric wires, optical fibers, etc. (cables) to terminal blocks or various devices.

[0003] For a robot to handle a connector, it is essential that the orientation and position of the connector be accurately recognized using image processing technology, and that the recognized data be provided to the robot in real time. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 6-188061 [Patent Document 2] Japanese Patent Application Publication No. 2018-180756 Summary of the Invention [Problem to be solved by the invention]

[0005] The present disclosure aims to provide a control device that estimates the characteristic points of a connector and further estimates the posture of the connector based on the estimated characteristic points, and a control device that provides a robot with data for operating the robot based on the posture of the connector. [Means for solving the problem]

[0006] In order to solve the above problem, a control device in a robot control device according to the present disclosure includes: an image acquisition unit that acquires an image including the connector; The system includes an estimation unit that inputs the acquired image into a trained feature point estimation learning model that has undergone machine learning to estimate feature points of connectors in the image, and acquires estimation results that estimate feature points of connectors included in the image from the feature point estimation learning model. [Effects of the Invention]

[0007] The control device in the robot control device according to the present disclosure can recognize the orientation of the connector and provide the robot with data for operating the robot based on the orientation of the connector. [Brief explanation of the drawings]

[0008] [Figure 1A] FIG. 1A is a block diagram of a robot control device in a connector gripping system according to an embodiment. [Figure 1B] FIG. 1B is a diagram illustrating the overall configuration of a connector holding system according to an embodiment. [Figure 2] FIG. 2 is a flowchart of a method for calculating the attitude and position of a connector according to an embodiment. [Figure 3A] Figure 3A shows an example of the output of a trained machine learning model for object detection. [Figure 3B] Figure 3B shows an example output of a trained machine learning model for object detection. [Figure 4] Figure 4 is an example image of a connector with numbered corners (vertices). [Figure 5] Figure 5 shows two examples of training data for the feature point estimation machine learning model. [Figure 6] Figure 6 shows examples (three examples) of training data and estimation results for the feature point estimation machine learning model. [Figure 7] Figure 7 is an example image of a connector with display of pose and position data. [Figure 8] FIG. 8 is a diagram illustrating an example of a U-net network structure. [Figure 9] FIG. 9 is a diagram schematically illustrating an example of a method using a machine learning model for extracting feature points of a connector from image data captured by an imaging unit of a robot control device. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, more detailed explanation than necessary may be omitted. For example, detailed explanation of well-known matters or redundant explanation of substantially the same configuration may be omitted. This is to avoid unnecessary redundancy in the following explanation and to facilitate understanding by those skilled in the art.

[0010] The inventors have provided the accompanying drawings and the following description to enable those skilled in the art to fully understand the present disclosure, and do not intend for them to limit the subject matter described in the claims.

[0011] 1. [Background to this disclosure] Automating connector insertion processes is required in the assembly and manufacturing of electronic devices such as home appliances, game consoles, tablets, and PCs. However, connectors attached to the ends of flexible cables are susceptible to changes in posture and position even with the slightest external force. Even for a short period of time, it is difficult for these postures and positions to remain constant. Furthermore, when multiple connectors are supplied in a randomly arranged container, the posture and position of each connector will not be consistent.

[0012] Since the orientation and position are not constant, it is not possible for a robot to handle a connector by teaching it. In order for a robot to handle a connector, data regarding the orientation and position of the connector that is recognized in real time must be continuously provided to the robot.

[0013] However, when using an imaging device such as a camera to recognize the orientation and position of a connector, for example, through matching processing, there is a problem that it is difficult to resolve the issue of robustness regarding the background of the connector, environmental lighting, etc.

[0014] Therefore, the inventors of the present application came up with the idea of ​​extracting an image portion relating to a minute connector from image data including the connector, and then using a trained machine learning model to recognize the orientation and position of only the connector. The orientation and position here refer to the orientation and position in three-dimensional space.

[0015] 2. [Embodiment] 2.1. Configuration of the embodiment 2.1.1.Configuration of the Connector Holding System The connector holding system according to the embodiment is a system for handling connectors. Fig. 1B is an overall configuration diagram of a connector holding system 2 according to the present embodiment.

[0016] 1B, a connector holding system 2 according to this embodiment includes an articulated robot 8 and a robot control device 4. A robot hand 10 is attached to the tip of the arm of the robot 8. The robot hand 10 holds a connector 14 attached to the tip of a cable 16 and handles it appropriately.

[0017] The robot 8 has a calculation unit 12. The calculation unit 12 of the robot 8 performs various calculations such as forward kinematics and inverse kinematics, and controls the entire robot 8 including the robot hand 10. The calculation unit 12 of the robot 8 is also connected to a control device 30 of the robot control device 4 (described later) via a wire (for example, a cable 20) or wirelessly, and controls the operation of the robot 8 based on data transmitted from the robot control device 4.

[0018] The structure of the robot hand 10 that grips the connector 14 is not particularly limited, and any known structure can be used, such as one with multiple rotary joints, a gripper with various structures that can open and close the fingers, or a robot hand that can grip by suction.

[0019] The robot control device 4 captures an image of the connector 14. The robot control device 4 calculates the posture and position of the connector 14 by capturing the image using a control device 30 (described later).

[0020] The connector 14 used in the present disclosure is not particularly limited in type or shape. The connector 14 includes, for example, those called plugs, jacks, receptacles, housings, etc. When the connector is attached to the end of a flexible linear object such as a cable, bending or twisting of the cable or the like is likely to cause an error in the gripping angle when the robot hand 10 grips the connector, so the merits of using the robot control device 4 according to this embodiment are particularly great. Furthermore, the connector 14 may be attached to the end of multiple flexible linear objects, such as when the connector 14 is attached to the end of a bundle of multiple cables.

[0021] In this embodiment, as will be described later, the robot control device 4 detects the connector, which is an object, using a trained machine learning model for object detection, and further derives the connector's posture using a trained feature point estimation machine learning model. The trained machine learning model for object detection must be trained so that it can identify connectors as objects. The trained feature point estimation machine learning model does not need to be trained for each connector type in order to accurately derive the connector's posture. Note that in this disclosure, the "XHP-4" manufactured by Japan Solderless Terminal Mfg. Co., Ltd. is used as an example of a connector.

[0022] Note that detection of connectors as objects can be achieved not only by the trained machine learning model for object detection described above, but also by using an image processing-based recognition processing model that extracts white areas using threshold processing in a situation where the background can be fixed as black and the object as white. Furthermore, detection can also be achieved using conventional techniques such as pattern matching and CAD matching. The trained feature point estimation learning model, which will be mentioned later, is a machine learning model that estimates the feature points of connectors contained in an image by inputting an image of the connector.

[0023] 2.1.2. Robot control device configuration 1A is a block diagram showing the configuration of a robot control device 4 according to this embodiment. The robot control device 4 includes an imaging unit 24, a control device 30, and a storage device .

[0024] The imaging unit 24 may be any device capable of capturing an image of the connector. The imaging unit 24 typically includes an imaging element such as a CMOS or CCD, and imaging control means for controlling the imaging element. The imaging element is preferably an area sensor. The imaging unit 24 also preferably includes a separate optical system such as a lens.

[0025] The control device 30 is, for example, a computer device. A general-purpose computer device can be used as this computer device, and includes, for example, an interface device 32 and a memory 36 as shown in FIG. 1A. The control device 30 may also control the image capture unit 24 .

[0026] The storage device 38 is a storage device such as a disk drive or flash memory that is provided outside or inside the control device 30, and stores various databases, various data sets, and various computer programs used by the control device 30. The storage device 38 stores, for example, image data captured by the imaging unit 24, which will be described later.

[0027] The interface device 32 is an interface unit that can input data from outside and output data to outside, and includes a network terminal, a video input terminal, a USB terminal, a keyboard, a pointing device, a mouse, etc. Various data is input / output from the imaging unit 24, from outside, to the imaging unit 24, or to outside via the interface device 6. For example, the acquired data may be image data including one or more objects or image data including a connector, which will be described later. After acquisition, this data may be stored in the storage device 38. The data stored in the storage device 38 may be input to the control device 30 via the interface device 32 as appropriate. The interface device 32 may also be implemented within a processing circuit.

[0028] Furthermore, various data generated by the control device 30 is appropriately recorded in the storage device 38. The various data is, for example, data output from a trained machine learning model, which will be described later. The various data generated by the control device 30 and appropriately recorded in the storage device 38 can be input again into the control device 30 via the interface device 32.

[0029] The control device 30 includes a processor as a processing circuit. The processor encompasses a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). Various processes of the control device 30 in the robot control device 4 according to this embodiment are realized by the processing circuit 34 executing various programs. The processing circuit 34 that executes the various processes may be realized by an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The processor may include an interface device 32, an image acquisition unit 35a, an estimation unit 35b, a posture calculation unit 35c, a memory 36, and a storage device 38.

[0030] As shown in FIG. 1A, the processing circuit of the control device 30 includes an image acquisition unit 35a and an estimation unit 35b. The processing circuit of the control device 30 may further include a posture calculation unit 35c. The image acquisition unit 35a acquires, for example, an image including a connector. The estimation unit 35b inputs the acquired image into a trained feature point estimation learning model that has undergone machine learning to estimate feature points of a connector in the image, and acquires estimation results that estimate each of the feature points of at least three connectors included in the image from the feature point estimation learning model. Furthermore, the posture calculation unit 35c estimates the posture of the connector based on the estimation results. The posture here is a three-dimensional posture.

[0031] Note that the estimation unit 35b and the separate image processing in the processing circuit may be configured such that the estimation unit 35b estimates feature points of (at least) one or two connectors included in the image using a trained feature point estimation learning model, and further, separate image processing in the processing circuit estimates feature points of (at least) two or one connectors, and these are passed to the posture calculation unit 35c as estimation results.

[0032] The memory 36 is a rewritable storage unit within the control device 30, and is configured, for example, by a random access memory (RAM) including a large number of semiconductor memory elements. The memory 36 temporarily stores specific computer programs, variable values, parameter values, etc., used when the processing circuit executes various processes. The memory 36 may also include a so-called read-only memory (ROM). The ROM pre-stores a computer program that realizes the processes of the control device 30, which will be described below. The processing circuit reads the computer program from the ROM and loads it into the RAM, enabling the processing circuit to execute the computer program.

[0033] The computer program that runs on the processing circuit of the control device 30 in the robot control device 4 according to this embodiment is written using a computer language such as Python. The computer language that writes the computer program that runs on the processing circuit of the control device 30 is not limited to these, and other computer languages ​​may of course be used.

[0034] Furthermore, in the control device 30 according to the present embodiment, a trained machine learning model for object detection and a trained feature point estimation learning model are constructed. As will be described later, the machine learning model for object detection according to the present embodiment is constructed using a network structure such as YOLO or SSD. Furthermore, as will be described later, the trained feature point estimation learning model according to the present embodiment is constructed using a network structure such as U-net. The machine learning model for object detection and the feature point estimation learning model are constructed using a computer language such as Python described above.

[0035] 2.2. Operation of the embodiment 2.2.1. Control Device Operation FIG. 2 is a flowchart of a calculation method for calculating the attitude and position of a connector in the control device 30 of the robot control device 4 according to the embodiment.

[0036] As described above, in the control device 30 of the robot control device 4 according to this embodiment, a trained machine learning model for object detection and a trained machine learning model for feature point estimation are constructed. The trained machine learning model for object detection is roughly trained using training data in which image data including images of one or more objects is used as an explanatory variable, and image data extracted from the one or more objects when a specific object is detected is used as a target variable.

[0037] In addition, the trained feature point estimation machine learning model is roughly trained using training data that uses image data including connectors as an explanatory variable, has the same number of output channels as the number of feature points in the connectors, and includes images of each feature point in each output channel, and uses image data as a target variable.

[0038] In other words, the trained feature point estimation machine learning model has been trained using training data in which image data including connectors is used as explanatory variable data and the positional information of each feature point of the connector in the image is used as objective variable data.

[0039] The objective variable data in the feature point estimation machine learning model can be said to be image data indicating the position of each feature point in the connector. As will be explained later, the objective variable data is, for example, image data indicating the probability of existence of each feature point in each pixel of the image.

[0040] The machine learning model for object detection and the machine learning model for feature point estimation will be explained later.

[0041] After starting (step S02), the processing circuit of the control device 30 in the robot control device 4 according to this embodiment prepares the above-mentioned trained machine learning model for object detection (step S04). Furthermore, the processing circuit of the control device 30 in the robot control device 4 according to this embodiment prepares the above-mentioned trained feature point estimation learning model (step S06).

[0042] Next, the processing circuit of the control device 30 checks whether image data including at least the connector captured by the imaging unit 24 exists (step S08), and if it exists (step S08: YES), the image acquisition unit 35a acquires the image data via the interface device 32 (step S10). If it does not exist (step S08: NO), the process ends (step S20).

[0043] Next, the estimation unit 35b performs object detection on the image data including at least the acquired connector (step S12). That is, the estimation unit 35b inputs the image data including at least the connector acquired by the image acquisition unit 35a to a machine learning model for trained object detection, and generates image data in which the connector is detected and cut out, which is output from the machine learning model (see FIGS. 3A and 3B).

[0044] That is, the estimation unit 35b generates a connector cutout image by cutting out the vicinity of the connector from an image including the connector. As will be described later, the connector cutout image is input to a trained feature point estimation learning model. Here, the estimation unit 35b generates the connector cutout image using a trained machine learning model for object detection that has been trained to detect specific objects in an image. As described above, the generation of the connector cutout image can also be achieved using an image processing recognition processing model, pattern matching, CAD matching, or the like.

[0045] FIG. 3B particularly shows an example of image data in which a connector has been detected and cut out.

[0046] Furthermore, if the image data acquired by the estimation unit 35b from the image acquisition unit 35a is guaranteed to include a sufficiently large image of only one connector, this object detection step (S12) may be omitted, and further, the step (S04) of preparing a trained machine learning model for object detection may also be omitted.

[0047] Next, the estimation unit 35b performs feature point extraction (feature point estimation) on the image data including the detected and cut-out connector (step S14). That is, the estimation unit 35b inputs the image data including the detected and cut-out connector to a trained feature point estimation machine learning model, and acquires image data output from the machine learning model, which has the same number of output channels as the number of feature points in the connector, and includes an image of each feature point in each output channel (see FIG. 5).

[0048] That is, the estimation unit 35b inputs image data including the detected and cut-out connector into a trained feature point estimation machine learning model to acquire position information of each feature point of the connector in the image. Here, the position information of each feature point of the connector in the image may be image data indicating the position of each feature point of the connector, or may be coordinate information.

[0049] In other words, the estimation unit 35b inputs an image including the acquired connector into a trained feature point estimation learning model that has undergone machine learning to estimate feature points of a connector in an image, and obtains an estimation result from the feature point estimation learning model that estimates the feature points (e.g., at least three feature points) of the connector included in the image.

[0050] Since the extraction of feature points is for deriving the three-dimensional posture of the connector, in this disclosure, the feature points are basically at least three points. Therefore, the trained feature point estimation learning model is basically trained using training data of at least three feature points of the connector. However, as mentioned above, if separate image processing for feature point extraction (estimation) is also used, the number of feature points in the training data may be smaller.

[0051] As will be described in detail later, in the robot control device 4 according to this embodiment, the number of feature points to be extracted is set to 8. In this case, the estimation unit 35b inputs an image including the acquired connector to a trained feature point estimation learning model that has undergone machine learning to estimate feature points of connectors in an image, and thereby acquires, from the trained feature point estimation learning model, an estimation result that estimates each of the feature points of at least 8 connectors included in the image.

[0052] By feature point extraction (feature point estimation), the three-dimensional pose of the connector can be derived.

[0053] Next, the orientation calculation unit 35c estimates the orientation and position of the connector, or at least the orientation, based on image data indicating the positions of the feature points of the connector (i.e., position information of the feature points) and dimensional data of the connector (step S16). In this embodiment, the orientation calculation unit 35c estimates the orientation and position (at least the orientation) as a PnP (Perspective n Point) problem.

[0054] The posture calculation unit 35c calculates coordinates for the operation of the robot 8 based on the estimated posture and position data of the connector (at least the posture data) (step S18). The coordinate data for the operation of the robot 8 is data for controlling the operation of the robot 8, and the operation controlled here is, for example, calculating the gripping position and gripping posture and causing the robot hand 10 at the tip of the arm of the robot 8 to grip the connector 14 (see FIG. 1B). The processing circuit of the control device 30 outputs the calculated coordinate data to the outside, for example, to the calculation unit 12 of the robot 8, via the interface device 32. Then, the process returns to step S08.

[0055] In addition, the processing circuit of the control device 30 may be configured to output the estimated connector posture and position data (at least the posture data) directly to the calculation unit 12 of the robot 8, and the calculation unit 12 of the robot 8, which receives the data, may calculate coordinates for the operation of the robot 8.

[0056] The above steps S02 to S20 are the processing operations of detecting a connector and estimating the attitude and position of the connector in the control device 30 of the robot control device 4 according to this embodiment.

[0057] 2.2.1.1. About machine learning models for object detection The following describes the machine learning model for object detection that is prepared in step S04 and used in step S12 in the flowchart of the calculation method for calculating the posture and position of a connector in the control device 30 of the robot control device 4 according to this embodiment, shown in FIG. 2.

[0058] A machine learning model for object detection is a model that uses image data containing images of one or more objects as explanatory variables (input) and image data of a specific object detected, for example, surrounded by a rectangle, as the objective variable (output).

[0059] 3A shows an example of the output of a trained machine learning model for object detection. When an image containing a dog, bicycle, truck, etc. is input, the machine learning model for object detection of the present disclosure outputs images of the specific objects, such as the dog, bicycle, and truck, surrounded by rectangles and tagged with "dog," "bicycle," and "truck," respectively, as shown in FIG.

[0060] 3B is also an example of the output of a trained machine learning model for object detection. When an image of a connector as shown in the present disclosure is input, the machine learning model for object detection of the present disclosure outputs an image of the connector surrounded by a rectangle and tagged with "connector."

[0061] A machine learning model for object detection can be constructed using a deep neural network such as YOLO (You Only Look Once) or SSD (Single Shot Detector).In addition, a machine learning model for object detection can be constructed not only using a deep neural network, but also using an image processing recognition model that extracts white areas using threshold processing in a situation where the background can be fixed as black and the object as white.

[0062] In the machine learning model for object detection in the control device 30 of the robot control device 4 according to this embodiment, YOLOv3 is used as a deep neural network. YOLOv3 is a neural network consisting of 75 layers of convolutions. In YOLOv3, the original image size is divided into grid cells at three scales as the output. Furthermore, each grid cell has three rectangular pieces of information. The rectangle information consists of the following three items. (1) Score of whether an object is present or not (2) Rectangle coordinates (3) Score for each label (representing the object class) In the learning stage of a machine learning model for object detection, these (1) to (3) are provided as training data. In addition, in the control device 30 of the robot control device 4 according to this embodiment, the class, i.e., the label, in (3) includes a connector. Because recognition is performed at multiple scales, i.e., three, YOLOv3 is able to achieve highly accurate recognition that is less affected by the size of the object.

[0063] As mentioned above, it is possible to detect connectors, which are objects, using conventional techniques that do not use machine learning models, such as pattern matching or CAD matching.

[0064] 2.2.1.2. About the trained machine learning model for feature point estimation Next, we will explain the feature point estimation machine learning model that is prepared in step S06 and used in step S14 of the flowchart of the calculation method for calculating the posture and position of a connector in the control device 30 of the robot control device 4 according to this embodiment, shown in Figure 2.

[0065] The feature point estimation machine learning model is a machine learning model that uses image data including connectors as explanatory variables (input), has the same number of output channels as the number of feature points in the connector, and uses image data containing images of each feature point in each output channel as the objective variable (output).

[0066] That is, the feature point estimation machine learning model is a machine learning model that uses image data including connectors as explanatory variable data and position information of each feature point of the connector in the image as objective variable data.

[0067] Figure 5 shows two examples of training data for a feature point estimation machine learning model. The top row shows one example, and the bottom row shows another. The explanatory variables are the image data of the connector. The image data of the connector is color image data, so although not shown in Figure 5, it becomes three-channel input data.

[0068] An example of connector image data is shown in the upper row (0) to (7). For ease of explanation, the same images are arranged in each of the upper rows (0) to (7). Another example of connector image data is shown in the lower row (0) to (7). For ease of explanation, the same images are also arranged in each of the lower rows (0) to (7).

[0069] In contrast to these, the objective variables in this embodiment are image data of feature points at the eight corners of the connector. Each feature point is expressed by a monochrome image, and since eight feature points are set, eight channels of output data are obtained, as shown in the upper and lower rows of Figure 5.

[0070] The image of feature points is a monochrome image in which pixel values ​​are scores ranging from 0 to 1, and the pixel value at corners (feature points) is "1." Therefore, eight feature points are represented by eight channels of image data. In this embodiment, the data of the objective variable in the feature point estimation machine learning model is image data indicating the probability of existence of each feature point at each pixel of the image. In other words, even in the training data, "corners (feature points)" have a small spread, and the points converted into a confidence map are used as the objective variable. Therefore, in Figures 5 and 6, the feature points, which are the objective variable, have a small magnitude and are output as, for example, heat map data. Note that estimated points are taught for invisible corners.

[0071] In this way, the data of the objective variable in the feature point estimation machine learning model is image data indicating the position of each feature point in the connector.

[0072] For example, in (0) at the left end of the top row of Figure 5, the input (explanatory variable) is the image data of the detected connector, and the output (objective variable) is assigned a black dot ((0)) of the feature point. The same is true for each of (1) to (7) in the top row. The black dots here are, for example, heat map data (the same applies below). Furthermore, for example, in (0) at the left end of the bottom row of Figure 5, the input (explanatory variable) is the image data of the detected connector, and the output (objective variable) is a black dot of the feature point ((0)). The same is true for each of (1) to (7) in the bottom row.

[0073] In this embodiment, the feature points are assigned the symbols (0) to (7) as follows: When the surface on which the guide 50 of the connector is provided is the top surface and the end surface of the connector is the facing front surface, the feature points are assigned the symbols (0), (1), (2), and (3) in order from the upper left corner clockwise on the end surface, as shown in Figures 4(A) and (B), and the feature points are assigned the symbols (4), (5), (6), and (7) in order from the upper left corner clockwise on the back surface.

[0074] When image data of a connector is input to a trained feature point estimation machine learning model, the point with the maximum pixel value in the image data of the feature points of each channel that is output becomes the feature point to be obtained as the output.

[0075] Figure 6 shows examples (three examples) of training data and estimation results for the feature point estimation machine learning model. In Fig. 6, the upper row (0) to (7), the middle row (0) to (7), and the lower row (0) to (7) show the same training data as in Fig. 5. In contrast, the upper, middle, and lower rows (a) in Fig. 6 are a compilation of the training data for each row. For ease of explanation, the training data for each feature point is marked with "0" to "7" and "◯." Furthermore, in Fig. 6, (b) in the upper, middle, and lower rows summarizes the feature points output when the image data of each connector is input to a trained feature point estimation machine learning model, i.e., it is an example of the estimation result. For ease of explanation, the estimation result of each feature point is marked with "0" to "7" and "◯".

[0076] Of course, the number of feature points may be more than eight.

[0077] A feature point estimation machine learning model can be constructed using a deep neural network such as an FCN (Fully Convolution Network). Unlike a CNN (Convolution Neural Network), a Fully Convolutional Neural Network (FCN) does not have a fully connected layer, and the network is composed only of convolutional layers. Segmentation is achieved using an FCN. Furthermore, the feature point estimation machine learning model is not limited to FCN, and may be a simple autoencoder, pix2pix, or other model. In other words, any model that inputs a three-channel color image and outputs an image (of feature points) with the same number of channels as the number of feature points is sufficient.

[0078] In the feature point estimation machine learning model in the control device 30 of the robot control device 4 according to this embodiment, U-net is used as a fully convolutional neural network (FCN). FIG. 8 is a diagram showing an example of the network structure of U-net. As shown in FIG. 8, the network structure of U-net is simple, repeating convolution → reduction, and then repeating convolution → enlargement again (repeat) to return to the original image size. One of its features is that highly accurate recognition is possible by using feature information before reduction together with features after enlargement.

[0079] 9 is a diagram schematically illustrating a method using a machine learning model to extract feature points of a connector from image data captured by the imaging unit 24 of the robot control device 4 according to this embodiment. As described above, the example of the network structure in this embodiment uses, as the machine learning model, YOLOv3 as a deep neural network and U-net as a fully convolutional neural network.

[0080] 2.2.1.3. Calculating Attitude and Position as a PnP Problem Next, we will explain the calculation as a PnP (Perspective n Point) problem in step S16 of the flowchart of the calculation method for calculating the posture and position of the connector in the control device 30 of the robot control device 4 according to this embodiment shown in Figure 2, which is set and used in estimating the posture and position (at least the posture) of the connector.

[0081] Based on the estimated and extracted at least eight feature points, the connector's pose and position are calculated as a PnP problem. The calculation method for the PnP problem is a method that directly determines the position and orientation of a connector (object) as viewed from the center of the lens of the camera (image capture unit) from the coordinates in the image of the connector (object) and its actual three-dimensional dimensions. In other words, the PnP problem refers to a problem setting that determines the position and orientation of the viewpoint from the relationship between a point cloud in three-dimensional space and a point cloud projected onto a two-dimensional plane. A method called EPnP can be cited as a highly accurate solution to the PnP problem.

[0082] In addition, position and orientation estimation using a monocular camera in the PnP solution requires at least three feature points.

[0083] FIG. 7 shows an example image of a connector along with the display of the attitude and position data calculated (estimated) by the calculation method as a PnP problem.

[0084] 2.3. Summary of embodiments The control device 30 according to this embodiment includes an image acquisition unit 35a that acquires an image including the connector 14, and an estimation unit 35b that inputs the acquired image into a trained feature point estimation learning model that has undergone machine learning to estimate feature points of the connector 14 in the image, and acquires an estimation result that estimates feature points (e.g., at least three feature points) of the connector included in the image from the feature point estimation learning model.

[0085] The control device in the robot control device described above can recognize the orientation and position of the connector in real time and provide the robot with data for handling the connector.

[0086] 3. [Other embodiments] As described above, the embodiments have been described as examples of the technology disclosed in the present application. However, the technology in the present disclosure is not limited to these, and can be applied to embodiments in which appropriate changes, substitutions, additions, omissions, etc. are made.

[0087] In the above-described embodiment, a connector holding system has been described, but application of the technology in the present disclosure is not limited to connector holding systems. For example, the technology in the present disclosure may also be applied to a system that performs connector connection work.

[0088] As described in the above embodiment, the machine learning model for object detection can be constructed not only by YOLOv3 but also by other deep neural networks, such as SSD (Single Shot Detector). The machine learning model for object detection can be constructed not only by deep neural networks but also by image processing-based recognition processing models that extract white areas using threshold processing in situations where the background can be fixed as black and the object as white. The machine learning model for feature point estimation can also be constructed not only by FCN but also by models such as simple autoencoders and pix2pix.

[0089] Furthermore, technology for detecting connectors, which are objects, can also be realized using conventional technology such as pattern matching and CAD matching.

[0090] In the robot control device 4, the imaging unit 24, the control device 30, and the storage device 38 may be configured as an integrated unit, or may be configured as separate units. In addition, the control device 30 may have both a function (estimation function) to estimate (calculate) the posture and position of a connector included in an image and a function (learning function) to learn a machine learning model for estimating the posture and position of the connector, or it may have only an estimation function without a learning function, or it may have only a learning function.

[0091] <Calculating posture and position using stereo (alternative method for PnP problem)> Furthermore, the method of calculating the orientation and position of the connector in the control device 30 of the robot control device 4 in step S16 of FIG. 2 can also use a stereo method instead of the calculation as the PnP problem described above. When the stereo method is used, the imaging unit 24 uses a stereo camera or the like that can capture images of the connector from different viewpoints. The imaging unit 24 acquires a first image and a second image of the connector (step S10). The control device 30 of the robot control device 4 performs object detection of the connector in each of the first image and the second image (step S12) and extracts each feature point of the connector in each image (step S14). By matching each feature point extracted in each image using the stereo method, it is possible to estimate (calculate) the coordinates (three-dimensional position) of each feature point of the connector and also estimate (calculate) its posture.

[0092] The accompanying drawings and detailed description are provided to explain the embodiments. Therefore, the components described in the accompanying drawings and detailed description may include not only components essential for solving the problem, but also components that are not essential for solving the problem in order to illustrate the above technology. Therefore, the fact that these non-essential components are described in the accompanying drawings or detailed description should not be interpreted as immediately indicating that these non-essential components are essential.

[0093] Furthermore, since the above-described embodiments are intended to illustrate the technology of the present disclosure, various modifications, substitutions, additions, omissions, etc. may be made within the scope of the claims or their equivalents. [Explanation of symbols]

[0094] 2 Connector holding system, 4 Robot control device, 8 Robot, 10 Robot hand, 12 Calculation unit, 14 Connector, 16 Cable, 20 Cable, 24 Imaging unit, 30 Control device, 32 Interface device, 36 Memory, 38 Storage device, 50 Guide.

Claims

1. an image acquisition unit that acquires an image including the connector; an estimation unit that inputs the acquired image into a trained feature point estimation learning model that has undergone machine learning to estimate feature points of a connector arbitrarily set for the connector in the image, and acquires an estimation result of the feature points of the connector included in the acquired image from the feature point estimation learning model; A control device comprising:

2. the estimation unit estimates feature points of at least three connectors included in the acquired image from the feature point estimation learning model; The control device according to claim 1 .

3. and a posture calculation unit that estimates a posture of the connector based on the estimation result. The control device according to claim 1 or 2.

4. The feature point estimation learning model is a machine learning model that is trained using training data of at least three feature points of a connector arbitrarily set for the connector in the image. The control device according to any one of claims 1 to 3.

5. The feature point estimation learning model is a machine learning model for trained feature point estimation that has been trained using training data, in which image data of an image including a connector is used as explanatory variable data, and position information of each feature point of the connector arbitrarily set for the connector in the image is used as objective variable data. The control device according to any one of claims 1 to 4.

6. The data of the objective variable is image data indicating the positions of the feature points of the connector arbitrarily set for the connector in the image. The control device according to claim 5 .

7. The data of the objective variable is image data indicating the probability of existence of each of the feature points in each pixel of the image. The control device according to claim 5 .

8. the feature point estimation learning model is a machine learning model that is trained using training data of at least eight feature points of a connector arbitrarily set for the connector in the image, the estimation unit inputs the acquired image into the feature point estimation learning model, and acquires, from the feature point estimation learning model, an estimation result of estimating each of the feature points of at least eight connectors included in the acquired image. The control device according to any one of claims 1 to 3.

9. the estimation unit generates a connector cutout image by cutting out the vicinity of the connector from the image including the connector, and inputs the connector cutout image to the feature point estimation learning model. The control device according to any one of claims 1 to 8.

10. The estimation unit generates the connector cutout image using a machine learning model for object detection that has been trained to detect a specific object in an image. The control device according to claim 9.

11. acquiring an image including a connector; a step of inputting the acquired image into a trained feature point estimation learning model that has undergone machine learning to estimate feature points of a connector arbitrarily set for the connector in the image, and acquiring, from the feature point estimation learning model, an estimation result of the feature points of the connector included in the acquired image; A connector feature point estimation method, including:

12. a memory storing a feature point estimation learning model for estimating feature points of a connector arbitrarily set for the connector in the image; a processing circuit that performs machine learning to convert the feature point estimation learning model into a trained model; A learning device comprising:

13. A computer device in which a trained feature point estimation learning model is constructed by machine learning for estimating feature points of a connector arbitrarily set for a connector in an image, obtaining image data for an image including a connector; inputting the acquired image into the feature point estimation learning model, and executing a process of acquiring, from the feature point estimation learning model, an estimation result of the feature points of connectors included in the acquired image; Computer program.

14. A trained feature point estimation learning model program that has undergone machine learning to estimate feature points of a connector arbitrarily set for a connector in an image, the feature point estimation learning model program causes a computer to execute a process of receiving an image including a connector and outputting an estimation result of estimating feature points of the connector included in the image; Computer program.

15. A control device according to claim 3 and an imaging unit, The control device acquiring image data including the connector captured by the imaging unit from the imaging unit; Calculating coordinates for the operation of an external robot based on the estimated connector posture and position data; transmitting the calculated coordinates to the external robot; Robot control device.

16. A control device according to claim 3, an imaging unit, and an articulated robot, the control device acquires image data including the connector captured by the imaging unit from the imaging unit, the robot includes a computing unit and a robot hand that holds a connector; The calculation unit controls the operation of the robot hand based on the posture data of the connector estimated by the control device. system.

Images