Robot control system and control method

Abstract

To provide a robot control system and a control method that are able to determine an appropriate position of an object whose shape changes with a lapse of time.SOLUTION: A robot control system 1 including a robot 10 that executes a task on an object S whose shape changes with a lapse of time, comprises: input means to which two-dimensional image data of the object S is inputted; reception means that receives a command for instructing the robot 10 to perform the task; acquisition means that acquires three-dimensional-coordinate data of the object S according to a reception result of the reception means; and mapping means that, in response to acquisition of the three-dimensional-coordinate data by the acquisition means, maps the three-dimensional-coordinate data acquired by the acquisition means onto the two-dimensional image data inputted through the input means.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a robot control system and a control method.

Background Art

[0002] Conventionally, industrial robots have been used to perform operations on objects. In Patent Document 1, identifying the position information of a specific object in an image captured by a camera, changing the position and orientation of a laser distance sensor based on the identified position information, and causing the laser distance sensor to measure this specific object after the position and orientation have been changed are disclosed. With the technology described in Patent Document 1, it is possible to reduce the large time cost and computational cost required for excessive measurement including objects other than a specific object.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In Patent Document 1, although there is a suggestion about applying the technology described in Patent Document 1 to a robot, the specific aspects thereof are not disclosed. Further, in Patent Document 1, it is not assumed that the shape of an object that is the work target changes over time.

[0005] The present disclosure has been made in view of the above-described circumstances, and an object thereof is to provide a robot control system and a control method capable of specifying an appropriate position for an object whose shape changes over time.

Means for Solving the Problems

[0006] <1>The robot control system according to Aspect 1 of the present disclosure is a robot control system including a robot that performs work on an object whose shape changes over time, and includes input means for inputting two-dimensional image data of the object, reception means for receiving a command for instructing the robot to perform the work, acquisition means for acquiring the three-dimensional coordinate data of the object according to the reception result of the reception means, and mapping means for mapping the three-dimensional coordinate data acquired by the acquisition means to the two-dimensional image data input by the input means according to the acquisition of the three-dimensional coordinate data by the acquisition means. It is characterized by comprising

Effect of the Invention

[0007] According to the present disclosure, it is possible to provide a robot control system and a control method capable of specifying an appropriate position for an object whose shape changes over time.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Modes for Carrying Out the Invention

[0009] Hereinafter, with reference to the drawings, a robot control system and a control method of the robot control system according to an embodiment of the present disclosure will be described. The robot control system according to the present embodiment includes a robot that is a control target. The robot according to the present embodiment executes, for example, work on an object attached to a work target. More specifically, the robot according to the present embodiment executes, for example, work for removing deposits in a melting furnace that is a work target. That is, in the present embodiment, the work target is a forehearth provided in the melting furnace. The object is slag, molten waste, etc. attached to the inside of the forehearth that is the work target. Note that the robot control according to the present embodiment is not limited to the above example and may be used in any other place.

[0010] The robot according to the present embodiment performs operations such as pushing and stirring an object, for example, using an end effector attached to the robot. Hereinafter, the operation of the end effector attached to the robot pushing an object is referred to as a pushing operation. The operation of the end effector attached to the robot stirring an object is referred to as a stirring operation. In the present embodiment, the object has a characteristic that its shape changes over time. In order to cope with such a change in the shape of the object, the work by the robot according to the present embodiment is preferably an irregular work. In the present embodiment, the robot is controlled to be operable, for example, by a user's operation. Alternatively, the robot may be controlled to perform work semi-automatically by a robot control system. Hereinafter, each component included in the robot control system according to the present embodiment will be described.

[0011] (Overview of the robot control system) FIG. 1 is a schematic diagram of a robot control system 1 according to an embodiment. As shown in FIG. 1, the robot control system 1 includes a robot 10, an end effector 20, imaging means 30, scanning means 40, and control means 50.

[0012] Robot 10 is placed in front of the work target T. In the present embodiment, for example, a known 6-axis vertical articulated robot is preferably used as the robot 10. As shown in FIG. 1, an end effector 20 is attached to the arm 11 of the robot 10, which is a 6-axis vertical articulated robot. In the present embodiment, the robot 10 operates so as to move the position of the end effector 20 and change the posture of the end effector 20.

[0013] Hereinafter, in the present embodiment, the position of the end effector 20 attached to the robot 10 is referred to as the position of the robot 10. The movement of the position of the end effector 20 by the operation of the robot 10 is referred to as the movement of the robot 10. The posture of the end effector 20 attached to the robot 10 is referred to as the posture of the robot 10. The change in the posture of the end effector 20 by the operation of the robot 10 is referred to as the change in the posture of the robot 10.

[0014] The end effector 20 is attached to the arm 11 of the robot 10. The end effector 20 comes into contact with the object S when performing work on the object S. In the present embodiment, the end effector 20 is a rod-shaped member.

[0015] The imaging means 30 images the object S located on the work target T and the periphery of the object S. By this, the imaging means 30 outputs, as two-dimensional image data visible by being displayed on a screen or the like, the shape of the object S and the situation around the object S. The output of the two-dimensional image data by the imaging means 30 is performed, for example, periodically at regular intervals. The two-dimensional image data output by the imaging means 30 is input to the input means 51A (described later) of the control means 50. For the imaging means 30, for example, a known ITV camera, a CCD camera, or the like is preferably used. In the vicinity of the work target T, the position and orientation of the imaging means 30 are preferably arranged at a fixed location, for example, during the operation of the robot control system 1. This enables the user of the robot control system 1 to always visually observe two-dimensional image data from a fixed viewpoint and also facilitates the mapping of three-dimensional coordinate data onto the two-dimensional image data. The mapping will be described later.

[0016] The scanning means 40 scans the object S located on the work target T and the shape of its surroundings. As a result, the scanning means 40 outputs the shape of the object S and the situation around the object S as three-dimensional coordinate data. The output of the three-dimensional coordinate data by the scanning means 40 is performed, for example, when the acquisition means 51B (described later) of the control means 50 determines to acquire the three-dimensional coordinate data. The three-dimensional coordinate data output by the scanning means 40 is acquired by the acquisition means 51B. For the scanning means 40, for example, a known 3D scanner or the like is preferably used. The scanning means 40 is inserted into the work target T, for example, when scanning the shape of the object S and its surroundings. This preferably enables more accurate three-dimensional coordinate data to be output. When the scanning means 40 does not scan the shape of the object S and its surroundings, it is preferably located outside the work target T. This preferably suppresses the scanning means 40 from being affected by the temperature of the work target T.

[0017] The control means 50 controls the operations of the robot 10, the imaging means 30, and the scanning means 40 respectively. Thus, the robot control system 1 according to the present embodiment operates. Further, the control means 50 performs processing to make it easier for the user to grasp the situation of the object S and its surroundings by associating (mapping) the two-dimensional image data output by the imaging means 30 with the three-dimensional coordinate data output by the scanning means 40. Hereinafter, the configuration of the control means 50 and the details of the mapping process by the control means 50 will be described.

[0018] FIG. 2 is a diagram showing an example of the hardware configuration of the control means 50 according to the embodiment. The control means 50 includes a control unit 51 including a processor 51a such as a CPU (Central Processing Unit) and a memory 51b connected by a bus, and executes a robot control program. The robot control program is a program for controlling the operations of the respective functional units included in the control means 50. The control means 50 functions as a device including the control unit 51, the user interface 52, and the storage unit 53 by executing the robot control program.

[0019] More specifically, the control means 50 reads out the robot control program stored in the storage unit 53 by the processor 51a, and stores the read robot control program in the memory 51b. By the processor 51a executing the robot control program stored in the memory 51b, the control means 50 functions as a device including the user interface 52, the control unit 51, and the storage unit 53.

[0020] The control unit 51 controls the operations of the respective functional units included in the control means 50. The control unit 51 controls, for example, the input / output in the user interface 52. The control unit 51 processes various kinds of information used for controlling each of the robot 10, the imaging means 30, and the scanning means 40, for example. Further, the control unit 51 performs processing for mapping the two-dimensional image data output by the imaging means 30 and the three-dimensional coordinate data output by the scanning means 40, and processing of various kinds of information associated with the mapping.

[0021] The user interface 52 includes a display unit 52a and an input unit 52b. The display unit 52a displays various kinds of information. The display unit 52a includes an output device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display, for example. The display unit 52a may be configured as an interface for connecting these output devices to the control means 50.

[0022] The input unit 52b is configured to include input terminals such as a mouse, a keyboard, and a touch panel. The input unit 52b may be configured as an interface that connects these input terminals to the control means 50. The input unit 52b receives the input of various information to the control means 50. The various information is information indicating how the input terminal has been operated (hereinafter referred to as "input terminal operation information"). Note that the display unit 52a and the input unit 52b may be configured as an integrated touch panel.

[0023] The input terminal operation information is, for example, information input by the user to instruct the robot 10 to perform work on the object S. That is, the user inputs an instruction for operating the robot 10 via the input unit 52b. The user appropriately clicks or taps, for example, the portion of the two-dimensional image data displayed on the display unit 52a where the object S is shown. By this, the target position of the robot 10 is determined. That is, the robot 10 moves the tip of the end effector 20 to the portion corresponding to the place set as the target position in the two-dimensional image data of the object S according to the thus set target position. The user inputs, for example, the content for instructing the posture of the robot 10 via the input unit 52b. The posture of the robot 10 is input, for example, by appropriately clicking or tapping a selection button or the like displayed on the display unit 52a. Alternatively, the posture of the robot 10 may be appropriately input via the input unit 52b. By this, the target posture of the robot 10 is determined. That is, the robot 10 changes the posture of the end effector 20 according to the thus set target posture. The user inputs details of the operation by the end effector 20, for example, in the part where the end effector 20 has moved. The details of the operation by the end effector 20 are input, for example, by appropriately clicking or tapping selection buttons or the like displayed on the display unit 52a. Alternatively, the details of the operation by the end effector 20 may be appropriately input via the input unit 52b. In this way, the work content to be performed by the robot 10 is determined. That is, the robot 10 performs work on the object S in accordance with the work content thus set.

[0024] The information input by the user as described above is converted by the control unit 51 into commands for operating the robot 10. When the commands thus converted are input to the robot 10, the robot 10 operates. Hereinafter, the user's input of an instruction for operating the robot 10 may be referred to as inputting a command. The command includes, for example, target position and orientation data and work content data. That is, the information input by the user as described above is classified by the control unit 51 into target position and orientation data and work content data.

[0025] The target position and orientation data is information indicating the target position and target orientation of the robot 10. The target position and target orientation of the robot 10 are specified by the user by the above-described method in order to set the relative position of the end effector 20 with respect to the object S when performing work on the object S. In this way, the input terminal operation information is, for example, information about the target position and target orientation of the robot 10 input by the user via the input unit 52b.

[0026] The work content data is information indicating the content of the work. The work content data includes type information and parameters such as intensity information and speed information. The type information is information indicating the type of operation of the robot 10 when the work on the object S is executed. The types of operations of the robot 10 include, for example, the pushing operation and the stirring operation described above. The intensity information is information indicating the intensity when performing the pushing operation and the stirring operation described above. The intensity when performing the pushing operation and the stirring operation is the strength of the force that brings the tip of the end effector 20 into contact with the object S when performing these operations. The intensity when performing these operations is preferably set stepwise, for example. The speed information is information indicating the speed when performing the pushing operation and the stirring operation described above. The speed when performing the pushing operation and the stirring operation is the speed at which the tip of the end effector 20 is moved when performing these operations. The speed when performing these operations is preferably set stepwise, for example. As described above, the input terminal operation information is, for example, information about whether the operation performed by the robot 10 that the user inputs via the input unit 52b is a pushing operation or a stirring operation, and the intensity and speed when performing the pushing operation and the stirring operation.

[0027] The storage unit 53 is configured using a non-temporarily computer-readable storage medium device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 53 stores, for example, various information related to the control means 50. The storage unit 53 stores, for example, a robot control program in advance. Alternatively, the storage unit 53 may store a learning data set described later.

[0028] In the present embodiment, the control means 50 operates as a device having each functional configuration described below in addition to the control of each configuration described above by executing the robot control program stored in the storage unit 53 by the control unit 51.

[0029] (Regarding the mapping process) Next, the details of the mapping between the two-dimensional image data and the three-dimensional coordinate data according to the present embodiment will be described. FIG. 3 is an example of two-dimensional image data output by the imaging means 30. FIG. 4 is an example of three-dimensional coordinate data output by the acquisition means 51B. FIG. 5 is an example showing the result of mapping by the mapping means 51C. Here, when the robot 10 performs an operation on the object S, it is necessary to accurately indicate the coordinates of the work target T and the object S. At this time, it is difficult for the user to accurately grasp the coordinates of the work target T and the object S located in the three-dimensional space only by two-dimensional image data. Also, even with only three-dimensional coordinate data, when the shape of the object S changes, it is difficult to accurately grasp the coordinates of the work target T and the object S by matching with a generally used model or the like. Further, with only three-dimensional coordinate data, it is difficult for the user to accurately grasp the coordinates of the work target T and the object S from the viewpoints of image quality, viewing angle, etc. Therefore, in the robot control system 1 according to the present embodiment, mapping processing is performed. In the present embodiment, mapping means associating two-dimensional image data and three-dimensional coordinate data with each other. That is, mapping means, for example, combining two-dimensional image data as shown in FIG. 3 and three-dimensional coordinate data as shown in FIG. 4, and it is also possible to generate an image that can be intuitively understood by the user by superimposing the three-dimensional coordinate data on the two-dimensional image data as shown in FIG. 5. By this, the robot 10 can perform an operation on the position in the three-dimensional coordinate data corresponding to the position specified by the user on the two-dimensional image data. Also, when the user performs an operation using the robot control system 1, it contributes to making it easier for the user to grasp the shape, situation, etc. of the work target T and the object S located on the work target T. In the example shown by FIGS. 3, 4, and 5, a rectangular parallelepiped object S placed on a table is used.

[0030] FIG. 6 is a block diagram showing an example of the functional configuration of the control unit 51. In the present embodiment, among the functional configurations provided in the control unit 51, those that contribute to mapping can be classified into a configuration for performing mapping processing, a configuration for determining whether to acquire three-dimensional coordinate data for performing mapping processing, a configuration for making it easier for the user to determine whether to start an operation by the robot 10, and a configuration for making it easier for the user to perform an operation by the robot 10 more in accordance with the intention. Hereinafter, the details of each of the above-described configurations will be described.

[0031] (Configuration for performing mapping processing) As a configuration for performing mapping processing between two-dimensional image data and three-dimensional coordinate data, the control unit 51 includes an input means 51A, an acquisition means 51B, and a mapping means 51C. The input means 51A receives two-dimensional image data of the object S. That is, the input means 51A receives the two-dimensional image data output by the imaging means 30 after imaging the object S located on the work target T and the periphery of the object S. The acquisition means 51B acquires three-dimensional coordinate data of the object S. That is, the acquisition means 51B acquires the three-dimensional coordinate data output by the scanning means 40 after scanning the object S located on the work target T and the periphery of the object S.

[0032] The mapping means 51C maps the three-dimensional coordinate data acquired by the acquisition means 51B to the two-dimensional image data input by the input means 51A in response to the acquisition of the three-dimensional coordinate data by the acquisition means 51B. The mapping means 51C maps, for example, the three-dimensional coordinate data to the two-dimensional image data as luminance information. The mapping means 51C, for example, based on the three-dimensional coordinate data, superimposes and displays only the coordinate acquisition area including the work target T and the object S on the two-dimensional image data captured by the imaging means 30. By doing so, when the user views the image displayed on the display unit 52a, the accuracy of the mapping result and the position of the object S in which part of the work target T can be easily grasped. The mapping means 51C, for example, based on the three-dimensional coordinate data, may display the part set as the target position of the robot 10 by the user in the two-dimensional image data captured by the imaging means 30 brighter than the other parts. By doing so, when the user views the image displayed on the display unit 52a, the target position of the robot 10 may be easily grasped.

[0033] (Configuration for determining whether to acquire three-dimensional coordinate data) Here, the acquisition of three-dimensional coordinate data by the acquisition means 51B and the mapping process by the mapping means 51C have a relatively large processing load on the control unit 51 and require time for processing. At this time, for example, when acquiring three-dimensional coordinate data and mapping it to two-dimensional image data for the shape of the object S that changes over time, the processing load of the system increases. More specifically, for example, if the three-dimensional coordinate data is updated while the mapping process by the mapping means 51C is in progress, a time difference until the mapping process is completed causes a delay in the mapping process.

[0034] In order to suppress such a delay in the mapping process, in the present embodiment, the control unit 51 is provided with the following respective configurations to enable the acquisition means 51B to determine whether to newly acquire three-dimensional coordinate data. Hereinafter, the details of each configuration for determining whether to acquire three-dimensional coordinate data for performing the mapping process will be described. That is, as a configuration for determining whether to acquire three-dimensional coordinate data for performing the mapping process, the control unit 51 includes a reception means 51D, a detection means 51E, a first timing means 51F, a second timing means 51G, a determination means 51H, and a discrimination means 51I.

[0035] The reception means 51D receives a command for instructing the robot 10 to perform work. Further, the reception means 51D receives, for example, the result of whether the user has input a command via the input unit 52b. In the present embodiment, the result received by the reception means 51D is referred to as the reception result. The acquisition means 51B acquires the three-dimensional coordinate data of the object S and its surroundings according to the reception result of the reception means 51D. That is, for example, when the reception means 51D outputs a reception result indicating that the user has input a command, the acquisition means 51B newly acquires three-dimensional coordinate data. By this, the three-dimensional coordinate data can be updated according to the input of a command for the user to instruct the robot 10 to perform work. Therefore, the three-dimensional coordinate data acquired immediately before the robot 10 starts working can be mapped to the two-dimensional image data. Therefore, the result of the mapping can be made more realistic.

[0036] The detection means 51E detects that the shape of the object S has changed since the acquisition means 51B acquired the three-dimensional coordinate data. In the present embodiment, it is preferable that the detection means 51E detects whether or not the shape of the object S has changed by using, for example, the two-dimensional image data output by the imaging means 30. That is, it is preferable that the detection means 51E detects a change in the shape of the object S by comparing the two-dimensional image data output a plurality of times as time elapses. In the present embodiment, the result of whether or not the detection means 51E has detected a change in the shape of the object S is referred to as a detection result. The acquisition means 51B may acquire the three-dimensional coordinate data of the object S according to the detection result of the detection means 51E. That is, the acquisition means 51B may determine whether or not to acquire the three-dimensional coordinate data of the object S based on whether or not the shape of the object S has changed.

[0037] Hereinafter, a specific example of acquisition of three-dimensional coordinate data based on the detection result will be described. That is, for example, when the detection means 51E outputs a detection result indicating that the shape of the object S has changed, the acquisition means 51B newly acquires three-dimensional coordinate data even if no command has been input by the user. By this, the three-dimensional coordinate data can be appropriately updated according to the change in the shape of the object S. Therefore, it is easy to make the three-dimensional coordinate data conform to the latest situation of the object S. Therefore, the result of the mapping can be made more surely realistic. Alternatively, when the detection means 51E outputs a detection result indicating that the shape of the object S has not changed, for example, the acquisition means 51B may not acquire new three-dimensional coordinate data even when the reception means 51D outputs a reception result indicating that the user has input a new command. By doing so, it is possible to suppress the acquisition of three-dimensional coordinate data more than necessary. Therefore, it is possible to more reliably suppress the occurrence of a delay in the mapping process.

[0038] The first timing means 51F measures the time elapsed since the reception means 51D received a command. The first timing means 51F preferably measures the time using, for example, the function of the processor 51a provided in the control means 50. In the present embodiment, the time measured by the first timing means 51F is referred to as the measurement result. The acquisition means 51B may acquire the three-dimensional coordinate data of the object S according to the measurement result of the first timing means 51F. That is, the acquisition means 51B may determine whether to acquire the three-dimensional coordinate data of the object S based on whether a predetermined time has elapsed since the reception means 51D received a command. The predetermined time may be arbitrarily determined by the user, for example.

[0039] Hereinafter, a specific example of acquiring three-dimensional coordinate data based on the measurement result will be described. That is, for example, when the measurement result by the first timing means 51F has elapsed a preset predetermined time, the acquisition means 51B acquires new three-dimensional coordinate data even if no command has been input by the user. By doing so, the three-dimensional coordinate data can be updated appropriately according to the passage of time. Therefore, it is easy to make the three-dimensional coordinate data conform to the latest situation of the object S. Therefore, the result of mapping can be made more reliably conform to reality. Alternatively, when the measurement result by the first measurement means 51F has not elapsed a preset predetermined time, for example, the acquisition means 51B may not acquire new three-dimensional coordinate data even when the reception means 51D outputs a reception result indicating that the user has newly input a command. By doing so, it is possible to suppress the acquisition of three-dimensional coordinate data more than necessary. Therefore, it is possible to more reliably suppress the occurrence of a delay in the mapping process.

[0040] The second measurement means 51G measures the time elapsed since the mapping means 51C executed the mapping process. The second measurement means 51G preferably measures time using, for example, the function of the processor 51a included in the control means 50. In the present embodiment, the time measured by the second measurement means 51G is referred to as the elapsed time. The acquisition means 51B may acquire the three-dimensional coordinate data of the object S in accordance with the elapsed time measured by the second measurement means 51G. That is, the acquisition means 51B may determine whether to acquire the three-dimensional coordinate data of the object S based on whether a predetermined time has elapsed since the mapping means 51C executed the mapping process. The predetermined time is preferably determined based on the time required for one mapping process, which depends on the processing ability of the processor 51a included in the control means 50, for example.

[0041] Hereinafter, a specific example of acquiring three-dimensional coordinate data based on the elapsed time will be described. That is, for example, when the elapsed time measured by the second measurement means 51G has not elapsed a preset predetermined time, the acquisition means 51B does not acquire new three-dimensional coordinate data even when the reception means 51D outputs a reception result indicating that the user has newly input a command. By doing so, it is possible to suppress the acquisition of new three-dimensional coordinate data while the mapping means 51C is performing the mapping process. Therefore, it is possible to more reliably suppress the occurrence of a delay in the mapping process. Alternatively, when the elapsed time measured by the second timing means 51G has passed a predetermined time set in advance, the acquisition means 51B may acquire new three-dimensional coordinate data even if no command is input by the user. This can suppress the occurrence of a time when the mapping process by the mapping means 51C is not performed. Therefore, it is easier to make the three-dimensional coordinate data conform to the latest situation of the object S. Therefore, the result of mapping can be made more surely conform to reality.

[0042] The elapsed time measured by the second timing means 51G may be used as a criterion for determining whether or not to perform the mapping process by the mapping means 51C. That is, the mapping means 51C may map the three-dimensional coordinate data acquired by the acquisition means 51B to the two-dimensional image data input by the input means 51A according to the elapsed time measured by the second timing means 51G. More specifically, for example, even when the acquisition means 51B has acquired three-dimensional coordinate data, if the elapsed time measured by the second timing means 51G has not passed a predetermined time set in advance, the mapping means 51C may not perform the mapping process using the acquired three-dimensional coordinate data. This can more surely suppress the occurrence of a delay in the mapping process.

[0043] The determination means 51H determines whether or not the mapping means 51C is in the middle of the mapping process. The determination means 51H preferably determines whether or not it is in the middle of the mapping process by monitoring the processing status of the mapping means 51C, for example. In the present embodiment, the result determined by the determination means 51H is referred to as a determination result. The acquisition means 51B may acquire the three-dimensional coordinate data of the object S according to the determination result of the determination means 51H. That is, the acquisition means 51B may determine whether or not to acquire the three-dimensional coordinate data of the object S based on whether or not the mapping means 51C is in the middle of the mapping process.

[0044] Hereinafter, a specific example of obtaining three-dimensional coordinate data based on the determination result will be described. That is, for example, when the determination means 51H outputs a determination result indicating that the mapping means 51C is in the middle of the mapping process, the acquisition means 51B does not acquire new three-dimensional coordinate data even if the reception means 51D outputs a reception result indicating that the user has newly input a command. By this, it is possible to suppress the acquisition of new three-dimensional coordinate data during the mapping process by the mapping means 51C. Therefore, it is possible to more reliably suppress the occurrence of a delay in the mapping process. Alternatively, for example, when the determination means 51H outputs a determination result indicating that the mapping means 51C is not in the middle of the mapping process, the acquisition means 51B may acquire new three-dimensional coordinate data even if no command is input by the user. By this, it is possible to suppress the occurrence of a time when the mapping process by the mapping means 51C is not performed. Therefore, it is possible to more easily make the three-dimensional coordinate data conform to the latest situation of the object S. Therefore, it is possible to more reliably make the mapping result conform to reality.

[0045] The discrimination means 51I discriminates the type information included in the command received by the reception means 51D. That is, the discrimination means 51I discriminates whether the instruction for the operation of the robot 10 input by the user via the input unit 52b is an instruction to perform a pushing operation or an instruction to perform a stirring operation. In the present embodiment, the result of the discrimination means 51I discriminating the type information of the command is referred to as the discrimination result. The acquisition means 51B may acquire the three-dimensional coordinate data of the object S according to the discrimination result of the discrimination means 51I. That is, the acquisition means 51B may determine whether to acquire the three-dimensional coordinate data of the object S based on whether the command input by the user is an instruction to perform a pushing operation or an instruction to perform a stirring operation.

[0046] Hereinafter, a specific example of obtaining three-dimensional coordinate data based on the discrimination result will be described. Here, whether to acquire the three-dimensional coordinate data by the acquisition means 51B is determined based on each of the information output by the above-described means. Thus, even when the reception means 51D outputs a reception result indicating that a new command has been input, the acquisition means 51B may not acquire new three-dimensional coordinate data. In such a case, for example, when the user continuously inputs a command for only one of the pushing operation or the stirring operation, new three-dimensional coordinate data is not acquired. On the other hand, when the command input by the user changes from one of the pushing operation or the stirring operation to the other, the acquisition means 51B preferentially acquires new three-dimensional coordinate data based on the determination based on the information output by the above-described means.

[0047] The above content will be described by taking the first timing means 51F as an example. That is, for example, when a pushing operation command is input by the user and a new pushing operation command is input before the elapsed time of the timing result by the first timing means 51F reaches a predetermined time, the acquisition means 51B does not acquire new three-dimensional coordinate data. On the other hand, in the same state, when a stirring operation command is newly input, the acquisition means 51B acquires new three-dimensional coordinate data.

[0048] Here, the shape of the object S is likely to change significantly when the operation by the robot 10 is switched. Therefore, by determining whether to acquire new three-dimensional coordinate data by the acquisition means 51B based on the determination result of the determination means 51I, it is possible to suppress overlooking the change in the shape of the object S. In addition, when new three-dimensional coordinate data is acquired based on the determination result of the determination means 51I and the mapping process is in progress, the ongoing mapping process may be aborted. Then, based on the newly acquired three-dimensional coordinate data, the mapping process may be started again.

[0049] However, regardless of the above, the acquisition means 51B may, for example, prioritize the acquisition of three-dimensional coordinate data according to the reception result over the acquisition of three-dimensional coordinate data according to the detection result, the timing result, etc. Such processing may be performed, for example, by being appropriately set by the user. By this, it may be possible to perform processing more in line with the user's intention.

[0050] (Configuration for facilitating determination of whether to start work) As a configuration for facilitating the user's determination of whether to start the work by the robot 10, the control unit 51 includes a notification means 51J, an output means 51K, a notification means 51L, and a reporting means 51M. The notification means 51J notifies the user that it is possible to acquire three-dimensional coordinate data by the acquisition means 51B. That is, the notification means 51J notifies the user that, based on the information output by each of the above configurations, there is no determination that the acquisition of three-dimensional coordinate data by the acquisition means 51B will not be performed.

[0051] As a method for the notification means 51J to notify the user of the above content, for example, there are a method of displaying characters, marks, etc. on the display unit 52a, and a method of voice using a speaker (not shown). Alternatively, when the terminal operated by the user is a mobile terminal, the above content may be notified by the mobile terminal vibrating in a specific pattern. The notification means 51J is preferably capable of appropriately controlling the display unit 52a, the speaker (not shown), or the vibration of the mobile terminal in order to implement any of the above methods.

[0052] The output means 51K outputs to the user that the acquisition means 51B is acquiring three-dimensional coordinate data, or that the mapping means 51C is in the middle of mapping processing, etc. That is, for example, when the user inputs a command to instruct the operation of the robot 10, the output means 51K outputs the reason so that the user can understand why the acquisition of three-dimensional coordinate data by the acquisition means 51B or the mapping processing by the mapping means 51C is not performed. More specifically, when it is determined that the acquisition means 51B does not acquire the three-dimensional coordinate data or the mapping means 51C does not perform the mapping process based on the information output by each of the above-described configurations, the output means 51K outputs the reason in a manner that can be grasped by the user. The method by which the output means 51K outputs the above content so that the user can grasp it may be the same as the method by which the notification means 51J notifies the user of the above content, for example.

[0053] The notification means 51L notifies the user that the acquisition of the three-dimensional coordinate data by the acquisition means 51B has been completed or that the mapping process by the mapping means 51C has been completed. By doing so, when a command for instructing the operation of the robot 10 is input to the user, for example, the notification means 51L enables the user to grasp that the acquisition of the three-dimensional coordinate data by the acquisition means 51B or the mapping process by the mapping means 51C is being performed. The method by which the notification means 51L notifies the user of the above content may be the same as the method by which the notification means 51J notifies the user of the above content, for example.

[0054] The reporting means 51M reports that a predetermined time has elapsed since the completion of the acquisition of the three-dimensional coordinate data by the acquisition means 51B or that a predetermined time has elapsed since the completion of the mapping process by the mapping means 51C. By doing so, when a command for instructing the operation of the robot 10 is input to the user, for example, the reporting means 51M enables the user to grasp that the acquisition of the three-dimensional coordinate data by the acquisition means 51B or the mapping process by the mapping means is being performed. The method by which the reporting means 51M reports the above content to the user may be the same as the method by which the notification means 51J notifies the user of the above content, for example.

[0055] When the user inputs a command to instruct the operation of the robot 10 by means of the functions of each of the above-described configurations, the user grasps whether the acquisition means 51B acquires three-dimensional coordinate data or the mapping means performs mapping processing. Based on this, the user may switch the operation of the robot 10 as follows, for example. That is, for example, when the acquisition of three-dimensional coordinate data and the mapping processing are being performed, the operation of the robot 10 is semi-automatically performed using the command generated by the generation means 51Q described later. Then, when the acquisition of three-dimensional coordinate data and the mapping processing are not being performed, the operation is switched so that the user performs all detailed operations related to the operation of the robot 10. By the user operating the robot 10 as described above, the work by the robot control system 1 according to the present embodiment can be efficiently performed.

[0056] (Configuration for facilitating work in accordance with intentions) As a configuration for facilitating the work by the robot 10 for the user in accordance with the intention, the control unit 51 includes a selection means 51N, a specifying means 51O, a memory control means 51P, and a generation means 51Q. The selection means 51N selects any one of the detection result of the detection means 51E, the measurement result of the first measurement means 51F, and the determination result of the determination means 51H. The selection means 51N makes a selection of any one of the detection result, the measurement result, and the determination result in accordance with the intention of the user. That is, the selection means 51N selects any one of the detection result, the measurement result, and the determination result based on, for example, an instruction input by the user via the input unit 52b. The acquisition means 51B may acquire three-dimensional coordinate data according to any one of the detection result, the timing result, and the determination result selected by the selection means 51N. That is, the acquisition means 51B may determine whether to acquire three-dimensional coordinate data by using any one of the detection result, the timing result, and the determination result selected by the selection means 51N. Thus, for example, the user can select, by the selection means 51N, the conditions when the acquisition means 51B updates the three-dimensional coordinate data. Therefore, the control of the robot 10 can be performed more in accordance with the user's intention.

[0057] The specifying means 51O specifies a target position in the two-dimensional image data input by the input means 51A. That is, the specifying means 51O specifies which part of the two-dimensional image data output by the imaging means 30 corresponds to the target position of the robot 10 input by the user via the input unit 52b. This makes it easier for the specifying means 51O to store in the storage unit 53 which part of the two-dimensional image data is set as the target position by the user.

[0058] The storage control means 51P associates the learning data and the teacher data with each other and stores them in the storage unit 53 as a learning data set. In the present embodiment, the learning data is the two-dimensional image data used for the specifying process of the specifying means 51O. And the teacher data is the command and the target position corresponding to the command specified by the specifying means 51O. That is, the storage control means 51P associates, among the commands input by the user, particularly the target position of the robot 10 and the type of operation of the robot 10 at the target position, with the two-dimensional image data used for the specifying process of the specifying means 51O and stores them in the storage unit 53. Thus, the details of the command once input by the user can be stored in association with the two-dimensional image data. The information stored in this way can, for example, simplify the user's operation by being used when the user performs the same work as the work related to the stored command. Alternatively, the information stored in this way can, for example, be used when verifying the operation of the robot 10 based on the command input by the user.

[0059] The generation means 51Q generates a command using the two-dimensional image data input by the input means 51A. That is, the generation means 51Q grasps the shape of the object S and the situation around the object S based on the two-dimensional image data, and automatically generates a command for performing an operation on the object S. The generation of the command by the generation means 51Q is performed based on, for example, the learning data set stored in the memory control means 51P. For example, first, the generation means 51Q selects, from the two-dimensional image data stored as learning data, those in which the shape of the object S and the situation around the object S are similar to the two-dimensional image data input by the input means 51A, using, for example, statistical processing or the like. Then, referring to the commands and target positions input by the user in the past and stored as teacher data corresponding to the selected two-dimensional image data which is the learning data, a command is generated. When the generation means 51Q generates a command in this way, for example, one of the plurality of stored learning data sets may be selected, and a command equivalent to the teacher data related to the selected learning data set may be generated. Alternatively, a plurality of learning data sets similar to the two-dimensional image data input by the input means 51A may be selected, and they may be appropriately combined to generate an optimal command. In this way, by automatically generating a command by the generation means 51Q, it is possible to contribute to the automation of the control of the robot 10 by the robot control system 1. Therefore, for example, the operation of the user can be simplified.

[0060] (Control method of robot control system) Next, a control method of the robot control system 1 according to the present embodiment will be described. That is, in the robot control system 1, a control method for performing mapping that associates the two-dimensional image data output by the imaging means 30 and the three-dimensional coordinate data output by the scanning means 40 will be described. When performing the mapping between two-dimensional image data and three-dimensional coordinate data, it is necessary to pre-convert (calibrate) the coordinate system of the three-dimensional coordinate data output by the scanning means 40 according to the coordinate system of the two-dimensional image data output by the imaging means 30 based on the installation locations of the imaging means 30 and the scanning means 40 in advance. That is, it is necessary to calculate the external parameters for converting the three-dimensional coordinate data into the coordinate system of the two-dimensional image data. Hereinafter, in the description of the control method of the robot control system 1, the preliminary preparations performed before operating the robot control system 1 will be described.

[0061] (Preliminary Preparations) The preliminary preparations performed before operating the robot control system 1 are carried out as follows. As the first step of the preliminary preparations, determine the installation locations of the imaging means 30 and the scanning means 40 with respect to the work object T. The installation locations of the imaging means 30 and the scanning means 40 are preferably at positions where the surrounding situation of the object S in the work object T can be grasped and that do not interfere with the operations of the robot 10 and the end effector 20.

[0062] As the second step of the preliminary preparations, calculate the internal parameters of the imaging means 30. That is, based on the characteristics of the image distortion, focal length, etc. in the imaging means 30, calculate the internal parameters for setting the coordinate system of the two-dimensional image data, which serves as a reference when converting the three-dimensional coordinate data.

[0063] As the third step of the preliminary preparations, obtain the calibration data between the coordinate system of the scanning means 40 and the coordinate system of the robot 10. The calibration data related to the third step is a mathematical formula for converting the coordinate system of the three-dimensional coordinate data acquired by the scanning means 40 into the coordinate system of the robot 10. The calibration data thus obtained is used when converting the coordinate system of the three-dimensional coordinate data into the coordinate system of the robot 10 during the operation of the robot 10 on the object S.

[0064] As the fourth step of the preliminary preparation, data for calibration is acquired from the imaging means 30 and the scanning means 40. That is, two-dimensional image data as shown in FIG. 3 and three-dimensional coordinate data as shown in FIG. 4 are respectively output as samples from the imaging means 30 and the scanning means 40 installed with respect to the work target T. As the fifth step of the preliminary preparation, as shown in FIGS. 3 and 4, calibration points CP in the two-dimensional image data and the three-dimensional coordinate data output in the fourth step are acquired. That is, in the two-dimensional image data and the three-dimensional coordinate data, at corresponding portions, portions that are not affected by the operation of the robot 10 are set as marks for calibration. By the above steps, preliminary preparation before operating the robot control system 1, that is, calculation of external parameters is performed.

[0065] (Operation steps of the robot control system) Next, the operation steps of the robot control system 1 according to the present embodiment will be described. FIG. 7 is a flowchart of the control method of the robot control system 1 according to the embodiment. The operation steps of the robot control system 1 include, as shown in FIG. 7, a determination step S1, an input step S2, an acquisition step S3, a calibration step S4, a mapping step S5, a specification step S6, a coordinate acquisition step S7, and an input step S8. The flow shown in FIG. 7 is started, for example, by starting the robot control system 1. The flow thus started is repeatedly performed, for example, until an instruction to end the flow is given by the user (SA: YES). In other words, the flow of the control method of the robot control system 1 according to the present embodiment is repeatedly performed when an instruction to end the flow is not given by the user (SA: NO).

[0066] FIG. 8 is a flowchart of the determination step S1. The determination step S1 is a step of determining whether to acquire three-dimensional coordinate data by the acquisition means 51B. The determination step S1 includes a reception step S1A. In the determination step S1, after obtaining the reception result in the reception step S1A, based on the respective information and results output by the above-described detection means 51E, first timing means 51F, second timing means 51G, determination means 51H, and discrimination means 51I, it is determined whether to acquire three-dimensional coordinate data.

[0067] The reception step S1A is a step of receiving a command for instructing the robot 10 to perform work on the object S. Further, the reception step S1A is a step in which the reception means 51D outputs, as a reception result, the result of whether the user has input a command via the input unit 52b. In the reception step S1A, for example, the command input by the user via the input unit 52b is received separately as the above-described target position and orientation data and work content data. The command received in the reception step S1A is stored in the storage unit 53 as a learning data set, for example, after being associated with the two-dimensional image data input in the input step S2, which is learning data as teacher data. In the determination step S1, after obtaining whether the reception result is a result (S1A: YES) indicating that the user has input a command or a result (S1A: NO) indicating that no command has been input, the information and results output by each of the following means are referred to, and the next step to transition to is determined.

[0068] (When the reception result is a result indicating that the user has input a command) First, the case where the reception result of the reception step S1A is a result (S1A: YES) indicating that the user has input a command will be described. After the reception step S1A, when the detection means 51E outputs a detection result indicating that the shape of the object S has changed (S1B: YES), it is determined that the acquisition means 51B acquires the three-dimensional coordinate data (S1: YES). When the detection means 51E outputs a detection result indicating that the shape of the object S has not changed (S1B: NO), whether the acquisition means 51B acquires the three-dimensional coordinate data is determined according to the determination result of the discrimination means 51I.

[0069] After the reception step S1A, when the measurement result by the first measurement means 51F has elapsed a preset predetermined time (S1C: YES), it is determined that the acquisition means 51B acquires the three-dimensional coordinate data (S1: YES). When the measurement result by the first measurement means 51F has not elapsed the preset predetermined time (S1C: NO), whether the acquisition means 51B acquires the three-dimensional coordinate data is determined according to the determination result of the discrimination means 51I.

[0070] After the reception step S1A, when the elapsed time measured by the second measurement means 51G has elapsed a preset predetermined time (S1D: YES), it is determined that the acquisition means 51B acquires the three-dimensional coordinate data (S1: YES). When the elapsed time measured by the second measurement means 51G has not elapsed the preset predetermined time (S1D: NO), whether the acquisition means 51B acquires the three-dimensional coordinate data is determined according to the determination result of the discrimination means 51I.

[0071] After the reception step S1A, when the determination means 51H outputs a determination result indicating that the mapping means 51C is not in the mapping process (S1E: NO), it is determined that the acquisition means 51B acquires the three-dimensional coordinate data (S1: YES). When the determination means 51H outputs a determination result indicating that the mapping means 51C is in the mapping process (S1E: YES), whether the acquisition means 51B acquires the three-dimensional coordinate data is determined according to the determination result of the discrimination means 51I.

[0072] When it is determined by the determination result of the determination means 51I whether to acquire the three-dimensional coordinate data by the acquisition means 51B according to each of the above means, if the determination means 51I determines that a command for only one of the pushing operation or the stirring operation is continuously input (S1F: YES), it is determined not to acquire new three-dimensional coordinate data (S1: NO). When the determination means 51I determines that the command input by the user has changed from one of the pushing operation or the stirring operation to the other (S1F: NO), it is determined to acquire the three-dimensional coordinate data by the acquisition means 51B, giving priority to the determination based on the information output by each of the above means (S1: YES).

[0073] (When the reception result is a result indicating that the user has not input a command) Next, the case where the reception result is a result (S1A: NO) indicating that the user has not input a command will be described. After the reception step S1A, when the detection means 51E outputs a detection result indicating that the shape of the object S has changed (S1G: YES), it is determined to acquire the three-dimensional coordinate data by the acquisition means 51B even if no command is input by the user (S1: YES). When the detection means 51E outputs a detection result indicating that the shape of the object S has not changed (S1G: NO), it is determined not to acquire new three-dimensional coordinate data (S1: NO).

[0074] After the reception step S1A, when the timing result by the first timing means 51F indicates that a preset predetermined time has elapsed (S1H: YES), it is determined to acquire the three-dimensional coordinate data by the acquisition means 51B even if no command is input by the user (S1: YES). When the timing result by the first timing means 51F indicates that the preset predetermined time has not elapsed (S1H: NO), it is determined not to acquire new three-dimensional coordinate data (S1: NO).

[0075] After the reception step S1A, when the elapsed time measured by the second timing means 51G has elapsed a predetermined time set in advance (S1I: YES), it is determined that the acquisition means 51B acquires three-dimensional coordinate data even if no command is input by the user (S1: YES). When the elapsed time measured by the second timing means 51G has not elapsed the predetermined time set in advance (S1I: NO), it is determined not to acquire new three-dimensional coordinate data (S1: NO).

[0076] After the reception step S1A, when the determination means 51H outputs a determination result indicating that the mapping means 51C is not in the mapping process (S1J: NO), it is determined that the acquisition means 51B acquires three-dimensional coordinate data even if no command is input by the user (S1: YES). When the determination means 51H outputs a determination result indicating that the mapping means 51C is in the mapping process (S1J: YES), it is determined not to acquire new three-dimensional coordinate data (S1: NO).

[0077] In the determination step S1, when it is determined not to acquire the three-dimensional coordinate data by the acquisition means 51B (S1: NO), the determination step S1 is performed again. In the determination step S1, when it is determined to acquire the three-dimensional coordinate data by the acquisition means 51B (S1: YES), the flow proceeds to the input step S2.

[0078] The input step S2 is a step in which two-dimensional image data of the object S is input. That is, in the input step S2, the input means 51A receives the two-dimensional image data output by the imaging means 30 and inputs it to the control unit 51. The two-dimensional image data input in this way is displayed on the display unit 52a as needed, for example.

[0079] The acquisition step S3 is a step of acquiring the three-dimensional coordinate data of the object S according to the reception result of the reception step S1A when it is determined in the determination step S1 that the three-dimensional coordinate data is to be acquired. That is, in the acquisition step S3, the acquisition means 51B acquires the three-dimensional coordinate data output by the scanning means 40 and inputs it to the control unit 51.

[0080] The calibration step S4 is a step of converting the three-dimensional coordinate data acquired in the acquisition step S3 into data that can be projected onto the two-dimensional image data. That is, first, based on the calibration data acquired in the third step of the above-mentioned preliminary preparation, the coordinate system of the three-dimensional coordinate data is converted into the coordinate system of the robot 10. Then, based on the calibration points CP shown in FIGS. 3 and 4 acquired in the fifth step of the above-mentioned preliminary preparation, the three-dimensional coordinate data is converted into the coordinate system of the two-dimensional image data. Further, based on the internal parameters of the imaging means 30 acquired in the second step of the above-mentioned preliminary preparation, the three-dimensional coordinate data converted into the coordinate system of the two-dimensional image data is converted into data that can be projected onto the two-dimensional image data. As data that can be projected onto the two-dimensional image data, specifically, for example, for the portion corresponding to the target position specified in the reception step S1A in the two-dimensional image data, an image is generated with the values of the x coordinate, y coordinate, and z coordinate in the coordinate system of the robot 10 as the luminance respectively.

[0081] The mapping step S5 is a step of mapping the three-dimensional coordinate data acquired in the acquisition step S3 to the two-dimensional image data input by the input means 51A. More specifically, in the mapping step S5, the data projectable onto the two-dimensional image data generated in the calibration step S4 is projected onto the two-dimensional image data displayed on the display unit 52a. By outputting a mapping result as shown in FIG. 5 in the mapping step S5, the user can grasp the portion corresponding to the target position specified in the reception step S1A by visually observing the display unit 52a.

[0082] The specifying step S6 specifies the target position of the robot 10 in the two-dimensional image data input by the input means 51A and displayed on the display unit 52a. This makes it easier for the user to more accurately grasp the target position of the robot 10 specified by the user. Alternatively, it becomes easier to store in the storage unit 53 which portion of the two-dimensional image data was set as the target position by the user.

[0083] The coordinate acquisition step S7 acquires the coordinates of the target position of the robot 10 in the two-dimensional image data specified in the specifying step S6. This acquires the target position in the coordinate system of the robot 10 when the robot 10 performs work on the object S. The input step S8 is a step of inputting the target position in the coordinate system of the robot 10 acquired in the coordinate acquisition step S7 to the robot 10. This enables the robot 10 to move the tip of the end effector 20 toward the target position. After the input step S8 is performed, the flow returns to the determination step S1 again. This causes each of the above steps to be repeated. By the above steps, mapping of three-dimensional coordinate data to two-dimensional image data and work by the robot 10 on the object S are performed.

[0084] As described above, according to the robot control system 1 according to the present embodiment, the acquisition of the three-dimensional coordinate data by the acquisition means 51B is performed according to the reception result of the reception means 51D regarding the command for instructing the robot 10 to perform the work. Then, the mapping of the three-dimensional coordinate data to the two-dimensional image data by the mapping means 51C is performed according to the acquisition of the three-dimensional coordinate data by the acquisition means 51B. That is, the acquisition of the three-dimensional coordinate data of the object S and the mapping of the acquired three-dimensional coordinate data to the two-dimensional image data are performed in response to receiving a command for instructing the robot 10 to perform the work. By performing the above processing, the two-dimensional image data and the three-dimensional coordinate data of the object S can be associated with each other. Therefore, it is possible to make it easier for the user to grasp the shape, position, etc. of the object S. In addition, for the object S whose shape changes over time, the three-dimensional coordinate data acquired immediately before the robot 10 starts working can be mapped to the two-dimensional image data. Therefore, the result of the mapping can be made more realistic. Therefore, it is possible to specify an appropriate position for the object S whose shape changes over time. Also, for example, compared with the case where the acquisition of the three-dimensional coordinate data and the mapping process to the two-dimensional image data are always performed regardless of the reception result of the command, an increase in the processing load of the robot control system 1 can be suppressed.

[0085] In addition, it further includes a detection means 51E for detecting that the shape of the object S has changed after the acquisition means 51B has acquired the three-dimensional coordinate data. The acquisition means 51B also acquires the three-dimensional coordinate data according to the detection result of the detection means 51E. That is, based on the detection result of the detection means 51E, when the shape of the object S changes, the acquisition means 51B appropriately updates the information of the three-dimensional coordinate data. Therefore, the result of the mapping can be made more surely realistic. In addition, the above-described effects can be enjoyed without constantly performing the acquisition and mapping processing of three-dimensional coordinate data. Therefore, while suppressing an increase in the processing load of the robot control system 1, the processing by the robot control system 1 can be made efficient.

[0086] In addition, it further includes a first time measuring means 51F for measuring the time elapsed since the reception means 51D received a command. The acquisition means 51B acquires three-dimensional coordinate data also according to the measurement result of the first time measuring means 51F. That is, based on the measurement result of the first time measuring means 51F, when a certain period of time has elapsed since the reception means 51D received a command, the acquisition means 51B appropriately updates the information of the three-dimensional coordinate data. Thus, the result of mapping can be made more surely conform to reality. In addition, the above-described effects can be enjoyed without constantly performing the acquisition and mapping processing of three-dimensional coordinate data. Therefore, while suppressing an increase in the processing load of the robot control system 1, the processing by the robot control system 1 can be made efficient.

[0087] In addition, the acquisition means 51B preferentially performs the acquisition of three-dimensional coordinate data according to the reception result over the acquisition of three-dimensional coordinate data according to the detection result and the measurement result, etc. That is, even when the detection means 51E does not detect a change in the shape of the object S and the measurement result of the first time measuring means 51F has not elapsed a certain period of time, when the reception means 51D receives a command, the information of the three-dimensional coordinate data is appropriately updated. Thereby, the acquisition of the three-dimensional coordinate data by the acquisition means 51B can be performed more in accordance with the operation of the robot 10 by the user. That is, by appropriately updating the information of the three-dimensional coordinate data when the user operates the robot 10, it is easier to make the result of mapping when the user performs work conform to reality.

[0088] Here, for example, if the three-dimensional coordinate data is updated while the mapping process by the mapping means 51C is in progress, it will cause a delay in the mapping process due to the time difference until the mapping process is completed. Therefore, it further includes a determination means 51H for determining whether the mapping means 51C is in the middle of the mapping process. The acquisition means 51B acquires the three-dimensional coordinate data according to the determination result of the determination means 51H. That is, for example, when the mapping means 51C is in the middle of the mapping process, the acquisition of the three-dimensional coordinate data is not performed. Thereby, it is possible to suppress the occurrence of a delay in the mapping process by the mapping means 51C.

[0089] It further includes a selection means 51N for selecting any one of the detection result of the detection means 51E, the measurement result of the first measurement means 51F, and the determination result of the determination means 51H. The acquisition means 51B acquires the three-dimensional coordinate data according to any one of the detection result, the measurement result, and the determination result selected by the selection means 51N. Thereby, the condition when the acquisition means 51B updates the three-dimensional coordinate data can be selected by the selection means 51N. Therefore, the control of the robot 10 can be performed more in accordance with the user's intention.

[0090] It further includes a second measurement means 51G for measuring the time elapsed since the mapping means 51C executed the mapping process. The mapping means 51C maps the three-dimensional coordinate data acquired by the acquisition means 51B to the two-dimensional image data input by the input means 51A according to the elapsed time measured by the second measurement means 51G. That is, the mapping means 51C does not perform a new mapping process when a certain time has not elapsed since the mapping means 51C executed the mapping process based on the elapsed time measured by the second measurement means 51G. Thereby, for example, it is possible to more reliably suppress the occurrence of a delay in the mapping process by the mapping means 51C.

[0091] Here, for example, if the three-dimensional coordinate data is updated while the mapping process by the mapping means 51C is in progress, it will cause a delay in the mapping process as described above. In order to suppress the occurrence of a delay in the mapping process, when the mapping means 51C is in the mapping process, the acquisition of the three-dimensional coordinate data may be controlled so as not to be performed. Therefore, it further includes a notification means 51J for notifying the user that the acquisition of the three-dimensional coordinate data by the acquisition means 51B is possible. By this, the user can more easily grasp the state of the robot control system 1. That is, for example, the user can more easily grasp whether the acquisition of the three-dimensional coordinate data corresponding to the input of the command for instructing the robot 10 to perform the work is performed. Therefore, for the user, it is easier to control the robot 10 as intended.

[0092] Also, the command for instructing the robot 10 to perform the work includes target position and orientation data indicating the target position and orientation of the robot 10, and work content data indicating the content of the work. Thereby, the user can appropriately operate the robot 10 with the posture and position of the robot 10 and the content of the work performed by the robot 10.

[0093] It also includes a specifying means 51O for specifying the target position in the two-dimensional image data input by the input means 51A. Thereby, for example, the target position of the robot 10 in the command instructed by the user can be reflected in the two-dimensional image data and visually displayed. Therefore, the user can more intuitively control the robot 10. Furthermore, a storage control means 51P is provided which stores, in a storage unit 53, by associating, two-dimensional image data used for the specifying process of the specifying means 51O and a command when a target position is input. Thereby, the details of the command once input by the user can be stored in association with the two-dimensional image data. The information stored in this way can be used, for example, when the user performs the same work as the work related to the stored command, thereby simplifying the user's operation. Alternatively, the information stored in this way can be used, for example, when verifying the operation of the robot 10 based on the command input by the user.

[0094] Also, a generating means 51Q is further provided which generates a command using the two-dimensional image data input by the input means 51A. Thereby, it is possible to contribute to the automation of the control of the robot 10 by the robot control system 1. Thus, for example, the operation of the user can be simplified.

[0095] Also, according to the control method of the robot control system 1 according to the present embodiment, the acquisition of the three-dimensional coordinate data in the acquisition step S3 is performed, for example, according to the reception result of the command for instructing the robot 10 to perform work in the reception step S1A. Then, the mapping of the three-dimensional coordinate data to the two-dimensional image data in the mapping step S5 is performed according to the acquisition of the three-dimensional coordinate data in the acquisition step S3. That is, the acquisition of the three-dimensional coordinate data of the object S and the mapping of the acquired three-dimensional coordinate data to the two-dimensional image data are performed in response to receiving a command for instructing the robot 10 to perform work. By performing the above-described processing, the two-dimensional image data and the three-dimensional coordinate data of the object S can be associated with each other. Thus, it is possible to make it easier for the user to grasp the shape, position, etc. of the object S. Also, for the object S whose shape changes over time, the three-dimensional coordinate data acquired immediately before the robot 10 starts working can be mapped to the two-dimensional image data. Thus, the result of the mapping can be made more realistic. In addition, for example, compared with the case of always performing the processes of acquiring three-dimensional coordinate data and mapping it to two-dimensional image data regardless of the reception result of a command, an increase in the processing load of the robot control system 1 can be suppressed.

[0096] Note that the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present disclosure.

[0097] In addition, within the scope not departing from the spirit of the present disclosure, it is possible to appropriately replace the components in the above-described embodiments with well-known components, and the above-described modification examples may also be appropriately combined.

[0098] (Appendix) The robot control system according to the above-described embodiment is understood as follows, for example. <1> A robot control system according to an aspect of the present disclosure is a robot control system including a robot that performs work on an object whose shape changes over time, the robot control system including input means for inputting two-dimensional image data of the object, reception means for receiving a command for instructing the robot to perform the work, acquisition means for acquiring the three-dimensional coordinate data of the object according to the reception result of the reception means, and mapping means for mapping the three-dimensional coordinate data acquired by the acquisition means to the two-dimensional image data input by the input means according to the acquisition of the three-dimensional coordinate data by the acquisition means.

[0099] The acquisition of the three-dimensional coordinate data by the acquisition means is performed according to the reception result of the reception means regarding the command for instructing the robot to perform the work. Then, the mapping of the three-dimensional coordinate data to the two-dimensional image data by the mapping means is performed according to the acquisition of the three-dimensional coordinate data by the acquisition means. That is, the acquisition of the three-dimensional coordinate data of the object and the mapping of the acquired three-dimensional coordinate data to the two-dimensional image data are performed in response to receiving a command for instructing the robot to perform the work. By performing the above processing, the two-dimensional image data of the object and the three-dimensional coordinate data can be associated with each other. Therefore, it is possible for the user to easily grasp the shape, position, etc. of the object. Also, for an object whose shape changes over time, the three-dimensional coordinate data acquired immediately before the robot starts working can be mapped to the two-dimensional image data. Therefore, the result of the mapping can be made more realistic. Accordingly, for an object whose shape changes over time, it is possible to specify an appropriate position. In addition, for example, compared with the case where the acquisition of three-dimensional coordinate data and the mapping process to two-dimensional image data are always performed regardless of the reception result of the command, an increase in the processing load of the robot control system can be suppressed.

[0100] <2>In the robot control system according to <1> above, it further includes a detection means for detecting that the shape of the object has changed after the acquisition means has acquired the three-dimensional coordinate data, and the acquisition means acquires the three-dimensional coordinate data in response to the detection result of the detection means. A configuration characterized by this may be adopted.

[0101] Also, it further includes a detection means for detecting that the shape of the object has changed after the acquisition means has acquired the three-dimensional coordinate data. The acquisition means acquires the three-dimensional coordinate data in response to the detection result of the detection means. That is, the acquisition means updates the information of the three-dimensional coordinate data as appropriate when the shape of the object changes based on the detection result of the detection means. Therefore, the result of the mapping can be made more surely realistic. In addition, the above effects can be enjoyed without always performing the acquisition of three-dimensional coordinate data and the mapping process. Therefore, while suppressing an increase in the processing load of the robot control system, the processing by the robot control system can be made efficient.

[0102] <3>In the robot control system according to <1> or <2> above, it further includes a first timing means for timing the time elapsed since the reception means received the command, and the acquisition means acquires the three-dimensional coordinate data according to the timing result of the first timing means. A configuration characterized by this may be adopted.

[0103] Further, it further includes a first timing means for timing the time elapsed since the reception means received the command. The acquisition means acquires the three-dimensional coordinate data according to the timing result of the first timing means. That is, based on the timing result of the first timing means, when a certain period of time has elapsed since the reception means received the command, the acquisition means appropriately updates the information of the three-dimensional coordinate data. Therefore, the result of mapping can be made more reliably conform to reality. Also, the above effects can be enjoyed without constantly performing the acquisition and mapping processing of the three-dimensional coordinate data. Therefore, while suppressing an increase in the processing load of the robot control system, the processing by the robot control system can be made efficient.

[0104] <4>In the robot control system according to any one of <1> to <3> above, it further includes a detection means for detecting that the shape of the object has changed after the acquisition means acquires the three-dimensional coordinate data, and a first timing means for timing the time elapsed since the reception means received the command. The acquisition means acquires the three-dimensional coordinate data according to the detection result of the detection means and the timing result of the first timing means, and the acquisition means gives priority to acquiring the three-dimensional coordinate data according to the reception result over acquiring the three-dimensional coordinate data according to the detection result and the timing result. A configuration characterized by this may be adopted.

[0105] Further, the acquisition means preferentially acquires the three-dimensional coordinate data according to the reception result over the acquisition of the three-dimensional coordinate data according to the detection result and the timing result. That is, even when the detection means does not detect a change in the shape of the object and the timing result of the first timing means has not elapsed for a certain period of time, if the reception means receives a command, the information of the three-dimensional coordinate data is appropriately updated. Thereby, the acquisition of the three-dimensional coordinate data by the acquisition means can be performed more in accordance with the operation of the robot by the user. That is, by appropriately updating the information of the three-dimensional coordinate data when the user operates the robot, it is easier to make the mapping result more in line with the actual situation when the user performs the work.

[0106] <5>In the robot control system according to any one of <1> to <4> above, it further includes a determination means for determining whether or not the mapping means is in the middle of the mapping process, and the acquisition means acquires the three-dimensional coordinate data according to the determination result of the determination means. A configuration characterized by this may be adopted.

[0107] Here, for example, if the three-dimensional coordinate data is updated during the mapping process by the mapping means, it may cause a delay in the mapping process due to the time difference until the mapping process is completed. Therefore, it further includes a determination means for determining whether or not the mapping means is in the middle of the mapping process. The acquisition means acquires the three-dimensional coordinate data according to the determination result of the determination means. That is, for example, when the mapping means is in the middle of the mapping process, the acquisition of the three-dimensional coordinate data is not performed. Thereby, it is possible to suppress a delay in the mapping process by the mapping means.

[0108] <6>In the robot control system according to any one of <1> to <5> above, detection means for detecting that the shape of the object has changed after the acquisition means has acquired the three-dimensional coordinate data, first timing means for measuring the time elapsed since the reception means received the command, determination means for determining whether or not the mapping means is in the middle of a mapping process, and selection means for selecting any one of the detection result of the detection means, the timing result of the first timing means, and the determination result of the determination means. The acquisition means may adopt a configuration characterized by acquiring the three-dimensional coordinate data in accordance with any one of the detection result, the timing result, and the determination result selected by the selection means.

[0109] Furthermore, it further includes selection means for selecting any one of the detection result of the detection means, the timing result of the first timing means, and the determination result of the determination means. The acquisition means acquires the three-dimensional coordinate data in accordance with any one of the detection result, the timing result, and the determination result selected by the selection means. Thereby, the conditions for the acquisition means to update the three-dimensional coordinate data can be selected by the selection means. Therefore, the control of the robot can be performed more in line with the user's intention.

[0110] <7>In the robot control system according to any one of <1> to <6> above, it further includes second timing means for measuring the time elapsed since the mapping means executed the mapping process. The mapping means maps the three-dimensional coordinate data acquired by the acquisition means to the two-dimensional image data input by the input means in accordance with the elapsed time measured by the second timing means. A configuration characterized by this may be adopted.

[0111] Furthermore, it further includes a second timing means for timing the time elapsed since the mapping means executed the mapping process. The mapping means also maps the three-dimensional coordinate data acquired by the acquisition means to the two-dimensional image data input by the input means according to the elapsed time measured by the second timing means. That is, based on the elapsed time measured by the second timing means, when a certain period of time has not elapsed since the mapping means executed the mapping process, the mapping means does not perform a new mapping process. Thereby, for example, it is possible to more reliably suppress the occurrence of a delay in the mapping process by the mapping means.

[0112] <8>In the robot control system according to any one of the above aspects <1> to <7>, it may further adopt a configuration characterized by further including a notification means for notifying the user that the acquisition means can acquire the three-dimensional coordinate data.

[0113] Here, for example, if the three-dimensional coordinate data is updated during the mapping process by the mapping means, it will cause a delay in the mapping process as described above. In order to suppress the occurrence of a delay in the mapping process, when the mapping means is in the middle of the mapping process, the acquisition of the three-dimensional coordinate data may be controlled not to be performed. Therefore, it further includes a notification means for notifying the user that the acquisition means can acquire the three-dimensional coordinate data. By this, the user can more easily grasp the state of the robot control system. That is, for example, the user can more easily grasp whether the acquisition of the three-dimensional coordinate data corresponding to the input of the command for instructing the robot to perform the work is performed. Therefore, for the user, it is possible to more easily control the robot as intended.

[0114] <9>In the robot control system according to any one of the above aspects <1> to <8>, the command may adopt a configuration characterized by including target position and orientation data indicating the target position and orientation of the robot and work content data indicating the content of the work.

[0115] In addition, the command for instructing the robot to perform work includes target position and orientation data indicating the target position and orientation of the robot, and work content data indicating the content of the work. Thereby, the user can appropriately operate the robot in terms of the robot's orientation and position, and the content of the work performed by the robot.

[0116] <10>In the robot control system according to <9> above, it may adopt a configuration further comprising a specifying means for specifying the target position in the two-dimensional image data input by the input means, a storage control means for storing in a storage unit the two-dimensional image data used in the specifying process of the specifying means, and the command, in association with each other.

[0117] Also, it is provided with a specifying means for specifying the target position in the two-dimensional image data input by the input means. Thereby, for example, the target position of the robot in the command instructed by the user can be visually displayed by being reflected in the two-dimensional image data. Thus, the user can more intuitively perform the control of the robot. Furthermore, it is provided with a storage control means for storing in a storage unit the two-dimensional image data used in the specifying process of the specifying means and the command when the target position is input, in association with each other. Thereby, the user can store the details of the command once input, in association with the two-dimensional image data. The information stored in this way can be used, for example, to simplify the user's operation when the user performs the same work as the work related to the stored command. Alternatively, the information stored in this way can be used, for example, when the user verifies the operation of the robot based on the command input.

[0118] <11>In the robot control system according to any one of <1> to <10> above, it may adopt a configuration further comprising a generating means for generating the command using the two-dimensional image data input by the input means.

[0119] Furthermore, it further includes a generation means for generating a command using the two-dimensional image data input by the input means. Thereby, it can contribute to the automation of robot control by the robot control system. Therefore, for example, the operation of the user can be simplified.

[0120] <12>The control method according to one aspect of the present disclosure is a control method for a robot control system including a robot that performs work on an object whose shape changes over time, the method comprising: an input step for inputting two-dimensional image data of the object; a reception step for receiving a command for instructing the robot to perform the work; an acquisition step for acquiring three-dimensional coordinate data of the object according to the reception result of the reception step; and a mapping step for mapping the three-dimensional coordinate data acquired in the acquisition step to the two-dimensional image data input in the input step according to the acquisition of the three-dimensional coordinate data in the acquisition step.

[0121] The acquisition of the three-dimensional coordinate data in the acquisition step is performed according to the reception result of the reception step for the command for instructing the robot to perform the work. And the mapping of the three-dimensional coordinate data to the two-dimensional image data in the mapping step is performed according to the acquisition of the three-dimensional coordinate data in the acquisition step. That is, the acquisition of the three-dimensional coordinate data of the object and the mapping of the acquired three-dimensional coordinate data to the two-dimensional image data are performed in response to receiving a command for instructing the robot to perform the work. By performing the above-described processing, the two-dimensional image data and the three-dimensional coordinate data of the object can be associated with each other. Therefore, it can be made easier for the user to grasp the shape, position, etc. of the object. In addition, for an object whose shape changes over time, the three-dimensional coordinate data acquired immediately before the robot starts working can be mapped to the two-dimensional image data. Therefore, the result of the mapping can be made more realistic. In addition, for example, compared with the case where the acquisition of three-dimensional coordinate data and the mapping process to two-dimensional image data are always performed regardless of the reception result of the command, an increase in the processing load of the robot control system can be suppressed.

Explanation of Signs

[0122] 1 Robot control system 10 Robot 11 Arm 20 End effector 30 Imaging means 40 Scanning means 50 Control means 51 Control unit 51a Processor 51A Input means 51b Memory 51B Acquisition means 51C Mapping means 51D Reception means 51E Detection means 51F First timing means 51G Second timing means 51H Judgment means 51I Discrimination means 51J Notification means 51K Output means 51L Notification means 51M Reporting means 51N Selection means 51O Identification means 51P Memory control means 51Q Generation means 52 User interface 52a Display unit 52b Input unit 53 Memory unit S Object S2 Input step S1A Reception step S1 Judgment step S3 Acquisition step S4 Calibration step S5 Mapping step S6 Identification step S7 Coordinate Acquisition Step S8 Input Step T Work Object

Claims

1. A robot control system including a robot that performs work on an object whose shape changes over time, input means for inputting two-dimensional image data of the object, reception means for receiving a command for instructing the robot to perform the work, acquisition means for acquiring the three-dimensional coordinate data of the object according to the reception result of the reception means, mapping means for mapping the three-dimensional coordinate data acquired by the acquisition means to the two-dimensional image data input by the input means according to the acquisition of the three-dimensional coordinate data by the acquisition means, A robot control system characterized by comprising the above.

2. detection means for detecting that the shape of the object has changed since the acquisition means acquired the three-dimensional coordinate data, further comprising, The acquisition means acquires the three-dimensional coordinate data also according to the detection result of the detection means. The robot control system according to claim 1, characterized by the above.

3. first timing means for timing the time elapsed since the reception means received the command, further comprising, The acquisition means acquires the three-dimensional coordinate data also according to the timing result of the first timing means. The robot control system according to claim 1, characterized by the above.

4. detection means for detecting that the shape of the object has changed since the acquisition means acquired the three-dimensional coordinate data, first timing means for timing the time elapsed since the reception means received the command, further comprising, The acquisition means acquires the three-dimensional coordinate data according to the detection result of the detection means and the timing result of the first timing means, The acquisition means preferentially performs the acquisition of the three-dimensional coordinate data according to the reception result over the acquisition of the three-dimensional coordinate data according to the detection result and the timing result. The robot control system according to claim 1, characterized by the above.

5. determination means for determining whether or not the mapping means is in the mapping process, further comprising, The acquisition means acquires the three-dimensional coordinate data also according to the determination result of the determination means. The robot control system according to claim 1, characterized by the above.

6. detection means for detecting that the shape of the object has changed since the acquisition means acquired the three-dimensional coordinate data, first timing means for timing the time elapsed since the reception means received the command, Determination means for determining whether or not the mapping means is in the middle of a mapping process; Selection means for selecting any one of the detection result of the detection means, the measurement result of the first measurement means, and the determination result of the determination means; further comprising: The acquisition means acquires the three-dimensional coordinate data according to any one of the detection result, the measurement result, and the determination result selected by the selection means. The robot control system according to claim 1, characterized in that.

7. Second measurement means for measuring the time elapsed since the mapping means executed the mapping process; further comprising: The mapping means maps the three-dimensional coordinate data acquired by the acquisition means to the two-dimensional image data input by the input means according to the elapsed time measured by the second measurement means. The robot control system according to claim 1, characterized in that.

8. Notification means for notifying the user that it is possible to acquire the three-dimensional coordinate data by the acquisition means; The robot control system according to any one of claims 1 to 7, further comprising:

9. The command includes target position and orientation data indicating the target position and orientation of the robot, and work content data indicating the content of the work. The robot control system according to any one of claims 1 to 7, characterized in that.

10. Specification means for specifying the target position in the two-dimensional image data input by the input means; Storage control means for associating and storing the two-dimensional image data used for the specification process of the specification means and the command in a storage unit; The robot control system according to claim 9, further comprising:

11. Generation means for generating the command using the two-dimensional image data input by the input means; The robot control system according to any one of claims 1 to 7, further comprising:

12. A control method for a robot control system including a robot that performs work on an object whose shape changes over time, the method comprising: An input step for inputting two-dimensional image data of the object; A reception step for receiving a command for instructing the robot to perform the work; An acquisition step for acquiring the three-dimensional coordinate data of the object according to the reception result of the reception step; A mapping step of mapping the three-dimensional coordinate data acquired in the acquisition step to the two-dimensional image data input in the input step in response to the acquisition of the three-dimensional coordinate data in the acquisition step; A control method characterized by comprising the above.

Images