Work system

Abstract

To provide a work system which enables early familiarization with remote operation, regardless of a training level.SOLUTION: A work system in which a pick-and-place process can be performed on an object through transmission of the operation of a master-side device 4 by an operator to a slave-side device 5 has display means 8 for displaying the work screen 80 that shows a work area where a hand part 7 constituting the slave-side device 5 is arranged, in the vicinity of the master-side device 4. The work screen 80 shows a virtual location display M on which the location information of an operation part 6 constituting the master-side device 4 is reflected in real time.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to an operation system that enables remote control of a pick-and-place process. [Background technology]

[0002] In recent years, in response to the expansion of the electronic commerce (EC) market due to the spread of product purchasing and service use over the Internet, logistics sites are rapidly moving toward automating logistics processes using work robots in order to solve problems such as increasing logistics volume and serious labor shortages.

[0003] Patent documents 1 to 3 propose position / force control devices that can realize a bilateral control function, which transmits the operation of a master device to a slave device and feeds back the reaction force input from an object to the slave device to the master device, making it possible for working robots to perform delicate tasks like humans.

[0004] Furthermore, Patent Document 4 proposes a work system in which, in automating the picking work of sorting individual items using a robot, a person (operator) remotely controls the work robot to support judgments and work that cannot be completed by artificial intelligence (AI). The work system of Patent Document 4 synthesizes and presents a predetermined mark image on the object in the real-time image captured by a camera installed on the slave side and displayed on the master side display, allowing the operator to easily identify the object requiring work assistance, and the operation of the operation unit of the master side device by the operator is transmitted to the hand unit of the slave side device, making it possible to perform picking work on the object. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 4696307 (pages 6 to 7, Figure 1) [Patent Document 2] Patent No. 6382203 (pages 8 to 10, Figure 1) [Patent Document 3] Patent No. 6598401 (pages 9 to 11, Figure 1) [Patent Document 4] JP 2018-153874 A (pages 5 to 6, Figure 1) Summary of the Invention [Problem to be solved by the invention]

[0006] In the work system of Patent Document 4, a series of operations called a pick-and-place process is mainly performed, in which a specific object at a specific position is picked up by the hand unit of the slave device, and while holding it, it is transported to a predetermined position and the object is placed there. However, when an operator operates the operation unit of the master device while watching a real-time image displayed on the master side display that captures the working area of ​​the hand unit of the slave device, there is a risk that the operator will feel a sense of incongruity between the operational feeling of operating the operation unit of the master device and the movement of the hand unit of the slave device. This sense of incongruity is particularly noticeable for operators with low levels of skill, and there is a problem that it takes time to become accustomed to remote operation.

[0007] The present invention has been made in view of the above-mentioned problems, and has an object to provide a work system that enables users to quickly become familiar with remote operation regardless of their level of training. [Means for solving the problem]

[0008] In order to solve the above problems, the working system of the present invention comprises: An operation system capable of performing a pick-and-place process on an object by transmitting an operation of a master device by an operator to a slave device, comprising: a display means for displaying a work screen showing a work area in which a hand unit constituting the slave device is disposed, the display means being disposed near the master device; The work screen is characterized in that a virtual position display is displayed that reflects position information of an operation unit constituting the master side device in real time. According to this feature, when an operator operates the operation part of the master device, the operator can check the position of the virtual operation part displayed on the work screen, and by comparing it with the movement movement of the hand part of the slave device, the operator can easily adapt the operational feel of operating the operation part of the master device to the actual movement movement of the hand part of the slave device, making it possible for the operator to quickly become familiar with remote operation regardless of his or her level of skill.

[0009] The present invention is characterized in that a position display screen that simply represents the work area is displayed in a small window on the work screen, and the virtual position display is displayed on the position display screen. According to this feature, since the position of the virtual operation part can be confirmed on the position display screen which shows the working area in a simplified manner, it is easy to grasp the position information of the operation part.

[0010] The position display screen is characterized in that, in addition to displaying a virtual position of the operation unit, a virtual position display that reflects position information of the hand unit in real time is simultaneously displayed. According to this feature, since the position of the virtual operation unit and the position of the virtual hand unit can be compared on the position display screen, it is easy to match the operational feeling of operating the operation unit of the master device to the movement of the actual hand unit of the slave device. Also, it is easy to grasp the difference in the movement amount of the hand unit relative to the displacement amount of the operation unit.

[0011] The position display screen is characterized in that at least two auxiliary lines of perpendicular axes are displayed. According to this feature, by using two-axis auxiliary lines on the position display screen as a reference, the visual aid effect is enhanced, making it easier to intuitively grasp the position information of the operation unit and hand unit.

[0012] When the hand unit is in an initial position, the virtual position display of the operation unit and the virtual position display of the hand unit are displayed overlapping each other in the same position on the position display screen. According to this feature, it becomes easier to compare the position of the virtual operation part and the position of the virtual hand part on the position display screen with reference to the initial position of the hand part.

[0013] A magnification display showing a conversion magnification of the amount of movement of the hand section relative to the amount of displacement of the operation section is displayed near the position display screen on the work screen. According to this feature, it becomes easier to grasp the difference in the amount of movement of the hand unit relative to the amount of displacement of the operation unit. [Brief description of the drawings]

[0014] [Figure 1] 1 is a schematic diagram showing a configuration of a work system according to an embodiment of the present invention; [Diagram 2] FIG. 2A is a diagram showing an outline of the structure of a grip operation unit of a master device, and FIG. 2B is a diagram showing an outline of the structure of a hand unit of a slave device. [Diagram 3] FIG. 13 is a diagram showing a work screen displayed on a master display in the first embodiment. [Figure 4] 4 is a diagram showing a state in which a hand unit is in an initial position on a position display screen displayed as a small window on the work screen of FIG. 3. FIG. [Diagram 5] 4 is a diagram showing a state in which a hand unit has been moved from an initial position on a position display screen displayed as a small window on the work screen of FIG. 3. FIG. [Figure 6] FIG. 4 is a diagram showing a scale selector displayed in a small window on the work screen of FIG. 3. [Figure 7] Fig. 7 is a diagram showing a work screen displayed on a display on the master side in Example 2. Fig. 7 is a diagram showing a state in which the hand unit is in an initial position. [Figure 8] Fig. 8 is a diagram showing a work screen displayed on the master side display in Example 2. Fig. 8 is a diagram showing a state in which the hand unit has moved from the initial position. [Figure 9] FIG. 13 is a diagram showing a real-time image of an object and a hand part of a slave device displayed on a master-side display in the third embodiment. [Figure 10] FIG. 13 is a diagram showing an outline of the procedure of a verification experiment in Example 3. [Figure 11] FIG. 13 is a graph showing the scores of the subjective evaluation for each operator in the (1) gripping step in the pick-and-place step in the verification experiment in Example 3. [Figure 12] FIG. 13 is a graph showing the scores of the subjective evaluations given to each operator in the (2) lifting step in the pick-and-place step in the verification experiment in Example 3, as a standard deviation graph. [Figure 13] FIG. 13 is a graph showing the scores of the subjective evaluations given to each operator in the (3) placing step in the pick-and-place step in the verification experiment in Example 3, as a standard deviation graph. [Figure 14] FIG. 13 is a graph showing the scores of the subjective evaluations given by each operator in the (4) arranging step in the pick-and-place step in the verification experiment in Example 3. [Figure 15] FIG. 13 is a graph showing the scores of the subjective evaluation for each operator in the step of pushing in the cushioning material (5) in the verification experiment in Example 3, as a standard deviation graph. [Figure 16] FIG. 13 is a graph showing the scores of the subjective evaluation in the (1) gripping step in the pick-and-place step as a standard deviation graph for each operator in the verification experiment in Example 4. [Figure 17] FIG. 13 is a graph showing the scores of the subjective evaluations given to each operator in the (2) lifting step in the pick-and-place step in the verification experiment in Example 4, as a standard deviation graph. [Figure 18] FIG. 13 is a graph showing the scores of the subjective evaluations for each operator in the (3) placing step in the pick-and-place step in the verification experiment in Example 4. [Figure 19]FIG. 13 is a graph showing the scores of the subjective evaluation responses in the (4) arranging step in the pick-and-place step as a standard deviation graph for each operator in the verification experiment in Example 4. [Figure 20] FIG. 13 is a graph showing the scores of the subjective evaluation for each operator in the step of pushing in the cushioning material (5) in the verification experiment in Example 4, as a standard deviation graph. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A mode for carrying out a work system according to the present invention will be described below based on an embodiment. EXAMPLES

[0016] First Embodiment A working system according to a first embodiment will be described with reference to FIGS.

[0017] The work system of this embodiment is a robot picking system for performing pick-and-place processes, and is configured as a work system that combines, for example, autonomous picking technology using a work robot equipped with AI, and remote operation technology in which an operator who receives a request for support remotely operates the work robot to support the picking process when it is difficult for the work robot to complete the picking process alone.

[0018] The configuration of the remote control technology of the working system of this embodiment will be described with reference to Fig. 1. As shown in Fig. 1, the working system of this embodiment is a master-slave type working system, and is mainly composed of control devices 1, 2, and 3, a master device 4, a slave device 5, an operation unit 6 of the master device 4, a hand unit 7 of the slave device 5, a display 8 (display means), and a streaming camera 9.

[0019] In this embodiment, the slave side device 5 is a working robot with a multi-joint arm structure, and a two-fingered robot hand having a pair of gripping pieces 70, 71 (see FIG. 2(b)) as a hand unit 7 is attached to the tip of the arm unit 50. The master side device 4 is a working robot with a multi-joint arm structure of the same structure and size as the slave side device 5, and an interface device having a pair of scissor-handle shaped operating pieces 60, 61 (see FIG. 2(a)) as an operating unit 6 is attached to the tip of the arm unit 40.

[0020] In this embodiment, the working robot with a multi-joint arm structure has six axes, but the number of axes is not limited to six. In the following explanation, the position of the hand is important in this embodiment, and the amount of movement and rotation of each joint axis is not important, so the explanation will be omitted and the amount of movement on the x-axis, y-axis along the horizontal plane, and the z-axis along the vertical direction that define the three-dimensional coordinate system will be explained (see FIG. 3).

[0021] 1, the control device 1 includes a motion control unit and a force feedback control unit. In addition, a control device 2 including a motion generation unit and a control device 3 including a video streaming unit are connected to the control device 1. In the working system of this embodiment, a master device 4 and a slave device 5 are connected via the control device 1, so that an operation of an operation unit 6 of the master device 4 by an operator is reproduced in real time as an operation of a hand unit 7 of the slave device 5. In detail, the amount of displacement and force of the operation unit 6 of the master side device 4 by the operator corresponds to the amount of movement and force of the hand unit 7 of the slave side device 5. For example, when the operator holds the handle portion 62 (see FIG. 2(a)) of the operation unit 6 of the master side device 4 with his / her right hand and places his / her thumb and index finger respectively through a pair of scissor-handle shaped operation pieces 60, 61 (see FIG. 2(a)), the operation unit 6 is displaced in three axial directions (x-axis, y-axis, z-axis) and the operation pieces 60, 61 are moved to perform a grasping operation to grasp an object. In this state, the hand unit 7 of the slave side device 5 can be moved in three axial directions and the pair of grasping pieces 70, 71 can be opened and closed to grasp the object.

[0022] In addition, the operation unit 6 and operation pieces 60, 61 of the master side device 4 are fed back, as "force sense" information, by processing in the force sense feedback control unit, external forces such as being pulled or pushed when the hand unit 7 comes into contact with an object, and the reaction force when an object is grasped by the grasping pieces 70, 71 of the hand unit 7.

[0023] More specifically, in this embodiment, the control device 1 equipped with a force feedback control unit acquires the external force acting on the inside of the gripping pieces 70, 71, i.e., the inner surfaces 70d, 71d (see FIG. 2(b)) of the gripping pieces 70, 71, and feeds it back as a force sense to the operation unit 6 of the master side device 4. Note that the detection of the external force acting on the inside of the gripping pieces 70, 71 is performed by the hand unit 7 or the arm unit 50 of the slave side device 5.

[0024] The control device 1 makes the motion control unit and the force feedback control unit cooperate to control the force required for opening and closing the operation pieces 60, 61 of the operation unit 6 of the master side device 4, thereby feeding back a reaction force to the operation unit 6. This allows the operator to sense information about the inner surfaces 70d, 71d of the gripping pieces 70, 71 gripping an object, that is, information about the shape and hardness of the object, the strength of the grip, and the like, from the operation unit 6.

[0025] When the control device 1 receives an external force from either of the inner surfaces 70d, 71d of the gripping pieces 70, 71 of the hand unit 7, the control device 1 feeds back the reaction force to one of the operation pieces 60, 61 of the operation unit 6, that is, the operation piece 61 through which the index finger is passed in this embodiment. This allows the master side device 4 to be simplified.

[0026] Furthermore, the control device 1 makes the operation control unit and the force feedback control unit cooperate with each other to control the force required for the displacement of the operation unit 6 of the master side device 4, thereby feeding back a reaction force to the operation unit 6. That is, when an external force of a predetermined magnitude or greater is applied to the inner surfaces 70d, 71d of the gripping pieces 70, 71 of the hand unit 7, the control device 1 controls the force required for the operation of the arm unit 40 of the master side device 4 to be large, thereby restricting or stopping the movement of the hand unit 7, thereby preventing a strong collision between the hand unit 7 and the target object or surrounding objects.

[0027] In addition, the control device 1 acquires the external force acting on the outside of the grip pieces 70, 71, i.e., the external force acting on the outer surfaces 70a, 71a (see FIG. 2(b)) of the grip pieces 70, 71, and feeds it back as a force sense to the operation unit 6 of the master side device 4. The detection of the external force acting on the outside of the grip pieces 70, 71 is performed by the hand unit 7.

[0028] In addition, the control device 1 acquires the external forces acting on the side surfaces 70b, 71b and 70c, 71c (see FIG. 2(b)) of the gripping pieces 70, 71, and feeds them back as a force sense to the operation unit 6 of the master side device 4. The external forces acting on the side surfaces of the gripping pieces 70, 71 are detected by the arm unit 50 of the slave side device 5.

[0029] The control device 1 makes the operation control unit and the force feedback control unit cooperate with each other to control the operation of the arm unit 40 and the hand unit 7 of the master side device 4, thereby feeding back the force sense to the operation unit 6. This allows the operator to sense from the operation unit 6 information that the outer surfaces 70a, 71a and both side surfaces 70b, 70c, 71b, 71c of the gripping pieces 70, 71 come into contact with an object or surrounding objects.

[0030] In addition, the control device 1 acquires the external force acting on the tips (lower ends) 70e, 71e (see FIG. 2(b)) of the gripping pieces 70, 71, and feeds it back as a force sense to the operation unit 6 of the master side device 4. The external force acting on the tips of the gripping pieces 70, 71 is detected by the arm unit 50 of the slave side device 5.

[0031] The control device 1 makes the motion control unit and the force feedback control unit cooperate with each other to control the motion of the arm unit 40 of the master side device 4, thereby feeding back the force sense to the operation unit 6. This enables the operator to sense from the operation unit 6 information that the tips 70e, 71e of the gripping pieces 70, 71 have come into contact with an object or a surrounding object.

[0032] In addition, the working system of this embodiment detects the external force acting on the hand unit 7 from the load torque, rotation speed, current value, etc. of each drive motor built into the arm unit 50 of the slave side device 5 and the hand unit 7, so that it is possible to obtain the external force at any position of the hand unit 7 without using a large number of sensors while simplifying the structure of the hand unit 7.

[0033] The control device 1 may also be capable of adjusting a reaction force fed back to the operation unit 6 against an external force acting on inner surfaces 70d, 71d of the gripping pieces 70, 71 of the hand unit 7, depending on the degree of opening and closing of the gripping pieces 70, 71 of the hand unit 7, i.e., the state of the gripping operation of the pair of scissor-handle-shaped operation pieces 60, 61 in the operation unit 6. Furthermore, the reference value of the degree of opening and closing of the gripping pieces 70, 71 related to the adjustment of the reaction force may be manually or automatically changeable depending on the size of the target object to be subjected to the pick-and-place process.

[0034] The control device 3 includes a video streaming unit. In the working system of this embodiment, a display 8 provided on the master side and a streaming camera 9 provided on the slave side are connected via the control device 3, so that a real-time image captured by the streaming camera 9 is displayed on the display 8 as a working screen 80 (see FIG. 3), and an operator can remotely operate the hand unit 7 of the slave side device 5 by using the operation unit 6 of the master side device 4 while watching the working screen 80. In addition, the control device 3 is connected to the control device 1, so that the control device 3 can display a position display screen 100 and a scale selector 101, which will be described later, and switch the conversion magnification on the working screen 80.

[0035] In this embodiment, the streaming camera 9 is provided at a location different from the hand unit 7 of the slave device 5, and captures the positional relationship between the target object and the work area of ​​the hand unit 7 of the slave device 5 in the pick-and-place process from diagonally above. As a result, the two-dimensional work screen 80 displayed on the display 8 makes it easy to visually grasp three-dimensional positional information of the target object and the hand unit 7, that is, positional information on three axes (x-axis, y-axis, z-axis).

[0036] As shown in Fig. 3, the work screen 80 displays a small window of a position display screen 100 in the upper right corner of the screen, which shows the work area in a simple manner. In this embodiment, the position display screen 100 reproduces the arrangement and outline (silhouette) of the containers 10A and 10B, the plastic bottles P1 and P2 and the box-shaped objects B1, B2 and B3 contained in the container 10A, and the plastic bottle P3, the box-shaped object B4 and the cushioning material C contained in the container 10B when the work area is viewed from above (see Fig. 4). In this embodiment, the position display screen 100 shows the work area in a plan view of the xy axes, but is not limited to this and may show a three-dimensional view with the z axis added.

[0037] 4, auxiliary lines X and Y (see dashed lines) of two perpendicular axes are displayed on the position display screen 100. The point (origin) where the auxiliary lines X and Y intersect corresponds to the initial position (home position) of the hand unit 7 of the slave device 5. In this embodiment, the position of the operation unit 6 of the master device 4 on the work screen 80 in FIG. 3 corresponds to the initial position of the hand unit 7 of the slave device 5.

[0038] The initial position of the operation unit 6 of the master side device 4 is not limited to being fixedly set in advance based on the structure of the master side device 4, but may be selectively set to a position where the operator can easily operate the operation unit 6 or the position where the previous work was completed as the initial position.

[0039] In this embodiment, the position display screen 100 simultaneously displays a master-side virtual position display M (virtual position display) reflecting position information of the operation unit 6 of the master-side device 4 in real time, and a slave-side virtual position display S (virtual position display) reflecting position information of the hand unit 7 of the slave-side device 5 in real time. Note that Fig. 4 shows a state in which the hand unit 7 is in an initial position, and the master-side virtual position display M and the slave-side virtual position display S are displayed with their centers overlapping at the same position on the position display screen 100, that is, the point (origin) where the auxiliary lines X and Y intersect at right angles.

[0040] In addition, in this embodiment, the master side virtual position display M and the slave side virtual position display S are represented by circles of different diameters, but this is not limited to this and they may be represented by other shapes such as polygons or points, and the master side virtual position display M and the slave side virtual position display S may be represented by different shapes.

[0041] Furthermore, in this embodiment, the diameter of the slave side virtual position display S in particular corresponds to the maximum opening angle of a pair of gripping pieces in the hand portion 7 of the slave side device 5, allowing the operator to intuitively align the object to perform a gripping operation.

[0042] In addition, in this embodiment, the master side virtual position display M and the slave side virtual position display S are represented by circles of different line types (the master side virtual position display M is a solid line, and the slave side virtual position display S is a dotted line) for the sake of convenience of explanation, but in an actual display 8, it is preferable that the master side virtual position display M and the slave side virtual position display S displayed on the position display screen 100 have different display styles such as color, transparency (transparency), line thickness, line type, etc., so that the operator can easily distinguish them.

[0043] The working system of this embodiment is capable of converting the displacement amount and force of the hand unit 7 of the slave device 5, which are output as a motion, into the displacement amount and force of the operation unit 6 of the master device 4, which are input to the working system, by output processing by the motion control unit of the control device 1 and the motion generation unit of the control device 2. Furthermore, the working system of this embodiment is capable of converting the displacement amount and force of the three axes of the hand unit 7 into the displacement amount and force of the three axes input by operating the operation unit 6, at different magnifications for each axis. In this embodiment, the two axes of left and right and front and back corresponding to the alignment direction of the hand unit 7, i.e., the two axes of the x-axis and y-axis (see FIG. 3), are converted at the same magnification (3 times).

[0044] Furthermore, in the working system of this embodiment, the output processing by the motion control unit of the control device 1 and the motion generation unit of the control device 2 can convert the amount of gripping operation of the pair of gripping pieces 70, 71 of the hand unit 7 of the slave device 5, which is output as an operation, into the amount of gripping operation of the pair of scissor-handle-shaped operation pieces 60, 61 of the operation unit 6 of the master device 4, which is input to the working system, separately from the conversion of the amount of movement and force of the hand unit 7 relative to the amount of displacement and force of the operation unit 6. Note that the conversion ratio of the gripping operation amount to the gripping operation amount may be freely set according to the difference in size and structure between the master device 4 and the slave device 5, but is preferably 1 to 2 times from the viewpoint of improving the motion accuracy of the gripping operation in the pick-and-place process.

[0045] 5, on the position display screen 100, the master side virtual position display M shows real-time position information of the operation unit 6 that has moved from the initial position, while the slave side virtual position display S showing the position information of the hand unit 7 is displayed at a position three times the master side virtual position display M. In other words, when the center of the master side virtual position display M is the end point of a vector starting from the initial position (origin), the center of the slave side virtual position display S is displayed at the end point of a vector in which the amount of movement from the start point in the x direction and y direction is tripled. Note that if the conversion magnification is 1x, the master side virtual position display M and the slave side virtual position display S move with their centers always overlapping.

[0046] Furthermore, the working system of the present invention is configured to be able to switch the conversion ratio of the movement amount of the hand unit 7 of the slave side device 5 to the displacement amount of the operation unit 6 of the master side device 4, and thereby it is possible to switch the conversion ratio of the movement amount of the hand unit 7 to the displacement amount of the operation unit 6 to make it easier for the operator to operate depending on the differences in size and structure between the master side device 4 and the slave side device 5, thereby improving workability.

[0047] Specifically, as shown in Fig. 3, a scale selector 101 (magnification display) showing the conversion magnification of the movement amount of the hand unit 7 relative to the displacement amount of the operation unit 6 is displayed near (below) the position display screen 100 on the work screen 80. As shown in Fig. 6, in this embodiment, the scale selector 101 allows selective switching of the conversion magnification of the displacements of the x, y and z axes, and the color of the button for the conversion magnification selected by the scale selector 101 (3x in this embodiment) is changed, thereby displaying the conversion magnification in a form that is easy for the operator to recognize.

[0048] Furthermore, when the conversion magnification is switched by the scale selector 101, it goes without saying that on the position display screen 100, the slave side virtual position display S is displayed at a position according to the conversion magnification relative to the master side virtual position display M which has moved from the initial position.

[0049] As described above, in the working system of the present invention, the position display screen 100, which simply represents the working area, is displayed in a small window on the working screen 80 on the master side, and the master side virtual position display M, which reflects the position information of the operating unit 6 in real time, and the slave side virtual position display S, which reflects the position information of the hand unit 7 in real time, are simultaneously displayed on the position display screen 100. This allows the position of the virtual operating unit 6 and the position of the virtual hand unit 7 to be compared on the position display screen 100, making it easy to adapt the operational feeling of operating the operating unit 6 to the movement of the actual hand unit 7. In addition, it is easy to grasp the difference in the movement amount of the hand unit 7 relative to the displacement amount of the operating unit 6, that is, the conversion magnification. Therefore, especially for operators with low levels of skill, the effect of assisting spatial recognition is high, and the sense of incongruity between the operational feeling of operating the operating unit 6 and the movement of the hand unit 7 is eliminated, and the time required to become familiar with remote operation can be shortened. Furthermore, the time required for the operator to complete the pick-and-place process is also shortened, improving workability. Even for highly skilled operators, the position display screen 100 is highly effective in assisting spatial recognition, and workability can be improved.

[0050] Moreover, the position of the virtual operation unit 6 and the position of the virtual hand unit 7 can be confirmed from the master-side virtual position display M and the slave-side virtual position display S displayed on the position display screen 100, which shows a simplified representation of the working area, making it easy to grasp the positional information of the hand unit 7 in the working area. In this embodiment, the position display screen 100 is configured as a plan view of the xy axes, thereby enhancing the visual aid effect of the plane (xy axes).

[0051] Furthermore, by using the two-axis auxiliary lines X and Y displayed on the position display screen 100 as a reference, the visual auxiliary effect of the plane (xy axes) is further enhanced, making it easier to intuitively grasp the position information of the hand unit 7.

[0052] Furthermore, when the hand unit 7 is in its initial position, the master side virtual position display M and the slave side virtual position display S are displayed at the same position on the position display screen 100, i.e., overlapping on the point (origin) where the auxiliary lines X and Y intersect perpendicularly, making it easier to compare the position of the virtual operation unit 6 and the position of the virtual hand unit 7 on the position display screen 100 based on the initial position of the hand unit 7.

[0053] In addition, by sequentially checking the positions of the master side virtual position display M and the slave side virtual position display S displayed on the position display screen 100, the displacement in the movement of the master side virtual position display M and the slave side virtual position display S can be grasped.

[0054] Furthermore, a scale selector 101 is displayed near the position display screen 100 on the work screen 80, making it easier to grasp the conversion magnification, that is, the difference between the amount of movement of the hand unit 7 relative to the amount of displacement of the operation unit 6. EXAMPLES

[0055] Next, a working system according to a second embodiment will be described with reference to Fig. 7 and Fig. 8. Note that a description of the same configuration as in the previous embodiment will be omitted.

[0056] In the work system of this embodiment, a master-side virtual position display M reflecting the position information of the operation unit 6 of the master-side device 4 in real time is displayed superimposed (overlay display) on a work screen 80 (see FIG. 7) shown on the display 8 (not shown). That is, in the work screen 80 of FIG. 7 showing a state in which the hand unit 7 is in the initial position, the master-side virtual position display M in the initial position is displayed superimposed just below the center of the hand unit 7. In this embodiment, a scale selector 101 is displayed in a small window in the upper right corner of the work screen 80.

[0057] According to this, by displaying the master side virtual position display M superimposed on the work screen 80, as shown in FIG. 8, when the operator operates the operation unit 6 of the master side device 4, the operator can check the position of the virtual operation unit 6 displayed on the work screen 80, and by comparing it with the movement movement of the hand unit 7 of the slave side device 5, it is easy to adapt the operational feel of operating the operation unit 6 to the movement movement of the actual hand unit 7, making it possible to quickly become familiar with remote operation regardless of the operator's level of skill.

[0058] In this embodiment, the work screen 80 may display a small window containing the position display screen 100 together with the scale selector 101 as in the first embodiment. EXAMPLES

[0059] Next, a working system according to a third embodiment will be described with reference to Fig. 9 to Fig. 15. Note that a description of the same configuration as in the previous embodiment will be omitted.

[0060] In this embodiment, a verification experiment will be described in which the displacement amounts and forces of three axes (x-axis, y-axis, z-axis) in the operation of the operation unit 6 of the master device 4 by the operator are converted under various conditions into the movement amounts and forces of three axes (x-axis, y-axis, z-axis) in the operation of the hand unit 7 of the slave device 5. Note that in this embodiment, the two axes of left and right and front and back corresponding to the alignment directions of the hand unit 7, i.e., the two axes of the x-axis and y-axis, are converted at the same magnification.

[0061] In addition to the displacement amounts and forces of the above three axes (x-axis, y-axis, z-axis), this embodiment also converts the amount of rotation and force of the rotation r-axis about the z-axis input based on the angle of the wrist orientation of the operator operating the operation unit 6 into the amount of rotation and force of the rotation r-axis output as the operation of the hand unit 7.

[0062] The following describes the work content of the pick-and-place process performed by the work system in the verification experiment. As shown in Fig. 9, two box-shaped containers 10 and 11 with their upper sides open are arranged side by side, and an object such as a plastic bottle placed in the left container 10 is moved to the right container 11. In this verification experiment, the left and right dimensions of the empty space in the container 11 are set to be slightly larger than the object, the plastic bottle, due to the box-shaped object B placed in advance in the container 11.

[0063] The operator operates the operation unit 6 of the master side device 4 to operate the hand unit 7 of the slave side device 5, and moves the plastic bottles P1 and P2 and the cushioning material C placed in the container 10 to an empty space in the container 11, as shown in FIG. 9. In detail, first, the plastic bottle P1 is picked up from inside the container 10, transported to an upper position of the empty space in the container 11, and the plastic bottle P1 is lowered into the empty space in the container 11 (see FIG. 10(a)). Next, the plastic bottle P2 is picked up from inside the container 10, transported to an upper position of the empty space in the container 11, and the plastic bottle P2 is lowered into the empty space in the container 11 (see FIG. 10(b)). Next, the plastic bottles P1 and P2 are pushed toward the inner wall of the container 11 on the front side in the real-time image displayed on the display 8, and aligned so as to close the gap (see FIG. 10(c)). Finally, the cushioning material C is picked from inside the container 10, transported to a position above the empty space in the container 11, and pushed into the gap in the empty space of the container 11, and the shape of the cushioning material C is adjusted (see Figure 10(d)), thereby completing the pick-and-place process.

[0064] During the above-mentioned pick-and-place process, experimental conditions (see Table 1) were set in which the displacement, rotation and force input of the operation unit 6 of the master device 4 was output as the movement, rotation and force of the hand unit 7 of the slave device 5 at various conversion magnifications, and objective and subjective evaluations were performed for each experimental condition.

[0065] [Table 1]

[0066] Each experimental condition is explained using Table 1. As shown in Table 1, in experimental conditions 1 to 4, experiments were conducted in which the displacement amount, rotation amount, and force were converted at a common conversion magnification (1x, 3x, 5x, 10x). In addition, in experimental conditions 5 to 7, in order to confirm the influence of the conversion of the displacement amount, rotation amount, and force of only the xy axis, z axis, and rotation r axis on the operational feeling, a condition was set in which the xy axis, z axis, and rotation r axis were separated and the magnification was converted. At this time, the conversion magnification was set to 5x. In the following, unless otherwise specified, the conversion magnification is set to 5x. In addition, in experimental conditions 8 to 10, in order to confirm the influence of the condition in which only the xy axis, z axis, and rotation r axis are not converted on the operational feeling, a condition was set in which the xy axis, z axis, and rotation r axis were separated and the magnification was converted. In addition, in experimental condition 11, a condition was set in which all except the displacement of the rotation r axis were converted to 5x.

[0067] As an "objective evaluation" of the verification experiment, the number of seconds (time required) taken for the series of tasks of lining up the PET bottles and pushing the cushioning material into them was measured. In addition, the end state was evaluated on a three-point scale as to whether the pick-and-place process of the PET bottles and the operation of pushing the cushioning material into them were completed or not: ◯: the work was done properly, △: the work was partially done (the bottles were lined up but not lined up properly, the cushioning material was pushed in but not pushed in enough, etc.), ×: the work was not done properly.

[0068] In addition, for the "subjective evaluation," operators were asked to complete a questionnaire to evaluate the four indicators of ease / lightness / simple / intuitiveness of each of the five steps in the pick-and-place process of PET bottles ((1) Grab → (2) Lift → (3) Place → (4) Arrange) and (5) Pushing in the cushioning material, with 10 being the highest and 0 being the lowest.

[0069] The verification experiment was conducted by operator A, who is familiar with remote operation (with at least one year of operation experience), and operator B, who has never performed remote operation. The results of the following verification experiment (objective evaluation, subjective evaluation) show the average values ​​of the results of performing tasks 3 to 10 times for each experimental condition.

[0070] The results of the objective evaluation (time required, achievement evaluation) are shown in Table 2.

[0071] [Table 2]

[0072] The results of the subjective evaluation (answer scores) are shown in Tables 3 to 7. Furthermore, the scores were converted into standard deviation graphs for each operator, which are shown in Figures 11 to 15. The reason for converting the scores into standard deviations is to make it easier to make uniform comparisons by eliminating biases between work processes and biases in the perceptions of individual operators.

[0073] [Table 3]

[0074] [Table 4]

[0075] [Table 5]

[0076] [Table 6]

[0077] [Table 7]

[0078] [Effect of displacement (translation and rotation) and force conversion magnification] As shown in Table 2, when comparing the required time, which is the result of objective evaluation, for experimental conditions 1 to 4 in which displacement (movement and rotation) and force are converted to 1x, 3x, 5x, and 10x with the same magnification, for operator A, who is an expert in remote operation, the required time increased as the magnification increased. Furthermore, for operator B, who is a novice in remote operation, there was no clear correlation between the magnification and the required time, and the required time was more than doubled especially when the magnification was 10x. Furthermore, in experimental condition 4, where the magnification was 10x, both operators A and B were unable to perform the pick-and-place process accurately.

[0079] As shown in Tables 3 to 6, for operator A, when the magnification of displacement and force was changed in each of steps (1) to (4) in the pick-and-place process, the answer scores for the indicators of easy, simple, and intuitive dropped significantly. Also, for the indicator of light, the answer score increased as the magnification increased in step (4) of arranging (see Table 6). Also, in step (5) of pushing in the cushioning material (see Table 7), the answer scores for the indicators of easy and light increased up to a magnification of 5 times, but the answer scores for the indicators of easy and intuitive decreased as the magnification increased.

[0080] For operator B, when the magnification of displacement and force was changed in the pick-and-place process, in (1) the gripping process (see Table 3) and (2) the lifting process (see Table 4), the answer scores for the indicators of easy and light were high up to a magnification of 5, but the answer scores for the indicators of easy and intuitive became lower as the magnification increased. Also, in (5) the process of pushing in the cushioning material (see Table 7), the answer scores for the indicators of easy and light were high up to a magnification of 5.

[0081] In this way, by increasing the displacement and force conversion ratio, the operator is assumed to feel that the amount of movement required for operation is reduced, making it lighter and easier. On the other hand, the amount of operation of the operation unit 6 of the master device 4 and the amount of movement of the hand unit 7 of the slave device 5, the distance between the target object and the hand unit 7, and the amount of force applied are not intuitive, making the operation difficult.

[0082] [Separate the axis to be transformed] As shown in Table 2, in the experimental conditions 5 to 10, in which only one or two of the three axes of the xy-axis, z-axis, and rotation r-axis, which are the motion axes of the hand unit 7 of the slave side device 5, were separated and the magnification of the displacement and force was converted to 5 times, both operators A and B required longer time to perform the task than in the experimental conditions 1 to 3, in which the magnification of the displacement and force of the three axes of the xy-axis, z-axis, and rotation r-axis were all converted to 1 to 5 times. In particular, in the experimental condition 7, in which only the xy-axis was converted to 5 times, and in the experimental condition 8, in which only the z-axis and r-axis were converted to 5 times, the operation was difficult, and both operators A and B could not complete the task in the experimental condition 7, and the operator B could not complete the task in the experimental condition 8. This is because the slave side device 5 moved too much compared to the operator's intention in the operation of aligning the plastic bottles in the process of (4) arranging and the operation of the process of pushing in the cushioning material in the pick-and-place process, and the operation was not accurate.

[0083] Here, for experimental condition 6, in which only the z-axis was converted by a factor of 5, the task was completed in a relatively short time. Converting only the z-axis direction has the effect of supplementing the movable area of ​​the master device, and may be effective in improving workability.

[0084] In addition, as shown in Tables 3 to 7 and Figures 11 to 15, for both operators A and B, in each of steps (1) to (4) in the PET bottle pick-and-place process (see Tables 3 to 6), experimental condition 6, in which only the z-axis was converted, tended to have high response scores for the indices of easy, simple, and intuitive, and it is presumed that the condition of converting the z-axis leads to ease of operation. Similar trends were also confirmed for step (5) of pushing in the cushioning material (see Table 7).

[0085] As shown in Table 2, in experimental condition 11, in which all values ​​except for the r-axis displacement (amount of rotation) were converted by a factor of 5, the time required for the task was shorter than in experimental conditions 5 to 10 above.

[0086] Also, as shown in Tables 3 to 7 and Figures 11 to 15, for both operators A and B, experimental condition 11, in which all displacements except for the r-axis were converted by a factor of 5 in each step of the PET bottle pick-and-place process (see Tables 3 to 6) and the cushioning material pushing process (see Table 7), had high response scores for the indices of easy, light, simple, and intuitive. In other words, it is presumed that the most intuitive and easiest to operate is to keep the displacement (amount of rotation) of the rotational r-axis, which is the angle of the wrist orientation of the operator operating the operation unit 6 of the master device 4, unchanged (1x) without conversion.

[0087] In addition, in the experimental condition 7 where only the xy axes were converted, it was confirmed that the hand unit 7 of the slave side device 5 shook too much from side to side during the operation. In this way, when the magnification of only the xy axes is large, it is not intuitive and makes delicate movements difficult, and was not suitable for the operation of this embodiment.

[0088] For these reasons, when separating and converting the three axes of motion of the hand section 7 of the slave device 5, the xy-axis, z-axis, and rotational r-axis, in the work of this embodiment, experimental condition 6, in which only the z-axis is converted by a factor of 5, and experimental condition 11, in which all displacements except for the r-axis are converted by a factor of 5, received high subjective evaluations and were effective for the work of this embodiment.

[0089] In addition, the work of this embodiment does not require strict alignment for the (3) placing step and (5) pushing in the cushioning material in the pick-and-place process, and does not require delicate movements, but only movements in the z-axis direction, i.e., the pick direction, while the (1) gripping step, (4) arranging step, and (5) pushing in the cushioning material in the pick-and-place process require alignment of the hand unit 7 on the slave side with the object, and delicate movements in the xy-axis direction, i.e., the alignment direction, are required. That is, by converting to amplify the displacement in the z-axis direction (pick direction), which does not require delicate movements, even if the displacement amount of the operation unit 6 of the master side device 4 is small, the movement amount of the hand unit 7 of the slave side device 5 is increased, leading to an improvement in the work speed. On the other hand, for the xy-axis direction (alignment direction), which requires delicate movements, by not converting the displacement, delicate movements in which the displacement amount of the operation unit 6 of the master side device 4 and the movement amount of the hand unit 7 of the slave side device 5 are the same are possible, and careful work can be performed. These conditions matched the work conditions of this example, and it is presumed that this resulted in high scores in the subjective evaluation, particularly for the ease and simplicity indicators.

[0090] In this way, at first glance, by separating the three axes of motion of the hand unit 7 of the slave device 5 operated by the operation unit 6 of the master device 4, that is, the x-y axis, the z-axis, and the rotational r-axis, the hand unit 7 of the slave device 5 operates with a different conversion magnification for each axis, which may make it appear that operation is not intuitive. However, by skillfully matching the work tasks and conditions, it is expected that benefits that outweigh this will arise and the operability may be improved.

[0091] As described above, from the results of the verification experiments, in the work system of the present invention, the amount of movement of the hand unit 7 of the slave side device 5 relative to the displacement amount of the operation unit 6 of the master side device 4 is converted to be larger for one or two of the three axes than the amount of movement of the other operation axes, so that in the pick-and-place process, the amount of movement of the operation axis (z-axis) of the hand unit 7 that is mainly related to work that does not require delicate movement of the hand unit 7 is converted to be larger than the amount of movement of the other operation axes (x-axis, y-axis) of the hand unit 7 that is mainly related to work that requires delicate movement of the hand unit 7, relative to the displacement amount of the operation unit 6, thereby reducing the operation area of ​​the operation unit 6, assisting in the operation of the operation unit 6, and improving workability.

[0092] In addition, in the pick-and-place process, the picking of an object by the hand unit 7 of the slave side device 5 does not require delicate movements, so the amount of movement of one axis of the hand unit 7 corresponding to the pick direction (z-axis direction) is greatly converted relative to the amount of displacement of the operation unit 6, thereby assisting the operation of the operation unit 6 and improving workability.

[0093] Furthermore, in the pick-and-place process, delicate operations are required for the work of moving the hand unit 7 of the slave side device 5 in two axial directions relative to the target object for alignment, so the amount of movement of one axis (z-axis) of the hand unit 7 corresponding to the pick direction is converted at a larger magnification with respect to the displacement amount of the operation unit 6 compared to the amount of movement of the other two axes (x-axis, y-axis) of the hand unit 7 corresponding to the alignment direction, and the amount of movement of the other two axes of the hand unit 7 corresponding to the alignment direction is converted at the same magnification, thereby enabling intuitive alignment by operating the operation unit 6 and improving workability. Note that the amount of movement of the other two axes (x-axis, y-axis) of the hand unit 7 corresponding to the alignment direction is 1x (same magnification), i.e., the magnification is not converted, making it easier to perform delicate operations of the hand unit 7.

[0094] Furthermore, the working system of the present invention is configured to be able to switch the conversion magnification of the movement amount of the hand unit 7 of the slave side device 5 relative to the displacement amount of the operation unit 6 of the master side device 4, thereby improving workability by switching the conversion magnification of the movement amount of the hand unit 7 relative to the displacement amount of the operation unit 6 so that the operator can easily perform the operation in accordance with differences in size and structure between the master side device 4 and the slave side device 5. Although not shown in the figures for ease of explanation, the working system of the present invention may be able to selectively switch not only the conversion magnification of the displacements of the x, y and z axes, but also the conversion magnification of the force.

[0095] Furthermore, the operator may be able to selectively switch the conversion ratio of the amount of gripping operation of the grip pieces 70, 71 in the hand unit 7 to the amount of gripping operation of the operation pieces 60, 61 in the operation unit 6. This makes it possible to optimize the positioning operation of the hand unit 7 and the gripping operation of the grip pieces 70, 71 in accordance with differences in structure and size between the master device 4 and the slave device 5.

[0096] In addition, in the working system of the present invention, it is preferable to select an optimal conversion magnification in accordance with the usage situation and the work task. For example, when the hand unit 7 of the slave side device 5 is moved from a position far from the target position, the conversion magnification of the displacement of the three axes (x-axis, y-axis, z-axis) is automatically increased so that the hand unit 7 of the slave side device 5 can efficiently move a long distance even if the amount of displacement caused by the operation of the operation unit 6 of the master side device 4 is small, and the conversion magnification of the displacement of the three axes (x-axis, y-axis, z-axis) is automatically switched to the conversion magnification optimized for the pick-and-place process as described above near the target position where the target object is placed, that is, in the operating area of ​​the hand unit 7 where the pick-and-place process of the target object is performed, so that delicate operation can be performed.

[0097] The automatic conversion magnification is switched by detecting, as a trigger (sensing information), that the hand unit 7 has approached the target object during the process in which the hand unit 7 goes to grab the target object for the pick-and-place process. Examples of triggers include the operation unit 6 being located at a position (three-dimensional coordinates) beyond the boundary of the operation area of ​​the operation unit 6 of the master side device 4 that is preset corresponding to the working area in which the pick-and-place process is performed, and the operation speed of the operation unit 6 reaching a predetermined speed or higher. Also, the distance between the target object and the hand unit 7 within a predetermined distance may be detected as a trigger by image analysis of real-time video captured by the streaming camera 9.

[0098] Furthermore, in a robot picking system such as the work system of the present invention, since the master device is often smaller in size and output than the slave device, the workability of the pick-and-place process can be further improved by making it possible to switch not only the displacement conversion magnification but also the force conversion magnification so that better operability can be felt on the operating unit of the master device. EXAMPLES

[0099] Next, a working system according to a fourth embodiment will be described with reference to Fig. 16 to Fig. 20. Note that a description of the same configuration as in the above embodiment will be omitted.

[0100] In this embodiment, a verification experiment for investigating the effect of force feedback on workability will be described.

[0101] In the verification experiment of this embodiment, the work content of the pick-and-place process performed by the work system is similar to that of the third embodiment, and therefore a detailed description thereof will be omitted.

[0102] During the above pick-and-place process, objective and subjective evaluations were performed under experimental condition 12 with force feedback and experimental condition 13 without force feedback.

[0103] As an "objective evaluation" of the verification experiment, the number of seconds (time required) taken for the series of tasks of lining up the PET bottles and pushing the cushioning material into them was measured. In addition, the end state was evaluated on a three-point scale as to whether the pick-and-place process of the PET bottles and the operation of pushing the cushioning material into them were completed or not: ◯: the work was done properly, △: the work was partially done (the bottles were lined up but not lined up properly, the cushioning material was pushed in but not pushed in enough, etc.), ×: the work was not done properly.

[0104] In addition, for the "subjective evaluation," operators were asked to complete a questionnaire to evaluate the four indicators of ease / lightness / simple / intuitiveness of each of the five steps in the pick-and-place process of PET bottles ((1) Grab → (2) Lift → (3) Place → (4) Arrange) and (5) Pushing in the cushioning material, with 10 being the highest and 0 being the lowest.

[0105] The verification experiment was conducted by operator A, who is familiar with remote operation (with at least one year of operation experience), and operator B, who has never performed remote operation. The results of the following verification experiment (objective evaluation, subjective evaluation) show the average values ​​of the results of performing tasks 3 to 10 times for each experimental condition.

[0106] The results of the objective evaluation (time required, achievement evaluation) are shown in Table 8.

[0107] [Table 8]

[0108] The results of the subjective evaluation (answer scores) are shown in Tables 9 to 13. Furthermore, the scores were converted into standard deviation graphs for each operator, which are shown in Figures 16 to 20. The reason for converting the scores into standard deviations is to make it easier to make uniform comparisons by eliminating biases between work processes and biases in the perceptions of individual operators.

[0109] [Table 9]

[0110] [Table 10]

[0111] [Table 11]

[0112] [Table 12]

[0113] [Table 13]

[0114] [Effect of force feedback] Comparing the results of experimental condition 12 and experimental condition 13 in Table 8, which shows the results of the objective evaluation, operator A, who is an expert in remote operation, required about four times as much time to complete the task under experimental condition 13 (111 s) without force feedback compared to experimental condition 12 (31 s) with force feedback. Operator B, who is a novice in remote operation, was unable to complete the task under experimental condition 13 without force feedback.

[0115] Furthermore, in Table 13, which shows the results of the subjective evaluation of the (5) cushioning material pushing in process in the pick and place process, comparing the results of experimental condition 12 and experimental condition 13, operator A's response scores for the indicators of easy, simple, and intuitive were lower in experimental condition 13 without force feedback compared to experimental condition 12 with force feedback. Note that for the indicator of lightness, the response score was higher in experimental condition 13 without force feedback compared to experimental condition 12 with force feedback. This is presumably because the software specifications for experimental condition 13 did not provide any force feedback at all, making the operation feel lighter.

[0116] In addition, for operator B, in the pick-and-place process (1) the grasping process (see Table 9), the experimental condition 13 without force feedback gave slightly higher scores in the indicators of easy, light, simple, and intuitive compared to experimental condition 12 with force feedback. However, in the other processes (2) the lifting process (see Table 10), (3) the placing process (see Table 11), and (4) the arranging process (see Table 12), the experimental condition 12 with force feedback tended to give higher scores in the indicators of easy, light, simple, and intuitive compared to experimental condition 13 without force feedback.

[0117] In addition, when observing the behavior of the slave side device 5 during work, under experimental condition 13 without force feedback, it was confirmed that both operators A and B had difficulty controlling the amount of force used during the pick-and-place process due to the lack of force feedback, such as knocking over a plastic bottle, pressing the hand portion 7 too hard against the inner wall of the container and shifting the container 10, and pushing in the cushioning material too hard and touching the bottom of the container 10 with the hand portion 7 (tips 70e, 71e of the gripping pieces 70, 71).

[0118] Thus, force feedback was effective from the standpoints of shortening the time required for tasks, improving the operability, protecting the target object, etc. In other words, it was confirmed that force feedback is an essential function for efficiently performing pick-and-place processes and pushing in cushioning materials in the remote operation of a work robot.

[0119] From the results of the verification experiments described above, in the working system of the present invention, the external force acquired by the hand unit 7 of the slave device 5 is fed back as a force sense to the operation unit 6 of the master device 4 operated by the operator while watching a real-time image on the display 8, so that the operator can feel information that the hand unit 7 has come into contact with an object or a surrounding object from the operation unit 6, and therefore the three-dimensional positional relationship between the object and the hand unit 7 can be accurately recognized by touch through intuitive operation of the operation unit 6, thereby improving workability. In particular, the operator can grasp the positional information of the object and the positional relationship with the hand unit 7 that is preferable for grasping the object by feeling information from the operation unit 6 that the inner surfaces 70d, 71d of the gripping pieces 70, 71 have come into contact with the object.

[0120] In addition, by obtaining the external force acting on the outside (outer surfaces 70a, 71a) and both side surfaces 70b, 70c, 71b, 71c of the pair of gripping pieces 70, 71 that constitute the hand unit 7 and feeding it back to the operation unit 6 of the master side device 4 as a force sense, it is possible to make it easier for the operation unit 6 to sense information that the gripping pieces 70, 71 that constitute the hand unit 7 of the slave side device 5 have come into contact with an object or surrounding objects.

[0121] In addition, the control device 1 obtains the external force acting on the tips 70e, 71e of the gripping pieces 70, 71 and feeds this back to the operation unit 6 of the master side device 4 as a force sense, making it easier for the operation unit to sense information that the gripping pieces 70, 71 that constitute the hand unit 7 of the slave side device 5 have come into contact with an object or surrounding objects.

[0122] In addition, regardless of whether the control device 1 receives an external force from the inner surfaces 70d, 71d of the gripping pieces 70, 71 of the hand unit 7, the control device 1 feeds back the reaction force to one of the operating pieces 60, 61 of the operating unit 6, which in this embodiment is the operating piece 61 through which the index finger is passed, thereby simplifying the master side device 4.

[0123] In addition, the control device 1 is capable of adjusting the reaction force fed back to the operating unit 6 in response to the external force acting on the inner surfaces 70d, 71d of the gripping pieces 70, 71 of the hand unit 7, depending on the degree of opening and closing of the gripping pieces 70, 71 of the hand unit 7, i.e., the state of the gripping operation of the pair of scissor-handle-shaped operating pieces 60, 61 in the operating unit 6.This makes it possible to switch the specifications of the force feedback required in the pick-and-place process between the alignment operation in which the operating unit 6 recognizes the three-dimensional positional relationship between the object and the hand unit 7 by feel, and the gripping operation in which the object is gripped by the gripping pieces 70, 71.

[0124] Furthermore, in the work system of the present invention, the streaming camera 9 that captures the movement of the hand unit 7 is provided in a location different from the hand unit 7, i.e., in a location where it can capture the positional relationship between the object and the hand unit 7 from diagonally above, and the positional relationship between the object and the hand unit 7 can be visually recognized from the real-time image captured by the streaming camera 9 and displayed on the display 8, while the three-dimensional positional relationship between the object and the hand unit 7 can be accurately recognized by touch through intuitive operation of the operation unit 6, thereby further improving workability through a remote operation interface.

[0125] Although the embodiments of the present invention have been described above with reference to the drawings, the specific configuration is not limited to these embodiments, and the present invention also includes modifications and additions that do not deviate from the gist of the present invention.

[0126] For example, in the first embodiment, a mode has been described in which the position display screen 100, which simply represents the work area, is displayed in a small window in the upper right corner of the work screen 80, but the present invention is not limited to this, and the position display screen 100 may be displayed at any position on the work screen 80. In addition, the position display screen 100 may be displayed semi-transparently near the hand unit 7 on the work screen 80 and made to follow the moving operation of the hand unit 7, so that the moving operation of the hand unit 7 on the work screen 80, the position of the virtual operation unit 6 based on the master side virtual position display M and the slave side virtual position display S displayed on the position display screen 100, and the position of the virtual hand unit 7 can be easily confirmed simultaneously. This allows the operator to simultaneously check the position display screen 100 without significantly moving his or her line of sight from the hand unit 7, and to concentrate on the operation of the hand unit 7 by the operation unit 6.

[0127] In addition, in the above embodiment, a two-fingered robot hand is attached as the hand unit 7 of the slave side device 5, but this is not limited thereto, and the hand unit may be freely selected depending on conditions such as the size and material of the object, and may be configured, for example, by a multi-fingered hand having three or more gripping pieces, a suction cup, etc.

[0128] In the above embodiment, a scissor-handle interface device is attached as the operation unit 6 of the master side device 4, but the structure of the operation unit may be freely selected according to the working conditions, etc., and may be configured with a glove structure, a trigger lever structure, etc.

[0129] In the above embodiment, the master device 4 and the slave device 5 constituting the operation system have the same structure and the same size. However, the present invention is not limited to this. The master device and the slave device may have different structures, sizes, and outputs.

[0130] In addition, in the above embodiment, a mode has been described in which a real-time image of an object photographed by the slave-side streaming camera 9 and the hand portion 7 of the slave-side device 5 is displayed on the master-side display 8. However, the present invention is not limited to this mode. For example, only a position display screen 100 that simply represents the working area may be displayed on the entire display 8.

[0131] In addition, in the above embodiment, the work system has been described as being configured by combining autonomous picking technology using a work robot equipped with AI and remote control technology in which an operator remotely controls the work robot, but this is not limited to the above and the work system may be configured using only remote control technology.

[0132] In the above embodiment, the pick direction in which the amount of movement of the hand unit 7 of the slave device 5 is converted at a large magnification relative to the amount of displacement of the operation unit 6 of the master device 4 in the pick-and-place process is the z-axis direction (up-down direction). However, the pick direction is not limited to this, and may be the x-axis direction (left-right direction) or the y-axis direction (depth direction) or may be a diagonal direction (for example, the direction of a vector having a y component and a z component) depending on the configurations of the master device and the slave device.

[0133] In addition, under the conditions of the above embodiment, it is preferable from the viewpoint of improving workability that the movement amount of the hand unit 7 of the slave device 5 relative to the operation amount of the operation unit 6 of the master device 4 is converted at a magnification of one axis among the three axes corresponding to the pick direction larger than the movement amount of the other two axes corresponding to the alignment directions, but this is not limited thereto, and in the pick-and-place process, for example, under conditions where delicate operation is required for one axis corresponding to the pick direction and delicate operation is not required for the two axes corresponding to the alignment directions, the two axes among the three axes corresponding to the alignment directions may be converted at a magnification larger than the movement amount of the other axis corresponding to the pick direction. In this case, the conversion magnification of the movement amount of the two axes corresponding to the alignment directions may be different.

[0134] Furthermore, the displacement (amount of movement) and the force do not need to be converted at the same magnification, and the displacement and the force may be converted at different magnifications.

[0135] In addition, in the above embodiment, a manner in which an external force acting on the hand unit 7 is detected from the load torque, rotation speed, current value, etc. of each drive motor built into the arm unit 50 and hand unit 7 of the slave side device 5 has been described. However, the present invention is not limited to this, and the detection of an external force acting on the hand unit may be performed in various detection manners, such as a manner in which a tactile sensor that detects pressure or frictional force is used. [Explanation of symbols]

[0136] 1 Control device 2. Control device 3. Control device 4 Master device 5 Slave device 6 Control section 7 Hand section 8 Display (display means) 9. Streaming Camera 10A, 10B Container 40,50 Arm section 60,61 Operation piece 62 Handle 70,71 Gripping piece 80 Work Screen 100 position display screen 101 Scale Selector M Master side virtual position display (Virtual position display) S Slave side virtual position display (Virtual position display) X,Y auxiliary lines

Claims

1. An operation system capable of performing a pick-and-place process on an object by transmitting an operation of a master device by an operator to a slave device, comprising: a display means for displaying a work screen showing a work area in which a hand unit constituting the slave device is disposed, the display means being disposed near the master device; A work system characterized in that a virtual position display reflecting position information of an operation unit constituting the master device in real time is displayed on the work screen.

2. 2. The work system according to claim 1, wherein a position display screen that simply represents the work area is displayed in a small window on the work screen, and the virtual position display is displayed on the position display screen.

3. 3. The work system according to claim 2, wherein the position display screen simultaneously displays a virtual position display that reflects position information of the hand unit in real time, in addition to the virtual position display of the operation unit.

4. 3. The work system according to claim 2, wherein at least two auxiliary lines of perpendicular axes are displayed on the position display screen.

5. 4. The working system according to claim 3, wherein when the hand unit is in an initial position, the virtual position display of the operation unit and the virtual position display of the hand unit are displayed overlapping at the same position on the position display screen.

6. A work system as described in any one of claims 3 to 5, characterized in that a magnification display showing a conversion magnification of the movement amount of the hand unit relative to the displacement amount of the operation unit is displayed near the position display screen on the work screen.

Images