Operation support device and operation support method

Abstract

To provide an operation support device and an operation support method that allow a user to easily operate a work device.SOLUTION: A working device 2, which can be operated by a user and includes a contact device 201 that comes into contact with a work object, includes a contact surface projection distance calculation unit 113 that calculates a distance between each part of the contact device 201 and the work object, and a display processing unit 4 that creates a contact surface projection distance image according to the distance value between each part of the contact device 201 and the work object, and outputs the created contact surface projection distance image to the display device 4. In the contact surface projected distance image, one of a color, density, a pattern, and a contour shape is associated with the distance.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a technique for an operation assistance device and an operation assistance method. [Background technology]

[0002] When gripping a workpiece with an end effector by remote control, it can be difficult for the operator to grasp the position of the end effector relative to the workpiece. Specifically, it is difficult for the operator to grasp the position of the end effector relative to the workpiece until the gripping position of the end effector is about to come into contact with the workpiece.

[0003] Patent Document 1 discloses a technique in which the center of a workpiece is assumed, and if the workpiece moves out of the field of view of the camera during teaching, the camera is directed toward the assumed center. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-214265 Summary of the Invention [Problem to be solved by the invention]

[0005] To address this issue, it is conceivable for the operator to view the workpiece from cameras in three or more directions. However, this method has the problem of increasing the effort required to install the cameras and the cost of the cameras. Furthermore, Patent Document 1 does not disclose any technology that solves the problem that it is difficult for the operator to grasp the position of the end effector relative to the workpiece until just before the gripping position of the end effector comes into contact with the workpiece.

[0006] The present invention has been made in view of the above background, and an object of the present invention is to enable a user to easily operate a work device. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the present invention provides a work device that can be operated by a user and has a contact part that comes into contact with a work object, the work device having: a calculation part that calculates the distance between each part of the contact part and the work object; an image creation part that creates an image according to the value of the distance between each part of the contact part and the work object; and an output processing part that outputs the image created by the image creation part to an output part. Other solutions will be described as appropriate in the embodiments. [Effects of the Invention]

[0008] According to the present invention, the user can easily operate the work device. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram (part 1) showing a work device and a work target. [Figure 2] FIG. 10 is a diagram (part 1) showing an example of a contact surface projected image displayed on a display device of the operation assistance device. [Figure 3] FIG. 2 is a second diagram showing the working device and the work object. [Figure 4] FIG. 10 is a diagram (part 2) showing an example of a contact surface projected image displayed on the display device of the operation assistance device. [Figure 5] 1 is a diagram showing a configuration of an operation assistance system according to a first embodiment. [Figure 6] FIG. 2 is a diagram illustrating a hardware configuration of the operation assistance device. [Figure 7] 3 is a flowchart showing the procedure of an operation assistance method according to the present embodiment. [Figure 8] 10 is a flowchart showing the detailed procedure of a contact projection distance calculation process. [Figure 9] 10 is a flowchart showing the detailed procedure of a contact surface projection image creation process. [Figure 10] FIG. 10 is a diagram (part 1) showing an operation support screen. [Figure 11] FIG. 2 is a diagram (part 2) showing the operation support screen. [Figure 12] FIG. 10 is a diagram showing another example of a contact surface projected image. [Figure 13] FIG. 11 is a diagram showing an example of an operation support screen in the second embodiment. [Figure 14A] FIG. 10 is a diagram (part 1) showing an example of an autonomous guided vehicle used in the third embodiment. [Figure 14B] FIG. 11 is a diagram (part 2) showing an example of an autonomous guided vehicle used in the third embodiment. [Figure 15A] FIG. 1 is a diagram (part 1) showing a method for transferring an object to an autonomous mobile guided vehicle. [Figure 15B] FIG. 10 is a diagram (part 2) showing a method for transferring an object to an autonomous mobile guided vehicle. [Figure 16] FIG. 11 is a diagram showing an example of an operation support screen displayed in the third embodiment. [Figure 17] FIG. 10 is a first diagram showing a working device and a work target in a fourth embodiment. [Figure 18] FIG. 13 is a second diagram showing the working device and the work target in the fourth embodiment. [Figure 19] FIG. 13 is a diagram showing an example of an image displayed on a display device of the operation assistance device in the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Next, modes for carrying out the present invention (referred to as "embodiments") will be described in detail with reference to the drawings as appropriate.

[0011] First Embodiment [overview] First, an outline of the operation assistance method according to this embodiment will be described with reference to FIGS.

[0012] 1 and 3 are diagrams showing the operation device 2 and the work target W. Also, FIGS. 2 and 4 are diagrams showing an example of a contact surface projection distance image 411 displayed on the display device 4 (see FIG. 5) of the operation assistance device 1 (see FIG. 5).

[0013] The working device 2 can be operated by a user, and in the example shown in Figs. 1 and 3, a manipulator is assumed. The working device 2 includes a contact device 201, which is a contact portion, a base 202, and an arm 203. The contact device 201 can move the work object W by contacting the work object W (for example, by gripping or suction). In Figs. 1 and 3, a suction pad is assumed as the contact device 201, but a parallel hand or the like may also be used. Note that the outline arrows in Figs. 1 and 3 indicate the direction of movement of the contact device 201.

[0014] 1 and 2 show an example in which the contact surface 201a (see FIG. 3) of the contact device 201 of the working device 2 is angled with respect to the work object W, and FIGS. 3 and 4 show an example in which the contact surface 201a of the contact device 201 of the working device 2 is approximately parallel to the contact surface of the work object W.

[0015] When the contact device 201 is a suction pad, the working device 2 can lift the work object W by adsorbing the contact device 201 to one surface of the work object W.

[0016] Furthermore, a contact object shape acquisition device 3 installed outside the working apparatus 2 acquires images of the work object W. The contact object shape acquisition device 3 may be a ToF (Time Of Flight) camera, a stereo camera 301, or a LiDAR (Light Detection And Ranging). The contact object shape acquisition device 3 records the three-dimensional shape of the work object W from a position where the entire work object W can be seen. Note that in this embodiment, the contact object shape acquisition device 3 is provided separately from the working apparatus 2, but the contact object shape acquisition device 3 may also be provided in the contact device 201, for example.

[0017] As shown in FIGS. 2 and 4, the display device 4 of the operation assistance device 1 displays a contact surface projection distance image 411 superimposed on a work object image 414, which is an image of the work object W. The contact surface projection distance image 411 is an image corresponding to the distance between each part of the contact surface 201a of the contact device 201 and the work object W. As can be seen from FIG. 4, the contact surface 201a is rectangular, but it may be circular, elliptical, or have other shapes. The contact surface 201a may also be formed by arranging multiple suction pads in a predetermined manner, such as in a matrix. In this case, it does not matter whether the multiple suction pads are densely or sparsely arranged, as long as they can lift the work object W.

[0018] 2 and 4, the contact surface projected distance image 411 represents the distance between each part of the contact surface 201a of the contact device 201 and the work object W by the shade of dots. Note that in Fig. 2 and Fig. 4, the distance between each part of the contact device 201 and the work object W is represented by the shade of dots, but in reality, the distance is represented by color, such as red representing a part of the contact surface 201a that is close and blue representing a part that is far away.

[0019] As shown in the example of Fig. 1, there are cases where the contact device 201 is at an angle to the contact device 201 of the work object W (a state that is not suitable for lifting the work object W). In such cases, the contact surface projection distance image 411, as shown in Fig. 2, expresses the distance between each position of the contact device 201 (specifically, each position of the contact surface 201a) and the work object W by the shade of dots. Note that in the example shown in Fig. 2, light dots indicate that the distance between the contact device 201 and the work object W is short, and dark dots indicate that the distance between the contact device 201 and the work object W is long.

[0020] When the contact device 201 is substantially parallel to the contact surface of the workpiece W as in the example shown in FIG. 3, the contact surface projected distance image 411 is made up of dots of substantially uniform density as shown in FIG.

[0021] [Operation Support System Z] FIG. 5 is a diagram showing the configuration of an operation support system Z according to the first embodiment.

[0022] The operation support system Z includes an operation support device 1, a working device 2, a contact object shape acquisition device 3, a display device 4 serving as an output unit, and an operation device 5.

[0023] The operation support device 1, which supports the operation of the work device 2, calculates the distance between each position of the contact device 201 and the work object W based on the design information of the contact device 201, the posture information of the work device 2, and the three-dimensional shape of the work object W acquired from the contact object shape acquisition device 3.

[0024] The operation assistance device 1 is configured from a PC (Personal Computer), a tablet terminal, etc. The operation assistance device 1 includes a contact surface shape storage unit 111, a contact surface attitude calculation unit 112, a contact surface projection distance calculation unit 113 which is a calculation unit, and a display processing unit 114 which is an image creation unit and an output processing unit. By using a tablet terminal as the operation assistance device 1, it becomes portable.

[0025] The contact surface shape storage unit 111 stores in advance design information (CAD (Computer Aided Design) information) of the contact device 201. Since the design information of the contact device 201 is stored in advance in this way in the contact shape storage unit, when the contact device 201 is changed, the design information can be easily replaced.

[0026] The contact surface posture calculation unit 112 calculates the posture of the contact surface 201a of the contact device 201 (see FIG. 3).

[0027] The contact surface projection distance calculation unit 113 mainly calculates the distance between each part of the contact surface 201a (each part of the contact portion) of the contact device 201 and the workpiece W.

[0028] The display processing unit 114 creates a contact surface projection distance image 411, which is an image corresponding to the distance values ​​between each part of the contact surface 201a of the contact device 201 and the work target W. Furthermore, the display processing unit 114 displays the created contact surface projection distance image 411 on the display device 4. Furthermore, the display processing unit 114 creates the contact surface projection distance image 411 and a distance information image 412, which is a legend of the distance relationship, and displays (outputs) it on the display device 4.

[0029] The operation assistance device 1 also acquires three-dimensional shape information of the work object W from the contact object shape acquisition device 3. As described above, the contact object shape acquisition device 3 is composed of a ToF camera, a stereo camera, LiDAR, etc. If a ToF camera is used, the contact object shape acquisition device 3 can be easily constructed at low cost. If a stereo camera is used, color images can be acquired. If LiDAR is used, three-dimensional information of the work object W (see FIGS. 1 and 3) can be acquired at high resolution.

[0030] The working device 2 is a robot such as a manipulator. As described above, the working device 2 is equipped with the contact device 201, which can lift the workpiece W. As described above, the contact device 201 uses a suction pad, a parallel hand, or the like. By using a suction pad, the workpiece W can be lifted vertically. Furthermore, by monitoring the positions of the claws of the parallel hand, the user can recognize the positions (grasping positions) of the claws relative to the workpiece W.

[0031] The display device 4 may be a PC monitor, a tablet terminal, or a monitor. A PC monitor is a monitor used in a laptop computer or the like. Using a tablet terminal as the display device 4 makes it portable and easier for the user to operate. Using a monitor as the display device 4 allows the monitor and the computing device to be used separately when using a high-performance computing device (desktop PC).

[0032] Furthermore, a contact surface projected distance image 411 and a distance information image 412 are displayed on the display device 4. The contact surface projected distance image 411 and the distance information image 412 will be described later.

[0033] The operation device 5 is a device with which the user operates the operation apparatus 2, and is configured from a teaching pendant, a tablet terminal, a smartphone, or the like. Using a teaching pendant as the operation device 5 enables detailed operation of the operation apparatus 2. Using a tablet terminal as the operation device 5 allows the user to easily install software that can operate the operation apparatus 2 while viewing the display on the tablet terminal.

[0034] [Hardware configuration of operation support device 1] FIG. 6 is a diagram showing the hardware configuration of the operation assistance device 1. As shown in FIG.

[0035] The operation assistance device 1 is configured, for example, by a PC (Personal Computer) or the like, and includes a memory 121 configured by a RAM (Random Access Memory) or the like. The operation assistance device 1 also includes a calculation device 122 configured by a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), or the like. The operation assistance device 1 also includes a storage device 123 configured by an HDD (Hard Disk Drive), an SSD (Solid State Drive), or the like, and a communication device 124. The communication device 124 communicates with the working device 2, the contact object shape acquisition device 3, and the like.

[0036] Then, the program stored in the storage device 123 is loaded into the memory 121, and the loaded program is executed by the arithmetic device 122. As a result, the components from the contact surface attitude calculation unit 112 to the display processing unit 114 shown in FIG. 5 are realized.

[0037] The contact surface shape storage unit 111 in FIG.

[0038] [flowchart] (Overall processing) 7 is a flowchart showing the procedure of the operation assistance method according to this embodiment. Note that in FIGS. 7 to 9, FIG. 5 will be referred to as appropriate.

[0039] The contact surface orientation calculation unit 112 acquires design information of the working device 2 (contact device 201) from the contact surface shape storage unit 111 (S1).

[0040] Next, the contact surface attitude calculation unit 112 acquires attitude information of the working device 2 from each of the sensors provided on the working device 2 (S2).

[0041] Next, the contact surface attitude calculation unit 112 performs contact surface attitude calculation processing by calculating the attitude of the contact surface 201a of the contact device 201 through coordinate transformation (S3). As a result of step S3, contact surface attitude information is generated.

[0042] Thereafter, the contact surface projection distance calculation unit 113 performs a contact surface projection distance calculation process (S4). In step S4, the contact surface projection distance calculation unit 113 calculates the contact surface projection distance based on the design information of the working device 2 acquired in step S1 and the contact surface posture information generated as a result of the calculation in step S3. The contact surface projection distance is the distance between each part of the contact device 201 and the work object W. The processing of step S4 will be described later. Step S4 is also a calculation step in the claims.

[0043] Then, the display processing unit 114 performs a contact surface projection distance image creation process (S5). The process of step S5 will be described later. Step S5 is also referred to as an image creation step in the claims.

[0044] Thereafter, the display processing unit 114 performs a display process (S6) to display the contact surface projected distance image 411 (see FIGS. 10 and 11) and the distance information image 412 (see FIG. 5) created in step S5 on the display device 4. Step S6 is an output processing step in the claims.

[0045] (Contact projection distance calculation process) FIG. 8 is a flowchart showing the detailed procedure of the contact projection distance calculation process (step S4) in FIG.

[0046] First, the contact surface projection distance calculation unit 113 converts the three-dimensional information (mesh or point cloud data) of the work object W acquired from the contact object shape acquisition device 3 into coordinates based on the projection direction of the contact surface 201a (S401). The coordinates based on the projection direction of the contact surface 201a are coordinates defined such that the origin of the coordinates is defined on the contact surface 201a of the contact device 201 and the normal direction of the contact surface 201a is the z coordinate pointing toward the work object W. That is, in step S401, the contact surface projection distance calculation unit 113 acquires three-dimensional information of the work object W as viewed from the direction of the contact surface 201a. The z coordinate is a coordinate set in the up-down direction with respect to the operation assistance device 1 and the work object W, i.e., perpendicular to the installation surface of the operation assistance device 1.

[0047] Next, the contact surface projection distance calculation unit 113 creates a contact object projection image by projecting the three-dimensional information of the contact object after the coordinate transformation in step S401 onto a two-dimensional image (S402). Projecting onto a two-dimensional image in step S402 means projecting a three-dimensional image of the work object W in the direction of the z coordinate defined in step S401. The contact object projection image created in this way is an image of the work object W projected onto the two-dimensional image from the direction of the contact surface 201a.

[0048] The contact surface projection distance calculation unit 113 also creates a contact surface projection image for the contact surface 201a by performing the same procedures as in steps S401 and S402 (S403). In step S403, an image is created in which the contact surface 201a is projected onto the contact surface (contacted surface) of the work object W.

[0049] The contact surface projection distance calculation unit 113 creates a difference image between the contact object projection image created in step S402 and the contact surface projection image created in step S403, and extracts only the area of ​​the contact surface 201a (S404). That is, in step S404, the contact surface projection distance calculation unit 113 extracts an area in the contact object projection image where the contact surface projection image overlaps.

[0050] Then, the contact surface projection distance calculation unit 113 converts the pixel values ​​of the differential region extracted in step S404 into point cloud or mesh data, and creates data based on the contact object shape acquisition device 3 by coordinate transformation (S405). In step S405, a three-dimensional image of the differential region relative to the contact object shape acquisition device 3 is created.

[0051] Furthermore, the contact surface projection distance calculation unit 113 calculates the maximum distance and the minimum distance between each position on the contact surface 201a and the work object W (S406). In step S406, the contact surface projection distance calculation unit 113 calculates the distance between each position on the contact surface 201a and the work object W based on the three-dimensional image of the difference region for the contact object shape acquisition device 3 created in step S405. Thereafter, the contact surface projection distance calculation unit 113 obtains the maximum and minimum values ​​of the calculated distances and sets them as the maximum distance and the minimum distance, respectively.

[0052] (Contact surface projection distance image creation process) FIG. 9 is a flowchart showing the detailed procedure of the contact surface projected distance image creation process (step S5) in FIG.

[0053] First, the display processing unit 114 performs a conversion process to convert the data acquired from the contact object shape acquisition device 3 into a distance image and a color image (S501). The distance image is an image in which information on the distance between the contact surface 201a of the contact device 201 and the work object W is embedded. Note that the conversion to a color image may be omitted. The distance calculated in step S406 is used as the distance in the distance image.

[0054] Thereafter, the display processing unit 114 converts the difference image calculated by the contact surface projection distance calculation unit 113 into an image (for example, color) according to the distance in the distance image (S502). The converted image is referred to as a contact surface projection distance image 411.

[0055] Next, the display processing unit 114 performs an overlay process to overlay the contact surface projected distance image 411 on the image acquired from the contact object shape acquisition device 3 (S503).

[0056] Next, the display processing unit 114 performs additional processing to create a distance information image 412 (see FIGS. 10 and 11) and an attitude information display area 413 (see FIGS. 10 and 11) and add them to the displayed image (S504). The distance information image 412, which will be described later, is a legend showing the relationship between the image pattern (color, density, etc.) in the contact surface projected distance image 411, each position on the contact surface 201a, and the distance from the work target W. The attitude information display area 413, which will be described later, is information showing the current attitude of the contact device 201.

[0057] [Screen example] Next, an example of the operation assistance screen 400 displayed on the display device 4 in the first embodiment will be described with reference to Fig. 10 and Fig. 11. Fig. 5 will also be referenced as appropriate. The examples shown in Fig. 10 to Fig. 13 show an example in which the operation assistance device 1 in Fig. 5 and the operation device 5 are integrated as a tablet terminal.

[0058] FIG. 10 is a diagram showing an operation support screen 400 when the contact surface 201a of the contact device 201 and the contact surface of the work object W form a predetermined angle.

[0059] The operation support screen 400 displays a main display section 410, operation buttons 401, a display switching section 421, a setting change section 422, and an operation mode change section 423.

[0060] The main display section 410 displays a contact surface projected distance image 411, a distance information image 412, and an attitude information display area 413.

[0061] As described above, the contact surface projection distance image 411 is displayed overlapping the work object image 414, which is an image of the work object W (see FIGS. 1 and 3). In the example shown in FIG. 10, the contact surface 201a of the contact device 201 and the contact surface of the work object W form a predetermined angle. Therefore, similar to FIG. 2, in the contact surface projection distance image 411, the distance between each position of the contact surface 201a of the contact device 201 and the work object W is expressed by the density of dots. Note that in the example shown in FIG. 10, the lighter the dot, the closer the contact surface 201a of the contact device 201 and the contact surface of the work object W are. Furthermore, the darker the dot, the farther the contact surface 201a of the contact device 201 and the contact surface of the work object W are.

[0062] 2, in the contact surface projection distance image 411 in Fig. 10, the distance between each position on the contact surface 201a of the contact device 201 and the work object W is represented by the density of dots, but in reality, they are differentiated by color. Also, in the contact surface projection distance image 411, the distance to the work object W may be represented by the density of a specific color.

[0063] The distance information image 412 also displays a legend of the shading of the dots in the contact surface projection distance image 411. If the range of the distance information image 412 is set to the minimum distance (lightest dot) to the maximum distance (darkest dot) in the contact surface projection distance image 411, the user can easily visually grasp the inclination and position of the contact device 201 relative to the work object W.

[0064] The attitude information display area 413 displays the current attitude information of the contact device 201 as coordinate information, including the "X" coordinate, "Y" coordinate, "Z" coordinate, "R" value, "P" value, and "Yo" value. It is desirable that the values ​​displayed in the attitude information display area 413 be the attitude information of the contact device 201 relative to the work object W. The "X" coordinate, "Y" coordinate, and "Z" coordinate are coordinates when an arbitrary location in the work area where the work device 2 is working is set as the origin. The "R" value indicates the angle of rotation around the roll axis, the "P" value indicates the angle of rotation around the pit axis, and the "Yo" value indicates the angle of rotation around the yaw axis.

[0065] By displaying the posture information display area 413, the user can specifically grasp the current posture of the contact device 201 and the distance between the contact surface 201a of the contact device 201 and the contact surface of the work object W. This allows the user to specifically predict how much further movement is required to contact the work object W.

[0066] The operation buttons 401 are used when manually operating the contact device 201. In the example shown in Fig. 10, the operation buttons 401 include an "X-" button, an "X+" button, a "Y-" button, a "Y+" button, a "Z-" button, and a "Z+" button. The operation buttons 401 also include an "R-" button, an "R+" button, a "P-" button, a "P+" button, a "Y-" button, and a "Yo+" button.

[0067] The "X-" button and "X+" button are buttons for moving the contact device 201 in the X coordinate direction. The "Y-" button and "Y+" button are buttons for moving the contact device 201 in the Y coordinate direction. The "Z-" button and "Z+" button are buttons for moving the contact device 201 in the Z coordinate direction. The "R-" button and "R+" button are buttons for rotating the contact device 201 around the roll axis. The "P-" button and "P+" button are buttons for rotating the contact device 201 around the pitch axis. The "Y-" button and "Y+" button are buttons for rotating the contact device 201 around the yaw axis.

[0068] The display switching unit 421 allows the user to switch from the currently used contact object shape acquisition device 3 to another contact object shape acquisition device 3, or to switch to an image that does not project the contact surface 201a, etc.

[0069] In the setting change section 422, the user can make screen settings such as changing the current operation support screen 400 to a menu screen.

[0070] The operation mode change unit 423 can switch the movement mode of the contact device 201, such as each coordinate of the contact device 201, movement of each axis, linear movement, arc movement, etc. When the movement mode of the contact device 201 is changed by the operation mode change unit 423, the display of the operation button 401 is also changed.

[0071] Fig. 11 is a diagram showing an operation assistance screen 400 when the contact surface 201a of the contact device 201 and the contact surface of the work target W are approximately parallel. In Fig. 11, the same components as those in Fig. 10 are denoted by the same reference numerals, and description thereof will be omitted.

[0072] 11 , the contact surface 201a of the contact device 201 and the contact surface of the work object W are substantially parallel, so the contact surface projection distance image 411 is displayed as dots of substantially uniform density. This allows the user to visually recognize that the contact surface 201a of the contact device 201 and the contact surface of the work object W have become substantially parallel. Note that if the contact device 201 is moved away from the work object W while the contact surface 201a of the contact device 201 and the contact surface of the work object W remain substantially parallel, the contact surface projection distance image 411 becomes dark dots while maintaining a uniform density. Conversely, if the contact device 201 is moved closer to the work object W while the contact surface 201a of the contact device 201 and the contact surface of the work object W remain substantially parallel, the contact surface projection distance image 411 becomes light dots while maintaining a uniform density.

[0073] Furthermore, the display range of the distance information image 412 changes in accordance with the display of the contact surface projected distance image 411. Specifically, the display range of the distance information image 412 becomes smaller as the density becomes uniform in the contact surface projected distance image 411. Such a change in the distance information image 412 may be made by the operation assistance device 1 or may be made manually by the user.

[0074] When the operation assistance device 1 changes the distance information image 412, the change is made in the following procedure. First, the user sets the minimum value of the display range (hereinafter referred to as the minimum range). For example, the user sets the minimum range to 10 mm. Then, the contact surface projection distance calculation unit 113 determines whether the difference between the nearest point and the farthest point in the contact surface projection distance image 411 is 10 mm or less. If the difference between the nearest point and the farthest point is 10 mm or less, the display processing unit 114 displays the distance information image 412 within a range of ±5 mm of the average value of the distance between the contact surface 201a of the contact device 201 and the contact surface of the work object W. This allows the user to recognize that the contact device 201 and the contact object are parallel if all the contact surface projection areas are the same color. Note that hereinafter, the distance between each position on the contact surface 201a of the contact device 201 and the contact surface of the work object W will be simply referred to as the distance, as appropriate.

[0075] In this way, the display processing unit 114 can change the range of distances indicated in the distance information image 412 according to the range of distances displayed in the contact surface projected distance image 411 displayed on the display device 4.

[0076] The range of the distance information image 412 can also be changed manually by the user using a mouse (not shown) or the like.

[0077] FIG. 12 is a diagram showing another example of the contact surface projected distance image 411. In FIG. 10, the contact surface projection distance image 411 uses density and color to represent the distance between each position on the contact surface 201a of the contact device 201 and the work object W. However, this is not limiting, and the distance between each position on the contact surface 201a of the contact device 201 and the work object W may be represented by different patterns as shown in FIG.

[0078] In this way, in the contact surface projected distance image 411, one of color, density, and pattern is associated with distance.

[0079] It is difficult for the user to grasp the positional relationship between the work object W and the contact device 201 from numerical values ​​alone. According to the first embodiment, a contact surface projection distance image 411 is displayed as an image corresponding to the distance value between each part of the contact device 201 and the work object W. This allows the user to easily visually recognize the distance between the contact device 201 and the work object W, and the angle of the contact device 201 relative to the work object W. This allows the user to easily operate the work device.

[0080] Furthermore, one of color, density, and pattern is associated with distance in the contact surface projected distance image 411. This allows the user to easily visually recognize the distance between the contact device 201 and the work object W, and the angle of the contact device 201 with respect to the work object W.

[0081] Then, the contact surface projected distance image 411 and the distance information image 412, which is a legend showing the distance relationship, are displayed. Furthermore, the display processing unit 114 changes the range of distances shown in the distance information image 412 according to the range of distances shown in the contact surface projected distance image 411. This allows the user to easily confirm the relationship between the pattern in the contact surface projected distance image 411 and the actual distance.

[0082] Second Embodiment Fig. 13 is a diagram showing an example of an operation support screen 400 in the second embodiment. In Fig. 13, the same components as those in Fig. 10 and Fig. 11 are denoted by the same reference numerals and description thereof will be omitted.

[0083] 13, the contact surface projection distance image 411a displayed on the main display unit 410a shows the distance between the contact surface 201a of the contact device 201 and the contact surface of the work object W in the form of contour lines. Also, the distance information image 412 shown in FIGS. 10 and 11 is omitted.

[0084] Third Embodiment Next, a third embodiment of the present invention will be described with reference to Figures 14A to 16. In the third embodiment, an example in which the working device 2 is an autonomous mobile guided vehicle 2b will be described.

[0085] [Autonomous mobile guided vehicle 2b] 14A and 14B are diagrams showing an example of an autonomous mobile guided vehicle 2b used in the third embodiment. Fig. 14A shows a side view, and Fig. 14B shows a front view. The outline arrow in Fig. 14A indicates the traveling direction of the autonomous mobile guided vehicle 2b.

[0086] The autonomous mobile guided vehicle 2b, which is the working device 2, is equipped with a stereo camera 301 and an active stereo camera 302 as a contact object shape acquisition device 3. The autonomous mobile guided vehicle 2b also has a transfer device 211, a transfer device control unit 212 for controlling the transfer device 211, and a base 220. The base 220 is equipped with wheels 221 for autonomous movement. Furthermore, contact sensors 222 are provided around the base 220. These contact sensors 222 detect contact between the base 220 and surrounding objects.

[0087] The autonomous mobile guided vehicle 2b is capable of autonomous movement using wheels 221. The autonomous mobile guided vehicle 2b transports the article C by placing the article C on a transfer device 211. The transfer device 211 is configured with, for example, rollers.

[0088] The autonomous mobile guided vehicle 2b can measure the distance to an object present in its traveling direction using the stereo camera 301 and the active stereo camera 302. Note that, although the third embodiment is described as including the stereo camera 301 and the active stereo camera 302 as the contact object shape acquisition device 3, this is not limiting. For example, a LiDAR may be used as the contact object shape acquisition device 3.

[0089] [Transfer method] 15A and 15B are diagrams showing a method for transferring an article C to an autonomously guided vehicle 2b.

[0090] 15A, the transported article C is placed on the transfer table 6. A transfer device 601 composed of rollers and the like is provided on the upper part of the transfer table 6. When the transfer device 601 is operated, the transported article C can be moved to the carry-out side 611.

[0091] As shown in FIG. 15B, the autonomous mobile guided vehicle 2b brings its front surface 231 (see FIG. 15A) into contact with the carry-out side 611 (see FIG. 15A) of the transfer table 6. The front surface 231 is the surface on the traveling direction side, and in FIGS. 14A and 14B, is the surface on which the stereo camera 301 and the active stereo camera 302 are provided. The front surface 231 corresponds to the contact surface 201a in the first embodiment.

[0092] Autonomous mobile guided vehicle 2b is controlled so that the entire front surface 231 of autonomous mobile guided vehicle 2b itself comes into contact with the carry-out side 611 of transfer table 6. Thereafter, transfer device 601 of transfer table 6 and transfer device 211 of autonomous mobile guided vehicle 2b are operated, and transported object C placed on transfer table 6 is moved to transfer device 211 of autonomous mobile guided vehicle 2b (white arrow in FIG. 15B).

[0093] In the third embodiment, the front surface 231 of the autonomous mobile guided vehicle 2b is configured to come into contact with the transfer table 6, but the side surface or rear surface of the autonomous mobile guided vehicle 2b may also be configured to come into contact with the transfer table 6.

[0094] [Screen example] Fig. 16 is a diagram showing an example of an operation support screen 400b displayed in the third embodiment. In Fig. 16, the same components as those in Figs. 10 to 13 are given the same reference numerals, and their explanations will be omitted. Fig. 16 is an example of a screen when the positional relationship between the autonomous mobile guided vehicle 2b and the transfer table 6 is as shown in Fig. 15A.

[0095] On the operation support screen 400b, an "X-" button, an "X+" button, a "θ-" button, and a "θ+" button are displayed as operation buttons 401b.

[0096] The "X-" button and "X+" button are buttons for moving the autonomous mobile guided vehicle 2b in the X coordinate direction. The "θ-" button and "θ+" button are buttons for moving the autonomous mobile guided vehicle 2b in the turning direction.

[0097] The main display section 410b displays a contact surface projected distance image 411b, a distance information image 412b, and a posture information display area 413b.

[0098] The main display section 410b displays an image of the legs of the transfer platform 6, which is the surface (corresponding to contact surface 201a) with which the autonomous mobile guided vehicle 2b comes into contact. A contact surface projection distance image 411b, in which the distance from the autonomous mobile guided vehicle 2b is shown as dot shading, is superimposed on the image of the legs. Note that the darker the dot shading, the greater the distance between the position of the autonomous mobile guided vehicle 2b and the transfer platform 6 (legs). Also, although the contact surface projection distance image 411b is shown as dot shading in FIG. 16, the distance to each position on the front surface 231 (see FIG. 15A) of the autonomous mobile guided vehicle 2b is actually expressed by color.

[0099] Further, a distance information image 412b is displayed in the main display section 410b. The distance information image 412b is similar to the distance information image 412 shown in FIGS.

[0100] Additionally, the posture information display area 413b displays the current posture information of the autonomous mobile guided vehicle 2b as coordinate information, including the "X" coordinate, "Y" coordinate, and "θ" angle. As in the first embodiment, it is desirable that each value displayed in the posture information display area 413b be posture information of the autonomous mobile guided vehicle 2b relative to the work target W (the transfer table 6 in the third embodiment). The "X" coordinate and "Y" coordinate are coordinates when an arbitrary location in the work area where the autonomous mobile guided vehicle 2b works is set as the origin. The "θ" angle indicates the orientation of the autonomous mobile guided vehicle 2b.

[0101] According to the third embodiment, when the user operates the autonomous mobile guided vehicle 2b, the autonomous mobile guided vehicle 2b can be easily positioned straight with respect to the transfer table 6.

[0102] <Fourth embodiment> Next, a fourth embodiment will be described with reference to FIGS.

[0103] 17 and 18 are diagrams showing the working device 2 and the work target W in the fourth embodiment. Also, FIG. 19 is a diagram showing an example of an image displayed on the display device 4 of the operation assistance device 1 in the fourth embodiment. In FIGS. 17 to 19, the same components as those in FIGS. 1 to 4 are given the same reference numerals, and their explanations will be omitted. Also, FIG. 5 will be referred to as appropriate.

[0104] As shown in FIG. 17, the contact object shape acquisition device 3 is installed at a predetermined location in the work space, and the user may perform work using the work device 2 while viewing the image acquired by the contact object shape acquisition device 3.

[0105] However, in such a case, as shown in FIG. 18, the working device 2 and the contact device 201 may get in the way and the user may not be able to see the work target W through the contact target shape acquisition device 3.

[0106] To cope with such a situation, there is a technique of installing multiple contact object shape acquisition devices 3 around the work object W to view the state of the work object W from various angles. However, increasing the number of contact object shape acquisition devices 3 increases costs. Also, the angle that is most convenient for the user to view varies depending on the positional relationship between the contact device 201 and the contact object shape acquisition device 3, so this does not necessarily lead to a solution.

[0107] To deal with such a case, in the fourth embodiment, the operation assistance device 1 pre-records the three-dimensional shape of the work object W at a position where the entire work object W can be seen, as shown in FIG. 17 (reference numeral 700).

[0108] 18, when the shadow of the working device 2 is cast on the work object W, the operation support device 1 displays a complement image 711 on the display device 4 as shown in Fig. 19. The complement image 711 is an image in which the parts that are not visible due to the shadow are complemented based on the recorded three-dimensional shape. This allows the user to easily visually recognize the parts that are shadowed by the contact object shape acquisition device 3.

[0109] The operation assistance methods shown in the first to fourth embodiments are performed to assist a user in manual operation when a situation arises where manual operation by the user is required for an autonomously operable working device 2 or an autonomous mobile guided vehicle 2b. The operation assistance methods shown in the first to fourth embodiments may also be used for teaching the working device 2 or the autonomous mobile guided vehicle 2b to learn an operation.

[0110] Moreover, the first to fourth embodiments can be combined with each other. [Explanation of symbols]

[0111] 1 Operation support device 2. Work equipment 2b Automated guided vehicle 3. Contact object shape acquisition device 4 Display device (output section) 5 Operating device 6 Transfer table 111 Contact surface shape retention part 112 Contact surface posture calculation section 113 Contact surface projection distance calculation section (calculation section) 114 Contact surface projection image creation unit (image creation unit, output processing unit) 121 memory 122 Arithmetic equipment 123 Storage device 124 communication equipment 201 Contact device (contact part) 201a Contact surface 202 Base 203 Arm 211 Transfer equipment 220 base 221 Wheels 222 Contact Sensor 231 Front 301 Stereo Camera 302 Active Stereo Camera 400 Operation support screen 400b Operation support screen 401 Operation button 401b Operation button 410 Main display 410b Main display 411 Contact Surface Projection Distance Image (Image) 411a Contact surface projection distance image (image) 411b Contact surface projected distance image (image) 412 Distance information image (legend) 412b Distance information image (legend) 413 Attitude information display area 413b Attitude information display area 414 Work object images 421 Display switching unit 422 Setting change section 423 Operation mode change unit 601 Transfer equipment 611 Carry-out side 700 code 711 complementary image C. Transported goods W Work object Z Operation Support System S4 Contact surface projection distance calculation process (calculation step) S5 Contact surface projection image creation process (image creation step) S6 Display processing (output processing step)

Claims

1. a calculation unit that calculates a distance between each part of the contact unit and the work object; an image creation unit that creates an image according to the value of the distance between each part of the contact unit and the work object; an output processing unit that outputs the image created by the image creation unit to an output unit; An operation assistance device having the above configuration.

2. In the image, One of color, density, and pattern is associated with the distance.

2. The operation assistance device according to claim 1.

3. The output processing unit displaying the image and the relationship between the distance as a legend on the output unit; The range of distances indicated in the legend is changed according to the range of distances indicated in the image output to the output unit.

2. The operation assistance device according to claim 1.

4. In the image, The distance is shown as a contour line shape 2. The operation assistance device according to claim 1.

5. The working device is an autonomous mobile guided vehicle.

2. The operation assistance device according to claim 1.

6. An operation assistance device that is operable by a user and assists in the operation of a working device having a contact portion that comes into contact with a work object, a calculation step of calculating a distance between each part of the contact portion and the work object in the work device; an image creation step of creating an image according to the distance between each part of the contact part and the work object; an output processing step of outputting the image created by the image creation step to an output unit; An operation support method for performing the above.

Images