Data processing device, data processing method and data processing system

Abstract

A data processing device 1 according to one embodiment of the present disclosure processes data for operating a movable part of a device in accordance with the position of a controller, and includes a control unit 13 configured so as to execute: the acquisition of controller information including a position when an operator uses the controller to operate the movable part; the acquisition of captured image data generated by an imaging device imaging the movable part during the operation of the movable part in accordance with the position; the generation of synthesized image data by superposing a controller image indicating the position included in the controller information onto a region corresponding to the movable part in the captured image data; and the display of the synthesized image data by a display device visible to the operator.

Description

[Technical Field]

[0001] The present disclosure relates to a data processing device, a data processing method, and a data processing system. [Background technology]

[0002] Patent Document 1 discloses a technology in which an operator remotely controls a robot using a controller worn by the operator, and an image generated by an imaging device installed in the robot is displayed on a terminal used by the operator. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-040155 Summary of the Invention [Problem to be solved by the invention]

[0004] When an operator operates a robot or other device while looking at captured images of the device's surroundings, the operator cannot see the controller in his or her hand. This makes it difficult for the operator to simultaneously recognize the position and orientation of the controller and the position and orientation of the device's movable parts, making it difficult to operate the device.

[0005] The present disclosure has been made in consideration of this point, and aims to provide a data processing device or the like that makes it easier for an operator of a device such as a robot to operate the device using a controller. [Means for solving the problem]

[0006] A data processing device according to one aspect of the present disclosure is a data processing device that processes data for operating movable parts of the device according to the position of a controller, and has a control unit configured to perform the following operations: when an operator operates a movable part using the controller, acquire controller information including the position; acquire captured image data generated by an imaging device capturing an image of the movable part while the movable part is being operated according to the position; generate composite image data by superimposing a controller image indicating the position included in the controller information on an area on the captured image data corresponding to the movable part; and display the composite image data on a display device visible to the operator. [Effects of the Invention]

[0007] According to the present disclosure, a data processing device or the like is provided that makes it easier for an operator of a device such as a robot to operate the device using a controller. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an overview of a data processing system. [Figure 2] FIG. 1 is a diagram illustrating a configuration of a data processing device. [Figure 3] FIG. 2 is a schematic diagram illustrating an operation device. [Figure 4] FIG. 4 is a schematic diagram of a controller image generated by a synthesis unit. [Figure 5] FIG. 2 is a schematic diagram of composite image data displayed on a display device. [Figure 6] FIG. 1 is a sequence diagram of a data processing method executed by a data processing system. [Figure 7] FIG. 1 is a flowchart showing a data processing method executed by a data processing device. [Figure 8] FIG. 2 is a schematic diagram of composite image data displayed on a display device. [Figure 9] FIG. 2 is a schematic diagram of composite image data displayed on a display device. DETAILED DESCRIPTION OF THE INVENTION

[0009] [Data Processing System S Overview] 1 is a diagram showing an overview of a data processing system S. The data processing system S is a system that provides a telexistence environment in which, for example, an operator U can operate a remote object in real time while making the operator U feel as if the object is nearby.

[0010] The data processing system S includes a data processing device 1, an operation device 2, and a robot 3. The data processing device 1 is a device, such as a computer, that controls the robot 3 in response to operations performed by an operator U on the operation device 2. The data processing device 1 may be installed in a room where either the operator U or the robot 3 is present, or may be installed in a location different from the location where the operator U is present and the location where the robot 3 is present.

[0011] The operation device 2 is a device worn by the operator U, and includes a display device 21, an operation device 22, and a communication unit 23. The display device 21 has a display (display device) that enables the operator U to view an image based on the composite image data generated by the data processing device 1, and is, for example, a goggle-type display device. The image displayed by the display device 21 is, for example, a celestial sphere image covering 360 degrees (all directions).

[0012] The robot 3 is a device that is remotely controlled by an operator U and operates based on control data received from the data processing device 1 via a network N. The robot 3 has an imaging device (hereinafter referred to as "camera 31") that generates captured image data, and an image transmission unit 32 that transmits the captured image data generated by the camera 31 to the data processing device 1. The robot 3 also has movable parts such as a head, hands, and arms that move in response to the movement of the operation device 2. The movable parts of the robot 3 move in synchronization with the position and orientation of the operation device 2.

[0013] Furthermore, the robot 3 transmits at least any one of robot status data, which is tactile data indicating tactile sensations detected by the robot 3, sound data indicating sounds collected by the robot 3, or joint status data indicating the status of the joints of the robot 3, to the operation device 2 via the data processing device 1. As an example, the robot 3 according to this embodiment performs the task of arranging products on shelves in response to the operation of an operator U in a store where many products are sold lined up on shelves, but the location where the robot 3 is installed and the content of the task are arbitrary.

[0014] The data processing device 1 acquires captured image data generated by the camera 31 capturing an image of the area ahead, and displays the acquired captured image data on the display device 21 of the operation device 2. When the operator U wears the display device 21, he or she cannot see the operation device 2 in hand. Therefore, the data processing device 1 generates composite image data by superimposing a controller image indicating the position and orientation of the operation device 22 on the captured image data generated by the camera 31. By having the data processing device 1 display the composite image data on the operation device 2, the operability for the operator U can be improved.

[0015] The display device 21 has a sensor that detects the angle of the display device 21 with respect to a reference orientation (hereinafter, sometimes referred to as the "device angle"). The reference orientation is, for example, the orientation of the display device 21 when the operator U is wearing the display device 21 and facing a predetermined orientation, and is, for example, the orientation at the time of start-up of the display device 21 or the orientation at the time of resetting. The display device 21 detects the angular difference between the reference orientation and the orientation of the display device 21 as the device angle. The device angle of the display device 21 is represented, for example, by a combination of two angles in a spherical coordinate system in three-dimensional space.

[0016] The display device 21 generates head operation data indicating the device angle of the display device 21 relative to a reference orientation. The display device 21 notifies the communication unit 23 of the head operation data at predetermined time intervals. The predetermined time interval is determined, for example, based on the speed at which the robot 3 can change the head angle, and the greater the speed at which the robot 3 can change the head angle, the shorter the predetermined time interval. Because the interval at which the display device 21 notifies the device angle corresponds to the speed at which the robot 3 can change the head angle, it is not necessary to detect the device angle more frequently than necessary, and therefore the power consumption of the operation device 2 can be reduced.

[0017] The operation device 22 (controller) is a device used by the operator U to operate the robot 3 with his hands and arms, and has sensors for detecting the movements of the operator U's hands and arms. The operation device 22 generates hand and arm operation data indicating the movements of the operator U's hands and arms. The operation device 22 notifies the communication unit 23 of the generated hand and arm operation data. The operation device 22 may accept an operation to move the robot 3 forward and backward or left and right. The operation device 22 also has elements for generating heat, pressure, vibration, or the like corresponding to the state of the robot 3 so that the operator U can sense the state of the robot 3.

[0018] The communication unit 23 has a communication controller that transmits operation data based on the content of the operation by the operator U to the data processing device 1 and receives composite image data from the data processing device 1. For example, the communication unit 23 synchronizes the head operation data notified from the display device 21 with the arm operation data notified from the operation device 22, and transmits operation data including the synchronized head operation data and arm operation data to the data processing device 1 at predetermined time intervals. The communication unit 23 also inputs the composite image data received from the data processing device 1 to the display device 21. The communication unit 23 may transmit operation data to the data processing device 1 for moving the robot 3 forward, backward, left, or right. The communication unit 23 may be included in either the display device 21 or the operation device 22, or may be housed in a housing different from that of the display device 21 and the operation device 22.

[0019] The robot 3 changes the orientation of its head to the same angle as the angle of the display device 21, based on head control data generated by the data processing device 1 based on head operation data transmitted from the operation device 2. The robot 3 also moves its hands and arms in the same way as the hands and arms of the operator U, based on arm control data generated by the data processing device 1 based on arm operation data transmitted from the operation device 2.

[0020] The camera 31 is provided, for example, on the head of the robot 3, and captures an image in front of the robot 3 to generate captured image data of a range including the movable parts of the robot 3. The camera 31 may also be an imaging device provided separately from the robot 3 at a position where it can capture an image of a range including the movable parts of the robot 3. The image transmission unit 32 has a communication controller for transmitting the captured image data via the network N. Although FIG. 1 illustrates an example in which the camera 31 has the image transmission unit 32, the image transmission unit 32 may be provided in any location.

[0021] An outline of the processing flow in the data processing system S will be described below with reference to Fig. 1. As described above, the operation device 2 generates operation data according to the movements of the head and arms of the operator U, and transmits the operation data to the data processing device 1.

[0022] The data processing device 1 generates control data for operating the robot 3 based on the operation data, and transmits the control data to the robot 3. The robot 3 operates based on the control data received from the data processing device 1. After operating based on the control data received from the data processing device 1, the robot 3 may transmit feedback information (e.g., ACK information) indicating that it has operated in accordance with the control data to the data processing device 1. While operating, the robot 3 transmits captured image data generated by the camera 31 to the data processing device 1 at predetermined time intervals (e.g., every 5 milliseconds).

[0023] The data processing device 1 transmits composite image data generated by superimposing a controller image indicating the position and orientation of the operation device 22 on an area corresponding to a movable part (for example, an arm of the robot 3) on the captured image data generated by the camera 31 to the operation device 2. The display device 21 displays the composite image data received from the data processing device 1. This allows the operator U to recognize the movable part of the robot 3 in the captured image data in association with the controller image corresponding to the operation device 22, even if the operator U is unable to see the operation device 22 in his or her hand, making it easier for the operator U to operate the robot 3.

[0024] [Configuration of data processing device 1] 2 is a diagram showing the configuration of the data processing device 1. The data processing device 1 has a communication unit 11, a storage unit 12, and a control unit 13. The control unit 13 has a controller information acquisition unit 131, an operation control unit 132, an image data acquisition unit 133, an identification unit 134, a synthesis unit 135, and a display control unit 136.

[0025] The communication unit 11 has a communication interface for transmitting and receiving various data between the operation device 2 and the robot 3 via the network N. The communication unit 11 inputs, for example, controller information received from the operation device 2 to the controller information acquisition unit 131. The communication unit 11 also inputs captured image data received from the robot 3 to the image data acquisition unit 133. The communication unit 11 also transmits control data input from the operation control unit 132 to the robot 3. The communication unit 11 also transmits composite image data input from the display control unit 136 to the operation device 2.

[0026] The storage unit 12 has storage media such as a ROM (Read Only Memory), a RAM (Random Access Memory), and an SSD (Solid State Drive). The storage unit 12 stores programs executed by the control unit 13. The storage unit 12 also temporarily stores captured image data received from the robot 3.

[0027] The control unit 13 has, for example, a CPU (Central Processing Unit) as a processor. The control unit 13 executes the programs stored in the storage unit 12, thereby functioning as a controller information acquisition unit 131, an operation control unit 132, an image data acquisition unit 133, an identification unit 134, a synthesis unit 135, and a display control unit 136.

[0028] The following describes in detail the processing executed by the data processing device 1. The following describes the processing by which the operator U operates the hands and arms of the robot 3 using the operation device 22, but the operator U may also operate the head of the robot 3 using the display device 21 in parallel.

[0029] The controller information acquisition unit 131 acquires controller information including the position and orientation of the operation device 22 when the operator U operates the movable parts of the robot 3 using the operation device 22 (controller).

[0030] 3 is a schematic diagram illustrating the operation device 22. The controller information acquisition unit 131 receives arm operation data from the operation device 22 via the communication unit 11, and acquires the received arm operation data as controller information including the position and orientation of the operation device 22. The controller information may also include information indicating whether or not one or more buttons of the operation device 22 are pressed. The controller information acquisition unit 131 inputs the acquired controller information to the operation control unit 132, the image data acquisition unit 133, and the identification unit 134.

[0031] The operation control unit 132 generates control data for operating the robot 3 based on the controller information input from the controller information acquisition unit 131. The control data is, for example, information indicating the amount and direction (i.e., vector) of movement or rotation of a movable part of the robot 3 according to the position and orientation of the operation device 22 indicated by the controller information. The operation control unit 132 may generate the control data by, for example, adding or subtracting a predetermined correction value to the value indicated by the controller information. The operation control unit 132 transmits the generated control data to the robot 3 via the communication unit 11.

[0032] The image data acquisition unit 133 acquires captured image data generated by the camera 31 capturing an image of the movable parts of the robot 3 while the movable parts of the robot 3 are being operated in accordance with the position and orientation of the operation device 22. The image data acquisition unit 133 inputs the acquired captured image data to the synthesis unit 135.

[0033] The identification unit 134 identifies the calculated position and orientation of the movable parts of the robot 3 based on the position and orientation of the operation device 22 included in the controller information. The position of the movable parts is, for example, the position of the wrist of the robot 3, and the orientation of the movable parts is, for example, the orientation of the hand based on the position of the wrist of the robot 3. The identification unit 134 calculates the calculated position and orientation of the movable parts of the robot 3, for example, by sequentially adding or subtracting the amount and direction (i.e., vector) of movement or rotation of the movable parts indicated by the control data generated by the operation control unit 132 to a predetermined initial position of the movable parts of the robot 3. The identification unit 134 is not limited to the specific method shown here, and may identify the calculated position and orientation of the movable parts of the robot 3 corresponding to the position and orientation of the operation device 22 by other methods.

[0034] The synthesis unit 135 generates a controller image indicating the position and orientation of the operation device 22 included in the controller information acquired by the controller information acquisition unit 131. Fig. 4 is a schematic diagram of the controller image IM1 generated by the synthesis unit 135. The controller image IM1 is an image indicating the position and orientation of at least one of the hand or wrist of the operator U holding the operation device 22, which corresponds to the position and orientation included in the controller information.

[0035] The synthesis unit 135 calculates the wrist center position A, for example, by adding or subtracting a predetermined correction value to the position of the operation device 22. The synthesis unit 135 calculates the hand direction B and the wrist angle C, for example, by adding or subtracting a predetermined correction value to the orientation of the operation device 22. The synthesis unit 135 generates a controller image IM1 that represents the calculated wrist center position A, hand direction B, and wrist angle C.

[0036] 4 shows the central position A of the wrist by the position of the spherical part, the hand direction B by the direction of the tip part connected to the spherical part, and the wrist angle C by the rotation angle of the spherical part. The controller image IM1 is not limited to the specific method shown here, and may show the central position A of the wrist, the hand direction B, and the wrist angle C by other methods.

[0037] The synthesis unit 135 generates synthesized image data by superimposing the generated controller image on an area on the captured image data that corresponds to the movable part. The area on the captured image data that corresponds to the movable part is coordinates on the captured image data that correspond to the calculated position and orientation of the movable part identified by the identification unit 134. The synthesis unit 135 calculates coordinates on the captured image data, for example, by converting the calculated position and orientation of the movable part according to a predetermined rule. The synthesis unit 135 superimposes the controller image on the calculated coordinates on the captured image data. This allows the data processing device 1 to allow the operator U to recognize the position and orientation of the operation device 22 even when the operator U cannot see the operation device 22.

[0038] When the movable part moves, that is, when the position or orientation of the movable part is changed, the synthesis unit 135 moves the controller image to an area on the captured image data that corresponds to the moved movable part. This allows the data processing device 1 to make the controller image follow the movable part, and allows the operator U to recognize that the controller image is being displayed according to the position and orientation of the operation device 22.

[0039] Furthermore, the synthesis unit 135 superimposes an operation image indicating a state in which the operation device 22 is being operated on the captured image data in association with the controller image. For example, the synthesis unit 135 superimposes an operation image representing each of one or more buttons of the operation device 22 near the controller image, and changes the display mode of the operation image depending on whether each of the one or more buttons is pressed based on the controller information. In this way, the data processing device 1 can allow the operator U to recognize the operation state of the operation device 22 even when the operator U cannot see the operation device 22. Furthermore, the data processing device 1 can allow the operator U to recognize the position, orientation, and operation state of the movable parts of the robot 3 by showing the controller image and the operation image to the operator U even when the movable parts of the robot 3 are hidden by the body of the robot 3 or other objects and cannot be seen.

[0040] The display control unit 136 transmits display data including the composite image data generated by the synthesis unit 135 to the operation device 2 via the communication unit 11, thereby causing the composite image data to be displayed on the display device 21 that is visible to the operator U.

[0041] 5 is a schematic diagram of the composite image data IM2 displayed by the display device 21. In the composite image data IM2, a controller image IM1 is displayed in an area corresponding to the movable part P. In addition, in the composite image data IM2, an operation image IM3 is displayed near the controller image IM1. Even if the operator U is unable to see the operation device 22, by looking at the composite image data IM2, the operator U can easily recognize the position and orientation of the operation device 22, as well as the operation state of the operation device 22.

[0042] [Data processing method sequence] 6 is a sequence diagram of a data processing method executed by the data processing system S. When the operator U operates the movable parts of the robot 3 using the operation device 22, the operation device 22 generates operation data (hand and arm operation data) indicating the movements of the hands and arms of the operator U (S1). The operation device 22 transmits the generated operation data to the data processing device 1 via the communication unit 23.

[0043] In the data processing device 1, the controller information acquisition unit 131 receives operation data from the operation device 22 via the communication unit 11, and acquires the received operation data as controller information including the position and orientation of the operation device 22. The controller information acquisition unit 131 inputs the acquired controller information to the operation control unit 132, the image data acquisition unit 133, and the identification unit 134.

[0044] The operation control unit 132 generates control data for operating the robot 3 based on the controller information input from the controller information acquisition unit 131 (S2). The operation control unit 132 transmits the generated control data to the robot 3 via the communication unit 11.

[0045] The robot 3 receives control data from the data processing device 1 and operates the movable parts in accordance with the received control data. After operating the movable parts in accordance with the control data received from the data processing device 1, the robot 3 may transmit feedback information (e.g., ACK information) indicating that the operation has been performed in accordance with the control data to the data processing device 1. The data processing device 1 stores the feedback information received from the robot 3 in the storage unit 12.

[0046] The camera 31 captures an image of the movable part while the movable part is being operated, thereby generating captured image data (S3). The image transmission unit 32 transmits the captured image data generated by the camera 31 to the data processing device 1.

[0047] In the data processing device 1, the image data acquisition unit 133 acquires captured image data generated by the camera 31 from the robot 3. The identification unit 134 identifies the calculated position and orientation of the movable parts of the robot 3 based on the position and orientation of the operation device 22 included in the controller information (S4).

[0048] The synthesis unit 135 generates a controller image indicating the position and orientation of the operation device 22 included in the controller information acquired by the controller information acquisition unit 131. The synthesis unit 135 generates synthesized image data by superimposing the generated controller image on an area on the captured image data that corresponds to the movable part (S5).

[0049] The display control unit 136 transmits display data including the composite image data generated by the composition unit 135 to the operation device 2 via the communication unit 11. In the operation device 2, the display device 21 displays the composite image data received from the data processing device 1 via the communication unit 23 (S6).

[0050] [Data processing method flow] Fig. 7 is a flowchart of the data processing method executed by the data processing device 1. The flowchart shown in Fig. 7 starts when the power of the controller device 2 is turned on.

[0051] The controller information acquisition unit 131 receives operation data from the operation device 22 via the communication unit 11, and acquires the received operation data as controller information including the position and orientation of the operation device 22 (S11). The image data acquisition unit 133 acquires captured image data generated by the camera 31 from the robot 3 (S12). S11 and S12 may be performed in the reverse order or in parallel.

[0052] The identification unit 134 identifies the calculated position and orientation of the movable parts of the robot 3 based on the position and orientation of the operation device 22 included in the controller information (S13). The synthesis unit 135 generates a controller image indicating the position and orientation of the operation device 22 included in the controller information acquired by the controller information acquisition unit 131 (S14). The synthesis unit 135 generates synthesized image data by superimposing the generated controller image on an area of ​​the captured image data corresponding to the movable parts (S15). The synthesis unit 135 also superimposes operation images representing one or more buttons of the operation device 22 near the controller image.

[0053] If the controller information indicates that one or more buttons are pressed on the operation device 22 (YES in S16), the synthesis unit 135 changes the display mode of the operation image corresponding to the button based on the controller information (S17). If the controller information does not indicate that one or more buttons are pressed on the operation device 22 (NO in S16), the synthesis unit 135 does not change the display mode of the operation image.

[0054] The display control unit 136 transmits display data including the composite image data generated by the composition unit 135 to the operation device 2 via the communication unit 11 (S18). The control unit 13 repeats the processes from S11 to S18 until the operator U performs an operation to end the operation on the operation device 2 (NO in S19). When the operator U performs an operation to end the operation (YES in S19), the control unit 13 ends the processes.

[0055] [Effects of the embodiment] In the data processing system S according to this embodiment, the data processing device 1 displays composite image data generated by superimposing a controller image indicating the position and orientation of the operation device 22 on an area corresponding to the movable parts on the captured image data generated by the camera 31, on the display device 21. This allows the operator U to recognize the movable parts of the robot 3 in relation to the controller image corresponding to the operation device 22 in the captured image data, even if the operator U cannot see the operation device 22 in his or her hand, making it easier to operate the robot 3.

[0056] [First Modification] In the above-described embodiment, there are errors in the motors, gears, etc. used to operate the robot 3, so the calculated position of the movable part calculated based on the position and orientation of the operating device 22 may deviate from the actual position of the movable part. The data processing system S according to this modification can make it easier for the operator U to operate the robot 3 by displaying the deviation (difference) between the actual position of the movable part and the calculated position of the movable part. The following mainly describes the parts that are different from the above-described embodiment.

[0057] In addition to determining the calculated position and orientation of the movable parts of the robot 3 using the above-described method, the determination unit 134 also determines the actual position and orientation of the movable parts of the robot 3. The determination unit 134, for example, executes a known image recognition process to extract areas in the captured image data that resemble image patterns of the movable parts in various predefined orientations. The determination unit 134 determines the position of the area in the captured image data that resembles a specific image pattern as the actual position of the movable part, and determines the orientation corresponding to the image pattern as the actual orientation of the movable part.

[0058] Furthermore, the identification unit 134 may calculate the position and orientation of the movable part using, for example, a sensor provided on the operating device 2 or the movable part, and identify the calculated position and orientation as the actual position and orientation. The sensor for identifying the position of the movable part is, for example, an angle sensor provided on each of the multiple joints of the movable part. The identification unit 134 acquires the angle of each of the multiple joints of the movable part from the angle sensor, and identifies the actual position and orientation of the movable part based on the acquired angle and a predefined structure of the movable part (such as the length of the arm).

[0059] The determination unit 134 determines the difference between the actual position and orientation of the movable part and the calculated position and orientation of the movable part. For example, the determination unit 134 determines the vector from the actual position of the movable part to the calculated position of the movable part as the position difference. Furthermore, the determination unit 134 determines the angle between the actual orientation of the movable part and the calculated orientation of the movable part as the orientation difference.

[0060] The display control unit 136 displays the difference between the actual position and orientation of the movable part and the calculated position and orientation of the movable part in association with the controller image on the composite image data displayed on the display device 21. For example, the display control unit 136 displays a difference image including a message indicating the amount and direction of the difference between the actual position and orientation of the movable part and the calculated position and orientation of the movable part by superimposing it on the composite image data. Furthermore, the synthesis unit 135 may superimpose the controller image on coordinates on the captured image data that correspond to the actual position and orientation of the movable part, instead of the calculated position and orientation of the movable part.

[0061] 8 is a schematic diagram of the composite image data IM2 displayed by the display device 21. In the composite image data IM2, a difference image IM4 including a message indicating the difference is displayed near the controller image IM1. The display control unit 136 may display the difference by displaying an arrow indicating the amount and direction of the difference instead of or in addition to the message, or by changing the color of the controller image IM1 depending on the magnitude of the difference. By looking at the difference image IM4, the operator U can easily recognize the discrepancy between the actual position and orientation of the movable part and the calculated position and orientation of the movable part.

[0062] The operation control unit 132 may operate the movable parts according to the position and orientation included in the controller information, on the condition that the difference is equal to or less than a predetermined value. That is, the operation control unit 132 allows the operator U to control the movement of the robot 3 when the difference is equal to or less than the predetermined value, and does not allow the operator U to control the movement of the robot 3 when the difference is greater than the predetermined value. This allows the data processing device 1 to prevent the robot 3 from moving when there is a large difference between the actual position and orientation of the movable parts and the calculated position and orientation of the movable parts, thereby improving safety.

[0063] Furthermore, the display control unit 136 may change the display mode (color, pattern, etc.) of the composite image data between a state in which the operator U is permitted to control the movement of the robot 3 and a state in which the operator U is not permitted to control the movement of the robot 3. This allows the data processing device 1 to make the operator U aware that the robot 3 should not be moved if the deviation is large.

[0064] After displaying the difference on the display device 21, the data processing device 1 may accept a correction of the difference from the operator U. For example, when the operator U performs an operation to start the correction on the operation device 22, the operation control unit 132 temporarily stops control of the movable parts, and the controller information acquisition unit 131 accepts an operation to move the position of the controller image from the operation device 2. When the position of the controller image matches the position of the movable parts, the operator U performs an operation to end the correction on the operation device 22. The controller information acquisition unit 131 stores the amount and direction of movement of the controller image during the correction in the storage unit 12 as the correction amount.

[0065] Furthermore, for example, when the operator U performs an operation to start correction on the operation device 22, the synthesis unit 135 may temporarily fix the position of the controller image, and the controller information acquisition unit 131 may accept an operation to move the movable part from the operation apparatus 2. With the position of the movable part coinciding with the position of the controller image, the operator U performs an operation to end the correction on the operation device 22. The controller information acquisition unit 131 stores the amount and direction by which the movable part has moved during correction in the storage unit 12 as the correction amount.

[0066] Thereafter, the operation control unit 132 generates control data by adding or subtracting the correction amount stored in the storage unit 12 to the position and orientation indicated by the controller information. This allows the data processing device 1 to correct the discrepancy between the actual position and orientation of the movable part and the calculated position and orientation of the movable part in accordance with the operation by the operator U.

[0067] Furthermore, the data processing device 1 may automatically correct the position and orientation of the operation device 22 indicated by the controller information in accordance with the difference identified by the identification unit 134. The difference correction may involve correcting the movement of the robot 3 or correcting the position of the controller image. When correcting the movement of the robot 3, the operation control unit 132 generates control data by, for example, adding or subtracting the value of the difference identified by the identification unit 134 to the position and orientation indicated by the controller information. This causes the movement of the robot 3 to be corrected in accordance with the difference.

[0068] When correcting the position of the controller image, the synthesis unit 135 superimposes the controller image at a position obtained by adding or subtracting the difference value identified by the identification unit 134 to or from the coordinates on the captured image data corresponding to the calculated position and orientation of the movable part. This corrects the position of the controller image according to the difference. This allows the data processing device 1 to automatically correct the discrepancy between the actual position and orientation of the movable part and the calculated position and orientation of the movable part.

[0069] [Second Modification] In the above-described embodiment, the data processing device 1 displays on the display device 21 an operation image showing a state in which the operation device 22 is being operated, whereas in this modification, the data processing device 1 displays on the display device 21 a suggested image showing the operation to be performed next on the operation device 22. The following mainly describes the parts that are different from the above-described embodiment.

[0070] The storage unit 12 pre-stores planned work information that associates one or more work contents that the operator U is scheduled to perform with operations to be performed on the robot 3 for each of the one or more work contents. The identification unit 134 acquires the planned work information from the storage unit 12 before the synthesis unit 135 generates synthetic image data. The identification unit 134 identifies the next work content that the operator U is scheduled to perform based on the planned work information.

[0071] The identification unit 134 identifies the next scheduled work content from one or more work contents indicated by the scheduled work information, for example, by receiving an input of the progress status of the work from the operation device 22. The identification unit 134 identifies an operation associated with the next scheduled work content in the scheduled work information.

[0072] The composition unit 135 superimposes an operation image indicating a state in which the operation device 22 is being operated, in association with the controller image, on the captured image data. Furthermore, the composition unit 135 superimposes a suggested image indicating the next operation to be performed on the operation device 22, in accordance with the work content identified by the identification unit 134, on the captured image data. The next operation to be performed on the operation device 22 is, for example, an operation to change the hand part of the robot 3 to be used in the work content scheduled to be performed next. The suggested image is, for example, an image displaying a message explaining the next operation to be performed on the operation device 22. Furthermore, the composition unit 135 may superimpose work information indicating the work content scheduled to be performed next on the captured image data.

[0073] The display control unit 136 transmits display data including the composite image data generated by the synthesis unit 135 to the operation device 2 via the communication unit 11, thereby causing the composite image data to be displayed on the display device 21 that is visible to the operator U.

[0074] 9 is a schematic diagram of composite image data IM2 displayed by the display device 21. In the composite image data IM2, an operation image IM3 is displayed near the controller image IM1. In addition, in the composite image data IM2, task information IM5 indicating the content of the next task to be performed is displayed, and a suggested image IM6 indicating the next operation to be performed on the operation device 22 is displayed near the operation image IM3. This allows the data processing device 1 to suggest to the operator U an operation according to the content of the next task to be performed, thereby facilitating the progress of the task.

[0075] The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, all or part of the device can be configured by functionally or physically distributing or integrating any unit. Furthermore, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effects of the new embodiments resulting from the combination also have the effects of the original embodiments.

Claims

1. a data processing device that processes data for operating a movable part of the device according to a position of a controller, acquiring controller information including the position when an operator operates the movable part using the controller; an imaging device acquiring captured image data generated by capturing an image of the movable part while the movable part is being operated according to the position; generating composite image data by superimposing a controller image indicating the position included in the controller information on an area of ​​the captured image data corresponding to the movable part; displaying the composite image data on a display device visible to the operator; determining a difference between an actual position of the moving part and a calculated position of the moving part calculated based on the position included in the controller information; 1. A data processing apparatus having a controller configured to perform the steps of:

2. The control unit obtaining the controller information including an orientation of the controller in addition to the position; generating the composite image data by superimposing the controller image indicating the position and the orientation included in the controller information; 2. The data processing device according to claim 1.

3. the controller image is an image indicating a direction of at least one of a hand or a wrist of the operator holding the controller, which direction corresponds to the direction included in the controller information.

3. The data processing device according to claim 2.

4. when the movable part has moved, the control unit superimposes the controller image on an area of ​​the captured image data that corresponds to the movable part after the movement; 2. The data processing device according to claim 1.

5. the control unit superimposes, on the captured image data, an image indicating a state in which the controller is being operated, in association with the controller image.

2. The data processing device according to claim 1.

6. the control unit superimposes, on the captured image data, an image indicating an operation to be performed next by the controller in accordance with the content of the work that the operator is scheduled to perform next.

2. The data processing device according to claim 1.

7. The control unit obtaining the controller information including an orientation of the controller in addition to the position; determining the difference between the actual position and orientation of the movable part and a calculated position and orientation of the movable part calculated based on the position and orientation included in the controller information; 2. The data processing device according to claim 1.

8. the control unit causes the display device to display the difference in association with the controller image.

2. The data processing device according to claim 1.

9. the control unit is further configured to correct the position included in the controller information corresponding to the difference, and then operate the movable part in accordance with the position included in the controller information.

2. The data processing device according to claim 1.

10. the control unit operates the movable part in accordance with the position included in the controller information, on condition that the difference is equal to or smaller than a predetermined value.

10. The data processing device according to claim 9.

11. A data processing method in which a computer processes data for operating a movable part of a device according to a position of a controller, comprising: acquiring controller information including the position when an operator operates the movable part using the controller; an imaging device acquiring captured image data generated by capturing an image of the movable part while the movable part is being operated according to the position; generating composite image data by superimposing a controller image indicating the position included in the controller information on an area of ​​the captured image data corresponding to the movable part; displaying the composite image data on a display device visible to the operator; determining a difference between an actual position of the moving part and a calculated position of the moving part calculated based on the position included in the controller information; Data processing methods, including:

12. A program for causing a computer to execute a data processing method for processing data for operating a movable part of a device according to a position of a controller, The data processing method includes: acquiring controller information including the position when an operator operates the movable part using the controller; an imaging device acquiring captured image data generated by capturing an image of the movable part while the movable part is being operated according to the position; generating composite image data by superimposing a controller image indicating the position included in the controller information on an area of ​​the captured image data corresponding to the movable part; displaying the composite image data on a display device visible to the operator; determining a difference between an actual position of the moving part and a calculated position of the moving part calculated based on the position included in the controller information; Including, the program.

13. 1. A data processing system comprising: an operating device associated with a controller for operating an apparatus; and a data processing device for processing data for operating a movable part of the apparatus according to a position of the controller, The data processing device includes: acquiring controller information including the position when an operator operates the movable part using the controller; an imaging device acquiring captured image data generated by capturing an image of the movable part while the movable part is being operated according to the position; generating composite image data by superimposing a controller image indicating the position included in the controller information on an area of ​​the captured image data corresponding to the movable part; transmitting the composite image data to the operation device; determining a difference between an actual position of the moving part and a calculated position of the moving part calculated based on the position included in the controller information; a controller configured to perform The operating device is a communication unit that transmits the controller information based on the content of the operation by the operator to the data processing device; a display unit that displays the composite image data transmitted by the data processing device and is visible to the operator; having Data processing system.

Images