Data processing device, data processing method, program and data processing system

Abstract

A data processing device 1 according to an embodiment of the present disclosure is provided with a control unit 13 configured to perform: acquiring captured image data generated on the basis of capturing of an image by a camera 31 provided in a robot 3; synthesizing display image data comprising at least a part of the captured image data acquired at a first point in time, and past captured image data comprising at least a part of the captured image data acquired at a second point in time earlier than the first point in time, to generate synthesized image data having a view field greater than that of the display image data; and causing the synthesized image data to be displayed on a display device 21 viewable by an operator U of the robot 3.

Description

[Technical Field]

[0001] The present disclosure relates to a data processing device, a data processing method, a program, and a data processing system. [Background technology]

[0002] Patent Document 1 discloses a technique for displaying captured images generated by an imaging device provided in a robot on a terminal used by an operator who remotely controls the robot. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-040155 Summary of the Invention [Problem to be solved by the invention]

[0004] When an operator operates a robot while looking at an image generated by an imaging device installed on the robot, if the range that the imaging device installed on the robot can capture is narrower than the range of the operator's field of view, there is a problem that the operator has difficulty operating the robot.

[0005] The present disclosure has been made in consideration of this point, and aims to provide a data processing device and the like that makes it easier for a robot operator to operate the robot. [Means for solving the problem]

[0006] A data processing device according to one embodiment of the present disclosure has a control unit configured to acquire captured image data generated based on imaging by an imaging device provided on a robot, generate composite image data having a wider angle of view than the display image data by combining display image data of at least a portion of the captured image data acquired at a first time point with past captured image data of at least a portion of the captured image data acquired at a second time point prior to the first time point, and display the composite image data on a display device visible to an operator of the robot. [Effects of the Invention]

[0007] According to the present disclosure, a data processing device and the like are provided that makes it easier for a robot operator to operate the robot. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an overview of a data processing system. [Figure 2] FIG. 10 is a diagram for explaining the device angle of the display device. [Figure 3] FIG. 2 is a diagram showing the relationship between captured image data and composite image data. [Figure 4] FIG. 1 is a diagram illustrating a configuration of a data processing device. [Figure 5] 10A and 10B are diagrams illustrating a method for storing captured image data in a storage unit. [Figure 6] FIG. 2 is a sequence diagram showing a processing flow of the data processing system. [Figure 7] 10 is a flowchart showing a processing flow in the data processing device. [Figure 8] FIG. 10 is a diagram for explaining the operation of the data processing device in the first modified example. [Figure 9] FIG. 10 is a diagram for explaining the operation of the data processing device in the second modified example. DETAILED DESCRIPTION OF THE INVENTION

[0009] [Data Processing System S Overview] Figure 1 is a diagram showing an overview of a data processing system S. The data processing system S is a system that provides a telexistence environment in which an operator U can operate objects in remote locations in real time while making the operator U feel as if the objects are nearby.

[0010] The data processing system S includes a data processing device 1, an operation device 2, and a robot 3. The data processing device 1 is a device, such as a computer, that controls the robot 3 in response to operations performed by an operator U on the operation device 2. The data processing device 1 may be installed in a room where either the operator U or the robot 3 is present, or may be installed in a location different from the location where the operator U is present and the location where the robot 3 is present.

[0011] The operation device 2 is a device worn by the operator U, and includes a display device 21, an operation device 22, and a communication unit 23. The display device 21 has a display that enables the operator U to view an image based on the composite image data generated by the data processing device 1, and is, for example, a goggle-type display device. The image displayed by the display device 21 is, for example, a celestial sphere image covering 360 degrees (all directions).

[0012] The robot 3 operates based on control data received from the data processing device 1 via the network N. The robot 3 has an imaging device (hereinafter referred to as "camera 31") that generates captured image data, and an image transmission unit 32 that transmits the captured image data generated by the camera 31 to the data processing device 1. Furthermore, the robot 3 transmits at least one of robot status data, which is tactile data indicating tactile sensations detected by the robot 3, sound data indicating sounds collected by the robot 3, or joint status data indicating the status of the joints of the robot 3, to the operation device 2 via the data processing device 1. As an example, the robot 3 according to this embodiment performs the task of arranging products on shelves in response to the operation of an operator U in a store where a large number of products are sold lined up on shelves, but the location where the robot 3 is installed and the content of the task are arbitrary.

[0013] The data processing device 1 acquires captured image data generated by the robot 3 by capturing an image of the area ahead using the camera 31, and displays the acquired captured image data on the display device 21 of the operation device 2. Because the angle of view of the captured image data generated by the robot 3 is narrower than the field of view of the operator U, displaying only the captured image data on the display device 21 would result in poor operability for the operator U. Therefore, the data processing device 1 generates composite image data by combining the captured image data generated by the robot 3 with previously generated captured image data that corresponds to the area surrounding the imaging area corresponding to the captured image data. By having the data processing device 1 display the composite image data on the operation device 2, the operability for the operator U can be improved.

[0014] The display device 21 has a sensor that detects the angle of the display device 21 with respect to a reference orientation (hereinafter, sometimes referred to as the "device angle"). The reference orientation is, for example, the orientation of the display device 21 when the operator U is wearing the display device 21 and facing a predetermined orientation, and is, for example, the orientation of the display device 21 at the time of startup or reset. The display device 21 detects the angular difference between the reference orientation and the orientation of the display device 21 as the device angle.

[0015] The device angle of display device 21 is expressed, for example, by a combination of two angles in a spherical coordinate system in three-dimensional space. FIG. 2 is a diagram for explaining the device angle of display device 21. The device angle of position G on the surface of the sphere shown in FIG. 2 is expressed as (θ, φ). θ is the angle between the X-axis direction and a line connecting position H, which is a projection of position G onto the XY plane, and origin O. φ is the angle between the Z-axis direction and the direction of the line connecting origin O and position G.

[0016] The positive direction in the X-axis direction corresponds to the front orientation of the display device 21 (i.e., the front orientation of the operator U) at the time the display device 21 is started up, and when the operator U is facing forward, the device angle of the display device 21 is expressed as (0,0). When the operator U is facing directly left (i.e., the positive direction in the Y-axis direction), the device angle of the display device 21 is expressed as (90,0).

[0017] The display device 21 generates head operation data indicating the device angle of the display device 21 relative to a reference orientation. The display device 21 notifies the communication unit 23 of the head operation data at predetermined time intervals. The predetermined time interval is determined, for example, based on the speed at which the robot 3 can change the head angle, and the greater the speed at which the robot 3 can change the head angle, the shorter the predetermined time interval. Because the interval at which the display device 21 notifies the device angle corresponds to the speed at which the robot 3 can change the head angle, it is not necessary to detect the device angle more frequently than necessary, and therefore the power consumption of the operation device 2 can be reduced.

[0018] The operation device 22 is a device used by the operator U to operate the robot 3 with his hands and arms, and has sensors for detecting the movements of the operator U's hands and arms. The operation device 22 generates hand and arm operation data indicating the movements of the operator U's hands and arms. The operation device 22 notifies the communication unit 23 of the generated hand and arm operation data. The operation device 22 may accept an operation to move the robot 3 forward and backward or left and right. The operation device 22 also has elements for generating heat, pressure, vibration, or the like corresponding to the state of the robot 3 so that the operator U can sense the state of the robot 3.

[0019] The communication unit 23 has a communication controller that transmits operation data based on the content of the operation by the operator U to the data processing device 1 and receives composite image data from the data processing device 1. For example, the communication unit 23 synchronizes the head operation data notified from the display device 21 with the arm operation data notified from the operation device 22, and transmits operation data including the synchronized head operation data and arm operation data to the data processing device 1 at predetermined time intervals. The communication unit 23 also inputs the composite image data received from the data processing device 1 to the display device 21. The communication unit 23 may transmit operation data to the data processing device 1 for moving the robot 3 forward, backward, left, or right. The communication unit 23 may be included in either the display device 21 or the operation device 22, or may be housed in a housing different from that of the display device 21 and the operation device 22.

[0020] The robot 3 changes the orientation of its head to the same angle as the angle of the display device 21, based on head control data generated by the data processing device 1 based on head operation data transmitted from the operation device 2. The robot 3 also moves its hands and arms in the same way as the hands and arms of the operator U, based on hand and arm control data generated by the data processing device 1 based on hand and arm operation data transmitted from the operation device 2.

[0021] The camera 31 is provided, for example, on the head, and generates captured image data by capturing an image in front of the robot 3. The image transmission unit 32 has a communication controller for transmitting the captured image data via the network N. Although FIG. 1 illustrates an example in which the camera 31 has the image transmission unit 32, the location where the image transmission unit 32 is provided is arbitrary.

[0022] The direction of the optical axis of the camera 31 changes as the direction of the head of the robot 3 changes based on the head control data, and the imaging area changes accordingly. In the following explanation, the angle of the direction of the line connecting the center of rotation of the head of the robot 3 (corresponding to the origin in FIG. 2) and the center position of the area to be imaged, relative to the direction of the front, is referred to as the "imaging angle." The robot 3 moves the head of the robot 3 so that the angle of the head of the operator U indicated by the head operation data (i.e., the device angle) matches the imaging angle.

[0023] An outline of the processing flow in the data processing system S will be explained below with reference to Fig. 1. As described above, the operation device 2 generates operation data according to the movements of the head and arms of the operator U and transmits the operation data to the data processing device 1. The data processing device 1 generates control data for operating the robot 3 based on the operation data and transmits the control data to the robot 3. The robot 3 operates based on the control data received from the data processing device 1.

[0024] While the robot 3 is operating, it transmits captured image data generated by the camera at predetermined time intervals (for example, every 5 milliseconds) to the data processing device 1. The robot 3 may transmit the captured image data to the data processing device 1 in association with information indicating the angle of the head of the robot 3 at the time the captured image data was generated.

[0025] The data processing device 1 generates composite image data by combining the captured image data received from the robot 3 with previously captured image data corresponding to an area surrounding the captured image area of ​​the captured image data, and transmits the composite image data to the operation device 2, and the display device 21 displays the composite image data. Since the composite image data includes image data of a wider range than the range that the robot 3 can capture, the operator U can have the same field of view as if he or she were actually looking at the work site where the robot 3 is working, making it easier to operate the robot 3.

[0026] 3A and 3B are diagrams illustrating the relationship between captured image data and composite image data. Image G1 in Fig. 3A is an image based on display image data generated by the robot 3 at a first time point, and shows a state in which bottles are lined up on a shelf. Images G2 and G3 in Fig. 3A are images based on captured image data corresponding to imaging areas adjacent to both sides of the imaging area of ​​the display image data, among multiple past captured image data generated by the robot 3 at a second time point prior to the first time point.

[0027] Fig. 3(b) shows a composite image obtained by combining display image data G1 with previously captured image data G2 and previously captured image data G3. Compared to when the operator U looks at image G1 shown in Fig. 3(a), when the operator U looks at the composite image shown in Fig. 3(b), the field of view of the operator U is wider, making it easier for the operator U to work with the robot 3. The configuration and operation of the data processing device 1 will be described in detail below.

[0028] [Configuration of data processing device 1] 4 is a diagram showing the configuration of the data processing device 1. The data processing device 1 has a communication unit 11, a storage unit 12, and a control unit 13. The control unit 13 has an operation data acquisition unit 131, a control data generation unit 132, an image data acquisition unit 133, a synthesis unit 134, and a display control unit 135.

[0029] The communication unit 11 has a communication interface for transmitting and receiving various data between the operation device 2 and the robot 3 via the network N. The communication unit 11 inputs, for example, operation data received from the operation device 2 to the operation data acquisition unit 131. The communication unit 11 also inputs captured image data received from the robot 3 to the image data acquisition unit 133. The communication unit 11 also transmits composite image data input from the display control unit 135 to the operation device 2.

[0030] The storage unit 12 has storage media such as a ROM (Read Only Memory), a RAM (Random Access Memory), and an SSD (Solid State Drive). The storage unit 12 stores programs executed by the control unit 13. The storage unit 12 also temporarily stores captured image data received from the robot 3. The storage unit 12 stores the captured image data acquired by the image data acquisition unit 133 in association with a device angle, which is the angle of the display device 21 when the captured image data was captured. Since the angle of the display device 21 is equivalent to the angle of the head of the robot 3, the storage unit 12 may store the captured image data in association with the angle of the head of the robot 3 at the time the captured image data was generated.

[0031] Fig. 5 is a diagram showing a method for storing captured image data in the storage unit 12. In Fig. 5, the device angle (θ, φ) of the display device 21 at the time when the robot 3 generated the captured image data is associated with the file name of the captured image data. Every time the communication unit 11 receives new captured image data from the robot 3, the captured image data stored in the storage unit 12 in association with the angle of the display device 21 at the time when the received captured image data was generated is updated. The angle of the display device 21 and the angle of the head of the robot 3 can be angles corresponding to any direction in three-dimensional space, so captured image data corresponding to 360 degrees (all directions) is stored in the storage unit 12.

[0032] 4, the control unit 13 will be described. The control unit 13 has, for example, a CPU (Central Processing Unit) as a processor. The control unit 13 executes programs stored in the storage unit 12, thereby functioning as an operation data acquisition unit 131, a control data generation unit 132, an image data acquisition unit 133, a synthesis unit 134, and a display control unit 135.

[0033] The operation data acquisition unit 131 acquires operation data indicating the operation content of the operator U via the communication unit 11. The operation data acquisition unit 131 acquires, for example, head operation data indicating the angle of the display device 21 and hand / arm operation data indicating the movement of the hand / arm of the operator U. The operation data acquisition unit 131 inputs the acquired operation data to the control data generation unit 132, the image data acquisition unit 133, and the synthesis unit 134.

[0034] The control data generation unit 132 generates control data for operating the robot 3 based on the operation data input from the operation data acquisition unit 131. The control data generation unit 132 may generate the control data by, for example, adding or subtracting a predetermined correction value to a value indicated by the operation data. The control data generation unit 132 transmits the generated control data to the robot 3 via the communication unit 11.

[0035] The image data acquisition unit 133 acquires captured image data generated based on image capture by the camera 31 provided on the robot 3 via the communication unit 11. The image data acquisition unit 133 stores the acquired captured image data in the storage unit 12. For example, as shown in FIG. 5, the image data acquisition unit 133 stores the captured image data in the storage unit 12 in association with the angle of the display device 21 indicated by the operation data at the time the captured image data was acquired.

[0036] When the image data acquisition unit 133 acquires the captured image data, it updates the captured image data stored in the storage unit 12 in association with the angle of the display device 21 indicated by the operation data at the time the captured image data was captured, to the captured image data acquired by the image data acquisition unit 133. That is, when the image data acquisition unit 133 acquires captured image data at an angle for which captured image data is already stored in the storage unit 12, it updates the captured image data stored in the storage unit 12 to the newly acquired captured image data. Note that, when the captured image data is associated with the angle of the head of the robot 3 at the time the robot 3 generated the captured image data, the image data acquisition unit 133 may store the captured image data in the storage unit 12 in association with the angle of the head of the robot 3.

[0037] However, when the position of the robot 3 changes, the area included in the captured image data generated by the robot 3 changes, and therefore, if past captured image data is combined after the position of the robot 3 has changed, inconsistencies will occur at the boundary lines of the combined multiple captured image data. Therefore, when the position of the robot 3 has moved, the image data acquisition unit 133 may initialize the captured image data stored in the storage unit 12.

[0038] For example, when detecting that an operation to move the position of the robot 3 has been performed based on the movement operation data acquired by the operation data acquisition unit 131, the image data acquisition unit 133 initializes the captured image data stored in the storage unit 12. For example, the image data acquisition unit 133 initializes the captured image data by deleting the captured image data stored in the storage unit 12. The image data acquisition unit 133 may delete the file name of the captured image data associated with the device angle as shown in FIG. 5 without deleting the captured image data. By the image data acquisition unit 133 initializing the captured image data when the position of the robot 3 has moved, it is possible to prevent an inconsistent composite image from being displayed on the display device 21.

[0039] Furthermore, even if the robot 3 is in the same position, the situation around the robot 3 may change over time. If past captured image data is combined despite the change in the situation around the robot 3, inconsistencies will occur at the boundary lines of the combined captured image data. Therefore, the image data acquisition unit 133 may initialize the captured image data stored in the storage unit 12 when the operator U finishes operating the robot 3 or when a predetermined time has passed. The predetermined time is a length of time during which the situation around the robot 3 may change, for example, one minute.

[0040] Whether the situation around the robot 3 changes may depend on the date, time, or day of the week. Therefore, the image data acquisition unit 133 may initialize the captured image data stored in the storage unit 12 when a predetermined time associated with at least either the date, time, or day of the week has elapsed.

[0041] Furthermore, the image data acquisition unit 133 may initialize the captured image data stored in the storage unit 12 when receiving an instruction to initialize the captured image data stored in the storage unit 12, which is input by the operator U via the operation device 22. By operating the image data acquisition unit 133 in this manner, the captured image data can be initialized at the operator U's discretion when the operator U begins to feel a decline in the quality of the composite image data.

[0042] Furthermore, while the operator U is not operating the robot 3, the situation around the robot 3 may change, or someone may move the robot 3. Therefore, when the operator U resumes operating the robot 3, it is expected that the captured image data stored in the storage unit 12 will not be consistent with the captured image data generated by the robot 3. Therefore, the image data acquisition unit 133 may initialize the captured image data stored in the storage unit 12 when the operator U logs out to end operation of the robot 3, or when the power of the robot 3 is turned off.

[0043] The combining unit 134 generates composite image data as shown in Fig. 3(b). Specifically, the combining unit 134 generates composite image data having a wider angle of view than the display image data by combining display image data, which is at least a part of the captured image data acquired by the image data acquisition unit 133 at a first time point, with previously captured image data, which is at least a part of the captured image data acquired by the image data acquisition unit 133 at a second time point prior to the first time point. The display image data, which is at least a part of the captured image data, is captured image data corresponding to at least a part of an imaging area corresponding to the captured image data. The display image data may be the same as the captured image data acquired by the image data acquisition unit 133 at the first time point.

[0044] The composition unit 134 generates composite image data by combining display image data based on captured image data generated by the camera of the robot 3 capturing images at a first capturing angle with previously captured image data based on captured image data generated by the camera capturing images at a second capturing angle different from the first capturing angle. The first capturing angle is, for example, the angle of the head of the robot 3 at the time when image G1 shown in FIG. 3(a) was generated, and the second capturing angle is the angle of the head of the robot 3 at the time when image G2 or image G3 shown in FIG. 3(a) was generated. If the horizontal angle of view of one piece of captured image data is, for example, 30 degrees and the first capturing angle corresponding to image G1 is (0,0), the second capturing angle corresponding to image G2 is (30,0), and the second capturing angle corresponding to image G3 is (330,0).

[0045] The synthesis unit 134 selects previously captured image data from the multiple previously captured image data stored in the storage unit 12 based on the relationship between the angle of the display device 21 indicated by the operation data acquired via the operation data acquisition unit 131 and multiple device angles corresponding to the multiple captured image data stored in the storage unit 12. Specifically, the synthesis unit 134 selects previously captured image data corresponding to a device angle that differs from the angle of the display device 21 indicated by the operation data by an angle corresponding to the angle of view of the display image data.

[0046] The synthesis unit 134 calculates an angle by adding or subtracting the angle of view (e.g., 30 degrees) of the captured image data to or from the angle of the display device 21 indicated by the head operation data, and selects the previously captured image data stored in the storage unit 12 in association with the device angle corresponding to the calculated angle. As in the above example, when the device angle of the display device 21 is (0,0), the synthesis unit 134 selects the previously captured image data on the left side associated with (30,0) obtained by adding (30,0) to (0,0) and the previously captured image data on the right side associated with (330,0) obtained by subtracting (30,0) from (0,0).

[0047] The composition unit 134 may further select previously captured image data corresponding to angles above and below, diagonally upward, and diagonally downward of the display image data. When the angle of view in the vertical direction is 60 degrees, the composition unit 134 selects captured image data stored in the storage unit 12 in association with a device angle (0, 60) as previously captured image data for the upper side. The composition unit 134 selects captured image data stored in the storage unit 12 in association with a device angle (0, 300) as previously captured image data for the lower side. The composition unit 134 selects captured image data stored in the storage unit 12 in association with a device angle (30, 60) as previously captured image data for the upper left side.

[0048] 3 shows only images G1, G2, and G3, but as described above, captured image data corresponding to 360 degrees (all directions) is stored in the storage unit 12. The synthesis unit 134 selects multiple pieces of captured image data to be used to construct the composite image data from the multiple pieces of captured image data stored in the storage unit 12 in association with each of multiple angles corresponding to all directions, based on the device angle or imaging angle corresponding to the captured image data acquired by the image data acquisition unit 133. The synthesis unit 134 creates celestial spherical image data corresponding to 360 degrees (all directions) by synthesizing the selected multiple pieces of captured image data.

[0049] As described above, the angle of the display device 21 is equivalent to the angle of the head of the robot 3. Therefore, the synthesis unit 134 may select previously captured image data to be used to generate composite image data by using an imaging angle equivalent to the angle of the head of the robot 3 instead of the angle of the display device 21.

[0050] In this case, the synthesis unit 134 selects previously captured image data from the multiple captured image data stored in the storage unit 12 based on the relationship between the first imaging angle at the first time point when the display image data was generated and the multiple imaging angles corresponding to the multiple captured image data stored in the storage unit 12. Specifically, the synthesis unit 134 selects previously captured image data corresponding to an imaging angle that differs from the first imaging angle by an angle corresponding to the angle of view of the display image data, similar to the case where the device angle of the display device 21 is used.

[0051] The combining unit 134 may generate combined image data in a manner that allows the operator U to distinguish between the display image data and the previously captured image data. For example, the combining unit 134 sets the display image data to color image data and the previously captured image data to monochrome image data, thereby enabling the operator U to distinguish between the display image data and the previously captured image data. The combining unit 134 may generate combined image data in which the luminance of the display image data is higher than the luminance of the previously captured image data. The combining unit 134 may generate combined image data that indicates a boundary line between the display image data and the previously captured image data.

[0052] By generating such composite image data by the synthesis unit 134, the operator U can recognize that the previously captured image data is previously captured image data. As a result, the operator U can operate the robot 3 while assuming that the state shown in the previously captured image data may differ from the current state, thereby improving safety.

[0053] The display control unit 135 displays the composite image data on the display device 21 that is visible to the operator U of the robot 3. For example, the display control unit 135 displays the composite image data on the display device 21 in a form that allows the display image data and the previously captured image data to be distinguished. The display control unit 135 transmits the composite image data generated by the synthesis unit 134 to the operation device 2 via the communication unit 11, thereby displaying the composite image data on the display device 21.

[0054] [Processing flow in data processing system S] Fig. 6 is a sequence diagram showing the flow of processing in the data processing system S. The sequence diagram shown in Fig. 6 starts from the point when the robot 3 is powered on.

[0055] The robot 3 generates captured image data by capturing images of the surroundings with the camera 31. The robot 3 generates captured image data corresponding to various angles, for example, by switching the angle of its head (S1), and transmits the generated captured image data to the data processing device 1 in association with imaging angle data indicating the angle of its head.

[0056] The data processing device 1 stores the captured image data received from the robot 3 in the storage unit 12. The data processing device 1 stores the captured image data in association with the imaging angle data, for example (S2). are stored in the storage unit 12, thereby generating the data shown in FIG.

[0057] When the operator U turns on the power of the operation device 2 and starts operating the robot 3, the operation device 2 generates operation data (S3) and transmits the generated operation data to the data processing device 1. The control data generation unit 132 generates control data based on the received operation data (S4) and transmits the generated control data to the robot 3.

[0058] The robot 3 transmits the captured image data to the data processing device 1. The synthesis unit 134 identifies the angle of the display device 21 or the angle of the head of the robot 3 at the time when the robot 3 generated the captured image data (S5). Based on the identified angle, the synthesis unit 134 selects captured image data that was previously generated and stored in the storage unit 12, and generates composite image data by synthesizing the latest captured image data acquired from the robot 3 with the captured image data that was previously generated (S6).

[0059] The display control unit 135 transmits the composite image data generated by the composition unit 134 to the operation device 2. The operation device 2 displays the received composite image data (S7).

[0060] In the above description, an example has been given in which the robot 3 generates captured image data corresponding to various angles while switching the head angle after the power of the robot 3 is turned on and before the operator U starts operation, but the robot 3 does not have to switch the head angle and generate captured image data before the operator U starts operation. In this case, the robot 3 starts generating captured image data after the operator U starts operation.

[0061] Until captured image data corresponding to various angles is accumulated in the storage unit 12, the synthesis unit 134 generates composite image data that includes only captured image data immediately after it has been generated by the robot 3 (i.e., real-time captured image data that is not stored in the storage unit 12). Thereafter, when the captured image data begins to be stored in the storage unit 12, the synthesis unit 134 generates composite image data that combines the captured image data immediately after it has been generated by the robot 3 with the previously captured image data stored in the storage unit 12. By operating the synthesis unit 134 in this manner, the operator U can start work promptly without having to wait until the data processing device 1 stores the captured image data in the storage unit 12.

[0062] [Processing flow in data processing device 1] 7 is a flowchart showing the flow of processing in the data processing device 1. The flowchart shown in FIG.

[0063] The image data acquisition unit 133 acquires captured image data from the robot 3 via the communication unit 11 (S11). Furthermore, the operation data acquisition unit 131 acquires operation data from the operation device 2 via the communication unit 11 (S12). The image data acquisition unit 133 acquires the operation data from the operation data acquisition unit 131 and determines whether the operation data indicates an operation to change the position of the robot 3 (S13). When the image data acquisition unit 133 determines that an operation to change the position of the robot 3 has been performed (YES in S13), it initializes the past captured image data stored in the storage unit 12 (S14).

[0064] When the image data acquisition unit 133 determines that no operation to change the position of the robot 3 has been performed (NO in S13), it does not initialize the past captured image data stored in the storage unit 12. Next, the synthesis unit 134 identifies the device angle of the display device 21 or the angle of the head of the robot 3 at the time when the acquired captured image data was generated, and calculates multiple device angles corresponding to the past captured image data to be synthesized based on the identified angles (S15).

[0065] The synthesis unit 134 determines whether or not previously captured image data corresponding to the calculated plurality of device angles is stored in the storage unit 12 (S16). When the synthesis unit 134 determines that previously captured image data corresponding to at least one device angle among the plurality of device angles is stored in the storage unit 12 (YES in S16), the synthesis unit 134 reads out the previously captured image data corresponding to the device angle from the plurality of previously captured image data stored in the storage unit 12. The synthesis unit 134 generates composite image data by synthesizing the read previously captured image data with the captured image data acquired by the image data acquisition unit 133 in S1 (S17).

[0066] If there is a device angle for which no previously captured image data exists among the multiple device angles calculated in S15, the synthesis unit 134 generates synthetic image data in which the area corresponding to that device angle is set to a predetermined pixel value (for example, a black or white pixel value). The synthesis unit 134 causes the display control unit 135 to transmit the synthetic image data (S18). If previously captured image data corresponding to all of the device angles calculated in S15 is not stored in the storage unit 12 (NO in S16), the synthesis unit 134 does not generate synthetic image data, and causes the display control unit 135 to transmit the captured image data acquired by the image data acquisition unit 133 in S11 (S19).

[0067] The control unit 13 repeats the processes from S11 to S19 until the operator U performs an operation to end the operation on the operation device 2 (NO in S20). When the operator U performs an operation to end the operation (YES in S20), the control unit 13 ends the process.

[0068] [First Modification] In the above explanation, an example has been given in which the synthesis unit 134 synthesizes the captured image data corresponding to the area photographed by the robot 3 with previously captured image data stored in the memory unit 12, but the synthesis unit 134 may also generate composite image data by synthesizing display image data generated by cutting out a partial area of ​​the captured image data acquired by the image data acquisition unit 133 with multiple previously captured image data corresponding to areas adjacent to the display image data.

[0069] Fig. 8 is a diagram for explaining the operation of the data processing device 1 in the first modified example. As shown in Fig. 8, the synthesis unit 134 generates display image data Gc by cutting out a predetermined partial area from the captured image data G0 acquired from the robot 3. The predetermined partial area is, for example, an area that includes the center position of the captured area of ​​the captured image data G0, which is of high importance when the operator U operates the robot 3.

[0070] The synthesis unit 134 calculates a plurality of device angles corresponding to a plurality of previously captured image data to be synthesized with the display image data Gc, based on the angle of view of the display image data. The synthesis unit 134 reads a plurality of previously captured image data corresponding to the calculated plurality of device angles from the storage unit 12. Specifically, the synthesis unit 134 reads from the storage unit 12 a piece of previously captured image data that is in contact with the display image data Gc, which is generated by cutting out the captured image data, above, below, left, right, and diagonally. The synthesis unit 134 generates synthesized image data by synthesizing the display image data Gc with the plurality of previously captured image data (image data in the area indicated by the dashed line in FIG. 8) read from the storage unit 12.

[0071] The synthesis unit 134 generates the display image data Gc by cutting out a portion of the captured image data G0 acquired from the robot 3 in this way, thereby reducing the area of ​​the moving image data that changes in real time, and thus the amount of data in the synthetic image data. As a result, the delay time when transmitting synthetic image data from the data processing device 1 to the controller device 2 is reduced, and the load of display processing on the controller device 2 is also reduced.

[0072] [Second Modification] 9 is a diagram for explaining the operation of the data processing device 1 in the second modified example. In the above explanation, the robot 3 transmits the captured image data G0 generated by the camera as is to the data processing device 1, but the robot 3 may also cut out a partial area of ​​the captured image data G0 and transmit the cut-out captured image data Gc to the data processing device 1. In this case, the image data acquisition unit 133 acquires captured image data Gc corresponding to a partial area of ​​the area captured by the camera.

[0073] By the robot 3 transmitting the extracted captured image data Gc to the data processing device 1, the amount of data transmitted from the robot 3 to the data processing device 1 is reduced, thereby shortening the delay time until the captured image data is displayed on the display device 21.

[0074] [Third Modification] In the above description, an example has been given in which the data processing device 1 generates composite image data, but the operation device 2 may store previously captured image data and have the functions of the image data acquisition unit 133 and the synthesis unit 134. In this case, the operation device 2 acquires captured image data transmitted by the robot 3, selects previously captured image data that is in contact with at least one of the upper, lower, left, and right sides of the captured image data based on the device angle corresponding to the acquired captured image data, and generates composite image data by synthesizing the captured image data and the previously captured image data.

[0075] [Fourth Modification] In the above description, the display device 21 transmits head operation data indicating the angle of the head of the operator U. However, the display device 21 may detect the direction of the line of sight of the operator U and transmit head operation data indicating the angle between the reference direction and the detected line of sight direction. The reference direction is the direction facing forward of the operator U.

[0076] The gaze direction of the operator U is determined based on the orientation of the head of the operator U relative to the front of the operator U and the gaze direction relative to the direction perpendicular to the display screen of the display device 21. Specifically, the display device 21 calculates the angle between the reference orientation and the gaze direction by adding the angle displaced from the orientation of the display device 21 at the time the display device 21 was started up and the angle between the direction perpendicular to the display screen of the display device 21 and the gaze direction. For example, when the head is turned 90 degrees to the right and the gaze is directed 30 degrees to the left relative to the front of the display device 21, the display device 21 determines the angle between the reference orientation and the gaze direction to be 60 degrees.

[0077] When the control data generation unit 132 receives the head operation data indicating the angle between the reference orientation and the gaze direction, it generates control data that sets the head angle to that angle and transmits the generated control data to the robot 3. The robot 3 changes the head angle based on the received control data, allowing the camera 31 to generate captured image data centered on the gaze direction of the operator U.

[0078] By configuring the data processing system S in this way, when the operator U moves his / her line of sight, he / she can view captured image data that matches the direction of the viewpoint in real time, and can also view images based on previously captured image data around the imaging area of ​​the captured image data, further improving operability.

[0079] [Effects of Data Processing System S] As described above, in the data processing device 1, the combining unit 134 combines display image data of at least a part of the captured image data acquired by the image data acquisition unit 133 at a first time point with previously captured image data of at least a part of the captured image data acquired by the image data acquisition unit at a second time point prior to the first time point, to generate composite image data with a wider angle of view than the display image data. Then, the display device 21 displays an image based on the composite image data generated by the combining unit 134.

[0080] By operating the data processing device 1 and the operation device 2 in this manner, the operator U can work while viewing captured images of a wider range than the range that can be captured by the robot 3, thereby improving workability and safety. Furthermore, since the data processing system S can reduce the range that the robot 3 must capture, it is possible to reduce the area of ​​the imaging element of the robot 3 to reduce costs, or to improve the resolution using an imaging element with the same imaging area.

[0081] The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, all or part of the device can be configured by functionally or physically distributing or integrating any unit. Furthermore, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effects of the new embodiments resulting from the combination also have the effects of the original embodiments.

Claims

1. acquiring captured image data generated based on imaging by an imaging device provided on the robot; generating composite image data having a wider angle of view than the display image data by combining display image data of at least a part of the captured image data acquired at a first time point with previous captured image data of at least a part of the captured image data acquired at a second time point prior to the first time point; displaying the composite image data on a display device visible to an operator of the robot; A data processing apparatus having a controller configured to execute the control unit generates the composite image data by combining the display image data based on the captured image data generated by the imaging device capturing an image at a first imaging angle and the previously captured image data based on the captured image data generated by the imaging device capturing an image at a second imaging angle different from the first imaging angle; the data processing device further includes a storage unit that stores the captured image data acquired by the control unit in association with a device angle, which is an angle of the display device when the captured image data was captured; The control unit is further configured to obtain operation data indicating an angle of the display device; The control unit selects the previously captured image data from the plurality of captured image data stored in the storage unit based on the relationship between the angle of the display device indicated by the operation data and the plurality of device angles corresponding to the plurality of captured image data stored in the storage unit.

2. the control unit selects the previously captured image data corresponding to the device angle that is different from the angle of the display device indicated by the operation data by an angle corresponding to the angle of view of the display image data.

2. The data processing device according to claim 1.

3. When the control unit acquires the captured image data, the control unit updates the captured image data stored in the storage unit in association with the angle of the display device indicated by the operation data at the time the captured image data was captured, to the captured image data acquired by the control unit.

3. A data processing device according to claim 1 or 2.

4. the control unit initializes the captured image data stored in the storage unit when the position of the robot is moved, when the operator finishes operating the robot, or when a predetermined time has elapsed.

4. The data processing device according to claim 3.

5. the data processing device further includes a storage unit that stores the captured image data acquired by the control unit in association with an imaging angle, which is an angle of the imaging device when the captured image data was captured; the control unit selects the previously captured image data from the plurality of captured image data stored in the storage unit based on a relationship between the first imaging angle and the plurality of imaging angles corresponding to the plurality of captured image data stored in the storage unit.

2. The data processing device according to claim 1.

6. the control unit selects the previously captured image data corresponding to the imaging angle that is different from the first imaging angle by an angle corresponding to the angle of view of the display image data.

6. A data processing device according to claim 5.

7. the control unit generates the composite image data in a manner that allows the display image data and the previously captured image data to be distinguished from each other.

2. The data processing device according to claim 1.

8. the control unit causes the display device to display the composite image data in a form in which the display image data and the previously captured image data can be distinguished from each other.

2. The data processing device according to claim 1.

9. the control unit generates the composite image data by combining the display image data generated by cutting out a partial area of ​​the captured image data acquired by the control unit with a plurality of pieces of the previously captured image data corresponding to areas adjacent to the display image data.

2. The data processing device according to claim 1.

10. the control unit acquires the captured image data corresponding to a part of an area captured by the imaging device.

2. The data processing device according to claim 1.

11. The computer executes acquiring captured image data generated based on imaging by an imaging device provided on the robot; generating composite image data having a wider angle of view than the display image data by combining display image data of at least a part of the captured image data acquired at a first time point with previously captured image data of at least a part of the captured image data acquired at a second time point prior to the first time point, wherein the display image data based on the captured image data generated by the imaging device capturing an image at a first imaging angle is combined with the previously captured image data based on the captured image data generated by the imaging device capturing an image at a second imaging angle different from the first imaging angle to generate the composite image data; displaying the composite image data on a display device visible to an operator of the robot; A data processing method comprising: acquiring operation data indicating an angle of a display device visible to an operator of the robot; storing the acquired captured image data in a storage unit in association with a device angle, which is the angle of the display device when the acquired captured image data was captured; selecting the previously captured image data from the plurality of captured image data stored in the storage unit based on a relationship between an angle of the display device indicated by the operation data and a plurality of device angles corresponding to the plurality of captured image data stored in the storage unit; The data processing method further comprises:

12. On the computer, acquiring captured image data generated based on imaging by an imaging device provided on the robot; generating composite image data having a wider angle of view than the display image data by combining display image data of at least a part of the captured image data acquired at a first time point with previously captured image data of at least a part of the captured image data acquired at a second time point prior to the first time point, wherein the display image data based on the captured image data generated by the imaging device capturing an image at a first imaging angle is combined with the previously captured image data based on the captured image data generated by the imaging device capturing an image at a second imaging angle different from the first imaging angle to generate the composite image data; displaying the composite image data on a display device visible to an operator of the robot; A program for executing acquiring operation data indicating an angle of a display device visible to an operator of the robot; storing the acquired captured image data in a storage unit in association with a device angle, which is the angle of the display device when the acquired captured image data was captured; selecting the previously captured image data from the plurality of captured image data stored in the storage unit based on a relationship between an angle of the display device indicated by the operation data and a plurality of device angles corresponding to the plurality of captured image data stored in the storage unit; The program further comprises causing the computer to execute the above steps.

13. A data processing system comprising: a robot; an operation device that transmits operation data indicating the content of an operation by an operator that operates the robot; and a data processing device that controls the robot based on the operation data received from the operation device, The robot an imaging device that generates captured image data; an image transmitting unit that transmits the captured image data to the data processing device; and The data processing device includes: acquiring the captured image data transmitted by the robot; generating composite image data having a wider angle of view than the display image data by combining display image data of at least a part of the captured image data acquired at a first time point with previously captured image data of at least a part of the captured image data acquired at a second time point prior to the first time point, wherein the display image data based on the captured image data generated by the imaging device capturing an image at a first imaging angle is combined with the previously captured image data based on the captured image data generated by the imaging device capturing an image at a second imaging angle different from the first imaging angle to generate the composite image data; displaying the composite image data on a display device visible to an operator of the robot; a controller configured to perform The control unit acquiring operation data indicating an angle of a display device visible to an operator of the robot; storing the acquired captured image data in a storage unit in association with a device angle, which is the angle of the display device when the acquired captured image data was captured; selecting the previously captured image data from the plurality of captured image data stored in the storage unit based on a relationship between an angle of the display device indicated by the operation data and a plurality of device angles corresponding to the plurality of captured image data stored in the storage unit; and further configured to: The operating device is an operation data transmission unit that transmits the operation data based on the content of the operation by the operator to the data processing device; a display unit that displays the composite image data transmitted by the data processing device; having Data processing system.

Images