Data processing device, data processing method, program, and data processing system

Abstract

A data processing apparatus (1) according to one embodiment of the present disclosure includes a control section (13) configured to perform the following processes: acquire camera image data generated based on imaging performed by a camera (31) provided to a robot (3); generate composite image data having a wider field of view than the display image data by synthesizing at least a portion of the display image data acquired at a first point in time and past camera image data acquired at a second point in time earlier than the first point in time; and cause the composite image data to be displayed on a display device (21) that can be visually confirmed by an operator (U) of the robot (3).

Description

Technical Field

[0001] This disclosure relates to a data processing apparatus, a data processing method, a program, and a data processing system. Background Technology

[0002] Patent document 1 discloses a technology that enables a terminal used by an operator to remotely operate a robot to display video images generated by a camera device installed on the robot.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2020-040155 Summary of the Invention

[0006] The problem the invention aims to solve

[0007] When an operator is manipulating a robot while viewing images generated by a camera device installed on the robot, there is a problem that the operator may have difficulty operating the robot if the area captured by the camera device is smaller than the operator's field of view.

[0008] This disclosure was made in view of this point, and its purpose is to provide a data processing device, etc., that makes it easy for the operator of the robot to operate the robot.

[0009] Solution for solving the problem

[0010] One aspect of the data processing apparatus disclosed herein includes a control unit configured to perform the following processing: acquiring video image data generated based on images captured by a camera device installed on a robot; generating composite image data with a wider viewing angle than the display image data by combining at least a portion of the display image data acquired at a first time point and at least a portion of the past video image data acquired at a second time point earlier than the first time point; and displaying the composite image data on a display device that allows the robot operator to visually confirm it.

[0011] The effects of the invention

[0012] According to this disclosure, a data processing device is provided that makes it easier for an operator to operate the robot. Attached Figure Description

[0013] Figure 1 This is a diagram showing an overview of the data processing system.

[0014] Figure 2It is a diagram used to illustrate the device angle of a display device.

[0015] Figure 3 This is a diagram showing the relationship between camera image data and synthetic image data.

[0016] Figure 4 This is a diagram showing the structure of a data processing device.

[0017] Figure 5 This is a diagram illustrating a method for storing camera image data in the storage unit.

[0018] Figure 6 It is a timing diagram showing the processing flow of the data processing system.

[0019] Figure 7 It is a flowchart illustrating the processing flow in the data processing device.

[0020] Figure 8 This is a diagram used to illustrate the operation of the data processing device in the first variation.

[0021] Figure 9 This is a diagram used to illustrate the operation of the data processing device in the second variation. Detailed Implementation

[0022] [Overview of Data Processing System S]

[0023] Figure 1 This is a diagram showing an outline of the data processing system S. The data processing system S is a system that provides a telepresence environment that allows the operator U to manipulate objects in real time while making the distant objects feel as if they are nearby.

[0024] The data processing system S includes a data processing device 1, an operating device 2, and a robot 3. The data processing device 1 is a device that controls the robot 3 based on the operations performed by the operator U in the operating device 2, and is, for example, a computer. The data processing device 1 can be installed indoors where either the operator U or the robot 3 is located, or it can be installed in a location different from the location of the operator U and the location of the robot 3.

[0025] Operating device 2 is a device worn by operator U, which includes display device 21, operating device 22, and communication unit 23. Display device 21 is a display that allows operator U to visually confirm an image obtained based on the synthetic image data generated by data processing device 1, for example, a goggle-type display device. The image displayed by display device 21 is, for example, a celestial image corresponding to 360 degrees (all directions).

[0026] Robot 3 performs actions based on control data received from data processing device 1 via network N. Robot 3 includes a camera device (hereinafter referred to as "camera 31") that generates image data, and an image transmission unit 32 that transmits the image data generated by camera 31 to data processing device 1. Furthermore, robot 3 transmits robot state data, representing at least one of the following: tactile data indicating tactile sensations detected by robot 3, sound data indicating sounds collected by robot 3, and joint state data indicating the joint states of robot 3, to operating device 2 via data processing device 1. As an example, in this embodiment, robot 3 performs the task of arranging goods on shelves according to the operation of operator U in stores where many goods are sold in a shelf-like state. The location where robot 3 is installed and the content of the task are arbitrary.

[0027] The data processing device 1 acquires video image data generated by the robot 3 using camera 31 to capture images of the front, and displays the acquired video image data on the display device 21 of the operating device 2. The viewing angle of the video image data generated by the robot 3 is narrower than the operator U's field of view; therefore, when only the video image data is displayed on the display device 21, the operator U's operability is poor. Therefore, the data processing device 1 generates composite image data by combining the video image data generated by the robot 3 with past video image data, which corresponds to the area surrounding the camera area corresponding to the video image data. By displaying the composite image data on the operating device 2 through the data processing device 1, the operator U's operability is improved.

[0028] Display device 21 has a sensor that detects the angle (hereinafter sometimes referred to as "device angle") of the orientation of display device 21 relative to a reference. The reference orientation is, for example, the orientation of display device 21 when an operator U is wearing display device 21 in a state of facing a predetermined orientation, such as the orientation of display device 21 at startup or the orientation at a time of reset. Display device 21 detects the angle difference between the reference orientation and the orientation of display device 21 as the device angle.

[0029] The device angle of the display device 21 is represented, for example, by a combination of two angles in a spherical coordinate system in three-dimensional space. Figure 2 This is a diagram used to illustrate the device angle of display device 21. Figure 2 The angles of the device for determining position G on the surface of the sphere are denoted as (θ, φ). θ is the angle between the line connecting position H (obtained by projecting position G onto the XY plane) and the origin O, and the X-axis direction. φ is the angle between the Z-axis direction and the direction of the line connecting the origin O and position G.

[0030] The positive direction of the X-axis corresponds to the orientation of the front of the display device 21 at the time of startup (i.e., the orientation of the operator U's front). When the operator U is facing forward, the device angle of the display device 21 is represented as (0, 0). When the operator U is facing due left (i.e., the positive direction of the Y-axis), the device angle of the display device 21 is represented as (90, 0).

[0031] Display device 21 generates head operation data representing the device angle of its orientation relative to a reference. Display device 21 notifies communication unit 23 of the head operation data at predetermined time intervals. These predetermined time intervals are determined, for example, based on the speed at which robot 3 can change its head angle; the faster robot 3 can change its head angle, the shorter the predetermined time interval. Since the interval at which display device 21 notifies the device angle corresponds to the speed at which robot 3 can change its head angle, it is not necessary to detect the device angle at an excessively high frequency, thus suppressing power consumption of the operating device 2.

[0032] The operating device 22 is a device for operator U to operate the hands and arms of robot 3, and has sensors for detecting the movement of operator U's hands and arms. The operating device 22 generates arm operation data representing the movement of operator U's hands and arms. The operating device 22 notifies the communication unit 23 of the generated arm operation data. The operating device 22 can accept operations to move robot 3 in the forward / backward or left / right directions. In addition, the operating device 22 also has elements for generating heat, pressure, or vibration corresponding to the state of robot 3, so that operator U can perceive the state of robot 3.

[0033] The communication unit 23 includes a communication controller that transmits operation data obtained based on the operations performed by the operator U to the data processing device 1, or receives synthetic image data from the data processing device 1. For example, the communication unit 23 synchronizes head operation data notified from the display device 21 and arm operation data notified from the operating device 22, and transmits the synchronized head operation data and arm operation data to the data processing device 1 at predetermined time intervals. Additionally, the communication unit 23 inputs synthetic image data received from the data processing device 1 to the display device 21. The communication unit 23 can transmit operation data for moving the robot 3 forward, backward, left, or right to the data processing device 1. Furthermore, the communication unit 23 can be included in either the display device 21 or the operating device 22, or it can be housed in a different housing than the display device 21 and the operating device 22.

[0034] Robot 3, based on head control data generated by data processing device 1 from head operation data sent from operating device 2, changes the orientation of its head to the same angle as the display device 21. Additionally, based on arm control data generated by data processing device 1 from arm operation data sent from operating device 2, Robot 3's hands and arms move in the same manner as the operator U's hands and arms.

[0035] A camera 31, for example, is mounted on the head to generate image data by capturing images of the front of the robot 3. The image transmitting unit 32 has a communication controller for transmitting the image data via network N. Figure 1 In the example shown, the camera 31 has an image transmitting unit 32, but the location of the image transmitting unit 32 is arbitrary.

[0036] Because the orientation of robot 3's head changes based on head control data, the orientation of the optical axis of camera 31 changes, resulting in a change in the imaging area. In the following description, the rotation center of robot 3's head will be connected (to... Figure 2 The angle between the direction of the straight line pointing from the origin (corresponding to the center of the area being photographed) and the direction of the center of the area being photographed, relative to the frontal orientation, is called the "camera angle". Robot 3 moves its head in a manner that matches the angle of the operator U's head (i.e., the device angle) as shown in the head operation data with the camera angle.

[0037] Below, refer to Figure 1 Here is a summary of the processing flow in the data processing system S. As described above, the operating device 2 generates operating data corresponding to the movements of the operator U's head and arms, and sends the operating data to the data processing device 1. The data processing device 1 generates control data for the robot 3 to move based on the operating data, and sends the control data to the robot 3. The robot 3 performs actions based on the control data received from the data processing device 1.

[0038] During its actions, robot 3 sends camera-generated image data to data processing device 1 at predetermined time intervals (e.g., 5-millisecond intervals). Robot 3 may also send the camera-generated image data to data processing device 1 in association with information indicating the angle of robot 3's head at the time the camera-generated image data was generated.

[0039] The data processing device 1 sends composite image data to the operating device 2, which is generated by combining the camera image data received from the robot 3 with past camera image data corresponding to the area surrounding the camera area of ​​the camera image data. The display device 21 displays the composite image data. The composite image data includes image data covering a wider range than that that the robot 3 can capture, so the operator U can ensure the same field of view as when actually watching the robot 3 operating, making it easy to operate the robot 3.

[0040] Figure 3 This is a diagram showing the relationship between camera image data and synthetic image data. Figure 3 Image G1 in (a) is an image based on display image data generated by robot 3 at the first time point, showing the arrangement of bottles placed on the shelf. Figure 3 Images G2 and G3 in (a) are images based on camera image data, which are camera image data from a plurality of past camera image data generated by robot 3 at a second time point earlier than the first time point, corresponding to the camera areas adjacent to both sides of the camera area of ​​the image data for display.

[0041] Figure 3 (b) shows the composite image obtained by combining image data G1 with past camera image data G2 and past camera image data G3. This is compared to the view of the operator U. Figure 3 In the case of image G1 shown in (a), viewed by operator U Figure 3 In the case of the composite image shown in (b), the operator U has a wide field of view, making it easy to use robot 3 for operations.

[0042] The structure and operation of the data processing device 1 will now be described in detail.

[0043] [Structure of Data Processing Device 1]

[0044] Figure 4 This is a diagram showing the structure of the data processing apparatus 1. The data processing apparatus 1 includes a communication unit 11, a storage unit 12, and a control unit 13. The control unit 13 includes an operation data acquisition unit 131, a control data generation unit 132, an image data acquisition unit 133, a synthesis unit 134, and a display control unit 135.

[0045] The communication unit 11 has a communication interface for sending and receiving various data with the operating device 2 and the robot 3 via the network N. For example, the communication unit 11 inputs operating data received from the operating device 2 to the operating data acquisition unit 131. Additionally, the communication unit 11 inputs camera image data received from the robot 3 to the image data acquisition unit 133. Furthermore, the communication unit 11 sends composite image data input from the display control unit 135 to the operating device 2.

[0046] Storage unit 12 has storage media such as ROM (Read Only Memory), RAM (Random Access Memory), and SSD (Solid State Drive). Storage unit 12 stores the program executed by control unit 13. In addition, storage unit 12 temporarily stores the camera image data received from robot 3. Storage unit 12 stores the angle of display device 21 at the time the camera image data acquired by image data acquisition unit 133 was captured, i.e., the device angle, in association with the camera image data. The angle of display device 21 is equivalent to the angle of robot 3's head, so storage unit 12 can store the camera image data in association with the angle of robot 3's head at the time the camera image data was generated.

[0047] Figure 5 This diagram illustrates a method for storing camera image data in the storage unit 12. Figure 5 In this process, the device angle (θ, φ) of the display device 21 at the time when the robot 3 generates the camera image data is associated with the filename of the camera image data. Whenever the communication unit 11 receives new camera image data from the robot 3, the camera image data stored in the storage unit 12, which is associated with the angle of the display device 21 at the time when the received camera image data was generated, is updated. The angle of the display device 21 and the angle of the robot 3's head can be angles corresponding to any orientation in three-dimensional space, so camera image data corresponding to 360 degrees (all directions) is stored in the storage unit 12.

[0048] return Figure 4 The control unit 13 will now be described. The control unit 13 has, for example, a CPU (Central Processing Unit) as a processor. The control unit 13 functions as an operation data acquisition unit 131, a control data generation unit 132, an image data acquisition unit 133, a compositing unit 134, and a display control unit 135 by executing the program stored in the storage unit 12.

[0049] The operation data acquisition unit 131 acquires operation data representing the operation content of the operator U via the communication unit 11. For example, the operation data acquisition unit 131 acquires head operation data representing the angle of the display device 21 and arm operation data representing the movement of the operator U's arm. The operation data acquisition unit 131 inputs the acquired operation data to the control data generation unit 132, the image data acquisition unit 133, and the synthesis unit 134.

[0050] The control data generation unit 132 generates control data for moving the robot 3 based on the operation data input from the operation data acquisition unit 131. For example, the control data generation unit 132 can generate control data by adding or subtracting a predetermined correction value from the value shown in the operation data. The control data generation unit 132 sends the generated control data to the robot 3 via the communication unit 11.

[0051] The image data acquisition unit 133 acquires camera image data generated by the camera 31 installed on the robot 3 via the communication unit 11. The image data acquisition unit 133 stores the acquired camera image data in the storage unit 12. For example, the image data acquisition unit 133... Figure 5 As shown, the image data is stored in the storage unit 12 in association with the angle of the display device 21, which is the operation data at the time the image data is acquired.

[0052] When the image data acquisition unit 133 acquires camera image data, it updates the camera image data stored in the storage unit 12 in association with the angle of the display device 21 shown in the operation data at the time the camera image data was captured, to the camera image data acquired by the image data acquisition unit 133. That is, when the image data acquisition unit 133 acquires camera image data with an angle that is already stored in the storage unit 12, it updates the camera image data stored in the storage unit 12 with the newly acquired camera image data. Furthermore, when the camera image data is associated with the angle of the robot 3's head at the time the robot 3 generates the camera image data, the image data acquisition unit 133 can store the camera image data in the storage unit 12 in association with the angle of the robot 3's head.

[0053] Furthermore, when the position of robot 3 changes, the regions included in the camera image data generated by robot 3 change. Therefore, if past camera image data is synthesized after the position of robot 3 changes, mismatches will occur at the boundary lines of the synthesized multiple camera image data. Therefore, when the position of robot 3 moves, the image data acquisition unit 133 can initialize the camera image data stored in the storage unit 12.

[0054] When the image data acquisition unit 133 detects an operation that moves the position of the robot 3 based on movement operation data acquired by the operation data acquisition unit 131, for example, it initializes the camera image data stored in the storage unit 12. The image data acquisition unit 133 initializes the camera image data, for example, by deleting the camera image data stored in the storage unit 12. Alternatively, the image data acquisition unit 133 may not delete the camera image data, but instead... Figure 5 The filenames of the camera image data associated with the device angle are deleted as shown. By initializing the camera image data in the event that the position of robot 3 has moved, mismatched composite images are prevented from being displayed on display device 21.

[0055] Furthermore, even if robot 3 remains in the same position, its surrounding environment may change over time. If past camera image data is synthesized despite changes in the robot 3's surrounding environment, mismatches will occur at the boundaries of the synthesized multiple camera image data sets. Therefore, the image data acquisition unit 133 can initialize the camera image data stored in the storage unit 12 when the operator U finishes operating on robot 3 or when a predetermined time has elapsed. The predetermined time is the duration during which the surrounding environment at robot 3's location may change, for example, one minute.

[0056] Whether the surrounding environment of robot 3 changes can vary depending on the date, time, or day of the week. Therefore, the image data acquisition unit 133 can initialize the camera image data stored in the storage unit 12 after a predetermined time associated with at least one of the date, time, and day of the week has elapsed.

[0057] Furthermore, the image data acquisition unit 133 can initialize the camera image data stored in the storage unit 12 upon receiving an instruction from the operator U inputting this instruction into the operation device 22 to initialize the camera image data stored in the storage unit 12. By operating the image data acquisition unit 133 in this way, the camera image data can be initialized according to the operator U's intention at the point when the operator U begins to perceive a decrease in the quality of the synthesized image data.

[0058] Furthermore, during the period when operator U is not operating robot 3, the surrounding environment of robot 3 may change, or someone may move robot 3. Therefore, it is assumed that at the point when operator U resumes operation of robot 3, the camera image data stored in storage unit 12 does not match the camera image data generated by robot 3. Therefore, image data acquisition unit 133 can initialize the camera image data stored in storage unit 12 if operator U logs out to end operation of robot 3, or if the power to robot 3 is turned off.

[0059] Synthesis section 134 generates as follows Figure 3 The composite image data shown in (b) is as follows. Specifically, the compositing unit 134 generates composite image data with a wider viewing angle than the display image data by compositing at least a portion of the display image data from the camera image data acquired by the image data acquisition unit 133 at a first time point and at least a portion of the past camera image data from the camera image data acquired by the image data acquisition unit 133 at a second time point earlier than the first time point. At least a portion of the display image data from the camera image data is camera image data corresponding to at least a portion of the camera area corresponding to the same camera image data. The display image data may be the same as the camera image data acquired by the image data acquisition unit 133 at the first time point.

[0060] The compositing unit 134 generates composite image data by combining display image data obtained from camera image data generated by the camera of robot 3 at a first camera angle, and past camera image data obtained from camera image data generated by the camera at a second camera angle different from the first camera angle. The first camera angle is, for example, when the camera was generated at a second camera angle different from the first camera angle. Figure 3 Image (a) shows the angle of the robot 3's head at time point G1. The second camera angle is during production. Figure 3 The angle of the robot 3's head at the time point shown in image G2 or image G3 in (a). In the case that the left-right viewing angle in a camera image data is, for example, 30 degrees, and the first camera angle corresponding to image G1 is (0, 0), the second camera angle corresponding to image G2 is (30, 0), and the second camera angle corresponding to image G3 is (330, 0).

[0061] The compositing unit 134 selects past image data from the plurality of past image data stored in the storage unit 12 based on the relationship between the angle of the display device 21 shown in the operation data acquired by the operation data acquisition unit 131 and the plurality of device angles corresponding to the same plurality of image data stored in the storage unit 12. Specifically, the compositing unit 134 selects past image data corresponding to the following device angles, where the difference between the device angle and the angle of the display device 21 shown in the operation data is the angle corresponding to the viewing angle of the image data to be displayed.

[0062] The compositing unit 134 calculates the angle obtained by adding or subtracting the viewing angle (e.g., 30 degrees) of the camera image data from the angle of the display device 21 shown by the head operation data, and selects past camera image data that is associated with the device angle corresponding to the calculated angle and stored in the storage unit 12. As in the example above, when the device angle of the display device 21 is (0, 0), the compositing unit 134 selects the past camera image data on the left side associated with (30, 0) obtained by adding (30, 0) to (0, 0), and the past camera image data on the right side associated with (330, 0) obtained by subtracting (30, 0) from (0, 0).

[0063] The compositing unit 134 can also select past camera image data corresponding to the vertical, diagonally upward, and diagonally downward angles of the displayed image data. When the vertical viewing angle is 60 degrees, the compositing unit 134 selects camera image data stored in the storage unit 12 associated with device angles (0, 60 degrees) as the upper past camera image data. The compositing unit 134 selects camera image data stored in the storage unit 12 associated with device angles (0, 30 degrees) as the lower past camera image data. The compositing unit 134 selects camera image data stored in the storage unit 12 associated with device angles (30, 60 degrees) as the left diagonally upward past camera image data.

[0064] exist Figure 3 In the example shown, only images G1, G2, and G3 are shown, but as described above, the storage unit 12 stores camera image data corresponding to 360 degrees (all directions). Therefore, the compositing unit 134 selects multiple camera image data from the multiple camera image data stored in the storage unit 12 that are associated with multiple angles corresponding to all directions, based on the device angle or camera angle corresponding to the camera image data acquired by the image data acquisition unit 133, to construct the composite image data. The compositing unit 134 creates celestial image data corresponding to 360 degrees (all directions) by compositing the selected multiple camera image data.

[0065] Furthermore, as described above, the angle of the display device 21 is equivalent to the angle of the robot 3's head. Therefore, the compositing unit 134 can use a camera angle equivalent to the angle of the robot 3's head instead of the angle of the display device 21 to select past camera image data for generating the composite image data.

[0066] In this case, the synthesis unit 134 selects past image data from the plurality of image data stored in the storage unit 12 based on the relationship between the first camera angle at the first time point in time when the display image data is generated and the plurality of camera angles corresponding to the same plurality of image data stored in the storage unit 12. Specifically, similar to the case of using the device angle of the display device 21, the synthesis unit 134 selects past image data corresponding to the following camera angle, which differs from the first camera angle by an angle corresponding to the viewing angle of the display image data.

[0067] The compositing unit 134 can generate composite image data in a form that distinguishes between display image data and past camera image data. For example, by setting the display image data to color image data and the past camera image data to monochrome image data, the compositing unit 134 enables the operator U to distinguish between the display image data and the past camera image data. The compositing unit 134 can generate composite image data in which the brightness of the display image data is higher than the brightness of the past camera image data. The compositing unit 134 can generate composite image data showing the boundary line between the display image data and the past camera image data.

[0068] By generating such synthetic image data through the synthesis unit 134, the operator U can identify that past camera image data is past camera image data. As a result, the operator U can operate the robot 3 assuming that the state shown in the past camera image data may be different from the state at the current time, thus improving safety.

[0069] The display control unit 135 displays the composite image data on a display device 21 that can be visually viewed by the operator U of the robot 3. The display control unit 135 displays the composite image data on the display device 21, for example, in a form that distinguishes between display image data and past camera image data. The display control unit 135 transmits the composite image data generated by the composite unit 134 to the operating device 2 via the communication unit 11, thereby displaying the composite image data on the display device 21.

[0070] [Processing flow in data processing system S]

[0071] Figure 6 This is a timing diagram illustrating the processing flow of the data processing system S. Figure 6 The timing diagram shown starts from the point when the power to robot 3 is turned on.

[0072] Robot 3 generates video image data by using camera 31 to capture images of its surroundings. For example, robot 3 generates video image data corresponding to various angles while switching the angle of its head (S1), and sends the generated video image data in association with video angle data representing the angle of the head to data processing device 1.

[0073] The data processing device 1 stores the camera image data received from the robot 3 in the storage unit 12 (S2). For example, the data processing device 1 generates data such as... Figure 5 The data shown.

[0074] When operator U turns on the power to the operating device 2 to begin operating the robot 3, the operating device 2 generates operation data (S3) and sends the generated operation data to the data processing device 1. The control data generation unit 132 generates control data based on the received operation data (S4) and sends the generated control data to the robot 3.

[0075] Robot 3 sends the camera image data to data processing device 1. Compositing unit 134 determines the angle of display device 21 or the angle of robot 3's head at the time point when robot 3 generates the camera image data (S5). Based on the determined angle, compositing unit 134 selects previously generated camera image data stored in storage unit 12, and generates composite image data by combining the latest camera image data obtained from robot 3 with the previously generated camera image data (S6).

[0076] The display control unit 135 sends the composite image data generated by the compositing unit 134 to the operation device 2. The operation device 2 displays the received composite image data (S7).

[0077] Furthermore, the above description illustrates the following scenario: after the power to robot 3 is turned on and before the operator U begins operation, robot 3 generates camera image data corresponding to various angles while switching the angle of its head. However, robot 3 may also generate camera image data without switching the angle of its head before the operator U begins operation. In this case, robot 3 begins generating camera image data after the operator U begins operation.

[0078] During the period until the storage unit 12 has accumulated camera image data corresponding to various angles, the compositing unit 134 generates composite image data that includes only the camera image data just generated by the robot 3 (i.e., real-time camera image data not yet stored in the storage unit 12). Afterwards, when the compositing unit 134 begins storing camera image data in the storage unit 12, it generates composite image data that combines the camera image data just generated by the robot 3 with past camera image data stored in the storage unit 12. By operating the compositing unit 134 in this way, the operator U does not need to wait for the data processing device 1 to store camera image data in the storage unit 12, and can quickly begin work.

[0079] [Processing flow in data processing device 1]

[0080] Figure 7 This is a flowchart illustrating the processing flow in the data processing device 1. Figure 7 The flowchart shown starts from the point when the power to the operating device 2 is turned on.

[0081] Image data acquisition unit 133 acquires camera image data from robot 3 via communication unit 11 (S11). Additionally, operation data acquisition unit 131 acquires operation data from operation device 2 via communication unit 11 (S12). Image data acquisition unit 133 acquires operation data from operation data acquisition unit 131 and determines whether the operation data indicates an operation to change the position of robot 3 (S13). When image data acquisition unit 133 determines that an operation to change the position of robot 3 has been performed (S13: "Yes"), it initializes the past camera image data stored in storage unit 12 (S14).

[0082] When the image data acquisition unit 133 determines that no operation to change the position of the robot 3 has been performed (S13: "No"), it does not initialize the past camera image data stored in the storage unit 12. Next, the compositing unit 134 determines the device angle of the display device 21 or the head angle of the robot 3 at the time when the acquired camera image data was generated, and calculates multiple device angles corresponding to the past camera image data that are the objects of compositing based on the determined angles (S15).

[0083] The compositing unit 134 determines whether the past camera image data corresponding to the calculated multiple device angles has been stored in the storage unit 12 (S16). If the compositing unit 134 determines that the past camera image data corresponding to at least one of the multiple device angles has been stored in the storage unit 12 (S16: "Yes"), the compositing unit 134 reads the past camera image data corresponding to the device angle from the multiple past camera image data stored in the storage unit 12. The compositing unit 134 generates composite image data by compositing the read past camera image data with the camera image data acquired by the image data acquisition unit 133 in S1 (S17).

[0084] If, among the multiple device angles calculated in S15, there is a device angle for which no past image data is available, the compositing unit 134 generates composite image data such that the area corresponding to that device angle has a predetermined pixel value (e.g., a black or white pixel value). The compositing unit 134 sends the composite image data to the display control unit 135 (S18). If, in the case that none of the past image data corresponding to all the device angles calculated in S15 is stored in the storage unit 12 (S16: "No"), the compositing unit 134 does not generate composite image data, and sends the image data acquired by the image data acquisition unit 133 in S11 to the display control unit 135 (S19).

[0085] During the period until the operator U performs the operation to end the operation on the operating device 2 (S20: "No"), the control unit 13 repeats the processing of S11 to S19. If the operator U performs the operation to end the operation (S20: "Yes"), the control unit 13 ends the processing.

[0086] [First Variation]

[0087] The above description illustrates the following situation: the compositing unit 134 combines the camera image data corresponding to the area captured by the robot 3 with the past camera image data stored in the storage unit 12. However, the compositing unit 134 may also generate composite image data by combining display image data generated by cutting out a portion of the camera image data acquired by the image data acquisition unit 133 with multiple past camera image data corresponding to the area adjacent to the display image data.

[0088] Figure 8 This is a diagram used to illustrate the operation of the data processing device 1 in the first modified example. For example... Figure 8As shown, the compositing unit 134 generates display image data Gc by cutting out a predetermined region from the camera image data G0 acquired from the robot 3. The predetermined region is, for example, a region of high importance when the operator U operates the robot 3, and includes the center position of the camera area of ​​the camera image data G0.

[0089] The compositing unit 134 calculates multiple device angles corresponding to multiple past camera images composited with the display image data Gc, based on the viewing angle of the display image data. The compositing unit 134 reads multiple past camera images corresponding to the calculated multiple device angles from the storage unit 12. Specifically, the compositing unit 134 reads from the storage unit 12 past camera images adjacent to the display image data Gc, which is generated by cropping from the camera image data, in the directions above, below, left, right, and diagonally adjacent to the display image data. The compositing unit 134 combines the display image data Gc and the multiple past camera images read from the storage unit 12 (in...) Figure 8 Image data from the area indicated by the dashed line is synthesized to generate composite image data.

[0090] By cutting out a portion of the camera image data G0 acquired from the robot 3 through the compositing unit 134 to generate display image data Gc, the area of ​​motion image data that is switched in real time is reduced, thus reducing the amount of composite image data. As a result, the delay time when sending composite image data from the data processing device 1 to the operation device 2 is reduced, and the load on the display processing in the operation device 2 is reduced.

[0091] [Second variation]

[0092] Figure 9 This diagram illustrates the operation of the data processing device 1 in the second variation. In the above description, the following scenario is illustrated: the robot 3 directly sends the camera-generated image data G0 to the data processing device 1; however, the robot 3 may also cut out a portion of the image data G0 and send the cut-out image data Gc to the data processing device 1. In this case, the image data acquisition unit 133 acquires the image data Gc corresponding to a portion of the area captured by the camera.

[0093] The cut-out camera image data Gc is sent to the data processing device 1 by the robot 3. The amount of data sent from the robot 3 to the data processing device 1 is reduced, thus shortening the delay time until the camera image data is displayed on the display device 21.

[0094] [Third variation]

[0095] The above description illustrates the case where the data processing device 1 generates composite image data. However, the operation device 2 may also store past camera image data and have the functions of image data acquisition unit 133 and synthesis unit 134. In this case, the operation device 2 acquires camera image data sent by the robot 3, and based on the device angle corresponding to the acquired camera image data, selects past camera image data adjacent to at least one of the sides above, below, left, and right of the camera image data, and generates composite image data obtained by combining the camera image data with the past camera image data.

[0096] [Fourth variation]

[0097] In the above description, the display device 21 sends head operation data indicating the angle of the operator U's head. However, it is also possible that the display device 21 detects the orientation of the operator U's gaze and sends head operation data indicating the angle between the reference orientation and the detected gaze orientation. The reference orientation is the orientation of the operator U's front.

[0098] The orientation of the operator U's gaze is determined based on the orientation of the operator U's head relative to the front of the operator U and the orientation of the gaze relative to the direction orthogonal to the display screen of the display device 21. Specifically, the display device 21 calculates the angle between the reference orientation and the gaze orientation by adding the angle of displacement of the display device 21 relative to the orientation of the display device 21 at the time of its startup, and the angle between the direction orthogonal to the display screen of the display device 21 and the gaze orientation. For example, if the head is facing 90 degrees to the right and the gaze is facing 30 degrees to the left relative to the front of the display device 21, then the display device 21 sets the angle between the reference orientation and the gaze orientation to 60 degrees.

[0099] When the control data generation unit 132 receives head operation data representing the angle between the reference orientation and the direction of the line of sight, it generates control data that sets that angle as the head angle and sends the generated control data to the robot 3. Based on the received control data, the robot 3 changes the angle of its head, thereby enabling the camera 31 to generate camera image data centered on the direction of the operator U's line of sight.

[0100] By configuring the data processing system S in this way, even when the operator U moves their line of sight, they can see the camera image data in real time that is aligned with the orientation of the viewpoint, and can also see images based on past camera image data around the camera area of ​​the image data, thus further improving operability.

[0101] [The effect of data processing system S]

[0102] As explained above, in the data processing apparatus 1, the compositing unit 134 combines at least a portion of the display image data from the camera image data acquired by the image data acquisition unit 133 at a first time point with at least a portion of the past camera image data from the camera image data acquired by the image data acquisition unit at a second time point earlier than the first time point to generate composite image data with a wider viewing angle than the display image data. Furthermore, the display device 21 displays an image based on the composite image data generated by the compositing unit 134.

[0103] By operating the data processing device 1 and the operating device 2 in this way, the operator U can perform tasks while viewing a camera image covering a larger area than that captured by the robot 3, thus improving workability and safety. Furthermore, the data processing system S can reduce the area that the robot 3 should capture, thereby reducing the size of the camera element on the robot 3 and lowering costs, or using a camera element of the same size to increase resolution.

[0104] The present invention has been described above using embodiments, but the scope of protection of the present invention is not limited to the scope described in the above embodiments, and various modifications and alterations can be made within its scope. For example, all or part of the device can be configured to be functionally or physically distributed / combined in any unit. In addition, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effects of new embodiments resulting from combinations are the same as the effects of the original embodiments.

Claims

1. A data processing apparatus comprising a control unit configured to perform the following processing: Acquire camera image data based on images captured by a camera device installed on the robot; By synthesizing at least a portion of the display image data from the camera image data acquired at a first time point and at least a portion of the past camera image data from the camera image data acquired at a second time point earlier than the first time point, composite image data with a wider viewing angle than the display image data is generated; and The synthesized image data is displayed on a display device that allows the robot operator to visually confirm it. The control unit generates the composite image data by combining the display image data obtained from the camera image data generated by the camera device shooting at a first camera angle, and the past camera image data obtained from the camera image data generated by the camera device shooting at a second camera angle different from the first camera angle. The data processing device further includes a storage unit that stores the angle of the display device (i.e., the device angle) at the time the camera image data acquired by the control unit is captured, in association with the camera image data. The control unit is configured to further perform the following processing: acquiring operational data representing the angle of the display device. The control unit selects the past camera image data from the plurality of camera image data stored in the storage unit based on the relationship between the angle of the display device shown in the operation data and the multiple device angles corresponding to the same plurality of camera image data stored in the storage unit.

2. The data processing apparatus according to claim 1, characterized in that, The control unit selects the past camera image data corresponding to the device angle below, the device angle being an angle that differs from the angle of the display device shown in the operation data by the angle corresponding to the viewing angle of the display image data.

3. The data processing apparatus according to claim 1 or 2, characterized in that, When the control unit acquires the camera image data, it updates the camera image data stored in the storage unit in association with the angle of the display device shown by the operation data at the time point when the camera image data was captured to the camera image data acquired by the control unit.

4. The data processing apparatus according to claim 3, characterized in that, The control unit initializes the camera image data stored in the storage unit when the robot's position moves, when the operator ends the operation of the robot, or when a predetermined time has elapsed.

5. The data processing apparatus according to claim 1 or 2, characterized in that, The control unit generates composite image data capable of distinguishing between the display image data and the past camera image data.

6. The data processing apparatus according to claim 1 or 2, characterized in that, The control unit causes the synthesized image data to be displayed on the display device in a form that allows it to distinguish between the display image data and the past camera image data.

7. The data processing apparatus according to claim 1 or 2, characterized in that, The control unit generates the composite image data by combining the display image data generated by cutting out a portion of the camera image data acquired by the control unit with a plurality of past camera image data corresponding to the region adjacent to the display image data.

8. The data processing apparatus according to claim 1 or 2, characterized in that, The control unit acquires the camera image data corresponding to a portion of the area captured by the camera device.

9. A data processing apparatus comprising a control unit configured to perform the following processing: Acquire camera image data based on images captured by a camera device installed on the robot; By synthesizing at least a portion of the display image data from the camera image data acquired at a first time point and at least a portion of the past camera image data from the camera image data acquired at a second time point earlier than the first time point, composite image data with a wider viewing angle than the display image data is generated; and The synthesized image data is displayed on a display device that allows the robot operator to visually confirm it. The control unit generates the composite image data by combining the display image data obtained from the camera image data generated by the camera device shooting at a first camera angle, and the past camera image data obtained from the camera image data generated by the camera device shooting at a second camera angle different from the first camera angle. The data processing device further includes a storage unit, which stores the camera angle (i.e., the camera angle) at the time the camera image data acquired by the control unit is captured, in association with the camera image data. The control unit selects the past camera image data from the plurality of camera image data stored in the storage unit based on the relationship between the first camera angle and the plurality of camera angles corresponding to the same plurality of camera image data stored in the storage unit.

10. The data processing apparatus according to claim 9, characterized in that, The control unit selects the past camera image data corresponding to the following camera angle, which differs from the first camera angle by an angle corresponding to the viewing angle of the display image data.

11. A data processing method, executed by a computer, the data processing method comprising the following steps: Acquire camera image data based on images captured by a camera device installed on the robot; By combining at least a portion of the display image data acquired at a first time point with at least a portion of the past image data acquired at a second time point earlier than the first time point, composite image data with a wider viewing angle than the display image data is generated. as well as The synthesized image data is displayed on a display device that allows the robot operator to visually confirm it. The composite image data is generated by combining the display image data obtained from the camera image data generated by the camera device shooting at a first camera angle, and the past camera image data obtained from the camera image data generated by the camera device shooting at a second camera angle different from the first camera angle. The computer also has a storage unit that stores the angle of the display device (i.e., the device angle) at the time the camera image data acquired by the computer was captured, in association with the camera image data. The data processing method further includes the following steps: acquiring operational data representing the angle of the display device. Specifically, the past video image data is selected from the plurality of video image data stored in the storage unit based on the relationship between the angle of the display device shown in the operation data and the multiple device angles corresponding to the same plurality of video image data stored in the storage unit.

12. A data processing method, executed by a computer, the data processing method comprising the following steps: Acquire camera image data based on images captured by a camera device installed on the robot; By combining at least a portion of the display image data acquired at a first time point with at least a portion of the past image data acquired at a second time point earlier than the first time point, composite image data with a wider viewing angle than the display image data is generated. as well as The synthesized image data is displayed on a display device that allows the robot operator to visually confirm it. The composite image data is generated by combining the display image data obtained from the camera image data generated by the camera device shooting at a first camera angle, and the past camera image data obtained from the camera image data generated by the camera device shooting at a second camera angle different from the first camera angle. The computer also has a storage unit that stores the camera angle (i.e., the camera angle) at the time the camera image data acquired by the computer was captured in association with the camera image data. Specifically, the past camera image data is selected from the plurality of camera image data stored in the storage unit based on the relationship between the first camera angle and the plurality of camera angles corresponding to the same plurality of camera image data stored in the storage unit.

13. A computer program product comprising a program for causing a computer to perform the following steps: Acquire camera image data based on images captured by a camera device installed on the robot; By synthesizing at least a portion of the display image data from the camera image data acquired at a first time point and at least a portion of the past camera image data from the camera image data acquired at a second time point earlier than the first time point, composite image data with a wider viewing angle than the display image data is generated; and The synthesized image data is displayed on a display device that allows the robot operator to visually confirm it. in, The composite image data is generated by combining the display image data obtained from the camera image data generated by the camera device shooting at a first camera angle, and the past camera image data obtained from the camera image data generated by the camera device shooting at a second camera angle different from the first camera angle. The computer also has a storage unit that stores the angle of the display device (i.e., the device angle) at the time the camera image data acquired by the computer was captured, in association with the camera image data. The program is also used to cause the computer to perform the following steps: acquiring operational data representing the angle of the display device. Specifically, the past video image data is selected from the plurality of video image data stored in the storage unit based on the relationship between the angle of the display device shown in the operation data and the multiple device angles corresponding to the same plurality of video image data stored in the storage unit.

14. A computer program product comprising a program for causing a computer to perform the following steps: Acquire camera image data based on images captured by a camera device installed on the robot; By synthesizing at least a portion of the display image data from the camera image data acquired at a first time point and at least a portion of the past camera image data from the camera image data acquired at a second time point earlier than the first time point, composite image data with a wider viewing angle than the display image data is generated; and The synthesized image data is displayed on a display device that allows the robot operator to visually confirm it. in, The composite image data is generated by combining the display image data obtained from the camera image data generated by the camera device shooting at a first camera angle, and the past camera image data obtained from the camera image data generated by the camera device shooting at a second camera angle different from the first camera angle. The computer also has a storage unit that stores the camera angle (i.e., the camera angle) at the time the camera image data acquired by the computer was captured in association with the camera image data. Specifically, the past camera image data is selected from the plurality of camera image data stored in the storage unit based on the relationship between the first camera angle and the plurality of camera angles corresponding to the same plurality of camera image data stored in the storage unit.

15. A data processing system comprising: a robot; an operating device that transmits operation data representing the content of an operation performed by an operator operating the robot; and a data processing device that controls the robot based on the operation data received from the operating device. in, The robot has the following characteristics: A camera device that generates camera image data; and The image transmitting unit sends the camera image data to the data processing device. The data processing device includes a control unit configured to perform the following processing: Acquire the camera image data sent by the robot; By combining at least a portion of the display image data acquired at a first time point with at least a portion of the past image data acquired at a second time point earlier than the first time point, composite image data with a wider viewing angle than the display image data is generated. as well as The synthesized image data is displayed on a display device that allows the robot operator to visually confirm it. The operating device has: An operation data transmission unit transmits operation data obtained based on the content of the operation performed by the operator to the data processing device; as well as The display unit displays the composite image data sent by the data processing device. The control unit generates the composite image data by combining the display image data obtained from the camera image data generated by the camera device shooting at a first camera angle, and the past camera image data obtained from the camera image data generated by the camera device shooting at a second camera angle different from the first camera angle. The data processing device further includes a storage unit that stores the angle of the display device (i.e., the device angle) at the time the camera image data acquired by the control unit is captured, in association with the camera image data. The control unit is configured to further perform the following processing: acquiring operational data representing the angle of the display device. The control unit selects the past camera image data from the plurality of camera image data stored in the storage unit based on the relationship between the angle of the display device shown in the operation data and the multiple device angles corresponding to the same plurality of camera image data stored in the storage unit.

16. A data processing system comprising: a robot; an operating device that transmits operation data representing the content of an operation performed by an operator operating the robot; and a data processing device that controls the robot based on the operation data received from the operating device. in, The robot has the following characteristics: A camera device that generates camera image data; and The image transmitting unit sends the camera image data to the data processing device. The data processing device includes a control unit configured to perform the following processing: Acquire the camera image data sent by the robot; By combining at least a portion of the display image data acquired at a first time point with at least a portion of the past image data acquired at a second time point earlier than the first time point, composite image data with a wider viewing angle than the display image data is generated. as well as The synthesized image data is displayed on a display device that allows the robot operator to visually confirm it. The operating device has: An operation data transmission unit transmits operation data obtained based on the content of the operation performed by the operator to the data processing device; as well as The display unit displays the composite image data sent by the data processing device. The control unit generates the composite image data by combining the display image data obtained from the camera image data generated by the camera device shooting at a first camera angle, and the past camera image data obtained from the camera image data generated by the camera device shooting at a second camera angle different from the first camera angle. The data processing device further includes a storage unit, which stores the camera angle (i.e., the camera angle) at the time the camera image data acquired by the control unit is captured, in association with the camera image data. The control unit selects the past camera image data from the plurality of camera image data stored in the storage unit based on the relationship between the first camera angle and the plurality of camera angles corresponding to the same plurality of camera image data stored in the storage unit.

Images