Remote machine operation system, visual presentation device, visual presentation method, and program

Abstract

This remote machine operation system (100) is provided with: an imaging unit (26) that captures an image of a remote machine (2) that is remotely operated; and a visual presentation device (1) for presenting an image to an operator operating the remote machine (2) by receiving the image captured by the image capturing unit (26) and displaying the received image, the visual presentation device (1) being provided with: a communication status monitoring unit (11) for detecting the communication status during the reception of the image; and a presented image control unit (12) that determines a central field-of-view region and a peripheral field-of-view region in a display region in which the image is displayed, and that displays the displayed image on the basis of the communication status detected by the communication status monitoring unit (11). An image parameter related to the transmission rate of the central field-of-view image displayed in the central field-of-view region and the peripheral field-of-view image displayed in the peripheral field-of-view region is determined.

Description

Technical Field

[0001] The present disclosure relates to a remote machine operating system, a visual presentation device, a visual presentation method and a program for presenting images related to a remote machine. Background Art

[0002] Conventionally, there are devices that present images to an observer who is observing an object. Patent Document 1 discloses a stereoscopic image observation device that presents images to both eyes of an observer to provide a stereoscopic image.

[0003] Prior art literature

[0004] Patent Document 1: Japanese Patent Application Laid-Open No. 9-80353 Summary of the invention

[0005] The technology described in Patent Document 1 is based on the premise that the image is directly transmitted from the camera that obtains the image to the image presentation device, and the image data can be used without worrying about the capacity. On the other hand, in order for the operator to observe the image at a remote location away from the camera, the image is transmitted using a communication line. When the technology described in Patent Document 1 is applied to a system in which the operator observes the image at a remote location, there is a problem that a wideband and stable communication is always required, and an image suitable for the communication conditions cannot be presented.

[0006] The present disclosure is completed in view of the above situation, and its purpose is to obtain a remote machine operating system that can present images suitable for communication conditions.

[0007] In order to solve the above-mentioned problems and achieve the purpose, the remote machine operating system disclosed in the present invention comprises: a camera for photographing the remote machine being remotely operated; and a visual presentation device for presenting the image to the operator operating the remote machine by receiving the image photographed by the camera and displaying the received image. The visual presentation device comprises: a communication status monitoring unit for detecting the communication status during the reception of the image; and a presentation image control unit for determining the central field of view area and the peripheral field of view area in the display area where the image is displayed, and determining the image parameters related to the transmission rate of the central field of view image displayed in the central field of view area and the peripheral field of view image displayed in the peripheral field of view area according to the communication status detected by the communication status monitoring unit.

[0008] The remote machine operating system disclosed in the present invention has the effect of being able to present images suitable for communication conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 This is a diagram showing a configuration example of a remote machine operating system according to the first embodiment.

[0010] Figure 2 A diagram used to illustrate the characteristics of human vision.

[0011] Figure 3 This is a diagram showing an example of the display area of ​​the video according to the first embodiment corresponding to the central visual field and the peripheral visual field, respectively.

[0012] Figure 4 This is a flowchart showing an example of the image control processing procedure in the visual presentation device of embodiment 1.

[0013] Figure 5 This is a diagram showing an example of video setting information according to the first embodiment.

[0014] Figure 6 This is a diagram showing a structural example of a computer system that implements the visual presentation device of embodiment 1.

[0015] Figure 7 This is a diagram showing a configuration example of a remote machine operating system according to the second embodiment.

[0016] Figure 8 This is a diagram showing a configuration example of a remote machine operating system according to a third embodiment.

[0017] Fig. 9 This is a diagram showing an example of video setting information according to the third embodiment.

[0018] Fig.10 This is a diagram showing a configuration example of a remote machine operating system according to a fourth embodiment.

[0019] Fig.11 This is a diagram showing a configuration example of a remote machine operating system according to a fifth embodiment.

[0020] Fig.12 This is a diagram showing an example of an object detected by the device peripheral condition monitoring device according to the fifth embodiment.

[0021] Fig.13 This is a diagram showing a structural example of a remote machine operating system according to a sixth embodiment.

[0022] Fig.14 This is a diagram showing a structural example of a remote machine operating system according to a seventh embodiment.

[0023] Fig.15 This is a diagram showing a structural example of a remote machine operating system according to an eighth embodiment.

[0024] Fig.16 This is a diagram showing an example of an image displayed by a visual presentation device according to embodiment 8 without performing enlargement or reduction of the image.

[0025] Fig.17This is a diagram showing an example of an image displayed by a visual presentation device according to embodiment 8 in which the image is enlarged or reduced.

[0026] Fig.18 This is a diagram showing a structural example of a remote machine operating system according to a ninth embodiment. DETAILED DESCRIPTION

[0027] Hereinafter, based on the accompanying drawings, the remote mechanical operation system, visual presentation device, visual presentation method and program of the implementation method will be described in detail.

[0028] Implementation method 1.

[0029] Figure 1 1 is a diagram showing a configuration example of a remote machine operating system according to Embodiment 1. The remote machine operating system 100 according to this embodiment includes a visual presentation device 1 and a remote machine 2. The remote machine 2 is a machine that can be remotely operated by an operator who is located at a place far from the remote machine 2. The visual presentation device 1 presents an image obtained by the remote machine 2 to the operator.

[0030] The remote machine 2 can be, for example, an operating machine that performs operations such as construction, assembly, transportation, packaging, and assistance; it can also be a machine that moves such as a vehicle, an unmanned aircraft (unmanned aircraft, underwater drone, space drone), etc.; it can also be a machine that performs both operations and movement.

[0031] Telemechanics 2 Figure 1 As shown in FIG. 1 , the camera unit 21, the communication unit 22, the drive unit 23, the drive control unit 24 and the status acquisition device 25 are provided. Figure 1 As shown in the lower part of , the remote machine 2 is, for example, a construction machine, but is not limited to this as described above.

[0032] The camera unit 21 includes an imaging unit 26, an image processing unit 27, and a communication unit 28. The imaging unit 26 is a camera that captures images of the remote machine 2 that is remotely operated, and is specifically, for example, a monocular camera. The communication unit 28 communicates with the visual presentation device 1. The image processing unit 27 adjusts the resolution, frame rate, etc. of the image captured by the imaging unit 26 according to an instruction from the visual presentation device 1 received via the communication unit 28, and sends the adjusted image to the visual presentation device 1 via the communication unit 28.

[0033] The communication unit 22 communicates with the visual presentation device 1. The communication unit 22 receives control information for controlling the remote machine 2 from the visual presentation device 1, for example, and outputs the received control information to the drive control unit 24. The drive control unit 24 controls the drive unit 23 according to the control information. The drive unit 23 is various actuators, for example, a motor for moving the arm, manipulator, etc. of the remote machine 2, a motor for moving the remote machine 2, etc.

[0034] The status acquisition device 25 is, for example, an acceleration sensor, a force sensor, an arm, etc., or a camera provided on an arm, a manipulator, etc. The status of the movement of the remote machine 2 is detected, and the detection result is output to the communication unit 22. For example, the communication unit 22 sends the detection result detected by the status acquisition device 25 to the visual presentation device 1. In addition, the detection result obtained by the status acquisition device 25 is used to determine the action mode showing the status of the movement of the remote machine 2 as described later. In addition, the action mode showing the status of the action can be represented by a combination of an operation mode and a moving speed of the remote machine 2. For example, the operation mode is determined according to the type of the operation status, such as whether the operator focuses on a part or grasps a wider range. In the case where the operation mode is manually set by the operator, the status acquisition device 25 may not be provided. In addition, as described later, in the case where the operation mode is determined based on the image obtained by the camera unit 21, the status acquisition device 25 may not be provided.

[0035] In addition, Figure 1 In the embodiment, the remote machine 2 includes both a communication unit 22 and a communication unit 28 in the camera unit 21, but the communication unit 28 in the camera unit 21 may not be provided. Instead, the image acquired by the camera unit 21 is sent to the visual presentation device 1 via the communication unit 22, and instructions to the camera unit 21 from the visual presentation device 1 are received via the communication unit 22.

[0036] In addition, Figure 1 In the embodiment, the camera unit 21 is provided in the remote machine 2, but the camera unit 21 may be provided independently of the remote machine 2. In addition, a plurality of camera units 21 may be provided, and the visual presentation device 1 may select which camera unit 21 to present an image to the operator, or the operator may select which camera unit 21 to present an image to the operator.

[0037] like Figure 1As shown, the visual presentation device 1 presents the image to the operator operating the remote machine 2 by receiving the image captured by the camera unit 26 that captures the remote machine 2 and displaying the received image. Specifically, the visual presentation device 1 communicates with the remote machine 2 to obtain the image captured at a remote location where the remote machine 2 is installed, and remotely operates the remote machine 2. The communication line between the remote machine 2 and the visual presentation device 1 can be a wireless line, a wired line, or a mixture of wireless lines and wired lines. For example, communication lines such as 5G (5th Generation: 5th generation mobile communication system) and Beyond 5G that achieve large-capacity and low-latency transmission can be used, and communication lines other than these can also be used. In addition, the communication line can also be changed according to the change of the position of the remote machine 2 or the visual presentation device 1. For example, the communication line used can also be switched during the movement as the remote machine 2 moves.

[0038] The visual presentation device 1 receives an image from the camera unit 21 of the remote machine 2 via a communication line, and displays the received image on a monitor or the like. Figure 1 The presentation unit 10 shown is used to present the image to the operator. On the other hand, the visual presentation device 1 may not always be able to communicate with the camera unit 21 in a wideband and stable communication environment. In this embodiment, the data amount per unit time of the image, that is, the image transmission rate (transmission rate) is controlled according to the communication conditions in consideration of the characteristics of human vision. As a result, an image suitable for the communication conditions can be provided to the operator. In addition, an image suitable for the communication conditions is, for example, an image that reduces the communication band and suppresses the reduction in visibility of the operator in consideration of the characteristics of human vision.

[0039] Figure 2 It is a diagram used to illustrate the visual characteristics of humans. Figure 2 As shown, in general, a region corresponding to the central visual field 5 corresponding to the vicinity of the center of the visual field 4 of the eye 3 of the person requires a high-definition image in human vision, and the peripheral visual field 6 outside the central visual field 5 is less susceptible to the influence of resolution than the central visual field 5. That is, if the image corresponding to the central visual field 5 is kept at a high resolution, even if the resolution of the image corresponding to the peripheral visual field 6 is lower than the resolution of the image corresponding to the central visual field 5, the operator can recognize it as a natural image. Therefore, by making the resolution of the image corresponding to the peripheral visual field 6 lower than the resolution of the image corresponding to the central visual field 5, it is possible to suppress the frequency band used in the transmission of the image and present a natural image to the operator.

[0040] In addition, the size of the central field of view 5 changes according to the situation. For example, the size of the central field of view 5 changes according to the work performed by a person and the speed at which the person moves. Therefore, when presenting an image to an operator, it is also preferable to set the range of the image corresponding to the central field of view 5 according to the work performed by the person and the speed at which the person moves. Therefore, in the present embodiment, for example, a mode corresponding to the operation situation such as the work performed by the remote machine 2 and the speed at which the remote machine 2 moves is determined as the operation mode, and the size (dimension) of the image corresponding to the central field of view 5 is set according to the operation mode.

[0041] Figure 3 FIG. 1 is a diagram showing an example of the display area of ​​the image of the present embodiment corresponding to the central field of view and the peripheral field of view. Figure 3 As shown, in this embodiment, the image display area 201 presented by the visual presentation device 1, that is, the image display area 201 displayed by the visual presentation device 1 is divided into a central field of view area 202 for displaying a central field of view image corresponding to the central field of view 5, and a peripheral field of view area 203 for displaying a peripheral field of view image corresponding to the peripheral field of view 6. Figure 3 As shown in FIG. 1 , the size of the central visual field area 202 is changed according to the operation mode, that is, according to the operation status of the remote machine 2. Figure 3 In the embodiment, it is assumed that the presentation unit 10 in the visual presentation device 1 is a dome-shaped projection device, and the image display area 201 is represented by a circle (within an angle determined from the direction of the operator's line of sight), but the image display area 201 is not limited to this shape. For example, the image display area 201 may also be a rectangle, and when the image display area 201 is a rectangle, the central visual field area 202 is also a circle.

[0042] In addition, Figure 3 2 illustrates a case where the operator's line of sight is in the center of the image display area 201, but the location of the central visual field area 202 moves depending on where the operator's line of sight corresponds to in the image display area 201. The operator's line of sight direction is set by the operator as a gaze point, as described later. In addition, the gaze point may be predetermined according to the conditions for using the visual presentation device 1. For example, in an environment where the position of the operator's face is roughly determined and the position of the object of the operation is also determined, the gaze point may be predetermined.

[0043] In this embodiment, the visual presentation device 1 considers the above-mentioned visual characteristics of the person and determines the image parameters related to the image according to the communication status. The image parameters include, for example, resolution, frame rate, color, and at least one of the position and size of the central visual field area 202. For example, in a situation where the operator focuses on a specific object shown in the image obtained by the camera unit 21, even if the image transmission rate of the peripheral visual field image is reduced by making the resolution of the peripheral visual field image lower than the resolution of the central visual field image or limiting the color information, the operator can feel that it is a natural image. Therefore, the visual presentation device 1 monitors the communication status, for example, when a sufficient transmission speed is obtained, a high-resolution color image is displayed at a high frame rate in the entire image display area 201, and when the transmission speed is reduced, the image transmission rate is reduced by changing at least one of the resolution, frame rate, color, and size of the central visual field area 202. As a result, it is possible to suppress the frequency band used in the transmission of the image and present a natural image to the operator.

[0044] Back to Figure 1 The visual presentation device 1 includes a presentation unit 10, a communication status monitoring unit 11, a presentation image control unit 12, a remote machine status monitoring unit 13, a control information generating unit 14, a communication unit 15, a gaze point setting device 16, and an operation input device 17.

[0045] The communication unit 15 communicates with the remote machine 2. For example, the communication unit 15 receives an image from the remote machine 2 and sends control information for controlling the remote machine 2 to the remote machine 2. In addition, when the communication unit 15 receives the detection result of the status acquisition device 25 of the remote machine 2, it outputs the received detection result to the image presentation control unit 12. The communication status monitoring unit 11 monitors the communication status in the communication unit 15 and notifies the image presentation control unit 12 of information indicating the communication information. The communication status monitoring unit 11 can monitor the transmission speed (communication frequency band) in the communication unit 15, or the communication quality such as the bit error rate and the received signal strength, or can monitor both. In addition, the communication status monitoring unit 11 monitors at least the communication in the direction of receiving information from the remote machine 2 and the communication status of the downlink communication in the visual presentation device 1. That is, the communication status monitoring unit 11 detects at least the communication status in the reception of the image. The communication status monitoring unit 11 can also monitor the communication status of both the downlink and the uplink. Furthermore, the communication status detected by the communication status monitoring unit 11 is preferably an accurate measured value, but may be an estimated value when it is difficult to detect an accurate measured value.

[0046] The remote machine status monitoring unit 13 detects the status of the operation of the remote machine 2. For example, the remote machine status monitoring unit 13 detects at least one of the moving speed of the remote machine 2 and the distance between the remote machine 2 and the object to be operated by the remote machine 2. Specifically, the remote machine status monitoring unit 13 receives the image captured by the imaging unit 26 of the camera unit 21 via the communication unit 15, monitors the status of the remote machine 2 using the received image, and notifies the image presentation control unit 12 of information indicating the status of the remote machine 2. The information indicating the status of the remote machine 2 is, for example, information indicating the moving speed of the remote machine 2 estimated from the image, the position of the operating object in the remote machine 2 estimated from the image, and the like. The operating object may be set in advance, or may be extracted by the remote machine status monitoring unit 13 according to a specified condition. For example, the remote machine status monitoring unit 13 detects the operating object from the image based on the image indicating the operating object, thereby detecting the position of the operating object. In addition, when the remote machine status monitoring unit 13 can calculate the distance between the work object and the remote machine 2 based on the size of the detected work object, etc., it can also estimate the distance and notify the presentation image control unit 12 as information indicating the status of the remote machine 2. In addition, the status acquisition device 25 in the remote machine 2 is also another example of the remote machine monitoring unit. That is, the remote machine monitoring unit that detects the status of the operation of the remote machine 2 can also be set in the remote machine 2. It can also be set in the visual presentation device 1. In addition, the remote machine status monitoring unit 13 and the status acquisition device 25 can also detect the relative distance and relative speed to the surrounding moving objects (not the objects of operation) and obstacles (including the remote machine 2 itself in operation) of the remote machine 2 being operated as the status of the operation.

[0047] The gaze point setting device 16 sets the gaze point of the operator. Specifically, the gaze point setting device 16 receives input of the gaze point from the operator and notifies the presentation image control unit 12 of information indicating the input gaze point. The gaze point setting device 16 may be, for example, an input device such as a joystick, a touch pad, a button, a keyboard, a mouse, a game controller, or a device that receives input by detecting the operator's posture based on an image obtained by photographing the operator or the muscle activity of the operator. The gaze point setting device 16 may be any device as long as it is an input device that can receive input from the operator, and is not limited to the above-mentioned example. The gaze point may be specified as a position in the image display area 201 displayed by the presentation unit 10, or as an object in the image displayed in the image display area 201. When specified as an object, the image processing unit 27 of the camera unit 21 extracts the object from the image to determine the position of the gaze point when receiving information indicating the gaze point from the visual presentation device 1 via the communication unit 28.

[0048] In the presented image control unit 12, the presented image control unit 12 sets the action mode, and determines the image parameters instructed to the camera unit 21 using the set action mode, the information indicating the communication status received from the communication status monitoring unit 11, and the information indicating the gaze point received from the gaze point setting device 16. For example, the presented image control unit 12 sets the action mode using at least one of the detection result sent from the status acquisition device 25 received from the communication unit 15 and the information indicating the status of the remote machine 2 received from the remote machine status monitoring unit 13. Alternatively, the action mode may be input from the operator via the gaze point setting device 16 or the operation input device 17, and the action mode may be set based on the input. The presented image control unit 12 determines the central visual field area 202 based on the gaze point set by the gaze point setting device. That is, the presented image control unit 12 determines the central visual field area 202 and the peripheral visual field area 203 in the display area for displaying the image. Furthermore, the presented image control unit 12 determines image parameters related to the transmission rate (image transmission rate) of the central view image and the peripheral view image based on the communication status detected by the communication status monitoring unit 11. The image parameters related to the transmission rate are parameters that affect the transmission rate among the image parameters.

[0049] The image parameters include, for example, resolution, frame rate, color, and at least one of the position and size of the central visual field area 202 as described above. The image presentation control unit 12 outputs the determined image parameters to the communication unit 15, and the communication unit 15 transmits the image parameters received from the image presentation control unit 12 to the camera unit 21. In addition, regarding these image parameters, only the changed parameters may be transmitted when they are changed, all the parameters may be transmitted when any parameters are changed, or all the parameters may be transmitted periodically. The camera unit 21 transmits the image to the visual presentation device 1 according to the image parameters indicated by the image presentation control unit 12. When the image is received from the camera unit 21 via the communication unit 15, the image presentation control unit 12 displays the image on the presentation unit 10.

[0050] The presentation unit 10 is a device that displays the image captured by the camera unit 21, and is, for example, a dome-shaped monitor, a display, a head-mounted display, a terminal such as a tablet, a smartphone, a projector, and a screen, and is not limited to these devices as long as it can display an image. When a device that can display an image in the up-down direction such as a dome-shaped monitor is used as the presentation unit 10, the operator can ensure the field of view in the up-down direction, so that, for example, when the remote machine 2 is a mobile machine, it is easy to confirm the surrounding area of ​​the lower part of the remote machine 2, that is, the foothold of the remote machine 2, which helps to support the remote operation of the movement of the remote machine 2.

[0051] The operation input device 17 is an input device that receives operation input of the remote machine 2 from the operator. It may be an input device that receives operation by detecting the operator's posture, or an input device such as a joystick, a touch panel, a button, a keyboard, a mouse, or a game controller. The operation input device 17 outputs operation information indicating the operation received from the operator to the control information generation unit 14. Figure 1 In FIG. 1 , the gaze point setting device 16 is described separately from the operation input device 17 , but the gaze point setting device 16 may be the same as the operation input device 17 or may be a part of the operation input device 17 .

[0052] The control information generating unit 14 generates control information for controlling the operation of the remote machine 2 using the operation information received from the operation input device 17 , and outputs the generated control information to the communication unit 15 . The control information is transmitted to the remote machine 2 via the communication unit 15 .

[0053] Next, the image control according to this embodiment will be described. Figure 4 1 is a flowchart showing an example of the image control processing process in the visual presentation device 1 of this embodiment. Figure 4 As shown, the visual presentation device 1 sets the action mode (step S1). Specifically, as described above, for example, the action mode can be set by the presentation image control unit 12 using at least one of the detection result sent from the status acquisition device 25 and the information indicating the status of the remote machine 2 received from the remote machine status monitoring unit 13, or can be set by the operator's input.

[0054] The visual presentation device 1 sets the image parameters according to the action mode (step S2). Specifically, the image presentation control unit 12 sets the image parameters according to the action mode. For example, the image presentation control unit 12 maintains the initial value of the image parameter of each action mode as the image setting information, and sets the image parameters corresponding to the action mode with reference to the image setting information.

[0055] Figure 5 FIG. 1 is a diagram showing an example of image setting information according to the present embodiment. Figure 5 In the example shown, each row of the image setting information represents one action mode, and the action mode is represented by a combination of an operation mode and a moving speed (moving speed of the remote machine 2). The image setting information includes the initial values ​​of the image parameters corresponding to each action mode. Figure 5In the image setting information, as initial values ​​of image parameters, resolution, size of the central field of view image (size of the central field of view area 202), and frame rate are included. The image parameters included in the image setting information are not limited to these. In addition, information indicating whether it is a color image or a monochrome image may be included as information related to color in the initial values ​​of the image parameters. In addition, the image parameters in the image setting information are set to values ​​corresponding to the case where the communication condition is optimal, that is, the case where the transmission speed is not restricted, for example. Here, as initial values, the resolution, frame rate, etc. are set to be the same in the central field of view area 202 and the peripheral field of view area 203, but the initial values ​​may be different in the central field of view area 202 and the peripheral field of view area 203. In this case, the image setting information includes the values ​​of the central field of view area 202 and the peripheral field of view area 203 as image parameters. For example, the image setting information is determined for each action mode in a manner that makes the operator feel easy to operate through experiments, etc., but the method of determining the image setting information is not limited to this example. For example, the image setting information may also be determined by learning from data during operation.

[0056] Back to Figure 4 Description. The visual presentation device 1 detects the communication status (step S3). In detail, the communication status monitoring unit 11 detects the communication status and notifies the detection result to the presentation image control unit 12. Next, the visual presentation device 1 determines whether the communication status has changed (step S4). In detail, the presentation image control unit 12 determines whether the communication status has changed. In addition, in the first step S4, it is assumed that the communication status is optimal and the image parameters are set, so the presentation image control unit 12 determines whether the communication status has changed from the situation where the communication status is optimal. In the second and subsequent steps S4, the presentation image control unit 12 determines whether the communication status has changed from the time of the last step S4. For example, it is determined whether the communication status has changed based on whether the transmission speed has changed by more than a certain amount. For example, the transmission speed can also be defined in a range of multiple stages, such as when the transmission speed is greater than Z1 (Mbps) and less than Z2 (Mbps), it is set to level 1, and when it is greater than Z2 (Mbps) and less than Z3 (Mbps), it is set to level 2, and when there is a change in level, it is determined that the communication status has changed.

[0057] When the communication status has changed (step S4 "yes"), the visual presentation device 1 changes the image parameters (step S5). In detail, the image presentation control unit 12 changes the image parameters in a manner that makes them correspond to the communication status. The image parameters of the change object include, for example, the resolution of the peripheral field of view image, the frame rate of the peripheral field of view image, the color of the peripheral field of view area, and at least one of the size of the central field of view area 201. For example, for each action mode, the priority of change is determined for each of these image parameters, and the image presentation control unit 12 changes the image parameters in order from high to low priority of change until the image transmission rate that can be transmitted at the current transmission speed is achieved. For example, in a certain action mode, when the priority of resolution change is the highest, the image presentation control unit 12 reduces the resolution of the peripheral field of view image in a manner that can be transmitted at the current transmission speed. In addition, the lower limit value of each image parameter when reducing is determined, and the image presentation control unit 12 changes the image parameter with the next higher priority (priority of change) when the image transmission rate that can be transmitted at the current transmission speed is not achieved even if the resolution of the peripheral field of view image is reduced to the lower limit value. In this way, by changing the image parameters in descending order of change priority to reduce the image transmission rate, the image presentation control unit 12 determines the image parameters in such a manner that the image transmission rate corresponds to the communication status. In addition, when the communication status changes in the direction of improvement, the image presentation control unit 12 may also change the image parameters in such a manner that the image transmission rate increases in order to achieve the image transmission rate corresponding to the communication status.

[0058] Next, the visual presentation device 1 determines whether the action mode has been changed (step S6). Specifically, the presentation image control unit 12 determines the action mode using at least one of the detection result sent from the status acquisition device 25 and the information indicating the status of the remote machine 2 received from the remote machine status monitoring unit 13, and determines whether the determined action mode is different from the currently set action mode. Alternatively, the presentation image control unit 12 determines whether the action mode input by the operator is different from the currently set action mode.

[0059] If the action mode has not been changed (step S6 "No"), the image presentation control unit 12 performs the processing from step S3 again. If the action mode has been changed (step S6 "Yes"), the image presentation control unit 12 determines whether the operation is completed (step S7). Specifically, the image presentation control unit 12 may determine that the operation is completed when there is an input from the operator indicating the end of the operation, or may include a mode indicating the end of the operation in the action mode, and determine that the operation is completed when the changed action mode is a mode indicating the end of the operation.

[0060] If the operation is not completed (step S7 "No"), the image presentation control unit 12 sets the image parameters according to the action mode (step S8), and performs the processing from step S3 again. In step S8, in detail, the image presentation control unit 12 can set the image parameters according to the image setting information as in step S2, or change a part of the image parameters indicated by the image setting information. For example, in step S8, only the size of the central visual field area 202 among the image parameters indicated by the image setting information can be changed to a value corresponding to the changed action mode.

[0061] If the job is completed (Yes in step S7 ), the presented image control unit 12 ends the process. If No in step S4 , the presented image control unit 12 advances the process to step S6 .

[0062] Through the above processing, the image parameters corresponding to the operation mode and communication status are set. Figure 4 The processing shown is only an example, and the specific processing process is not limited to Figure 4 In the above example, the image setting information includes the initial value of the image parameter when the communication condition is optimal for each operation mode, but the image setting information may be determined for each communication condition. For example, the image setting information may be determined for each level of the above communication condition, and in step S2, the image presentation control unit 12 sets the image parameters using the image setting information corresponding to the communication condition.

[0063] In addition, in the example described above, the image obtained by photographing the remote machine 2 is used as an example for explanation, but the image presented by the visual presentation device 1 is not limited to the image obtained by photographing the remote machine 2, but can also be a landscape, a person, an animal, or other things photographed by a camera. For example, the camera can also be set on a person's shoulder, in a forest, in a river, in space, etc., and the communication unit 15 of the visual presentation device 1 receives the image from the camera. In this case, for example, the image parameters of the central field of view image and the peripheral field of view image are set according to the communication conditions, such as reducing the image transmission rate of the peripheral field of view image when the communication conditions deteriorate, so that a suitable image can be presented according to the communication conditions. In the case where the camera is mounted on a person's shoulder, etc., the image parameters of the central field of view image and the peripheral field of view image can also be determined according to the person's moving speed.

[0064] Next, the hardware configuration of the visual presentation device 1 of this embodiment will be described. The visual presentation device 1 of this embodiment causes the computer system to function as the visual presentation device 1 by executing a program as a computer program describing the processing in the visual presentation device 1 on the computer system. Figure 61 is a diagram showing a configuration example of a computer system that implements the visual presentation device 1 of this embodiment. Figure 6 As shown, the computer system includes a control unit 101, an input unit 102, a storage unit 103, a display unit 104, a communication unit 105, and an output unit 106, which are connected via a system bus 107. The control unit 101 and the storage unit 103 constitute a processing circuit.

[0065] exist Figure 6 In the embodiment, the control unit 101 is a processor such as a CPU (Central Processing Unit), which executes a program describing the processing in the visual presentation device 1 of this embodiment. In addition, a part of the control unit 101 can also be implemented by dedicated hardware such as a GPU (Graphics Processing Unit) and an FPGA (Field-Programmable Gate Array). As described above, the input unit 102 can be an input device such as a joystick, a touch pad, a button, a keyboard, a mouse, a game controller, or an input device that accepts input by detecting the operator's posture based on the image obtained by photographing the operator or the muscle activity of the operator. The storage unit 103 includes various memories such as RAM (Random Access Memory), ROM (Read Only Memory), and storage devices such as a hard disk, which store the program to be executed by the above-mentioned control unit 101, the necessary data obtained during the processing, etc. In addition, the storage unit 103 is also used as a temporary storage area for programs. As described above, the display unit 104 is, for example, a dome-shaped monitor, a display, a head-mounted display, a tablet, a smartphone, or other terminal, a projector, or a screen. In addition, the display unit 104 and the input unit 102 may be integrated and implemented by a touch panel or the like. The communication unit 105 is a receiver and a transmitter for performing communication processing. The output unit 106 is a speaker or the like. In addition, Figure 6 is an example, the structure of the computer system is not limited to Figure 6 Example.

[0066] Here, an example of the operation of the computer system until the program of the present embodiment becomes executable is described. In the computer system adopting the above structure, the computer program is installed in the storage unit 103 from a CD-ROM or DVD-ROM placed in a CD (Compact Disc)-ROM drive or DVD (Digital Versatile Disc)-ROM drive (not shown). And when the program is executed, the program read from the storage unit 103 is stored in the main storage area of ​​the storage unit 103. In this state, the control unit 101 performs the processing of the visual presentation device 1 of the present embodiment according to the program stored in the storage unit 103.

[0067] In addition, in the above description, a CD-ROM or DVD-ROM is used as a recording medium to provide a program that records the processing in the visual presentation device 1, but it is not limited to this. It is also possible to use a program provided via a transmission medium such as the Internet, depending on the structure of the computer system, the capacity of the program provided, etc.

[0068] For example, the program of this embodiment causes the visual presentation device 1 to execute: a step of detecting the communication status in the reception of an image; and a step of dividing the display area for displaying the image into a central field of view area 202 and a peripheral field of view area 203, and determining image parameters related to the transmission rates of the central field of view image displayed in the central field of view area 202 and the peripheral field of view image displayed in the peripheral field of view area 203 according to the detected communication status.

[0069] By Figure 6 The control unit 101 shown executes Figure 6 The computer program stored in the storage unit 103 shown in the figure is used to implement Figure 1 The communication status monitoring unit 11, the image presentation control unit 12, the remote machine status monitoring unit 13 and the control information generating unit 14 are shown. Figure 1 The communication status monitoring unit 11, the image display control unit 12, the remote machine status monitoring unit 13 and the control information generating unit 14 shown in the figure also use Figure 6 The storage unit 103 is shown. Figure 1 The communication unit 15 shown is Figure 6 The communication unit 105 shown is implemented. Figure 1 The gaze point setting device 16 and the operation input device 17 shown in the figure are Figure 6 The input unit 102 shown is implemented. Figure 1 The presenting section 10 shown is Figure 6 The display unit 104 shown is implemented.

[0070] In addition, Figure 1In the example shown, the gaze point setting device 16, the operation input device 17, and the presentation unit 10 are included in the visual presentation device 1, but at least one of the gaze point setting device 16, the operation input device 17, and the presentation unit 10 may not be included in the visual presentation device 1. In this case, the visual presentation device 1 exchanges information by connecting with at least one of the gaze point setting device 16, the operation input device 17, and the presentation unit 10 that are not included in the visual presentation device 1. Figure 1 In the example shown, the communication status monitoring unit 11 and the presented image control unit 12 are included in the visual presentation device 1 , but either or both of them may be included in the remote machine 2 .

[0071] Figure 1 The image processing unit 27 in the remote machine 2 shown in the figure is provided with, for example, Figure 6 The control unit 101 and the storage unit 103 are implemented by a processing circuit. The processing circuit is also a computer system. Figure 1 The communication unit 22 and the communication unit 28 shown are a receiver and a transmitter.

[0072] As described above, the visual presentation device 1 of the present embodiment divides the image display area 201 into the central visual field area 202 and the peripheral visual field area 203, and determines the image parameters displayed in each area according to the operation status and communication status of the remote machine 2. Therefore, it is possible to present an image suitable for the communication status. In particular, when the actual transmission speed is a constant speed compared to the image transmission rate corresponding to the currently set image parameters, the image parameters are determined in such a way that the image transmission rate of the peripheral visual field image is reduced, thereby suppressing the frequency band used in the transmission of the image and presenting a natural image to the operator.

[0073] Implementation method 2.

[0074] Figure 7 1 is a diagram showing a configuration example of a remote machine operating system according to Embodiment 2. A remote machine operating system 100a according to this embodiment includes a visual presentation device 1a and a remote machine 2 similar to Embodiment 1. Components having the same functions as those in Embodiment 1 are denoted by the same reference numerals as those in Embodiment 1 and repeated descriptions are omitted. The following mainly describes the differences from Embodiment 1.

[0075] The visual presentation device 1a is the same as the visual presentation device 1 of the first embodiment except that the gaze point setting device 16 is replaced with a sight line measuring device 18 as another example of the gaze point setting device. The sight line measuring device 18 as an example of the gaze point setting device measures the sight line (the direction of the sight line) of the operator and outputs the measurement result to the presentation image control unit 12. The sight line is information indicating the gaze point, and the presentation image control unit 12 determines the gaze point based on the measurement result, and uses the gaze point to determine the central visual field area 202. In addition, the sight line measuring device 18 may not be included in the visual presentation device 1a, and the visual presentation device 1a may obtain the measurement result from the sight line measuring device 18. In addition, the visual presentation device 1a may also include a gaze point setting device 16, and it may be set to be able to switch between the gaze point setting device 16 and the gaze sight line measuring device 18. The operation of this embodiment other than the above description is the same as that of the first embodiment.

[0076] The visual presentation device 1a of this embodiment uses the sight line measuring device 18 to determine the gaze point, so there is no need for the operator to set the gaze point, which can reduce the burden on the operator.

[0077] Implementation method 3.

[0078] Figure 8 1 is a diagram showing a configuration example of a remote machine operating system according to Embodiment 3. The remote machine operating system 100b of this embodiment includes the same visual presentation device 1 and remote machine 2a as those of Embodiment 1. The same reference numerals as those of Embodiment 1 are attached to the components having the same functions as those of Embodiment 1 and repeated descriptions are omitted. The following mainly describes the differences from Embodiment 1.

[0079] In the remote machine 2a, except that the camera unit 21 is replaced by a camera unit 21a, it is the same as the remote machine 2 of the embodiment 1. In the camera unit 21a, except that the camera unit 26 and the image processing unit 27 are replaced by a compound eye camera unit 29 and an image processing unit 27a, it is the same as the camera unit 2 of the embodiment 1. The compound eye camera unit 29 is a compound eye camera. The compound eye camera unit 29 is, for example, a three-eye camera, and can provide a stereoscopic view image and a plane view image by photographing an image for the left eye, an image for the right eye, and an image for a plane view. Alternatively, the compound eye camera unit 29 can also photograph an image for the left eye and an image for the right eye, and use the image for the left eye and the image for the right eye to produce a plane view image. The communication unit 28 sends the image photographed by the compound eye camera unit 29 to the visual presentation device 1.

[0080] In this embodiment, the image parameters may include whether a stereoscopic view is provided, that is, whether the image is a stereoscopic view image or a flat view image. Specifically, for example, the image parameters may include information indicating whether the central view image and the peripheral view image are respectively provided as stereoscopic view images. Fig. 9 : is a diagram showing an example of image setting information of the present embodiment. For example, in the case where the remote machine 2a is moved in a state where the remote machine 2a is close to an object, the operator can easily grasp the distance between the remote machine 2a and the object when using a stereoscopic view image, and the operation of avoiding interference between the remote machine 2a and the object becomes easy. On the other hand, in the case where the remote machine 2a is a vehicle or the like and moves at a high speed, the necessity of a stereoscopic view image is low. In addition, for example, depending on the action mode, if the central field of view image is made a stereoscopic view image, the operator can recognize a natural image without any sense of disobedience even if the peripheral field of view image is a flat view image.

[0081] In the present embodiment, the image presentation control unit 12 may also change the presence or absence of a stereoscopic view as an image parameter according to the communication status. For example, when a stereoscopic view image is displayed in the entire image display area 201, when the transmission speed is reduced, the image transmission rate of the peripheral field of view image is reduced by changing the peripheral field of view image to a planar view image. In addition, the image presentation control unit 12 may also add the presence or absence of a stereoscopic view to the image parameters described in Embodiment 1, and determine the priority of the change of each image parameter as described in Embodiment 1, thereby changing the image parameters in descending order of priority so as to achieve an image transmission rate corresponding to the communication status.

[0082] When the image processing unit 27a of the camera unit 21a is instructed by the visual presentation device 1 whether to change the stereoscopic view, the image processing unit 27a adjusts the image output to the communication unit 28 according to the instruction. For example, when the image processing unit 27a receives an instruction to set the central field of view image as a stereoscopic view image and the peripheral field of view image as a planar view image, the portion corresponding to the central field of view image becomes a stereoscopic view image, and the portion corresponding to the peripheral field of view image becomes a planar view image. In addition, as described above, the compound eye camera unit 29 may shoot the left eye image and the right eye image, and the image processing unit 27a may use the left eye image and the right eye image to infer and create a planar view image corresponding to the image in the middle of the two. In addition, the compound eye camera unit 29 may be equipped with a compound eye camera for stereoscopic view images and a monocular camera for planar view, and the image processing unit 27a may use the images shot by these two cameras as stereoscopic view images and planar view images.

[0083] As the presentation unit 10 of the visual presentation device 1, an anaglyph, deflection glasses, active shutter glasses, a head mounted display, a lens device, etc. may also be used. The stereoscopic images of multiple cameras may also include images synthesized by an image synthesis function. The operations of this embodiment other than those described above are the same as those of the first embodiment.

[0084] The remote machine operating system 100b of this embodiment uses the compound eye camera unit 29, so it is possible to present a stereoscopic image to the operator. In addition, by setting the central field of view image as a stereoscopic image and the peripheral field of view image as a flat field of view image, it is possible to reduce the image transmission rate compared to the case where the stereoscopic image is displayed in the entire image display area 201.

[0085] Implementation method 4.

[0086] Fig.10 1 is a diagram showing a configuration example of a remote machine operating system according to Embodiment 4. The remote machine operating system 100c of this embodiment includes the same visual presentation device 1 and remote machine 2b as Embodiment 1. The same reference numerals as Embodiments 1 and 3 are given to components having the same functions as Embodiments 1 and 3, and repeated descriptions are omitted. The following mainly describes the differences from Embodiments 1 and 3.

[0087] The remote machine 2b is the same as the remote machine 2a of Embodiment 3 except that a camera unit 21b is provided instead of the camera unit 21a. The camera unit 21b is the same as the camera unit 21a of Embodiment 3 except that a camera driving unit 30 is added.

[0088] The camera driving unit 30 controls at least one of the orientation of the compound eye camera unit 29, that is, the direction of the camera, the distance of the compound eye camera unit 29 (the distance between the two cameras of the compound eye camera), and the convergence angle. In the case where the camera driving unit 30 can change the orientation of the compound eye camera unit 29, the camera driving unit 30 receives, for example, from the communication unit 22 information indicating the orientation of the compound eye camera unit 29, which is input to the visual presentation device 1 through the operation of the operator, and changes the orientation of the compound eye camera unit 29 according to the received information. Thus, the orientation of the compound eye camera unit 29 can be set according to the operator's request. In the case where the camera driving unit 30 can change the distance of the compound eye camera unit 29, the camera driving unit 30 receives, for example, from the communication unit 22 information indicating the distance of the compound eye camera unit 29, which is input to the visual presentation device 1 through the operation of the operator, and changes the distance of the compound eye camera unit 29 according to the received information. Thus, the depth of the stereoscopic view image can be set according to the operator's request. In addition, the distance of the compound eye camera unit 29 can also be set by the image presentation control unit 12 according to the action mode. For example, the image parameter can also include the distance of the compound eye camera unit 29. Thus, it is possible to set a suitable depth resolution according to the action mode. In addition, in a case where the camera drive unit 30 is capable of changing the convergence angle of the compound eye camera unit 29, the camera drive unit 30 receives information representing the convergence angle of the compound eye camera unit 29 input to the visual presentation device 1 through the operation of the operator from the communication unit 22, for example, and changes the convergence angle of the compound eye camera unit 29 according to the received information. Sometimes the stereoscopic viewing angle becomes comfortable when the convergence angle is adjusted, but sometimes the resolution of depth perception is higher when the convergence angle is not adjusted. Therefore, by setting the convergence angle to be set through the operation of the operator, it is possible to provide an image corresponding to the operator's expectations. The actions of this embodiment other than the above description are the same as those of embodiment 1.

[0089] In addition, Fig.10 , an example of using the gaze point setting device 16 is shown, but the sight line measurement device 18 described in Embodiment 2 may be used instead of the gaze point setting device 16. In addition, when a monocular camera is used as in Embodiment 1, a camera driving unit 30 may be added, and the direction of the imaging unit 26 may be changed.

[0090] Implementation method 5.

[0091] Fig.11 1 is a diagram showing a configuration example of a remote machine operating system according to Embodiment 5. A remote machine operating system 100d according to this embodiment includes a visual presentation device 1b and a remote machine 2b similar to Embodiment 4. Components having the same functions as Embodiments 1 and 4 are denoted by the same reference numerals as Embodiments 1 and 4, and repeated descriptions are omitted. The following mainly describes the differences from Embodiments 1 and 4.

[0092] The visual presentation device 1b is the same as the visual presentation device 1 of Embodiment 1 except that a device peripheral condition monitoring device 19 is added. The device peripheral condition monitoring device 19 detects whether there is an object in the field of view of the operator corresponding to the central field of view image. For example, the device peripheral condition monitoring device 19 detects whether there is an object between the operator and the presentation unit 10 that displays the image and in the area corresponding to the central field of view area 202. The device peripheral condition monitoring device 19 may be, for example, a camera and an image processing device that captures the space between the operator and the presentation unit 10, or a distance sensor or the like. In the case where the device peripheral condition monitoring device 19 is a camera and an image processing device, the image processing device determines whether there is an object at a position corresponding to the central field of view area 202 between the operator and the presentation unit 10 based on the image captured by the camera, and outputs the determination result as a detection result to the presentation image control unit 12.

[0093] Fig.12 FIG. 1 is a diagram showing an example of an object detected by the device peripheral condition monitoring device 19 of the present embodiment. Fig.12 In the example shown, an object 211 exists in the area corresponding to the central visual field area 202 between the operator 210 and the presentation unit 10. When the stereoscopic view image is displayed as the central visual field image, the presentation image control unit 12 reduces the resolution of the peripheral visual field image when it is determined that the object 211 exists in the area corresponding to the central visual field area 202 between the operator 210 and the presentation unit 10 based on the detection result of the device peripheral condition monitoring device 19. This is based on the following experimental results: when the object 211 exists in the area corresponding to the central visual field area 202 between the operator 210 and the presentation unit 10 based on the detection result of the device peripheral condition monitoring device 19, the operator 210 cannot achieve a stereoscopic view, but can continue to achieve a stereoscopic view when the resolution of the peripheral visual field image is low. Therefore, when the presented image control unit 12 displays a stereoscopic view image as the central view image, if it is determined based on the detection result of the device surrounding situation monitoring device 19 that an object 211 exists in the area corresponding to the central view area 202 between the operator 210 and the presentation unit 10, the operator can continue to realize the stereoscopic view of the central view image by reducing the resolution of the peripheral view image. That is, when the central view image is a stereoscopic view image, the presented image control unit reduces the resolution of the peripheral view image when the device surrounding situation monitoring device 19 detects the existence of an object. In addition, the area corresponding to the central view area 202 includes the vicinity of the central view area 202 which is a peripheral view close to the central view and is out of the stereoscopic view.

[0094] In addition, Fig.11 , an example using the gaze point setting device 16 is shown, but the sight line measuring device 18 described in Embodiment 2 may be used instead of the gaze point setting device 16. In addition, in the remote mechanical operating system 100b described in Embodiment 3, the device surrounding condition monitoring device 19 may be added similarly, and the operation described in this embodiment may be performed by the presented image control unit 12. In addition, when a monocular camera is used as in Embodiment 1, the device surrounding condition monitoring device 19 may be added, and the operation described in this embodiment may be performed by the presented image control unit 12.

[0095] Implementation method 6.

[0096] Fig.13 1 is a diagram showing a configuration example of a remote machine operating system according to Embodiment 6. The remote machine operating system 100e of this embodiment includes a visual presentation device 1b and a remote machine 2c similar to Embodiment 5. Components having the same functions as Embodiments 1, 4, and 5 are denoted by the same reference numerals as Embodiments 1, 4, and 5, and repeated descriptions are omitted. The following mainly describes the differences from Embodiments 1, 4, and 5.

[0097] The remote machine 2c is the same as the remote machine 2b of Embodiment 4 except that a camera unit 21c is provided instead of the camera unit 21b. The camera unit 21c is the same as the camera unit 21b of Embodiments 4 and 5 except that a zoom control unit 31 is added.

[0098] The zoom control unit 31 controls the zoom-in and zoom-out of the image captured by the compound eye camera unit 29. The zoom control unit 31 can control the zoom-in and zoom-out by optical zooming or by digital zooming. In the case where the zoom control unit 31 controls the zoom-in and zoom-out by optical zooming, for example, the compound eye camera unit 29 can change the focus by driving the optical system, or a variable focus lens can be used in the compound eye camera unit 29. In the case where the zoom control unit 31 controls the zoom-in and zoom-out by digital zooming, the zoom control unit 31 zooms in by cutting out and stretching a part of the captured image. For example, the zoom (zoom-in, zoom-out) can also be set by the presentation image control unit 12 according to the action mode. For example, the image parameters can also include zooming. Thus, the zoom can be set according to the action mode, and the operator's depth perception and operability can be improved. In addition, the zoom control unit 31 can also receive information indicating zooming in or out input to the visual presentation device 1b through the operator's operation from the communication unit 22, and control the zoom of the compound eye camera unit 29 according to the received information. In addition, the zoom control unit 31 may control the compound eye imaging unit 29 in cooperation with the zooming and reducing of the image. For example, the camera driving unit 30 may control at least one of the distance between cameras and the convergence angle in the compound eye imaging unit 29 according to the magnification determined by the zoom control unit 31.

[0099] In addition, as one of the methods for reducing the image transmission rate when the communication condition deteriorates, a method of reducing the range of the image captured as the central field of view and stretching the image of the area to enlarge it can be adopted, and a method of reducing the entire range of the image captured and stretching the image of the area to enlarge it can also be adopted. That is, as one of the image parameters changed according to the communication condition, scaling can also be included. The operation of this embodiment other than the above description is the same as that of embodiment 5.

[0100] In addition, Fig.13 , an example of using the gaze point setting device 16 is shown, but the sight line measurement device 18 described in Embodiment 2 may be used instead of the gaze point setting device 16. In addition, when a monocular camera is used as in Embodiment 1, the zoom control described in this embodiment may also be applied. In addition, the zoom control described in this embodiment may be added to the remote mechanical operating systems 100b and 100c described in Embodiments 3 and 4.

[0101] Implementation method 7.

[0102] Fig.141 is a diagram showing a configuration example of a remote machine operating system according to Embodiment 7. A remote machine operating system 100f according to this embodiment includes a visual presentation device 1b and a remote machine 2d similar to Embodiment 5. Components having the same functions as Embodiments 1, 5, and 6 are denoted by the same reference numerals as Embodiments 1, 5, and 6, and repeated descriptions are omitted. The following mainly describes points different from Embodiments 1, 5, and 6.

[0103] The remote machine 2d is the same as the remote machine 2c of the sixth embodiment except that a distance measuring sensor 32 is added. The distance measuring sensor 32 measures, for example, the distance between the remote machine 2d and an object such as an object to be operated by the remote machine 2d (operation object). For example, when the remote machine 2d has an end effector, the distance measuring sensor 32 measures the distance between the end effector and the operation object. The distance measuring sensor 32 outputs the measurement result to the zoom control unit 31. The zoom control unit 31 determines a magnification based on the measurement result, and controls the enlargement and reduction of the image captured by the compound eye camera unit 29 based on the determined magnification. For example, the zoom control unit 31 enlarges the image when the distance between the remote machine 2d and the operation object becomes shorter, and reduces the image when the distance between the remote machine 2d and the operation object becomes longer, based on the measurement result. For example, by determining a magnification indicating a ratio of enlargement or reduction for each distance between the remote machine 2d and the operation object, the zoom control unit 31 controls the enlargement and reduction of the image captured by the compound eye camera unit 29.

[0104] Thus, for example, when the end effector of the remote machine 2d approaches the operation object, the image is enlarged without the need for the operator's operation, so the operator's load can be reduced and the operability can be improved. In addition, when the remote machine 2d is a drone, when the operation object is set to be a place to land such as the ground, by enlarging when approaching the place to land, the operator's load can be reduced and the operability can be improved. In addition, when the remote machine 2d is a vehicle, when driving in a place with an obstacle, by enlarging when approaching the obstacle, the operator's load can be reduced and the operability can be improved. The actions of this embodiment other than the above description are the same as those of embodiment 6. In addition, the operator can also set the on / off (ON / OFF) of the automatic zoom control using the measurement results of the ranging sensor 32 (whether to set it to be effective or invalid).

[0105] Alternatively, the measurement result of the distance measuring sensor 32 may be sent to the visual presentation device 1b via the communication unit 22, and the presentation image control unit 12 receives the measurement result via the communication unit 15, and uses the received measurement result to determine the zoom ratio. The zoom control unit 31 receives the determined ratio via the communication unit 15 and the communication unit 28, and controls the zoom of the compound eye camera unit 29 according to the received ratio.

[0106] In addition, Fig.14 , an example using the gaze point setting device 16 is shown, but the sight line measurement device 18 described in Embodiment 2 may be used instead of the gaze point setting device 16. In addition, when a monocular camera is used as in Embodiment 1, the zoom control using the measurement result of the ranging sensor 32 described in this embodiment may also be applied. In addition, the zoom control using the measurement result of the ranging sensor 32 described in this embodiment may be added to the remote mechanical operating systems 100b and 100c described in Embodiments 3 and 4.

[0107] Implementation method 8.

[0108] Fig.15 1 is a diagram showing a configuration example of a remote machine operating system according to an eighth embodiment. A remote machine operating system 100g according to the eighth embodiment includes a visual presentation device 1c and a remote machine 2d similar to that of the seventh embodiment. Components having the same functions as those of the first and seventh embodiments are denoted by the same reference numerals as those of the first and seventh embodiments, and repeated descriptions are omitted. The following mainly describes the differences from the first and seventh embodiments.

[0109] The visual presentation device 1c is the same as the visual presentation device 1b of Embodiment 7 except that a gain adjustment unit 20 is added. In this embodiment, the measurement result of the distance measuring sensor 32 is sent to the visual presentation device 1c via the communication unit 22, and the presentation image control unit 12 receives the measurement result via the communication unit 15. The gain adjustment unit 20 determines the zoom factor based on the measurement result in the same manner as in Embodiment 7, and determines the gain of the motion operation of the remote machine 2d (the gain of the operation for moving the remote machine 2d) based on the factor, changes the control information generated by the control information generation unit 14 based on the determined gain, and sends the changed control information to the remote machine 2d via the communication unit 15. In addition, the gain adjustment unit 20 sends the determined zoom factor to the camera unit 21c of the remote machine 2d via the communication unit 15.

[0110] In addition, the gain adjustment unit 20 determines the zoom ratio here, but the zoom control unit 31 may determine the zoom ratio based on the measurement result of the distance sensor 32, and send the determined zoom ratio to the visual presentation device 1c via the communication unit 28. In this case, the gain adjustment unit 20 of the visual presentation device 1c determines the gain of the action of the remote machine 2d according to the zoom ratio received via the communication unit 15, changes the control information generated by the control information generation unit 14 according to the determined gain, and sends the changed control information to the remote machine 2d via the communication unit 15. In addition, the gain adjustment unit 20 may be provided in the remote machine 2d, and the remote machine 2d may determine the gain according to the zoom ratio received from the zoom control unit 31, change the control information, and output the changed control information to the drive control unit 24.

[0111] Fig.16 1 is a diagram showing an example of an image displayed by the visual presentation device 1c of this embodiment without performing image enlargement and reduction. Fig.16 In the example shown, the remote machine 2d is a construction machine, and a shovel (bucket) is gradually approaching soil as an operation object. The size of the display graphic 301 representing the remote machine 2d in the video does not change.

[0112] Fig.17 1 is a diagram showing an example of an image displayed by the visual presentation device 1c of the present embodiment in the case of enlarging and reducing the image. Fig.17 In the example shown, Fig.16 The same situation as the example shown, but as the shovel approaches the soil as the operation object, the image is enlarged and the size of the display graphic 301 also changes. Fig.16 and Fig.17 As you know, in Fig.17 In the example shown, the display makes it easy to understand the situation around the shovel. However, when the operator operates based on the image, the amplitude of the action is determined according to the distance in the displayed image. Fig.17 When the image is zoomed in as shown, the ratio of the distance in the displayed image to the distance in the actual area being photographed changes, so there is a possibility that the operator's intended action and the control amount operated by the operator may be inconsistent. For example, when the operator operates based on the zoomed screen, the operator's operation amount becomes larger than when the operator operates based on the non-magnified screen. Therefore, the gain adjustment unit 20 reduces the control amount of the action of the remote machine 2d corresponding to the operator's operation amount according to the zoom ratio, so that the operator's intended action is easily realized, and the operability is improved.

[0113] In addition, Fig.15, an example using the gaze point setting device 16 is shown, but the sight line measuring device 18 described in Embodiment 2 may be used instead of the gaze point setting device 16. In addition, when a monocular camera is used as in Embodiment 1, the operation described in this embodiment may also be applied. In addition, the operation described in this embodiment may also be applied to the remote mechanical operating systems 100b and 100c described in Embodiments 3 and 4.

[0114] Implementation method 9.

[0115] Fig.18 1 is a diagram showing a configuration example of a remote machine operating system according to Embodiment 9. The remote machine operating system 100h of this embodiment includes a visual presentation device 1c and a remote machine 2e similar to Embodiment 8. Components having the same functions as Embodiments 1 and 8 are denoted by the same reference numerals as Embodiments 1 and 8, and repeated descriptions are omitted. The following mainly describes the differences from Embodiments 1 and 8.

[0116] The remote machine 2e is the same as the remote machine 2d of the eighth embodiment except that the camera unit 21c is replaced with a camera unit 21d. The camera unit 21d is the same as the camera unit 21c of the eighth embodiment except that an automatic tracking unit 33 is added.

[0117] The automatic tracking unit 33 controls the imaging direction of the compound eye imaging unit 29 so that the determined object is included in the central field of view image. The automatic tracking unit 33 controls the imaging direction of the compound eye imaging unit 29, for example, so that the determined object becomes the center of the field of view of the compound eye imaging unit 29. The determined object is, for example, the end effector of the remote machine 2e. The automatic tracking unit 33 detects the determined object from the image captured by the compound eye imaging unit 29, and controls the imaging direction of the compound eye imaging unit 29 so that, for example, a determined part such as the center of the determined object becomes the center of the field of view of the compound eye imaging unit 29. Thus, the end effector of the remote machine 2e is automatically tracked, so that the operation load of the operator can be reduced. In addition, the determined object is not limited to the end effector, as long as it is an object that the operator wants to focus on. In addition, the operator can set the automatic tracking on / off, that is, whether to make the automatic tracking effective or ineffective. In addition, the determined object can also be set by the operator. For example, the determined object may be specified by the operator specifying a region to be the determined object while the image is being displayed.

[0118] In addition, Fig.18, an example using the gaze point setting device 16 is shown, but the sight line measuring device 18 described in Embodiment 2 may be used instead of the gaze point setting device 16. In addition, when a monocular camera is used as in Embodiment 1, the operation described in this embodiment may also be applied. In addition, the operation described in this embodiment may also be applied to the remote mechanical operating systems 100b and 100c described in Embodiments 3 and 4.

[0119] The configuration described in the above embodiment is merely an example, and may be combined with other known technologies, the embodiments may be combined with each other, and part of the configuration may be omitted or changed without departing from the gist.

[0120] Explanation of symbols

[0121] 1, 1a, 1b, 1c: visual presentation device; 2, 2a, 2b, 2c, 2d, 2e: remote machine; 10: presentation unit; 11: communication status monitoring unit; 12: presentation image control unit; 13: remote machine status monitoring unit; 14: control information generation unit; 15, 22, 28: communication unit; 16: gaze point setting device; 17: operation input device; 18: sight line measurement device; 19: device surrounding status monitoring device; 20: gain adjustment unit; 21, 21a, 21b, 21c, 21d: camera Machine unit; 23: driving unit; 24: driving control unit; 25: status acquisition device; 26: camera unit; 27, 27a: image processing unit; 29: compound eye camera unit; 30: camera driving unit; 31: zoom control unit; 32: ranging sensor; 33: automatic tracking unit; 100, 100a, 100b, 100c, 100d, 100e, 100f, 100g, 100h: remote mechanical operating system; 201: image display area; 202: central field of view area; 203: peripheral field of view area.

Claims

1. A remote mechanical operating system, characterized in that: have: A camera for photographing the remote machine being remotely operated; and a visual presentation device that receives the image captured by the camera and displays the received image, thereby presenting the image to an operator operating the remote machine, The visual presentation device comprises: a communication status monitoring unit configured to detect a communication status during reception of the image; and The image presentation control unit determines a central field of view area and a peripheral field of view area in a display area where the image is displayed, and determines image parameters related to the transmission rates of the central field of view image displayed in the central field of view area and the peripheral field of view image displayed in the peripheral field of view area based on the communication status detected by the communication status monitoring unit.

2. The remote mechanical operating system according to claim 1, characterized in that: The remote mechanical operating system includes a gaze point setting device, which sets the gaze point of the operator. The presented image control unit determines the central visual field area according to the gaze point set by the gaze point setting device.

3. The remote mechanical operating system according to claim 2, characterized in that: The gaze point setting device is a line of sight measuring device that measures the line of sight of the operator.

4. The remote mechanical operating system according to any one of claims 1 to 3, characterized in that: The remote machine operating system includes a remote machine status monitoring unit that detects a status of an operation of the remote machine. The image parameter is determined using the detection result of the remote machine condition monitoring unit.

5. The remote mechanical operating system according to claim 4, characterized in that: The remote machine condition monitoring unit detects a moving speed of the remote machine.

6. The remote mechanical operating system according to claim 4 or 5, characterized in that: The remote machine state monitoring unit detects a distance between the remote machine and an object to be operated by the remote machine.

7. The remote mechanical operating system according to any one of claims 1 to 6, characterized in that: The image parameters include at least one of a resolution of the peripheral visual field image, a frame rate of the peripheral visual field image, a color of the peripheral visual field area, and a size of the central visual field area.

8. The remote mechanical operating system according to any one of claims 1 to 7, characterized in that: The remote machine operating system includes an automatic tracking unit that controls the imaging direction of the camera so that the determined object is included in the central field of view image.

9. The remote mechanical operating system according to any one of claims 1 to 8, characterized in that: The remote machine operating system includes a zoom control unit that controls enlargement and reduction of the image captured by the camera.

10. The remote mechanical operating system according to claim 9, characterized in that: The remote machine operating system includes a distance measuring sensor that measures the distance between the remote machine and an object to be operated by the remote machine. The zoom control unit controls enlargement and reduction of the image according to a magnification determined based on a measurement result of the distance measuring sensor.

11. The remote mechanical operating system according to claim 9 or 10, characterized in that: The remote machine operating system includes a gain adjustment unit that adjusts a gain of a motion operation of the remote machine according to a magnification set by the zoom control unit.

12. The remote mechanical operating system according to any one of claims 1 to 8, characterized in that: The camera is a compound eye camera, The image parameters include information indicating whether to set the central field of view image and the peripheral field of view image as stereoscopic images.

13. The remote mechanical operating system according to any one of claims 9 to 11, characterized in that: The camera is a compound eye camera, The image parameters include information indicating whether to set the central field of view image and the peripheral field of view image as stereoscopic images.

14. The remote mechanical operating system according to claim 13, characterized in that: The remote machine operating system includes a camera driving unit that controls at least one of a direction of imaging of the compound eye camera, a distance between cameras of the compound eye camera, and a convergence angle of the compound eye camera.

15. The remote mechanical operating system according to claim 13 or 14, characterized in that: The zoom control unit controls the camera in cooperation with the zooming in and out of the image.

16. The remote mechanical operating system according to any one of claims 12 to 15, characterized in that: The remote machine operating system includes a device peripheral condition monitoring device, the device peripheral condition monitoring device detecting whether there is an object in the field of view of the operator corresponding to the central field of view image, When the central field of view image is a stereoscopic view image, the presented image control unit reduces the resolution of the peripheral field of view image when the presence of the object is detected by the device periphery situation monitoring device.

17. A visual presentation device for presenting an image to an operator operating a remote machine by receiving an image from a camera that photographs a remote machine being remotely operated and displaying the received image, wherein the visual presentation device comprises: a communication status monitoring unit configured to detect a communication status during reception of the image; and The image presentation control unit determines a central field of view area and a peripheral field of view area in a display area where the image is displayed, and determines image parameters related to the transmission rates of the central field of view image displayed in the central field of view area and the peripheral field of view image displayed in the peripheral field of view area based on the communication status detected by the communication status monitoring unit.

18. A visual presentation method, in a visual presentation device, wherein the visual presentation device receives an image from a camera that photographs a remote machine being remotely operated, and displays the received image, thereby presenting the image to an operator operating the remote machine, wherein the visual presentation method comprises: A step of detecting a communication status in receiving the image; as well as The steps of determining a central field of view area and a peripheral field of view area in a display area for displaying the image, and determining image parameters related to the transmission rates of the central field of view image displayed in the central field of view area and the peripheral field of view image displayed in the peripheral field of view area according to the detected communication status.

19. A program for causing a visual presentation device to be executed, the visual presentation device receiving an image from a camera that photographs a remote machine being remotely operated and displaying the received image, thereby presenting the image to an operator operating the remote machine, the program being characterized in that: a step of detecting a communication status in receiving the image; and The steps of determining a central field of view area and a peripheral field of view area in a display area for displaying the image, and determining image parameters related to the transmission rates of the central field of view image displayed in the central field of view area and the peripheral field of view image displayed in the peripheral field of view area according to the detected communication status.

Images