Remote operation support system and remote operation support method
The remote control support system addresses the challenge of adjusting camera display for remote operators by providing controlled zoom and stabilized images, enhancing the operator's understanding of the working machine's environment for improved operational efficiency.
Patent Information
- Application Number
- JP2021045536
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-03-19
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2041-03-19
AI Technical Summary
Operators remotely controlling working machines face challenges in accurately grasping the working status due to difficulties in adjusting camera display magnification and orientation, leading to insufficient understanding of the surrounding situation, especially when trying to display driving and caution images simultaneously.
A remote control support system that includes a remote control device with an actual imaging device and imaging function control elements, which adjusts zoom magnification based on operation modes to provide a bird's-eye view in one state and allows arbitrary magnification in another, while stabilizing images to prevent delays and enhance understanding.
Enables operators to accurately grasp the environment of the working machine through controlled zoom functions and stabilized images, improving operational efficiency and reducing delays in understanding the surroundings.
Smart Images

Figure 0007581996000001 
Figure 0007581996000002 
Figure 0007581996000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a technique for assisting an operator in remotely controlling a work machine such as a hydraulic excavator. [Background technology]
[0002] 2. Description of the Related Art There has been proposed a technique in which an operator operates a work machine using a remote control device while referring to a remote image showing the working status of the work machine (see, for example, Patent Document 1). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2018-021395 A [Patent Document 2] JP 2017-021517 A Summary of the Invention [Problem to be solved by the invention]
[0004] Compared to actually being on board a work machine, it is difficult to check the work situation when remotely operating the machine. For this reason, there is a demand to display an enlarged detailed image of a specific location (ATT and / or the work target object). For this reason, the operator changes the display magnification and direction of the camera to aim at a specific location and display an enlarged detailed image.
[0005] Meanwhile, a remote operator is required to grasp the surrounding situation as accurately as possible while performing work. Therefore, after checking the work situation, the operator cancels the setting that increased the display magnification of the camera and returns it to the minimum (initial) display magnification, and then performs an operation to display an image with as wide a field of view as possible on the display device.
[0006] However, if the operator forgets to cancel the setting that increased the camera's display magnification, or if the camera's display magnification does not fully return to the minimum (initial) display magnification, it will be difficult to grasp the surrounding situation, and the surrounding situation necessary for performing the work will not be fully grasped.
[0007] When attempting to simultaneously display a driving image and a caution image as in conventional technology 2, it is difficult for the operator to grasp where the caution image is located in the driving image, and the caution image cannot be displayed filling the entire screen, which makes it difficult for the operator to check the surrounding situation.
[0008] Therefore, an object of the present invention is to provide a system etc. that can enable an operator to grasp the environment of a work machine in an appropriate form taking into account the state of the work machine. [Means for solving the problem]
[0009] The present invention Remote operation support The system is A remote operation support system including a remote operation device having a remote operation mechanism that is operated to operate a working mechanism of a working machine, and an actual machine imaging device that is mounted on the working machine and obtains an image including at least a part of the working mechanism and outputs the image to a remote image output device provided in the remote operation device. 1. A system comprising: a remote control device provided in the remote operation device and recognizing an operation mode for adjusting a zoom magnification of the actual imaging device; The remote control mechanism is operated to perform a task in which the working mechanism is operated. First state In , When the zoom magnification is changed according to the operation mode and is equal to or smaller than a designated magnification, the zoom magnification of the real imaging device is changed according to the operation mode, and when the zoom magnification is changed according to the operation mode and exceeds the designated magnification, the zoom magnification of the real imaging device is maintained at the designated magnification. A first imaging function control element for The operation in which the remote control mechanism is operated to operate the working mechanism is not being performed. Second state In , the actual imaging device The zoom magnification is changed according to the operation mode. A second imaging function control element; It is equipped with:
[0010] The configuration Remote operation supportAccording to the system, in the first state, the operator can obtain an overview of the environment of the work machine through the captured image output to the output interface, and can then operate the work machine through the input interface. Meanwhile, in the second state, the zoom function of the actual machine imaging device is exercised in response to the operator's zoom operation through the input interface, so that a captured image of any display magnification is output through the output interface, allowing the operator to obtain an overview of the environment of the work machine in a desired form.
[0011] The above-mentioned configuration Remote operation support In the system, The first imaging function control element stops a zoom function as an imaging function of the actual imaging device in the first state. It is preferred.
[0012] The configuration Remote operation support According to the system, in the first state, the display magnification of the captured image output to the output interface is fixed to a specified display magnification or less, allowing the operator to grasp the environment of the work machine from a bird's-eye view and then operate the work machine through the input interface.
[0013] The above-mentioned configuration Remote operation support In the system, The second imaging function control element changes a change speed of the display magnification of the captured image in a zoom function as an imaging function of the actual machine imaging device in accordance with a work progress status of the work machine in the second state. It is preferred.
[0014] The configuration Remote operation support According to the system, in the second state, the zoom-in and / or zoom-out speed is adjusted depending on the difference in the work progress of the work machine, thereby improving the convenience for the operator to grasp the environment of the work machine.
[0015] The above-mentioned configuration Remote operation support In the system, the first imaging function control element stops an image stabilization function as an imaging function of the actual imaging device in the first state; The second imaging function control element, in the second state, causes a camera shake correction function as an imaging function of the actual imaging device to be exercised. It is preferred.
[0016] The configuration Remote operation support According to the system, in the first state, the image stabilization function of the real image capturing device is stopped, thereby preventing an increase in the image processing cost of the captured image. This avoids a delay in transmission from the real image capturing device to the output interface, and thus an output delay of the captured image at the output interface. As a result, a situation in which the operator's understanding of the environment of the work machine and thus the corresponding operation of the work machine through the input interface are hindered is avoided. On the other hand, in the second state, the image stabilization function of the real image capturing device is exercised, thereby allowing the operator to easily and accurately understand the environment of the work machine through the image stabilization-corrected captured image output at the output interface. [Brief description of the drawings]
[0017] [Figure 1] FIG. 1 is an explanatory diagram regarding the configuration of a remote operation support system. [Diagram 2] FIG. 2 is an explanatory diagram relating to the configuration of a remote control device. [Diagram 3] FIG. 2 is an explanatory diagram relating to the configuration of a work machine. [Figure 4] FIG. 2 is an explanatory diagram regarding functions of a remote operation support system (imaging function control system). [Diagram 5] FIG. [Figure 6] 5A and 5B are explanatory diagrams relating to a control mode of a zoom function of the actual imaging device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0018] (Configuration of remote operation support system) The remote operation support system shown in Fig. 1 is composed of a remote operation support server 10 and a remote operation device 20 for remotely operating a work machine 40. The remote operation support server 10, the remote operation device 20, and the work machine 40 are configured to be able to communicate with each other via a network. The mutual communication network of the remote operation support server 10 and the remote operation device 20 and the mutual communication network of the remote operation support server 10 and the work machine 40 may be the same as or different from each other.
[0019] (Configuration of remote operation support server) The remote operation support server 10 includes a database 102, a first support processing element 121, and a second support processing element 122. The database 102 stores captured image data and the like. The database 102 may be configured as a database server separate from the remote operation support server 10. Each support processing element is configured as a calculation processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes a calculation process (described later) on the data according to the software. In this embodiment, the remote operation support server 10 constitutes an "imaging function control system", and the second support processing element 122 constitutes a "first imaging function control element" and a "second imaging function control element".
[0020] (Configuration of remote control device) The remote operation device 20 includes a remote control device 200, a remote input interface 210, and a remote output interface 220. The remote control device 200 is configured with an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing on the data according to the software.
[0021] The remote input interface 210 includes a remote control mechanism 211. The remote output interface 220 includes a remote image output device 221, an audio output device 222, and a remote wireless communication device 224.
[0022] The remote operation mechanism 211 includes a traveling operation device, a slewing operation device, a boom operation device, an arm operation device, and a bucket operation device. Each operation device has an operation lever that receives a rotation operation. The operation lever (travel lever) of the traveling operation device is operated to move the lower traveling body 410 of the work machine 40. The travel lever may also serve as a travel pedal. For example, a travel pedal fixed to the base or lower end of the travel lever may be provided. The operation lever (slewing lever) of the slewing operation device is operated to move a hydraulic slewing motor that constitutes the slewing mechanism 430 of the work machine 40. The operation lever (boom lever) of the boom operation device is operated to move the boom cylinder 442 of the work machine 40. The operation lever (arm lever) of the arm operation device is operated to move the arm cylinder 444 of the work machine 40. The operation lever (bucket lever) of the bucket operation device is operated to move the bucket cylinder 446 of the work machine 40.
[0023] The operating levers constituting the remote control mechanism 211 are arranged around a seat St on which the operator sits, as shown in Fig. 2. The seat St is in the form of a high-back chair with armrests, but may be in the form of a low-back chair without a headrest, or in the form of a chair without a backrest, or in any other form on which the operator can sit.
[0024] A pair of left and right travel levers 2110 corresponding to the left and right crawlers are arranged side by side in front of the seat St. One operation lever may serve as multiple operation levers. For example, the left operation lever 2111 provided in front of the left frame of the seat St shown in FIG. 2 may function as an arm lever when operated in the front-rear direction, and may function as a rotation lever when operated in the left-right direction. Similarly, the right operation lever 2112 provided in front of the right frame of the seat St shown in FIG. 2 may function as a boom lever when operated in the front-rear direction, and may function as a bucket lever when operated in the left-right direction. The lever pattern may be arbitrarily changed by an operation instruction from the operator.
[0025] The left-side operating lever 2111 and the right-side operating lever 2112 are provided at the positions gripped by the operator with a zoom-in switch SW+ and a zoom-out switch SW-, respectively, for adjusting the zoom-in angle and the zoom-out angle of the actual imaging device 412, as described below.
[0026] 2, the remote image output device 221 is composed of a central remote image output device 2210, a left remote image output device 2211, and a right remote image output device 2212, each having a substantially rectangular screen, and disposed respectively in front of, diagonally forward left of, and diagonally forward right of, the seat St. The shapes and sizes of the screens (image display areas) of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be the same or different.
[0027] As shown in Fig. 2, the right edge of the left remote image output device 2211 is adjacent to the left edge of the central remote image output device 2210, such that the screens of the central remote image output device 2210 and the left remote image output device 2211 form an inclination angle θ1 (e.g., 120°≦θ1≦150°). As shown in Fig. 2, the left edge of the right remote image output device 2212 is adjacent to the right edge of the central remote image output device 2210, such that the screens of the central remote image output device 2210 and the right remote image output device 2212 form an inclination angle θ2 (e.g., 120°≦θ2≦150°). The inclination angles θ1 and θ2 may be the same or different.
[0028] The screens of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be parallel to the vertical direction or may be inclined relative to the vertical direction. At least one of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be composed of a plurality of divided image output devices. For example, the central remote image output device 2210 may be composed of a pair of image output devices adjacent to each other above and below, each having a substantially rectangular screen.
[0029] The sound output device 222 is composed of one or more speakers, and is composed of a central sound output device 2220, a left sound output device 2221, and a right sound output device 2222, which are arranged behind the seat St, at the rear of the left armrest, and at the rear of the right armrest, respectively, as shown in Fig. 2. The specifications of the central sound output device 2220, the left sound output device 2221, and the right sound output device 2222 may be the same or different.
[0030] (Work machine configuration) The work machine 40 comprises an actual machine control device 400, an actual machine input interface 41, an actual machine output interface 42, and a work mechanism 440. The actual machine control device 400 comprises an image processing device 30. The image processing device 30 comprises a state detection element 31, an image prediction element 32, and an image compression element 34. Each of the components of the actual machine control device 400 and the image processing device 30 is configured by an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing on the data in accordance with the software.
[0031] The work machine 40 is, for example, a crawler excavator (construction machine), and as shown in Fig. 3, includes a crawler-type lower traveling body 410, and an upper rotating body 420 that is rotatably mounted on the lower traveling body 410 via a rotating mechanism 430. A cab 424 (operator's cabin) is provided on the front left side of the upper rotating body 420. A work mechanism 440 is provided in the front center of the upper rotating body 420.
[0032] The real machine input interface 41 includes a real machine operation mechanism 411, a real machine imaging device 412, and a positioning device 414. The real machine operation mechanism 411 includes a plurality of operation levers arranged around a seat arranged inside the cab 424 in the same manner as the remote control mechanism 211. A drive mechanism or robot that receives a signal according to the operation mode of the remote control lever and moves the real machine operation lever based on the received signal is provided in the cab 424. The real machine imaging device 412 is installed, for example, inside the cab 424, and images the environment including at least a part of the working mechanism 440 through the front window and a pair of left and right side windows. A part or all of the front window and the side windows may be omitted. The positioning device 414 is configured with a GPS and, if necessary, a gyro sensor or the like.
[0033] The real device output interface 42 includes a real device wireless communication device 422 .
[0034] 3, a working mechanism 440 as a working mechanism includes a boom 441 mounted on the upper rotating body 420 so as to be able to rise and fall, an arm 443 rotatably connected to the tip of the boom 441, and a bucket 445 rotatably connected to the tip of the arm 443. A boom cylinder 442, an arm cylinder 444, and a bucket cylinder 446 each composed of an extendable hydraulic cylinder are mounted on the working mechanism 440.
[0035] The boom cylinder 442 is interposed between the boom 441 and the upper rotating body 420 so as to extend and retract when supplied with hydraulic oil to rotate the boom 441 in the hoisting direction. The arm cylinder 444 is interposed between the arm 443 and the boom 441 so as to extend and retract when supplied with hydraulic oil to rotate the arm 443 about a horizontal axis relative to the boom 441. The bucket cylinder 446 is interposed between the bucket 445 and the arm 443 so as to extend and retract when supplied with hydraulic oil to rotate the bucket 445 about a horizontal axis relative to the arm 443.
[0036] (function) The functions of the remote operation support system and the imaging function control system having the above configuration will be described with reference to the flowchart shown in Fig. 4. In the flowchart, a block "C●" is used for the sake of simplicity, and means transmission and / or reception of data, and means a conditional branch in which processing in a branching direction is executed on the condition that the data is transmitted and / or received.
[0037] In the remote operation device 20, it is determined whether or not a designation operation has been performed by the operator through the remote input interface 210 (FIG. 4 / STEP 210). The "designation operation" is, for example, an operation such as tapping on the remote input interface 210 for the operator to designate the work machine 40 that he or she intends to remotely operate. If the determination result is negative (FIG. 4 / STEP 210...NO), the series of processes ends. On the other hand, if the determination result is positive (FIG. 4 / STEP 210...YES), an environment confirmation request is transmitted to the remote operation support server 10 through the remote wireless communication device 224 (FIG. 4 / STEP 212).
[0038] When the environment confirmation request is received in the remote operation support server 10, the first support processing element 121 transmits the environment confirmation request to the corresponding work machine 40 (FIG. 4 / C10).
[0039] When an environment confirmation request is received in the work machine 40 via the actual machine wireless communication device 422 (Figure 4 / C40), the actual machine control device 400 acquires an image via the actual machine imaging device 412, and the captured image data that has been subjected to the image processing is transmitted to the remote control device 10 via the actual machine wireless communication device 422 (Figure 4 / STEP 410).
[0040] In the remote operation support server 10, when the first support processing element 121 receives captured image data (FIG. 4 / C11), the second support processing element 122 transmits environmental image data corresponding to the captured image to the remote operation device 20 (FIG. 4 / STEP 110). The environmental image data is not only the captured image data itself, but also image data representing a simulated environmental image generated based on the captured image.
[0041] When the remote operation device 20 receives environmental image data through the remote wireless communication device 224 (FIG. 4 / C21), the remote control device 200 outputs an environmental image corresponding to the environmental image data to the remote image output device 221 (FIG. 4 / STEP 212).
[0042] As a result, for example, as shown in FIG. 5, an environmental image showing a boom 441, arm 443, and a pile of rubble or earth and sand (the object of work by bucket 445), which are part of the work mechanism 440, in front of the cab 424, is output to the remote image output device 221 through the window frame that defines the cab 424.
[0043] In the remote operation device 20, the remote control device 200 recognizes the zoom operation mode of the remote operation mechanism 211, and transmits a zoom command corresponding to the zoom operation mode to the remote operation support server 10 through the remote wireless communication device 224 (FIG. 4 / STEP 214). The zoom operation mode is specified by the operation postures of the zoom-in switch SW+ and the zoom-out switch SW- provided at the gripping positions of the left operation lever 2111 and the right operation lever 2112, respectively (see FIG. 2). By continuously pressing the zoom-in switch SW+, the zoom magnification γ of the real imaging device 412 increases gradually or in stages. By continuously pressing the zoom-out switch SW-, the zoom magnification γ of the real imaging device 412 decreases gradually or in stages.
[0044] When a zoom command is received in the remote operation support server 10 (FIG. 4 / C12), the second support processing element 122 determines the state of the work machine 40 based on the communication result with the work machine 40 (FIG. 4 / STEP120). The "first state" is a state in which the work machine 40 can be operated (or a state in which the work machine 40 is working). The "second state" is a state in which the work machine 40 cannot be operated (or a state in which the work machine 40 has stopped working). For example, the first state and the second state of the work machine 40 may be determined depending on whether or not an operation cutoff lever constituting the actual machine operation mechanism 411 of the work machine 40 is in a predetermined position. The first state and the second state of the work machine 40 may also be determined depending on whether or not a remote hydraulic cutoff lever simulating a hydraulic cutoff lever as a remote input interface is in a predetermined position.
[0045] When it is determined that the work machine 40 is in the first state (FIG. 4 / STEP 120...1), a first zoom command is transmitted to the work machine 40 by the second support processing element 122 (first imaging function control element) (FIG. 4 / STEP 121). On the other hand, when it is determined that the work machine 40 is in the second state (FIG. 4 / STEP 120...2), a second zoom command is transmitted to the work machine 40 by the second support processing element 122 (second imaging function control element) (FIG. 4 / STEP 122).
[0046] In the work machine 40, when a second zoom command is received through the actual machine wireless communication device 422 (FIG. 4 / C42), the actual machine control device 400 controls the zoom function of the actual machine imaging device 412 in a normal manner according to the operation manner of the zoom-in switch SW+ and / or the zoom-out switch SW-, and adjusts the zoom magnification γ (FIG. 4 / STEP412).
[0047] For example, when the zoom operation amount changes as shown by the solid line in FIG. 6, the zoom magnification is adjusted so as to be proportional to the zoom operation amount (proportionality coefficient=gain coefficient) as shown by the two-dot chain line in FIG. 6. The zoom operation amount increases when the zoom-in switch SW+ is operated, and decreases when the zoom-out switch SW- is operated. This changes the width of the real space area reflected in the captured image acquired through the real machine imaging device 412, and further in the environmental image output by the remote image output device 221 (see FIG. 4 / STEP 410→...→STEP 212, FIG. 5). For example, when the zoom magnification γ is increased by zooming in the real machine imaging device 412, the real space area reflected in the environmental image becomes smaller or narrower than before. Also, when the zoom magnification γ is reduced by zooming out the real machine imaging device 412, the real space area reflected in the environmental image becomes larger or wider than before.
[0048] On the other hand, when the work machine 40 receives a first zoom command through the actual machine wireless communication device 422 (Figure 4 / C41), the zoom magnification of the actual machine imaging device 412 is controlled by the actual machine control device 400 in a manner that is more restrictive than the normal manner, although it depends on the operation manner of the zoom-in switch SW+ and / or the zoom-out switch SW- (Figure 4 / STEP 411).
[0049] For example, when the zoom operation amount changes as shown by the solid line in FIG. 6, the zoom designated magnification γ 0 In the state where the zoom magnification is equal to or less than the specified zoom magnification γ 0 (time t 1 From time t 2 During the period up to the zoom magnification γ, the zoom magnification γ 0 In addition, as shown by the dashed line in FIG. 6, the zoom magnification γ of the actual imaging device 412 is maintained at the specified magnification γ 0 It may be controlled to the following value (for example, a constant value): Furthermore, regardless of the operation manner of the zoom-in switch SW+ and / or the zoom-out switch SW-, the zoom function may be stopped and the zoom magnification of the actual imaging device 412 may be fixed to the minimum magnification or a magnification close to the minimum magnification.
[0050] In the remote operation device 20, the remote control device 200 recognizes the operation mode of the remote operation mechanism 211, and a remote operation command corresponding to the operation mode is transmitted to the remote operation support server 10 via the remote wireless communication device 224 (FIG. 4 / STEP 220).
[0051] In the remote operation support server 10, when the second support processing element 122 receives the remote operation command, the first support processing element 121 transmits the remote operation command to the work machine 40 (FIG. 4 / C14).
[0052] In the work machine 40, when an operation command is received by the actual machine control device 400 via the actual machine wireless communication device 422 (FIG. 4 / C44), the operation of the work mechanism 440 and the like is controlled (FIG. 4 / STEP 420). For example, the work of scooping up soil in front of the work machine 40 with the bucket 445, rotating the upper rotating body 410, and dropping the soil from the bucket 445 is performed.
[0053] (Action and effect) According to the imaging function control system constituting the remote operation support system of this configuration, when the work machine 40 is in the first state, the display magnification (zoom magnification γ) of the captured image (environmental image) output to the remote output interface 220 is controlled only within a range equal to or less than the designated display magnification, or is fixed to a value equal to or less than the designated display magnification (see FIG. 4 / STEP 120...1→STEP 121→C41→STEP 411, and the dashed and dotted lines in FIG. 6). This allows the operator to get an overview of the environment of the work machine 40 through the captured image (environmental image) output to the remote output interface 220 (remote image output device 221), and then operate the work machine 40 through the remote input interface 210 (remote operation mechanism 211).
[0054] On the other hand, when the work machine 40 is in the second state, the zoom function of the actual machine imaging device 412 is exerted in response to zoom operation by the operator via the remote input interface 210 (zoom-in switch SW+ and zoom-out switch SW-) (FIG. 4 / STEP 120..2->STEP 122->C42->STEP 412, see the two-dot chain line in FIG. 6). As a result, a captured image (environmental image) at any display magnification is output via the remote output interface 220, allowing the operator to grasp the environment of the work machine 40 in the form he or she desires.
[0055] (Another embodiment of the present invention) In the above embodiment, the imaging function control system is constituted by the remote operation support server 10, and the first imaging function control element and the second imaging function control element are constituted by the second support processing element 122. However, in another embodiment, the imaging function control system may be constituted by the work machine 40 and / or the remote operation device 20, and the first imaging function control element and / or the second imaging function control element may be constituted by the actual machine control device 400 and / or the remote control device 200.
[0056] When the work machine 40 is in the second state, the rate of change of the zoom magnification γ of the imaging function 412 of the actual imaging device may be changed depending on the progress of the work of the work machine 40. For example, when the work machine 40 is using the work mechanism 440 to excavate a pile of rubble or earth and sand as shown in FIG. 5 (first state), if the operator wants to check the state of the excavation surface during excavation, the work may be temporarily interrupted (second state) and a zoom operation may be performed via the remote input interface 210 (zoom-in switch SW+ and zoom-out switch SW-), and the rate of change of the zoom magnification γ accompanying this zoom operation may be increased above the normal rate. If the excavation surface becomes deeper as the work progresses, the rate of change of the zoom magnification γ accompanying the zoom operation may be further increased.
[0057] According to the imaging function control system having this configuration, in the second state, the zoom-in and / or zoom-out speed is adjusted according to the difference in the work progress of the work machine, improving the convenience for the operator when understanding the environment of the work machine. In particular, when the work is interrupted during work to check the work target, the interruption time of the work can be shortened, improving the work efficiency.
[0058] The second support processing element 122 (first imaging function control element) sends a "first image stabilization command" along with the first zoom command to the work machine 40 (see Figure 4 / STEP 121), stopping the image stabilization function as an imaging function of the actual imaging device 412, and the second support processing element 122 (second imaging function control element) sends a "second image stabilization command" along with the second zoom command to the work machine 40 (see Figure 4 / STEP 122), thereby enabling the image stabilization function as an imaging function of the actual imaging device 412.
[0059] According to the imaging function control system having this configuration, in the first state, the image stabilization function of the actual imaging device 412 is stopped, thereby preventing an increase in image processing costs for the captured image. This avoids a transmission delay from the actual imaging device 412 to the remote output interface 220 (remote image output device 221), and thus an output delay of the captured image (environmental image) in the remote output interface 220 (FIG. 4 / STEP 410→C11→STEP 110→C21→STEP 212, see FIG. 5). As a result, a situation in which the operator's understanding of the environment of the work machine 40, and thus the corresponding operation of the work machine 40 via the remote input interface 210 (remote operation mechanism 211) are hindered is avoided.
[0060] On the other hand, in the second state, the image stabilization function of the actual machine imaging device 412 is utilized, allowing the operator to easily and accurately grasp the environment of the work machine 40 through the image stabilization-compensated captured image (environmental image) output from the remote output interface 220 (see FIG. 5).
[0061] In the above embodiment, when the work machine 40 is in the first state, the display magnification of the captured image (environmental image) output to the remote output interface 220 is controlled only within a range equal to or less than a specified display magnification, but such control to limit the display magnification may be performed only immediately after the work machine 40 enters the first state. As a result, even in the first state, a captured image (environmental image) with any display magnification is output through the remote output interface 220, allowing the operator to grasp the environment of the work machine 40 in a desired form, and immediately after the work machine 40 enters the first state, there is no problem in the operator's understanding of the environment of the work machine 40, and therefore in operating the work machine 40 through the remote input interface 210 (remote operation mechanism 211) accordingly. [Explanation of symbols]
[0062] 10: Remote operation support server, 20: Remote operation device, 40: Work machine, 102: Database, 121: First support processing element, 122: Second support processing element (First imaging function control element, Second imaging function control element), 200: Remote control device, 210: Remote input interface, 211: Remote operation mechanism, 220: Remote output interface, 221: Remote image output device, 222: Remote sound output device, 400: Actual machine control device, 410: Actual machine input interface, 420: Actual machine output interface, 424: Cab (operator's compartment), 440: Work mechanism, 445: Bucket (working part), SW+: Zoom in switch, SW-: Zoom out switch.
Claims
1. A remote operation support system comprising: a remote operation device having a remote operation mechanism operated to operate a working mechanism of a working machine; and a real machine imaging device mounted on the working machine and acquiring an image including at least a part of the working mechanism, the image being output to a remote image output device provided in the remote operation device, a remote control device provided in the remote operation device and recognizing an operation mode for adjusting a zoom magnification of the actual imaging device; a first imaging function control element that, in a first state in which the remote control mechanism is operated to operate the working mechanism and a task is being performed, changes the zoom magnification of the real imaging device in accordance with the operation mode when the zoom magnification is changed in accordance with the operation mode and is equal to or smaller than a designated magnification, and maintains the zoom magnification of the real imaging device at the designated magnification when the zoom magnification is changed in accordance with the operation mode and is greater than the designated magnification; a second imaging function control element that changes the zoom magnification of the actual imaging device in accordance with the operation mode in a second state in which the operation mechanism is operated by operating the remote operation mechanism and the operation mechanism is not being performed; A remote operation support system comprising:
2. 2. The remote operation support system according to claim 1, The first imaging function control element stops a zoom function as an imaging function of the actual imaging device in the first state. Remote operation support system.
3. 3. The remote operation support system according to claim 1, The second imaging function control element changes a change speed of the display magnification of the captured image in a zoom function as an imaging function of the actual machine imaging device in accordance with a work progress status of the work machine in the second state. Remote operation support system.
4. In the remote operation support system according to any one of claims 1 to 3, the first imaging function control element stops a camera shake correction function as an imaging function of the actual imaging device in the first state; The second imaging function control element, in the second state, causes a camera shake correction function as an imaging function of the actual imaging device to be exercised. Remote operation support system.
5. A remote operation assistance method for assisting remote operation of a work machine using a remote operation device comprising: a remote operation mechanism operated to operate a work mechanism of the work machine; and a remote image output device for outputting an image including at least a part of the work mechanism acquired by an actual machine imaging device mounted on the work machine, the method comprising: an operation mode recognition process for recognizing an operation mode for adjusting a zoom magnification of the actual imaging device by a remote control device provided in the remote operation device; a first imaging control process for changing the zoom magnification of the real imaging device in accordance with the operation mode when the zoom magnification is equal to or smaller than a designated magnification in a first state in which the remote control mechanism is operated to operate the working mechanism, and for maintaining the zoom magnification of the real imaging device at the designated magnification in a state in which the zoom magnification is changed in accordance with the operation mode and exceeds the designated magnification in a first state in which the remote control mechanism is operated to operate the working mechanism; a second imaging control process for changing the zoom magnification of the actual imaging device in accordance with a state of operation in a second state in which the operation mechanism is operated by operating the remote control mechanism and the operation is not being performed; A remote operation support method comprising the steps of:
Citation Information
Patent Citations
Remote control system and main control device
EP3754120A1
Camera visual field angle controller for remote-controlled machine
JP1997247515A
Monitoring camera
JP2011059403A
Remote control system and method
JP2017021517A
Remote operation system of construction machine
JP2018021395A
