Remote operation device, remote operation assistance server, remote operation assistance system and remote operation assistance method
By controlling the display mode of the captured image in the remote operation device, a part of the cab column is displayed near the frame of the image output device, the problem of operator confusing the column and the frame is solved, and the accuracy of space recognition is improved.
Patent Information
- Application Number
- CN202180014247.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-02-18
- Filing Date
- 2021-01-07
- Publication Date
- 2025-02-25
- Estimated Expiration
- 2041-01-07
AI Technical Summary
When the real camera shooting device is arranged inside the cab, the left and right pair of columns located on the front side of the cab will be reflected in the image output device of the remote operation device, causing the operator to confuse the columns with the frame of the image output device.
The remote control device controls the way in which the captured image in front of the cab obtained by the real-machine camera device is displayed on the designated image output device, so that a part of the column overlapping with the frame of the image output device is displayed in the designated image area in the strip shape .
The possibility of misidentification or confusion between the operator's cab post that the captured image is reflected and the frame of the image output device is reduced, thereby improving the accuracy of the operator's identification space.
Smart Images

Figure CN115088251B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a remote operation assisting server for assisting an operator in performing work using a work machine by communicating with a client assigned to the operator. Background Art
[0002] The following technical solution has been proposed (for example, refer to Patent Document 1), in which a captured image showing the situation at a work site captured by a real-machine shooting device mounted on a work machine is displayed on an image output device arranged in an adjacent manner at the side edge of a remote operating device.
[0003] Prior art literature
[0004] Patent Literature
[0005] Patent Document 1: Japanese Patent Application Publication No. 2018-207244 Summary of the invention
[0006] Problems to be solved by the invention
[0007] However, when the actual machine shooting device is configured inside the cab, a pair of left and right pillars located on the front side of the cab will be reflected in the shooting image displayed by the image output device that constitutes the remote operation device, and the left and right borders (Bezel) of the image output device and the pair of pillars may be confused and mistaken by the operator.
[0008] Therefore, an object of the present invention is to provide a remote control device or the like that can reduce the possibility of misidentification between a pair of pillars of a cab reflected in a captured image and a frame of an image output device displaying the pillars.
[0009] Means for solving problems
[0010] The remote operation device of the present invention comprises a remote operation mechanism for an operator to operate a working machine remotely, one or more image output devices and a remote control device arranged around the remote operation mechanism, and the remote operation device is characterized in that:
[0011] The remote control device performs the following control: a captured image obtained by a real-machine shooting device installed inside the cab of the working machine and showing the front of the cab is displayed on a designated image output device among the one or more image output devices, and a display method of the captured image in the designated image output device is variably controlled so that a part of at least one of a pair of pillars constituting the cab that overlaps with a designated image area, that is, the entirety of a designated part, is displayed on the designated image output device, wherein the designated image area extends in a strip-like manner between left and right borders.
[0012] According to the remote operation device having such a structure, a photographic image showing the appearance of the front of the cab obtained by a real machine photographing device mounted inside the cab of the working machine is displayed on a designated image output device. Therefore, the operator can operate the remote operation mechanism while grasping the situation in front of the cab of the working machine or real machine that is the object of remote operation through the photographic image displayed on the designated image output device. In addition to being a space where the operator rides in order to directly operate the working machine instead of remotely operating it, the "cab" also refers to a space that the operator does not actually ride in but is equivalent to the space.
[0013] Furthermore, the display mode of the captured image on the designated image output device is variably controlled so that a portion of at least one pillar constituting the cab that overlaps with the designated image region extending in a strip shape between the left and right frames, that is, the entire designated portion, is displayed on the designated image output device. Thus, at least the left and right frames of the designated image output device and a portion of the at least one pillar can be clearly identified in the designated image region extending transversely between the left and right frames. Therefore, the possibility that the operator will misidentify or confuse the pair of pillars of the cab reflected in the captured image with the frame of the image output device in the captured image showing the pillars can be reduced.
[0014] Brief description of the attached figure
[0015] Figure 1 It is an explanatory diagram related to the configuration of a remote operation support system as one embodiment of the present invention.
[0016] Figure 2 This is an explanatory diagram regarding the structure of the remote control device.
[0017] Figure 3 This is an explanatory diagram relating to the structure of a working machine.
[0018] Figure 4 This is a diagram for explaining the functions of the remote operation assistance system.
[0019] Figure 5This is an explanatory diagram regarding the first display method of a captured image.
[0020] Figure 6 This is an explanatory diagram regarding the second display method of the captured image.
[0021] Figure 7 This is an explanatory diagram regarding the third display method of the captured image.
[0022] Figure 8 It is an explanatory diagram regarding the fourth display method of the captured image.
[0023] Fig. 9 This is an explanatory diagram regarding the fifth display method of captured images. DETAILED DESCRIPTION
[0024] (Composition of remote operation assistance system)
[0025] Figure 1 The remote operation support system shown as one embodiment of the present invention is composed of a remote operation support server 10 and a remote operation device 20 for remotely operating a working machine 40. The remote operation support server 10, the remote operation device 20, and the working machine 40 are configured to be able to communicate with each other through a network. The mutual communication network between the remote operation support server 10 and the remote operation device 20 and the mutual communication network between the remote operation support server 10 and the working machine 40 may be the same network or different networks.
[0026] (Configuration of remote operation support server)
[0027] The remote operation support server 10 includes a database 102, a first auxiliary processing element 121, and a second auxiliary processing element 122. The database 102 stores and saves captured image data, etc. The database 102 may also be composed of a database server different from the remote operation support server 10. Each auxiliary processing element is composed of an operation processing device (a single-core processor or a multi-core processor or a processor core constituting the above processor), reads necessary data and software from a storage device such as a memory, and performs the operation processing described later according to the software on the data.
[0028] (Composition of remote control device)
[0029] The remote operation device 20 includes a remote control device 200, a remote input interface 210, and a remote output interface 220. The remote control device 200 is composed of a computing device (a single-core processor or a multi-core processor or a processor core constituting the above processor), reads necessary data and software from a storage device such as a memory, and performs computing processing in accordance with the software with the data as the object. The remote input interface 210 includes a remote operation mechanism 211. The remote output interface 220 includes an image output device 221 and a remote wireless communication device 222.
[0030] The remote operating mechanism 211 includes an operating device for traveling, an operating device for rotating, an operating device for an arm, an operating device for an arm, and an operating device for a bucket. Each operating device has an operating lever that receives a rotation operation. The operating lever (travel lever) of the operating device for traveling is operated to move the lower traveling body 410 of the working machine 40. The travel lever may also serve as a travel pedal. For example, a travel pedal fixed to the base or the lower end of the travel lever may also be provided. The operating lever (rotation lever) of the operating device for rotating is operated to actuate the hydraulic rotary motor of the rotating mechanism 430 constituting the working machine 40. The operating lever (boom lever) of the operating device for the arm is operated to actuate the boom cylinder 442 of the working machine 40. The operating lever (arm lever) of the operating device for the arm is operated to actuate the arm cylinder 444 of the working machine 40. The operating lever (bucket lever) of the operating device for the bucket is operated to actuate the bucket cylinder 446 of the working machine 40.
[0031] like Figure 2 As shown, the operating levers constituting the remote operating mechanism 211 are arranged around a seat St for the operator to sit on, for example. The seat St may be a high-back chair with armrests, a low-back chair without a headrest, or a chair without a backrest, etc., whichever seat is suitable for the operator to sit on.
[0032] A pair of left and right travel levers 2110 corresponding to the left and right crawlers are arranged side by side in front of the seat St. One operating lever can also serve as multiple operating levers. Figure 2 The left operating lever 2111 provided in front of the left frame of the seat St shown in the figure functions as a boom lever when operated in the front-rear direction, and functions as a swing lever when operated in the left-right direction. Figure 2 The right operating lever 2112 provided in front of the right frame of the seat St shown functions as a boom lever when operated in the front-rear direction, and functions as a bucket lever when operated in the left-right direction. The lever mode can be arbitrarily changed according to the operator's operation instruction.
[0033] For example Figure 2As shown, the image output device 221 is composed of a central image output device 2210 having a substantially rectangular screen arranged in front of the seat St, a left image output device 2211 having a substantially rectangular screen arranged in the left oblique front of the seat St, and a right image output device 2212 having a substantially rectangular screen arranged in the right oblique front. The shapes and sizes of the screens (image display areas) of the central image output device 2210, the left image output device 2211, and the right image output device 2212 may be the same or different.
[0034] like Figure 2 As shown, the screen of the central image output device 2210 is inclined at an angle θ1 (for example, 120°≤θ1≤150°) with the screen of the left image output device 2211, and the right edge of the left image output device 2211 is adjacent to the left edge of the central image output device 2210. Figure 2 As shown, the left edge of the right image output device 2212 is adjacent to the right edge of the central image output device 2210 in such a manner that the screen of the central image output device 2210 and the screen of the right image output device 2212 are inclined at an angle θ2 (for example, 120°≤θ2≤150°). The inclination angle θ1 and the inclination angle θ2 may be the same or different.
[0035] The screens of the central image output device 2210, the left image output device 2211, and the right image output device 2212 may be parallel to the vertical direction or inclined to the vertical direction. At least one of the central image output device 2210, the left image output device 2211, and the right image output device 2212 may be composed of a plurality of image output devices. For example, the central image output device 2210 may be composed of a pair of image output devices adjacent to each other in the upper and lower directions and having a substantially rectangular screen. The image output devices 2210 to 2212 may further include a speaker (sound output device).
[0036] (Structure of operating machinery)
[0037] The working machine 40 includes a real machine control device 400, a real machine input interface 41, a real machine output interface 42, and a working mechanism 440. The real machine control device 400 is composed of an operation processing device (a single-core processor or a multi-core processor or a processor core constituting the processor), reads necessary data and software from a storage device such as a memory, and performs operation processing in accordance with the software on the data.
[0038] The working machine 40 is, for example, a crawler excavator (construction machine). Figure 2As shown, the vehicle comprises a crawler-type lower traveling body 410 and an upper revolving body 420 rotatably mounted on the lower traveling body 410 via a revolving mechanism 430. A cab 424 (operator's cab) is provided at the front left side of the upper revolving body 420. A working attachment 440 is provided at the front center of the upper revolving body 420.
[0039] The real machine input interface 41 includes a real machine operating mechanism 411 and a real machine photographing device 412. The real machine operating mechanism 411 includes a plurality of operating levers arranged in the same manner as the remote operating mechanism 211 around a seat arranged inside the cab 424. A driving mechanism or a robot is provided in the cab 424 to receive a signal corresponding to the operation method of the remote operating lever and to operate the real machine operating lever based on the received signal. The real machine photographing device 412 is, for example, provided inside the cab 424, and photographs an environment including at least a part of the working mechanism 440 through a front window and a pair of left and right side windows defined by a pair of left and right pillars 4240 (including symbols "L" and "R" when distinguishing between left and right) located on the front side of the cab 424. Some or all of the front and side windows may also be omitted.
[0040] The real machine output interface 42 includes a real machine wireless communication device 422 .
[0041] The working attachment 440 as a working mechanism includes a boom 441 which is raisably mounted on the upper rotating body 420, an arm 443 which is rotatably connected to the top of the boom 441, and a bucket 445 which is rotatably connected to the top of the arm 443. The working attachment 440 is equipped with a boom cylinder 442, an arm cylinder 444, and a bucket cylinder 446 which are composed of telescopic hydraulic cylinders.
[0042] The boom cylinder 442 is interposed between the boom 441 and the upper slewing body 420, so that it can be extended and retracted by receiving the supply of hydraulic oil, thereby rotating the boom 441 in the lifting direction. The arm cylinder 444 is interposed between the arm 443 and the boom 441, so that it can be extended and retracted by receiving the supply of hydraulic oil, thereby rotating the arm 443 around the horizontal axis relative to the boom 441. The bucket cylinder 446 is interposed between the bucket 445 and the arm 443, so that it can be extended and retracted by receiving the supply of hydraulic oil, thereby rotating the bucket 445 around the horizontal axis relative to the arm 443.
[0043] (Function)
[0044] use Figure 4 The flowchart shown in the figure explains the functions of the remote operation support system having the above configuration. In the flowchart, the block "C●" is a mark used to simplify the description, and the mark refers to a conditional branch that executes processing in the branch direction based on the transmission and / or reception of data.
[0045] In the remote operation device 20, it is determined whether the operator OP has performed a designated operation via the remote input interface 210 ( Figure 4 / Step 202). The "designated operation" is, for example, an operation such as a tapping operation performed by the operator OP in the remote input interface 210 to designate the work machine 40 to be remotely operated in the overhead work environment image. Figure 4 / Step (STEP) 202: No), the process of determining whether there is a designated operation is repeated. On the other hand, when the determination result is positive ( Figure 4 / Step 202: Yes (YES)), send an environment confirmation request to the remote operation assistance server 10 through the remote wireless communication device 222 ( Figure 4 / Step 204).
[0046] In the remote operation support server 10, when receiving the environment confirmation request, the first auxiliary processing component 121 sends the environment confirmation request to the corresponding working machine 40 ( Figure 4 / C10).
[0047] In the working machine 40, when the real machine wireless communication device 422 receives the environment confirmation request ( Figure 4 / C40), the real machine control device 400 obtains the captured image through the real machine capturing device 412 ( Figure 4 / Step 402). The real machine control device 400 transmits the captured image data ( Figure 4 / Step 404).
[0048] In the remote operation assistance server 10, when the first auxiliary processing component 121 receives the captured image data ( Figure 4 / C11), the second auxiliary processing component 122 sends the captured image data to the remote operation device 20 ( Figure 4 / Step 102). Instead of the captured image data, environmental image data representing a simulated environmental image generated based on the captured image may be sent to the remote operating device 20. At this time, the second auxiliary processing component 122 may send to the remote operating device 20 an instruction for splitting and displaying the captured image corresponding to the captured image data on the image output devices 2210 to 2212.
[0049] In the remote operation apparatus 20, when the remote wireless communication device 222 receives the captured image data ( Figure 4 / C21), the remote control device 200 controls the split display mode of the captured image corresponding to the captured image data on the three image output devices 2210 to 2212 ( Figure 4 / Step 206).
[0050] Thus, for example Figure 5 As shown, the captured image of the left column 4240L and the right column 4240R constituting the cab 424 and the boom 441, the arm 443 and the bucket 445 as part of the working attachment 440 of the working mechanism in front of the column 4240 is divided into three parts in the horizontal direction and displayed on the central image output device 2210, the left image output device 2211 and the right image output device 2212 respectively. Figure 5 , a central image output device 2210, a left image output device 2211, and a right image output device 2212 are shown when facing the image output devices from a direction perpendicular to the screen of the central image output device 2210.
[0051] For example, Figure 5 As shown, the split display mode of the shooting device is controlled as follows: in the central image output device 2210 as the designated image output device, all designated parts of the designated part PPL of the left column 4240L overlapping with the designated image area R and the designated part PPR of the right column 4240R overlapping with the designated image area R are displayed, wherein the above-mentioned designated image area R includes a bucket 445 as a working part, and the designated image area R extends horizontally between the left side frame 2210L and the right side frame 2210R.
[0052] In the remote operation device 20, the operation mode of the remote operation mechanism 211 is recognized by the remote control device 200 ( Figure 4 / Step 208), and sending a remote operation instruction corresponding to the operation mode to the remote operation auxiliary server 10 through the remote wireless communication device 222 ( Figure 4 / Step 210).
[0053] In the remote operation support server 10, when the second auxiliary processing component 122 receives the remote operation instruction, the first auxiliary processing component 121 sends the remote operation instruction to the working machine 40 ( Figure 4 / C12).
[0054] In the working machine 40, when an operation instruction is received by the real machine control device 400 via the real machine wireless communication device 422 ( Figure 4 / C42), controlling the operation of the job attachment 440, etc. ( Figure 4 / Step 406) For example, an operation is performed in which the soil in front of the working machine 40 is scooped up by the bucket 445 and the soil is dumped from the bucket 445 after the upper rotating body 420 is rotated.
[0055] (Effect)
[0056] According to the remote operation support system having such a configuration, and the remote operation support server 10 and the remote operation device 20 constituting the remote operation support system, the captured image is displayed on the central image output device 2210 (designated image output device) (refer to Figure 4 / Step 402 → Step 404 → C11 → Step 102 → C21 → Step 206, refer to Figure 5 ), wherein the captured image represents the situation in front of the cab 424 and is obtained by the real machine capturing device 412 mounted inside the cab 424 of the working machine 40. Therefore, the operator can grasp the situation in front of the cab 424 of the working machine 40 or the real machine as the remote operation target through the captured image displayed on the central image output device 2210 (designated image output device), while operating the remote operation mechanism 211 (refer to Figure 4 / Step 208 → Step 210 → C12 → C42 → Step 406, refer to Figure 5 ).
[0057] Furthermore, the display mode of the captured image in the central image output device 2210 (designated image output device) is variably controlled so that the entire designated portion of the designated portion PPL, PPR, which is a portion of each of the pair of pillars 4240L, 4240R constituting the cab 424, overlapping with the designated image area R extending in a strip shape between the left and right frames 2210L, 2210R is displayed on the central image output device 2210 (designated image output device) (refer to Figure 5 ).
[0058] Thus, at least the left and right frames 2210L, 2210R of the central image output device 2210 (designated image output device) and a portion PPL, PPR of the pillars 4240L, 4240R can be clearly recognized in the designated image region R extending laterally between the left and right frames 2210L, 2210R. Therefore, the possibility of the operator misidentifying or confusing the pair of pillars of the cab 424 reflected in the captured image with the frames 2210L, 2210R of the central image output device 2210 in the captured image showing the pillars can be reduced.
[0059] In particular, when the operator's line of sight moves between the central image output device 2210 and the left image output device 2211, it is possible to reduce the possibility that the operator will mistakenly recognize or confuse the left side frame 2210L of the central image output device 2210 for the left side pillar 4240L constituting the cab 424 (see Figure 5). Similarly, when the operator's line of sight moves between the central image output device 2210 and the right image output device 2212, it is possible to reduce the possibility that the operator will mistakenly recognize or confuse the right side frame 2210R of the central image output device 2210 as the right side pillar 4240R constituting the cab 424.
[0060] Furthermore, using the designated portions PPL and PPR of the pillars 4240L and 4240R reflected in the captured image and the bucket 445 (working part) of the working attachment 440 (working mechanism) constituting the working machine 40 as indicators, space recognition can be performed, and the space recognition includes recognition of the position, direction, orientation, size, and shape of the bucket 445 in the actual space and recognition of the distance between the bucket 445 and the ground or the like as the working object. As described above, since the possibility of the operator misidentifying or confusing the pair of pillars 4240L and 4240R of the cab 424 reflected in the captured image with the frame 2210L and 2210R of the central image output device 2210 in the captured image showing the pillars 4240L and 4240R can be reduced, the accuracy of the operator's space recognition can be improved.
[0061] (Other embodiments of the present invention)
[0062] In the above-mentioned embodiment, the central image output device 2210 arranged at the front center of the remote operating mechanism 211 is defined as the designated image output device, but the left image output device 2211 arranged at the left oblique front of the remote operating mechanism 211 and / or the right image output device 2212 arranged at the right oblique front may be defined as the designated image output device on the basis of or in place of the central image output device 2210.
[0063] In the above embodiment, the image output device 221 is composed of three image output devices 2210, 2211, and 2212 (or a group of image output devices). However, as another embodiment, the image output device 221 may be composed of only one image output device (for example, the central image output device 2210), or may be composed of two image output devices or four or more image output devices.
[0064] In the above embodiment, the bucket 445 is defined as the working part, but other components of the working attachment 440 (working mechanism) (for example, the boom 441 and / or the arm 443 ) may be defined as the working part in addition to or in place of the bucket 445 .
[0065] The remote control device 20 may also include a sight line detection device for detecting the sight line of the operator. Figure 6As shown, the remote control device 200 may also define the following image area extending in a strip shape in the horizontal direction as a designated image area R, which image area includes a part or all of the trajectory Tr of the operator's sight target on the central image output device 2210 (designated image output device) during the second designated period (for example, 1 second to 1 minute) detected by the sight detection device. Figure 6 As shown, the remote control device 200 can also variably control the display mode of the captured image in the central image output device 2210 (designated image output device) so as to display the entire designated parts PPL, PPR of at least one side of the pillars 4240L, 4240R overlapping with the designated image area R.
[0066] According to the remote control device having such a configuration, the designated parts PPL, PPR of at least one of the pillars 4240L, 4240R reflected in the captured image and the object located at the sight target of the operator can be used as indicators to perform spatial recognition, which includes recognition of the position, direction, orientation, size, shape of the bucket 445 (working part) in the actual space and recognition of the interval between the bucket 445 and the working object. As described above, since the possibility of the operator misidentifying or confusing the pair of pillars of the cab 424 reflected in the captured image with the frame of the image output device in the captured image showing the pillars can be reduced, the accuracy of the operator's spatial recognition can be improved.
[0067] The designated image area R may also be defined in various other ways. For example, the remote control device 200 may define the following image area extending in a strip shape in the central image output device 2210 (designated image output device) as the designated image area R, and the image area includes a part or all of the position trajectory of the bucket 445 (working unit) during the first designated period (e.g., 1 second to 1 minute).
[0068] Furthermore, the remote control device 200 may define the following image area as a designated image area R, which is an area extending in a strip shape in a direction parallel to the displacement direction of the working part (e.g., bucket 445) of the working mechanism (e.g., working attachment 440) constituting the working machine 40 in the designated image output device (e.g., central image output device 2210) during the first designated period. Fig. 9As shown, in the central image output device 2210, an image region extending in a stripe shape in a direction parallel to the displacement direction (see the thick arrow) of the bucket 445 (working unit) in the first designated period (here, a direction inclined to the upper right) is defined as a designated image region R. And the display method of the captured image on the central image output device 2210 is controlled so that all designated portions of the designated portion PPL overlapping the designated image region R of the left column 4240L and the designated portion PPR overlapping the designated image region R of the right column 4240R are displayed on the image output device.
[0069] Moreover, the remote control device 200 may also define the following image area as a designated image area R, which is an area extending in a band-like direction parallel to the displacement direction of the line of sight target of the operator during a second designated period on a designated image output device (for example, the central image output device 2210) detected by the line of sight detection device.
[0070] The remote control device 200 may variably control the display mode of the captured image in the central image output device 2210 (designated image output device) according to different operation modes of the remote operation mechanism 211 operated by the operator.
[0071] According to the remote control device 20 having such a configuration, even when the state of the working machine 40 changes in accordance with the operator's operation of the remote control mechanism 211, and thus the situation in front of the cab 424 shown in the captured image changes, the display mode of the captured image can be variably controlled so that all the specified parts of the specified parts PPL, PPR of at least one of the pillars 4240L, 4240R are displayed in the specified image area R in the central image output device 2210 (specified image output device). Thus, the possibility of the operator misidentifying or confusing the pair of pillars 4240L, 4240R of the cab 424 reflected in the captured image that changes in accordance with the operation of the remote control mechanism 211 with the frame 2210L, 2210R of the central image output device 2210 on which the pillars 4240L, 4240R are displayed can be reduced.
[0072] The remote control device 200 may also perform the following control: when there is a rotation operation for rotating the upper rotating body 420 equipped with the cab 424 relative to the lower traveling body 410 in the remote operating mechanism 211 (see Figure 3), the remote control device 200 variably controls the display mode of the captured image so that a specified portion PPL (or PPR) of the column 4240L (or 4240R) on one side of the rotation direction of the upper rotating body 420 among the left and right pair of columns 4240L, 4240R is displayed more heavily in the central image output device 2210 (specified image output device) than a specified portion PPR (or PPL) of the column 4240R (or 4240L) on the other side.
[0073] According to the remote control device 20 having such a configuration, when the upper rotating body 420 of the working machine 40 rotates rightward relative to the lower traveling body 410 in response to the operator's rotating operation of the remote control mechanism 211, as shown in FIG. Figure 7 As shown in FIG. 1 , the front view of the cab 424 represented by the captured image is displaced as a whole in the direction opposite to the rotation direction (see the displacement direction arrow in the captured image) (see the white arrow). Figure 7 As shown, in the captured image displayed on the central image output device 2210, although at least a part of the designated part PPL of the left pillar 4240L is not displayed, the entire designated part PPR of the right pillar 4240R is displayed. Therefore, in the captured image displayed on the central image output device 2210, the area of the right pillar 4240R is larger than the area of the left pillar 4240L. Figure 7 In accordance with Figure 5 The implementation method of FIG. 1 defines a specified image region R, but it can also be implemented as Figure 6 Or other implementations are used to define the designated image area R.
[0074] Similarly, when the upper rotating body 420 of the working machine 40 rotates to the left relative to the lower traveling body 410 in response to the operator's rotating operation of the remote operating mechanism 211, as shown in FIG. Figure 8 As shown, the front of the cab 424 shown in the captured image is displaced as a whole in the direction opposite to the rotation direction (see the arrow in the captured image displacement direction). Figure 8 As shown, although at least a part of the designated part PPR of the right pillar 4240R is not displayed in the captured image displayed on the central image output device 2210, the entire designated part PPL of the left pillar 4240L is displayed. Therefore, in the captured image displayed on the central image output device 2210, the area of the left pillar 4240L is larger than that of the right pillar 4240R. Figure 8 In accordance with Figure 6 The implementation method of FIG. 1 defines a specified image region R, but it can also be defined as Figure 5 Or other implementations are used to define the designated image area R.
[0075] As a result, it is possible to reduce the possibility that the operator will misidentify or confuse the pillar 4240L or 4240R on one side of the operator's cab 424 projected in the rotation direction according to the rotation operation of the remote operating mechanism 211 with the frame 2210L or 2210R of the central image output device 2210 on which the pillar 4240L or 4240R is displayed. Since there is a tendency that the operator's sight target or attention is inclined to the right side of the central image output device 2210 when the upper rotating body 420 rotates rightward, and the operator's sight target or attention is inclined to the left side of the central image output device 2210 when the upper rotating body 420 rotates leftward, the control of the display method is particularly meaningful.
[0076] The remote control device 200 controls and splits the captured image so that it is displayed on the right image output device 2212, the central image output device 2210 and the left image output device 2211, and can variably control the split display method of the captured image on the right image output device 2212, the central image output device 2210 and the left image output device 2211.
[0077] When the operator's line of sight moves between the central image output device 2210 and the left image output device 2211, the possibility that the operator mistakenly recognizes or confuses the left side frame 2210L of the central image output device 2210 for the left side pillar 4240L constituting the cab 424 can be reduced. Similarly, when the operator's line of sight moves between the central image output device 2210 and the right image output device 2212, the possibility that the operator mistakenly recognizes or confuses the right side frame 2210R of the central image output device 2210 for the right side pillar 4240R constituting the cab 424 can be reduced.
[0078] The remote control device 200 uses the image area extending in a strip as a designated image area, and variably controls the display method of the captured image in the designated image output device, so that the entire designated part of at least one side of the column 4240L, 4240R is displayed on the designated image output device (for example, the central image output device 2210), wherein the above-mentioned image area includes the bucket 445 of the working attachment 440 constituting the working machine 40.
[0079] By using the designated parts of at least one of the pillars 4240L, 4240R reflected in the captured image and the working unit 445 of the working mechanism 440 constituting the working machine 40 as indicators, the possibility of the operator misidentifying or confusing between a pair of pillars 4240L, 4240R of the cab 424 reflected in the captured image and the frames 2210L, 2210R of the designated image output device displaying the pillars can be reduced, so that spatial recognition including the position, direction, orientation, size, shape of the working unit 445 in the actual space and the interval between the working unit 445 and the working object can be performed, thereby improving the accuracy of the operator's spatial recognition.
[0080] The remote control device 200 variably controls the display mode of the captured image in the designated image output device in the following manner: that is, a strip-shaped extended image area including a part or all of the position trajectory of the working unit during the first designated period on the designated image output device (for example, the central image output device 2210) is used as a designated image area, and the entire designated part of at least one of the pillars 4240L, 4240R is displayed.
[0081] The remote control device 200 variably controls the display mode of the captured image in the designated image output device in the following manner: that is, an image area extending in a strip shape in a direction parallel to the displacement direction of the working unit 445 during the first designated period on the designated image output device (for example, the central image output device 2210) is used as a designated image area, and the entire designated portion of at least one of the pillars 4240L, 4240R is displayed.
[0082] The remote operation device 20 also has a line of sight detection device for detecting the operator's line of sight. The remote control device 200 variably controls the display mode of the captured image in the designated image output device in the following manner: that is, an image area extending in a strip shape including a part or all of the trajectory of the operator's line of sight target on the designated image output device (for example, the central image output device 2210) during the second designated period detected by the line of sight detection device is used as a designated image area, and the entire designated part of at least one of the pillars 4240L, 4240R is displayed.
[0083] By using the designated part of at least one of the pillars 4240L, 4240R reflected in the captured image and the object located at the operator's sight target as an index or mark, spatial recognition can be performed, and the spatial recognition includes recognition of the position, direction, orientation, size, shape of the working unit 445 in the actual space and recognition of the interval between the working unit 445 and the working object. In this way, the possibility of the operator misidentifying or confusing the pair of pillars 4240L, 4240R of the cab 424 reflected in the captured image and the frame 2210L, 2210R of the image output device displaying the pillars can be reduced. Therefore, the accuracy of the operator's spatial recognition can be improved.
[0084] The remote control device 200 variably controls the display mode of the captured image in the designated image output device in the following manner: that is, an image area extending in a band in a direction parallel to the displacement direction of the line of sight target of the operator on the designated image output device (for example, the central image output device 2210) during the second designated period detected by the line of sight detection device is used as the designated image area, and the entire designated portion of at least one of the pillars 4240L, 4240R is displayed.
[0085] The remote control device 200 variably controls the display mode of the captured image on the designated image output device according to the operator's operation mode on the remote operation mechanism.
[0086] According to the operation of the remote operating mechanism 211 by the operator, even when the situation in front of the cab 424 shown in the captured image changes due to the change in the operating state of the working machine 40, the display mode of the captured image can be variably controlled so that the entire designated portion of at least one of the pillars is included in the designated image area on the designated image output device. Thus, the possibility of the operator misidentifying or confusing the pair of pillars 4240L, 4240R of the cab 424 reflected in the captured image that changes in response to the operation of the remote operating mechanism and the frames 2210L, 2210R of the image output device displaying the pillars can be reduced.
[0087] When there is a rotation operation for rotating the upper rotating body 420 equipped with the cab 424 relative to the lower traveling body 410 in the remote operating mechanism 211, the remote control device 200 can variably control the display method of the captured image in the designated image output device so that the column on one side of the rotation direction of the upper rotating body 420 in the left and right pair of columns 4240L, 4240R is displayed more heavily on the designated image output device (for example, the central image output device 2210) than the column on the other side.
[0088] In response to the operator's rotation operation of the remote control mechanism 211, when the upper rotating body 420 of the working machine 40 rotates relative to the lower traveling body 410 and the entire front view of the cab 424 shown in the captured image is displaced in the direction opposite to the rotation direction, the pillar 4240L (4240R) on one side in the rotation direction is displayed more heavily than the pillar 4240R (4240L) on the other side in the designated image area on the designated image output device (for example, the central image output device 2210). Thus, for example, in the captured image displayed on the designated image output device, although at least a part of the designated portion of the other side pillar 4240R (4240L) is not displayed, the entire designated portion of the pillar 4240L (4240R) on one side is displayed. In addition, the display method of the captured image on the designated image output device may be controlled so that the area of one pillar 4240L (4240R) (the number of pixels constituting the image area corresponding to the one pillar) is larger than the area of the other pillar 4240R (4240L) (the number of pixels constituting the image area corresponding to the other pillar). As a result, it is possible to reduce the possibility of the operator misidentifying or confusing one pillar 4240L (4240R) of the operator's cab 424 projected in the rotation direction according to the rotation operation of the remote operating mechanism 211 with the frame 2210L (2210R) of the image output device on which the pillar is displayed.
[0089] Explanation of symbols
[0090] 10…remote operation auxiliary server; 20…remote operation device; 40…operating machinery; 41…real machine input interface; 42…real machine output interface; 102…database; 121…first auxiliary processing element; 122…second auxiliary processing element; 200…remote control device; 210…remote input interface; 211…remote operation mechanism; 220…remote output interface; 221…image output device; 400…real machine control device; 410…lower traveling body; 420…upper rotating body; 424…operating cab (operating cab); 440…operating attachment (working mechanism); 445…bucket (operating part); 2210…central image output device (designated image output device); 2211…left image output device; 2212…right image output device; 4240L…left column; 4240R…right column; PPL, PPR…designated part of the column.
Claims
1. A remote operation device comprising a remote operation mechanism for an operator to operate a working machine remotely, one or more image output devices arranged around the remote operation mechanism, and a remote control device, The remote control device is characterized in that: The remote control device performs the following control: a captured image obtained by a real machine shooting device mounted on the inside of the cab of the working machine, showing the front of the cab, is displayed on a designated image output device among the one or more image output devices, and a display mode of the captured image in the designated image output device is variably controlled so that a portion of each of a pair of pillars constituting the cab that overlaps with a designated image area, namely, a designated portion of a left pillar and a designated portion of a right pillar, is displayed on the designated image output device, wherein: The designated image area extends in a band shape between left and right frames which are part of a housing of the designated image output device on the designated image output device.
2. The remote control device according to claim 1, characterized in that: The remote operation device includes the following image output devices as the plurality of image output devices: a central image output device as the designated image output device arranged in front of the remote operation mechanism; a left image output device and a right image output device adjacent to left and right side edges of the central image output device and arranged obliquely left and right in front of the remote operation mechanism, respectively; The remote control device performs the following control: the captured image is divided and displayed on the right image output device, the central image output device and the left image output device, and the divided display mode of the captured image in the right image output device, the central image output device and the left image output device is variably controlled.
3. The remote control device according to claim 1 or 2, characterized in that: The remote control device performs the following control: taking the image area extending in a strip shape on the designated image output device as the designated image area, variably controlling the display mode of the captured image in the designated image output device so that the entire designated part of the column of at least one side is displayed on the designated image output device, wherein the image area includes the working part constituting the working mechanism of the working machine.
4. The remote control device according to claim 3, characterized in that: The remote control device performs the following control: taking an image area extending in a strip shape on the designated image output device as the designated image area, variably controlling the display mode of the captured image in the designated image output device so that the entire designated portion of at least one of the columns is displayed on the designated image output device, wherein the image area includes a portion or all of the position trajectory of the working unit during the first designated period.
5. The remote control device according to claim 4, characterized in that: The remote control device performs the following control: an image area extending in a band shape in a direction parallel to the displacement direction of the working unit during the first designated period on the designated image output device is used as the designated image area, and the display mode of the captured image in the designated image output device is variably controlled so that the entire designated part of the at least one side of the column is displayed on the designated image output device.
6. The remote control device according to any one of claims 1 to 5, characterized in that: The remote control device further includes a sight line detection device for detecting the sight line of the operator. The remote control device performs the following control: taking an image area extending in a strip shape on the designated image output device as the designated image area, variably controlling the display mode of the captured image in the designated image output device so that the entire designated portion of the pillar on at least one side is displayed on the designated image output device, wherein the image area includes a part or all of a trajectory of the operator's line of sight target during a second designated period on the designated image output device detected by the line of sight detection device.
7. The remote control device according to claim 6, characterized in that: The remote control device performs the following control: taking an image area extending in a band in a direction parallel to the displacement direction of the line of sight target on the designated image output device during the second designated period detected by the line of sight detection device as the designated image area, and variably controlling the display mode of the captured image in the designated image output device so that the entirety of the designated portion of the pillar on at least one side is displayed on the designated image output device.
8. The remote control device according to any one of claims 1 to 7, characterized in that: The remote control device variably controls a display mode of the captured image on the designated image output device according to a difference in an operation mode of the operator on the remote operation mechanism.
9. The remote control device according to claim 8, characterized in that: The remote control device performs the following control: when there is a rotation operation in the remote operating mechanism for rotating the upper rotating body equipped with the cab relative to the lower traveling body, the display mode of the captured image in the designated image output device is variably controlled so that the column on one side of the rotation direction of the upper rotating body among the pair of left and right columns is displayed on the designated image output device more heavily than the column on the other side.
10. A remote operation assistance server, used to assist remote operation of a working machine through a remote operation device, characterized in that: have: a first auxiliary processing element that acquires a captured image based on communication with the working machine, the captured image being an image showing a state in front of the cab acquired by a real machine imaging device mounted inside a cab constituting the working machine; as well as The second auxiliary processing element variably controls the display mode of the captured image in the designated image output device based on the communication with the remote operating device so that a portion of each of a pair of pillars constituting the cab that overlaps with a designated image area, i.e., a designated portion of a left pillar and a designated portion of a right pillar, that is, the entirety of a designated portion of the left pillar and a designated portion of the right pillar are displayed in the designated image output device, wherein the designated image area is an image area extending in a band-like manner between left and right borders which are part of a frame of the designated image output device on a designated image output device among the one or more image output devices constituting the remote operating device.
11. A remote operation assistance system, characterized in that: The remote operation assistance system is composed of the remote operation device according to any one of claims 1 to 9 and the remote operation assistance server according to claim 10.
12. A remote operation assisting method for assisting remote operation of a working machine through a remote operation device, characterized in that: have: A first auxiliary process is to obtain a captured image based on communication with the working machine, the captured image being an image showing a front side of the cab obtained by a real machine imaging device mounted inside a cab constituting the working machine; as well as The second auxiliary processing is to variably control the display mode of the captured image in the designated image output device based on the communication with the remote operation device so that a portion of each of a pair of pillars constituting the cab that overlaps with a designated image area, i.e., a designated portion of a left pillar and a designated portion of a right pillar, that is, an entirety of the designated portion, of the left pillar, of the designated image area, which is an image area extending in a band-like manner between left and right borders which is a part of a frame of the designated image output device on a designated image output device among the one or more image output devices constituting the remote operation device.
Citation Information
Patent Citations
Display system, display method, and remote control system
JP2018207244A
Working vehicle, display unit and failure determination method
JP2019068236A
Construction machine
JP2019167733A
