Job assistance server, job assistance method

Abstract

The present invention provides a work assistance server and a work assistance method. The work assistance server (10) assists work in a manner that can selectively remotely operate multiple work machines (40) as operation objects in response to operations performed on a remote operation device (20) having an image output device (221). The first auxiliary processing element (121) of the work assistance server (10) identifies the position of each work machine (40), and the second auxiliary processing element (122) displays images of the positions of the first work machine (40a) and the second work machine (40b) on the image output device (221) when the operation object of the remote operation device (20) is switched from the first work machine (40a) to the second work machine (40b).

Description

[Technical field]

[0001] The present invention relates to a work assisting server for assisting remote operation of a work machine and a work assisting method for assisting work of the work machine. [Background Technology]

[0002] Conventionally, a plurality of fixed-point cameras are installed at a work site where a remote excavator or other working machine is working, so as to be able to observe the entire site from a bird's-eye view or to monitor the movement or work of a specific excavator.

[0003] For example, the construction site management system of the following patent document 1 has multiple camera devices and information prompting devices. The multiple camera devices are respectively arranged above the construction site to take bird's-eye view of the construction site. Moreover, all locations where manned vehicles can travel in the construction site are photographed by at least one camera device.

[0004] In addition, the information display device can generate an overall bird's-eye view image showing the entire construction site based on the images captured by the multiple camera devices, and display the overall bird's-eye view image depicting the transport vehicle on the vehicle-mounted terminal of the loading vehicle. The information display device is, for example, installed inside a building at the construction site (Patent Document 1 / Paragraphs 0008-0010, Figure 1 ).

[0005] [Prior art literature]

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2019-151968 [Summary of the invention]

[0008] Problems to be solved by the invention

[0009] The images (videos) of such a camera are helpful to grasp the positions of multiple working machines at the work site. However, when operating a working machine through a remote operating device, there is a situation where the operating object is switched. Specifically, the operating object machine is switched from the currently operated working machine to the next scheduled working machine. At this time, it is required to grasp not only the position of the scheduled operating machine, but also its posture and surrounding conditions. However, in the information prompting device described in the above-mentioned patent document 1, it is difficult to grasp this information.

[0010] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide a work support server that enables a worker to recognize the conditions around a plurality of work machines to be operated, the relative positional relationship, etc., and thereby support the work.

[0011] Means for solving problems

[0012] The first invention is a work assistance server, which assists the work of a plurality of work machines as the operation objects in a manner that can selectively perform remote operations on the work machines as the operation objects in response to operations performed on a remote operation device having a display device, and is characterized in that it comprises: a first auxiliary processing element, which identifies the position of the work machine as the operation object of the remote operation device; and a second auxiliary processing element, which displays an image of the positions of the first work machine and the second work machine simultaneously captured on the display device when the operation object of the remote operation device is switched from the first work machine to the second work machine.

[0013] The work support server of the present invention includes a first support processing element and a second support processing element to support work performed by operating a work machine as an operation target through a remote control device. The first support processing element recognizes the position of the work machine as an operation target.

[0014] In addition, when the operation object is switched from the operation of the first operation machine to the operation of the second operation machine, the second auxiliary processing element performs display control to display an image that simultaneously captures the positions of the first operation machine and the second operation machine, and displays the image on the display device. Thus, the operation auxiliary server enables the operator to recognize the relative positional relationship of multiple operation machines, etc., so as to perform the operation safely and reliably.

[0015] The second invention is a work assisting method, which assists the work of a plurality of work machines as the operation objects in a manner that can selectively perform remote operation on the plurality of work machines as the operation objects in response to the operation performed on a remote operating device having a display device, and is characterized in that it comprises: a first step of identifying the position of the work machine as the operation object of the remote operating device; and a second step of displaying an image of the positions of the first work machine and the second work machine simultaneously captured on the display device when the operation object of the remote operating device is switched from the first work machine to the second work machine.

[0016] In the operation assisting method of the present invention, a first step and a second step are provided in order to assist the operation performed by operating the operation machine as the operation object through the remote operation device. In the first step, the position of the operation machine as the operation object is identified. Then, in the second step, when the operation object is switched from the operation of the first operation machine to the operation of the second operation machine, a display control is performed to display an image that simultaneously captures the positions of the first operation machine and the second operation machine, and the image is displayed on the display device. Thus, the present operation assisting method enables the operator to identify the relative positional relationship of multiple operation machines, etc., so as to perform the operation safely and reliably.

[0017] [Brief Description of the Drawings]

[0018] Figure 1 It is a diagram for explaining the outline of the work support system according to the embodiment of the present invention.

[0019] Figure 2 It is a diagram for explaining the structure of an operating mechanism of a working machine.

[0020] Figure 3 It is a figure explaining the detail of a working machine.

[0021] Figure 4 This is an example of a bird's-eye view of a work site from above.

[0022] Figure 5 This is a flowchart of each process performed by the work support server and the remote operation device.

[0023] Figure 6 This is a flowchart of the display processing of the working machine switching process.

[0024] Figure 7 This is a flowchart of the process of determining the first condition.

[0025] Figure 8 This is an example of a captured image displayed on the image output device (1).

[0026] Fig. 9 This is an example of a captured image displayed on the image output device (2). [Specific implementation method]

[0027] Hereinafter, the details of the work support server according to the present invention will be described with reference to the drawings.

[0028] First, refer to Figure 1, a work support server 10 and a work support system 1 including the same according to an embodiment of the present invention are described. In addition to the work machine 40, the work support system 1 is also composed of the following parts: the work support server 10; a remote operation device 20, which is used to remotely operate the work machine 40; and a plurality of equipment cameras 30, which are set at a predetermined position on the work site to photograph the work machine 40.

[0029] The work support server 10, the remote operation device 20, the equipment camera 30, and the work machine 40 are configured to be able to communicate with each other via the network NW. The work support server 10 automatically selects an appropriate equipment camera 30 according to the switching of the work machine 40, which is the remote operation target of the remote operation device 20, to another work machine 40, the movement of the work machine 40 in response to the remote operation of the remote operation device 20, the work content of the work machine 40 performed by the remote operation of the remote operation device 20, etc. In addition, the work support server 10 outputs the captured image to the image output device 221 described later in order to assist the operation of the operator OP of the remote operation device 20.

[0030] The work support server 10 includes a database 102, a first support processing component 121, a second support processing component 122, and a server wireless communication device 124. The database 102 stores images captured by the device camera 30. The database 102 may be formed by a database server separate from the work support server 10.

[0031] The auxiliary processing elements 121 and 122 are composed of a calculation processing device (a single-core processor or a multi-core processor, or a processor core constituting the same). Furthermore, the auxiliary processing elements 121 and 122 read necessary data and software from a storage device such as a memory, and perform calculation processing in accordance with the software with the data as an object. In addition, the server wireless communication device 124 instructs the image output device 221 to display or receives a captured image from the device camera 30 via the network NW.

[0032] Next, the remote operation device 20 includes a remote control device 200, a remote input interface 210, and a remote output interface 220. The remote control device 200 is composed of a computing device (a single-core processor or a multi-core processor, or a processor core constituting the same). In addition, the remote control device 200 reads necessary data and software from a storage device such as a memory, and performs computing processing according to the software with the data as the object.

[0033] The remote input interface 210 includes a remote operation mechanism 211. The remote output interface 220 includes an image output device 221 and a remote wireless communication device 222. The remote wireless communication device 222 transmits an operation signal to the working machine 40 or receives a captured image from the machine camera 30 via the network NW.

[0034] The remote control mechanism 211 includes a travel control device, a rotation control device, an arm control device, an arm control device, and a bucket control device. Each control device has an operation lever that receives a rotation operation. The operation lever (travel lever) of the travel control device can be operated to operate the lower mobile body 427 of the working machine 40 (see Figure 3 ). The travel rod may also serve as a travel pedal. For example, a travel pedal fixed to the base or lower end of the travel rod may also be provided.

[0035] The operating lever (rotation lever) of the swing operating device is used to operate the hydraulic swing motor constituting the swing mechanism 430 of the working machine 40. In addition, the operating lever (boom lever) of the boom operating device is used to operate the boom cylinder 442 of the working machine 40 (see Figure 3 ).

[0036] The operating lever (arm lever) of the arm operating device is used to operate the arm cylinder 444 of the working machine 40. The operating lever (bucket lever) of the bucket operating device is used to operate the bucket cylinder 446 of the working machine 40 (see Figure 3 ).

[0037] Here, Figure 2 An example of the operation levers constituting the remote operation mechanism 211 and a seat St for the operator OP to sit on is shown. The seat St is in the form of a high-back chair with armrests. The seat St may also be in any form in which the operator OP can sit, such as a low-back chair without a headrest, or a seat without a backrest.

[0038] In front of the seat St, a pair of left and right travel levers 2110 corresponding to the left and right crawlers are arranged in a horizontal arrangement. One operating lever may also serve as multiple operating levers. For example, the left operating lever 2111 provided in front of the left frame of the seat St may also function as a boom lever when operated in the front-rear direction, and may function as a bucket lever when operated in the left-right direction.

[0039] Similarly, the right operating lever 2112 provided in front of the right frame of the seat St can also function as a boom lever when operated in the front-rear direction, and as a swing lever when operated in the left-right direction. In addition, the lever mode can also be arbitrarily changed according to the operation instruction of the operator OP.

[0040] like Figure 2 As shown, the image output device 221 is composed of a central image output device 2210, a left image output device 2211, and a right image output device 2212, each of which has a substantially rectangular screen and is disposed in front of, obliquely left in front of, and obliquely right in front of the seat St. The shapes and sizes of the screens (image display areas) of the central image output device 2210, the left image output device 2211, and the right image output device 2212 may be the same or different.

[0041] In addition, the right edge of the left image output device 2211 is adjacent to the left edge of the central image output device 2210 so that the screen of the central image output device 2210 and the screen of the left image output device 2211 are inclined at an angle θ1 (for example, 120°≤θ1≤150°). Similarly, the left edge of the right image output device 2212 is adjacent to the right edge of the central image output device 2210 so that the screen of the central image output device 2210 and the screen of the right image output device 2212 are inclined at an angle θ2 (for example, 120°≤θ2≤150°). The inclination angles θ1 and θ2 may be the same or different.

[0042] The screens of the central image output device 2210, the left image output device 2211, and the right image output device 2212 may be parallel to the vertical direction or inclined to the vertical direction. In addition, at least one of the central image output device 2210, the left image output device 2211, and the right image output device 2212 may be composed of a plurality of image output devices divided into a plurality of image output devices. For example, the central image output device 2210 may be composed of a pair of image output devices adjacent to each other in the upper and lower directions and having a substantially rectangular screen. In addition, the image output devices 2210 to 2212 may further include a speaker (sound output device).

[0043] return Figure 1 The device camera 30 includes a photographing device 300 and a camera wireless communication device 310. The photographing device 300 has a predetermined photographing range and may also be set with a field of view that can be moved up, down, left, and right. The photographing device 300 basically always continues photographing during operation, but may also start photographing when the device camera 30 is selected.

[0044] In addition, the camera wireless communication device 310 transmits the captured image captured by the image capturing device 300 to the server wireless communication device 124 of the operation support server 10 and the remote wireless communication device 222 of the remote operation device 20 via the network NW. The display switching instructions of the captured object, direction, zoom-in, zoom-out, etc. may be performed by the operation support server 10 while performing the image analysis, or may be instructed by the operator OP.

[0045] Next, the working machine 40 includes a real machine control device 400, a real machine input interface 410, a real machine output interface 420, and a working mechanism (working attachment) 440. The real machine control device 400 is composed of a computing device (a single-core processor or a multi-core processor, or a processor core constituting the same). In addition, the real machine control device 400 reads necessary data and software from a storage device such as a memory, and performs computing processing in accordance with the software with the data as an object.

[0046] then, Figure 3 An example of a working machine 40 according to the present embodiment is shown.

[0047] The working machine 40 is, for example, a crawler excavator (construction machine), and includes a crawler-type lower moving body 427 and an upper rotating body 435 rotatably mounted on the lower moving body 427 via a rotating mechanism 430. In addition, a control room (operating room) 425 is provided on the front left side of the upper rotating body 435, and a working mechanism 440 is provided in the front center of the upper rotating body 435.

[0048] return Figure 1 The real machine input interface 410 includes a real machine operation mechanism 411 and a real machine shooting device 412. The real machine operation mechanism 411 includes a plurality of operation levers arranged around a seat arranged inside the control room 425 in the same manner as the remote operation mechanism 211. A drive mechanism or a robot that receives a signal corresponding to the operation form of the remote operation lever and moves the real machine operation lever based on the received signal is set in the control room 425.

[0049] The real machine photographing device 412 is installed inside the control room 425, and photographs the environment including at least a part of the working mechanism 440 through the front window of the control room 425. In addition to the real machine photographing device 412, the working machine 40 may also have an omnidirectional camera, which is composed of a plurality of photographing devices and can photograph the surrounding area of ​​the working machine 40 in all directions (360°).

[0050] The real machine output interface 420 is provided with a real machine wireless communication device 422. The real machine wireless communication device 422 receives a signal corresponding to the operation form of the remote operating lever from the remote wireless communication device 222 of the remote operating device 20 via the network NW. The signal is further sent to the real machine control device 400, and the working machine 40 operates according to the signal. In addition, a driving mechanism or a robot that operates the real machine operating lever based on the signal is set in the control room 425.

[0051] The working mechanism (working attachment) 440 includes a boom 441 mounted on the upper rotating body 435, an arm 443 rotatably connected to the front end of the boom 441, and a bucket 445 rotatably connected to the front end of the arm 443. The working mechanism 440 is equipped with a boom cylinder 442, an arm cylinder 444, and a bucket cylinder 446, which are composed of telescopic hydraulic cylinders. In addition, the positioning device 460 is a device for detecting the position of the working machine 40. The positioning device 460 is composed of, for example, a GNSS receiver (GNSS: Global Navigation Satellite System).

[0052] The boom cylinder 442 is interposed between the boom 441 and the upper slewing body 435, and is extended and retracted by receiving the supply of hydraulic oil to rotate the boom 441 in the up-and-down direction. The arm cylinder 444 is interposed between the arm 443 and the boom 441, and is extended and retracted by receiving the supply of hydraulic oil to rotate the arm 443 around the horizontal axis relative to the boom 441.

[0053] The bucket cylinder 446 is interposed between the bucket 445 and the arm 443 , and is extended and retracted by receiving supply of hydraulic oil to rotate the bucket 445 around a horizontal axis relative to the arm 443 .

[0054] Here, Figure 4 An example of arrangement of the equipment cameras 30 at a work site is shown.

[0055] Figure 4 The bird's-eye view image of the work site from above is shown. At the work site, the equipment camera 30a and the equipment camera 30b are set at separate locations. In addition, the area R surrounded by the locations R1 to R3 and shown by the dotted line is the entire area that can be photographed by the two equipment cameras 30a and 30b. In the area R, there are the first working machine 40a and the second working machine 40b. In addition, the equipment camera can also be further set at a position different from the equipment cameras 30a and 30b.

[0056] Next, refer to Figure 5 , a flowchart of the functions of the work support system 1 of this embodiment is described. Figures 6 to 9 , the switching of the device camera 30, the display of the captured image, etc. are explained.

[0057] In addition, blocks “C11” to “C13” in the flowchart are used for simplification of description, and mean transmission and / or reception of data, and mean that processing in a branch direction is executed on the condition of transmission and / or reception of the data.

[0058] First, the operator OP starts working the first working machine 40 a in the remote operating device 20 ( STEP 201 ). At this time, the remote operating device 20 transmits a “work start signal” to the work support server 10 .

[0059] The operation support server 10 receives the operation start signal (C11) and issues a start instruction for sending the captured image (STEP101). Here, the real machine image capturing device 412 mounted on the first operation machine 40a is selected. In addition, the image capturing device to be selected is not limited to the real machine image capturing device 412. For example, a first camera that is set at a predetermined position on the operation site and includes the first operation machine 40a in the image capturing range may be selected together with the real machine image capturing device 412.

[0060] Next, the remote operation device 20 starts outputting the image captured by the image capturing device selected in STEP 101 to the image output device 221 (STEP 202). Thus, the operator OP can observe the image output device 221 to confirm the situation in front of the first working machine 40a from the inside of the operating room 425 of the first working machine 40a through the front window of the operating room 425.

[0061] When selecting an image capturing device other than the real machine image capturing device 412, the operator OP can simultaneously check the status of the first working machine 40a from another viewpoint. For example, when selecting the equipment camera 30a, the operator OP can also check the status around the first working machine 40a.

[0062] Next, when the operator OP performs an operation to switch from the work of the first working machine 40a to the work of the second working machine 40b, the remote operation device 20 transmits a “working machine switching signal” to the work support server 10 (STEP 203 ).

[0063] The work support server 10 receives the work machine switching signal (C12), and executes a display process S of outputting an output showing the operation target switching process to the image output device 221 (STEP 102).

[0064] Here, refer to Figure 6 and Figure 7 , the flowchart regarding the above-mentioned display processing S is explained.

[0065] When the display process S is started, first, it is determined whether the first condition ( Figure 6 / STEP 301). Depending on whether the first condition is satisfied, the output method of the operation target switching process for the image output device 221 is changed.

[0066] The first condition is, for example, that the first working machine 40a and the second working machine 40b are captured in the image captured by the device camera 30. When the device camera 30 has a zoom-in / zoom-out function, the first condition is that the first working machine 40a and the second working machine 40b are captured in the image captured in the state of the maximum range that can be captured when zoomed out.

[0067] The determination of whether the first condition is satisfied is specifically performed as follows. The work support server 10 performs image analysis processing on each captured image captured by the device camera 30, and determines whether there is a real machine image Qi ( Figure 7 / STEP401).

[0068] If the determination result is positive ( Figure 7 / STEP401…YES), the first auxiliary processing component 121 determines whether the real machine image Qi corresponds to any one of the first working machine 40a and the second working machine 40b ( Figure 7 / STEP 402). Specifically, it is determined whether the actual spatial positions of the first working machine 40a and the second working machine 40b corresponding to the real machine image Qi are consistent with or correspond to the actual spatial positions of the first working machine 40a and the second working machine 40b registered in the database 102.

[0069] The actual spatial position of the working machine 40 corresponding to the actual machine image Qi is obtained by transforming the position (e.g., center of gravity) of the actual machine image Qi in the image coordinate system into the actual space coordinate system. At this time, the actual spatial distance from the equipment camera 30 to the working machine 40 is estimated based on the size of the actual machine image Qi in each captured image.

[0070] In addition, during the coordinate transformation, the position of the working machine 40 in the coordinate system of the equipment camera 30 is identified, and a coordinate conversion factor (matrix or quaternion) representing the position and posture of the imaging device in the real space coordinate system is used. The coordinate conversion factor is registered in the database 102 in association with the identifier of the imaging device (or even the image identifier).

[0071] When mutual communication between the remote operating device 20 (client) and the working machine 40 is established, the identifiers of the remote operating device 20 (or its operator) and the working machine 40 are registered in the database 102 in association with the actual spatial position of the working machine 40 .

[0072] The actual spatial position of the working machine 40 is measured by the positioning device 460 mounted on the working machine 40. The working machine 40 sends the actual spatial position or its time series to the working support server 10, thereby updating the actual spatial position of the working machine 40 registered in the database 102. Thus, the actual spatial positions of the first working machine 40a and the second working machine 40b can be retrieved or identified from the database 102 based on the identifier.

[0073] If the determination result is positive ( Figure 7 / STEP402…YES), the job support server 10 considers that the first condition ( Figure 7 / STEP403). In this case, the work support server 10 adds a label to each captured image and saves it in the database 102. That is, the work support server 10 stores the captured image ( Figure 7 / STEP404).

[0074] Then, a photographic image that captures the positional relationship between the first working machine 40a and the second working machine 40b and the condition of the working site between the first working machine 40a and the second working machine 40b is selected from the photographic images stored in the database 102 ( Figure 7 / STEP405).

[0075] Selection of captured images ( Figure 7 / STEP 405) For example, the following criteria (1) to (4) are used. Each time each criterion is met, the score is increased, and the one with the higher score is selected.

[0076] (1) The first working machine 40a and the second working machine 40b are simultaneously displayed.

[0077] (2) The area of ​​the actual machine image Qi is relatively large.

[0078] (3) The first working machine 40a and the second working machine 40b are displayed in a horizontal arrangement.

[0079] (4) The first working machine 40a and the second working machine 40b do not overlap.

[0080] On the other hand, if the determination result of the presence or absence of the real machine image Qi is negative ( Figure 7 / STEP401 ... NO), or if the result of determination as to whether the real machine image Qi corresponds to any one of the first working machine 40a and the second working machine 40b is negative ( Figure 7 / STEP402…NO), the job support server 10 considers that the first condition ( Figure 7 / STEP406).

[0081] When the first condition is met ( Figure 6 / STEP301…YES), the work support server 10 issues a change instruction for sending the captured image ( Figure 6 / STEP302). Specifically, the work support server 10 selects the selection of the captured image ( Figure 7 / STEP405) is used as the device camera 30 for photographing the selected photographed image.

[0082] Then, the work support server 10 instructs the equipment camera 30 to zoom in on the first work machine 40a. Figure 8 The image shown in (a) is switched to the image of the front of the first working machine 40a viewed from the inside of the operating room 425 of the first working machine 40a through the front window of the operating room 425. Figure 8 (b) is an enlarged image of the first working machine 40a photographed by the equipment camera 30.

[0083] Next, the work support server 10 instructs the device camera 30 to zoom out until the second work machine ( Figure 6 / STEP303). Therefore, Figure 8 As shown in (c), the first working machine 40a and the second working machine 40b are captured within the imaging range of the equipment camera 30, and the captured images are output to the image output device 221.

[0084] Next, the work support server 10 instructs the device camera 30 to zoom in on the second work machine 40b as the center ( Figure 6 / STEP304). Thus, Figure 8 As shown in (d), the enlarged image of the second working machine 40b captured by the equipment camera 30 is output to the image output device 221.

[0085] Finally, the work support server 10 issues a change instruction for sending the captured image ( Figure 6 / STEP305). Thus, Figure 8 As shown in (e), an image captured from inside the operator's cab 425 of the second operating machine 40b and viewed through the front window of the operator's cab 425 is output to the image output device 221.

[0086] The display of the operation object switching process is performed as described above, and the display process S ends (STEP 102). In this way, the work support server 10 (first support processing component 121) controls the display of the operation object switching process for the operator OP to recognize.

[0087] Furthermore, the work support server 10 (second support processing unit 122) may flash or color the images of the first and second work machines 40a and 40b to display them, thereby allowing the operator OP to easily understand the current positions and positional relationship of the work machines 40a and 40b.

[0088] In addition, if Figure 4 As shown, an arrow mark M may be displayed from the first working machine 40a toward the second working machine 40b to indicate that the remote operation target of the remote operation device 20 is changed from the first working machine 40a to the second working machine 40b.

[0089] If the first condition ( Figure 6 / STEP301…NO), the work support server 10 issues a change instruction for sending the captured image ( Figure 6 / STEP 306) Specifically, the work support server 10 selects the bird's-eye view image data of the work site where the first work machine 40a is located, and further instructs the server to zoom in on the first work machine 40a.

[0090] The bird's-eye view image data may be, for example, a photograph taken in advance by an airplane or map information. Thus, an image obtained by enlarging the bird's-eye view image data with the position of the first working machine 40a as the center is output to the image output device 221.

[0091] Here, Fig. 9 A display example of the image output device 221 in this case is shown.

[0092] The image output to the image output device 221 is from Fig. 9 The image shown in (a) is switched to the image of the front of the first working machine 40a viewed from the inside of the operating room 425 of the first working machine 40a through the front window of the operating room 425. Fig. 9 (b) shows an image obtained by enlarging the bird's-eye view image data of the work site where the first work machine 40a is located, with the position of the first work machine 40a as the center. A simulated machine image of the work machine 40a is displayed at the position of the first work machine 40a.

[0093] Next, the work support server 10 reduces the bird's-eye view image of the work site where the first work machine 40a is located to a position where the second work machine is reflected ( Figure 6 / STEP307). A simulated machine image of the working machine 40b is displayed at the position of the second working machine 40b. Fig. 9 The bird's-eye view image showing the positions of the first working machine 40 a and the second working machine 40 b as shown in (c) is output to the image output device 221 .

[0094] In addition, there is a case where the work site where the first work machine 40a is located is far away from the work site where the second work machine 40b is located. In this case, after zooming out the bird's-eye view image, Fig. 9 As shown in (c'), the image is zoomed out to the position where the first working machine 40a and the second working machine 40b are projected on the map.

[0095] Next, the work support server 10 instructs the user to enlarge the bird's-eye view image at the position of the second work machine 40b ( Figure 6 / STEP308). Thus, Fig. 9 As shown in (d), the bird's-eye view image in which the position of the second working machine 40b is enlarged is output to the image output device 221.

[0096] Finally, the job assistance server 10 will Fig. 9 As shown in (e), an image of the front of the second working machine 40b observed from the inside of the operating room 425 of the second working machine 40b through the front window of the operating room 425 is output to the image output device 221.

[0097] The display of the operation object switching process is performed as described above, and the display process S ends (STEP 102). In this way, the work support server 10 (first support processing component 121) controls the display of the operation object switching process for the operator OP to recognize.

[0098] In addition, the operation support server 10 (second auxiliary processing element 122) may also flash the images of the first operation machine 40a and the second operation machine 40b in the bird's-eye view image, or display the images in a colorized manner, so that the operator OP can easily understand the current positions, positional relationships, etc. of the operation machines 40a and 40b.

[0099] In addition, an arrow mark M pointing from the first working machine 40a toward the second working machine 40b may be displayed to indicate that the remote operation target of the remote operation device 20 is changed from the first working machine 40a to the second working machine 40b.

[0100] Figure 8 and Fig. 9 The images displayed on the image output device 221 shown may be switched automatically and continuously, or may be switched by an operator's operation (eg, button operation).

[0101] Furthermore, even if it is determined that the first condition ( Figure 6 / STEP301…YES), it is also possible to perform the operation without satisfying the first condition ( Figure 6 / STEP301…NO). In this case, for example, the process when the first condition is satisfied and the process when the first condition is not satisfied can be selected at the same time. In this case, any one of the display processes can be selected according to the operator's preference.

[0102] Thereafter, the operator OP ends the work of the second working machine 40b in the work support server 10 (STEP 204). At this time, a "work end signal" is transmitted to the work support server 10.

[0103] The operation support server 10 receives the operation end signal (C13) and issues an instruction to stop sending the captured image (STEP 103). The operation support server 10 may also issue an instruction to stop sending to all cameras at the operation site, not limited to the second camera currently sending the captured image. The above terminates a series of processing of the operation support server 10.

[0104] Finally, the remote operation device 20 stops outputting the captured image (STEP 205). As a result, the display of the captured image on the image output device 221 ends. The above is the end of a series of processes of the remote operation device 20.

[0105] In the above embodiment, the operation support server 10 controls to switch the plurality of equipment cameras 30 provided at the predetermined positions of the operation site to display the process of changing the operation object from the first operation machine 40a to the second operation machine 40b. However, if there is a camera that replaces the equipment camera 30, the camera may be used. For example, in the case where the camera of the third operation machine 40c can photograph the first operation machine 40a and the second operation machine 40b, the camera of the third operation machine 40c may be selected to display the process.

[0106] Thus, the work support server 10 uses the images captured by the equipment camera 30 and the real machine image capturing device 412 of the work machine 40 to display the switching of the work machine 40 on the image output device 221. Thus, the operator OP can easily understand the situation around the work machine 40, the relative position relationship, etc.

[0107] Preferably, in the work assistance server of the present invention, the first auxiliary processing component grasps the positions and shooting directions of multiple equipment cameras arranged at the work site where the work machine is located, and selects the equipment cameras that simultaneously capture the positions based on the positions and shooting directions of the equipment cameras and the positions of the first work machine and the second work machine, and the second auxiliary processing component performs equipment camera display processing to display the images taken by the equipment cameras selected by the first auxiliary processing component on the display device.

[0108] According to this structure, the first auxiliary processing element can identify the positions of the first working machine and the second working machine, and thus can select a device camera that simultaneously captures the positions from a plurality of device cameras arranged at the working site. In addition, the second auxiliary processing element can show the positions of the first working machine and the second working machine to the operator based on the images taken by the device cameras. Thus, the work support server can show the conditions around the working machines.

[0109] In addition, preferably, in the work assistance server of the present invention, based on the respective positions of the first working machine and the second working machine identified by the first auxiliary processing element, the equipment camera captures an overhead image overlooking the respective positions, and the second auxiliary processing element performs overhead image display processing to display the captured overhead image on the display device.

[0110] According to this configuration, a bird's-eye view image that allows identification of the positions of the first and second working machines is displayed on the display device of the remote operating device. Thus, the work support server allows the operator to identify the relative positional relationship of the working machines.

[0111] In addition, preferably, in the work assistance server of the present invention, it is possible to execute: equipment camera display processing, through the above-mentioned first auxiliary processing component, grasp the position and shooting direction of the equipment camera configured at the work site where the above-mentioned work machine is located, and select the above-mentioned equipment camera that simultaneously captures the positions of the above-mentioned first work machine and the above-mentioned second work machine according to the position of the equipment camera and the shooting direction, and the respective positions of the above-mentioned first work machine and the above-mentioned second work machine, and through the above-mentioned second auxiliary processing component, make the above-mentioned display device display the captured image captured by the equipment camera selected by the above-mentioned first auxiliary processing component; and bird's-eye view image display processing, through the above-mentioned first auxiliary processing component, based on the respective positions of the above-mentioned first work machine and the above-mentioned second work machine, select the bird's-eye view image that simultaneously captures the respective positions, and through the above-mentioned second auxiliary processing component, make the above-mentioned display device display the bird's-eye view image selected by the above-mentioned first auxiliary processing component, and the above-mentioned second auxiliary processing component performs the above-mentioned bird's-eye view image display processing when the above-mentioned equipment camera display processing cannot be performed or when it is determined that the above-mentioned equipment camera display processing is not set to be performed.

[0112] When the positions of the first working machine and the second working machine can be confirmed by the images captured by the device camera, the work support server executes the device camera display processing to cause the display device to display the images captured simultaneously at the positions of the working machines.

[0113] On the other hand, when the positions of the first and second working machines cannot be confirmed by the images captured by the device camera, the operation support server performs a bird's-eye view image display process to display the bird's-eye view image instead of the captured image. Thus, the operation support server enables the operator to recognize the relative positional relationship of the working machines. In addition, even when the positions of the first and second working machines can be confirmed by the images captured by the device camera, the operation support server can display the bird's-eye view image according to the setting.

[0114] In addition, preferably, in the work assistance server of the present invention, the first auxiliary processing component determines that the device camera display processing cannot be performed when any device camera configured at the work site where the first working machine is located cannot capture the respective positions of the first working machine and the second working machine.

[0115] The display device preferentially displays the image captured by the device camera in real time. However, if the positions of the first working machine and the second working machine cannot be captured at the same time, the first auxiliary processing element determines that the device camera display processing cannot be performed. In this case, the work auxiliary server can, for example, enable the operator to reliably identify the relative positions of the working machines by displaying a bird's-eye view image.

[0116] In addition, preferably, in the work assistance server of the present invention, the above-mentioned first auxiliary processing element performs conversion image display processing, and the conversion image display processing is to convert the images of the above-mentioned positions captured at the same time and displayed on the above-mentioned display device in the order of the following states: a state in which the above-mentioned first working machine is enlarged, a state in which the above-mentioned first working machine is reduced so that the above-mentioned second working machine is also reflected, and a state in which the above-mentioned second working machine is enlarged.

[0117] The work support server can allow the operator to reliably recognize the work machine to be focused on by performing the conversion image display process. The work support server can allow the operator to reliably recognize the change of the control target from the first work machine to the second work machine, and can display the relative positional relationship of the work machines.

[0118] [Explanation of symbols]

[0119] 1: work support system; 10: work support server; 20: remote operation device; 30, 30a, 30b: equipment camera; 40, 40a, 40b: work machine; 102: database; 121: first auxiliary processing element; 122: second auxiliary processing element; 123: server wireless communication device; 200: remote control device; 210: remote input interface; 220: remote output interface; 221: image output device; 222: remote wireless communication device ; 300: photographing device; 310: camera wireless communication equipment; 400: actual machine control device; 410: actual machine input interface; 412: actual machine photographing device; 420: actual machine output interface; 425: control room; 427: lower moving body; 430: rotating mechanism; 435: upper rotating body; 440: working mechanism (working attachment); 445: bucket; 460: positioning device; 2110: travel lever; 2111: left operating lever; 2112: right operating lever.

Claims

1. A work assisting server for assisting remote operation of a work machine, which assists the operation of the work machines as operation objects in a manner that can selectively remotely operate a plurality of work machines as operation objects in response to an operation performed on a remote operation device having a display device, and comprises: a first auxiliary processing element that recognizes a position of the working machine that is an operation target of the remote operation device; and The second auxiliary processing element displays an image simultaneously capturing positions of the first working machine and the second working machine on the display device when the operating target of the remote operating device is switched from the first working machine to the second working machine.

2. The work support server for supporting remote operation of a work machine according to claim 1, wherein: The first auxiliary processing component grasps the positions and shooting directions of a plurality of equipment cameras arranged at the work site where the working machine is located, and selects the equipment cameras that simultaneously capture the respective positions according to the positions and shooting directions of the equipment cameras and the respective positions of the first working machine and the second working machine. The second auxiliary processing component performs a device camera display process of displaying an image captured by the device camera selected by the first auxiliary processing component on the display device.

3. The work support server for supporting remote operation of a work machine according to claim 2, wherein: Based on the positions of the first working machine and the second working machine identified by the first auxiliary processing element, the device camera captures a bird's-eye view image of the positions. The second auxiliary processing element performs a bird's-eye view image display process for displaying the captured bird's-eye view image on the display device.

4. The work support server for supporting remote operation of a work machine according to claim 1, which is capable of executing: Equipment camera display processing, through the first auxiliary processing component, grasping the position and shooting direction of the equipment camera arranged at the work site where the working machine is located, and according to the position and shooting direction of the equipment camera and the respective positions of the first working machine and the second working machine, selecting the equipment camera that simultaneously captures the respective positions, and through the second auxiliary processing component, making the display device display the captured image captured by the equipment camera selected by the first auxiliary processing component; and The bird's-eye view image display processing includes selecting, based on the respective positions of the first working machine and the second working machine, bird's-eye view images simultaneously captured at the respective positions by the first auxiliary processing element, and causing the display device to display the bird's-eye view image selected by the first auxiliary processing element by the second auxiliary processing element. The second auxiliary processing component performs the bird's-eye view image display process when the device camera display process cannot be performed or when it is determined that the device camera display process is not set to be performed.

5. The work support server for supporting remote operation of a work machine according to claim 4, wherein: The first auxiliary processing component determines that the device camera display process cannot be performed when any device camera arranged at the work site where the first work machine is located cannot capture the positions of the first work machine and the second work machine.

6. The work support server for supporting remote operation of a work machine according to any one of claims 1 to 5, wherein: The first auxiliary processing element performs a conversion image display process, and the conversion image display process is to convert the image of each position captured while being displayed on the display device in the following order: a state in which the first working machine is enlarged, a state in which the first working machine is reduced so that the second working machine is also reflected, and a state in which the second working machine is enlarged.

7. A method for assisting the operation of a work machine, which assists the operation of the work machines as the operation objects in a manner that can selectively remotely operate a plurality of work machines as the operation objects in response to an operation performed on a remote operation device having a display device, comprising: The first step is to identify the position of the working machine to be operated by the remote operating device; and In the second step, when the operation target of the remote operation device is switched from the first working machine to the second working machine, an image simultaneously capturing the positions of the first working machine and the second working machine is displayed on the display device.

Images