Job assistance server and job assistance system

Abstract

The present invention provides an operation assistance server and an operation assistance system. It enables an operator of a remote operation device to identify which remote operation device the work machine displayed on the output interface of the remote operation device is the remote operation object of. For example, a first work environment image representing the work site situation obtained by a photographing device mounted on the work machine is output on the output interfaces of a plurality of remote operation devices. In response to an identification image request accompanied by an identifier from a first remote operation device, it is determined whether the actual machine image (Qi) belonging to the cooperative work machine identified by the identifier is included in the first work environment image. When the actual machine image (Qi) belongs to the cooperative work machine, a first identification image is output to the output interface of at least one of the first and second remote operation devices. The "first identification image" represents the information that the actual machine image (Qi) is a cooperative work machine.

Description

Technical Field

[0001] The present invention relates to a work support server for assisting in remotely operating a work machine using a remote operation device. Background Art

[0002] There has been proposed a terminal device for a remote monitoring support system (for example, refer to Patent Document 1): This terminal device for a remote monitoring support system is used to achieve information sharing with sufficient accuracy between a worker who patrols inside a plant for inspection and a person who waits outside the work site. The terminal device includes: an image input unit for inputting image data of the site; an input operation selection unit such as a pen or a mouse; a detection unit for detecting whether new images are acquired; a communication control unit for wirelessly transmitting and receiving data to and from the outside; and an input / output screen display unit for displaying an input screen for inputting specified data.

[0003] Prior Art Documents

[0004] Patent Documents

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2005-242830 Summary of the Invention

[0006] Problems to be Solved by the Invention

[0007] However, when a captured image obtained by a photographing device is displayed on an output interface (image display device) constituting a remote operation device and a plurality of work machines are reflected in the captured image, an operator may misidentify between the work machine that is the operation target in the displayed image and the surrounding work machines that are non-operation targets. In particular, when a plurality of operators operate work machines respectively, the machine that is the operation target of one of the operators and the other machines that are the operation targets of other operators need to be recognized by their respective operators, but it is difficult to share the same recognition among the plurality of operators. This is particularly significant when a plurality of work machines of the same model of the same manufacturer are simultaneously displayed on the output interface. In this case, from the viewpoint of performing the target work, the operation mode of the work machine using the remote operation device based on the operator's misidentification may become an inappropriate operation mode.

[0008] Therefore, an object of the present invention is to provide a server and a system that can enable an operator of a remote operation device to recognize which remote operation device the work machine displayed on the output interface constituting the remote operation device is the remote operation target of.

[0009] Means for Solving the Problems

[0010] The present invention relates to a work assistance server for assisting in remotely operating a work machine using a remote operation device.

[0011] The work assistance server of the present invention is used to assist in the remote operation of each of a plurality of work machines, where each of the plurality of work machines uses a remote operation device. The work assistance server includes:

[0012] A database in which the respective identifiers of the remote operation device and the cooperative work machine, which is a work machine that cooperates with the remote operation device, are stored and held in association with the actual spatial position of the cooperative work machine.

[0013] A first assistance processing element that, based on communication with each of the plurality of remote operation devices, causes the output interfaces of each of the plurality of remote operation devices to output a work environment image corresponding to a captured image of the work site obtained by a capturing device; and

[0014] A second assistance processing element that performs the following processing: when a marker image request accompanied by the identifier of the cooperative work machine is received based on communication with each of the plurality of remote operation devices, identifies the actual spatial position of the cooperative work machine stored and held in association with the identifier in the database, determines whether the actual spatial position of the cooperative work machine is included in the actual spatial capture range of the capturing device, and when the determination result is affirmative, causes the output interfaces of each of the plurality of remote operation devices to output a marker image indicating that the cooperative work machine exists at a specified position corresponding to the actual spatial position of the cooperative work machine in the work environment image.

[0015] According to the work assistance server having the above configuration, a work environment image showing the situation of the work site obtained by the capturing device is output on the respective output interfaces of the plurality of remote operation devices. Based on the marker image requests accompanied by identifiers sent from the plurality of remote operation devices respectively, it is determined whether the actual spatial position of the cooperative work machine identified by the identifier is included in the actual spatial capture range of the capturing device. When the determination result is affirmative, a marker image indicating the presence of the cooperative work machine is output on the respective output interfaces of the plurality of remote operation devices. Thus, each of the plurality of operators of each of the plurality of remote operation devices can respectively identify the presence of a cooperative work machine that cooperates with at least one of the remote operation devices in the work environment image.

[0016] Therefore, it is possible for a plurality of operators to commonly identify the presence of a cooperative work machine that cooperates with one remote operation device or with each remote operation device in the work environment image output on the output interface of the remote operation device.

[0017] Brief Description of the Drawings

[0018] Figure 1 It is an explanatory diagram of the configuration of an operation assistance system as an embodiment of the present invention.

[0019] Figure 2 It is an explanatory diagram of the structure of a remote operation device.

[0020] Figure 3 It is an explanatory diagram of the structure of a work machine.

[0021] Figure 4 It is an explanatory diagram of the first function of the operation assistance system.

[0022] Figure 5 It is an explanatory diagram of the second function of the operation assistance system.

[0023] Figure 6 It is an explanatory diagram of the first work environment image obtained by a photographing device mounted on a work machine.

[0024] Figure 7 It is an explanatory diagram of the first work environment image obtained by a photographing device installed at a work site.

[0025] Figure 8 It is an explanatory diagram of an aerial photograph image.

[0026] Figure 9A It is an explanatory diagram of a first identification image.

[0027] Figure 9B It is an explanatory diagram of a second identification image.

[0028] Figure 10A It is an explanatory diagram of a first work environment image including an object machine.

[0029] Figure 10B It is an explanatory diagram of another first work environment image including an object machine.

[0030] Figure 11 It is an explanatory diagram of the photographing range of a photographing device mounted on a work machine installed at a work site and an object machine. Detailed Embodiment

[0031] (Configuration of the Operation Assistance System)

[0032] Figure 1The work assistance system shown as an embodiment of the present invention is composed of a work assistance server 10 and a plurality of remote operation devices 20 for remotely operating a plurality of work machines 40. The work assistance server 10, the remote operation devices 20, and the work machines 40 are configured to be able to communicate with each other via a network. Each of the plurality of remote operation devices 20 can also be configured to be able to communicate with each other via a network different from the communication network of the work assistance server 10.

[0033] (Configuration of the work assistance server)

[0034] The work assistance server 10 includes a database 102, a first assistance processing element 121, and a second assistance processing element 122. In addition to storing and holding the positions and / or movement tracks of the respective work machines 40, the database 102 also stores and holds captured images obtained by cameras provided in the respective work machines 40, work environment images and path guidance images provided to the respective operators who operate the work machines 40, and the like. The database 102 may also be constituted by a database server different from the work assistance server 10. Each assistance processing element is constituted by an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the processor), reads necessary data and software from a storage device such as a memory, and performs arithmetic processing described later according to the software with the data as an object.

[0035] (Structure of the work machine)

[0036] The work machine 40 includes an actual machine control device 400, an actual machine input interface 41, an actual machine output interface 42, a working mechanism 440, and a positioning device 460. The actual machine control device 400 is constituted by an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the processor), reads necessary data and software from a storage device such as a memory, and performs arithmetic processing according to the software with the data as an object.

[0037] The work machine 40 is, for example, a crawler excavator (construction machinery), as Figure 3 shown, the crawler excavator includes a crawler-type lower traveling body 410 and an upper revolving body 420 rotatably mounted on the lower traveling body 410 via a slewing mechanism 430. A cab (driver's cab) 424 is provided at the front left portion of the upper revolving body 420. A working attachment 440 is provided at the front central portion of the upper revolving body 420.

[0038] The actual machine input interface 41 is equipped with an actual machine operating mechanism 411 and an actual machine photographing device 412. The actual machine operating mechanism 411 has a plurality of operating levers arranged around the seat disposed inside the cab 424 in the same manner as the remote operating mechanism 211. A driving mechanism or a robot is provided inside the cab 424, and the driving mechanism or the robot receives a signal corresponding to the operation mode of the remote operating lever and operates the actual machine operating lever based on the received signal. The actual machine photographing device 412 is provided, for example, inside the cab 424, and photographs the environment including at least a part of the working mechanism 440 through the front window of the cab 424.

[0039] The actual machine output interface 42 is equipped with an actual machine wireless communication device 422.

[0040] The work attachment 440 as a working mechanism includes: a boom 441 that can be mounted on the upper swing body 420 in a liftable manner; an arm 443 that is rotatably connected to the tip of the boom 441; and a bucket 445 that is rotatably connected to the tip of the arm 443. A boom cylinder 442, an arm cylinder 444, and a bucket cylinder 446 formed of a telescopic hydraulic cylinder are installed on the work attachment 440.

[0041] The boom cylinder 442 is interposed between the boom 441 and the upper swing body 420, and expands and contracts by receiving the supply of working oil, so that the boom 441 rotates in the lift direction. The arm cylinder 444 is interposed between the arm 443 and the boom 441, and expands and contracts by receiving the supply of working oil, so that the arm 443 rotates relative to the boom 441 about a horizontal axis. The bucket cylinder 446 is interposed between the bucket 445 and the arm 443, and expands and contracts by receiving the supply of working oil, so that the bucket 445 rotates relative to the arm 443 about a horizontal axis.

[0042] (Configuration of the remote operation device)

[0043] The remote operation device 20 that constitutes the client includes a remote control device 200, a remote input interface 210, and a remote output interface 220. The remote control device 200 is constituted by an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the processor), reads necessary data and software from a storage device such as a memory, and performs arithmetic processing according to the software on the data. The remote input interface 210 is equipped with a remote operation mechanism 211. The remote output interface 220 is equipped with an image output device 221 and a remote wireless communication device 222.

[0044] The client can also be constituted by a portable terminal such as a smartphone or a tablet terminal or a wearable terminal such as a VR goggles that cooperates with the remote operation device 20 or has a mutual communication function. The portable terminal or the wearable terminal may also have a function of communicating with the operation assistance server 10.

[0045] The remote operation mechanism 211 includes an operation device for traveling, an operation device for slewing, an operation device for boom, an operation device for arm, and an operation device for bucket. Each operation device has an operation lever that receives a rotational operation. By operating the operation lever (travel lever) of the operation device for traveling, the lower traveling body 410 of the work machine 40 is moved. The travel lever can also serve as a travel pedal. For example, a travel pedal fixed to the base or the lower end of the travel lever may be provided. To operate the hydraulic slewing motor that constitutes the slewing mechanism 430 of the work machine 40, the operation lever (slewing lever) of the operation device for slewing is operated. To operate the boom cylinder 442 of the work machine 40, the operation lever (boom lever) of the operation device for boom is operated. To operate the arm cylinder 444 of the work machine 40, the operation lever (arm lever) of the operation device for arm is operated. To operate the bucket cylinder 446 of the work machine 40, the operation lever (bucket lever) of the operation device for bucket is operated.

[0046] As Figure 2 shown, each operation lever constituting the remote operation mechanism 211 is arranged, for example, around the seat St for the operator to sit on. The seat St is in the form of a high-back seat with armrests, and may also be in the form of a low-back chair without a headrest, or a chair without a backrest, etc., in any form that allows the remote operator OP2 to sit.

[0047] A pair of left and right travel levers 2110 corresponding to the left and right crawlers are arranged side by side in front of the seat St. One operation lever can also serve as multiple operation levers. For example, when the right operation lever 2111 shown in Figure 2 front of the right side frame of the seat St is operated in the front-rear direction, it functions as a boom lever, and when operated in the left-right direction, it functions as a bucket lever. Similarly, when the left operation lever 2112 shown in Figure 2 front of the left side frame of the seat St is operated in the front-rear direction, it functions as an arm lever, and when operated in the left-right direction, it functions as a slewing lever. The lever mode can be arbitrarily changed according to the operator's operation instruction.

[0048] For example, as Figure 2 shown, the image output device 221 is composed of a right front image output device 2211 arranged at the right front of the seat St, a front image output device 2212 arranged in front of the seat St, and a left front image output device 2213 arranged at the left front of the seat St. The image output devices 2211 to 2213 may also be provided with speakers (voice output devices).

[0049] (Function)

[0050] Use Figure 4 and Figure 5 The flowchart shown in Figure 5 is used to illustrate the functions of the job assistance system with the above configuration. In this flowchart, the block "C●" is a label used for simplified notation, which refers to the transmission and / or reception of data, and also refers to a conditional branch that executes processing in the branch direction based on the transmission and / or reception of this data.

[0051] (First function (remote operation of construction machinery))

[0052] In the remote operation device 20, it is determined whether the operator has performed a first specified operation through the remote input interface 210 ( Figure 4 / Step (STEP) 200). The "first specified operation" is, for example, an operation performed on the remote input interface 210 for the operator to select the construction machinery 40 that the operator intends to remotely operate. For example, this operation is an operation such as a tap performed by the operator through the remote input interface 210. If the determination result is negative ( Figure 4 / Step 200... No (NO)), a series of processes is ended.

[0053] On the other hand, if the determination result is positive ( Figure 4 / Step 200... Yes (YES)), the remote operation device 20 obtains candidates for construction machinery that can be the target machinery for remote operation by the remote operation device 20 via the communication network, and displays various information such as the identifier of the construction machinery, the location, model, and operating status of the construction machinery on the remote output interface 220 ( Figure 4 / Step (STEP) 201).

[0054] Next, in the remote operation device 20, it is determined whether the operator has performed a second specified operation through the remote input interface 210 ( Figure 4 / Step (STEP) 202). The "second specified operation" refers to an operation for specifying the construction machinery to be remotely operated from the candidates for construction machinery that can be the target machinery for remote operation. For example, this operation is an operation such as a tap performed through the remote input interface 210. If the determination result is negative ( Figure 4 / Step (STEP) 202... No), a series of processes is ended. On the other hand, if the determination result is positive ( Figure 4 / Step 202... Yes), a job environment image request is sent to the job assistance server 10 through the remote wireless communication device 222 ( Figure 4 / Step 203). The job environment image request includes at least one of the identifier of the remote operation device 20 and the identifier of the operator.

[0055] In the job assistance server 10, when a job environment image request is received, the job environment image request is sent to the corresponding construction machine 40 by the first assistance processing element 121( Figure 4 / C10).

[0056] In the construction machine 40, when a job environment image request is received through the on-site wireless communication device 422( Figure 4 / C41), the on-site control device 400 acquires a captured image through the on-site imaging device 412( Figure 4 / Step (STEP) 402). The on-site control device 400 sends captured image data representing the captured image to the job assistance server 10 through the on-site wireless communication device 422( Figure 4 / Step (STEP) 404).

[0057] In the job assistance server 10, when the captured image data is received( Figure 4 / C11), the first job environment image data corresponding to the captured image data (data representing all or part of the captured image itself, or simulated first job environment image data generated based on the image) is sent to the remote operation device 20( Figure 4 / Step (STEP) 112).

[0058] In the remote operation device 20, when the first job environment image data is received through the remote wireless communication device 222( Figure 4 / C20), the first job environment image corresponding to the first job environment image data is output to the image output device 221( Figure 4 / Step (STEP) 204).

[0059] Thus, for example, as Figure 6 shown, a job environment image including a part of the working attachment 440 such as the boom 441, the arm 443, the bucket 445, and the arm cylinder 444 of the working attachment is displayed on the image output device 221.

[0060] In addition, based on the captured image obtained by the imaging device C provided at the job site (refer to Figure 8 ), for example, as Figure 7 shown, an image representing the situation of the job site where the actual machine images Q1 to Q4 of multiple (4) construction machines 40 exist is additionally or alternatively output to the image output device 221 as the first job environment image.

[0061] Furthermore, for example, as Figure 8As shown, an aerial photographed image or an aerial view showing the overall situation of the work site can also be output to the image output device 221 additionally or alternatively as the first work environment image, where the aerial photographed image or the aerial view shows a real machine image of the work machine 40 existing at the work site or icons Q1 to Q4. The aerial photographed image can also be obtained, for example, by a photographing device mounted on a drone or a photographing device installed on a structure such as a mounting rod at the work site. The photographing position and the field of view angle of the photographed image as the first work environment image can also be arbitrarily changed respectively. The aerial view can also be a diagram generated based on the aerial photographed image.

[0062] In the remote operation device 20, the remote control device 200 identifies the operation mode of the remote operation mechanism 211 ( Figure 4 / Step (STEP) 206), sends a signal corresponding to the operation mode of the remote operation lever to the work machine 40, and prepares to drive the real machine operation lever based on the sent signal. Then, in the remote operation device 20, it is determined whether the operator has performed a third specified operation via the remote input interface 210 ( Figure 4 / Step (STEP) 208). The "third specified operation" refers to an operation for enabling the work machine 40 selected as the operation object to be operable by the remote operation device 20. For example, this operation is an operation such as a tap on the remote input interface 210. The remote operation of the work machine 40 by the remote operation device 20 is started through the "third specified operation". Then, a remote operation instruction corresponding to this operation mode can be sent to the work assistance server 10 through the remote wireless communication device 222 ( Figure 4 / Step (STEP) 210).

[0063] In the work assistance server 10, when receiving this remote operation instruction, the first assistance processing element 121 sends this remote operation instruction to the work machine 40 ( Figure 4 / C12).

[0064] In the work machine 40, when the real machine control device 400 receives an operation instruction through the real machine wireless communication device 422 ( Figure 4 / C42), it controls the actions of the work attachment 440, etc. ( Figure 4 / Step (STEP) 406). For example, it performs an operation of shoveling the soil in front of the work machine 40 with the bucket 445, turning the upper slewing body 420, and then dumping the soil from the bucket 445.

[0065] (Second function (output of real machine specified image))

[0066] In the remote operation device 20 (first client), it is determined whether the operator has performed a fourth specified operation via the remote input interface 210 ( Figure 5 / Step 210). The "fourth specified operation" is one of the following operations: an operation for sharing the same identification among multiple operators regarding which operator operates this machine, which is the operation object of one operator himself / herself, and which operator operates another machine, which is the operation object of another operator. For example, this operation is an operation such as a tap performed through the remote input interface 210. For example, the act that the operator of the first client (first remote operation device 20) requests to communicate with the operator of the second client (second remote operation device 20) and the operator of the second client (second remote operation device 20) agrees to this request becomes an opportunity for the fourth specified operation.

[0067] Next, an identification image request can be sent to the operation assistance server 10 through the remote wireless communication device 222 ( Figure 5 / Step (STEP) 212). This identification image request corresponds to the operation where the operator selects the target actual machine for which identification sharing is to be performed. For example, the operator selects the target actual machine for which identification sharing is to be performed based on the identifier of the construction machine, as well as various information such as the position, model, and operating status of the construction machine displayed on the remote output interface 220. When the remote output interface 220 displays candidates for the target actual machine, by at least displaying the working machine 40, which is the operation object of the first client, and the working machine 40, which is the operation object of the operator of the second client (second remote operation device 20) that is communicating with each other, and highlighting them, it becomes easier to share the identification of this machine and the other machine that is communicating. In the identification image request, in addition to the image identifier for identifying the first working environment image output to the image output device 221, it also includes at least one identifier of the first identifier for identifying the first client or its operator and the second identifier for identifying the second client or its operator.

[0068] In the operation assistance server 10, when the identification image request is received ( Figure 5 / C13), the first auxiliary processing element 121 retrieves from the database 102 based on the first identifier and identifies the actual spatial position of the target actual machine ( Figure 5 / Step (STEP) 122).

[0069] When the mutual communication between the remote operation device 20 (client) and the construction machine 40 is established, the respective identifiers of the remote operation device 20 (or its operator) and the construction machine 40 are registered in the database 102 in association with the actual spatial position of the construction machine 40. The actual spatial position of the construction machine 40 is measured by the GPS mounted on the construction machine 40 and the positioning device 460 that uses an acceleration sensor as needed. The construction machine 40 sends the actual spatial position or the time series of the actual spatial position to the operation support server 10, thereby updating the actual spatial position of the construction machine 40 registered in the database 102. Thus, the actual spatial position of the target actual machine can be identified based on the first identifier and the second identifier, respectively.

[0070] Next, the first auxiliary processing element 121 determines whether the target actual machine is included in the first operation environment image identified by the image identifier included in the identification image request ( Figure 5 / step (STEP) 123). The case where the target actual machine is included in the first operation environment image means that, respectively, as Figure 10A and Figure 10B shown, the target actual machine 40 is included in the first operation environment image (the operation environment image obtained by the imaging device 412 mounted on a construction machine 40 different from the target actual machine).

[0071] Specifically, first, the shooting range of the first operation environment image is determined. When the first operation environment image is captured by the imaging device 412 of the construction machine 40, the actual spatial position of the construction machine 40 and the shooting direction of the imaging device 412 (the up-and-down and left-and-right mounting directions relative to the construction machine 40) are obtained using the positioning device 460. When the first operation environment image is captured by the imaging device C installed at the Figure 8 operation site shown, the actual spatial position of the imaging device C and the shooting direction of the imaging device C are obtained.

[0072] Next, based on the actual spatial position of the construction machine 40 registered in the database 102, it is determined whether the target actual machine is included in the shooting range of the first operation environment image in the actual space coordinate system ( Figure 5 / step (STEP) 123).

[0073] If the determination result is affirmative ( Figure 5 / step 123... yes), the position of the target actual machine in the actual space coordinate system is transformed into the position in the first operation environment image coordinate system. Then, image analysis processing is performed on the first operation environment image, and it is determined whether the actual machine image Qi representing the construction machine 40 is extracted at the position of the target actual machine in the first operation environment image coordinate system ( Figure 5 / Step (STEP) 124).

[0074] When the determination result is affirmative ( Figure 5 / Step 124... Yes), the second auxiliary processing element 122 generates a first identification image indicating that the actual machine image Qi is the target actual machine (this machine or other machines), and sends the data representing the first identification image to the first client through the second auxiliary processing element 122. Figure 5 / Step (STEP) 125). Then, a specified notice indicating that the first identification image is overlapped on the first working environment image is sent. Figure 5 / Step (STEP) 132).

[0075] In the remote operation device 20 (the first client), when the first identification image data is received through the remote wireless communication device 222 ( Figure 5 / C21), the first identification image that emphasizes the presence of the target actual machine in the first working environment image is output through the image output device 221 that constitutes the remote output interface 220. Figure 5 / Step (STEP) 214). Thus, for example, as Figure 9A shown, the result of the texture image T overlapping the first working environment image within the area surrounded by the set EC of the edge points of the actual machine image Qi is output as the first identification image. The color of the texture image T can be changed according to the operator's preference. In addition, for example, as Figure 9B shown, the result of the marker image M overlapping the first working environment image beside the actual machine image Qi is output as the first marker image. The marker image M can be set to a corresponding mark (text) according to the operator's preference.

[0076] For example, as Figure 11 shown, when the determination result is negative ( Figure 5 / Step 124... No), in addition to indicating the case where the working machine 40-3 is not facing the target actual machine 40-0, it also indicates the case of being blocked by other working machines or obstacles.

[0077] When the target actual machine is not included in the shooting range of the first working environment image in the actual space coordinate system ( Figure 5 / Step 123... No), or when the position of the target actual machine in the first working environment image coordinate system does not extract the actual machine image Qi representing the working machine 40 ( Figure 5 / Step 124... No), it is determined whether the target actual machine is included in the shooting range of the second working environment image in the actual space coordinate system. Figure 5 / Step (STEP) 126). Here, the second operation environment image is the operation environment image provided to the second client, and it is the operation environment image of the working machine 40 that the operator of the second client takes as the object of remote operation. Since the second operation environment image for the second client is the same as the first operation environment image for the first client, the description of the method for generating its image is omitted.

[0078] When the determination result is affirmative ( Figure 5 / Step 126… is (YES)), the position of the object real machine in the actual space coordinate system is transformed into the position in the second operation environment image coordinate system. Then, taking the second operation environment image as the object, image analysis processing is performed to determine whether a real machine image Qi of the working machine 40 is extracted at the position of the object real machine in the second operation environment image coordinate system Figure 5 / Step (STEP) 127).

[0079] When the determination result is affirmative ( Figure 5 / Step 127… is), the second auxiliary processing element 122 generates a second identification image indicating that this real machine image Qi is the object real machine (this machine or other machines), and the second auxiliary processing element 122 sends the data representing the second identification image to the first client Figure 5 / Step (STEP) 128). Then, a specified notification indicating that the second identification image is overlapped on the second operation environment image is sent Figure 5 / Step 132).

[0080] In the remote operation device 20, when the second identification image data is received through the remote wireless communication device 222 ( Figure 5 / C22), the image output device 221 constituting the remote output interface 220 outputs the second identification image in which the existence of the object real machine is emphasized in the second operation environment image instead of or together with the second operation environment image Figure 5 / Step 216). Thus, for example, as Figure 9A shown, the result of the texture image T overlapping the second operation environment image within the area surrounded by the set EC of the edge points of the real machine image Qi is output as the second marked image. The color of the texture image T can be changed according to the preference of the operator. In addition, for example, as Figure 9B shown, the result of the marked image M overlapping the second operation environment image beside the real machine image Qi is output as the second marked image. The marked image M can be set to the corresponding mark (character) according to the preference of the operator.

[0081] When the object real machine is not included in the shooting range of the second operation environment image in the actual space coordinate system (Figure 5 / Step 126…No), or when the position of the actual machine of the object in the second work environment image coordinate system fails to extract the actual machine image Qi representing the working machine 40 Figure 5 / Step 127…No), determine whether the actual machine of the object is included in the shooting range of the third work environment image in the actual space coordinate system Figure 5 / Step 129). Here, the third work environment image is a work environment image not provided to the first client and the second client that is communicating with the first client. The third work environment image is a work environment image that can be provided to the second client that does not communicate with the first client, or a work environment image that can be provided to the first client and the second client that is communicating with the first client.

[0082] If the determination result is affirmative Figure 5 / Step 129…Yes), transform the position of the actual machine of the object in the actual space coordinate system into the position in the third work environment image coordinate system. Then, perform image analysis processing on the third work environment image, and determine whether the position of the actual machine of the object in the third work environment image coordinate system extracts the actual machine image Qi representing the working machine 40 Figure 5 / Step 130). If the determination result is affirmative Figure 5 / Step 130…Yes), the second auxiliary processing element 122 generates a third identification image indicating that the actual machine image Qi is the actual machine of the object (this machine or other machines), and the second auxiliary processing element 122 sends data representing the third identification image to the first client Figure 5 / Step 131). Then, send a specified notification indicating that the third identification image is overlapped on the third work environment image Figure 5 / Step 132).

[0083] In the remote operation device 20, when receiving the third identification image data through the remote wireless communication device 222 Figure 5 / C23), the image output device 221 constituting the remote output interface 220 outputs the third identification image in which the existence of the actual machine of the object is emphasized and displayed in the second work environment image instead of or together with the third work environment image Figure 5 / Step 218). Thus, for example, as Figure 9A shown, the result of the texture image T overlapping the second work environment image in the area surrounded by the set EC of the edge points of the actual machine image Qi is output as the second marker image. The color of the texture image T can be changed according to the preference of the operator. In addition, for example, as Figure 9BAs shown, the result of the marked image M overlapping with the second work environment image beside the actual machine image Qi is output as the second marked image. The marked image M can be set to corresponding marks (characters) according to the operator's preference.

[0084] When the object actual machine is not included in the shooting range of the third work environment image in the actual space coordinate system ( Figure 5 / step (STEP) 129... No), when the position of the object actual machine in the third work environment image coordinate system fails to extract the actual machine image Qi representing the work machine 40 ( Figure 5 / step 130... No), a specified notice equivalent to the meaning that the actual machine image of the object actual machine does not exist in the first to third work environment images is sent to the remote operation device 20 ( Figure 5 / step (STEP) 132).

[0085] In the remote operation device 20, when the specified notice is received through the remote wireless communication device 222 ( Figure 5 / C24), the specified notice is displayed through the image output device 221 constituting the remote output interface 220. The specified notice is one of the following notices: a notice of which image among the first to third work environment images contains the object actual machine, or information that the object actual machine is not contained in any of the first to third work environment images.

[0086] In the remote operation device 20, the image output device 221 may have the following function: it can switch the screens of the first to third work environment images regardless of whether the object actual machine is included in the shooting range.

[0087] The texture image T and the marked image M are common images in the second client that communicates with the first client. For example, when the red texture image T is output as the first identification image on the actual machine image Qi through the image output device 221 of the first client, on the image output device 221 of the second client, the texture image T of the actual image Qi also becomes red.

[0088] (Effect)

[0089] According to the work assistance system having this configuration and the work assistance server 10 constituting this work assistance system, the first work environment image representing the situation of the work site obtained by the shooting device (for example, the shooting device 412 mounted on the work machine 40) is output to the remote output interfaces 220 of the respective remote operation devices 20 (refer to Figure 4 / step (STEP) 204 and Figures 6 - 8)。Based on an identification image request accompanied by an identifier from the first remote operation device 20, it is determined whether the actual machine image Qi belonging to the cooperative work machine identified by the identifier is included in the first work environment image (refer to Figure 5 / Step (STEP) 123, Figure 5 / Step 124). When the actual machine image Qi belongs to the cooperative work machine, the first identification image is output to the remote output interface 220 of at least one of the first remote operation device and the second remote operation device 20 (refer to Figure 5 / Step (STEP) 214, Figure 9A and Figure 9B ). The "first identification image" indicates the information that the actual machine image Qi is a cooperative work machine.

[0090] Thus, the operator of the first client (the first remote operation device 20) sends an identification image request accompanied by the first identifier to the job assistance server 10 by operating the remote input interface 210, so that it is possible to confirm whether one of the actual machine images Qi included in the first work environment image output to the remote output interface 220 of the first client belongs to the first client identified by the first identifier or the machine that is the operation object of the operator, that is, this machine. Further, the operator of the first client sends a marked image request accompanied by the second identifier to the job assistance server 10 by operating the remote input interface 210, so that it is possible to confirm whether one of the actual machine images Qi included in the first work environment image output to the remote output interface 220 of the first client belongs to the second client identified by the second identifier or the machine that is the operation object of the operator, that is, another machine.

[0091] Similarly, the operator of the first client (the first remote operation device 20) sends an identification image request accompanied by the first identifier to the job assistance server 10 by operating the remote input interface 210, so that the operator of the second client can confirm the following: whether one of the actual machine images Qi included in the first work environment image output to the remote output interface 220 of the second client belongs to the first client identified by the first identifier or another machine operated by the operator. Further, the operator of the first client sends an identification image request accompanied by the second identifier to the job assistance server 10 by operating the remote input interface 210, so that the operator of the second client can confirm the following: whether one of the actual machine images Qi included in the first work environment image output to the remote output interface 220 of the second client belongs to the second client identified by the second identifier or the machine that is the operation object of the operator, that is, this machine.

[0092] Therefore, the following recognition sharing can be achieved: multiple operators can respectively recognize which remote operation device 20 the working machine 40 in the first working environment image projected onto the output interface constituting the remote operation device is the remote operation target of.

[0093] Furthermore, when all of the one or more actual machine images Qi included in the first working environment image do not belong to the cooperative working machine, the second working environment image including the actual machine image Qj corresponding to the cooperative working machine is output to the remote output interface 220 of the remote operation device 20 together with the second identification image (see Figure 5 / Step (STEP) 126 to Step (STEP) 128, Step 216). The "second identification image" represents the information that the actual machine image Qj is a cooperative working machine.

[0094] Thus, the operator of the first client (the first remote operation device 20) sends an identification image request accompanied by a first identifier to the job assistance server 10 by operating the remote input interface 210, so that it is possible to confirm whether a certain actual machine image among the actual machine images Qj included in the second working environment image output to the remote output interface 220 of the first client belongs to this machine. Furthermore, the operator of the first client sends an identification image request accompanied by a second identifier to the job assistance server 10 by operating the remote input interface 210, so that it is possible to confirm whether a certain actual machine image among the actual machine images Qj included in the second working environment image output to the remote output interface 220 of the first client belongs to another machine.

[0095] Similarly, the operator of the first client sends an identification image request accompanied by a first identifier to the job assistance server 10 by operating the remote input interface 210, so that the operator of the second client can confirm whether a certain actual machine image among the actual machine images Qj included in the second working environment image output to the remote output interface 220 of the second client belongs to another machine. Furthermore, the operator of the first client sends an identification image request accompanied by a second identifier to the job assistance server 10 by operating the remote input interface 210, so that the operator of the second client can confirm whether a certain actual machine image among the actual machine images Qj included in the second working environment image output to the remote output interface 220 of the second client belongs to this machine.

[0096] Therefore, the following recognition sharing can be achieved: multiple operators can respectively recognize which remote operation device 20 the working machine 40 in the second working environment image projected onto the output interface constituting the remote operation device is the remote operation target of.

[0097] (Other Embodiments of the Present Invention)

[0098] In the above-described embodiment, the operation support server 10 is composed of one or more servers different from the remote operation device 20 and the construction machine 40, respectively (see Figure 1 ). However, as another embodiment, the operation support server 10 may also be a component of the remote operation device 20 or the construction machine 40. Each component 121 and component 122 of the operation support server 10 may also be components of two or more devices that can communicate with each other in the remote operation device 20 and the construction machine 40.

[0099] The second auxiliary processing element 122 may also perform the following identification: when an identification image request without an identifier is received based on communication with the first client (the first remote operation device 20) (see Figure 5 / C13), based on the position of the actual machine image Qi in the first operation environment image and the respective actual space positions of the plurality of construction machines 40 stored in the database 102, identify which construction machine 40 among the plurality of construction machines 40 the actual machine image Qi belongs to. Then, the second auxiliary processing element 122 outputs, based on communication with at least one of the first remote operation device and the second remote operation device, i.e., the remote operation device 20, an identification image indicating which construction machine 40 the actual machine image Qi belongs to on the remote output interface 220 of the at least one remote operation device 20.

[0100] According to the operation support server 10 and the like having this configuration, when the construction machine 40 is reflected in the operation environment image output to the output interface of the first remote operation device, identify which remote operation device 20 or which operator's remote operation object the construction machine 40 is. Then, output an identification image indicating the identification result to the output interface of at least one of the first remote operation device and the second remote operation device. As a result, each operator can easily identify which remote operation object the actual machine image output to the output interface of the remote operation device belongs to.

[0101] In the above-described embodiment, based on the actual space position of the construction machine 40 measured by the positioning device 460, it is determined whether the target actual machine is included in the first to third operation environment images. However, as another embodiment, the following processing may also be performed: performing image analysis processing on each operation environment image, and thereby extracting the actual machine image Qi representing the construction machine 40 in the first operation environment image.

[0102] For example, in the case where image analysis processing is performed on the first work environment image, as a result, in the first work environment image, the actual machine images Qi (i = 1, 2,...) representing the work machine 40 are extracted. Figure 7 )

[0103] Next, based on the position of the actual machine image Qi in the first work environment image coordinate system, the actual spatial position of the work machine 40 corresponding to the actual machine image Qi is obtained. At this time, based on the size of the actual machine image Qi in the first work environment coordinates, the actual spatial distance from the imaging device (e.g., imaging device 412) to the work machine 40 is estimated. In the case where the distance to the imaging object obtained by a distance measuring sensor such as a TOF sensor is included as the pixel value of the first work environment image, the actual spatial distance from the imaging device (e.g., imaging device 412) to the work machine 40 can also be estimated based on this pixel value. Moreover, by transforming the position of the work machine 40 in the imaging device coordinate system into the coordinates in the actual space coordinate system, the position of the target actual machine (work machine 40) in the actual space coordinate system is identified or calculated. At the time of this coordinate transformation, the position of the target actual machine in the imaging device coordinate system is identified, and a coordinate transformation factor (matrix or quaternion) representing the position and orientation of the imaging device in the actual space coordinate system is used. The coordinate transformation factor is registered in the database 102 in association with the identifier of the imaging device (and further the image identifier).

[0104] Then, it is determined whether there is an actual machine image Qi representing the work machine 40 in the first work environment image, where the work machine 40 has an actual spatial position that is the same as or corresponding to the actual spatial position of the target actual machine registered in the database 102.

[0105] In the above-described embodiment, the case where the work machine that is the object of the above-described identification image request is one is set. However, the work machines that are the objects of the above-described identification image request can also be plural. For example, the texture image T of the identification image of the first work machine is given red, the texture image T of the identification image of the second work machine is given white, etc., so that multiple operators can share the identification.

[0106] In the above-described embodiment, in the case where a designation notice indicating that there is no actual machine image corresponding to the target actual machine in the first to third work environment images is sent to the remote operation device 20. Figure 5 / Step (STEP) 132), the identification image cannot be set. However, as another embodiment, it can also be set such that even when there is no actual machine image corresponding to the target actual machine in the first to third operation environment images, the identification image can be set. For example, there may be a situation where there are few imaging devices used at the work site, or a situation where the work machine 40 is being transported to the work site by a transporter and has not yet reached the work site. In this case, by presetting the identification image, there is no need to re-perform the operation of identification sharing, so the workability can be improved.

[0107] The second auxiliary processing element 122 performs the following selection and output processing: when the determination result that the actual spatial position of the cooperative work machine is not included in the actual spatial imaging range of the imaging device is negative, the imaging device whose actual spatial position of the cooperative work machine can be included in the actual spatial imaging range is determined as the designated imaging device, and the identification image indicating the information that the cooperative work machine exists at the designated position in the operation environment image obtained by the designated imaging device is output to the respective remote output interfaces 220 of the plurality of remote operation devices 20.

[0108] In this case, when the actual spatial imaging range of one imaging device does not include the actual spatial position of the cooperative work machine, the alternative operation environment image obtained by another imaging device that can include the actual spatial position of the cooperative work machine in the actual spatial imaging range can be output to the remote output interfaces 220 of the respective remote operation devices 20. Therefore, it is possible to achieve identification sharing regarding the existence of the cooperative work machine among multiple operators.

[0109] The second auxiliary processing element 122 determines the actual machine imaging device 412 mounted on at least one of the plurality of work machines 40 as the designated imaging device.

[0110] In this case, when the operation environment image respectively output on the respective remote output interfaces 220 of the plurality of remote operation devices 20 does not include the cooperative work machine that is the target of the identification image request, the operation environment image can be output on the remote output interface 220 of another remote operation device 20 that includes the cooperative work machine. Thus, it is possible to achieve identification sharing regarding the existence of the cooperative work machine that is the target of the identification image request among the multiple operators of the respective plurality of remote operation devices 20.

[0111] When the actual machine imaging device 412 mounted on at least one of the work machines 40 cannot be determined as the designated imaging device, the second auxiliary processing element 122 determines the imaging device that is not mounted on any of the plurality of work machines 40 as the designated imaging device.

[0112] In this case, when the imaging device mounted on at least one work machine is not determined as the designated imaging device, the work environment image obtained by the imaging device (for example, the imaging device installed at the work site) that is not mounted on any of the multiple work machines, i.e., the designated imaging device, can be output to the output interfaces of the respective multiple remote operation devices. Thereby, it is possible to achieve recognition sharing among multiple operators of the existence of the cooperative work machine that is the object of the identification image request.

[0113] When the work environment images output on the remote output interfaces 220 of one remote operation device 20 and the other remote operation device 20 are common, the second auxiliary processing element 122 performs the following processing: outputting the information indicating the other remote operation device 20 to the remote output interface 220 of one remote operation device 20.

[0114] In this case, when the work environment image output to the remote output interface 220 of one remote operation device 20 is also output on the remote output interface 220 of the other remote operation device 20, the information indicating the other remote operation device 20 is output to the remote output interface 220 of this one remote operation device 20.

[0115] When the identification images output on the remote output interfaces 220 of one remote operation device 20 and the other remote operation device 20 indicate the existence of the same cooperative work machine, the second auxiliary processing element 122 performs the following processing: outputting the identification image in a common manner on the remote output interfaces 220 of one remote operation device 20 and the other remote operation device 20.

[0116] In this case, since the identification image indicating the existence of the same cooperative work machine is output in a common manner on the remote output interfaces 220 of the multiple remote operation devices 20, it is possible to achieve recognition sharing among multiple operators of the existence of the cooperative work machine that is the object of the identification image request.

[0117] The second auxiliary processing element 122 performs the following processing: outputting multiple identification images in a mutually recognizable manner on the remote output interfaces 220 of the multiple remote operation devices 20.

[0118] In this case, since multiple identification images are output in a mutually recognizable manner on the remote output interfaces of the multiple remote operation devices 20, it is possible to achieve recognition sharing among multiple operators of the identification and existence of each of the multiple cooperative work machines that are the objects of the identification image request.

[0119] The second auxiliary processing element 122 performs the following processing: Based on the position of the actual machine image corresponding to the working machine in the work environment image and the actual spatial positions of the cooperative working machines stored in the database, it determines whether the actual machine image belongs to the cooperative working machine, and on the condition that the determination result is affirmative, it outputs an identification image indicating the existence of the cooperative working machine on the remote output interfaces 220 of the respective remote operation devices 20.

[0120] In this case, it is possible to let the operator recognize whether the actual machine image included in the work environment image output on the output interface of a remote operation device belongs to the cooperative working machine that is the remote operation object of the remote operation device by whether there is an output marker image.

[0121] The second auxiliary processing element 122 identifies which working machine 40 the actual machine image belongs to based on the position of the actual machine image corresponding to the working machine 40 in the work environment image and the actual spatial positions of the respective working machines 40 stored in the database 102, and based on the communication with each of the remote operation devices of the plurality of remote operation devices 20, it outputs an identification image indicating which working machine 40 the actual machine image belongs to on at least the remote output interface 220 of one remote operation device 20.

[0122] In this case, when the working machine 40 is reflected in the work environment image output on the remote output interface 220 constituting the remote operation device 20, it is possible to identify which remote operation device 20 or which operator's remote operation object the working machine 40 belongs to. Then, the identification image indicating the identification result is output to the output interface of at least one of the first remote operation device and the second remote operation device. As a result, it is possible to let each operator easily recognize which remote operation object of the remote operation device the actual machine image output to the output interface constituting each remote operation device belongs to.

[0123] Symbol Explanation

[0124] 10... Work assistance server; 20... Remote control operation device (client); 40... Working machine; 102... Database; 121... First auxiliary processing element, 122... Second auxiliary processing element; 210... Remote input interface; 220... Remote output interface; 410... Actual machine input interface; 412... Actual machine photographing device; 420... Actual machine output interface; 440... Work attachment (working mechanism).

Claims

1. An operation assistance server for assisting the remote operation of multiple construction machines respectively, wherein, each of the multiple construction machines uses a remote operation device, and the operation assistance server is characterized by comprising: a database in which the identification identifiers of the multiple remote operation devices and the cooperative construction machines that cooperate with the multiple remote operation devices and are the operation objects of the multiple remote operation devices respectively are stored in association with the actual spatial positions of the cooperative construction machines; a first assistance processing element that, based on communication with each of the multiple remote operation devices, causes the remote output interfaces of the multiple remote operation devices that cooperate with the cooperative construction machines respectively to output an operation environment image corresponding to a captured image of the operation site obtained by a capturing device provided in each of the cooperative construction machines; and a second assistance processing element that performs the following processing: when an identification image request accompanied by the identification identifier of the cooperative construction machine is received based on communication with each of the multiple remote operation devices, identifies the actual spatial position of the cooperative construction machine stored in the database in association with the identification identifier, determines whether the actual spatial position of the cooperative construction machine is included in the actual spatial capture range of the capturing device, and when the determination result is affirmative, causes the remote output interfaces of the multiple remote operation devices respectively to output an identification image indicating that the cooperative construction machine exists at a specified position corresponding to the actual spatial position of the cooperative construction machine in the operation environment image and for preventing misidentification with other construction machines.

2. The operation assistance server according to claim 1, wherein, the second assistance processing element performs the following selection output processing: when the determination result as to whether the actual spatial position of the cooperative construction machine is included in the actual spatial capture range of the capturing device is negative, determines a capturing device mounted on another construction machine whose actual spatial position can be included in the actual spatial capture range as a specified capturing device, and outputs the identification image indicating the specified position of the cooperative construction machine in the operation environment image obtained by the specified capturing device to the remote output interfaces of the multiple remote operation devices respectively.

3. The operation assistance server according to claim 2, wherein, when the capturing device mounted on the other construction machine cannot be determined as the specified capturing device, the second assistance processing element determines a capturing device not mounted on any of the multiple construction machines as the specified capturing device.

4. The operation assistance server according to claim 3, wherein, When the work environment images output on the respective remote output interfaces of one remote operation device and other remote operation devices are common, the second auxiliary processing element performs the following processing: causing information indicating the other remote operation device to be output on the remote output interface of the one remote operation device.

5. The work assistance server according to any one of claims 1 to 4, wherein, when the identification images output on the respective remote output interfaces of one remote operation device and other remote operation devices indicate the presence of the same cooperative work machine, the second auxiliary processing element performs the following processing: outputting the identification image on the respective remote output interfaces of the one remote operation device and other remote operation devices in a common manner.

6. The work assistance server according to any one of claims 1 to 4, wherein, the second auxiliary processing element performs the following processing: outputting a plurality of the identification images on the respective remote output interfaces of the plurality of remote operation devices in a mutually recognizable manner.

7. The work assistance server according to any one of claims 1 to 4, wherein, the second auxiliary processing element performs the following processing: based on the position of the actual machine image corresponding to the work machine in the work environment image and the actual spatial position of the cooperative work machine stored in the database, determining whether the actual machine image belongs to the cooperative work machine, and outputting, on the respective remote output interfaces of the plurality of remote operation devices, the identification image indicating the presence of the cooperative work machine on the condition that the determination result is affirmative.

8. The work assistance server according to any one of claims 1 to 4, wherein, the second auxiliary processing element identifies which work machine belongs to the actual machine image based on the position of the actual machine image corresponding to the work machine in the work environment image and the actual spatial positions of the plurality of work machines stored in the database, and based on communication with the respective plurality of remote operation devices, outputs, on the remote output interface of at least one remote operation device, the identification image indicating which work machine belongs to the actual machine image.

9. A work assistance system, wherein, it is composed of the work assistance server according to any one of claims 1 to 8 and the remote operation device.

Images