Job assistance server, job assistance method, and job assistance system

Abstract

The present invention provides a work assistance server, a work assistance method and a work assistance system. When using the work assistance server, etc., it is possible to accumulate experience based on simulated operations while referring to the operation of a work machine by oneself or other operators. A second operator can specify a time series of a work environment image from an archive registered in a database (102) through a remote input interface (210). Thus, the second operator can view the time series of the specified work environment image on a remote output interface (220). The second operator can output a time series of simulated images representing the action mode and environmental change mode of a virtual work site of a work machine to the remote output interface (220) according to the simulated operation mode of a remote operating mechanism (211).

Description

Technical Field

[0001] The present invention relates to a work support server for supporting an operator to improve the operation technique of a work machine based on communication with a client. Background Art

[0002] The following display device for construction machinery has been proposed (for example, refer to Patent Document 1): a display device that prompts an unskilled operator to perform the best operation to promote efficient excavation work. Specifically, a "target traction force value", "actual excavation traction force value relative to the target traction force value", and "lifting force value" in the excavation work that are pre-set in correspondence with the respective operation amounts of a first operating unit and a second operating unit are comparatively displayed, wherein the first operating unit operates the traction force of the vehicle body; and the second operating unit instructs and operates the working machine at the front of the vehicle body. As a result, the operator is prompted to perform operations that improve fuel efficiency and high working efficiency in each stage of the operation (driving of the vehicle, lifting of the working machine).

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2015-040422 Summary of the invention

[0006] Problems to be solved by the invention

[0007] However, from the viewpoint of improving operating skills, it is preferable for an operator with little experience in operating a working machine to accumulate experience through simulated operations while referring to situations in which experienced operators operate the working machine.

[0008] Therefore, an object of the present invention is to provide a server or the like that can accumulate experience based on simulated operations while referring to situations in which the operator himself or another operator operates a working machine.

[0009] Means for solving problems

[0010] The present invention relates to a work support server for supporting an operator to improve the operation technique of a work machine based on communication with a client.

[0011] The work assistance server of the present invention comprises: a first auxiliary processing element, which, based on communication with a first client used by a first operator to operate a first work machine, identifies a time sequence of camera images obtained by a camera device cooperating with the first work machine and representing the movement mode and environmental change mode of the first work machine, and cumulatively stores and retains the time sequence of work environment images corresponding to the camera images in a database; and a second auxiliary processing element, which, based on communication with a second client, outputs a time sequence of designated work environment images specified by an input interface of the second client in the time sequence of the work environment images stored in the database to an output interface of the second client, and outputs a time sequence of simulated images representing the movement mode and environmental change mode of the second work machine in a virtual work site corresponding to a simulated operation input by the second operator through the input interface of the second client at a designated time point in the time sequence of the designated work environment images to the output interface of the second client.

[0012] The work support system of the present invention includes the work support server of the present invention, the first client, and the second client.

[0013] According to the operation assistance server and operation assistance system of the present invention (hereinafter appropriately referred to as "operation assistance server, etc."), the second operator can specify a time series of operation environment images from the time series of operation environment images or archived dynamic images corresponding to the captured images representing the operation mode, movement mode and environmental change mode of the first operation machine performed by the first operator registered in the database through the input interface of the second client. The time series of operation environment images can also be defined, for example, in the following manner: the time series of operation environment images of the first operation machine from the execution start time point to the execution end time point of the specified task constitutes the archive unit. The second operator can view the time series of the specified operation environment images through the output interface of the second client. In addition to being another operator different from the second operator, the first operator can also be the same operator as the second operator, that is, himself.

[0014] The second operator can specify a designated working environment image at a designated time point in the time sequence of designated working environment images through the input interface of the second client, and can perform simulated operation of the second working machine in a virtual working scene formed based on the designated working environment image at the designated time point. The second operator can make the output interface of the second client output a time sequence of simulated images representing the action mode and environmental change mode of the virtual working scene of the second working machine according to the simulated operation mode.

[0015] In this way, the second operator can browse the time series of designated working environment images for reference, and can specify further designated working environment images for reference from the time series of designated working environment images, and perform simulated operation of the second working machine in the virtual working scene reproduced based on the designated working environment images, thereby improving the operating technology of the working machine.

[0016] Brief description of the attached figure

[0017] Figure 1 It is an explanatory diagram about the structure of the work support system which is one embodiment of the present invention.

[0018] Figure 2 This is an explanatory diagram regarding the structure of the remote control device.

[0019] Figure 3 This is an explanatory diagram about the structure of a working machine.

[0020] Figure 4 This is an explanatory diagram regarding the first function of the work support system.

[0021] Figure 5 This is an explanatory diagram regarding the second function of the work support system.

[0022] Figure 6 This is an explanatory diagram regarding the time series of work environment images.

[0023] Figure 7 This is an illustration of the time series of simulated images. DETAILED DESCRIPTION

[0024] (Composition of the work support system)

[0025] Figure 1 The work support system as one embodiment of the present invention shown is composed of a work support server 10 and a plurality of remote operating devices 20 for remotely operating a plurality of work machines 40. The work support server 10, the remote operating device 20 and the work machines 40 are configured to be able to communicate with each other via a network.

[0026] (Structure of the job support server)

[0027] The operation support server 10 includes a database 102, a first auxiliary processing element 121, and a second auxiliary processing element 122. In addition to storing the positions and tracks of the plurality of operation machines 40, the database 102 also stores and maintains captured images, operation environment images, and path guidance images. The database 102 may also be composed of a database server different from the operation support server 10. Each auxiliary processing element is composed of an operation processing device (a single-core processor or a multi-core processor, or a processor core constituting these processors), and reads necessary data and software from a storage device such as a memory, and performs the operation processing described later in accordance with the software on the data.

[0028] (Composition of remote control device)

[0029] The remote operation device 20 constituting the client includes a remote control device 200, a remote input interface 210, and a remote output interface 220. The remote control device 200 is composed of a computing device (a single-core processor or a multi-core processor, or a processor core constituting these processors), and reads necessary data and software from a storage device such as a memory, and performs computing processing in accordance with the software with the data as the object. The remote input interface 210 includes a remote operation mechanism 211. The remote output interface 220 includes an image output device 221 and a remote wireless communication device 222.

[0030] The client may be constituted by a mobile terminal that cooperates with the remote operation device 20 or has a mutual communication function. The mobile terminal may have a communication function for communicating with the work support server 10.

[0031] The remote control mechanism 211 includes a travel control device, a slewing control device, an arm control device, an arm control device, and a bucket control device. Each control device has an operating lever that receives a rotation operation. The operating lever (travel lever) of the travel control device is operated to move the lower travel body 41 of the working machine 40. The travel lever can also serve as a travel pedal. For example, a travel pedal fixed to the base or the lower end of the travel lever can also be provided. The operating lever (slewing lever) of the slewing control device is operated to move the hydraulic slewing motor constituting the slewing mechanism 43 of the working machine 40. The operating lever (boom lever) of the arm control device is operated to move the arm cylinder 442 of the working machine 40. The operating lever (arm lever) of the arm control device is operated to move the arm cylinder 444 of the working machine 40. The operating lever (bucket lever) of the bucket control device is operated to move the bucket cylinder 446 of the working machine 40.

[0032] like Figure 2As shown, the operating levers constituting the remote operating mechanism 211 are arranged around a seat St for the operator to sit on, for example. The seat St may be a high-back chair with armrests, a low-back chair without a headrest, or a chair without a backrest, etc., in any form in which the operator can sit.

[0033] A pair of left and right travel levers 2110 corresponding to the left and right crawlers are arranged side by side in front of the seat St. One operating lever can also serve as multiple operating levers. Figure 2 The right operating lever 2111 in front of the right frame of the seat St shown in the figure functions as a boom lever when operated in the front-rear direction, and functions as a bucket lever when operated in the left-right direction. Figure 2 The left operating lever 2112 provided in front of the left frame of the seat St shown functions as an arm lever when operated in the front-rear direction, and functions as a swing lever when operated in the left-right direction. The lever mode can be arbitrarily changed according to the operator's operation instruction.

[0034] For example, Figure 2 As shown, the image output device 221 is composed of a right oblique front image output device 2211, a front image output device 2212, and a left oblique front image output device 2213, which are respectively arranged at the right oblique front, front, and left oblique front of the seat St. The image output devices 2211 to 2213 may further include a speaker (sound output device).

[0035] (Composition of working machines)

[0036] The working machine 40 includes a real machine control device 400, a real machine input interface 410, a real machine output interface 420, and a working mechanism 440. The real machine control device 400 is composed of an operation processing device (a single-core processor or a multi-core processor, or a processor core constituting the processor), reads necessary data and software from a storage device such as a memory, and performs operation processing in accordance with the software on the data.

[0037] The working machine 40 is, for example, a crawler excavator (construction machine). Figure 2 As shown, it has a crawler-type lower traveling body 41 and an upper revolving body 42 rotatably mounted on the lower traveling body 41 via a revolving mechanism 43. A cab (driver's cab) 424 is provided on the front left side of the upper revolving body 42. A working attachment 44 is provided in the front center of the upper revolving body 42.

[0038] The real machine input interface 410 includes a real machine operating mechanism 411 and a real machine photographing device 412. The real machine operating mechanism 411 includes a plurality of operating levers arranged around a seat arranged inside the cab 424 in the same manner as the remote operating mechanism 211. A driving mechanism or a robot is provided in the cab 424, and the driving mechanism or the robot receives a signal corresponding to the operation mode of the remote operating lever and moves the real machine operating lever based on the received signal. The real machine photographing device 412 is, for example, provided inside the cab 424, and photographs the environment including at least a part of the working mechanism 440 through the front window of the cab 424.

[0039] The real machine output interface 420 includes a real machine wireless communication device 422 .

[0040] The working attachment 44 as a working mechanism includes a boom 441 which is raisably mounted on the upper rotating body 42, an arm 443 which is rotatably connected to the top of the boom 441, and a bucket 445 which is rotatably connected to the top of the arm 443. The working attachment 44 is equipped with a boom cylinder 442, an arm cylinder 444, and a bucket cylinder 446 which are composed of telescopic hydraulic cylinders.

[0041] The boom cylinder 442 is interposed between the boom 441 and the upper slewing body 42, so that it can be extended and retracted by receiving the supply of hydraulic oil, thereby rotating the boom 441 in the lifting direction. The arm cylinder 444 is interposed between the arm 443 and the boom 441, so that it can be extended and retracted by receiving the supply of hydraulic oil, thereby rotating the arm 443 around the horizontal axis relative to the boom 441. The bucket cylinder 446 is interposed between the bucket 445 and the arm 443, so that it can be extended and retracted by receiving the supply of hydraulic oil, thereby rotating the bucket 445 around the horizontal axis relative to the arm 443.

[0042] (Function)

[0043] use Figure 4 and Figure 5 The flowchart shown illustrates the functions of the work assisting system of the above structure. In the flowchart, a box such as "C●" is used for simplicity of description, and the box refers to the sending and / or receiving of data, and refers to a conditional branch for processing the branch direction with the sending and / or receiving of the data as a condition.

[0044] The concept of "recognizing" information by each constituent element (computational processing resource or hardware resource) of the present invention includes: receiving the information; reading or retrieving the information from a storage device, etc.; writing (storing and maintaining it) or registering the information to a storage device, etc.; performing computational processing on the output signal from the sensor and / or the basic information received or retrieved according to a prescribed algorithm, thereby preparing the information in all usable ways for subsequent processing such as inferring, judging, identifying, measuring, and predicting the information.

[0045] (First function (registration of work environment image))

[0046] In the remote operation device 20 as the first client, it is determined whether the operator has performed a designated operation through the remote input interface 210 ( Figure 4 / Step (STEP) 200). The "specified operation" is an operation to select the working machine 40 that cooperates with the remote operating device 20, such as a touch operation such as tapping, sliding, flicking, or pinching to zoom in / pinch to zoom out on the touch panel that constitutes the remote input interface 210. The touch panel displays, for example, a real machine image of the working machine 40 that can be the object of remote operation, and determines whether the display position of the real machine image in the touch panel has been operated. In the case where the specified operation is interrupted, etc., and the determination result is negative ( Figure 4 / Step (STEP) 200 ... No (NO)), a series of processing ends. On the other hand, if the determination result is positive ( Figure 4 / Step 200 ... Yes (YES)), send a work environment image request to the work support server 10 via the remote wireless communication device 222 ( Figure 4 / Step (STEP) 202), the working environment image request is used to request a captured image required for operating the working machine 40 cooperating with the remote operating device 20. The working environment image request includes at least one of an identifier of the remote operating device 20 and an identifier of the operator.

[0047] In the operation support server 10, when receiving the operation environment image request, the first auxiliary processing component 121 sends the operation environment image request to the operation machine 40 corresponding to the operation environment image request ( Figure 4 / C10).

[0048] In the working machine 40, when the working environment image request is received by the real machine wireless communication device 422 ( Figure 4 / C41), the real machine control device 400 obtains the captured image through the real machine capturing device 412 ( Figure 4 / Step (STEP) 402). The real machine control device 400 transmits the captured image data ( Figure 4 / Step (STEP) 404).

[0049] In the work support server 10, when the captured image data is received ( Figure 4 / C11), sends the work environment image data corresponding to the captured image data to the remote operation device 20 (data representing the entire image or a portion of the captured image itself, or a simulated work environment image generated based on the entire image or a portion of the image) ( Figure 4 / Step (STEP) 112). In addition, the time series of the working environment image data is registered or stored in the database 102 ( Figure 4 / Step (STEP) 114).

[0050] In the remote operation device 20, when the work environment image data is received via the remote wireless communication device 222 ( Figure 4 / C20), outputs the working environment image corresponding to the working environment image data to the image output device 221 ( Figure 4 / Step (STEP) 204). Thus, for example, Figure 6 As shown, a working environment image captured by a real machine shooting device 412 mounted on a working machine 40 cooperating with a remote operating device 20 is displayed on an image output device 221, wherein the working environment image includes a part of the working attachment 44 as a working mechanism, namely, a boom 441, an arm 443, a bucket 445 and an arm cylinder 444.

[0051] In the remote operation device 20, the operation mode of the remote operation mechanism 211 is recognized by the remote control device 200 ( Figure 4 / Step (STEP) 206), and a remote operation instruction corresponding to the operation mode is sent to the operation support server 10 through the remote wireless communication device 222 ( Figure 4 / Step (STEP) 208).

[0052] In the operation support server 10, when the remote operation instruction is received, the remote operation instruction is sent to the operation machine 40 through the first auxiliary processing component 121 ( Figure 4 / C12).

[0053] In the working machine 40, when an operation instruction is received by the real machine control device 400 via the real machine wireless communication device 422 ( Figure 4 / C42), controls the operation of the attachment 44, etc. ( Figure 4 / Step (STEP) 406). For example, the bucket 445 scoops up the soil in front of the working machine 40, and after rotating the upper rotating body 42, the soil is dumped from the bucket 445 (see Figure 6 ).like Figure 6 As shown, when the operation is performed, for example, the working environment image (dynamic image or intermittent static image) at the time point t=tk (k=1, 2, 3, 4, 5, 6) is displayed on the image output device 221. The working environment image at t=t1 shows the situation that the working attachment 44 is extended forward and the bucket 445 is pressed to the ground just after the bucket 445 is moved to the excavation start position. The working environment image at t=t2 shows the situation that the operation of making the dipper arm 443 approach the front of the cab 424 is performed. The working environment image at t=t3 shows the situation that the bucket 445 is in a holding posture for holding the soil after the sand and soil have fully entered the bucket 445. The working environment image at t=t4 shows the situation that the left rotation is performed while the operation of lifting the boom 441 is performed. The working environment image at t=t5 shows the situation that the operation of moving the dipper arm 443 away from the cab 424 is performed after the left rotation, and the bucket 445 is located directly above the soil discharge position. The working environment image at t=t6 shows a state where the bucket is in a soil discharge posture and soil is being dumped from the bucket 445 .

[0054] (Second function (analog image output))

[0055] In the remote operation device 20 as the second client, it is determined whether the operator has performed a playback operation through the remote input interface 210 ( Figure 5 / Step (STEP) 210). The "reproduction operation" includes, for example, a touch operation for specifying a work environment image from a library of work environment images registered in the database 102, wherein the database 102 is displayed on a touch panel constituting the remote input interface 210 and the remote output interface 220. The remote input interface 222 can start reproduction, stop reproduction, pause reproduction, fast forward, and rewind the work environment image.

[0056] If the determination result is negative (Figure / Step 210 ... No), the series of processing is terminated. On the other hand, if the determination result is positive (Figure / Step 210 ... Yes), a reproduction request ( Figure 5 / Step (STEP) 212). The reproduction request accompanying the reproduction operation includes an image identifier for identifying the work environment image designated by the operator.

[0057] In the job support server 10, when a reproduction request is received ( Figure 5 / C13), the second auxiliary processing component 122 retrieves the time series of the work environment images to be reproduced from the database 102 as the time series of the designated work environment based on the image identifier included in the reproduction request ( Figure 5 / step 120).

[0058] The second auxiliary processing element 122 sends the time series data representing the designated working environment image to the remote operation device 20. Figure 5 / Step (STEP) 121).

[0059] In the remote operation device 20, when the work environment image data is received via the remote wireless communication device 222 ( Figure 5 / C21), the image output device 221 reproduces or outputs the working environment image corresponding to the working environment image data ( Figure 5 / Step (STEP) 214). Thus, for example, Figure 6 As shown, respective working environment images (dynamic images or intermittent static images) at different time points t=t1, t2, t3, t4, t5 and t6 of a part of the working attachment 44 as the working mechanism, namely, the boom 441, the boom 443, the bucket 445 and the boom cylinder 444 are displayed on the image output device 221 in a time series order.

[0060] The remote control device 200 determines whether or not a playback stop operation has been performed by operating the remote input interface 210. Figure 5 / Step (STEP) 216).

[0061] If the determination result is negative ( Figure 5 / Step 216…No), a reproduction request without reproduction operation is sent to the work support server 10 via the remote wireless communication device 222. As a result, the image output device 221 continuously outputs the time series of the work environment image (refer to Figure 5 / Step 212→→C13→Step 120→Step 121→C21→Step 214).

[0062] On the other hand, if the determination result is positive ( Figure 5 / Step (STEP) 216 ... Yes), the remote control device 200 sends a reproduction stop request ( Figure 5 / Step (STEP) 218). In this case, a reproduction request without a reproduction operation is no longer sent to the work support server 10, and as a result, the reproduction of the time series of the work environment image is stopped in the image output device 221. The reproduction stop request includes the reproduction stop time point of the time series of the work environment image as a designated time point.

[0063] In the job assist server 10, when receiving the reproduction stop request ( Figure 5 / C14), the second auxiliary processing component 122 identifies the simulation status ( Figure 5 / Step 122). For example, Figure 6 The operation mode and environment change mode of the work machine (second work machine) in the virtual work site corresponding to the work environment image at the designated time point t=tk in the time series of the work environment images are identified as the simulation situation.

[0064] Furthermore, the second auxiliary processing element 122 sends the simulated image data ( Figure 5 / Step 123 (STEP 123)).

[0065] In the remote operation device 20, when the remote wireless communication device 222 constituting the remote output interface 220 receives the analog image data ( Figure 5 / C22), the analog image is outputted in the image output device 221 constituting the remote output interface 220 ( Figure 5 / Step (STEP) 220). Thus, the image output device 221 outputs, for example, Figure 7 The simulated image showing the operation mode of the work machine (second work machine) and the environment change mode in the virtual work site at the time point t = τj (j = 1, 2, ...) is shown. At this time, in the image output device 221, the simulated image can be output instead of the work environment image in the stopped state, or the work environment image in the stopped state can be output in a corner of the simulated image.

[0066] The remote control device 20 determines whether a simulated stop operation has been performed via the remote input interface ( Figure 5 / Step (STEP) 222).

[0067] If the determination result is positive ( Figure 5 / Step 222…Yes), the remote control device 200 sends a simulation stop request to the work support server 10 via the wireless communication device 222 ( Figure 5 / Step (STEP) 228).

[0068] On the other hand, if the determination result is negative ( Figure 5 / Step 222…No), the remote control device 200 identifies the operation mode of the remote operating mechanism 211 ( Figure 5 / Step 224), and sending a simulation operation instruction corresponding to the operation mode to the operation auxiliary server 10 through the remote wireless communication device 222 ( Figure 5 / Step 226).

[0069] In the work support server 10, when receiving the simulation operation instruction ( Figure 4 / C14), the second auxiliary processing element 122 identifies the simulation status corresponding to the simulation operation instruction ( Figure 5 / STEP122). For example, when the remote control mechanism 211 is operated to lift the boom 441 relative to the upper rotating body 42 to lift the entire working attachment 44, the working attachment 44 recognizes the working condition in this manner as a simulation condition in the simulation.

[0070] The second auxiliary processing element 122 sends data representing the simulated image to the remote operation device 20 ( Figure 5 / Step (STEP) 123). Then, the second auxiliary processing element 122 determines whether there is a simulation stop request ( Figure 5 / Step (STEP) 124). If the determination result is positive ( Figure 5 / Step 124 ... Yes), a series of processing ends. In the case where the determination result is negative ( Figure 5 / Step 124…No), repeatedly execute the processing after receiving the simulation operation instruction (refer to Figure 5 / C14→STEP122→STEP123).

[0071] In the remote operation device 20, when analog image data is received via the remote wireless communication device 222 constituting the remote output interface 220 ( Figure 5 / C22), outputs the analog image to the image output device 221 constituting the remote output interface 220 ( Figure 5 / Step (STEP) 220). Thus, for example, an image output device 221 outputs, for example, Figure 7 The simulated image at the time point t=τj (j=1, 2, ...) is shown.

[0072] The working environment image at t=τ1 shows, in a simulated image, a situation in which the working attachment 44 is extended forward and the bucket 445 is pressed to the ground just after the bucket 445 is moved to the excavation start position. The working environment image at t=τ2 shows, in a simulated image, a situation in which the boom 443 is moved close to the front of the cab. The working environment image at t=τ3 shows, in a simulated image, a situation in which the bucket 445 is put in a holding posture to hold the soil after the sand and soil have fully entered the bucket 445. The working environment image at t=τ4 shows, in a simulated image, a situation in which the boom 441 is lifted while being turned left. The working environment image at t=τ5 shows, in a simulated image, a situation in which the boom 443 is moved away from the cab after being turned left and the bucket 445 is positioned directly above the soil discharge position. The working environment image at t=τ6 shows, in a simulated image, a situation in which the bucket is in a soil discharge posture and soil is dumped from the bucket 445.

[0073] (Effect)

[0074] According to the work support system having the above-mentioned structure and the work support server 10 constituting the work support system, the time series of the work environment images representing the operation mode and the environment change mode of the first work machine are registered in the database 102 (refer to Figure 4 / Step (STEP) 114), wherein the first operator remotely operates the first working machine through the remote operating device 20 constituting the first client. As a result, a time series archive of various working environment images is cumulatively stored in the database 102. The "first working machine" refers to the working machine 40 that is remotely operated by the first operator and performs work at the actual working site.

[0075] The second operator of the remote operation device 20 constituting the second client can specify a time series of work environment images from the archive registered in the database 102 through the remote input interface 210 (see Figure 5 / Step 210). Thus, the second operator can view the time series of the designated work environment images in the remote output interface 220 (see Figure 5 / Step 220 and Figure 7 ).

[0076] The second operator can perform simulated operation of the second working machine in a virtual working site based on a designated working environment image through the remote operating mechanism 211 constituting the remote input interface 210 (see Figure 3). The "second working machine" refers to a working machine that is simulated and operated by the second operator and performs actions at the virtual working site. The second operator can output a time series of simulated images showing the action mode of the working machine at the virtual working site and the environmental change mode in the remote output interface 220 according to the simulated operation mode (refer to Figure 5 / STEP224→STEP226→C14→STEP122→STEP123→STEP124→C22→STEP220 and Figure 7 ).

[0077] In this way, the second operator can improve his operating skills of the working machinery by performing the following operations: viewing the time series of working environment images for reference, and further specifying a working environment image for reference from the time series of working environment images, and performing simulated operation of the working machinery in the virtual working site reproduced based on the specified working environment image.

[0078] (Other embodiments of the present invention)

[0079] In the above-described embodiment, the work support server 10 is composed of one or more servers that are independent of the remote operation device 20 and the work machine 40 (see Figure 1 ). However, as another embodiment, the operation support server 10 may also be a component of the remote operation device 20 or the operation machine 40. The components 121 and 122 of the operation support server 10 may also be components of two or more devices in the remote operation device 20 and the operation machine 40 that can communicate with each other.

[0080] The second auxiliary processing element 122 may also overlap the time series of the simulation image with the designated time series after the designated time point in the time series of the designated working environment image based on the communication with the remote operating device 20 constituting the second client, and output it to the remote output interface 220 constituting the remote operating device 20. The second operator can overlap the simulation image representing the action mode and environmental change mode of the virtual working site of the second working machine with the designated time series of the working environment image after the designated time point and output it to the remote output interface 220 according to the simulation operation mode through the remote operating mechanism 211.

[0081] Therefore, the second operator can recognize the difference between the action mode of the first working machine to be used as a reference, or the action mode and environmental change mode of the first operator, and the action mode of the second working machine, or the simulated action mode of the second operator, and the environmental change mode of the virtual working site, thereby further improving the operating technology.

[0082] The second auxiliary processing element 122 may also evaluate the difference between the action mode of the first working machine represented by the specified time series of the specified working environment image and the action mode of the second working machine represented by the time series of the simulation image. For example, the interval between the specified parts (e.g., bucket 445) of the working attachment 44 of the first working machine and the second working machine may be converted into the actual spatial distance, and the cumulative value or time average value of the conversion result may be used as the difference for evaluation.

[0083] Furthermore, when the difference exceeds a threshold value, the second auxiliary processing element 122 may output information to the remote output interface 220 to confirm whether it is necessary to perform the simulated operation of the second working machine again.

[0084] Thus, when the difference between the action mode of the first working machine or the action mode and environmental change mode of the first operator and the action mode of the second working machine or the simulated action mode and environmental change mode of the second operator at the virtual working site exceeds a threshold, the need to redo the operation is confirmed through the remote input interface 210, thereby further improving the operation technique of the second operator. In addition, when it is confirmed that the operation needs to be redoed, the reproduction of the specified time series of the specified working environment image can also be output on the remote output interface 220. Thus, the second operator can redo the simulated operation, thereby further improving the operation technique.

[0085] The second auxiliary processing element 122 can also be based on the communication with the remote operating device 20 constituting the second client, so that the operating state of the remote operating mechanism 211 constituting the second client at the start time of the simulated operation of the second working machine (for example, the inclination angle or position of each rod 2110, 2111, 2112) matches the operating state of the remote operating mechanism 211 at a specified time point of the remote operating device 20 constituting the first client.

[0086] Thus, the second operator can start the simulated operation of the second working machine from the following operation state as the starting point, wherein the operation state is the operation state of the remote operation mechanism 211 of the remote operation device 20 constituting the second client that matches the operation state of the remote operation mechanism 211 of the remote operation device 20 constituting the first client by the first operator. Therefore, it is easier for the second operator to learn the operation technology of the first operator, and the operation technology of the second operator is further improved.

[0087] Explanation of symbols

[0088] 10…operation auxiliary server, 20…remote operation device (first client, second client), 40…operation machinery, 102…database, 121…first auxiliary processing element, 122…second auxiliary processing element, 210…remote input interface, 220…remote output interface, 410…real machine input interface, 412…real machine shooting device, 420…real machine output interface, 440…operation attachment (working mechanism).

Claims

1. A work support server for assisting an operator to improve the operation technique of a work machine based on communication with a client. The operation support server is characterized in that it has: A first auxiliary processing element, which identifies a time series of camera images obtained by a camera device cooperating with the first working machine and indicating a mode of movement and a mode of environmental change of the first working machine based on communication with a first client used by a first operator to operate the first working machine, and cumulatively stores and retains the time series of working environment images corresponding to the camera images in a database; and, A second auxiliary processing element, based on communication with a second client, outputs a time sequence of one of the working environment images stored in the database and specified by the input interface of the second client as a time sequence of specified working environment images to the output interface of the second client, identifies the action mode and environmental change mode of a second working machine having a working mechanism configured in a virtual working site as a simulated condition based on the specified working environment image at the specified time point specified by the input interface of the second client in the time sequence of the specified working environment images, identifies the simulated condition corresponding to the simulated operation mode of the second operating mechanism possessed by the second operator through the second client in the virtual working site, and outputs a time sequence of simulated images representing the working situation of the second working machine including the working situation of the working mechanism to the output interface of the second client according to the simulated operation mode representing the simulated condition, wherein the second client is used to accept the operation of the second operator, the second operator is used to operate the second working machine, and the second working machine includes an operation for operating the working mechanism, The second auxiliary processing element overlaps the time series of the simulation images with a designated time series after the designated time point in the time series of the designated working environment images, and outputs the overlapped images to the output interface of the second client.

2. The operation support server according to claim 1, characterized in that: When the difference between the action mode of the first working machine represented by the specified time series of the specified working environment image and the action mode of the second working machine represented by the time series of the simulation image exceeds a threshold, the second auxiliary processing element will output information confirming whether it is necessary to re-perform the simulation operation of the second working machine to the output interface of the second client.

3. The operation support server according to claim 1 or 2, characterized in that: The first auxiliary processing element identifies a time sequence of an operation state of a first operating mechanism constituting the first client based on communication with the first client, The second auxiliary processing element matches the operating state of the second operating mechanism constituting the second client at the start time of the simulated operation of the second working machine with the operating state of the first operating mechanism at a time point corresponding to one of the specified working environment images based on the communication with the second client.

4. A work assistance system, characterized in that: The work support server comprises the work support server according to any one of claims 1 to 3, the first client, and the second client.

5. A work assisting method for assisting an operator to improve the operation technique of a work machine based on communication with a client. The operation assisting method is characterized in that the following assisting processes are performed: A first auxiliary process, based on communication with a first client used by a first operator to operate a first working machine, identifies a time series of camera images obtained by a camera device cooperating with the first working machine and indicating a mode of movement and a mode of environmental change of the first working machine, and cumulatively stores and retains the time series of working environment images corresponding to the camera images in a database; and, The second auxiliary processing is based on the communication with the second client, so that the time sequence of the working environment images stored in the database and the time sequence of one of the working environment images specified by the input interface of the second client are output as the time sequence of the designated working environment images to the output interface of the second client, and based on the designated working environment image at the designated time point specified by the input interface of the second client in the time sequence of the designated working environment images, the action mode and environmental change mode of the second working machine having a working mechanism configured in the virtual working site are identified as simulation conditions, and the simulation condition corresponding to the simulation operation mode of the second operating mechanism possessed by the operator through the second client in the virtual working site is identified, and according to the simulation operation mode representing the simulation condition, the time sequence of the simulation image representing the working situation of the second working machine including the working situation of the working mechanism is output to the output interface of the second client, the second client is used to accept the operation of the operator, the operator is used to make the second working machine work, and the second working machine includes the operation for making the working mechanism work, In the second auxiliary processing, the time series of the simulated images is made to coincide with the time series of the specified operation. The designated time sequence after the designated time point in the time sequence of the industrial environment image overlaps, And output to the output interface of the second client.

Images