Job support server, job support method, and job support system

Abstract

The present invention provides a server and a system that enable multiple operation-related personnel to share information related to objects that require attention in an operation area. The output interface (e.g., a remote output interface (220)) of a second client (e.g., a remote operation device (20)) outputs an operation environment image, which shows a portion of a captured image acquired by a camera (e.g., a camera (612)) of a first client (e.g., an operator terminal (60)), namely, an extended form of an object in an object image area (R) in actual space and object-related information.

Description

Technical Field

[0001] The present invention relates to a work support server for supporting sharing of information related to a work site among a plurality of workers by communicating with a plurality of clients respectively assigned to the plurality of workers. Background Art

[0002] A terminal device for a remote monitoring support system is proposed, which is used to realize information sharing with sufficient accuracy between workers who are performing maintenance in a factory and people waiting outside the work site (for example, refer to Patent Document 1). The terminal device includes: an image input unit for inputting image data of the site; an input operation selection unit, which is a pen or mouse type; a detection unit, which detects whether a new image is acquired; a communication control unit, which transmits and receives data with the outside via wireless; and an input / output screen display unit, which displays an input screen for inputting specified data.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2005-242830 Summary of the invention

[0006] Technical problem to be solved by the invention

[0007] However, it is preferable that information on objects requiring attention in a work area in which a plurality of work-related personnel are involved can be shared among a plurality of work-related personnel.

[0008] Therefore, an object of the present invention is to provide a server and a system capable of sharing information on objects requiring attention in a work area with a plurality of people involved in the work.

[0009] Solutions for solving the above technical problems

[0010] The present invention relates to a work support server for supporting sharing of information related to a work site among a plurality of workers by communicating with a plurality of clients respectively assigned to the plurality of workers.

[0011] The work support server of the present invention comprises: a first support processing element, which identifies, based on communication with the first client among the multiple clients, the existence of an object in an object object image area, which is a part of a captured image acquired by a camera, and object object related information related to the object, and the actual spatial position and actual spatial posture of the camera, and estimates the extended form of the object in the actual space based on the actual spatial position and actual spatial posture of the camera; and a second support processing element, which causes the output interface of the second client to output a work environment image based on communication with the second client among the multiple clients, and the work environment image shows the extended form of the object and the object object related information estimated by the first support processing element.

[0012] The work support system of the present invention is composed of the work support server of the present invention and the plurality of clients.

[0013] According to the job support server and the job support system of the present invention (hereinafter appropriately referred to as "job support server, etc."), a job environment image is output to the output interface of the second client, and the job environment image shows a part of the captured image obtained by the shooting device of the first client, that is, the extended form of the object object in the actual space and the object object related information in the object object image area.

[0014] Therefore, for example, when each operator recognizes that there is an object around him, he can use the client as the first client to quickly obtain a captured image of the object. Then, other operators can use the client as the second client to quickly recognize the extended form of the object in the actual space and the object-related information through the working environment image output by the output interface. Furthermore, for example, since multiple operators located at a common site each use the client as the first client, a map rich in information related to various objects can be shared among the multiple operators. Thus, for example, when an operator uses a working machine to perform a work, he can smoothly perform the work while being aware of the extended form of the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is an explanatory diagram about the structure of the work support system which is one embodiment of the present invention.

[0016] Figure 2 This is an explanatory diagram regarding the structure of the remote control device.

[0017] Figure 3 This is an explanatory diagram regarding the structure of a working machine.

[0018] Figure 4This is an explanatory diagram regarding the first function of the work support system.

[0019] Figure 5 This is an explanatory diagram regarding the second function of the work support system.

[0020] Figure 6 This is an explanatory diagram regarding a captured image and an image area of ​​a target object.

[0021] Figure 7 This is an explanatory diagram regarding the first environment image.

[0022] Figure 8 This is an illustration of a work environment image.

[0023] Fig. 9 This is an explanatory diagram regarding the second environment image. DETAILED DESCRIPTION

[0024] (Composition of the work support system)

[0025] Figure 1 The work support system shown as one embodiment of the present invention is composed of a work support server 10, a remote operation device 20 for remotely operating a work machine 40, and an operator terminal 60. "Multiple clients" may be composed of one or more remote operation devices 20 and one or more operator terminals 60, or may be composed of multiple remote operation devices 20 or multiple operator terminals 60. The work support server 10, the remote operation device 20, the work machine 40, and the operator terminal 60 are configured to be able to communicate with each other via a network.

[0026] (Configuration of the work support server)

[0027] The operation support server 10 includes a database 102, a first support processing element 121, and a second support processing element 122. The database 102 stores and maintains "captured images", "actual spatial positions and actual spatial postures of the camera 612", "extended forms of the target object image area in the captured images", and "target object related information related to the target object existing in the target object image area". The database 102 can be composed of a database server independent of the operation support server 10. Each support processing element is composed of an operation processing device (a single-core processor or a multi-core processor or a processor core constituting the processor), reads necessary data and software from a storage device such as a memory, and performs the operation processing described later according to the software on the data.

[0028] (Composition of remote control device)

[0029] The remote operation device 20 constituting one client includes a remote control device 200, a remote input interface 210, and a remote output interface 220. The remote control device 200 is composed of a computing device (a single-core processor or a multi-core processor or a processor core constituting the processor), reads necessary data and software from a storage device such as a memory, and performs computing processing based on the software with the data as the object. The remote input interface 210 includes a remote operation mechanism 211. The remote output interface 220 includes an image output device 221 and a remote wireless communication device 222.

[0030] This one client may be constituted by a mobile terminal that cooperates with the remote operation device 20 or has a mutual communication function. This mobile terminal has the same configuration as the operator terminal 60 described later.

[0031] The remote control mechanism 211 includes a travel control device, a slewing control device, an arm control device, an arm control device, and a bucket control device. Each control device has an operating lever that receives a rotation operation. The operating lever (travel lever) of the travel control device is operated to move the lower travel body 410 of the working machine 40. The travel lever may also serve as a travel pedal. For example, a travel pedal fixed to the base or the lower end of the travel lever may also be provided. The operating lever (slewing lever) of the slewing control device is operated to move the hydraulic slewing motor of the slewing mechanism 430 constituting the working machine 40. The operating lever (boom lever) of the arm control device is operated to move the arm cylinder 442 of the working machine 40. The operating lever (arm lever) of the arm control device is operated to move the arm cylinder 444 of the working machine 40. The operating lever (bucket lever) of the bucket control device is operated to move the bucket cylinder 446 of the working machine 40.

[0032] Each operating lever constituting the remote operating mechanism 211 is, for example, Figure 2 As shown, the seat St is arranged around the operator seat. The seat St may be a high-back chair with armrests, a low-back chair without a headrest, or a chair without a backrest, etc., in any form in which the remote operator OP2 can sit.

[0033] A pair of left and right travel levers 2110 corresponding to the left and right crawlers are arranged side by side in front of the seat St. One operating lever can also serve as multiple operating levers. Figure 3 The right operating lever 2111 provided in front of the right frame of the seat St shown in the figure functions as a boom lever when operated in the front-rear direction, and functions as a bucket lever when operated in the left-right direction. Figure 2The left operating lever 2112 provided in front of the left frame of the seat St functions as an arm lever when operated in the front-rear direction and functions as a swing lever when operated in the left-right direction. The lever mode can be arbitrarily changed according to the operator's operation instruction.

[0034] For example, Figure 2 As shown, the image output device 221 is composed of a right oblique front image output device 2211, a front image output device 2212, and a left oblique front image output device 2213, which are respectively arranged at the right oblique front, front, and left oblique front of the seat St. The image output devices 2211 to 2213 may further include a speaker (sound output device).

[0035] (Composition of working machines)

[0036] The working machine 40 includes a real machine control device 400, a real machine input interface 410, a real machine output interface 420, and a working mechanism 440. The real machine control device 400 is composed of an operation processing device (a single-core processor or a multi-core processor or a processor core constituting the processor), reads necessary data and software from a storage device such as a memory, and performs operation processing based on the software on the data.

[0037] The working machine 40 is, for example, a crawler excavator (construction machine). Figure 3 As shown, it has a crawler-type lower traveling body 410 and an upper revolving body 420 rotatably mounted on the lower traveling body 410 via a revolving mechanism 430. A cab (operator's cab) 424 is provided on the front left side of the upper revolving body 420. A working attachment 440 is provided in the front center of the upper revolving body 420.

[0038] The real machine input interface 410 includes a real machine operating mechanism 411 and a real machine photographing device 412. The real machine operating mechanism 411 includes a plurality of operating levers arranged around a seat arranged inside the cab 424, similarly to the remote operating mechanism 211. A driving mechanism or a robot is provided in the cab 424, which receives a signal corresponding to the operation form of the remote operating lever and actuates the real machine operating lever based on the received signal. The real machine photographing device 412 is, for example, provided inside the cab 424, and photographs an environment including at least a part of the working mechanism 440 through the front window of the cab 424.

[0039] The real device output interface 420 includes a real device wireless communication device 422 .

[0040] The working attachment 440 as a working mechanism includes a boom 441 which is raisably mounted on the upper rotating body 420, an arm 443 which is rotatably connected to the front end of the boom 441, and a bucket 445 which is rotatably connected to the front end of the arm 443. The working attachment 440 is equipped with a boom cylinder 442, an arm cylinder 444, and a bucket cylinder 446 which are composed of telescopic hydraulic cylinders.

[0041] The boom cylinder 442 is interposed between the boom 441 and the upper slewing body 420, so that it can be extended and retracted by receiving the supply of hydraulic oil, thereby rotating the boom 441 in the lifting direction. The arm cylinder 444 is interposed between the arm 443 and the boom 441, so that it can be extended and retracted by receiving the supply of hydraulic oil, thereby rotating the arm 443 around the horizontal axis relative to the boom 441. The bucket cylinder 446 is interposed between the bucket 445 and the arm 443, so that it can be extended and retracted by receiving the supply of hydraulic oil, thereby rotating the bucket 445 around the horizontal axis relative to the arm 443.

[0042] (Construction of operator terminal)

[0043] The operator terminal 60 constituting other clients is a terminal device such as a smartphone or a tablet terminal, and includes a control device 600, an input interface 610, and an output interface 620. The control device 600 is composed of an operation processing device (a single-core processor or a multi-core processor or a processor core constituting the processor), reads necessary data and software from a storage device such as a memory, and performs operation processing based on the software on the data.

[0044] The input interface 610 includes a camera 612. The input interface 610 is composed of touch panel buttons and switches, etc. The output interface 620 includes an image output device 621 (and a sound output device if necessary) and a wireless communication device 622.

[0045] (Function)

[0046] (First function (output of work environment image))

[0047] use Figure 4 as well as Figure 5 The flowchart shown in the figure illustrates the functions of the work support system of the above structure. In the flowchart, for the sake of simplicity, a box such as "C·" is used to refer to the sending and / or receiving of data, and refers to a conditional branch that executes the processing in the branch direction with the sending and / or receiving of data as a condition. Figure 4 , a flowchart is shown in which the operator terminal 60 constitutes the "first client" and the remote operation device 20 constitutes the "second client". Conversely, the operator terminal 60 may constitute the "second client" and the remote operation device 20 may constitute the "first client".

[0048] The concept of "recognizing" information by each constituent element (computational processing resource or hardware resource) of the present invention includes: receiving the information; reading or retrieving the information from a storage device, etc.; writing (storing and maintaining it) or registering the information to a storage device, etc.; performing computational processing on the output signal from the sensor and / or the basic information received or retrieved according to a prescribed algorithm, thereby preparing the information in all usable ways for subsequent processing such as inference, judgment, identification, measurement, and prediction of the information.

[0049] In the remote operation device 20, the operator OP determines whether the first designated operation ( Figure 4 / Step 202). For example, the first designated operation is an operation such as a tap on a predetermined portion of the remote input interface 210. In the case where the determination result is positive ( Figure 4 / Step 202‥Yes), a first environment confirmation request is sent to the operation support server 10 via the remote wireless communication device 222 ( Figure 4 / Step 204).

[0050] In the operation support server 10, when the first environment confirmation request is received ( Figure 4 / C10), the first support processing element 121 sends the first environment image data ( Figure 4 / Step 102).

[0051] In the remote control device 20, when the first environment image data is received via the remote wireless communication device 222 ( Figure 4 / C20), outputs the first environment image corresponding to the first environment image data to the image output device 221 ( Figure 4 / Step 206). Thus, for example, Figure 7 As shown, an overhead map or an overhead shot image showing the overall situation of the work site is output in the image output device 221 constituting the remote output interface 220. The position in the overhead map can be determined by latitude and longitude. The overhead shot image includes an image or icon Q1 showing the location of the first working machine 40, an image or icon Q2 showing the location of the second working machine 40, and an image OP1 showing the location of the first operator. The overhead shot image can be obtained, for example, by a shooting device mounted on an unmanned aerial vehicle or a shooting device installed on a structure such as a pole at the work site. The shooting position and viewing angle of the shot image serving as the first environmental image can be arbitrarily changed. An overhead map can also be generated based on the overhead shot image.

[0052] In the operator terminal 60, the first operator performs a photographing operation through the input interface 610, and in response, a photographed image is acquired by the photographing device 612 ( Figure 4 / Step 602). In addition, the captured image is output to the image output device 621 constituting the output interface 620 ( Figure 4 / Step 604). Thus, for example, Figure 6 As shown, a captured image is output to the image output device 621, and the captured image shows the situation at the work site where the trace of the work machine 40 has become a depression. In the case where the remote operation device 20 corresponds to the "first client", a captured image (second captured image) showing the surrounding situation of the work machine 40 obtained by the real machine capturing device 412 mounted on the work machine 40 is output to the image output device 221 constituting the remote output interface 220 (refer to Fig. 9 ).

[0053] The actual spatial position and actual spatial posture ( Figure 4 / Step 606). The actual spatial position (latitude and longitude) of the camera 612 is measured based on the output signal of a positioning sensor such as a GPS. The actual spatial posture of the camera 612 is measured based on the output signal of a three-axis orientation sensor or an acceleration sensor.

[0054] The control device 600 determines whether the target object image area R ( Figure 4 / Step 608). For example, in the operator terminal 60, the extension form or outline of the target object image area can be identified by identifying the trajectory of the first operator's fingertip or pen on the touch screens constituting both the input interface 610 and the output interface 620. Figure 6 As shown, the first operator can specify a substantially stage-shaped target object image region R in the captured image. When the remote operation device 20 corresponds to the "first client", the second environment image (refer to FIG. 1 ) outputted from the output interface 620 as a captured image acquired by the real machine capturing device 412 mounted on the working machine 40 Fig. 9 ), the target object image region R and the input target object related information can be specified by operating the input interface 610 (refer to Figure 4 / Step 606).

[0055] If the determination result is positive ( Figure 4 / Step 608‥Yes), the control device 600 determines whether the target object related information ( Figure 4 / Step 612). For example, the object-related information can be identified by identifying the contact form of the first operator's fingertip or pen in the touch keyboard constituting both the input interface 610 and the output interface 620. Thus, information indicating the type or property of the object such as "gravel", "rubble", "building material", "sand", "hole", "depression", "puddle" or "rock" can be identified as the object-related information.

[0056] If the determination result is positive ( Figure 4 / Step 612‥Yes), data representing "captured image", "actual spatial position and actual spatial posture of the camera 612", "extended form of the object image region R in the captured image", and "object-related information related to the object existing in the object image region" are sent to the operation support server 10 through the wireless communication device 622 constituting the output interface 602. In the case where the remote operation device 20 corresponds to the "first client", data representing "captured image", "actual spatial position and actual spatial posture of the camera 412", "extended form of the object image region R in the captured image", and "object-related information related to the object existing in the object image region" are sent to the operation support server 10 through the remote wireless communication device 222 constituting the remote output interface 220.

[0057] In the case where the work support server 10 receives data indicating “captured image data”, “actual spatial position and actual spatial posture of the imaging device 612”, “extended form of the target object image region R”, and “target object related information” respectively ( Figure 4 / C13), the first support processing element 121 estimates the extended shape of the target object in the actual space ( Figure 4 / Step 108). For example, it is assumed that the height of the imaging device 612 based on the ground that the operator is in contact with is at a predetermined height ranging from the standard waist position to the head position of a person standing or sitting on the ground. In addition, it is assumed that the object exists in a horizontal plane (two-dimensional space) at the same height as the ground. That is, the object image area R specified in the captured image is assumed to have a width corresponding to the viewing angle of the imaging device 612 and to extend two-dimensionally in the area projected on the horizontal plane.

[0058] On the basis of these assumptions, in addition to the extension form of the image area R of the object object in the captured image coordinate system, the extension form of the object object (the expansion method in the latitude direction and longitude direction respectively) is estimated based on the actual spatial position and actual spatial posture of the shooting device 612, and the viewing angle or focal distance. In the case where the object object is a structure located higher than the ground or a depression sunken than the ground, although the estimation accuracy based on this assumption becomes lower, the approximate extension form of the object object in the actual space can be grasped.

[0059] A captured image (or a distance image as a captured image) to which the distance to the target object measured by the distance measuring sensor (or the distance measuring sensor as the imaging device 612) mounted on the operator terminal 60 is assigned as a pixel value can be transmitted from the operator terminal 60 to the operation support server 10. In this case, the extended form of the target object or its surface in the actual space based on the actual spatial position of the imaging device 612 can be estimated with higher accuracy.

[0060] Next, the second support processing element 122 generates a work environment image based on the first environment image, and the work environment image shows the estimation result of the extended form of the object in the actual space performed by the first support processing element 121 and the object related information ( Figure 4 / Step 110). On this basis, the second support processing component 122 sends the work environment image data to the remote operation device 20 ( Figure 4 / Step 112).

[0061] In the remote operation device 20, when the work environment image data is received via the remote wireless communication device 222 constituting the remote output interface 220 ( Figure 4 / C21), outputting the working environment image in the image output device 221 constituting the remote output interface 220 ( Figure 4 / Step 208). Thus, for example, Figure 8 As shown, the first working environment image is output to the image output device constituting the remote output interface 220 (see Figure 7 ) represents an image of an extended form of an object image region R in a real space of the object and object related information such as “existence of a depression” associated with the object image region R.

[0062] In the operator terminal 60, when the control device 600 determines that the target object related information has not been input through the operation of the first operator in the input interface 610 ( Figure 4 / Step 612‥No), data representing "captured image", "actual spatial position and actual spatial posture of the camera 612", and "extended form of the target object image area R in the captured image" are sent to the operation support server 10 through the wireless communication device 622 constituting the output interface 620 ( Figure 4 / Step 614).

[0063] In the case where the work support server 10 receives data indicating "captured image data", "actual spatial position and actual spatial posture of the imaging device 612", and "extended form of the target object image region R", respectively ( Figure 4 / C12), the first support processing element 121 determines whether the target object related information can be recognized ( Figure 4 / Step 106). For example, by texture analysis of the target object image region R, the type or property of the object existing in the target object image region R can be identified as the target object related information. Figure 4 / Step 106‥Yes), performing estimation of the extended shape of the target object in the actual space ( Figure 4 / Step 108) and subsequent processing. If the determination result is negative ( Figure 4 / Step 106‥No), the work environment image data is not sent to the remote operation device 20, and the work environment image is not output in the remote operation device 20 (refer to Figure 8 ), and ends Figure 4 A series of processing shown.

[0064] In the operator terminal 60, when the control device 600 determines that the target object image region R is not specified by the first operator's operation in the input interface 610 ( Figure 4 / Step 608‥No), the wireless communication device 622 constituting the output interface 620 sends data indicating the "captured image" and the "actual spatial position and actual spatial posture of the camera 612" to the work support server 10 ( Figure 4 / Step 610).

[0065] When the work support server 10 receives data indicating the "captured image" and the "actual spatial position and actual spatial posture of the imaging device 612" respectively ( Figure 4 / C11), through the first support processing element 121, it is determined whether the target object image area can be recognized ( Figure 4 / Step 104). For example, a portion of the captured image can be identified as the target object image region by image analysis processing such as pattern matching with the captured image as the target. In the case where the determination result is positive ( Figure 4 / Step 104‥Yes), executing a determination as to whether the target object related information can be recognized ( Figure 4 / Step 106) and subsequent processing. If the determination result is negative ( Figure 4 / Step 104‥No), the work environment image data is not sent to the remote operation device 20, and the work environment image is not output in the remote operation device 20 (refer to Figure 8 ), and ends Figure 4 A series of processing shown.

[0066] (Second function (remote operation of working machinery))

[0067] In the remote operation device 20, the operator OP determines whether the second designated operation ( Figure 5 / Step 220). For example, in the working environment image (refer to Figure 7 ) in which the "second designated operation" is an operation such as a tap of an image Q1 or Q2 of the remote input interface 210 used by the operator to designate the work machine 40 that the operator wants to remotely operate. The second designated operation and the first designated operation may be operations of different forms or operations of the same form. In the case where the determination result is negative ( Figure 5 / Step 220‥No), the series of processing ends. On the other hand, if the determination result is positive ( Figure 5 / Step 220‥Yes), send a second environment confirmation request to the operation support server 10 via the remote wireless communication device 222 ( Figure 5 / Step 222).

[0068] In the operation support server 10, when the second environment confirmation request is received, the first support processing component 121 sends the second environment confirmation request to the operation machine 40 ( Figure 5 / C14).

[0069] In the working machine 40, when an environment confirmation request is received via the real machine wireless communication device 422 ( Figure 5 / C41), the real machine control device 400 obtains the captured image through the real machine shooting device 412 ( Figure 5 / Step 402). The real machine control device 400 transmits the captured image data ( Figure 5 / Step 404).

[0070] In the work support server 10, when the captured image data is received ( Figure 5 / C15), sends the second environment image data corresponding to the captured image data to the remote control device 20 (data representing all or part of the captured image itself or a simulated environment image generated based on it) ( Figure 5 / Step 112).

[0071] In the remote control device 20, when the second environment image data is received via the remote wireless communication device 222 ( Figure 5 / C23), outputs the second environment image corresponding to the second environment image data to the image output device 221 ( Figure 5 / Step 224). Thus, for example, Fig. 9 As shown, an environment image including a boom 441 , an arm 443 , a bucket 445 , and an arm cylinder 444 , which are part of a working attachment 440 as a working mechanism, is displayed on the image output device 221 .

[0072] In the remote operation device 20, the operation state of the remote operation mechanism 211 is recognized by the remote control device 200 ( Figure 5 / Step 226), and transmits a remote operation instruction corresponding to the operation form to the operation support server 10 via the remote wireless communication device 222 ( Figure 5 / Step 228).

[0073] In the operation support server 10, when the remote operation command is received, the remote operation command is sent to the operation machine 40 via the first support processing component 121 ( Figure 5 / C16).

[0074] In the working machine 40, when an operation instruction is received through the real machine wireless communication device 422 ( Figure 5 / C42), the actual machine control device 400 controls the operation of the work attachment 440, etc. ( Figure 5 / Step 406) For example, an operation is performed in which the soil in front of the working machine 40 is scooped up by the bucket 445 and the soil is dumped from the bucket 445 after the upper rotating body 420 is rotated.

[0075] (Effect)

[0076] According to the thus constructed work support system and the work support server 10 constituting the same, a work environment image is outputted to an output interface (e.g., remote output interface 220) of a second client (e.g., remote operation device 20), wherein the work environment image shows a portion of a captured image acquired by a capturing device (e.g., capturing device 612) of a first client (e.g., operator terminal 60), namely, an extended form of an object in an actual space in an object image region R and object related information (see Figure 4 / Step 602‥Yes→Step 610, Step 614, Step 616→‥→Step 112→Step 208, Figure 8 ).

[0077] Therefore, for example, when each operator recognizes that there is a target object around him / her, he / she can quickly obtain a captured image of the target object using the client as the first client (see Figure 4 / Step 602 and Figure 6 ). In addition, other operators can use the client as the second client to quickly identify the extended form of the object in the actual space and the object related information through the working environment image output by the output interface (refer to Figure 4 / Step 208 and Figure 8 ). In addition, for example, since multiple workers at a common site use the client as the first client, a map rich in information about various objects can be shared among the multiple workers. Thus, for example, when a worker uses a work machine to perform work, the worker can smoothly perform the work while being aware of the extended form of the object.

[0078] (Other Embodiments of the Present Invention)

[0079] In the above-described embodiment, the work support server 10 is composed of one or more servers that are independent of the remote operation device 20, the work machine 40, and the operator terminal 60 (see Figure 1 ), as another embodiment, the operation support server 10 may also be a component of the remote operation device 20, the operation machine 40 or the operator terminal 60. Each component 110, 121 and 122 of the operation support server 10 may be two or more components of the remote operation device 20, the operation machine 40 and the operator terminal 60 that can communicate with each other.

[0080] In the above-described embodiment, it is assumed that the target object image region R and the input target object related information can be specified in the first client (for example, the operator terminal 60 or the remote operation device 20). As another embodiment, on the premise that the target object image region R and the input target object related information are not specified in the first client, the captured image obtained by the camera can be sent to the operation support server 10 together with the data indicating the actual spatial position and the actual spatial posture of the camera (see Figure 4 / Step 610).

[0081] In addition, on the premise that the object-related information is not input in the first client, the captured image obtained by the camera can also be sent to the operation support server 10 together with the object image area R and the data indicating the actual spatial position and the actual spatial posture of the camera. Figure 4 / Step 614). Furthermore, on the premise that the target object image region R is not specified in the first client, the captured image acquired by the camera can be sent to the operation support server 10 together with the target object association information and the data indicating the actual spatial position and the actual spatial posture of the camera.

[0082] The first support processing element 121 identifies, based on communication with the first client (for example, the operator terminal 60 or the remote operation device 20), an extended form of an object object in an object object image area specified by an input interface (210, 610) of the first client and object object association information related to the object in a captured image output by an output interface (220, 620) of the first client.

[0083] Thus, for example, as described above, when each operator recognizes that there is an object around him / her, he / she can use the client as the first client to quickly obtain a captured image of the object. In addition, each operator can specify a portion of the image area where the object exists in the captured image of the object output by the output interface of the first client as the object image area through the input interface, and input object-related information. Therefore, the existence of the object noticed by each operator and the object-related information can be transmitted with higher accuracy through other operators.

[0084] The first support processing element 121 identifies the presence of an object in an object object image area, which is a part of a captured image acquired by a real machine shooting device 412 mounted on the working machine 40, and object object association information related to the object, as well as the actual spatial position and actual spatial posture of the real machine shooting device 412, based on communication with the remote operation device 20 serving as the first client for remotely operating the working machine 40.

[0085] Thus, based on the captured images acquired by the real machine imaging device 412 mounted on the working machine 40 , a plurality of workers can share the extended form of the target object and the target object related information around the working machine 40 .

[0086] Description of Reference Numerals

[0087] 10 Operation Support Server

[0088] 20 Remote control device

[0089] 40 Operating machinery

[0090] 60 Operator Terminal

[0091] 102 Database

[0092] 121 1st support processing unit

[0093] 122 Second support processing unit

[0094] 210 Remote input interface

[0095] 220 Remote output interface

[0096] 410 Real machine input interface

[0097] 412 Real Machine Shooting Device

[0098] 420 real machine output interface

[0099] 440 Operation Attachment (Working Organization)

[0100] 610 Input Interface

[0101] 612 Filming Device

[0102] 620 output interface.

Claims

1. A work support server for supporting sharing of information related to a work site among a plurality of workers including a first worker and a second worker by communicating with a plurality of clients including a first client assigned to the first worker among a plurality of workers and a second client assigned to the second worker among the plurality of workers, It is characterized in that have: The first supporting processing element, based on communication with the first client, identifies a portion of the captured image, namely, a target object image area, which is specified by the operation of the first operator via an input interface mounted on the first client for the captured image acquired by the capturing device mounted on the first client, target object related information about the target object existing in the target object image area, which is input via the input interface by the operation of the first operator, and an actual spatial position and an actual spatial posture of the capturing device when acquiring the captured image, which are measured by a control device of the first client, Based on the actual spatial position and the actual spatial posture of the camera, an extension form of the object in the latitude and longitude directions in the actual space is estimated; a second support processing element for generating a work environment image showing the extended form of the target object estimated by the first support processing element and the target object related information; Based on the communication with the second client, the output interface of the second client is caused to output the bird's-eye view map or the bird's-eye view captured image as the work environment image.

2. The operation support server according to claim 1, It is characterized in that The first support processing element identifies the existence of the object in the object object image area, which is a part of the captured image acquired by the shooting device mounted on the working machine, and the object object association information related to the object, and the actual spatial position and actual spatial posture of the shooting device based on the communication with the remote operation device serving as the first client for remotely operating the working machine.

3. A work support system, It is characterized in that The system is composed of the work support server according to claim 1 or 2 and the plurality of clients.

4. A work support method for supporting sharing of information related to a work site among a plurality of workers including a first worker and a second worker by communicating with a plurality of clients including a first client assigned to the first worker among a plurality of workers and a second client assigned to the second worker among the plurality of workers, It is characterized in that have: The first support processing, based on communication with the first client, identifies a portion of the captured image, namely, a target object image area, which is specified by the operation of the first operator via an input interface mounted on the first client for the captured image acquired by the capturing device mounted on the first client, target object related information about the target object existing in the target object image area, which is input via the input interface by the operation of the first operator, and an actual spatial position and an actual spatial posture of the capturing device when acquiring the captured image, which are measured by a control device of the first client, Based on the actual spatial position and the actual spatial posture of the camera, an extension form of the object in the latitude and longitude directions in the actual space is estimated; a second support process of generating a work environment image showing the extended form of the target object estimated by the first support process and the target object related information; Based on the communication with the second client, the output interface of the second client is caused to output the bird's-eye view map or the bird's-eye view captured image as the work environment image.

Images