Remote operation support server and remote operation support system

Abstract

To provide a system and the like which can reduce a psychological burden on an operator who remotely controls a work machine when changing an image range acquired by an actual imaging device mounted on the work machine.SOLUTION: An advance notice expressing the future change mode of an actual space area reflected in a work environment image displayed on a remote image output device 221 is output to a remote output interface 220. An operator who is in contact with an arrow-shaped icon image A1 expressing the displacement mode of a virtual sound source Vss and / or the actual space area displaced according to the displacement mode of the actual space area can grasp in advance that the actual space area reflected in the work environment image is displaced according to the displacement mode in the future.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a technique for notifying an operator of a condition of a work machine. [Background technology]

[0002] With a view to providing a crane that realizes improved safety and improved work efficiency, a technology has been proposed that tracks a subject (e.g., a sub-hook) in an image captured by a camera, cuts out the area containing the subject, and displays it on a display device (see, for example, Patent Document 1).

[0003] From the standpoint of issuing just the right amount of appropriate warning regarding obstacles around a work machine, a technology has been proposed in which the blind spot area from the driver's seat of the work machine (hydraulic excavator) is calculated based on information about the posture and movement of the work machine, the visibility level for the obstacle is determined based on the blind spot area and the positional relationship between the obstacle and the work machine, and the warning level is corrected in accordance with the visibility level (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-156533 A [Patent Document 2] Patent No. 6581139 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when a work machine is remotely operated, if the image range captured by the actual imaging device mounted on the work machine and provided to the operator suddenly changes due to tracking of an object, etc., this may increase the psychological burden, such as anxiety, on the operator.

[0006] Therefore, an object of the present invention is to provide a system etc. that can reduce the psychological burden on an operator remotely operating a work machine when changing the image range acquired by an actual machine imaging device mounted on the work machine. [Means for solving the problem]

[0007] The remote operation support server of the present invention comprises: A server for supporting remote operation of a work machine using a remote operation device, a first support processing element that acquires a captured image representing an environment of the work machine through an actual machine imaging device mounted on the work machine, and displays a work environment image corresponding to the captured image on a remote image output device constituting the remote operation device; a second support processing element that recognizes a future change in a real space area reflected in the work environment image displayed on the remote image output device by the first support processing element, and outputs a notice indicating a future change in the real space area to a remote output interface constituting the remote operation device; It is equipped with:

[0008] According to the remote operation support server having the above configuration, a notice indicating a future change in the real space area reflected in the work environment image displayed on the remote image output device is output to a remote output interface constituting the remote operation device. For example, the spatial occupation mode of the real space area reflected in the work environment image may change depending on the optical axis direction and / or the shooting magnification of the real machine imaging device, switching of one of the multiple real machine imaging devices corresponding to the work environment image, image processing by the first support processing element, and the like. Then, the operator who comes into contact with the notice can be made to know in advance the future change mode of the real space area reflected in the work environment image. Therefore, the psychological burden of the operator when the spatial occupation mode of the real space area reflected in the work environment image displayed on the remote image output device changes immediately thereafter in accordance with the notice is reduced.

[0009] In the remote operation support server having the above configuration, The second support processing element recognizes a future change state of a real space area reflected in the work environment image according to a specified state detected by an actual machine surroundings monitoring device mounted on the work machine. It is preferable.

[0010] According to the remote operation support server having this configuration, an operator who receives a notice of the displacement of the output location can be made to understand in advance that the real space area reflected in the work environment image will be displaced in accordance with the displacement mode in the future, and that the work machine is in a designated state. This reduces the psychological burden on the operator when the real space area reflected in the work environment image displayed on the remote image output device is displaced as predicted.

[0011] In the remote operation support server having the above configuration, The second support processing element recognizes a future displacement state of the real space area as a future change state of the real space area, and displaces an output location of the notice configured by the remote output interface in accordance with the future displacement state of the real space area. It is preferable.

[0012] According to the remote operation support server having the above configuration, an operator who has received a notice of the displacement of the output location can be made to understand in advance that the real space area reflected in the work environment image will be displaced in the future according to the displacement mode. For example, the spatial occupancy mode of the real space area reflected in the work environment image can be displaced according to a change in the optical axis direction of the real machine imaging device, switching of one of the multiple real machine imaging devices corresponding to the work environment image, image processing by the first support processing element, etc. Therefore, the psychological burden of the operator when the real space area reflected in the work environment image displayed on the remote image output device is displaced as predicted can be reduced.

[0013] In the remote operation support server having the above configuration, The second support processing element further recognizes a future expansion / contraction state of the real space area as a future change state of the real space area, and displaces the output location of the notice configured by the remote output interface according to the future displacement state of the real space area before or after displacing the output location of the notice on the remote output interface according to the future expansion / contraction state of the real space area. It is preferable.

[0014] According to the remote operation support server having the above configuration, an operator who has received a notice of the output location being displaced in two stages can be made to understand in advance that the real space area reflected in the work environment image will be displaced in the first displacement mode and will be enlarged or reduced in the second displacement mode in the future. This reduces the psychological burden on the operator when the real space area reflected in the work environment image displayed on the remote image output device is displaced and enlarged or reduced as predicted.

[0015] In the remote operation support server having the above configuration, The second support processing element displaces a virtual sound source that emits the sound as the notice to a remote sound output device that constitutes the remote output interface. It is preferable.

[0016] According to the remote operation support server of this configuration, an operator who receives a notice of the displacement of the virtual sound source can be made to understand in advance that the real space area reflected in the work environment image will be displaced (and even expanded or contracted) in the future according to the displacement mode. For example, when the virtual sound source is displaced to the right (left, up, down), the operator can be made to understand that the real space area reflected in the work environment image will be displaced to the right (left, up, down) immediately afterwards. At this time, the notice is conveyed to the operator through hearing rather than vision, so that the situation in which the visual attention of the operator is distracted from the work environment image is avoided. Therefore, the psychological burden on the operator when the real space area reflected in the work environment image displayed on the remote image output device is displaced as predicted is reduced.

[0017] In the remote operation support server having the above configuration, The second support processing element recognizes a future displacement state of the real space area as a future change state of the real space area in accordance with a switching state from one real machine imaging device to another real machine imaging device among the plurality of real machine imaging devices mounted on the work machine. It is preferred.

[0018] According to the remote operation support server having the above configuration, the operator who has received the notice of the change in the output mode can be made to understand in advance that the future expansion / contraction mode of the real space area reflected in the work environment image will change in accordance with the change in the output mode due to the switching of the real machine imaging device. This reduces the psychological burden on the operator when the real space area reflected in the work environment image displayed on the remote image output device is expanded or contracted as predicted.

[0019] In the remote operation support server having the above configuration, The second support processing element recognizes a future expansion / contraction state of the real space area as a future change state of the real space area, and changes an output state of the notice to the remote output interface in accordance with the future expansion / contraction state of the real space area. It is preferred.

[0020] According to the remote operation support server having the above configuration, the operator who receives the notice of the change in the output mode can be made to understand in advance that the future enlargement / reduction mode of the real space area reflected in the work environment image will change according to the change in the output mode. For example, the spatial occupancy mode of the real space area reflected in the work environment image can be enlarged or reduced according to the shooting magnification of the real machine imaging device, switching of one of the multiple real machine imaging devices corresponding to the work environment image, image processing by the first support processing element, etc. Therefore, the psychological burden of the operator when the real space area reflected in the work environment image displayed on the remote image output device is enlarged or reduced as predicted can be reduced.

[0021] In the remote operation support server having the above configuration, The second support processing element changes at least one of the sound pressure and the frequency of the sound as the notice in a remote sound output device constituting the remote output interface. It is preferable.

[0022] According to the remote operation support server of this configuration, an operator who is exposed to a notice as sound in which the sound pressure and / or the frequency change can be made to know in advance that the real space area reflected in the work environment image will expand or shrink in accordance with the change in the future. For example, when the sound pressure gradually increases, the operator can be made to know in advance that the real space area will shrink. On the other hand, when the sound pressure gradually decreases, the operator can be made to know in advance that the real space area will expand. In this case, the notice is conveyed to the operator not through sight but through hearing, so that the situation in which the visual attention of the operator is distracted from the work environment image is avoided. Therefore, the psychological burden of the operator when the real space area reflected in the work environment image displayed on the remote image output device expands or shrinks as predicted is reduced. [Brief description of the drawings]

[0023] [Figure 1] FIG. 1 is an explanatory diagram regarding the configuration of a remote operation support system. [Diagram 2] FIG. 2 is an explanatory diagram relating to the configuration of a remote control device. [Diagram 3] FIG. 2 is an explanatory diagram relating to the configuration of a work machine. [Figure 4] FIG. 2 is an explanatory diagram regarding functions of the remote operation support system according to the first embodiment. [Diagram 5] FIG. [Figure 6] FIG. 4 is an explanatory diagram relating to a first displacement mode of the virtual sound source. [Figure 7] FIG. 11 is an explanatory diagram relating to a second displacement mode of the virtual sound source. [Figure 8] FIG. 11 is an explanatory diagram regarding functions of a remote operation support system according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] (Configuration of remote operation support system) The remote operation support system shown in Fig. 1 is composed of a remote operation support server 10, a remote operation device 20, and a work machine 40 that can be remotely controlled via the remote operation device 20. 40 Only one of them may be a component of the remote operation support system. The remote operation support server 10, the remote operation device 20, and the work machine 40 are configured to be able to communicate with each other via a network. The mutual communication network of the remote operation support server 10 and the remote operation device 20 and the mutual communication network of the remote operation support server 10 and the work machine 40 may be the same as or different from each other.

[0025] (Configuration of remote operation support server) The remote operation support server 10 includes a database 102, a first support processing element 121, and a second support processing element 122. The database 102 stores captured image data and the like. The database 102 may be configured as a database server separate from the remote operation support server 10. Each support processing element is configured as an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing (described later) on the data in accordance with the software.

[0026] (Configuration of remote control device) The remote operation device 20 includes a remote control device 200, a remote input interface 210, and a remote output interface 220. The remote control device 200 is configured with an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing on the data according to the software.

[0027] The remote input interface 210 includes a remote control mechanism 211. The remote output interface 220 includes a remote image output device 221, a remote audio output device 222, and a remote wireless communication device 224.

[0028] The remote operation mechanism 211 includes a traveling operation device, a slewing operation device, a boom operation device, an arm operation device, and a bucket operation device. Each operation device has an operation lever that receives a rotation operation. The operation lever (travel lever) of the traveling operation device is operated to move the lower traveling body 410 of the work machine 40. The travel lever may also serve as a travel pedal. For example, a travel pedal fixed to the base or lower end of the travel lever may be provided. The operation lever (slewing lever) of the slewing operation device is operated to move a hydraulic slewing motor that constitutes the slewing mechanism 430 of the work machine 40. The operation lever (boom lever) of the boom operation device is operated to move the boom cylinder 442 of the work machine 40. The operation lever (arm lever) of the arm operation device is operated to move the arm cylinder 444 of the work machine 40. The operation lever (bucket lever) of the bucket operation device is operated to move the bucket cylinder 446 of the work machine 40.

[0029] The operating levers constituting the remote control mechanism 211 are arranged around a seat St on which the operator sits, as shown in Fig. 2. The seat St is in the form of a high-back chair with armrests, but may be in the form of a low-back chair without a headrest, or in the form of a chair without a backrest, or in any other form on which the operator can sit.

[0030] A pair of left and right travel levers 2110 corresponding to the left and right crawlers are arranged side by side in front of the seat St. One operation lever may serve as multiple operation levers. For example, the left operation lever 2111 provided in front of the left frame of the seat St shown in FIG. 2 may function as an arm lever when operated in the front-rear direction, and may function as a rotation lever when operated in the left-right direction. Similarly, the right operation lever 2112 provided in front of the right frame of the seat St shown in FIG. 2 may function as a boom lever when operated in the front-rear direction, and may function as a bucket lever when operated in the left-right direction. The lever pattern may be arbitrarily changed by an operation instruction from the operator.

[0031] 2, the remote image output device 221 is composed of a central remote image output device 2210, a left remote image output device 2211, and a right remote image output device 2212, each having a substantially rectangular screen, and disposed respectively in front of, diagonally forward left of, and diagonally forward right of, the seat St. The shapes and sizes of the screens (image display areas) of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be the same or different.

[0032] As shown in Fig. 2, the right edge of the left remote image output device 2211 is adjacent to the left edge of the central remote image output device 2210, such that the screens of the central remote image output device 2210 and the left remote image output device 2211 form an inclination angle θ1 (e.g., 120°≦θ1≦150°). As shown in Fig. 2, the left edge of the right remote image output device 2212 is adjacent to the right edge of the central remote image output device 2210, such that the screens of the central remote image output device 2210 and the right remote image output device 2212 form an inclination angle θ2 (e.g., 120°≦θ2≦150°). The inclination angles θ1 and θ2 may be the same or different.

[0033] The screens of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be parallel to the vertical direction or may be inclined relative to the vertical direction. At least one of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be composed of a plurality of divided image output devices. For example, the central remote image output device 2210 may be composed of a pair of image output devices adjacent to each other above and below, each having a substantially rectangular screen.

[0034] The remote sound output device 222 is composed of one or more speakers, and for example, as shown in Fig. 2, is composed of a central sound output device 2220, a left sound output device 2221, and a right sound output device 2222, which are arranged behind the seat St, at the rear of the left armrest, and at the rear of the right armrest, respectively. The specifications of the central sound output device 2220, the left sound output device 2221, and the right sound output device 2222 may be the same or different. The remote sound output device 222 outputs a virtual sound source V, which is sound whose position can be displaced around the seat St. ss To achieve this.

[0035] (Work machine configuration) The work machine 40 comprises an actual machine control device 400, an actual machine input interface 41, an actual machine output interface 42, and a work mechanism 440. The actual machine control device 400 comprises an image processing device 30. The image processing device 30 comprises a state detection element 31, an image prediction element 32, and an image compression element 34. Each of the components of the actual machine control device 400 and the image processing device 30 is configured by an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing on the data in accordance with the software.

[0036] The work machine 40 is, for example, a crawler excavator (construction machine), and as shown in Fig. 3, includes a crawler-type lower traveling body 410, and an upper rotating body 420 that is rotatably mounted on the lower traveling body 410 via a rotating mechanism 430. A cab 424 (operator's cabin) is provided on the front left side of the upper rotating body 420. A work mechanism 440 is provided in the front center of the upper rotating body 420.

[0037] The real machine input interface 41 includes a real machine operation mechanism 411, a real machine imaging device 412, and a real machine surroundings monitoring device 414. The real machine operation mechanism 411 includes a plurality of operation levers arranged around a seat arranged inside a cab 424 in the same manner as the remote control mechanism 211. A drive mechanism or robot that receives a signal according to the operation mode of the remote control lever and moves the real machine operation lever based on the received signal is provided in the cab 424. The real machine imaging device 412 is installed, for example, inside the cab 424, and images the environment including at least a part of the working mechanism 440 through the front window and a pair of left and right side windows. Some or all of the front window and the side windows may be omitted. The actual machine surroundings monitoring device 414 is composed of a distance measuring sensor etc. for detecting the presence of a target object (e.g., a worker, other work machinery and / or a vehicle, etc.) in an actual space area (e.g., the right side area, left side area and / or rear area and / or blind spot area of ​​the cab 424) which cannot be imaged by the actual machine imaging device 412 in a basic position (e.g., a position in which the optical axis is directed forward).

[0038] The real device output interface 42 includes a real device wireless communication device 422 .

[0039] 3, a working mechanism 440 as a working mechanism includes a boom 441 mounted on the upper rotating body 420 so as to be able to rise and fall, an arm 443 rotatably connected to the tip of the boom 441, and a bucket 445 rotatably connected to the tip of the arm 443. A boom cylinder 442, an arm cylinder 444, and a bucket cylinder 446 each composed of an extendable hydraulic cylinder are mounted on the working mechanism 440.

[0040] The boom cylinder 442 is interposed between the boom 441 and the upper rotating body 420 so as to extend and retract when supplied with hydraulic oil to rotate the boom 441 in the hoisting direction. The arm cylinder 444 is interposed between the arm 443 and the boom 441 so as to extend and retract when supplied with hydraulic oil to rotate the arm 443 about a horizontal axis relative to the boom 441. The bucket cylinder 446 is interposed between the bucket 445 and the arm 443 so as to extend and retract when supplied with hydraulic oil to rotate the bucket 445 about a horizontal axis relative to the arm 443.

[0041] (function) The functions of the remote operation support system having the above configuration will be described with reference to the flowchart shown in Fig. 4. In the flowchart, a block "C●" is used for the sake of simplicity, and means transmission and / or reception of data, and means a conditional branch in which processing in the branching direction is executed on the condition that the data is transmitted and / or received.

[0042] In the remote operation device 20, an environment confirmation request is transmitted to the remote operation assistance server 10 via the remote wireless communication device 224 (FIG. 4 / STEP 210). For example, a requirement for starting transmission of the environment confirmation request may be determined to be a designation operation by the operator via the remote input interface 210. The "designation operation" is, for example, an operation such as tapping on the remote input interface 210 for designating the work machine 40 that the operator intends to remotely operate.

[0043] When the environment confirmation request is received in the remote operation support server 10, the first support processing element 121 transmits the environment confirmation request to the corresponding work machine 40 (FIG. 4 / C10).

[0044] When an environment confirmation request is received in the work machine 40 via the actual machine wireless communication device 422 (Figure 4 / C40), an image is acquired by the actual machine control device 400 via the actual machine imaging device 412, and image data representing the image is transmitted to the remote control device 10 via the actual machine wireless communication device 422 (Figure 4 / STEP 410).

[0045] In the remote operation support server 10, when the first support processing element 121 receives captured image data (FIG. 4 / C11), the second support processing element 122 transmits work environment image data corresponding to the captured image to the remote operation device 20 (FIG. 4 / STEP 110). The work environment image data is not only the captured image data itself, but also image data representing a simulated work environment image generated based on the captured image.

[0046] When the remote operation device 20 receives work environment image data through the remote wireless communication device 224 (FIG. 4 / C21), the remote control device 200 outputs a work environment image corresponding to the work environment image data to the remote image output device 221 (FIG. 4 / STEP 212).

[0047] As a result, for example, as shown in FIG. 5, a work environment image showing a boom 441, arm 443, and a pile of rubble or earth and sand (the object of work by the bucket 445), which are part of the work mechanism 440, in front of the cab 424, is output to the remote image output device 221 (in particular, the central remote image output device 2210) through the window frame that defines the cab 424.

[0048] In the work machine 40, the actual machine control device 400 determines whether the work machine 40 is in a designated state (FIG. 4 / STEP 411). At this time, it may be determined which of a plurality of different designated states that the designated state corresponds to. The "designated state" is a state that is set or programmed in advance such that the actual machine control device 400 causes the actual machine imaging device 412 to perform its pan function and / or tilt function, as well as the zoom function, in response to the work machine 40 entering a designated state.

[0049] For example, a state in which the real machine surroundings monitoring device 414 detects the presence of a target object in a blind spot area of ​​the real machine imaging device 412 in the normal position may be defined as the specified state. The left side area of ​​the cab 424 is the blind spot area of ​​the real machine imaging device 412 in the normal position, and when the presence of a target object is detected in the left side area, the optical axis of the real machine imaging device 412 is directed leftward by the pan function, and the optical axis is directed downward (or upward) by the tilt function as necessary. Similarly, the right side area of ​​the cab 424 is the blind spot area of ​​the real machine imaging device 412 in the normal position, and when the presence of a target object is detected in the right side area, the optical axis of the real machine imaging device 412 is directed rightward by the pan function, and the optical axis is directed downward (or upward) by the tilt function as necessary.

[0050] For example, a state in which the presence of a designated type of target object is detected in a real space region to be imaged by the real machine imaging device 412 may be defined as the designated state. When the presence of a first designated type of target object (e.g., a human being such as a worker) is detected, the zoom-in function of the real machine imaging device 412 increases the display magnification of the image region including the target object. When the presence of a second designated type of target object (e.g., another work machine) is detected, the zoom-out function of the real machine imaging device 412 decreases the display magnification of the image region including the target object. Together with the zoom function of the real machine imaging device 412, a pan function and / or a tilt function may be controlled to adjust the position of the target object in the captured image.

[0051] For example, the specified state may be defined as a state in which the amount of displacement and / or the displacement speed of a target object is detected to be equal to or greater than a predetermined value in a real space region that is the subject of imaging by the real imaging device 412. When the amount of displacement of a target object (e.g., the bucket 445) exceeds a predetermined value, the direction of the optical axis of the real imaging device 412 is changed by a pan function and / or a tilt function of the real imaging device 412 so as to follow the target object.

[0052] When it is determined that the work machine 40 is in the designated state (FIG. 4 / STEP 411...YES), the real machine control device 400 recognizes the control mode of the attitude change function (pan function and / or tilt function) and / or zoom function of the real machine imaging device 412 corresponding to the designated state, and transmits the recognition result to the remote operation support server 10 via the real machine wireless communication device 422 (FIG. 4 / STEP 412). The correspondence between the (type of) designated state and the control mode of the attitude change function and / or zoom function of the real machine imaging device 412 may be stored in the storage device or database 102 constituting the real machine control device 400.

[0053] If it is determined that the work machine 40 is not in the specified state (FIG. 4 / STEP 411...NO), the control mode of the zoom function of the actual machine imaging device 412 and the like is not recognized and the subsequent processing is executed.

[0054] When the remote operation support server 10 receives the control state of the attitude change function and / or zoom function of the real machine imaging device 412 (FIG. 4 / C12), the second support processing element 122 recognizes the change state of the real space area reflected in the captured image (and thus the work environment image) and transmits it to the work machine 40 (FIG. 4 / STEP112). When the direction of the optical axis changes left and right due to the pan function of the real machine imaging device 412, it is recognized that the real space area reflected in the captured image is similarly displaced left and right. When the direction of the optical axis changes up and down due to the tilt function of the real machine imaging device 412, it is recognized that the real space area reflected in the captured image is similarly displaced up and down. When the zoom magnification increases due to the zoom-in function of the real machine imaging device 412, it is recognized that the real space area reflected in the captured image is reduced. When the zoom magnification decreases due to the zoom-out function of the real machine imaging device 412, it is recognized that the real space area reflected in the captured image is enlarged.

[0055] When the remote operation device 20 receives a change in the real space area through the remote wireless communication device 224 (FIG. 4 / C22), the remote control device 200 outputs a notice indicating the change through the remote output interface 220 (FIG. 4 / STEP 214).

[0056] As a result, for example, when a real space area reflected in a captured image is recognized as being displaced to the left as a change in the real space area, as shown in FIG. 6, the virtual sound source V ss is controlled to move to the left (the same direction as the displacement direction of the real space). In this case, the virtual sound source V ss In addition to or instead of the displacement of the virtual sound source V, as shown in FIG. 6, a left-facing arrow-shaped figure or icon A1 (same direction as the displacement direction of the real space area) may be output to the image output device 221. Also, when the designated icon is output to the remote image output device 211, ss may be output so as to be moved in the same direction as

[0057] In addition, for example, when a reduction in the real space area reflected in the captured image is recognized as a change in the real space area, as shown in FIG. 7, the virtual sound source V ss The sound is controlled so that it gradually becomes stronger and / or the frequency of the sound gradually becomes higher with the virtual sound source V ss 7, an image area S enlarged by zooming in and / or an arrow-shaped figure or icon A2 pointing outward from the image area S may be output to the image output device 221. Also, a designated icon may be output so as to be enlarged or reduced in accordance with the enlargement or reduction of the real space area or the image area in the remote image output device 211.

[0058] Conversely, when the expansion of the real space area reflected in the captured image is recognized as a change in the real space area, the virtual sound source V ss In this case, the virtual sound source V ss In addition to or instead of the intensity and / or frequency of the sound, an image area reduced by zooming out and / or an arrow-shaped figure or icon pointing inward of the image area may be output to the image output device 221.

[0059] Thereafter, the remote control device 200 transmits a notice that the notice has been outputted through the remote output interface 220 to the remote operation support server 10 through the remote wireless communication device 224 (FIG. 4 / STEP 216).

[0060] When the advance notice output completion notification is received in the remote operation support server 10, the advance notice output completion notification is transmitted to the work machine 40 by the second support processing element 122 (FIG. 4 / C14).

[0061] When the work machine 40 receives a notice of advance output via the actual machine wireless communication device 422 (Figure 4 / C44), the actual machine control device 400 controls the attitude change function and / or zoom function of the actual machine imaging device 412 in a manner corresponding to the specified state (Figure 4 / STEP 414).

[0062] This changes the real space area reflected in the captured image acquired through the real machine imaging device 412, and ultimately in the work environment image output by the remote image output device 221 (FIG. 4 / STEP 410→...→STEP 212, see FIG. 5).

[0063] For example, when the optical axis direction changes from the front of the work machine 40 to the diagonally forward left by control of the pan function of the real machine imaging device 412, the real space area reflected in the work environment image changes from the front area to the diagonally forward left area of ​​the work machine 40. Also, when the optical axis direction changes from the forward horizontal direction to the diagonally downward forward by control of the tilt function of the real machine imaging device 412, the real space area reflected in the work environment image changes to a lower area than before.

[0064] For example, the zoom-in function of the real machine imaging device 412 is controlled to increase the zoom magnification, thereby reducing the real space area reflected in the work environment image (or enlarging the image area). Also, the zoom-out function of the real machine imaging device 412 is controlled to reduce the zoom magnification, thereby enlarging the real space area reflected in the work environment image (or enlarging the image area).

[0065] In the remote operation device 20, the remote control device 200 recognizes the operation mode of the remote operation mechanism 211, and a remote operation command corresponding to the operation mode is transmitted to the remote operation support server 10 via the remote wireless communication device 224 (FIG. 4 / STEP 220).

[0066] In the remote operation support server 10, when the second support processing element 122 receives the remote operation command, the first support processing element 121 transmits the remote operation command to the work machine 40 (FIG. 4 / C16).

[0067] In the work machine 40, when an operation command is received by the actual machine control device 400 via the actual machine wireless communication device 422 (FIG. 4 / C46), the operation of the work mechanism 440 and the like is controlled (FIG. 4 / STEP 420). For example, the work of scooping up soil in front of the work machine 40 with the bucket 445, rotating the upper rotating body 410, and dropping the soil from the bucket 445 is performed.

[0068] (Action and effect) According to the remote operation support system having this configuration, a preview showing the future change state of the real space area reflected in the work environment image displayed on the remote image output device 221 is output to the remote output interface 220.

[0069] Specifically, a virtual sound source V ss And / or, the operator who comes into contact with the arrow-shaped icon image A1 representing the displacement mode of the real space area can be made to know in advance that the real space area reflected in the work environment image will be displaced according to the displacement mode in the future (see FIG. 6). For example, the spatial occupancy mode of the real space area reflected in the work environment image can be displaced according to a change in the optical axis direction of the real machine imaging device 412 (see FIG. 5). This reduces the psychological burden on the operator when the real space area reflected in the work environment image displayed on the remote image output device 221 is displaced as predicted.

[0070] Furthermore, a virtual sound source V emitting a sound whose intensity and / or frequency change according to the expansion and contraction state of the real space area is ssAnd / or, the operator who comes into contact with the arrow-shaped icon image A2 representing the enlargement / reduction mode of the real space area can be made to know in advance that the real space area reflected in the work environment image will be enlarged or reduced in accordance with the enlargement / reduction mode in the future (see FIG. 7). For example, the spatial occupancy mode of the real space area reflected in the work environment image can be enlarged or reduced according to a change in the shooting magnification of the real machine imaging device 412. This reduces the psychological burden on the operator when the real space area reflected in the work environment image displayed on the remote image output device 221 is enlarged or reduced as predicted.

[0071] (Another embodiment of the present invention) In the above embodiment, the first support processing element 121 and the second support processing element 122 were configured by the remote operation support server 10, but in other embodiments, the first support processing element 121 and / or the second support processing element 122 may be configured by the work machine 40 and / or the remote operation device 20.

[0072] In the above embodiment, in response to recognizing that the work machine 40 is in a specified state, the actual machine control device 400 recognizes the control mode of the attitude change function and / or zoom function of the actual machine imaging device 412 corresponding to the specified state (see Figure 4 / STEP 411...YES → STEP 412), but in other embodiments, the remote operation support server 10 or the second support processing element 122 may recognize the control mode of the attitude change function and / or zoom function of the actual machine imaging device 412 corresponding to the specified state.

[0073] In this case, a series of processes may be executed according to the flowchart shown in Fig. 8 instead of the flowchart shown in Fig. 4. In the flowchart, a block "C●" is used for the sake of simplicity, and means transmission and / or reception of data, and means a conditional branch in which a process in a branching direction is executed on the condition that the data is transmitted and / or received. In Fig. 8, the same reference numerals are used for processes common to Fig. 4, and the description thereof will be omitted.

[0074] If it is determined that the work machine 40 is in the designated state (FIG. 8 / STEP 411...YES), the actual machine control device 400 transmits the designated state to the remote operation support server 10 via the actual machine wireless communication device 422 (FIG. 8 / STEP 413).

[0075] When the remote operation support server 10 receives a designated state (FIG. 8 / C13), the second support processing element 122 recognizes the control state of the attitude change function (pan function and / or tilt function) and / or zoom function of the actual machine imaging device 412 corresponding to the designated state (FIG. 8 / STEP 111). Furthermore, the second support processing element 122 recognizes the change state of the real space area reflected in the captured image (and thus the work environment image) corresponding to the control state of the attitude change function and / or zoom function of the actual machine imaging device 412, and transmits it to the work machine 40 (FIG. 8 / STEP 112).

[0076] Thereafter, when the remote operation support server 10 receives a notice of completion of advance output from the remote operation device 20 (FIG. 8 / C15), the second support processing element 122 generates a control command for the attitude change function and / or zoom function of the actual machine imaging device 412 in a form corresponding to the specified state, and transmits it to the work machine 40 (FIG. 8 / STEP 114).

[0077] When a control command is received in the work machine 40 through the real machine wireless communication device 422 (Figure 8 / C44), the real machine control device 400 controls the attitude change function and / or zoom function of the real machine imaging device 412 in accordance with the control command (Figure 8 / STEP 414).

[0078] This changes the real space area reflected in the captured image acquired through the real machine imaging device 412, and ultimately in the work environment image output by the remote image output device 221 (FIG. 4 / STEP 410→...→STEP 212, see FIG. 5).

[0079] In the above embodiment, in response to recognizing that the work machine 40 is in a specified state, the actual machine control device 400 recognizes the control mode of the attitude change function and / or zoom function of the actual machine imaging device 412 corresponding to the specified state (see Figure 4 / STEP 411...YES → STEP 412), but in another embodiment, the remote operation device 20 or the remote control device 200 may recognize the control mode of the attitude change function and / or zoom function of the actual machine imaging device 412 corresponding to the specified state.

[0080] In the above embodiment, the pan function and / or tilt function of the real machine imaging device 412 continuously displaces the real space area reflected in the captured image and the corresponding work environment image, but in another embodiment, a plurality of real machine imaging devices having different imaging areas may be mounted on the work machine 40, and the real machine imaging device that obtains the captured image that is the basis of the work environment image may be switched from one real machine imaging device to another real machine imaging device, so that the real space area reflected in the work environment image may be discontinuously displaced. For example, one real machine imaging device may be switched to another real machine imaging device whose imaging area is the real space area where the target object is detected by the real machine surroundings monitoring device 414. In this case, too, the advance notice output by the remote output interface 220 reduces the psychological burden on the operator even if the work environment image output to the remote image output device 221 is discontinuously displaced or changed.

[0081] In the above embodiment, the real space area reflected in the captured image and the corresponding work environment image is continuously displaced by the pan function and / or tilt function of the real machine imaging device 412, but in another embodiment, the real space area reflected in the work environment image may be displaced and / or enlarged / reduced by image processing performed by the first support processing element 121 when generating the work environment image. In this case as well, the psychological burden on the operator is reduced even when the work environment image output to the remote image output device 221 is displaced and / or enlarged / reduced by the advance notice output by the remote output interface 220.

[0082] In the above embodiment, the zooming in and out of the real image capture device 412 is ss In another embodiment, the real imaging device 412 has only a panning function, and the change in the panning angle is controlled by the virtual sound source V ss The zooming in and out of the real image pickup device 412 before and after the left and right displacement of the virtual sound source V ss Similarly, in another embodiment, the real image capture device 412 has only a tilt function, and the change in the tilt angle is notified by the virtual sound source V ss The zoom-in and zoom-out of the real imaging device 412 occurs before and after the up and down displacement of the virtual sound source V ss This may be announced by a left or right displacement of the [Explanation of symbols]

[0083] 10: Remote operation support server, 20: Remote operation device, 40: Work machine, 102: Database, 121: First support processing element, 122: Second support processing element, 200: Remote control device, 210: Remote input interface, 211: Remote operation mechanism, 220: Remote output interface, 221: Remote image output device, 222: Remote sound output device, 400: Actual machine control device, 410: Actual machine input interface, 420: Actual machine output interface, 424: Cab (operating room), 440: Work mechanism, 445: Bucket (working part).

Claims

1. A remote operation support system for supporting remote operation of a work machine using a remote operation device, The remote operation device and a remote operation support server are configured, The remote control device is a remote image output device that displays a work environment image corresponding to an image captured by an actual machine imaging device having an attitude changing function mounted on the work machine, the image capturing device representing the environment of the work machine; a remote output interface constituting the remote control device; the remote operation support server includes a processing device that recognizes a future change state of a real space area reflected in the work environment image displayed on the remote image output device based on a control state of the attitude change function of the real machine imaging device, A notice indicating a future change in the real space area recognized by the arithmetic processing device is outputted through the remote output interface. Remote operation support system.

2. 2. The remote operation support system according to claim 1, the arithmetic processing device recognizes a future displacement state of the real space region as a future change state of the real space region; The output location of the notice output through the remote output interface is displaced according to the future displacement mode of the real space area recognized by the arithmetic processing device. Remote operation support system.

3. A remote operation support system for supporting remote operation of a work machine using a remote operation device, The remote operation device and a remote operation support server are configured, The remote control device is a remote image output device that displays a work environment image corresponding to an image representing the environment of the work machine acquired by an actual machine imaging device mounted on the work machine; a remote output interface constituting the remote control device; the remote operation support server includes a processing device that recognizes a future displacement state of a real space area reflected in the work environment image displayed on the remote image output device as a future change state of the real space area, A notice representing a future change manner of the real space area recognized by the arithmetic processing device is output through the remote output interface, and the output location of the notice is displaced according to the future change manner of the real space area. Remote operation support system.

4. 4. The remote operation support system according to claim 2, The arithmetic processing device further recognizes a future expansion / contraction state of the real space region as a future change state of the real space region, The output location of the notice output through the remote output interface is displaced according to the future displacement of the real space area before or after the output location of the notice is displaced according to the future expansion / contraction of the real space area. Remote operation support system.

5. In the remote operation support system according to any one of claims 2 to 4, The remote output interface includes a remote sound output device that displaces a virtual sound source that emits the sound as the warning. Remote operation support system.

6. In the remote operation support system according to any one of claims 2 to 5, The arithmetic processing device recognizes a future displacement state of the real space area as a future change state of the real space area in accordance with a switching state from one real machine imaging device to another real machine imaging device among the plurality of real machine imaging devices mounted on the work machine. Remote operation support system.

7. In the remote operation support system according to any one of claims 1 to 6, The arithmetic processing device recognizes a future change state of a real space area reflected in the work environment image according to a specified state detected by an actual machine surroundings monitoring device mounted on the work machine. Remote operation support system.

8. The remote operation support system according to any one of claims 1 to 7, The output mode of the notice outputted through the remote output interface is changed according to the future expansion / contraction mode of the real space area as the future change mode of the real space area recognized by the arithmetic processing device. Remote operation support system.

9. The remote operation support system according to claim 8, The remote output interface includes a remote sound output device that changes at least one of the sound pressure and the frequency of the sound as the notice. Remote operation support system.

10. A remote operation support server for supporting remote operation of a work machine using a remote operation device, a first support processing element that acquires an image representing an environment of the work machine through an actual machine imaging device having an attitude changing function mounted on the work machine, and displays a work environment image corresponding to the image on a remote image output device constituting the remote operation device; The first support processing element recognizes a future change state of a real space area reflected in the work environment image displayed on the remote image output device based on a control state of the attitude change function of the real machine imaging device, and outputs to a remote output interface constituting the remote operation device, a second support processing element that outputs a notice indicating a future change state of the real space area; A remote operation support server comprising:

11. A remote operation support server for supporting remote operation of a work machine using a remote control device. hand, a first support processing element that acquires an image representing an environment of the work machine through an actual machine imaging device mounted on the work machine, and displays a work environment image corresponding to the image on a remote image output device constituting the remote operation device; a second support processing element that recognizes a future displacement state of the real space area as a future change state of the real space area reflected in the work environment image displayed on the remote image output device by the first support processing element, outputs a notice representing the future change state of the real space area to a remote output interface constituting the remote operation device, and displaces an output location in accordance with the future displacement state of the real space area; A remote operation support server comprising:

Images