Work support system and work support complex system

Abstract

The present application provides a system capable of reliably notifying a worker of a movement mode of a work machine regardless of the length of the interval between the work machine and the worker. An unmanned aerial vehicle (60) projects an identification image (M) on a surrounding area of the worker (for example, a ground surface located near the worker to a degree that the worker can visually recognize). The identification image (M) is an image indicating the movement mode of the work machine (40). Therefore, compared to a case where the identification image (M) is projected at a place regardless of the position of the worker, it is possible to reliably notify the worker of the movement mode of the work machine (40) regardless of the length of the interval between the work machine (40) and the worker.

Description

Technical Field

[0001] This invention relates to a technology for notifying operators of the status of operating machinery. Background Technology

[0002] When machinery is moved by a remote operator or through actual machine operation, it is necessary to inform the operator of the machinery's direction of travel to ensure the safety of the operator at the work site. Therefore, a technology is being considered that projects an image of the transport vehicle's navigation route onto the road surface by an unmanned aerial vehicle (UAV) while the transport vehicle is operating in an unmanned driving mode (see, for example, Patent Document 1).

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent No. 6707600 Summary of the Invention

[0006] The technical problem that the invention aims to solve

[0007] However, when the operator leaves the navigation route of the work machinery, it becomes difficult to inform the operator that the navigation route has been projected onto the road surface, and thus difficult to inform the operator of the direction of travel of the work machinery.

[0008] Therefore, the object of the present invention is to provide a system that can reliably notify the operator of the movement mode of the working machine regardless of the length of the interval between the working machine and the operator.

[0009] Solution to the above technical problems

[0010] The work support system of the present invention includes:

[0011] The first support processing element identifies the positions of the operator and the machine in time sequence;

[0012] The second support processing element controls the flight maneuver function of the unmanned aerial vehicle (UAV) to move the UAV to a position below and facing the operator, as identified by the first support processing element. It also controls the UAV's identification projection function to project an identification image representing the movement mode of the work machinery onto the area surrounding the operator. The movement mode of the work machinery is determined based on the time sequence of the position of the work machinery identified by the first support processing element.

[0013] According to this operational support system, an unmanned aerial vehicle projects a marker image onto the area surrounding the operator (e.g., the ground near the operator to a degree that the operator can visually recognize). The "marker image" is an image indicating the movement pattern of the operating machinery. Therefore, compared to projecting a marker image in a location independent of the operator's position, it is possible to reliably notify the operator of the operating machinery's movement pattern regardless of the distance between the operating machinery and the operator. Attached Figure Description

[0014] Figure 1 This is a schematic diagram illustrating the configuration of a work support composite system as one embodiment of the present invention.

[0015] Figure 2 This is an explanatory diagram of the configuration of a remote operation device.

[0016] Figure 3 It is an explanatory diagram of the structure of the operating machinery.

[0017] Figure 4 This is an explanatory diagram of the first function of the job support system.

[0018] Figure 5 This is an explanatory diagram of the second function of the job support system.

[0019] Figure 6 It is an explanatory diagram of the working environment.

[0020] Figure 7 This is an illustration of the navigation function of unmanned aerial vehicles for operating machinery.

[0021] Figure 8A This is an explanatory diagram of the first method for determining the operator who becomes the recipient of a notification regarding the actual machine status.

[0022] Figure 8B This is an explanatory diagram of the second method for determining the operator who becomes the recipient of the notification regarding the actual machine status.

[0023] Figure 9A This is an illustration of the first method of notifying operators of the actual status of an unmanned aerial vehicle.

[0024] Figure 9B This is an illustration of the second method of notifying operators of the actual status of an unmanned aerial vehicle.

[0025] Figure 9C This is an illustration of the third method of notifying operators of the actual status of an unmanned aerial vehicle.

[0026] Figure 10A This is an illustration of the fourth method of notifying operators of the actual status of an unmanned aerial vehicle.

[0027] Figure 10B This is an illustration of the fifth method of notifying operators of the actual status of an unmanned aerial vehicle.

[0028] Figure 11A This is an illustration of the first cooperative mode of multiple unmanned aerial vehicles.

[0029] Figure 11B This is an illustration of a second collaborative method involving multiple unmanned aerial vehicles.

[0030] Figure 12 This is an illustration of the sixth method of notifying operators of the actual status of an unmanned aerial vehicle. Detailed Implementation

[0031] (The structure of a remote operating system)

[0032] Figure 1 The work support system shown as an embodiment of the present invention comprises a work support server 10 for supporting remote operation of the work machinery 40 by the remote operating device 20. The work support server 10 is configured such that the remote operating device 20, the work machinery 40, the unmanned aerial vehicle 60, and the mobile terminal device 80 can each communicate via a common or separate network.

[0033] In this invention, the constituent element (hardware) "identifying" specified information means receiving the specified information; reading or retrieving the specified information from an internal storage device (e.g., a memory) and / or an external storage device (e.g., an external database server); calculating, estimating, predicting, identifying the specified information by performing arithmetic processing on the received, read, retrieved, etc. information; and encompassing all arithmetic processing for preparing the specified information in a form that can be used in subsequent arithmetic processing.

[0034] (Composition of the job support server)

[0035] The job support server 10 includes a database 102, a first support processing element 121, and a second support processing element 122. The database 102 stores and holds captured image data, etc. The database 102 can be configured as a database server independent of the job support server 10. Each support processing element is configured as a processing unit (a single-core processor, a multi-core processor, or a processor core constituting the processor), which reads necessary data and software from storage devices such as memory, and performs processing operations based on the data, as described later in the software description.

[0036] (Composition of the remote operation device)

[0037] The remote operation device 20 includes a remote control device 200, a remote input interface 210, and a remote output interface 220. The remote control device 200 is composed of a processing unit (a single-core processor, a multi-core processor, or a processor core that constitutes a processor), which reads necessary data and software from storage devices such as memory, and performs arithmetic processing based on the software using the data as the object.

[0038] The remote input interface 210 includes a remote operation mechanism 211. The remote output interface 220 includes a remote image output device 221, a remote audio output device 222, and a remote wireless communication device 224.

[0039] The remote control mechanism 211 includes a travel control device, a slewing control device, a boom control device, a stick control device, and a bucket control device. Each control device has a lever for rotating operation. The travel control device's lever (travel lever) is operated to move the lower traveling body 410 of the work machinery 40. The travel lever can also serve as a travel pedal. For example, a travel pedal fixed to the base or lower end of the travel lever can also be provided. The slewing control device's lever (slewing lever) is operated to operate the hydraulic slewing motor constituting the slewing mechanism 430 of the work machinery 40. The boom control device's lever (boom lever) is operated to move the boom cylinder 442 of the work machinery 40. The stick control device's lever (armever) is operated to move the stick cylinder 444 of the work machinery 40. The bucket control device's lever (bucket lever) is operated to move the bucket cylinder 446 of the work machinery 40.

[0040] For example, Figure 2 As shown, the levers constituting the remote operation mechanism 211 are arranged around the seat St for the operator to sit on. The seat St can be a high-backed chair with armrests, a low-backed chair without a headrest, or a chair without a backrest, or any other type of seating where the operator can sit.

[0041] A pair of left and right travel levers 2110 are arranged side by side in front of the seat St, corresponding to the left and right tracks. One lever can also function as multiple levers. For example, Figure 2 The left-side operating lever 2111, located in front of the left side frame of the seat St, can function as a boom lever when operated in the forward / backward direction, and as a swivel lever when operated in the left / right direction. Similarly, Figure 2The right-side operating lever 2112, located in front of the right-side frame of the seat St, can function as a boom lever when operated in the forward / backward direction and as a bucket lever when operated in the left / right direction. The lever mode can be changed arbitrarily according to the operator's instructions.

[0042] For example, Figure 2 As shown, the remote image output device 221 consists of a central remote image output device 2210, a left remote image output device 2211, and a right remote image output device 2212, each with a roughly rectangular screen, respectively positioned in front of, to the left diagonally in front of, and to the right diagonally in front of the seat St. The shape and size of the screens (image display areas) of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be the same or different.

[0043] like Figure 2 As shown, the right edge of the left remote image output device 2211 is adjacent to the left edge of the central remote image output device 2210, so that the image of the central remote image output device 2210 and the image of the left remote image output device 2211 form a tilt angle θ1 (e.g., 120°≤θ1≤150°). Figure 2 As shown, the left edge of the right remote image output device 2212 is adjacent to the right edge of the central remote image output device 2210, so that the screen of the central remote image output device 2210 and the screen of the right remote image output device 2212 form a tilt angle θ2 (e.g., 120°≤θ2≤150°). The tilt angles θ1 and θ2 can be the same or different.

[0044] The screens of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 can be parallel or tilted relative to the vertical direction. At least one of the image output devices 2210, 2211, and 2212 can be composed of multiple image output devices. For example, the central remote image output device 2210 can be composed of a pair of vertically adjacent image output devices with a generally rectangular screen.

[0045] The remote audio output device 222 consists of one or more speakers, such as... Figure 2As shown, it consists of a central remote audio output device 2220, a left-side remote audio output device 2221, and a right-side remote audio output device 2222, respectively located behind the seat St, behind the left armrest, and behind the right armrest. The specifications of the central remote audio output device 2220, the left-side remote audio output device 2221, and the right-side remote audio output device 2222 can be the same or different.

[0046] (Composition of operating machinery)

[0047] like Figure 1 As shown, the operating machinery 40 includes a machine control device 400, a machine input interface 41, and a machine output interface 42. The machine control device 400 is composed of a computing processing unit (a single-core processor, a multi-core processor, or a processor core that constitutes a processor), which reads necessary data and software from storage devices such as memory, and performs computing processing based on the software using the data as the object.

[0048] Construction machinery 40 includes, for example, tracked excavators (construction machinery) with hybrid drive, such as hydraulic, electric, or a combination of hydraulic and electric powertrains. Figure 3 As shown, the device includes a tracked lower traveling body 410 and an upper rotating body 420 rotatably mounted on the lower traveling body 410 via a rotating mechanism 430. A driver's cab 424 is provided on the front left side of the upper rotating body 420. A working mechanism 440 is provided on the front center of the upper rotating body 420.

[0049] The machine input interface 41 includes a machine operation mechanism 411, a machine imaging device 412, and a machine status sensor group 414. The machine operation mechanism 411 has multiple joysticks arranged in the same manner as the remote operation mechanism 211 around the seat located inside the cab 424. A drive mechanism or robot is provided in the cab 424 to receive signals corresponding to the operation mode of the remote joysticks and to actuate the joysticks based on these received signals. The machine imaging device 412 is, for example, located inside the cab 424, and captures images of the environment, including at least a portion of the operating mechanism 440, through the front window and a pair of side windows. Alternatively, part or all of the front window (or window frame) and side windows may be omitted. The actual machine status sensor group 414 consists of an angle sensor, a slewing angle sensor, an external force sensor, and a 3-axis acceleration sensor. The angle sensor is used to measure the rotation angle (lifting and lowering angle) of the boom 441 relative to the upper slewing body 420, the rotation angle of the stick 443 relative to the boom 441, and the rotation angle of the bucket 445 relative to the stick 443. The slewing angle sensor is used to measure the slewing angle of the upper slewing body 420 relative to the lower traveling body 410. The external force sensor is used to measure the external force acting on the bucket 445. The 3-axis acceleration sensor is used to measure the 3-axis acceleration acting on the upper slewing body 420.

[0050] The actual output interface 42 includes an actual image output device 421 and an actual wireless communication device 422. The actual image output device 421 is, for example, located inside the cab 424 and near the front window (see reference). Figure 6 (and Figure 9). The actual image output device 421 can also be omitted.

[0051] The working mechanism 440 includes: a boom 441, which is mounted on the upper rotating body 420 for lifting and lowering; a stick 443, which is rotatably connected to the front end of the boom 441; and a bucket 445, which is rotatably connected to the front end of the stick 443. The working mechanism 440 is equipped with a boom cylinder 442, a stick cylinder 444, and a bucket cylinder 446, all composed of telescopic hydraulic cylinders. In addition to the bucket 445, various accessories such as hydraulic shears, cutters, and magnetic tools can be used as the working unit.

[0052] Boom cylinder 442 is located between boom 441 and upper slewing body 420, allowing it to extend and retract by receiving a supply of working oil, thereby causing boom 441 to rotate in the lifting direction. Stick cylinder 444 is located between stick 443 and boom 441, allowing it to extend and retract by receiving a supply of working oil, thereby causing stick 443 to rotate about a horizontal axis relative to boom 441. Bucket cylinder 446 is located between bucket 445 and stick 443, allowing it to extend and retract by receiving a supply of working oil, thereby causing bucket 445 to rotate about a horizontal axis relative to stick 443.

[0053] (The composition of unmanned aerial vehicles)

[0054] The unmanned aerial vehicle 60 is a rotorcraft, equipped with multiple blades, electric motors (actuators) for rotating the blades, and batteries for supplying power to the electric motors, etc. The unmanned aerial vehicle 60 can be remotely operated via a remote operation device consisting of a remote input interface 210 and / or a physical input interface 41. The unmanned aerial vehicle 60 can also be a component of a work machinery 40. In this case, the work machinery 40 may also include a platform for the unmanned aerial vehicle 60 to take off and land.

[0055] The unmanned aerial vehicle 60 includes an aircraft control device 600, an aircraft input interface 610, and an aircraft output interface 620. The aircraft control device 600 consists of a computing processing unit (a single-core processor, a multi-core processor, or a processor core that constitutes a processor), which reads necessary data and software from storage devices such as memory, and performs computing processing based on the data and the software.

[0056] The aircraft input interface 610 includes an aircraft imaging device 612 and an aircraft status sensor group 614. The aircraft imaging device 612 is configured to orient its optical axis in all directions via an actuator. The aircraft status sensor group 614 consists of positioning sensors, velocity sensors and / or acceleration sensors, gyroscope sensors, etc., used to measure the vertical and horizontal position, velocity, and / or attitude of the unmanned aerial vehicle 60. Latitude (Y coordinate value) and longitude (X coordinate value) in the world coordinate system (or actual spatial coordinate system) are measured using a positioning device such as GPS mounted on the unmanned aerial vehicle 60. Absolute altitude or barometric altitude (Z coordinate value) in the world coordinate system is measured using a TOF sensor or a barometric pressure sensor.

[0057] The aircraft output interface 620 includes an aircraft wireless communication device 622, an aircraft projection device 624, and an aircraft sound output device 626. The aircraft projection device 624 is configured to include a light-emitting element such as an LED and / or laser, a driver for controlling the light-emitting action of the light-emitting element, and an optical system drive mechanism for orienting light from the light-emitting element in a specified direction, projecting a marked image onto a predetermined area below the unmanned aerial vehicle 60. The aircraft sound output device 626 is configured to output sound (e.g., directional sound) towards a predetermined area below the unmanned aerial vehicle 60.

[0058] (Composition of mobile terminal devices)

[0059] The mobile terminal device 80 is, for example, a smartphone, designed to be of a size and weight that can be carried by an operator. The mobile terminal device 80 includes a terminal control device 800, a terminal input interface 810, and a terminal output interface 820. The terminal control device 800 is composed of a processing unit (a single-core processor, a multi-core processor, or a processor core constituting a processor), which reads necessary data and software from storage devices such as memory, and performs processing based on the software using that data.

[0060] The terminal input interface 810 includes a terminal imaging device 812 and a terminal status sensor group 814. The terminal status sensor group 814 consists of positioning sensors, speed sensors and / or acceleration sensors, gyroscope sensors, etc., used to measure the position, speed, and / or posture of the mobile terminal device 80 in the vertical and horizontal directions. The terminal output interface 820 includes a terminal image output device 821 and a terminal wireless communication device 822.

[0061] (Function 1)

[0062] Figure 4 This is a flowchart describing the first function of the basic functions of the work support system, which operates through the cooperation of the work support server 10, remote operating device 20, and work machinery 40. In this flowchart, for the sake of simplicity, boxes such as "C●" are used to indicate the transmission and / or reception of data, and to indicate conditional branches in which processing in a branch direction is performed based on the transmission and / or reception of such data.

[0063] In the remote operation device 20, it is determined whether a specified operation has been performed by the operator through the remote input interface 210. Figure 4 / Step 210). For example, "specified operation" is used to specify an operation such as a tap in the remote input interface 210 for the operating machinery 40 that the operator intends to remotely operate. In the case where the determination result is negative ( Figure 4 / Step 210 (No) ends the series of processes. On the other hand, if the determination result is positive ( Figure 4 / Step 210 (Yes), send an environment confirmation request to the job support server 10 via the remote wireless communication device 224 ( Figure 4 / Step 212).

[0064] In the operation support server 10, upon receiving an environmental confirmation request, the first support processing element 121 sends the environmental confirmation request to the corresponding operation machine 40. Figure 4 / C10).

[0065] In the operating machinery 40, when an environmental confirmation request is received via the actual machine wireless communication device 422 ( Figure 4 / C40), the actual control device 400 acquires the captured image through the actual shooting device 412 ( Figure 4 / Step 410). The machine control unit 400 can also acquire images via the aircraft imaging device 612 mounted on the unmanned aerial vehicle 60, based on communication between the working machine 40 and the unmanned aerial vehicle 60 flying around it. The machine control unit 400 then sends the image data representing the captured image to the work support server 10 via the machine wireless communication device 422. Figure 4 / Step 412).

[0066] In the job support server 10, when the captured image data is received by the first support processing element 121 ( Figure 4 / C11), the second support processing element 122 sends environmental image data corresponding to the captured image to the remote operation device 20. Figure 4 / Step 110). Environmental image data is not only the captured image data itself, but also image data representing simulated environmental images generated based on the captured images.

[0067] In the remote operation device 20, when environmental image data is received via the remote wireless communication device 224 ( Figure 4 / C21), the remote control device 200 outputs the environmental image corresponding to the environmental image data to the remote image output device 221 (C21), the remote control device 200 outputs the environmental image to the remote image output device 221 (C21). Figure 4 / Step 214).

[0068] Therefore, for example Figure 6 As shown, the environmental image of the boom 441, stick 443 and bucket 445, which are part of the working mechanism 440, is output to the remote image output device 221.

[0069] In the remote operation device 20, the remote control device 200 identifies the operation mode of the remote operation mechanism 211. Figure 4 / Step 216), and send a remote operation command corresponding to this operation mode to the job support server 10 via the remote wireless communication device 224. Figure 4 / Step 218).

[0070] In the operation support server 10, upon receiving the remote operation command by the second support processing element 122, the first support processing element 121 sends the remote operation command to the operating machine 40. Figure 4 / C12).

[0071] In the operating machinery 40, when the actual machine control device 400 receives an operation command via the actual machine wireless communication device 422 ( Figure 4 / C41), controls the actions of the operating mechanism 440, etc. Figure 4 / Step 414). For example, performing an operation in which the bucket 445 scoops up the soil in front of the working machine 40 and dumps the soil from the bucket 445 after the upper rotating body 420 is rotated.

[0072] (Function 2)

[0073] Figure 5 This is a flowchart describing the second function of the operational support system, which operates through the cooperation of the operational support server 10, remote operating device 20, operational machinery 40, unmanned aerial vehicle 60, and mobile terminal device 80. In this flowchart, for the sake of simplicity, boxes such as "C●" are used to indicate data transmission and / or reception, and to indicate conditional branches in which processing in a branch direction is performed based on the transmission and / or reception of such data.

[0074] In the operating machinery 40, the machine control device 400 acquires machine positioning data (determined by latitude and longitude) representing the position of the operating machinery 40 through positioning sensors constituting the machine status sensor group 414. Figure 5 / Step 420). The machine control device 400 sends the machine positioning data to the operation support server 10 via the machine wireless communication device 422. Figure 5 / Step 422).

[0075] In the job support server 10, upon receiving real-machine location data ( Figure 5 / C14), the actual machine position as the location of the working machine 40 is identified by the first support processing element 121. Figure 5 / Step 121).

[0076] Next, the second support processing element 122 generates actual navigation flight commands and sends them to the corresponding unmanned aerial vehicle 60. Figure 5 (Step 122) The actual aircraft navigation flight command includes a time series of the unmanned aerial vehicle (UAV) 60's position, i.e., the UAV target position track, used to navigate the work machinery 40 according to the actual aircraft target position track registered in the database 102. The actual aircraft target position track can also be set in the remote operation device 20 via the remote input interface 210 and then registered in the database 102. Alternatively, instead of the actual aircraft target position track, a time series of the future position of the work machinery 40 predicted based on the past time series of the actual aircraft position can be used, i.e., the predicted position track.

[0077] To avoid interference between the work machinery 40 and the unmanned aerial vehicle 60, the actual navigation flight commands may also include the measurement results of the position and attitude in the work machinery coordinate system (a coordinate system whose position and attitude are fixed relative to the upper rotating body 420) of the boom 441, stick 443, and bucket 445. These measurement results are calculated, for example, based on the angles of the connection mechanism (or joint mechanism) between the upper rotating body 420 and boom 441, the connection mechanism between boom 441 and stick 443, and the connection mechanism between stick 443 and bucket 445, as well as the dimensions of the boom 441, stick 443, and bucket 445.

[0078] In the unmanned aerial vehicle 60, when the aircraft control device 600 receives actual aircraft navigation flight commands via the aircraft wireless communication device 622 ( Figure 5 / C61), according to the target position orbit (or actual target position orbit) contained in the instruction, control the flight mode of the unmanned aerial vehicle 60. Figure 5 / Step 621). Specifically, the aircraft control device 600 controls the rotation of the actuator (electric motor) and multiple blades powered by the actuator according to the flight command signal. Thus, the unmanned aerial vehicle 60 can fly, ascend or descend, or remain suspended in the air by the airflow generated by the rotation of the multiple blades. Therefore, as... Figure 7 As shown, the unmanned aerial vehicle 60 flies in front of the work machinery 40 in a manner that navigates or guides the work machinery 40 from the first position P1 to the second position P2.

[0079] In the remote operating device 20, the operator refers to an image of the work environment (reference image)... Figure 6 The remote control mechanism 211 is operated by manipulating the travel direction of the unmanned aerial vehicle 60 as reflected in the image, thereby remotely controlling the work machinery 40 by tracking the unmanned aerial vehicle 60 (see reference). Figure 4 (Steps 216 → 218 → ... → 414). Alternatively, the operator in the cockpit 424 can operate the control sticks and other components of the machine's control mechanism 411 by referring to the direction of travel of the unmanned aerial vehicle 60 located in front of the cockpit 424, thereby controlling the work machinery 40 in a way that tracks the unmanned aerial vehicle 60. Furthermore, the work machinery 40 can also be controlled by the machine control device 400 to include the unmanned aerial vehicle 60 in the work environment image or a designated image area that is part of it, i.e., to make the work machinery 40 track the unmanned aerial vehicle 60.

[0080] In the mobile terminal device 80, the terminal control device 800 acquires terminal positioning data (determined by latitude and longitude) representing the location of the mobile terminal device 80 (and consequently, the location of the operator carrying the mobile terminal device) via positioning sensors constituting the terminal status sensor group 814. Figure 5 / Step 820). The terminal control device 800 sends terminal location data to the job support server 10 via the terminal wireless communication device 822. Figure 5 / Step 822).

[0081] The operator's location can also be identified based on the position in images acquired by the on-machine camera 412 mounted on the work machinery 40 and / or the drone camera 612 mounted on the drone 60, and the position of the work machinery 40 and / or the drone 60. From this perspective, the functions of the mobile terminal device 80 or a portion thereof can be omitted. In the work support server 10, upon receiving terminal positioning data ( Figure 5 / C16), the location of the mobile terminal device 80 is identified by the first support processing element 121. Figure 5 / Step 123).

[0082] Next, the second support processing element 122 determines whether there are any operators whose current location (or predicted future location) is included in the first designated area R1. Figure 5 (Step 124). The first designated area R1 is an area defined based on the position of the operating machine 40. For example, as... Figure 8A and Figure 8B As shown, the current position of the working machine 40 can also be included and biased towards the direction of travel of the working machine 40 (see reference). Figure 8A and Figure 8B The area marked with a dashed arrow is defined as the first designated area R1. Alternatively, the area relative to the working machine 40 in its direction of travel or away from it can also be defined as the first designated area R1. The shape of the first designated area R1 on the horizontal plane or ground can be various shapes besides a circle, such as an ellipse, triangle, rectangle, trapezoid, or regular N-angle (N being an integer greater than 5). The size of the first designated area R1 can be determined based on the spatial volume or width occupied by the working machine 40 and / or the speed of movement of the working machine 40.

[0083] like Figure 8A As shown, if the first operator W1 is included in the first designated area R1, the determination result is positive for the first operator W1. On the other hand, as Figure 8A As shown, if the second operator W2 deviates from the first designated area R1, the determination result is negative for the second operator W2.

[0084] It can also replace this judgment process ( Figure 5 / Step 124) or, in addition to this determination process, determine whether the second designated region R2 overlaps with the first designated region R1, wherein the second designated region R2 is the predicted location of the operator based on the time series of the operator's location, or an area extended based on this. For example, as Figure 8B As shown, if the second designated area R2 (W1), extended based on the predicted position trajectory of the first operator W1 (refer to the dotted arrow), does not overlap with the first designated area R1, the determination result is negative for the first operator W1. On the other hand, as... Figure 8B As shown, when the second designated area R2 (W2), extended from the predicted position trajectory of the second worker W2 (refer to the double-dotted arrow), overlaps with the first designated area R1, the determination result is positive for the second worker W2. The shape of the second designated area R2 on the horizontal plane or ground can be various shapes besides a circle, such as an ellipse, triangle, rectangle, trapezoid, or regular N-angle (N is an integer greater than 5). The size of the second designated area R2 can also be determined based on the worker's movement speed. For example, the second designated area can be set as the work area specified for the worker based on the work plan.

[0085] In the case where the judgment result is negative ( Figure 5 / Step 124‥No), repeat the actual aircraft navigation flight command and transmission processing corresponding to the reception of actual aircraft positioning data ( Figure 5 / C14→Step 121→Step 122).

[0086] On the other hand, if the determination result is affirmative ( Figure 5 / Step 124 (Yes), send a real-time status notification command containing the location of the corresponding operator to the corresponding unmanned aerial vehicle 60. Figure 5 (Step 126). At this time, the second support processing element 122 can also output information about moving the unmanned aerial vehicle 60 to the operator's position from below and facing position to the remote output interface 220.

[0087] In the unmanned aerial vehicle 60, when the aircraft control device 600 receives a real-time status notification command via the aircraft wireless communication device 622 ( Figure 5 / C62), to control the flight mode of the unmanned aerial vehicle 60 in a manner that is toward the position of the operator contained in the instruction ( Figure 5 / Step 622). Therefore, as Figure 8A As shown, the unmanned aerial vehicle 60 flies towards the corresponding first operator W1 (refer to the dashed arrow). Furthermore, as... Figure 8BAs shown, the unmanned aerial vehicle 60 is flying toward the corresponding second operator W2 (refer to the dashed arrow).

[0088] Furthermore, when the unmanned aerial vehicle 60 reaches the operator's position from below and facing him, the aircraft projection device 624 is controlled by the aircraft control device 600 to project a marker image onto the area surrounding the operator. Figure 5 (Step 623). The "identification image" is an image representing the movement pattern of the working machine 40, determined by its speed (including direction of movement), acceleration or deceleration, and / or distance from the operator. For example, the movement pattern of the working machine 40 can be identifiable by the shape, color, or pattern of the identification image, or any combination thereof, or by differences in their temporal variations. The operator's surrounding area is an area extended from the operator's perspective; for example, it can also be an area tilted towards the location of the working machine 40 from the operator's viewpoint.

[0089] Thus, for example, in Figure 8A In the situation shown, as Figure 9A As shown, image M is a projection of the area S(R1) surrounding the first operator W1 by unmanned aerial vehicle 60. Furthermore, in Figure 8B In the situation shown, as Figure 9B As shown, the image M is a projection of the surrounding area S(R2) of the second operator W2 by the unmanned aerial vehicle 60.

[0090] If the operator's position is contained in the first designated area R1, the projection method of the identification image M can be differentiated based on whether the second designated area R2 corresponding to the operator's predicted position trajectory overlaps with the first designated area R1.

[0091] The second support processing element 122 can also cause the unmanned aerial vehicle 60 to project a location marker image M onto the location of the work machinery 40 (current position and / or predicted future position) identified by the first support processing element 121, based on the operator's position, or the orientation relative to its time series within the operator's surrounding area. For example, if the current position of the work machinery 40, as seen from the operator, is in the east, the location marker image M can be projected in the east direction within the operator's surrounding area. Subsequently, if the work machinery 40, as seen from the operator, moves to the northeast direction, the location marker image M can be projected in the northeast direction within the operator's surrounding area.

[0092] The second support processing element 122 can also, based on the positions of the work machinery 40 and the operator identified by the first support processing element 121, cause the unmanned aerial vehicle 60 to project an image M representing the orientation of the work machinery 40 (current position and / or predicted future position) or its time series relative to the operator's position onto the area surrounding the operator. For example, as Figure 10A As shown, when the orientation of the working machine 40 in the time series is a straight line relative to the current position of the working machine 40, the unmanned aerial vehicle 60 can project a marker image M representing the shape of the straight line onto the surrounding area of ​​the operator. Furthermore, as... Figure 10B As shown, when the orientation of the working machine 40 in the time series is a curve relative to the current position of the working machine 40, that is, when the working machine 40 turns to the right or left, the unmanned aerial vehicle 60 can also project the identification image M representing the curve shape onto the surrounding area of ​​the operator.

[0093] like Figure 9C As shown, when both the first operator W1 and the second operator W2 are contained within the first designated area R1, and the distance between them is small enough that the surrounding areas S(R1) and S(R2) overlap, the unmanned aerial vehicle 60 projects a marker image M onto the overlapping area. Alternatively, when both the first operator W1 and the second operator W2 are contained within the first designated area R1, and the distance between them is large enough that the surrounding areas S(R1) and S(R2) do not overlap, the unmanned aerial vehicle 60 projects marker images M onto each of the surrounding areas S(R1) and S(R2) sequentially. Alternatively, the projection time of the marker image M can be controlled such that the more operators are targeted, the longer the projection time. The projection order of the marker images M can also be determined based on the distance between the operating machine 40 and each operator (e.g., the shorter the distance, the earlier the projection).

[0094] In the unmanned aerial vehicle 60, when the aircraft control device 600 receives a real-time status notification command via the aircraft wireless communication device 622 ( Figure 5 / C62), or alternatively, the projection of the identification image M or, in addition to the projection of the identification image M, the directional sound output from the aircraft sound output device 626 toward the position of the operator included in the instruction.

[0095] In the mobile terminal device 80 carried by the operator who is the projected object of the identification image M, the terminal control device 800 determines whether there is an identification confirmation operation through the terminal input interface 810. Figure 5 / Step 824).

[0096] In the case where the judgment result is negative ( Figure 5 / Step 824‥No), repeat the process of obtaining terminal location data ( Figure 5 The processing after step 820. On the other hand, if the determination result is positive ( Figure 5 / Step 824 (Yes), send identification confirmation data to the job support server 10 ( Figure 5 / Step 826).

[0097] In the job support server 10, upon receiving the identification confirmation data ( Figure 5 / C18), the second support processing element 122 sends a real-time status notification stop command to the corresponding unmanned aerial vehicle 60 ( ). Figure 5 / Step 128).

[0098] In the unmanned aerial vehicle 60, when the aircraft control device 600 receives a stop command from the actual aircraft status notification via the aircraft wireless communication device 622 ( Figure 5 / C64), stop the unmanned aerial vehicle 60 from projecting the identification image M of the area surrounding the operator contained in the instruction. Figure 5 / Step 624). Without a stop command from the actual aircraft status, the projection of the marker image M by the UAV 60 automatically stops after a specified period, starting from the initial projection of the marker image M by the UAV 60. Afterwards, control of the flight mode of the UAV 60 corresponding to the received navigation flight commands is executed. Figure 5 / C61→Step 621). For the combination of the working machine 40 and the operator that are the projection objects of the identification image continuously within a specified number of times (e.g., once), the projection of the identification image indicating the movement mode of the working machine 40 onto the area surrounding the operator may be omitted.

[0099] (Effect)

[0100] According to the configured operation support system, the unmanned aerial vehicle 60 projects a marker image M (refer to) onto the area surrounding the operator (e.g., the ground near the operator to a degree that the operator can visually recognize). Figures 9A to 9C The identification image M is an image representing the movement mode of the operating machinery 40. Therefore, compared to projecting the identification image M onto a location independent of the operator's position, it is possible to reliably notify the operator of the movement mode of the operating machinery 40 regardless of the length of the distance between the operating machinery 40 and the operator.

[0101] (Other embodiments of the present invention)

[0102] In the above embodiment, the job support system is constituted by the job support server 10. However, in other embodiments, at least a portion of the functions of the job support system may also be performed by the remote operation device 20, the work machine 40, the unmanned aerial vehicle 60, and / or the mobile terminal device 80. For example, regarding the second function (see...) Figure 5 Alternatively, the unmanned aerial vehicle 60 can perform the function of the second support processing element 122.

[0103] In the above embodiment, one working machine 40 and one unmanned aerial vehicle 60 cooperate. However, in other embodiments, one working machine 40 may also cooperate with multiple unmanned aerial vehicles 60.

[0104] For example, such as Figure 11A As shown, when there is one operator among multiple operators W1 and W2 whose current location is contained in the first designated area R1, one unmanned aerial vehicle 60-1 can be assigned to the projection processing of the identification image M (refer to...). Figure 5 (Steps 622-623), another unmanned aerial vehicle 60-2 is assigned to the navigation flight of the working machine 40 (refer to...) Figure 5 (Step 621). Additionally, as... Figure 11B As shown, when there are two operators W1 and W2 whose current location is contained in the first designated area R1, one UAV 60-1 and another UAV 60-2 can be assigned to the projection processing of the identification image M (see reference). Figure 5 / Step 622→Step 623).

[0105] The second support processing element 122 can also enable the work machinery 40, which indicates the mode of movement through the identification image M projected by the UAV 60 onto the area surrounding the operator, to output identification notifications in a manner that the operator can visually recognize. In this case, the second support processing element 122 can control the UAV 60 to project the identification image M in a manner that corresponds to the work machinery 40's output of the identification notification. Specifically, it can be controlled so that the color and / or flashing mode of the identification notification (light or text on an LED panel) implemented by the light-emitting device constituting the physical output interface 42 of the work machinery 40 is consistent with or reproduces a certain correlation with the color and / or flashing mode of the identification image M projected by the UAV 60.

[0106] According to this operational support system, the operator can identify the movement mode of the operational machine 40 by visually recognizable identification notifications output by the operational machine 40, as indicated by the identification image M projected onto the operator's surrounding area by the unmanned aerial vehicle 60. Therefore, even when multiple operational machines 40 are present at the work site, the operator can clearly determine which operational machine 40's movement mode should be monitored.

[0107] In the above embodiment, based on the presence of an identification confirmation operation in the mobile terminal device 80, the unmanned aerial vehicle 60 stops projecting the identification image M (refer to...). Figure 5 / Step 824 → Step 826 → → Step 624), but as another implementation, the UAV 60 may automatically stop projecting the marker image M after a specified period has elapsed since the UAV 60 began projecting the marker image M. From this point of view, the mobile terminal device 80 or some of its functions may also be omitted.

[0108] In the above embodiment, according to the second support processing element 122, in the case where there is no operator whose current location (or future predicted location) is included in the first designated area R1 ( Figure 5 / Step 124 (No), the projection of the identification image M by the unmanned aerial vehicle 60 is not performed. However, as another implementation, the flight mode of the unmanned aerial vehicle 60 can also be controlled by the aircraft control device 600 based on the aircraft target position trajectory (or the actual aircraft target position trajectory) contained in the actual aircraft navigation flight command received through the aircraft wireless communication device 622. Figure 5 (Step 621), and the unmanned aerial vehicle 60 projects the identification image M.

[0109] In addition, such as Figure 12 As shown, when the movement mode of the work machine 40 identified by the first support processing element indicates that the work machine 40 is not moving, the unmanned aerial vehicle 60 can also project an identification image M indicating that the work machine 40 is stopped, as a movement mode of the work machine 40.

[0110] This can be achieved in the job support system of the present invention.

[0111] The first support processing element identifies a first designated area based on a time series of the positions of the operating machinery identified by the first support processing element.

[0112] The second support processing element requires that the location of the operator, as identified by the first support processing element, be included in the first designated area, to enable the unmanned aerial vehicle to project the identification image onto the area surrounding the operator.

[0113] According to the operation support system, the movement mode of the operation machinery can be identified by projecting an image onto the area surrounding the operator, provided that the operator's position is included in a first designated area determined by a time series based on the position of the operation machinery.

[0114] This can be achieved in the job support system of the present invention.

[0115] The first support processing element identifies a second designated area based on a time series of the operator's location identified by the first support processing element.

[0116] The second support processing element causes the unmanned aerial vehicle to project the marked image onto the ground of the first designated area in a different manner, depending on whether at least a portion of the first designated area or at least a portion of the time sequence of the location of the operating machinery is included in the second designated area identified by the first support processing element.

[0117] According to this operational support system, based on the time sequence of the location of the operating machinery or whether a portion of a first designated area determined based on that time sequence is contained within a second designated area determined based on the time sequence of the operator's location, the unmanned aerial vehicle projects identification images around the operator's location in different ways. By varying the projection methods of the identification images, the operator can identify whether a portion of the first designated area is contained within the second designated area (e.g., whether operating machinery heading towards the operator passes through the second designated area).

[0118] This can be achieved in the job support system of the present invention.

[0119] The second support processing element causes the unmanned aerial vehicle to project the marked image onto the location of the operating machinery, or the location of its orientation relative to its time series, in the area surrounding the operator, based on the operator's position and identified by the first support processing element.

[0120] The operation support system configured in this way enables the operator to identify the location of the operating machinery in the orientation of the projected image based on the operator's position.

[0121] This can be achieved in the job support system of the present invention.

[0122] The second support processing element, based on the respective positions of the operating machinery and the operator identified by the first support processing element, causes the unmanned aerial vehicle to project the identification image onto the area surrounding the operator. The identification image represents the position of the operating machinery or the orientation of its time sequence relative to the position of the operator.

[0123] According to the operation support system, by projecting an image (specifically, its design and / or projection method) onto the operator's surrounding area from an unmanned aerial vehicle, the operator can identify the location of the work machinery based on the operator's position or the orientation of its time sequence as the movement mode of the work machinery.

[0124] This can be achieved in the job support system of the present invention.

[0125] The second support processing element enables the work machine to output a notification in a manner that the operator can visually recognize, the work machine being a work machine whose mode of movement is indicated by the marked image projected by the unmanned aerial vehicle onto the area surrounding the operator.

[0126] According to this operational support system, through visually recognizable notifications output by the operating machinery, the operator can identify the movement of the machinery indicated by a marker image projected onto the operator's surrounding area by an unmanned aerial vehicle (UAV). In the presence of multiple operating machines at the work site, this allows the operator to identify which machine's movement should be monitored.

[0127] This can be achieved in the job support system of the present invention.

[0128] The second support processing element enables the unmanned aerial vehicle to project the identification image onto the area surrounding the operator in a manner that corresponds to the working machinery outputting the identification notification, which is represented by the identification image and the movement method.

[0129] According to this operational support system, the operator can identify which machine's movement is indicated by a visually recognizable marker output by the machine itself, projected onto the operator's surrounding area by an unmanned aerial vehicle (UAV). Therefore, even when multiple machines are present at the work site, the operator can clearly determine which machine's movement should be monitored.

[0130] This can be achieved in the job support system of the present invention.

[0131] When the second support processing element moves the unmanned aerial vehicle (UAV) to a position where the operator's position, as identified by the first support processing element, is below the UAV, it outputs information indicating that the UAV was moved to the position where the operator's position is below the UAV to the output interface accessible to the operator of the work machinery.

[0132] The operational support system based on this configuration can reduce or eliminate the operator's sense of disharmony with changes in the flight mode of the unmanned aerial vehicle.

[0133] The operation support composite system of the present invention consists of the operation support system and the unmanned aerial vehicle.

[0134] Explanation of reference numerals in the attached figures

[0135] 10 Operation Support Server

[0136] 20 remote operating devices

[0137] 40 operating machines

[0138] 41 Actual Input Interface

[0139] 42 actual machine output interfaces

[0140] 60 unmanned aerial vehicles

[0141] 80 mobile terminal devices

[0142] 102 Database

[0143] 121 First Support Processing Component

[0144] 122 Second Support Processing Component

[0145] 200 remote control device

[0146] 210 Remote Input Interface

[0147] 211 Remote Operation Mechanism

[0148] 220 Remote Output Interface

[0149] 221 Remote Image Output Device

[0150] 222 Remote Audio Output Device

[0151] 224 Remote Wireless Communication Equipment

[0152] 410 Lower Running Body

[0153] 412 Real-world shooting device

[0154] 414 Real-world status sensor group

[0155] 420 Upper Rotary Body

[0156] 421 Real-world image output device (information output device)

[0157] 422 Real-world wireless communication equipment

[0158] 440 Operational Mechanism (Operational Attachments)

[0159] 445 Bucket (Operations Department)

[0160] 610 aircraft input interface

[0161] 620 aircraft output interface

[0162] 622 aircraft wireless communication equipment

[0163] 800 terminal control device

[0164] 810 Terminal Input Interface

[0165] 820 terminal output interface.

Claims

1. A job support system, characterized in that it is An operation support system equipped with the following components: The first support processing element identifies the positions of the operator and the machine in a time sequence, and identifies a first designated area determined based on the time sequence of the machine's position; The second support processing element controls the flight maneuvers of the unmanned aerial vehicle (UAV) to move it to a position below and facing the operator, as identified by the first support processing element. Furthermore, by controlling the UAV's marker projection function, and requiring that the operator's position, as identified by the first support processing element, is contained within a first designated area, the UAV projects a marker image representing the movement pattern of the work machinery onto the area surrounding the operator. The movement pattern of the work machinery is determined based on a time sequence of the work machinery's position identified by the first support processing element. The size of the first designated area is determined based on the size of the space occupied by the operating machinery and / or the speed of movement of the operating machinery.

2. A job support system, characterized in that, have: The first support processing element identifies the positions of the operator and the machine in a time sequence, and identifies a first designated area determined based on the time sequence of the machine's position; The second support processing element controls the flight maneuver function of the unmanned aerial vehicle (UAV) to make the UAV fly according to the time sequence of the UAV's position, i.e., the target position trajectory of the UAV, which is used for navigation based on the time sequence of the position of the work machinery identified by the first support processing element. This is conditional on the operator's position, identified by the first support processing element, being contained within the first designated area. The UAV then moves to a position where the operator's position, identified by the first support processing element, is below and facing the UAV. Furthermore, by controlling the UAV's marker projection function, a marker image indicating the movement mode of the work machinery is projected onto the area surrounding the operator. The movement mode of the work machinery is determined based on the time sequence of the work machinery's position identified by the first support processing element.

3. The job support system as described in claim 2, characterized in that, The first support processing element identifies a second designated area based on a time series of the operator's location identified by the first support processing element. The second support processing element causes the unmanned aerial vehicle to project the marked image onto the ground of the first designated area in a different manner, depending on whether at least a portion of the first designated area or at least a portion of the time sequence of the location of the operating machinery is included in the second designated area identified by the first support processing element.

4. The operation support system as described in any one of claims 1 to 3, characterized in that, The second support processing element causes the unmanned aerial vehicle to project the marked image onto the location of the operating machinery, or the location of its orientation relative to its time series, in the area surrounding the operator, based on the operator's position and identified by the first support processing element.

5. The operation support system as described in any one of claims 1 to 3, characterized in that, The second support processing element, based on the respective positions of the operating machinery and the operator identified by the first support processing element, causes the unmanned aerial vehicle to project the identification image onto the area surrounding the operator. The identification image represents the position of the operating machinery or the orientation of its time sequence relative to the position of the operator.

6. The operation support system according to any one of claims 1 to 3, characterized in that, The second support processing element enables the work machine to output an identification notification in a manner that the operator can visually recognize, wherein the work machine is a work machine whose movement mode is indicated by the identification image projected by the unmanned aerial vehicle onto the area surrounding the operator.

7. The job support system as described in claim 6, characterized in that, The second support processing element enables the unmanned aerial vehicle to project the identification image onto the area surrounding the operator in a manner that corresponds to the working machinery outputting the identification notification, which is represented by the identification image and the movement method.

8. The operation support system as described in any one of claims 1 to 3, characterized in that, When the second support processing element moves the unmanned aerial vehicle (UAV) to a position where the operator's position, as identified by the first support processing element, is below the UAV, it outputs information indicating that the UAV was moved to the position where the operator's position is below the UAV to the output interface accessible to the operator of the work machinery.

9. A job support composite system, characterized in that, It comprises the operation support system as described in any one of claims 1 to 3 and the unmanned aerial vehicle.

Images