Work support system and work support complex system
The remote control system uses a work support server and imaging devices to display actual machine identification information, addressing the challenge of distinguishing between work machines during remote control, thereby ensuring correct connections and operations.
Patent Information
- Application Number
- JP2021058131
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-03-30
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2041-03-30
AI Technical Summary
When remotely controlling multiple work machines, operators face difficulties in distinguishing between intended and unintended machines due to similar visual appearances and lack of direct visual confirmation, leading to incorrect slave connections.
A remote control system with a work support server and imaging devices on each machine captures images, extracts actual machine identification information, and displays it alongside selection information to ensure correct machine identification.
Enables operators to visually confirm the match between selected and actual machine identities, preventing incorrect connections and ensuring accurate remote control operations.
Smart Images

Figure 0007582019000001 
Figure 0007582019000002 
Figure 0007582019000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a remote operation system having a function of mutual communication between a plurality of work machines and a remote operation device for remotely operating one of the plurality of work machines that is to be remotely operated. [Background technology]
[0002] Conventionally, a work machine has been operated remotely without an operator riding on the work machine. For example, Patent Document 1 discloses a remote operation system for such a work machine in which a master and a slave are provided so as to be able to communicate with each other, and an operator operates an operation lever provided on the master, whereby the slave, located in the driver's seat of the work machine, operates the operation lever of the work machine. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2019-176401 A Summary of the Invention [Problem to be solved by the invention]
[0004] Incidentally, when one master is used to remotely operate two or more work machines, the master and the slaves provided on each of the work machines must be connected so that they can communicate with each other.
[0005] Furthermore, when operating one work machine, the operator selects one work machine from among a plurality of work machines, and switches between the work machines to be operated.
[0006] However, for example, if the slave connection settings are made incorrectly, the machine may be connected to a different slave than the one selected by the operator. When the operator actually gets on the work machine, he or she can check the appearance of the work machine and the state of the work site where the work machine is located before getting on, but when remotely operating the work machine, these checks are not possible, so it is difficult for the operator to notice that the machine has been connected to an unintended work machine (different slave). The imaging device Even if you try to check the images, they are similar and difficult to distinguish.
[0007] The present invention has been made in consideration of such problems, and has an object to provide a remote system that is capable of outputting information on a work machine selected by an operator and on slaves connected to a master. [Means for solving the problem]
[0008] In order to achieve this object, The first aspect The remote operation system includes a plurality of work machines, a remote operation device for remotely operating one of the plurality of work machines that is to be remotely operated, and A work support server; a remote control system having a mutual communication function with each of the plurality of work machines, and a work machine that is a candidate for a remote control target among the plurality of work machines is provided with an actual machine imaging device capable of capturing an image, and a remote control system having a remote control function for remotely controlling the work machine, the remote control system being provided with a remote control function for remotely controlling the work machine. A description as character information assigned to the candidate work machine. It is equipped with the actual machine identification information and The work support server includes: A method for determining whether or not one work machine to be remotely operated has been selected from among the plurality of work machines, and when it is determined that the one work machine has been selected, a method for connecting a cooperative work machine identified by an identifier corresponding to selection information, which is information about the selected one work machine, to the remote operation device, and acquiring the image captured by the actual machine imaging device of the cooperative work machine. Processing and The image captured by the actual machine imaging device of the collaborative work machine and the selection information to the output interface of the remote control device. Processing and executing It is characterized by:
[0009] According to the remote control system of the present invention, an image including actual machine identification information and Selection Information is output from an output interface of a remote control device, so that the image is displayed on the remote control device, for example. and Selection Information Therefore, the operator can display the image and Selection Information Therefore, by checking whether the selection information and the actual machine identification information are different, the operator can confirm whether the work machine connected to the remote control device is the same as the work machine to be operated.
[0010] A remote operation system according to a second aspect of the present invention is a remote operation system having mutual communication functions with a plurality of work machines, a remote operation device for remotely operating one of the plurality of work machines to be the remotely operated target, and a work support server, wherein a work machine that is a candidate to be the remotely operated target among the plurality of work machines is equipped with an actual machine imaging device capable of capturing images, and actual machine identification information that is captured so as to be included in the image captured by the actual machine imaging device, and the work support server determines whether or not one of the plurality of work machines to be the remotely operated target has been selected, and when it is determined that the one work machine has been selected, the remote control device, acquiring the image captured by the actual machine imaging device of the coordinated work machine, and outputting the image captured by the actual machine imaging device of the coordinated work machine to an output interface of the remote control device; and determining whether or not the image captured by the actual machine imaging device of the coordinated work machine includes actual machine identification information, and if it is determined that the image includes the actual machine identification information, determining consistency between the actual machine identification information and the selection information, and outputting the determination result of the consistency to the output interface of the remote control device.
[0011] According to this aspect, the result of determining whether the selection information and the actual device identification information are consistent is transmitted to the remote control device, so that, for example, the result of the determination can be displayed on the remote control device.
[0012] A remote operation system according to a third aspect of the present invention is a remote operation system having mutual communication functions between a plurality of work machines, a remote operation device for remotely operating one of the plurality of work machines which is to be the remotely operated target, and a work support server, wherein a work machine which is a candidate to be the remotely operated target among the plurality of work machines is equipped with an actual machine imaging device capable of capturing images, and actual machine identification information which is captured so as to be included in the image captured by the actual machine imaging device, and the work support server determines whether or not a work machine which is to be the remotely operated target among the plurality of work machines has been selected, and when it is determined that the one work machine has been selected, connects a cooperating work machine identified by an identifier corresponding to selection information which is information on the selected one work machine to the remote operation device, and transmits an image captured by the actual machine imaging device of the cooperating work machine to the remote operation device. The present invention is characterized in that the method executes the following processes: a process of acquiring the image, and outputting the image captured by the actual machine imaging device of the coordinated work machine to an output interface of the remote control device; and, when the actual machine identification information included in the image differs from the actual machine identification information corresponding to the selection information, a process of acquiring a candidate image captured by the actual machine imaging device of a work machine other than the coordinated work machine among the plurality of work machines, acquiring consistency information from the remote control device regarding whether the work machine equipped with the actual machine imaging device that captured the candidate image is a true coordinated work machine, outputting the candidate image to the output interface of the remote control device, and, when the consistency information indicates the true coordinated work machine, rewriting the identifier corresponding to the coordinated work machine to the identifier of the work machine equipped with the actual machine imaging device that captured the candidate image.
[0013] According to this aspect, the identifier corresponding to the cooperative work machine is changed to the identifier of the work machine equipped with the actual machine imaging device that captured the candidate image based on the operator's intention, so that the operator can recognize the timing when the identifier is changed.
[0014] In the remote operation systems according to the second and third aspects of the present invention, it is preferable that the work support server executes a process of outputting the selection information to an output interface of the remote operation device. According to this aspect, for example, the selection information can be displayed on the remote control device, which allows the operator to visually check the image including the actual machine identification information and the selection information.
[0015] In addition, in the remote control system of the present invention, it is preferable that the work support server executes a process of acquiring the selection information inputted to an input interface of the remote control device, and a process of outputting the acquired selection information to an output interface of the remote control device. [Brief description of the drawings]
[0016] [Figure 1] 1 is a diagram illustrating the configuration of a work support complex system according to an embodiment of the present invention; [Diagram 2] FIG. 2 is an explanatory diagram showing an example of a data structure stored in a database of a work support server. [Diagram 3] FIG. 2 is an explanatory diagram relating to a configuration of a remote control device. [Figure 4] FIG. 2 is an explanatory diagram showing an example of a screen display of a remote control device. [Diagram 5] FIG. 2 is an explanatory diagram relating to the configuration of a work machine. [Figure 6] FIG. 11 is an explanatory diagram showing an example of an image captured by a real imaging device. [Figure 7] 13 is a flowchart showing an example of a work support process performed by the work support server. [Figure 8] 13 is a flowchart showing an example of another work support process performed by the work support server. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS EXAMPLES
[0017] (Configuration of remote control system) The work support system as one embodiment of the present invention shown in Fig. 1 is configured with a work support server 10 for supporting remote operation of a first work machine 40 and a second work machine 50 by a remote operation device 20. The work support server 10, the remote operation device 20, the first work machine 40, and the second work machine 50 are configured to be able to communicate with each other via a common or separate network. In this figure, the work support server 10 is connectable to the first work machine 40 or the second work machine 50, but the work support server 10 only needs to be connectable to two or more work machines, and there is no limit to the number of connected machines.
[0018] In this embodiment, the work support server 10 is described as a remote operation system having a function of communicating with a plurality of work machines and a remote operation device for remotely operating one of the plurality of work machines that is to be remotely operated. Also, in this embodiment, an example of a work support system in which a composite system is configured by the work support server 10, the remote operation device 20, a first work machine 40, and a second work machine 50 is described.
[0019] The components (hardware) of the present invention "acquire" various information, which encompasses all types of computational processing for preparing various information in a form usable in subsequent computational processing, such as receiving the information, reading or searching the information from an internal storage device (e.g., a memory) and / or an external storage device (e.g., an external database server), and performing computational processing on the received, read, searched, etc. information to calculate, estimate, predict, identify, etc. the information.
[0020] (Work support server configuration) The work support server 10 includes a database 102, a first support processing element 121, and a second support processing element 122. The database 102 stores and holds captured image data and the like. The database 102 may be configured as a database server separate from the work support server 10. As shown in FIG. 2, the database 102 holds an identifier (e.g., an IP address assigned to each work machine) for identifying a work machine on a network assigned to each first work machine 40 or second work machine 50, and actual machine identification information including information on the work machine corresponding to the identifier assigned to each first work machine 40 or second work machine 50. Note that the actual machine identification information may be any information capable of identifying one work machine from another work machine, and may be, for example, a unique number such as a serial number, a two-dimensional code holding information such as a unique number, or a figure such as a circle or triangle.
[0021] The first support processing element 121 and the second support processing element 122 are composed of a calculation processing device (a single-core processor or a multi-core processor or a processor core that constitutes the same), read the necessary data and software from a storage device such as a memory, and perform the calculation processing described below in accordance with the software on the data.
[0022] (Configuration of remote control device) The remote operation device 20 includes a remote control device 200, a remote input interface 210, and a remote output interface 220. The remote control device 200 is configured with an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing on the data according to the software.
[0023] The remote input interface 210 includes a remote operation mechanism 211 and a handheld terminal 2240. The remote output interface 220 includes a remote image output device 221, a remote audio output device 222, a remote wireless communication device 224, and a handheld terminal 2240. The handheld terminal 2240 serves as both the remote input interface 210 and the remote output interface 220.
[0024] The remote operation mechanism 211 includes a traveling operation device, a slewing operation device, a boom operation device, an arm operation device, and a bucket operation device. Each operation device has an operation lever that receives a rotation operation. The operation lever (travel lever) of the traveling operation device is operated to move the lower traveling body 410 of the work machine 40. The travel lever may also serve as a travel pedal. For example, a travel pedal fixed to the base or lower end of the travel lever may be provided. The operation lever (slewing lever) of the slewing operation device is operated to move a hydraulic slewing motor that constitutes the slewing mechanism 430 of the work machine 40. The operation lever (boom lever) of the boom operation device is operated to move the boom cylinder 442 of the work machine 40. The operation lever (arm lever) of the arm operation device is operated to move the arm cylinder 444 of the work machine 40. The operation lever (bucket lever) of the bucket operation device is operated to move the bucket cylinder 446 of the work machine 40.
[0025] The operation levers constituting the remote operation mechanism 211 are arranged around a seat St on which the operator OP1 sits, as shown in Fig. 3. The seat St is in the form of a high-back chair with armrests, but may be in the form of a low-back chair without a headrest, or in the form of a chair without a backrest, or any other form of seating section on which the operator OP1 can sit.
[0026] A pair of left and right travel levers 2110 corresponding to the left and right crawlers are arranged side by side in front of the seat St. One operation lever may serve as multiple operation levers. For example, the left operation lever 2111 provided in front of the left frame of the seat St shown in FIG. 2 may function as an arm lever when operated in the front-rear direction, and may function as a rotation lever when operated in the left-right direction. Similarly, the right operation lever 2112 provided in front of the right frame of the seat St shown in FIG. 3 may function as a boom lever when operated in the front-rear direction, and may function as a bucket lever when operated in the left-right direction. The lever pattern may be arbitrarily changed by an operation instruction from the operator OP1.
[0027] The handheld terminal 2240 is a display device that is installed diagonally forward and to the right when viewed from the seating position of the seat St, as shown in Fig. 3 for example. The handheld terminal 2420 has an input means such as a touch panel. As shown in Fig. 4 for example, the handheld terminal 2240 is capable of selecting the first work machine 40 or the second work machine 50 to be connected to the remote operation device 20 by using an input means such as a touch panel. Handheld terminal 2240 The display screen and input surface of the handheld terminal 2240 are directed toward the seat St. The handheld terminal 2240 may be configured so that the angle of the display screen can be adjusted.
[0028] Specifically, as shown in FIG. 2240 On the display screen, images of a plurality of work machines A to C that are candidates for connection are displayed in the upper column 22401, Bottom columnIn 22402, work machines A to C are displayed surrounded by an outer frame. In the upper section 22401, a map showing the relative positions of the multiple work machines A to C is displayed. Bottom column In 22402, the machine number (actual machine identification information) and specifications of each of the work machines A to C are displayed. Bottom column When the operator OP1 taps on any one of the images of the work machines A to C in 22402 and selects a work machine to be connected to the remote operation device 20, Bottom column The display form (color of the outer frame, thickness of the outer frame line, blinking of the work machine icon, background color, etc.) of the work machines A to C selected by the tap in 22402 is changed so as to make them distinguishable from non-selected work machines.
[0029] 3, the remote image output device 221 is composed of a central remote image output device 2210, a left remote image output device 2211, and a right remote image output device 2212, each having a substantially rectangular screen, which are respectively arranged in front of the seat St, diagonally forward to the left, and diagonally forward to the right. The shapes and sizes of the screens (image display areas) of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be the same or different.
[0030] The screens of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be parallel to the vertical direction or may be inclined relative to the vertical direction. At least one of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be composed of a plurality of divided image output devices. For example, the central remote image output device 2210 may be composed of a pair of image output devices adjacent to each other above and below, each having a substantially rectangular screen.
[0031] The remote audio output device 222 is composed of one or more speakers, and is composed of a central remote audio output device 2220, a left remote audio output device 2221, and a right remote audio output device 2222, which are arranged behind the seat St, at the rear of the left armrest, and at the rear of the right armrest, respectively, as shown in Fig. 2. The specifications of the central remote audio output device 2220, the left remote audio output device 2221, and the right remote audio output device 2222 may be the same or different.
[0032] (Configuration of the first work machine) 1, the first work machine 40 is equipped with an actual machine control device 400, an actual machine input interface 41, and an actual machine output interface 42. The actual machine control device 400 is configured with an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing on the data in accordance with the software.
[0033] The first work machine 40 is, for example, a hydraulic, electric or hybrid drive type crawler excavator (construction machine) that combines hydraulic and electric drive systems, and as shown in Fig. 5, includes a crawler type lower traveling body 410 and an upper rotating body 420 that is rotatably mounted on the lower traveling body 410 via a rotating mechanism 430. A cab 424 (operator's cabin) is provided on the front left side of the upper rotating body 420. A work mechanism 440 is provided in the front center of the upper rotating body 420.
[0034] The real machine input interface 41 includes a real machine operation mechanism 411, a real machine imaging device 412, and a real machine state sensor group 414. The real machine operation mechanism 411 includes a plurality of operation levers arranged around a seat arranged inside the cab 424 in the same manner as the remote control mechanism 211.
[0035] A drive mechanism or robot is provided in the cab 424 for receiving a signal corresponding to the operation mode of the remote control lever and moving the actual machine control lever based on the received signal.
[0036] The actual machine imaging device 412 is installed, for example, inside the cab 424, and captures an image of the environment including at least a part of the operating mechanism 440 through the front window and a pair of left and right side windows. Some or all of the front window (or window frame) and the side windows may be omitted.
[0037] The actual machine imaging device 412 captures an image including a sticker ST bearing actual machine identification information (No. △△ in FIG. 6) as a unique number assigned to the first work machine 40. The sticker ST bearing the actual machine identification information is affixed, for example, to the front window of the first work machine 40. Images captured by the actual machine imaging device 412 are sequentially transmitted to the work support server 10. Note that the actual machine identification information is not limited to the sticker ST, and may be engraved, for example, on a plate or on the frame of the first work machine 40.
[0038] Furthermore, multiple pieces of actual machine identification information may be provided on the first work machine 40. For example, a sticker ST bearing a unique number and a two-dimensional code holding the unique number may be affixed. When multiple pieces of actual machine identification information are provided on a work machine in this manner, it is not necessary for all pieces of actual machine identification information to be written on the same sticker ST, and for example, multiple stickers ST bearing one piece of actual machine identification information may be affixed.
[0039] The actual machine status sensor group 414 is 420It is composed of angle sensors for measuring the rotation angle (raising and lowering angle) of the boom 441 relative to the boom 441, the rotation angle of the arm 443 relative to the boom 441, and the rotation angle of the bucket 445 relative to the arm 443, a rotation angle sensor for measuring the rotation angle of the upper rotating body 420 relative to the lower running body 410, an external force sensor for measuring the external force acting on the bucket 445, and a three-axis acceleration sensor for measuring the three-axis acceleration acting on the upper rotating body 420.
[0040] The actual machine output interface 42 includes an actual machine image output device 421 and an actual machine wireless communication device 422. The actual machine image output device 421 is disposed, for example, inside the cab 424 and near the front window. The actual machine image output device 421 may be omitted.
[0041] The working mechanism 440 as an operating mechanism includes a boom 441 mounted on the upper rotating body 420 so as to be able to rise and fall, an arm 443 rotatably connected to the tip of the boom 441, and a bucket 445 rotatably connected to the tip of the arm 443. A boom cylinder 442, an arm cylinder 444, and a bucket cylinder 446 each composed of an extendable hydraulic cylinder are attached to the working mechanism 440. In addition to the bucket 445, various attachments such as a nibbler, a cutter, and a magnet may be used as the working unit.
[0042] The boom cylinder 442 is interposed between the boom 441 and the upper rotating body 420 so as to extend and retract when supplied with hydraulic oil to rotate the boom 441 in the hoisting direction. The arm cylinder 444 is interposed between the arm 443 and the boom 441 so as to extend and retract when supplied with hydraulic oil to rotate the arm 443 about a horizontal axis relative to the boom 441. The bucket cylinder 446 is interposed between the bucket 445 and the arm 443 so as to extend and retract when supplied with hydraulic oil to rotate the bucket 445 about a horizontal axis relative to the arm 443.
[0043] (Configuration of the second work machine) 1, the second work machine 50 is equipped with an actual machine control device 500, an actual machine input interface 51, and an actual machine output interface 52. The actual machine control device 500 is configured with an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing on the data in accordance with the software.
[0044] The second work machine 50 is, for example, a hydraulic, electric, or hybrid drive type crawler excavator (construction machine) that combines hydraulic and electric drive systems, and as shown in Fig. 5, includes a crawler type lower traveling body 510 and an upper rotating body 520 that is rotatably mounted on the lower traveling body 510 via a rotating mechanism 530. A cab 524 (operator's cabin) is provided on the front left side of the upper rotating body 520. A work mechanism 540 is provided in the front center of the upper rotating body 520.
[0045] The real machine input interface 51 includes a real machine operation mechanism 511, a real machine imaging device 512, and a real machine state sensor group 514. The real machine operation mechanism 511 includes a plurality of operation levers arranged around a seat arranged inside the cab 524 in the same manner as the remote control mechanism 211.
[0046] A drive mechanism or robot is provided in the cab 524 for receiving a signal corresponding to the operation mode of the remote control lever and moving the actual machine control lever based on the received signal.
[0047] The actual machine imaging device 512 is installed, for example, inside the cab 524, and images the environment including at least a part of the operating mechanism 540 through the front window and a pair of left and right side windows. The front window (or window frame) and the side windows may be omitted in part or in whole.
[0048] The actual machine imaging device 512 captures an image including a sticker ST bearing the actual machine identification information assigned to the second work machine 50. The sticker ST bearing the actual machine identification information is affixed, for example, to the front window of the second work machine 50. The images captured by the actual machine imaging device 512 are sequentially transmitted to the work support server 10 and stored in the database 102. Note that the actual machine identification information is not limited to the sticker ST, and may be engraved, for example, on a plate or on the frame of the second work machine 50.
[0049] The actual machine status sensor group 514 is 520 It is composed of angle sensors for measuring the rotation angle (raising and lowering angle) of the boom 541 relative to the boom 541, the rotation angle of the arm 543 relative to the boom 541, and the rotation angle of the bucket 545 relative to the arm 543, a rotation angle sensor for measuring the rotation angle of the upper rotating body 520 relative to the lower running body 510, an external force sensor for measuring the external force acting on the bucket 545, and a three-axis acceleration sensor for measuring the three-axis acceleration acting on the upper rotating body 520.
[0050] The actual machine output interface 52 includes an actual machine image output device 521 and an actual machine wireless communication device 522. The actual machine image output device 521 is disposed, for example, inside the cab 524 and near the front window. The actual machine image output device 521 may be omitted.
[0051] The working mechanism 540 as an operating mechanism includes a boom 541 mounted on the upper rotating body 520 so as to be able to rise and fall, an arm 543 rotatably connected to the tip of the boom 541, and a bucket 545 rotatably connected to the tip of the arm 543. A boom cylinder 542, an arm cylinder 544, and a bucket cylinder 546 each composed of an extendable hydraulic cylinder are attached to the working mechanism 540. In addition to the bucket 545, various attachments such as a nibbler, a cutter, and a magnet may be used as the working unit.
[0052] The boom cylinder 542 is interposed between the boom 541 and the upper rotating body 520 so as to extend and retract when supplied with hydraulic oil to rotate the boom 541 in the hoisting direction. The arm cylinder 544 is interposed between the arm 543 and the boom 541 so as to extend and retract when supplied with hydraulic oil to rotate the arm 543 about a horizontal axis relative to the boom 541. The bucket cylinder 546 is interposed between the bucket 545 and the arm 543 so as to extend and retract when supplied with hydraulic oil to rotate the bucket 545 about a horizontal axis relative to the arm 543.
[0053] The work support processing of the work support system achieved by the cooperation of the work support server 10, the remote control device 20, the first work machine 40 and the second work machine 50 configured as described above will be described with reference to the flowchart shown in FIG.
[0054] The work support system starts the work support process with the activation of the remote operation device 20 as a trigger. When the remote operation device 20 is activated by the operator OP1 (step S101: Y), the remote operation device 20 displays, for example, the first work machine 40 and the second work machine 50, which are candidates for connection with the remote operation device 20, on the display screen of the remote terminal 2240 in a selectable manner. The operator OP1 selects, for example, the first work machine 40 or the second work machine 50, which is a candidate for connection, by touching the image of the work machine displayed on the remote terminal 2240. The remote operation device 20 determines whether or not a work machine has been selected (step S102). When the remote operation device 20 determines that a work machine has been selected (step S102: Y), it transmits information on the work machine selected by the operator OP1 (selection information) to the work support server 10 (step S103).
[0055] When the work support server 10 receives the selection information, it stores it in the database 102. The work support server 10 starts work support processing when it receives the selection information transmitted from the remote operation device 20. The first support processing element 121 of the work support server 10 determines whether or not a work machine has been selected (step S201).
[0056] The first support processing element 121 of the work support server 10 makes a judgment in step S201 based on whether or not selection information is stored by referring to the database 102. When the first support processing element 121 of the work support server 10 judges that a work machine has not been selected (step S201: N), it ends the work support processing.
[0057] When the first support processing element 121 of the work support server 10 determines that a work machine has been selected (step S201: Y), it refers to the database 102 to acquire selection information (step S202), selects a work machine corresponding to the selection information based on the actual machine identification information, and acquires an image captured by an actual machine imaging device of a cooperative work machine identified by the identifier of the selected work machine (step S203). That is, the first support processing element 121 can acquire selection information in which one of a plurality of work machines, a first work machine 40 or a second work machine 50, which is to be remotely operated, is selected. Furthermore, the first support processing element 121 can connect the cooperative work machine identified by an identifier corresponding to the selection information to the remote operation device 20. Furthermore, the first support processing element 121 can acquire an image captured by an actual machine imaging device of one cooperative work machine connected to the remote operation device 20.
[0058] The second support processing element 122 of the work support server 10 transmits the acquired selection information and image to the remote operation device 20 (step S204), and ends the work support processing.
[0059] The remote operation device 20 displays, for example, the image transmitted from the work support server 10 in step S204 on the remote image output device 221. The operator OP1 visually checks the image displayed on the remote image output device 221 and determines whether the real machine represented by the sticker ST in the image captured by the real machine imaging device of the cooperative work machine connected to the remote operation device 20 is a real machine. identification It is then possible to verify whether the information matches the selection information of the first work machine 40 or the second work machine 50 .
[0060] Specifically, the operator OP1 can visually check whether the actual machine identification information contained in the image displayed on the remote image output device 221 matches the actual machine identification information displayed on the local terminal 2240 together with the selection information.
[0061] In addition, as long as the operator OP1 can confirm the actual machine identification information included in the image and the actual machine identification information corresponding to the selection information, the actual machine identification information does not necessarily need to be displayed on the handheld terminal 2240 together with the selection information. For example, the remote operation device 20 may output the selection information and the actual machine identification information corresponding to the selection information by voice from the remote audio output device 222. Also, the operator OP1 may refer to a paper medium on which the selection information and the actual machine identification information corresponding to the selection information are written, and confirm the correspondence between the selection information and the actual machine identification information included in the image. Matching Verification may be performed.
[0062] If the operator OP1 confirms that there is no match, he or she can take measures such as interrupting the remote operation, restarting the work support system, and contacting the system administrator in order to resolve the mismatch between the actual device identification information and the selection information.
[0063] Further, the remote operation device 20 displays, for example, the content of the operation input to the terminal 2240 at step S102 (content that either the first work machine 40 or the second work machine 50 has been selected) on the terminal 2240. As shown in the example in FIG. Bottom column The display form (color, frame thickness, blinking) of the outer frame of the selected work machine A to C is changed by tapping on 22402, so that the operator OP1 can visually confirm the operation content displayed on the terminal 2240. R This allows the operator OP1 to know which work machine he has selected. confirmation In addition, the contents of the operation may be displayed on the remote image output device 221.
[0064] Furthermore, the remote operation device 20 may display the selection information received from the second support processing element 122 on the handheld terminal 2240 or the remote image output device 221. The operator OP1 can confirm the selection information recognized by the first support processing element 121 of the work support server 10 by visually checking the received selection information. If the received selection information does not match the work machine selected by the operator OP1 in step S102, the state in which the selection information does not match can be resolved by taking measures such as interrupting the remote operation, restarting the work support system, contacting the system administrator, etc. EXAMPLES
[0065] In the first embodiment, the operator OP1 was able to determine whether the actual machine identification information matched the selection information by visually checking the image displayed on the image output device of the remote operation device 20. In the second embodiment, the work support server 10 determines whether the actual machine identification information matched the selection information.
[0066] As shown in FIG. 8, the work support system starts the work support process with the activation of the remote operation device 20 as a trigger. When the remote operation device 20 is activated by the operator OP1 (step S301: Y), the remote operation device 20 displays, for example, on the remote terminal 2240, the first work machine 40 and the second work machine 50 that are candidates for connection with the remote operation device 20, in a selectable manner. The operator OP1 selects, for example, the first work machine 40 or the second work machine 50 that is a candidate for connection by touching the image of the work machine displayed on the remote terminal 2240. The remote operation device 20 determines whether or not a work machine has been selected (step S302). When the remote operation device 20 determines that a work machine has been selected (step S302: Y), it transmits information on the work machine selected by the operator OP1 (selection information) to the work support server 10 (step S303).
[0067] When the work support server 10 receives the selection information, it stores it in the database 102. The work support server 10 starts work support processing when it receives the selection information transmitted from the remote operation device 20. The first support processing element 121 of the work support server 10 determines whether or not a work machine has been selected (step S401).
[0068] The first support processing element 121 of the work support server 10 makes a judgment in step S401 based on whether or not selection information is stored by referring to the database 102. When the first support processing element 121 of the work support server 10 judges that a work machine has not been selected (step S401: N), it ends the work support processing.
[0069] When the first support processing element 121 of the work support server 10 determines that a work machine has been selected (step S401: Y), it refers to the database 102 to acquire selection information (step S402), selects the work machine corresponding to the selection information based on the actual machine identification information, and acquires an image captured by the actual machine imaging device of the cooperative work machine identified by the identifier of the selected work machine (step S403).
[0070] The first support processing element 121 of the work support server 10 has a function of extracting actual machine identification information as character information from an acquired image by character recognition means such as OCR (Optical Character Recognition). The first support processing element 121 determines whether or not the acquired image contains actual machine identification information, and if the actual machine identification information is included, extracts the character information assigned as the actual machine identification information. In other words, the first support processing element 121 can determine whether or not the actual machine identification information is included in the image, and if the actual machine identification information is present, can acquire the actual machine identification information assigned to each of the work machines.
[0071] The second support processing element 122 of the work support server 10 determines whether the actual machine identification information extracted from the image by the first support processing element 121 is consistent with the identifier of the work machine corresponding to the selection information acquired in step S402 (step S404). In other words, the first support processing element 121 can recognize the actual machine identification information from an image captured by the actual machine imaging device of one cooperative work machine to be remotely operated among the multiple work machines.
[0072] Furthermore, the second support processing element 122 can acquire an identifier corresponding to the selection information by referring to the database 102. The second support processing element 122 can acquire an identifier corresponding to the actual machine identification information by referring to the database 102 based on the actual machine identification information extracted from the image by the first support processing element 121.
[0073] The second support processing element 122 is capable of determining the consistency between an identifier corresponding to the selection information and an identifier corresponding to the actual machine identification information extracted from the image, thereby determining the consistency between the identifier of the work machine corresponding to the selection information and the work machine to which the remote control device 20 is connected.
[0074] When the second support processing element 122 of the work support server 10 determines in step S404 that the extracted actual machine identification information is consistent with the identifier of the work machine corresponding to the selection information (step S404: Y), it transmits the selection information, the image and the determination result to the remote operation device 20, and causes it to be output to the remote output interface 220 (step S405), and ends the work support processing. That is, the second support processing element 122 of the work support server 10 can transmit the determination result of consistency in step S404 to the remote operation device 20, and causes it to be output to the remote output interface 220.
[0075] When the remote operation device 20 receives the transmission result indicating that the information matches, the remote operation device 20 displays the determination result on, for example, the terminal at hand 2240. The operator OP1 can confirm that the actual device identification information and the selection information match by visually checking the displayed determination result. The transmission result indicating that the information matches may be output to the remote image output device 221.
[0076] If the second support processing element 122 of the work support server 10 determines in step S404 that the extracted actual machine identification information does not match the identifier of the work machine corresponding to the selection information (step S404: N), it transmits the determination result to the remote operation device 20 and outputs it to the remote output interface 220 (step S406).
[0077] When the remote operation device 20 receives a transmission result indicating that there is no consistency, it displays the determination result on, for example, the terminal at hand 2240. The operator OP1 can confirm that the actual machine identification information and the identifier of the work machine corresponding to the selected information do not match by visually checking the determination result. When the operator OP1 confirms that there is no match, he or she can resolve the state in which the actual machine identification information and the identifier corresponding to the selected information do not match by taking measures such as interrupting the remote operation, restarting the work support system, and contacting a system administrator. The transmission result indicating that there is no consistency may be output to the remote image output device 221.
[0078] Furthermore, the remote operation device 20 may display, for example, the operation content input to the terminal at hand 2240 in step S102 (information that the first work machine 40 or the second work machine 50 is selected), i.e., the selection information, on the terminal at hand 2240. The selection information displayed on the terminal at hand 2240 is not limited to this, and for example, the selection information transmitted from the second support processing element 122 may be displayed on the remote image output device 221.
[0079] A state in which the actual machine identification information does not match the identifier of the work machine corresponding to the selection information can occur, for example, when the connection settings are made incorrectly, such as when an incorrect identifier is assigned when assigning an identifier to each work machine, or when a data problem occurs due to an input error when registering the identifier in the database 102.
[0080] The method of resolving the situation in which the actual machine identification information and the identifier of the work machine corresponding to the selection information do not match described above involves interrupting work, but if continuing work is given priority, the following flow can be adopted.
[0081] The second support processing element 122 of the work support server 10 judges whether or not a matching process for matching the actual machine identification information with an identifier corresponding to the selection information has been started (step S407). The matching process is started, for example, by the operator OP1 touching the display screen of the terminal 2240 at hand of the remote operation device 20 (selecting the processing start button). The remote operation device 20 generates execution instruction information for executing the matching process or no-processing information for not executing the matching process based on the selection of the operator OP1, and transmits it to the work support server 10. The work support server 10 makes the judgment of step S407 based on the execution instruction information or the no-processing information.
[0082] If the operator OP1 has not started the matching process (step S407: N), the work support server 10 ends the work support process. On the other hand, if the operator OP1 has started the matching process (step S407: Y), the process proceeds to step S408.
[0083] The first support processing element 121 of the work support server 10 causes images (hereinafter also referred to as candidate images) captured by an actual imaging device of a work machine other than the work machine to which the remote operation device 20 is connected to be transmitted to the work support server 10 and stored in the database 102. That is, the first support processing element 121 acquires candidate images captured by the actual imaging device of the other work machine (step S408). The second support processing element 122 transmits these candidate images to the remote operation device 20 and causes the remote output interface 220 to output them sequentially.
[0084] The remote operation device 20 causes the remote image output device 221 to display these candidate images in a selectable manner, and also causes the handheld terminal 2240 to display a message prompting the operator to select a candidate image desired to be connected and to select actual machine identification information. The operator OP1 selects a candidate image including actual machine identification information corresponding to the selection information, for example, based on the actual machine identification information of a sticker ST included in the candidate image, and selects the actual machine identification information. Furthermore, if there is no candidate image desired to be connected among these candidate images, the operator OP1 selects that fact. If it is selected that there is no candidate image desired to be connected, the remote operation device 20 generates work machine non-selected information.
[0085] The remote operation device 20 transmits the candidate image and actual machine identification information or work machine non-selection information selected by the operator OP1 (hereinafter, the candidate image, actual machine identification information, and work machine non-selection information are also referred to as matching information) to the work support server 10. The second support processing element 122 of the work support server 10 determines whether or not a work machine has been selected based on the candidate image and actual machine identification information or the work machine non-selection information transmitted from the remote operation device 20 (step S409).
[0086] That is, the first support processing element 121 of the task support server 10 of The matching information can be acquired from the remote input interface 210. In this embodiment, a work machine whose actual machine identification information included in the candidate image and whose actual machine identification information corresponding to the selection information are consistent is referred to as a true work machine.
[0087] If the operator OP1 does not select a candidate image having actual machine identification information corresponding to the selection information (step S409: N), the work support server 10 ends the work support process.
[0088] When a candidate image including actual machine identification information corresponding to the selection information is selected (step S409: Y), the second support processing element 122 of the work support server 10 connects the remote control device 20 to the work machine having the candidate image with actual machine identification information corresponding to the selection information, and rewrites the identifier in the database 102 before the connection switching to the identifier after the connection switching (step S410).
[0089] Incidentally, the identifier of the previously connected work machine (the identifier before the rewrite process in step S410) may not necessarily match the identifier of the work machine after the connection has been switched, even if it is rewritten. Therefore, it is advisable to add a note such as "must check database 102" so that the system administrator, etc. can check it later.
[0090] The second support processing element 122 of the work support server 10 transmits completion information indicating that the rewriting of the identifier has been completed to the remote operation device 20, and ends the work support processing. That is, when the matching information acquired by the first support processing element 121 is a true cooperative work machine, the second support processing element 122 of the work support server 10 is capable of rewriting the identifier corresponding to the cooperative work machine to the identifier of the work machine equipped with the actual machine imaging device that captured the candidate image.
[0091] When the remote operation device 20 receives the completion information, it displays, for example, on the display screen of the terminal 2240, a message to the effect that the identifier of the connected work machine has been changed and prompts the operator OP1 to confirm. By displaying such a message, the operator OP1 can check the connected work machine again after changing the connected work machine. Therefore, the operator OP1 can remotely operate the work machine after recognizing the need to confirm whether the selected information and the actual machine identification information are the same or different.
[0092] In the above description of the first and second embodiments, the work support processing is executed by the work support server 10. However, the work support processing may be executed by the remote control device 20, or may be executed by the first work machine 40 or the second work machine 50.
[0093] Also, it has been described that the images are captured by the actual machine imaging device 412 of the first work machine 40 or the actual machine imaging device 512 of the second work machine 50. However, the images may be captured by an imaging device other than the actual machine imaging device 412 of the first work machine 40 or the actual machine imaging device 512 of the second work machine 50 and acquired.
[0094] Examples of such imaging devices include a fixed-point imaging device provided in a position capable of imaging the exterior of the first work machine 40 or the second work machine 50, and an imaging device provided on an aircraft such as a drone.
[0095] When such imaging devices are used, the sticker ST on which the identification information is printed is preferably provided in a position visible from outside the first work machine 40 and the second work machine 50. Examples of such positions include the side of the upper revolving body 420 of the first work machine 40 and the upper revolving body 520 of the second work machine 50. Note that, when a connection operation is performed on the first work machine 40 or the second work machine 50 to the remote control device 20, it is preferable that these imaging devices be notified of this. In this manner, it is possible to start transmitting images from the imaging devices to the work support server 10.
[0096] The sticker ST may be a two-dimensional code or a figure. When a two-dimensional code is attached to the sticker ST, the first support processing element 121 of the work support server 10 extracts the actual machine identification information from the two-dimensional code. When a figure is attached to the sticker ST, the first support processing element 121 of the work support server 10 may recognize the difference in the figure and call up the actual machine identification information corresponding to the figure by referring to the database 102.
[0097] In the above first and second embodiments, when a new work machine is connected to the remote control device 20, or when a work machine that was connected to the remote control device 20 is switched to another work machine, the operator is asked to confirm whether the actual machine identification information represented by the sticker ST in the image captured by the actual machine image capture device of the work machine matches the selection information. TaO A notification may be sent to the remote output interface 220 to prompt P1. [Explanation of symbols]
[0098] 10. Work support server 121 - First support processing element 122 - Second support processing element 20. Remote control device 40, 50...Work machines
Claims
1. A remote operation system having a function of mutual communication between a plurality of work machines, a remote operation device for remotely operating one of the plurality of work machines that is to be remotely operated, and a work support server, A work machine that is a candidate for a remote operation target among the plurality of work machines is equipped with an actual machine imaging device capable of capturing an image, and actual machine identification information that is captured so as to be included in the image captured by the actual machine imaging device and is described as character information assigned to the candidate work machine, The work support server determines whether or not one work machine to be a remotely operated target has been selected from the multiple work machines, and when it is determined that the one work machine has been selected, connects a cooperative work machine identified by an identifier corresponding to selection information, which is information about the selected one work machine, to the remote operation device, and executes a process of acquiring the image captured by the actual machine imaging device of the cooperative work machine, and a process of outputting the image captured by the actual machine imaging device of the cooperative work machine and the selection information to an output interface of the remote operation device. A remote control system comprising:
2. A remote operation system having a function of mutual communication between a plurality of work machines, a remote operation device for remotely operating one of the plurality of work machines that is to be remotely operated, and a work support server, A work machine that is a candidate for a remote operation target among the plurality of work machines is equipped with an actual machine imaging device capable of capturing an image, and actual machine identification information that is captured so as to be included in the image captured by the actual machine imaging device, the work support server determines whether or not one of the multiple work machines to be remotely operated has been selected, and when it is determined that the one work machine has been selected, connects a cooperating work machine identified by an identifier corresponding to selection information, which is information about the selected one work machine, to the remote operation device, acquires the image captured by the actual machine imaging device of the cooperating work machine, and outputs the image captured by the actual machine imaging device of the cooperating work machine to an output interface of the remote operation device; and determines whether or not the image captured by the actual machine imaging device of the cooperating work machine includes actual machine identification information, and when it is determined that the image includes the actual machine identification information, determines consistency between the actual machine identification information and the selection information, and outputs a result of the determination of consistency to the output interface of the remote operation device.
3. A remote operation system having a function of mutual communication between a plurality of work machines, a remote operation device for remotely operating one of the plurality of work machines that is to be remotely operated, and a work support server, A work machine that is a candidate for a remote operation target among the plurality of work machines is equipped with an actual machine imaging device capable of capturing an image, and actual machine identification information that is captured so as to be included in the image captured by the actual machine imaging device, The work support server determines whether or not one work machine to be remotely operated has been selected from the multiple work machines, and when it is determined that the one work machine has been selected, connects a cooperating work machine identified by an identifier corresponding to selection information, which is information on the selected one work machine, to the remote operation device, acquires the image captured by the actual machine imaging device of the cooperating work machine, and outputs the image captured by the actual machine imaging device of the cooperating work machine to an output interface of the remote operation device; and acquiring a candidate image captured by the actual machine imaging device of a work machine other than the cooperating work machine among the plurality of work machines, acquiring consistency information from the remote control device as to whether the work machine equipped with the actual machine imaging device that captured the candidate image is a true cooperating work machine, outputting the candidate image to an output interface of the remote control device, and, when the consistency information indicates the true cooperating work machine, rewriting the identifier corresponding to the cooperating work machine to an identifier of the work machine equipped with the actual machine imaging device that captured the candidate image.
4. A remote control system as described in claim 2 or claim 3, characterized in that the work support server executes a process to output the selection information to an output interface of the remote control device.
5. A remote control system as described in any one of claims 1 to 4, characterized in that the work support server executes a process of acquiring the selection information inputted into an input interface of the remote control device, and a process of outputting the acquired selection information to an output interface of the remote control device.
6. A work support server capable of communicating with each of a plurality of work machines and a remote operation device for remotely operating one of the plurality of work machines that is to be remotely operated, A work machine that is a candidate for a remote operation target among the plurality of work machines is equipped with an actual machine imaging device capable of capturing an image, and actual machine identification information that is captured so as to be included in the image captured by the actual machine imaging device and is described as character information assigned to the candidate work machine, a work support server which executes the following processes: determining whether or not one work machine to be remotely operated has been selected from the multiple work machines; and, when it is determined that the one work machine has been selected, connecting a cooperating work machine identified by an identifier corresponding to selection information, which is information about the selected one work machine, to the remote operation device, acquiring the image captured by the actual machine imaging device of the cooperating work machine; and outputting the image captured by the actual machine imaging device of the cooperating work machine and the selection information to an output interface of the remote operation device.
Citation Information
Patent Citations
Communication method for work machine at remote location
JP1995087004A
Remote control system for work machine
JP2019176401A
Control system
JP2019183561A
Operation system for vehicle
JP2021026398A
