Imaging function control system and imaging function control method
The system addresses the challenge of camera magnification changes by controlling the swing function of the imaging device, ensuring clear surroundings visibility and reducing operator discomfort.
Patent Information
- Application Number
- JP2021045532
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-03-19
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2041-03-19
AI Technical Summary
The challenge in remote operation of working machines is that excessive or insufficient freedom in the swing function of the imaging device can lead to difficulties in understanding the surroundings due to changes in camera magnification, hindering effective operation.
A system that limits the swing function of the actual imaging device in certain states and controls its optical axis direction based on the operation mode, ensuring appropriate imaging for better situational awareness.
This system ensures the operator maintains a clear understanding of the surroundings by limiting the swing function when necessary, allowing for smooth operation and reducing operator discomfort from image changes.
Smart Images

Figure 0007581995000001 
Figure 0007581995000002 
Figure 0007581995000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a technique for controlling an imaging function of an actual machine imaging device mounted on a work machine such as a hydraulic excavator. [Background technology]
[0002] A technology has been proposed in which an operator operates a work machine using a remote control device while referring to a remote image showing the working status of the work machine (see, for example, Patent Document 1). When an operator remotely operates a work machine, the sense of grasping the situation around the work machine is different compared to when the operator is on board the work machine (actual machine) and operates the actual machine. For this reason, the system is configured to change the direction of the actual machine imaging device mounted on the work machine so that the operator can confirm the situation around the work machine by changing the direction of his or her line of sight in various ways, just as when operating the actual machine. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2018-021395 A Summary of the Invention [Problem to be solved by the invention]
[0004] However, if the degree of freedom of the swing function of the actual machine imaging device is excessively high or low, the change in the image captured through the actual machine imaging device may cause problems in grasping the surrounding situation if you forget to cancel the setting that increased the display magnification of the work machine's camera or if the camera's display magnification does not fully return to the minimum (initial) display magnification, resulting in insufficient understanding of the surrounding situation necessary for performing work.
[0005] Therefore, an object of the present invention is to provide a system etc. that can improve the convenience for the operator to grasp the situation around the work machine by controlling the imaging function of the actual machine imaging device mounted on the work machine in an appropriate form taking into account the condition of the work machine. [Means for solving the problem]
[0006] The imaging function control system of the present invention comprises: An imaging function control device including a remote control device having a remote control mechanism that is operated to operate a working mechanism of a working machine, and an actual machine imaging device that is mounted on the working machine and obtains an image including at least a part of the working mechanism and outputs the image to a remote image output device provided in the remote control device. 1. A system comprising: a remote control device provided in the remote control device and recognizing an operation mode of a swing operation for adjusting an optical axis direction of the actual imaging device; A swing function is provided for changing the optical axis direction of the real-machine imaging device to an angle according to the operation mode of the swing operation. Imaging function control element And, The imaging function control element The remote control device is operated to operate the working mechanism. In the first state, No work is being performed in which the remote control device is operated to operate the working mechanism. than the second state, The above Limit swing function.
[0007] According to the imaging function control system of this configuration, in the first state, the swing function of the real imaging device is restricted as compared to the second state. "Swing function" means at least one of the pan function and the tilt function. In other words, in the first state, when the real imaging device has both a pan function and a tilt function, at least one of the two functions is restricted as compared to the second state, and when the real imaging device has only one of the pan function and the tilt function, that one function is restricted. As a result, Remote Image Output Device The amount of change in the real space reflected in the image is suppressed, so the operator's attention is distracted in the first state. Remote Control Mechanism This can prevent a situation in which the operation of the work machine is hindered through the use of the vehicle.
[0008] In the imaging function control system having the above configuration, The imaging function control element, in the first state, ,before When the swing operation is stopped, the attitude of the actual image capturing device is changed by exerting a swing function. The posture of the real imaging device at the start of the swing operation. While returning to a predetermined initial posture, in the second state ,beforeWhen the swing operation is stopped, the posture of the real-machine imaging device is maintained as it is. It is preferred.
[0009] According to the imaging function control system having the above configuration, when the swing operation is stopped in the first state, the attitude of the real imaging device is The attitude of the real imaging device at the start of a swing operation. As a result, the operator is allowed to grasp the state of the real space area corresponding to the posture at the time when the swing operation of the real machine imaging device is stopped, and then the operator is allowed to perform the operation. Rus The operator can grasp the state of the real space area according to the initial attitude of the real machine image capturing device without the need for a swing operation. On the other hand, when the swing operation is stopped in the second state, the attitude of the real machine image capturing device is maintained as it is. As a result, during the period when the operation of the work machine is stopped, Can The real-machine imaging device can be caused to exert a swing function in accordance with any swing operation mode, allowing the operator to grasp the state of the real space area reflected in the captured image acquired through the real-machine imaging device.
[0010] In the imaging function control system having the above configuration, The imaging function control element, in the first state, ,before When the swing operation is stopped, the attitude of the real machine imaging device is The posture of the real imaging device at the start of the swing operation. During the time until the robot returns to the predetermined initial attitude, the captured image obtained through the actual machine imaging device is Remote Image Output Device or outputting the captured image at the time when the swing operation is stopped to the Remote Image Output Device to output continuously It is preferred.
[0011] According to the imaging function control system having the above configuration, In the first state ,vinegar When the tilting operation is stopped, the captured image when the orientation of the real image capture device is returned to the initial orientation is Remote Image Output Device (Or the captured image when the operation is stopped is continuously output through the display.) As a result, the discomfort felt by the operator due to the captured image changing even though the swing operation is stopped can be reduced. [Brief description of the drawings]
[0014] [Figure 1] FIG. 1 is an explanatory diagram regarding the configuration of a remote operation support system. [Diagram 2] FIG. 2 is an explanatory diagram relating to the configuration of a remote control device. [Diagram 3] FIG. 2 is an explanatory diagram relating to the configuration of a work machine. [Figure 4] FIG. 2 is an explanatory diagram regarding functions of a remote operation support system (imaging function control system). [Diagram 5] FIG. [Figure 6] FIG. 4 is an explanatory diagram regarding a control mode of a swing function of the actual imaging device. [Figure 7] FIG. 13 is an explanatory diagram relating to functions of a remote operation support system (imaging function control system) according to another embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0015] (Configuration of remote operation support system) The remote operation support system shown in Fig. 1 is composed of a remote operation support server 10 and a remote operation device 20 for remotely operating a work machine 40. The remote operation support server 10, the remote operation device 20, and the work machine 40 are configured to be able to communicate with each other via a network. The mutual communication network of the remote operation support server 10 and the remote operation device 20 and the mutual communication network of the remote operation support server 10 and the work machine 40 may be the same as or different from each other.
[0016] (Configuration of remote operation support server) The remote operation support server 10 includes a database 102, a first support processing element 121, and a second support processing element 122. The database 102 stores and holds captured image data and the like. The database 102 may be configured as a database server separate from the remote operation support server 10. Each support processing element is configured as an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing (described later) on the data according to the software. In this embodiment, the remote operation support server 10 constitutes an "imaging function control system," and the second support processing element 122 constitutes an "imaging function control element."
[0017] (Configuration of remote control device) The remote operation device 20 includes a remote control device 200, a remote input interface 210, and a remote output interface 220. The remote control device 200 is configured with an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing on the data according to the software.
[0018] The remote input interface 210 includes a remote control mechanism 211. The remote output interface 220 includes a remote image output device 221, an audio output device 222, and a remote wireless communication device 224.
[0019] The remote operation mechanism 211 includes a traveling operation device, a slewing operation device, a boom operation device, an arm operation device, and a bucket operation device. Each operation device has an operation lever that receives a rotation operation. The operation lever (travel lever) of the traveling operation device is operated to move the lower traveling body 410 of the work machine 40. The travel lever may also serve as a travel pedal. For example, a travel pedal fixed to the base or lower end of the travel lever may be provided. The operation lever (slewing lever) of the slewing operation device is operated to move a hydraulic slewing motor that constitutes the slewing mechanism 430 of the work machine 40. The operation lever (boom lever) of the boom operation device is operated to move the boom cylinder 442 of the work machine 40. The operation lever (arm lever) of the arm operation device is operated to move the arm cylinder 444 of the work machine 40. The operation lever (bucket lever) of the bucket operation device is operated to move the bucket cylinder 446 of the work machine 40.
[0020] The operating levers constituting the remote control mechanism 211 are arranged around a seat St on which the operator sits, as shown in Fig. 2. The seat St is in the form of a high-back chair with armrests, but may be in the form of a low-back chair without a headrest, or in the form of a chair without a backrest, or in any other form on which the operator can sit.
[0021] A pair of left and right travel levers 2110 corresponding to the left and right crawlers are arranged side by side in front of the seat St. One operation lever may serve as multiple operation levers. For example, the left operation lever 2111 provided in front of the left frame of the seat St shown in FIG. 2 may function as an arm lever when operated in the front-rear direction, and may function as a rotation lever when operated in the left-right direction. Similarly, the right operation lever 2112 provided in front of the right frame of the seat St shown in FIG. 2 may function as a boom lever when operated in the front-rear direction, and may function as a bucket lever when operated in the left-right direction. The lever pattern may be arbitrarily changed by an operation instruction from the operator.
[0022] A pan switch SW1 and a tilt switch SW2 for adjusting the pan angle and tilt angle of the real imaging device 412 are provided at the positions of the left operation lever 2111 and the right operation lever 2112 that are gripped by the operator, as described below. Both the pan switch SW1 and the tilt switch SW2 may be provided on only one of the left operation lever 2111 and the right operation lever 2112. Each of the pan switch SW1 and the tilt switch SW2 is configured so that its operating position is returned to its initial operating position by a biasing element such as an actuator or a spring under certain conditions. If the real imaging device 412 has only one of the pan function and the tilt function as a swing function, one of the pan switch SW1 and the tilt switch SW2 may be omitted.
[0023] 2, the remote image output device 221 is composed of a central remote image output device 2210, a left remote image output device 2211, and a right remote image output device 2212, each having a substantially rectangular screen, and disposed respectively in front of, diagonally forward left of, and diagonally forward right of, the seat St. The shapes and sizes of the screens (image display areas) of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be the same or different.
[0024] As shown in Fig. 2, the right edge of the left remote image output device 2211 is adjacent to the left edge of the central remote image output device 2210, such that the screens of the central remote image output device 2210 and the left remote image output device 2211 form an inclination angle θ1 (e.g., 120°≦θ1≦150°). As shown in Fig. 2, the left edge of the right remote image output device 2212 is adjacent to the right edge of the central remote image output device 2210, such that the screens of the central remote image output device 2210 and the right remote image output device 2212 form an inclination angle θ2 (e.g., 120°≦θ2≦150°). The inclination angles θ1 and θ2 may be the same or different.
[0025] The screens of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be parallel to the vertical direction or may be inclined relative to the vertical direction. At least one of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be composed of a plurality of divided image output devices. For example, the central remote image output device 2210 may be composed of a pair of image output devices adjacent to each other above and below, each having a substantially rectangular screen.
[0026] The sound output device 222 is composed of one or more speakers, and is composed of a central sound output device 2220, a left sound output device 2221, and a right sound output device 2222, which are arranged behind the seat St, at the rear of the left armrest, and at the rear of the right armrest, respectively, as shown in Fig. 2. The specifications of the central sound output device 2220, the left sound output device 2221, and the right sound output device 2222 may be the same or different.
[0027] (Work machine configuration) The work machine 40 comprises an actual machine control device 400, an actual machine input interface 41, an actual machine output interface 42, and a work mechanism 440. The actual machine control device 400 comprises an image processing device 30. The image processing device 30 comprises a state detection element 31, an image prediction element 32, and an image compression element 34. Each of the components of the actual machine control device 400 and the image processing device 30 is configured by an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing on the data in accordance with the software.
[0028] The work machine 40 is, for example, a crawler excavator (construction machine), and as shown in Fig. 3, includes a crawler-type lower traveling body 410, and an upper rotating body 420 that is rotatably mounted on the lower traveling body 410 via a rotating mechanism 430. A cab 424 (operator's cabin) is provided on the front left side of the upper rotating body 420. A work mechanism 440 is provided in the front center of the upper rotating body 420.
[0029] The real machine input interface 41 includes a real machine operation mechanism 411, a real machine imaging device 412, and a positioning device 414. The real machine operation mechanism 411 includes a plurality of operation levers arranged around a seat arranged inside the cab 424 in the same manner as the remote control mechanism 211. A drive mechanism or robot that receives a signal according to the operation mode of the remote control lever and moves the real machine operation lever based on the received signal is provided in the cab 424. The real machine imaging device 412 is installed, for example, inside the cab 424, and images the environment including at least a part of the working mechanism 440 through the front window and a pair of left and right side windows. A part or all of the front window and the side windows may be omitted. The positioning device 414 is configured with a GPS and, if necessary, a gyro sensor or the like.
[0030] The real device output interface 42 includes a real device wireless communication device 422 .
[0031] 3, a working mechanism 440 as a working mechanism includes a boom 441 mounted on the upper rotating body 420 so as to be able to rise and fall, an arm 443 rotatably connected to the tip of the boom 441, and a bucket 445 rotatably connected to the tip of the arm 443. A boom cylinder 442, an arm cylinder 444, and a bucket cylinder 446 each composed of an extendable hydraulic cylinder are mounted on the working mechanism 440.
[0032] The boom cylinder 442 is interposed between the boom 441 and the upper rotating body 420 so as to extend and retract when supplied with hydraulic oil to rotate the boom 441 in the hoisting direction. The arm cylinder 444 is interposed between the arm 443 and the boom 441 so as to extend and retract when supplied with hydraulic oil to rotate the arm 443 about a horizontal axis relative to the boom 441. The bucket cylinder 446 is interposed between the bucket 445 and the arm 443 so as to extend and retract when supplied with hydraulic oil to rotate the bucket 445 about a horizontal axis relative to the arm 443.
[0033] (function) The functions of the remote operation support system and the imaging function control system having the above configuration will be described with reference to the flowchart shown in Fig. 4. In the flowchart, a block "C●" is used for the sake of simplicity, and means transmission and / or reception of data, and means a conditional branch in which processing in a branching direction is executed on the condition that the data is transmitted and / or received.
[0034] In the remote operation device 20, an environment confirmation request is transmitted to the remote operation assistance server 10 via the remote wireless communication device 224 (FIG. 4 / STEP 210). For example, a requirement for starting transmission of the environment confirmation request may be determined to be a designation operation by the operator via the remote input interface 210. The "designation operation" is, for example, an operation such as tapping on the remote input interface 210 for designating the work machine 40 that the operator intends to remotely operate.
[0035] When the environment confirmation request is received in the remote operation support server 10, the first support processing element 121 transmits the environment confirmation request to the corresponding work machine 40 (FIG. 4 / C10).
[0036] When an environment confirmation request is received in the work machine 40 via the actual machine wireless communication device 422 (Figure 4 / C40), an image is acquired by the actual machine control device 400 via the actual machine imaging device 412, and image data representing the image is transmitted to the remote control device 10 via the actual machine wireless communication device 422 (Figure 4 / STEP 410).
[0037] In the remote operation support server 10, when the first support processing element 121 receives captured image data (FIG. 4 / C11), the second support processing element 122 transmits environmental image data corresponding to the captured image to the remote operation device 20 (FIG. 4 / STEP 110). The environmental image data is not only the captured image data itself, but also image data representing a simulated environmental image generated based on the captured image.
[0038] When the remote operation device 20 receives environmental image data through the remote wireless communication device 224 (FIG. 4 / C21), the remote control device 200 outputs an environmental image corresponding to the environmental image data to the remote image output device 221 (FIG. 4 / STEP 212).
[0039] As a result, for example, as shown in FIG. 5, an environmental image showing a boom 441, arm 443, and a pile of rubble or earth and sand (the object of work by bucket 445), which are part of the work mechanism 440, in front of the cab 424, is output to the remote image output device 221 through the window frame that defines the cab 424.
[0040] In the remote operation device 20, the remote control device 200 recognizes the swing operation mode of the remote operation mechanism 211, and transmits a swing command corresponding to the swing operation mode to the remote operation support server 10 through the remote wireless communication device 224 (FIG. 4 / STEP 214). The swing operation mode is specified by the operation postures of the pan switch SW1 and the tilt switch SW2 provided at the gripping positions of the left operation lever 2111 and the right operation lever 2112, respectively (see FIG. 2). The switches SW1 and SW2 are, for example, slide switches or lever-type switches capable of detecting the operation posture (amount of operation), which detect the amount of operation to one side and the other side of the switch, and are in a neutral position when no operation is performed.
[0041] Pan switch SW1 operating position range [θ p1 ,θ p2 ] at any operating position θ p In response to this, the pan angle φ of the actual imaging device 412 is p The pan angle range [φ p1 ,φ p2 ] at pan angle f p A swing command is generated to control the angle to (θp). p is defined, for example, by the relation (01).
[0042] f p (θ p )={(φ p2 -φ p1 ) / (θ p2 -θ p1 )}(θ p -θ p1 )+φ p1 ...(01).
[0043] Tilt switch SW2 operating position range [θ t1 ,θ t2 ], the tilt angle φ of the actual imaging device 412 is adjusted according to an arbitrary operation posture θt in the t The tilt angle range [φ t1 ,φ t2 ] at the tilt angle f t (θ t) is generated. t is defined, for example, by the relation (02).
[0044] f t (θ t )={(φ t2 -φ t1 ) / (θ t2 -θ t1 )}(θ t -θ t1 )+φ t1 ...(02).
[0045] When the swing command is received in the remote operation support server 10, the second support processing element 122 (imaging function control element) transmits the swing command to the work machine 40 (FIG. 4 / C12).
[0046] When the work machine 40 receives a normal swing command through the actual machine wireless communication device 422 (FIG. 4 / C42), the actual machine control device 400 controls the swing function of the actual machine imaging device 412 (more precisely, the pan drive mechanism and / or tilt drive mechanism of the actual machine imaging device 412), and the pan angle φ p and / or tilt angle φ t is adjusted (Figure 4 / STEP 412).
[0047] As a result, the real space area reflected in the captured image acquired through the real imaging device 412 and, in turn, in the environmental image output by the remote image output device 221 changes (see FIG. 4 / STEP 410 → . . . →STEP 212, FIG. 5). For example, the pan angle φ of the real imaging device 412 p When the optical axis direction is changed from the front of the work machine 40 to the left diagonally forward by the control of the above, the real space area reflected in the environmental image changes from the front area of the work machine 40 to the left diagonally forward area. t When the optical axis direction is changed from the forward horizontal direction to a forward diagonally downward direction by the control of (a), the real space area reflected in the environmental image is changed to a lower area than before.
[0048] In the remote operation device 20, the remote control device 200 determines whether the swing operation has been stopped (FIG. 4 / STEP 216). For example, it is determined whether the swing operation has been stopped depending on whether the amount of change in the operating attitude of each of the pan switch SW1 and the tilt switch SW2 has become equal to or less than a threshold value (a state in which neither the pan switch SW1 nor the tilt switch SW2 is being operated) has continued for a specified period of time.
[0049] If the determination result is positive (FIG. 4 / STEP 216...YES), the remote control device 200 transmits a swing stop command to the remote operation support server 100 (FIG. 4 / STEP 218). On the other hand, if the determination result is negative (FIG. 4 / STEP 216...NO), the swing stop command is not generated and the subsequent process is executed.
[0050] When the remote operation support server 10 receives a swing stop command (FIG. 4 / C14), the second support processing element 122 determines the state of the work machine 40 based on the communication result with the work machine 40 (FIG. 4 / STEP 120). The "first state" is a state in which the work machine 40 can be operated (or a state in which the work machine 40 is working). The "second state" is a state in which the work machine 40 cannot be operated (or a state in which the work machine 40 has stopped working). For example, the first state and the second state of the work machine 40 may be determined depending on whether or not a hydraulic cutoff lever constituting the actual machine operation mechanism 411 of the work machine 40 is in a predetermined position. The first state and the second state of the work machine 40 may also be determined depending on whether or not a remote hydraulic cutoff lever simulating a hydraulic cutoff lever as a remote input interface is in a predetermined position.
[0051] When it is determined that the work machine 40 is in the first state (FIG. 4 / STEP 120...1), the second support processing element 122 transmits an initial attitude return command to the work machine 40 (FIG. 4 / STEP 124). On the other hand, when it is determined that the work machine 40 is in the second state (FIG. 4 / STEP 120...2), the initial attitude return command is not generated and the subsequent processing is executed.
[0052] When the work machine 40 receives an initial attitude return command through the actual machine wireless communication device 422 (FIG. 4 / C44), the actual machine control device 400 controls the swing function of the actual machine imaging device 412, and the pan angle φ p and / or tilt angle φ t is adjusted to an initial angle or initial posture (FIG. 4 / STEP 414).
[0053] For example, as shown in Figure 6, at time t=t s Consider the case where a swing operation is started at time t=t0, the swing operation is stopped at time t=t1, the swing operation is resumed at time t=t2, and the swing operation is stopped again at time t=t3.
[0054] In this case, if the state of the working machine 40 is the first state, as shown by the dashed line in FIG. 6, the attitude φ of the actual machine imaging device 412 is equal to (φ p , φ t ) is controlled to return to the initial attitude φ0 at time t=t0, and to change from time t=t1 according to the operation manner of the pan switch SW1 and / or the tilt switch SW2, and then to return to the initial attitude φ0 again at time t=t2. Therefore, the operator adjusts the attitude φ of the real imaging device 412 from the initial attitude φ0 by starting the operation of the pan switch SW1 and / or the tilt switch SW2 at time t=t1. In response to the attitude φ of the real imaging device 412 returning to the initial attitude φ0, the second support processing element 122 may transmit an initial attitude return command to the remote operation device 20 so that the operation attitudes θ of the pan switch SW1 and the tilt switch SW2 are returned to the initial operation attitude θ0.
[0055] On the other hand, if the state of the working machine 40 is the second state, then as shown by the two-dot chain line in FIG. 6, the attitude φ of the actual machine imaging device 412 is φ=(φ p , φ t) is maintained at the same attitude φ(t1) at time t=t0, and is controlled so that it changes from time t=t1 in accordance with the operation mode of the pan switch SW1 and / or the tilt switch SW2, and then is maintained at the same attitude φ(t2) at time t=t2. Therefore, by resuming the operation of the pan switch SW1 and / or the tilt switch SW2 at time t=t1, the operator adjusts the attitude φ of the actual imaging device 412 from the attitude φ(t0) at the time when the operation was stopped immediately before.
[0056] In the remote operation device 20, the remote control device 200 recognizes the operation mode of the remote operation mechanism 211, and a remote operation command corresponding to the operation mode is transmitted to the remote operation support server 10 via the remote wireless communication device 224 (FIG. 4 / STEP 220).
[0057] In the remote operation support server 10, when the second support processing element 122 receives the remote operation command, the first support processing element 121 transmits the remote operation command to the work machine 40 (FIG. 4 / C16).
[0058] In the work machine 40, when an operation command is received by the actual machine control device 400 via the actual machine wireless communication device 422 (FIG. 4 / C46), the operation of the work mechanism 440 and the like is controlled (FIG. 4 / STEP 420). For example, the work of scooping up soil in front of the work machine 40 with the bucket 445, rotating the upper rotating body 410, and dropping the soil from the bucket 445 is performed.
[0059] (Action and effect) According to the imaging function control system constituting the remote operation support system of this configuration, in the first state, the swing function of the actual imaging device 412 is restricted compared to the second state (Figure 4 / STEP 120...1 → STEP 124 → C44 → STEP 414, see Figure 6).
[0060] In detail, when the work machine 40 is in the first state, after the operation of the pan switch SW1 and the tilt switch SW2 is stopped, the attitude φ of the real machine imaging device 412 when the operation is resumed is limited to the initial attitude φ0. As a result, after the operator is allowed to grasp the state of the real space area corresponding to the attitude when the swing operation of the real machine imaging device is stopped, the operator can grasp the state of the real space area corresponding to the initial attitude of the real machine imaging device without the need for the operator to perform a swing operation via the remote input interface 21 (pan switch SW1 and / or tilt switch SW2).
[0061] On the other hand, when the work machine 40 is in the second state, after the operation of the pan switch SW1 and the tilt switch SW2 is stopped, the attitude φ of the real machine imaging device 412 when the operation is resumed remains the attitude φ at the time when the operation was stopped immediately before. As a result, during the period when the operation of the work machine 40 is stopped, the real machine imaging device 412 is caused to exert a swing function in response to an arbitrary swing operation mode through the remote input interface 21 (pan switch SW1 and / or tilt switch SW2) by the operator, and the operator can grasp the state of the real space area reflected in the captured image acquired through the real machine imaging device 412.
[0062] (Another embodiment of the present invention) In the above embodiment, the imaging function control system is constituted by the remote operation support server 10, and the imaging function control element is constituted by the second support processing element 122. However, in another embodiment, the imaging function control system may be constituted by the work machine 40 and / or the remote operation device 20, and the imaging function control element may be constituted by the actual machine control device 400 and / or the remote control device 200.
[0063] In the above embodiment, the swing function is restricted in the first state compared to the second state in that the attitude φ of the real imaging device 412 is restricted when the swing operation of the remote control device 20 is resumed. However, in other embodiments, the swing function may be restricted in that the adjustment range and / or change speed of the attitude φ of the real imaging device 412 is restricted.
[0064] In this case, a series of processes may be executed according to the flowchart shown in Fig. 7 instead of the flowchart shown in Fig. 4. In the flowchart, a block "C●" is used for the sake of simplicity, and means transmission and / or reception of data, and means a conditional branch in which a process in a branching direction is executed on the condition that the data is transmitted and / or received. In Fig. 7, the same reference numerals are used for processes common to Fig. 4, and the description thereof will be omitted.
[0065] When a swing command is received in the remote operation support server 10 (FIG. 7 / C13), the second support processing element 122 judges the state of the work machine 40 based on the results of communication with the work machine 40 (FIG. 7 / STEP 120). When the work machine 40 is judged to be in the "first state" (FIG. 7 / STEP 120...2), the second support processing element 122 transmits a first swing command to the work machine 40 (FIG. 7 / STEP 121). On the other hand, when the work machine 40 is judged to be in the "second state" (FIG. 7 / STEP 120...2), the second support processing element 122 transmits a second swing command to the work machine 40 (FIG. 7 / STEP 122).
[0066] In the working machine 40, when a second swing command is received through the actual machine wireless communication device 422 (FIG. 7 / C42), the swing function of the actual machine imaging device 412 is controlled in a normal manner by the actual machine control device 400 in accordance with the operation manner of the pan switch SW1 and / or the tilt switch SW2, and the pan angle φ p and / or tilt angle φ tis adjusted (FIG. 7 / STEP 412). On the other hand, when the work machine 40 receives a first swing command through the real machine wireless communication device 422 (FIG. 7 / C41), the swing function of the real machine imaging device 412 is controlled by the real machine control device 400 in a more restricted manner than the normal manner in accordance with the operation manner of the pan switch SW1 and / or the tilt switch SW2 (FIG. 7 / STEP 411). At this time, the swing function is restricted in a form in which the adjustment range and / or the change speed of the attitude φ of the real machine imaging device 412 is limited, compared to when the work machine 40 is in the first state.
[0067] In the first state, when the swing operation is stopped (see times t=t0 and t=t2 in FIG. 6), until the attitude φ of the real machine imaging device 412 returns to a predetermined initial attitude φ0, the captured image or environmental image acquired through the real machine imaging device 412 may be intermittently output to the remote output interface 22, or the captured image at the time when the swing operation was stopped may be continuously output to the remote output interface 22. As a result, the discomfort felt by the operator due to the captured image or environmental image changing even though the swing operation is stopped may be reduced.
[0068] In the first state, when a swing operation is input via the remote input interface 21 (pan switch SW1 and / or tilt switch SW2), the position and / or attitude of the actual machine imaging device 412 may be controlled to a predetermined use position and / or use attitude by exerting the swing function. This allows the operator to easily perform a swing operation in parallel with work by the work machine 40. [Explanation of symbols]
[0069] 10: Remote operation support server, 20: Remote operation device, 40: Work machine, 102: Database, 121: First support processing element, 122: Second support processing element (imaging function control element), 200: Remote control device, 210: Remote input interface, 211: Remote operation mechanism, 220: Remote output interface, 221: Remote image output device, 222: Remote sound output device, 400: Actual machine control device, 410: Actual machine input interface, 420: Actual machine output interface, 424: Cab (operator's compartment), 440: Work mechanism, 445: Bucket (working part), SW1: Pan switch, SW2: Tilt switch.
Claims
1. An imaging function control system comprising: a remote control device having a remote control mechanism that is operated to operate a working mechanism of a working machine; and an actual machine imaging device that is mounted on the working machine and obtains an image including at least a part of the working mechanism and outputs the image to a remote image output device provided in the remote control device, a remote control device provided in the remote control device and recognizing an operation mode of a swing operation for adjusting an optical axis direction of the actual imaging device; An imaging function control element that exerts a swing function of changing the optical axis direction of the real imaging device to an angle according to the operation mode of the swing operation, The imaging function control element, in a first state in which the remote control device is operated to operate the working mechanism, limits the swing function more than in a second state in which the remote control device is not operated to operate the working mechanism. Imaging function control system.
2. 2. The imaging function control system according to claim 1, The imaging function control element, in the first state, restores the attitude of the real imaging device to a predetermined initial attitude, which is the attitude of the real imaging device at the start of the swing operation, by exerting a swing function when the swing operation is stopped, while, in the second state, maintains the attitude of the real imaging device as it is when the swing operation is stopped. Imaging function control system.
3. 2. The imaging function control system according to claim 1, When the swing operation is stopped in the first state, the imaging function control element causes the remote image output device to intermittently output the captured image acquired through the real imaging device until the attitude of the real imaging device returns to a predetermined initial attitude, which is the attitude of the real imaging device at the start of the swing operation, or causes the remote image output device to continuously output the captured image at the time when the swing operation is stopped. Imaging function control system.
4. A method for controlling an imaging function of an actual imaging device that is mounted on a work machine and obtains an image including at least a part of a work mechanism and outputs the image to a remote image output device provided in a remote operation device that is equipped with a remote operation mechanism that is operated to operate a work mechanism of the work machine, comprising: The process includes a process for recognizing an operation mode of a swing operation for adjusting an optical axis direction of the real imaging device by a remote control device provided in the remote operation device, and an imaging control process for exerting a swing function for changing an optical axis direction of the real imaging device to an angle corresponding to the operation mode, The imaging control process restricts the swing function in a first state in which the remote control mechanism is operated to operate the working mechanism, more than in a second state in which the remote control mechanism is not operated to operate the working mechanism. An imaging function control method comprising the steps of:
Citation Information
Patent Citations
shovel
EP3399744A1
Monitor for construction machine
JP1996020975A
Remote operation system of construction machine
JP2018021395A
Work machine
JP2018053537A
shovel
WO2017115808A1
