Photographing function control system and photographing function control method

Abstract

The present application provides a kind of shooting function control system and shooting function control method. By controlling the shooting function of the real shooting device mounted on the working machine in an appropriate manner in view of the state of the working machine, the improvement of the convenience for the operator to master the situation of the surroundings of the working machine can be realized. In the first state, compared with the second state, the swing function of the real shooting device (412) is limited. When the working machine (40) is in the first state, after the operation of the translation switch (SW1) and the tilt switch (SW2) stops, the posture (φ) of the real shooting device (412) when the operation starts again is limited to the initial posture (φ0). On the other hand, when the working machine (40) is in the second state, after the stop operation of the translation switch (SW1) and the tilt switch (SW2), the posture (φ) of the real shooting device (412) when the operation starts again is maintained on the posture (φ) at the last operation stop.

Description

Technical Field

[0001] This invention relates to a technology for controlling the shooting function of a real-machine shooting device mounted on hydraulic excavators and other operating machinery. Background Technology

[0002] One proposed technique involves an operator using a remote control device to operate a work machine while referring to a remote image showing its operational status (e.g., see Patent Document 1). The operator's experience of understanding the surroundings of the work machine differs when remotely operating it compared to physically operating it from a seat on the machine. Therefore, to allow the operator to adjust their line of sight to various angles to assess the surroundings of the work machine, similar to operating the machine physically, the system is configured to change the orientation of the camera mounted on the work machine.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2018-021395 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] However, if the swing function of the shooting device is too high or too low, when the shape of the captured image obtained by the shooting device changes, for example, if the magnification of the camera of the working machine is not turned off or the magnification of the camera is not fully restored to the minimum (initial) magnification, the understanding of the surrounding situation will be hindered, resulting in insufficient understanding of the surrounding situation required for the operation to be carried out.

[0008] Therefore, the object of the present invention is to provide a system in which, in view of the state of the working machinery, the shooting function of the real-time shooting device mounted on the working machinery is controlled in an appropriate manner, thereby improving the convenience for the operator to grasp the surrounding situation of the working machinery.

[0009] Methods for solving problems

[0010] The shooting function control system of the present invention is a system for controlling the shooting function of a real-machine shooting device mounted on a working machine for acquiring images that are output to an output interface. The working machine is controlled to operate according to operations input through an input interface.

[0011] The shooting function control system is characterized in that...

[0012] The device includes a shooting function control element that controls the swing function, which is the shooting function of the actual shooting device, based on the swing operation input through the input interface.

[0013] The shooting function control element performs the following control: in the first state, when the machine is operable or the machine is in operation, compared to the second state, when the machine is not operable or the machine is not in operation, the swing function of the shooting function of the actual shooting device is restricted.

[0014] According to the shooting function control system with this structure, in the first state, the oscillation function of the actual shooting device is restricted compared to the second state. "Oscillation function" refers to at least one of a translation function and a tilt function. That is, in the first state, compared to the second state, if the actual shooting device has both translation and tilt functions, at least one of these functions is restricted; if the actual shooting device only has either a translation or tilt function, that function is restricted. As a result, the amount of change in the actual spatial area reflected in the output interface is controlled, and correspondingly, it is possible to avoid a situation where the operator's inattention is distracted in the first state, thus hindering the operation of the working machinery input through the input interface.

[0015] Brief description of the attached diagram

[0016] Figure 1 This is an explanatory diagram of the structure of a remote operation assistance system.

[0017] Figure 2 This is an explanatory diagram of the structure of the remote operation device.

[0018] Figure 3 It is an explanatory diagram related to the structure of the operating machinery.

[0019] Figure 4 This is an explanatory diagram of the functions of the remote operation assistance system (shooting function control system).

[0020] Figure 5 It is an explanatory diagram of the working environment.

[0021] Figure 6 This is an explanatory diagram of the control method for the swing function of the actual shooting device.

[0022] Figure 7 This is an explanatory diagram relating to the functions of the remote operation assistance system (shooting function control system) in other embodiments. Detailed Implementation

[0023] (Structure of a remote operation assistance system)

[0024] Figure 1 The remote operation assistance system shown consists of a remote operation assistance server 10 and a remote operation device 20 for remotely operating the work machinery 40. The remote operation assistance server 10, the remote operation device 20, and the work machinery 40 are configured to communicate with each other via a network. The communication network between the remote operation assistance server 10 and the remote operation device 20, and the communication network between the remote operation assistance server 10 and the work machinery 40, can use the same communication network or different communication networks.

[0025] (Structure of a remote operation auxiliary server)

[0026] The remote operation assistance server 10 includes a database 102, a first auxiliary processing element 121, and a second auxiliary processing element 122. The database 102 stores and saves captured image data, etc. The database 102 may also be configured as a database server independent of the remote operation assistance server 10. Each auxiliary processing element is configured as a processing unit (a single-core processor or a multi-core processor, or a processor core constituting such a processor), reads necessary data and software from a storage device such as a memory, and performs computational processing according to the software, described later, on the object of that data. In this embodiment, the remote operation assistance server 10 constitutes a "capture function control system," and the second auxiliary processing element 122 constitutes a "capture function control element."

[0027] (Structure of the remote operation device)

[0028] The remote operation device 20 includes a remote control device 200, a remote input interface 210, and a remote output interface 220. The remote control device 200 is composed of a processing unit (a single-core processor or a multi-core processor, or a processor core constituting the processor), which reads necessary data and software from storage devices such as memory, and performs arithmetic processing according to the software based on the data.

[0029] The remote input interface 210 includes a remote operation mechanism 211. The remote output interface 220 includes a remote image output device 221, a remote audio output device 222, and a remote wireless communication device 224.

[0030] The remote control mechanism 211 includes a travel control device, a slewing control device, a boom control device, a stick control device, and a bucket control device. Each control device has a lever for receiving rotational operations. The travel control device's lever (travel lever) is operated to move the lower traveling body 410 of the work machinery 40. The travel lever can also serve as a travel pedal. For example, a travel pedal fixed to the base or lower end of the travel lever can be provided. The slewing control device's lever (slewing lever) is operated to actuate the hydraulic slewing motor constituting the slewing mechanism 430 of the work machinery 40. The boom control device's lever (boom lever) is operated to actuate the boom cylinder 442 of the work machinery 40. The stick control device's lever (arm lever) is operated to actuate the stick cylinder 444 of the work machinery 40. The bucket control device's lever (bucket stick) is operated to actuate the bucket cylinder 446 of the work machinery 40.

[0031] like Figure 2 As shown, the levers constituting the remote operation mechanism 211 are arranged, for example, around a seat St for the operator to sit on. The seat St can be a high-backed chair with armrests, a low-backed chair without a headrest, or a chair without a backrest, or any other type of seating where the operator can sit.

[0032] A pair of left and right travel levers 2110 are arranged in front of the seat St, corresponding to the left and right tracks. One lever can function as multiple control levers. For example, when controlling... Figure 2 The left-side operating lever 2111, located in front of the left side frame of the seat St, functions as an arm when operated in the forward / backward direction and as a slewing lever when operated in the left / right direction. Similarly, when operating on... Figure 2 The right-side operating lever 2112, located in front of the right-side frame of the seat St, functions as a boom lever when operated in the forward / backward direction and as a bucket lever when operated in the left / right direction. The lever mode can be changed arbitrarily according to the operator's instructions.

[0033] On the operator-held portions of the left and right operating levers 2111 and 2112, respectively, are provided a pan switch SW1 and a tilt switch SW2 for adjusting the pan and tilt angles of the camera shooting device 412, as described later. Alternatively, only one of the left and right operating levers 2111 and 2112 may be provided with both pan and tilt switches SW1 and SW2. The pan and tilt switches SW1 and SW2 are configured to return the operating posture to its initial position under certain conditions via an actuator or spring. If the camera shooting device 412 only has one of the pan or tilt functions as a swing function, one of the pan and tilt switches SW1 or SW2 may be omitted.

[0034] For example Figure 2 As shown, the remote image output device 221 consists of a central remote image output device 2210, a left remote image output device 2211, and a right remote image output device 2212, each with a roughly rectangular screen. The central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 are respectively positioned in front of, diagonally to the left front of, and diagonally to the right front of the seat St. The shape and size of the screen (image display area) of each of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 can be the same or different.

[0035] like Figure 2 As shown, the right edge of the left remote image output device 2211 is adjacent to the left edge of the central remote image output device 2210, with the screen of the central remote image output device 2210 and the screen of the left remote image output device 2211 forming a tilt angle θ1 (e.g., 120°≤θ1≤150°). Figure 2 As shown, the screens of the central remote image output device 2210 and the right remote image output device 2212 are tilted at an angle θ2 (e.g., 120°≤θ2≤150°), with the left edge of the right remote image output device 2212 adjacent to the right edge of the central remote image output device 2210. These tilt angles θ1 and θ2 can be the same or different angles.

[0036] The screens of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 can be parallel or tilted relative to the vertical direction. At least one of the image output devices 2210, 2211, and 2212 can also be composed of multiple image output devices. For example, the central remote image output device 2210 can also be composed of a pair of vertically adjacent image output devices with a generally rectangular screen shape.

[0037] The remote audio output device 222 consists of one or more speakers. For example, Figure 2 As shown, the remote audio output device 222 consists of a central audio output device 2220, a left audio output device 2221, and a right audio output device 2222, respectively located behind the seat St, behind the left armrest, and behind the right armrest. The specifications of the central audio output device 2220, the left audio output device 2221, and the right audio output device 2222 can be the same or different.

[0038] (Structure of the operating machinery)

[0039] The operating machinery 40 includes a machine control device 400, a machine input interface 41, a machine output interface 42, and a working mechanism 440. The machine control device 400 includes an image processing device 30. The image processing device 30 includes a status detection element 31, an image prediction element 32, and an image compression element 34. The components of the machine control device 400 and the image processing device 30 are respectively composed of a computing processing device (a single-core processor or a multi-core processor, or a processor core constituting the processor), which reads necessary data and software from storage devices such as memory, and performs calculations according to the software based on the data.

[0040] Construction machinery 40, for example, is a tracked excavator (construction machinery), such as Figure 3 As shown, the device includes a tracked lower traveling body 410 and an upper rotating body 420 rotatably mounted on the lower traveling body 410 via a rotating mechanism 430. A driver's cab 424 is provided on the front left side of the upper rotating body 420. A working mechanism 440 is provided on the front center of the upper rotating body 420.

[0041] The machine input interface 41 includes a machine operation mechanism 411, a machine imaging device 412, and a machine positioning device 414. The machine operation mechanism 411 has multiple joysticks arranged similarly to the remote operation mechanism 211 around the seat located inside the cab 424. A drive mechanism or robot that receives signals corresponding to the operation mode of the remote joysticks and moves the machine joysticks based on these received signals is located in the cab 424. The machine imaging device 412 is located, for example, inside the cab 424, and captures images of the environment containing at least a portion of the operating mechanism 440 through the front window and a pair of side windows. This environment may omit some or all of the front and side windows. The machine positioning device 414 consists of a GPS device and, if necessary, a gyroscope sensor, etc.

[0042] The actual output interface 42 is equipped with actual wireless communication device 422.

[0043] like Figure 3 As shown, the working mechanism 440, which serves as the working mechanism, includes: a boom 441, which is mounted on the upper rotating body 420 for lifting and lowering; a stick 443, which is rotatably connected to the top of the boom 441; and a bucket 445, which is rotatably connected to the top of the stick 443. The working mechanism 440 is equipped with a boom cylinder 442, a stick cylinder 444, and a bucket cylinder 446, all of which are telescopic hydraulic cylinders.

[0044] Boom cylinder 442 is located between boom 441 and upper slewing body 420, allowing it to extend and retract by receiving a supply of working oil, thereby causing boom 441 to rotate in the lifting direction. Stick cylinder 444 is located between stick 443 and boom 441, allowing it to extend and retract by receiving a supply of working oil, thereby causing stick 443 to rotate about a horizontal axis relative to boom 441. Bucket cylinder 446 is located between bucket 445 and stick 443, allowing it to extend and retract by receiving a supply of working oil, thereby causing bucket 445 to rotate about a horizontal axis relative to stick 443.

[0045] (Function)

[0046] use Figure 4 The flowchart shown illustrates the functions of the remote operation assistance system and the shooting function control system with the above-described structure. In this flowchart, the "C●" block is a symbol used for simplification and refers to the transmission and / or reception of data, as well as a conditional branch that executes branch direction processing based on the transmission and / or reception of such data.

[0047] In the remote operation device 20, an environment confirmation request is sent to the remote operation assistance server 10 via the remote wireless communication device 224. Figure 4 / Step 210 (STEP 210)). For example, the situation where the operator performs a specified operation through the remote input interface 210 may be determined as a necessary condition for initiating the sending of the environmental confirmation request. "Specified operation" is, for example, an operation such as a click on the remote input interface 210 by the operator to specify the work machinery 40 that they want to perform remote operation.

[0048] In the remote operation assistance server 10, upon receiving an environmental confirmation request, the first auxiliary processing element 121 sends the environmental confirmation request to the target machine 40. Figure 4 / C10).

[0049] In the operating machinery 40, when an environmental confirmation request is received via the actual machine wireless communication device 422 ( Figure 4 / C40), the actual control device 400 acquires the captured image through the actual shooting device 412, and sends the captured image data representing the captured image to the remote operation assistance server 10 through the actual wireless communication device 422. Figure 4 / Step 410).

[0050] In the remote operation assistance server 10, when the first assistance processing element 121 receives the captured image data ( Figure 4 / C11), the second auxiliary processing element 122 sends environmental image data corresponding to the captured image to the remote operation device 20 ( / C11), Figure 4 / Step 110). Environmental image data is not only the captured image data itself, but also image data representing simulated environmental images generated based on the captured images.

[0051] In the remote operation device 20, when environmental image data is received via the remote wireless communication device 224 ( Figure 4 / C21), the remote control device 200 outputs the environmental image corresponding to the environmental image data to the remote image output device 221 (C21). Figure 4 / Step 212).

[0052] Therefore, for example Figure 5 As shown, by defining the window frame of the cab 424, an environmental image of the boom 441, stick 443 (which is part of the working mechanism 440), and a pile of rubble or sand (which is the object of the work of the bucket 445) that is reflected in front of the cab 424 is output to the remote image output device 221.

[0053] In the remote operation device 20, the remote control device 200 identifies the swing operation mode of the remote operation mechanism 211, and sends a swing command corresponding to the swing operation mode to the remote operation auxiliary server 10 via the remote wireless communication device 224. Figure 4 / Step 214). Determine the swing operation mode based on the operating posture of the translation switch SW1 and tilt switch SW2 located on the respective gripping parts of the left operating lever 2111 and the right operating lever 2112 (refer to...). Figure 2 The translation switch SW1 and tilt switch SW2 are, for example, slide switches or lever switches capable of detecting operating postures (operation amounts). These switches can detect operation amounts to one side and operation amounts to the other side, and are in a neutral position when no operation is performed.

[0054] Based on the operating posture range of translation switch SW1 [θ] p1 θ p2 Any operation posture θ in ] p Generates the translation angle φ for the actual shooting device 412. p Controlled within the translation angle range [φ] p1 φ p2 The translation angle f in ] p (θ p The swing command of function f. p For example, it can be defined by relation (01).

[0055] f p (θ p )={(φ p2 -φ p1 ) / (θ p2 -θ p1 )}(θ p -θ p1 )+φ p1 ‥(01).

[0056] Based on the operating posture range of tilt switch SW2 [θ] t1 θ t2 Any operation posture θ in ] t Generates the tilt angle φ for the actual shooting device 412. t Controlled within the tilt angle range [φ] t1 , φ t2 The tilt angle f in ] t (θ t The swing command of function f. t For example, it can be defined by relation (02).

[0057] f t (θ t )={(φ t2 -φ t1 ) / (θ t2 -θ t1 )}(θ t -θ t1 )+φt1 ‥(02).

[0058] In the remote operation assistance server 10, upon receiving a swing command, the second auxiliary processing element 122 (shooting function control element) sends the swing command to the operating machine 40. Figure 4 / C12).

[0059] In the operating machinery 40, when a normal swing command is received via the actual machine wireless communication device 422 ( Figure 4 / C42), the actual control device 400 controls the swing function of the actual shooting device 412 (more precisely, the translation drive mechanism and / or tilt drive mechanism of the actual shooting device 412), and adjusts its translation angle φ. p and / or tilt angle φ t ( Figure 4 / Step 412).

[0060] As a result, the actual spatial area in the captured image obtained by the actual shooting device 412, and subsequently reflected in the environmental image output to the remote image output device 221, changes (see reference). Figure 4 Step 410 → … → Step 212 Figure 5 For example, in the optical axis direction, by the translation angle φ of the actual shooting device 412. p When the direction of the camera changes from in front of the machine 40 to diagonally forward to the left, the actual spatial area reflected in the environmental image changes from the area in front of the machine 40 to the area diagonally forward to the left. Furthermore, the tilt angle φ of the actual camera 412 along the optical axis direction... t When the image is controlled to change from horizontal forward to diagonally downward, the actual spatial area reflected in the environmental image changes to a lower area than before.

[0061] In the remote operation device 20, the remote control device 200 determines whether the swing operation has stopped. Figure 4 / Step 216). For example, it is determined whether the swing operation has stopped based on whether the change in the operating posture of the translation switch SW1 and the tilt switch SW2 is below a threshold (the state in which the translation switch SW1 and the tilt switch SW2 are not operated) has lasted for a specified period.

[0062] If the judgment result is affirmative ( Figure 4 / Step 216…Yes (YES)), the remote control device 200 sends a swing stop command to the remote operation assistance server 10. Figure 4 / Step 218). On the other hand, if the determination result is negative ( Figure 4 / Step 216…No (NO)), do not generate a swing stop command and execute subsequent processing.

[0063] In the remote operation assistance server 10, upon receiving a swing stop command ( Figure 4 / C14), the second auxiliary processing element 122 determines the status of the working machine 40 based on the communication results with the working machine 40. Figure 4 / Step 120). "First state" refers to the state in which the machine 40 can be operated (or the state in which the machine 40 is operating). "Second state" refers to the state in which the machine 40 cannot be operated (or the state in which the machine 40 is stopped). For example, whether the machine 40 is in the first state or the second state can be determined by whether the hydraulic cutting lever of the actual machine operating mechanism 411 constituting the machine 40 is in a prescribed posture. Alternatively, whether the machine 40 is in the first state or the second state can also be determined by whether the remote hydraulic cutting lever, which serves as a remote input interface and mimics the hydraulic cutting lever, is in a prescribed posture.

[0064] Under the condition that the operating machinery 40 is in the first state ( Figure 4 / Step 120…1), the second auxiliary processing element 122 sends an initial posture recovery command to the working machine 40 ( Figure 4 / Step 124). On the other hand, if it is determined that the operating machinery 40 is in the second state ( Figure 4 / Step 120…2), do not generate initial pose recovery instructions and execute subsequent processing.

[0065] In the operating machine 40, when the initial posture recovery command is received via the actual machine wireless communication device 422 ( Figure 4 / C44), the actual machine control device 400 controls the swing function of the actual machine shooting device 412, and moves it by an angle φ. p and / or tilt angle φ t Adjust to the initial angle or initial posture ( Figure 4 / Step 414).

[0066] For example, such as Figure 6 As shown, consider the following case: at time t = t s The swinging operation begins, stops at time t=t1, restarts at time t=t2, and stops at time t=t3.

[0067] In this case, if the operating machinery 40 is in the first state, then as follows: Figure 6 As shown by the single-dotted line in the middle, the following control is performed: the posture φ of the actual shooting device 412 = (φ p φ tThe device returns to its initial posture φ0 at time t=t1. After the operation of translation switch SW1 and / or tilt switch SW2 changes from time t=t2, it returns to its initial posture φ0 again at time t=t3. Therefore, the operator starts operating translation switch SW1 and / or tilt switch SW2 at time t=t2, thereby adjusting the posture of the camera device 412 from the initial posture φ0. Alternatively, if the camera device 412 returns to its initial posture φ0 based on its posture φ, the second auxiliary processing element 122 sends an initial posture recovery command to the remote operation device 20 to restore the operating posture θ of translation switch SW1 and tilt switch SW2 to their initial operating posture θ0.

[0068] On the other hand, if the operating machinery 40 is in the second state, then as follows: Figure 6 As shown by the double-dotted line, the following control is performed: the posture φ of the actual shooting device 412 = (φ p φ t The device maintains its original posture φ(t1) at time t=t1. After the operation of the translation switch SW1 and / or tilt switch SW2 changes starting from time t=t2, the original posture φ(t3) is maintained at time t=t3. Therefore, when the operator starts operating the translation switch SW1 and / or tilt switch SW2 at time t=t2, the posture φ of the actual shooting device 412 is adjusted from the posture φ(t1) at the time when the most recent operation stopped.

[0069] In the remote operation device 20, the remote control device 200 identifies the operation mode of the remote operation mechanism 211, and sends a remote operation command corresponding to the operation mode to the remote operation auxiliary server 10 via the remote wireless communication device 224. Figure 4 / Step 220).

[0070] In the remote operation auxiliary server 10, when the second auxiliary processing element 122 receives the remote operation command, the first auxiliary processing element 121 sends the remote operation command to the operating machine 40. Figure 4 / C16).

[0071] In the operating machinery 40, when the actual machine control device 400 receives the operation command through the actual machine wireless communication device 422 ( Figure 4 / C46), controls the actions of the operating mechanism 440, etc. Figure 4 (Step 420). For example, performing an operation in which the bucket 445 scoops up the soil in front of the working machine 40 and dumps the soil from the bucket 445 after the upper rotating body 420 is rotated.

[0072] (Effects)

[0073] According to the shooting function control system constituting the remote operation assistance system having the above structure, compared with the second state, the swing function of the actual shooting device 412 is limited in the first state (see reference). Figure 4 / Step 120…1→Step 124→C44→Step 414, refer to Figure 6 ).

[0074] In detail, when the machine 40 is in the first state, after the operation of the translation switch SW1 and tilt switch SW2 stops, the posture φ of the actual shooting device 412 when the operation restarts is limited to the initial posture φ0. As a result, after the operator has grasped the actual spatial area corresponding to the posture when the swing operation of the actual shooting device stops, the operator can grasp the actual spatial area corresponding to the initial posture of the actual shooting device without having to perform swing operation through the remote input interface 21 (translation switch SW1 and / or tilt switch SW2).

[0075] On the other hand, when the machine tool 40 is in the second state, after the operation of the translation switch SW1 and tilt switch SW2 has stopped, the posture φ when the actual shooting device 412 is restarted remains at the posture φ at the most recent operation stop. As a result, during the period when the operation of the machine tool 40 is stopped, the actual shooting device 412 can perform a swing function according to any swing operation mode performed by the operator via the remote input interface 210 (translation switch SW1 and / or tilt switch SW2), thereby enabling the operator to grasp the actual spatial area reflected in the captured image obtained by the actual shooting device 412.

[0076] (Other embodiments of the present invention)

[0077] In the above embodiment, the remote operation auxiliary server 10 constitutes the shooting function control system, and the second auxiliary processing element 122 constitutes the shooting function control element. However, as another embodiment, the shooting function control system may also be constituted by the operating machine 40 and / or the remote operation device 20, and the shooting function control element may be constituted by the actual machine control device 400 and / or the remote control device 200.

[0078] In the above embodiment, compared with the second state, the first state restricts the swing function by limiting the posture φ of the real camera 412 when the swing operation in the remote operation device 20 starts again. However, as another embodiment, the swing function can also be restricted by limiting the adjustment range and / or change speed of the posture φ of the real camera 412.

[0079] In this case, it can also be done according to Figure 7 The flowchart shown replaces Figure 4 The flowchart shown executes a series of processes. In this flowchart, the "C●" block is a symbol used for simplification and refers to the sending and / or receiving of data, as well as conditional branches that execute processing in a specific direction based on the sending and / or receiving of that data. Figure 7 In China, targeting and Figure 4 Common treatments use the same reference numerals, and their descriptions are omitted.

[0080] In the remote operation assistance server 10, upon receiving a swing command ( Figure 7 / C13), the second auxiliary processing element 122 determines the status of the working machine 40 based on the communication results with the working machine 40. Figure 7 / STEP120). Under the condition that the operating machinery 40 is determined to be in "first state" ( Figure 7 / Step 120…1), the first swing command is sent to the working machine 40 through the second auxiliary processing element 122 ( Figure 7 / Step 121). On the other hand, if it is determined that the operating machinery 40 is in the "second state" ( Figure 7 / Step 120…2), the second auxiliary processing element 122 sends a second swing command to the working machine 40 ( Figure 7 / Step 122).

[0081] In the operating machinery 40, when a second swing command is received via the actual machine wireless communication device 422 ( Figure 7 / C42), the actual camera control device 400 controls the swing function of the actual camera shooting device 412 in a normal manner according to the operation mode of the translation switch SW1 and / or tilt switch SW2, and adjusts its translation angle φ. p and / or tilt angle φ t ( Figure 7 / Step 412). On the other hand, in the working machine 40, when the first swing command is received via the actual machine wireless communication device 422 ( Figure 7 / C41), the camera control device 400 controls the swing function of the camera shooting device 412 in a manner that corresponds to the operation mode of the pan switch SW1 and / or tilt switch SW2, but is more restrictive than the usual mode. Figure 7 / Step 411). At this time, compared with the situation where the working machine 40 is in the first state, the swing function is restricted in a way that the adjustment range and / or change speed of the posture of the actual shooting device 412 are limited.

[0082] In the first state, when the swinging operation has stopped (refer to...) Figure 6At times t=t1 and t=t3, the captured images or environmental images obtained by the actual shooting device 412 can be intermittently output to the remote output interface 220 during the period until the posture φ of the actual shooting device 412 returns to the predetermined initial posture φ0, or the captured images when the swing operation stops can be continuously output to the remote output interface 22. As a result, even when the swing operation stops, the operator's sense of incongruity due to changes in the captured images or environmental images can be reduced.

[0083] In the first state, when a swing operation is input via the remote input interface 210 (translation switch SW1 and / or tilt switch SW2), the position and / or posture of the live-action shooting device 412 can also be controlled at a specified operating position and / or operating posture by utilizing the swing function. This allows the operator to easily perform swing operations in parallel with the operation of the working machine 40.

[0084] In the shooting function control system having the above structure, it is preferable that the shooting function control element performs the following control: in the first state, when the swing operation is stopped through the input interface, the posture of the actual shooting device is restored to the predetermined initial posture by exercising the swing function; on the other hand, in the second state, when the swing operation is stopped through the input interface, the posture of the actual shooting device is maintained as before.

[0085] According to the shooting function control system with this structure, when the swing operation stops in the first state, the posture of the actual shooting device returns to its initial posture. As a result, after the operator has grasped the actual spatial area corresponding to the posture of the actual shooting device when the swing operation stops, the operator can grasp the actual spatial area corresponding to the initial posture of the actual shooting device without needing to perform a swing operation through the input interface. On the other hand, when the swing operation stops in the second state, the posture of the actual shooting device is maintained as before. As a result, during the period when the operation of the working machinery is stopped, the actual shooting device can perform a swing function according to any swing operation performed by the operator through the input interface, allowing the operator to grasp the actual spatial area reflected in the captured image obtained by the actual shooting device.

[0086] In the shooting function control system having the above structure, it is preferable that the shooting function control element performs the following control: when the swing operation is stopped through the input interface in the first state, during the period until the posture of the actual shooting device is restored to the predetermined initial posture, the shooting image obtained by the actual shooting device is intermittently output to the output interface, or the shooting image when the swing operation stops is continuously output to the output interface.

[0087] According to the shooting function control system with this structure, when the swing operation is stopped via the input interface in the first state, the image captured when the actual shooting device returns to its initial position is output intermittently via the output interface (or, the image captured when the operation stops is output continuously). As a result, the operator's sense of incongruity regarding the change in the captured image even though the swing operation has stopped can be reduced.

[0088] In the shooting function control system having the above structure, it is preferable that the shooting function control element performs the following control: when the above swing operation is input through the above input interface in the above first state, the swing function is used to make the posture of the actual shooting device become the prescribed usage posture.

[0089] According to the shooting function control system with this structure, in the first state, the posture of the actual shooting device can be made into a prescribed operating posture regardless of the magnitude of the swing operation input through the input interface. This allows the operator to easily perform the swing operation in parallel with the operation of the machine.

[0090] Symbol Explanation

[0091] 10: Remote operation auxiliary server; 20: Remote operation device; 40: Working machinery; 102: Database; 121: First auxiliary processing element; 122: Second auxiliary processing element (shooting function control element); 200: Remote control device; 210: Remote input interface; 211: Remote operation mechanism; 220: Remote output interface; 221: Remote image output device; 222: Remote audio output device; 400: Real machine control device; 41: Real machine input interface; 42: Real machine output interface; 424: Cab (driver's cab); 440: Working mechanism; 445: Bucket (working part); SW1: Translation switch; SW2: Tilt switch.

Claims

1. A shooting function control system, comprising: a remote operating device having a remote operating mechanism for operating a working mechanism of a machine; and a real-time shooting device mounted on the machine for acquiring a captured image, including at least a portion of the working mechanism, which is output to a remote image output device of the remote operating device. The shooting function control system is characterized by having: The remote control device of the remote operation device recognizes the operation mode of the swing operation used to adjust the optical axis direction of the actual shooting device; and The shooting function control element functions to change the optical axis direction of the actual shooting device to an angle corresponding to the swing operation mode. The shooting function control element performs the following control: in a first state where the operation of the working mechanism is in progress by operating the remote operating device, compared with a second state where the operation of the working mechanism is not in progress by operating the remote operating device, the swing function is restricted.

2. The shooting function control system according to claim 1, characterized in that, The shooting function control element performs the following control: in the first state, when the swing operation is stopped, the swing function is used to restore the posture of the actual shooting device to the predetermined initial posture of the actual shooting device when the swing operation begins; on the other hand, in the second state, when the swing operation is stopped, the posture of the actual shooting device is maintained as before.

3. The shooting function control system according to claim 1, characterized in that, The shooting function control element performs the following control: when the swing operation is stopped in the first state, during the period until the posture of the actual shooting device is restored to the predetermined initial posture of the actual shooting device when the swing operation begins, the captured image obtained by the actual shooting device is intermittently output to the remote image output device, or the captured image when the swing operation stops is continuously output to the remote image output device.

4. A shooting function control method for controlling the shooting function of a live shooting device, the live shooting device being mounted on a working machine for acquiring a captured image of at least a portion of the working mechanism of the working machine, which is output to a remote image output device of a remote operating device, the remote operating device having a remote operating mechanism for operating the working mechanism to cause it to move. The shooting function control method is characterized in that... The steps performed by the arithmetic processing device include: The remote control device of the remote operation device identifies the swing operation mode, and the swing operation is used to adjust the optical axis direction of the actual shooting device; In addition, the shooting control processing enables the optical axis direction of the actual shooting device to change to an angle corresponding to the operating mode, thus providing a swing function. The shooting control process includes the following steps: in a first state where the operation of the working mechanism is in progress by operating the remote operating mechanism, compared with a second state where the operation of the working mechanism is not in progress by operating the remote operating mechanism, the swing function is restricted.

Images