Shooting system and robotic system

Abstract

The photographing system (100) comprises: an unmanned aerial vehicle (120); a photographing device (110) mounted on the unmanned aerial vehicle (120) for photographing a robot (200) operating an object; a display device (140) disposed at a position away from the unmanned aerial vehicle (120) for displaying images photographed by the photographing device (110) to a user operating the robot (200); and a control device (150) for controlling the actions of the photographing device (110) and the unmanned aerial vehicle (120). The control device obtains information related to the action of the robot (200), i.e., action-related information, and moves the unmanned aerial vehicle (120) by changing the position and orientation of the photographing device (110) in accordance with the action-related information.

Description

Technical Field

[0001] The present disclosure relates to a camera system and a robot system. Background Art

[0002] Conventionally, there are technologies for displaying an image of an object captured by a camera on a monitor while accepting remote control of a robot from a user visually checking the monitor. For example, Patent Document 1 discloses a robot system that displays an image of the robot on a monitor based on image data acquired by an unmanned aerial vehicle (UAV) equipped with a camera capable of changing its posture.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2019-93471

[0004] The robot system disclosed in Patent Document 1 includes a robot manipulator for operating the robot and an aircraft manipulator for operating the drone. For example, to simultaneously operate the robot and the drone, the operator must manually operate both the robot manipulator and the aircraft manipulator, making simultaneous operation difficult. Summary of the Invention

[0005] An object of the present disclosure is to provide an imaging system and a robot system that simplify the operation of an unmanned aerial vehicle equipped with an imaging device.

[0006] A shooting system involved in one form of the present disclosure includes: an unmanned aerial vehicle; a shooting device, mounted on the above-mentioned unmanned aerial vehicle, for shooting a robot working on an object; a display device, arranged at a position away from the above-mentioned unmanned aerial vehicle, for displaying images shot by the above-mentioned shooting device to a user operating the above-mentioned robot; and a control device, for controlling the movement of the above-mentioned shooting device and the above-mentioned unmanned aerial vehicle, the above-mentioned control device obtains information related to the movement of the above-mentioned robot, that is, movement-related information, and moves the above-mentioned unmanned aerial vehicle in a manner that changes the position and orientation of the above-mentioned shooting device corresponding to the above-mentioned movement-related information. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 It is a perspective view showing an example of the configuration of a robot system according to an exemplary embodiment.

[0008] Figure 2 It is a perspective view showing an example of the structure of an unmanned aerial vehicle according to an exemplary embodiment.

[0009] Figure 3 This is a block diagram showing an example of the functional structure of an imaging system according to an exemplary embodiment.

[0010] Figure 4This is a flowchart showing an example of the operation of the imaging system according to the exemplary embodiment.

[0011] Figure 5 It is a perspective view showing an example of the configuration of a robot system according to Modification 1 of the exemplary embodiment.

[0012] Figure 6 This is a block diagram showing an example of the functional configuration of an imaging system according to Modification 1.

[0013] Figure 7 This is a perspective view showing another example of the flight restriction area for the unmanned aerial vehicle in Modification 1.

[0014] Figure 8 It is a perspective view showing an example of the configuration of a robot system according to Modification 2 of the exemplary embodiment.

[0015] Figure 9 This is a block diagram showing an example of the functional configuration of an imaging system according to Modification 2.

[0016] Figure 10 This is a flowchart showing an example of the operation of the imaging system according to Modification 2. DETAILED DESCRIPTION

[0017] Hereinafter, the exemplary embodiments of the present disclosure will be described with reference to the accompanying drawings. In addition, the exemplary embodiments described below are all examples that represent generality or specificity. In addition, among the constituent elements in the following exemplary embodiments, the constituent elements that are not recorded in the independent claims representing the highest concept are described as arbitrary constituent elements. In addition, each figure in the accompanying drawings is a schematic diagram and is not necessarily a strict illustration. Moreover, in each figure, there is a case where substantially the same constituent elements are marked with the same figure mark, and repeated descriptions are omitted or simplified. In addition, in this specification and claims, "device" can refer not only to one device, but also to a system composed of multiple devices.

[0018] [Structure of the robot system]

[0019] The configuration of a robot system 1 according to an exemplary embodiment will be described. Figure 1 1 is a perspective view showing an example of the structure of the robot system 1 according to the exemplary embodiment. Figure 1 As shown, the robot system 1 includes an imaging system 100 , a robot 200 , a robot operating device 300 , and a robot control device 400 . The imaging system 100 includes an imaging device 110 , an unmanned aerial vehicle 120 , an imaging input device 130 , a display device 140 , and an imaging control device 150 .

[0020] In this exemplary embodiment, robot 200 is an industrial robot comprising a robot arm 210, a base 220, and an end effector 230, but the present invention is not limited thereto. Alternatively, robot 200 may be a service robot, a medical robot, a drug development robot, or a humanoid robot. Service robots are robots used in various service industries, such as nursing, medical care, cleaning, security, guidance, rescue, cooking, and product delivery.

[0021] The base 220 is fixed on a supporting surface to support the robot arm 210. The supporting surface of the base 220 can be an immovable surface such as the ground, or a movable surface on a movable device such as a traveling device. The robot arm 210 has at least one joint 211 and at least one degree of freedom. The robot arm 210 has a joint driving device 212 for driving each joint 211. The joint driving device 212 includes a servo motor as an electric motor for driving the joint 211. The servo motor includes: an encoder for detecting the amount of rotation of the motor; and a current sensor for detecting the current value of the motor. In addition, in the servo motor, the current sensor is not necessary.

[0022] The robot arm 210 is configured such that an end effector 230 is mounted at the front end of the robot arm 210. The robot arm 210 is capable of moving the end effector 230 so that the position and posture of the end effector 230 can be freely changed. The end effector 230 is configured to perform various actions on an object (also referred to as a "workpiece") W, such as gripping, suction, spraying of liquids such as paint, welding, and injection of sealants, depending on the application of the end effector 230.

[0023] Furthermore, in this exemplary embodiment, the robot arm 210 is a vertical multi-jointed robot arm with six degrees of freedom (DOF) including six rotational joints 211, but the present invention is not limited thereto. The robot arm 210 may be of any type, such as a horizontal multi-jointed robot arm, a polar coordinate robot arm, a cylindrical coordinate robot arm, or a rectangular coordinate robot arm. The joints 211 of the robot arm 210 may be any joint, such as direct-acting joints. The number of joints 211 of the robot arm 210 may be any number, such as five or fewer or seven or more.

[0024] The robot operating device 300 is located at a distance from the robot 200 and is used to remotely operate the robot 200. The robot operating device 300 can be located at a location where a user P using the robot operating device 300 can directly visually confirm the robot 200, or it can be located at a location where the user P cannot directly visually confirm the robot 200. For example, the robot operating device 300 can be located in a space separated from the space where the robot 200 is located, or in a space located away from that space.

[0025] The robot operating device 300 receives input of various instructions, information, and data, and outputs them to the robot control device 400. For example, the robot operating device 300 can receive input from a user P. For example, the robot operating device 300 can be connected to other devices and receive input from the devices. For example, the robot operating device 300 can have a known input mechanism such as a joystick, a button, a touch panel, a joystick, a motion capture device, a camera, and a microphone. For example, the robot operating device 300 can have a teaching pendant that is one of the teaching devices, a smart device such as a smartphone or a tablet, a personal computer, and a terminal device such as a dedicated terminal device. For example, when the robot 200 is controlled in a master-slave manner, the robot operating device 300 can have a host. For example, the host can be configured to perform the same or similar actions as the robot arm 210.

[0026] The robot control device 400 controls the movement of the robot 200. The robot control device 400 is connected to the robot 200 and the robot operating device 300 via wired or wireless communication. However, any wired or wireless communication method may be used. The robot control device 400 processes commands, information, and data input via the robot operating device 300. The robot control device 400 may also be configured to connect to an external device and receive and process commands, information, and data from the device.

[0027] For example, the robot control device 400 controls the movement of the robot 200 according to the aforementioned commands, information, and data. It also controls the supply of power and other information to the robot 200. It also manages information used to manage the robot 200. For example, the robot control device 400 controls the driving of the joint drive devices 212 of the robot arm 210 and the driving of the end effector 230. The robot control device 400 determines the rotational drive amount of each joint 211 required to move the end effector 230 of the robot arm 210 and drives each joint drive device 212 according to the determined rotational drive amount. Furthermore, the robot control device 400 uses the rotational drive amount and current values detected by the encoders and current sensors of the servo motors of each joint drive device 212 as feedback information to control the drive direction, drive amount, drive speed, drive acceleration, and drive torque of each joint 211. This allows the robot arm 210 to move the end effector 230 to a desired position and posture and to apply a desired force through the end effector 230. Furthermore, the robot control device 400 may obtain the current value of the servo motor from another mechanism such as a circuit that controls the current supplied to the servo motor.

[0028] The robot control device 400 also outputs various commands, information, and data to the robot operating device 300 and / or the display device 140 of the imaging system 100. For example, the robot control device 400 causes the display device 140 to visually and / or auditorily present various commands, information, and data. For example, the robot control device 400 may output images for operating the robot 200, images indicating the status of the robot 200, and images for managing the robot 200.

[0029] The robot control device 400 includes a computer. Furthermore, the robot control device 400 may include circuits for controlling the power supplied to the robot 200; devices for controlling power other than electricity, such as air pressure and hydraulic pressure, supplied to the robot 200; and devices for controlling the supply of materials such as cooling water and paint to the robot 200. Devices other than the computer may also be provided separately from the robot control device 400.

[0030] Figure 2 It is a perspective view showing an example of the structure of an unmanned aerial vehicle 120 according to an exemplary embodiment. Figure 3 1 is a block diagram showing an example of the functional structure of the imaging system 100 according to the exemplary embodiment. Figures 1 to 3 As shown, the imaging device 110 of the imaging system 100 is mounted on an unmanned aerial vehicle 120. The imaging device 110 includes a camera 111 for capturing still and / or moving digital images, a gimbal 112 for supporting the camera 111 on the unmanned aerial vehicle 120, and a camera control device 113. In this exemplary embodiment, the camera 111 is a visible light camera only, but other types of cameras, such as a 3D camera, can also be used to capture 3D images including positional information of the subject within the image.

[0031] The gimbal 112 is configured to be mounted on the unmanned aerial vehicle 120 and to carry the camera 110. The gimbal 112 is operable to freely change the orientation of the camera 110. In this exemplary embodiment, the gimbal 112 is electrically driven and includes an electric motor for driving it. The gimbal 112 may also be configured to change the posture of the camera 110 relative to the unmanned aerial vehicle 120 in at least the yawing direction DY and at least the pitching direction DP. Furthermore, the gimbal 112 may also be configured to change the posture of the camera 110 in the rolling direction DR. The orientation of the camera 110 may be the orientation of the optical axis center of the camera of the camera 110, specifically, the three-dimensional direction of the optical axis center in three-dimensional space, but is not limited thereto. For example, the orientation of the camera 110 may correspond to the posture of the camera 110. In addition, the position of the camera 110 may also be the three-dimensional position of the camera 110 in three-dimensional space.

[0032] The camera control device 113 controls the driving of the camera 111 and the gimbal 112. The camera control device 113 controls the driving of the camera 111 and the gimbal 112 based on information and commands received from the imaging control device 150 via wireless communication, causing the camera 111 and the gimbal 112 to operate in accordance with the information and commands. The camera control device 113 transmits operation information indicating the results of the operation of the camera 111 and the gimbal 112 to the imaging control device 150 via wireless communication. The camera control device 113 wirelessly communicates with the imaging control device 150 using the first communicator 124 of the unmanned aerial vehicle 120, described later. The camera control device 113 includes a computer and a circuit for controlling the power supplied to the camera 111 and the gimbal 112.

[0033] The unmanned aerial vehicle 120 is capable of flying in a manner that allows the position and orientation of the mounted camera 110 to be freely changed. Although not limited to this, in this exemplary embodiment, the unmanned aerial vehicle 120 is a drone equipped with multiple rotors 121, for example, four rotors 121 arranged in a plane and capable of rotating along the plane. The drone is capable of flying in any direction, including horizontal and vertical directions, stationary flight in the air, and movement in any orientation while stationary. Furthermore, the unmanned aerial vehicle 120 includes: a wing drive device 122 that rotationally drives each rotor 121; a position detection device 123; a first communicator 124; and an aircraft control device 125.

[0034] An example of the wing drive device 122 is an electric motor. The first communicator 124 performs wireless communication between the unmanned aerial vehicle 120 and the imaging control device 150. Specifically, the first communicator 124 relays wireless communication between the aircraft control device 125 and the imaging control device 150, and wireless communication between the camera control device 113 of the imaging device 110 and the imaging control device 150. For example, the first communicator 124 may include a communication circuit.

[0035] The position detection device 123 detects the position and posture of the UAV 120 and outputs the information to the aircraft control device 125. The structure of the position detection device 123 is not particularly limited, as long as it can detect the position and posture of the UAV 120. For example, a known technology may be used. For example, the position detection device 123 may be configured to include a position estimation system using the GPS (Global Positioning System), a position estimation system using radio waves other than GPS, a position estimation system using image recognition technology, a position estimation system using acceleration and angular velocity, another position estimation system, or a combination of two or more of the above position estimation systems, and use this position estimation system to detect the position and posture of the UAV 120.

[0036] An example of a position estimation system using GPS is a system that determines the position and posture of the UAV 120 by transmitting and receiving information with the GPS. An example of a position estimation system using radio waves other than GPS is a system that estimates the position and posture of the UAV 120 by receiving radio waves from a radio transmitter located in the space where the robot 200 is located. An example of a position estimation system using image recognition technology is a system that uses a three-dimensional camera to detect surrounding objects and their positions through image recognition to estimate the position and posture of the UAV 120. An example of a position estimation system using acceleration and angular velocity is a system that estimates the position and posture of the UAV 120 using the detection results of an acceleration sensor and an angular velocity sensor.

[0037] The aircraft control device 125 controls the driving of each wing drive device 122. Based on information and instructions received from the imaging control device 150 via wireless communication via the first communicator 124, the aircraft control device 125 controls the driving of each wing drive device 122, causing the unmanned aerial vehicle 120 to fly in accordance with this information and instructions. Furthermore, the aircraft control device 125 transmits information on the position and posture of the unmanned aerial vehicle 120 detected by the position detection device 123 to the imaging control device 150 via the first communicator 124. The aircraft control device 125 includes a computer and a circuit for controlling the power supplied to each wing drive device 122.

[0038] Furthermore, the UAV 120 only needs to have a structure that allows it to fly while flexibly changing at least its position and posture. For example, the UAV 120 may have a structure similar to a helicopter with two rotating wings that rotate in intersecting directions. The UAV 120 performs flight operations under the control of the imaging control device 150. For example, the UAV 120 can fly according to operations input to the imaging input device 130. The UAV 120 can also fly automatically, i.e., autonomously, according to the program control of the imaging control device 150.

[0039] The imaging input device 130 receives inputs and operations for operating the imaging system 100 from the user P. The imaging input device 130 receives inputs of various instructions, information, data, etc., and outputs them to the imaging control device 150. For example, the imaging input device 130 receives inputs for operating the camera 111, the gimbal 112, and the unmanned aerial vehicle 120, and outputs operation information indicating the content of the accepted operation to the imaging control device 150. The imaging input device 130 is arranged near the robot operating device 300 and can have the same structure as the structure illustrated in the robot operating device 300. The robot operating device 300 can also include the imaging input device 130 and have the function of the imaging input device 130. The imaging input device 130 is an example of an input device.

[0040] The display device 140 presents the image captured by the camera 110 to the user P in a perceptible manner. The display device 140 is arranged near the robot operating device 300 and at a position away from the camera 110. The display device 140 can also present the user P with instructions, information, and data received from the robot control device 400 in a perceptible manner. For example, the display device 140 has displays such as a liquid crystal display (Liquid Crystal Display) and an organic or inorganic EL display (Electro-Luminescence Display) for visual prompts. The display device 140 can also have a sound output device such as a speaker for auditory prompts. The display device 140 can also be configured to provide tactile prompts.

[0041] The shooting control device 150 is connected to the shooting device 110, the unmanned aerial vehicle 120, the shooting input device 130, and the display device 140 via wired communication or wireless communication. In addition, any wired communication and wireless communication can also be used. In this exemplary embodiment, the shooting control device 150 is configured outside the unmanned aerial vehicle 120, and performs wireless communication with the shooting device 110 and the unmanned aerial vehicle 120, and performs wired communication with the shooting input device 130 and the display device 140. However, the shooting control device 150 can also be mounted on the unmanned aerial vehicle 120 and perform wireless communication with the shooting input device 130 and the display device 140. The shooting control device 150 is an example of a control device.

[0042] The imaging control device 150 described above includes a computer. Furthermore, the imaging control device 150 may include circuits for controlling the power supplied to the imaging input device 130 and the display device 140. Devices other than the computer may be provided separately from the imaging control device 150. The imaging control device 150 may be incorporated into the robot control device 400, the imaging input device 130, or the robot operating device 300, or may be provided separately therefrom.

[0043] For example, the computers of the robot control device 400 and the imaging control device 150 include circuits or processing circuits with processors and memory. These circuits or processing circuits receive and send instructions, information, and data to and from other devices. They input signals from various devices and output control signals to various controlled objects. Memory consists of semiconductor memories such as volatile and non-volatile memories, as well as storage devices such as hard disks and SSDs. For example, memory stores programs and various data executed by the circuits or processing circuits.

[0044] The functions of the circuit or processing circuit can also be realized by a computer system composed of a processor such as a CPU (Central Processing Unit), a volatile memory such as a RAM (Random Access Memory), and a non-volatile memory such as a ROM (Read-Only Memory). The computer system can also realize the functions of the circuit or processing circuit by having the CPU use the RAM as a work area to execute the program recorded in the ROM. In addition, part or all of the functions of the circuit or processing circuit can be realized by the above-mentioned computer system, or by a dedicated hardware circuit such as an electronic circuit or an integrated circuit, or by a combination of the above-mentioned computer system and the hardware circuit. The robot control device 400 and the shooting control device 150 can execute each process through centralized control by a single computer, or can execute each process through distributed control by the cooperation of multiple computers.

[0045] For example, the various functions of the robot control device 400 and the shooting control device 150 can be implemented by a microcontroller, an MPU (Micro Processing Unit), an LSI (Large Scale Integration), a system LSI, a PLC (Programmable Gate Array), and a logic circuit. The multiple functions of the robot control device 400 and the shooting control device 150 can be individually single-chip, or they can be single-chip in a manner that includes some or all of them. In addition, each circuit can be a general-purpose circuit or a dedicated circuit. As an LSI, an FPGA (Field Programmable Gate Array) that is programmable after the LSI is manufactured, a configurable processor that can reconfigure the connections and / or settings of the circuit units inside the LSI, or an ASIC (Application Specific Integrated Circuit) that integrates multiple functional circuits into one for a specific purpose can also be used.

[0046] [Functional structure of the imaging system]

[0047] The functional structure of the imaging system 100 is described. Figure 3As shown, the imaging control device 150 of the imaging system 100 includes, as functional components, a drive control unit 151, a position and posture calculation unit 152, a display control unit 153, and a storage unit 154. Furthermore, the imaging control device 150 includes a second communicator 155. The drive control unit 151 includes, as functional components, a camera control unit 1511 and an aircraft control unit 1512. The position and posture calculation unit 152 includes, as functional components, a first detection unit 1521, a second detection unit 1522, and a determination unit 1523.

[0048] The second communicator 155 wirelessly communicates with the first communicator 124 of the UAV 120, and relays wireless communication between the imaging control device 150 and the UAV 120. For example, the second communicator 155 may include a communication circuit.

[0049] The function of the storage unit 154 is realized by a memory or the like of the computer of the imaging control device 150. The functions of the functional components of the imaging control device 150 other than the storage unit 154 are realized by a processor or the like of the computer.

[0050] The storage unit 154 stores various information and allows the stored information to be read. For example, the storage unit 154 can store programs and various data. For example, the storage unit 154 can store programs, data, and information for operating each device of the imaging system 100.

[0051] The first detection unit 1521 of the position and posture calculation unit 152 detects the position and posture of the end effector 230 of the robot 200 and outputs it to the determination unit 1523. Although not limited to this, the first detection unit 1521 detects the position and posture of the front end portion of the robot arm 210 as the position and posture of the end effector 230. The first detection unit 1521 is configured to receive feedback information of the joint drive device 212 of the robot arm 210 from the robot control device 400. The first detection unit 1521 uses the rotational drive amount of each joint drive device 212 included in the feedback information to calculate the position and posture of the connection portion between the robot arm 210 and the end effector 230, that is, the front end portion of the robot arm 210. For example, the position of the front end can be used Figure 1 The first coordinate system is defined by the coordinate axes X1, Y1, and Z1. The posture of the front end can be expressed using the following example: Figure 1The second coordinate system is defined by the coordinate axes X2, Y2, and Z2. The posture of the front end can be represented by the angles of the coordinate axes X2, Y2, and Z2 relative to the coordinate axes X1, Y1, and Z1 as posture angles. Here, the feedback information of the joint drive device 212 of the robot arm 210 and the information on the position and posture of the end effector 230 are examples of motion-related information. The above-mentioned feedback information is an example of information on the position and posture of the end effector 230.

[0052] Here, motion-related information refers to information related to the motion of the robot 200. The motion-related information may include: information used to cause the robot 200 to move; information indicating the results of the robot 200's motion; or both. Specifically, the motion-related information may include information indicating the motion of the robot 200. Examples of information used to cause the robot 200 to move include information about operations input to the robot manipulation device 300, etc., to cause the robot 200 to move; and instructions for target motions output from the robot control device 400 to the robot 200 to cause the robot 200 to move. Examples of information indicating the results of the robot 200's motion include feedback information output from the robot 200 and detection results of the robot 200's motion by sensors, etc. The motion of the robot 200 may include the motion of the robot arm 210, the motion of the joints of the robot arm 210, the motion of the joint drive device 212 of the robot arm 210, the position and posture of the end effector 230, the motion of the end effector 230, or a combination of two or more of these.

[0053] The first detection unit 1521 may also detect the position and posture of the end effector 230 using operation information input to the robot operating device 300 to move the robot arm 210, target action instructions output from the robot control device 400 to the robot arm 210 to move the robot arm 210, or a combination thereof.

[0054] Alternatively, the robot controller 400 may calculate the position and posture of the end effector 230 , and the first detection unit 1521 may receive the calculation results of the position and posture from the robot controller 400 .

[0055] Second detection unit 1522 detects the current position and posture of camera 111 of imaging device 110 and outputs this information to determination unit 1523. Second detection unit 1522 receives information on the current position and posture of unmanned aerial vehicle 120 and information on the movements of camera 111 and gimbal 112 of imaging device 110 from unmanned aerial vehicle 120 via wireless communication. Second detection unit 1522 uses this received information to detect the position and posture of camera 111. The position and posture of camera 111 can be expressed using a first coordinate system.

[0056] The determination unit 1523 determines the target position and posture of the camera 111 of the imaging device 110 in accordance with the motion-related information and outputs the information to the drive control unit 151. Specifically, the determination unit 1523 determines the target position and posture of the camera 111 so that the position and posture of the camera 111 change in accordance with changes in the motion-related information. For example, the determination unit 1523 determines the target position and posture of the camera 111 so that the position and posture of the camera 111 follow the motion of the robot 200 indicated by the motion-related information. Specifically, the determination unit 1523 determines the target position and posture of the camera 111 so that the position and posture of the end effector 230 and the position and posture of the camera 111 have a predetermined relationship, that is, a predetermined relative relationship.

[0057] As the predetermined relative relationship, the determination unit 1523 reads information indicating the relative position and posture of the camera 111 relative to the position and posture of the end effector 230 from the storage unit 154. Using this predetermined relative relationship information, the detection results of the end effector's 230 position and posture received from the first detection unit 1521, and the detection results of the camera 111's current position and posture received from the second detection unit 1522, the determination unit 1523 calculates a target position and posture of the camera 111 that satisfies the predetermined relative relationship based on the detection results of the end effector's position and posture. The target position and posture can be expressed using the first coordinate system. This allows the relative position and posture of the camera 111 to be maintained relative to the end effector 230.

[0058] Here, the determination unit 1523 determines the predetermined relative relationship in accordance with a command input by the user P to the imaging input device 130. For example, if the position and posture of the end effector 230 is in a first state and the position and posture of the camera 111 is in a second state, and the command is input to the imaging input device 130, the determination unit 1523 sets the relationship between the first state and the second state as the predetermined relative relationship, calculates information representing this relationship, such as a relational expression between the position and posture of the end effector 230 and the position and posture of the camera 111, and stores the information in the storage unit 154.

[0059] Furthermore, when user P inputs an operation to move the position and posture of camera 111 to imaging input device 130, decision unit 1523 receives operation information indicating the details of the operation from imaging input device 130 and determines the target position and posture of camera 111 to be moved in accordance with the operation information. The relationship between the operation information and the position and posture of camera 111 is pre-stored in storage unit 154, and decision unit 1523 uses this information to calculate the target position and posture. The operation information includes information about the operation performed by user P to visually confirm parts of robot 200, such as end effector 230, through camera 111 while robot 200 is in motion.

[0060] Furthermore, when determining unit 1523 receives camera 111 operation information from imaging input device 130 while executing control for determining the target position and orientation of camera 111 using a predetermined relative relationship, it may prioritize control for determining the target position and orientation of camera 111 based on the operation information. Specifically, determining unit 1523 receives camera 111 operation information from imaging input device 130 as a correction instruction to correct the position and orientation of camera 111, thereby changing the target position and orientation of camera 111 corresponding to the predetermined relative relationship to the target position and orientation of camera 111 based on the operation information. Thus, the target position and orientation of camera 111 determined using the predetermined relative relationship is corrected using the target position and orientation of camera 111 based on the operation information. In other words, user P can use imaging input device 130 to correct the target position and orientation of camera 111.

[0061] Furthermore, the determination unit 1523 may use the target position and posture of the camera 111 according to the operation information instead of the target position and posture of the camera 111 using the predetermined relative relationship, and determine the target position and posture of the camera 111 according to the operation information as the corrected target position and posture of the camera 111. The determination unit 1523 may also determine the target position and posture of the camera 111 according to the operation information as the corrected target position and posture of the camera 111 by adding the target position and posture of the camera 111 according to the operation information to the target position and posture of the camera 111 using the predetermined relative relationship.

[0062] After correcting the target position and orientation of camera 111, decision unit 1523 may reflect the correction results in the determination of the target position and orientation of camera 111 corresponding to the predetermined relative relationship. For example, decision unit 1523 may change the predetermined relative relationship to reflect the correction results. Alternatively, decision unit 1523 may determine the target position and posture of camera 111 without reflecting the correction results in the predetermined relative relationship after correcting the target position and orientation of camera 111.

[0063] The vehicle control unit 1512 of the drive control unit 151 generates an action command for the unmanned aerial vehicle 120 for moving the camera 111, based on the target position and posture of the camera 111 received from the determination unit 1523. The vehicle control unit 1512 outputs the action command to the camera control unit 1511 and the vehicle control device 125 of the unmanned aerial vehicle 120. The action command includes an instruction for the target position and posture of the unmanned aerial vehicle 120, and the target position and posture can be expressed using the first coordinate system. For example, the target position of the unmanned aerial vehicle 120 can be a position for moving the position of the camera 111 to the target position. The target posture of the unmanned aerial vehicle 120 can also be a posture for bringing the orientation of the camera 111 closer to the target posture. Since the unmanned aerial vehicle 120 is a drone, it can rotate horizontally and change the orientation of the camera 111 in the horizontal direction.

[0064] The camera control unit 1511 generates motion commands for the gimbal 112 to adjust the posture of the camera 111 based on the target posture of the camera 111 received from the determination unit 1523. These motion commands include motion commands for each component of the gimbal 112. The camera control unit 1511 transmits these motion commands to the camera control device 113 of the unmanned aerial vehicle 120. Based on the target posture information of the unmanned aerial vehicle 120 and the target posture information of the camera 111 received from the vehicle control unit 1512, the camera control unit 1511 determines the target posture of the camera 111 relative to the unmanned aerial vehicle 120 and determines the motions of each component of the gimbal 112 to achieve this target posture. For example, the camera control unit 1511 determines the target posture of the camera 111 relative to the mounting portion of the gimbal 112 on the unmanned aerial vehicle 120. In addition, when the camera 111 is mounted on a device that can move the position and posture of the camera 111, the camera control unit 1511 can also determine the target position and posture of the camera 111 based on the unmanned aerial vehicle 120 and output it to the camera control device 113.

[0065] The camera control unit 1511 generates an operation command for operating the camera 111 and transmits it to the camera control device 113 of the UAV 120. The operation command may include commands for starting and stopping the camera 111 from shooting, and commands for zooming in and out of the camera 111.

[0066] The display control unit 153 receives image data captured by the camera 110 from the camera 110, outputs the image data to the display device 140, and displays an image corresponding to the image data. The display control unit 153 may also perform image processing on the image data received from the camera 110 and output the processed image data to the display device 140.

[0067] [Camera system operation]

[0068] The operation of the imaging system 100 according to the exemplary embodiment will be described. Figure 4 This is a flowchart illustrating an example of the operation of the imaging system 100 according to an exemplary embodiment. First, in step S101, the imaging control device 150 operates in an initial setting mode that determines a predetermined relative relationship between the end effector 230 of the robot 200 and the camera 111 of the imaging device 110. For example, the imaging control device 150 begins the initial setting mode in response to a start command input by a user P to the imaging input device 130.

[0069] Next, in step S102, the imaging control device 150 receives a command to determine the initial position and initial posture of the camera 111 of the imaging device 110. Specifically, the imaging control device 150 activates the imaging device 110 and the unmanned aerial vehicle 120 and operates them according to an operation input to the imaging input device 130. The user P visually checks the image captured by the imaging device 110 on the display device 140 and operates the imaging input device 130 to cause the unmanned aerial vehicle 120 to change the position and posture of the camera 111 and the gimbal 112 of the imaging device 110 to change the posture of the camera 111. For example, when a desired image, such as a desired image of the end effector 230, is displayed on the display device 140, the user P inputs a command to the imaging input device 130 to determine the current position and posture of the camera 111 as the initial position and initial posture of the camera 111.

[0070] Next, in step S103, the imaging control device 150 determines the initial position and initial posture of the camera 111. During control execution, the imaging control device 150 receives information on the position and posture of the unmanned aerial vehicle 120 and the motion information of the camera 111 and the gimbal 112 from the unmanned aerial vehicle 120 and the imaging device 110. Based on the information on the position and posture of the unmanned aerial vehicle 120 and the motion information of the gimbal 112 detected when the instruction to determine the initial position and initial posture of the camera 111 was received in step S102, the imaging control device 150 detects the position and posture of the camera 111 and determines this position and posture as the initial position and initial posture.

[0071] Next, in step S104, the imaging control device 150 detects the position and posture of the end effector 230. Upon receiving the command to determine the initial position and initial posture of the camera 111 in step S102, the imaging control device 150 outputs a command to the robot control device 400 requesting motion information of the robot arm 210 as information on the position and posture of the end effector 230. The robot control device 400 outputs this motion information to the imaging control device 150. The motion information of the robot arm 210 is information on the rotation amount of each joint 211 detected by each joint drive device 212 of the robot arm 210. The imaging control device 150 uses this motion information to calculate the position and posture of the end effector 230.

[0072] Next, in step S105, the imaging control device 150 uses the information about the initial position and initial posture of the camera 111 and the information about the position and posture of the end effector 230 to determine a relational expression representing the predetermined relative relationship, that is, to determine the predetermined relative relationship. The imaging control device 150 stores the information about the predetermined relative relationship in the storage unit 154.

[0073] Next, in step S106 , the imaging control device 150 ends the initial setting mode and starts operating in the normal operation mode.

[0074] Next, in step S107 , the imaging control device 150 starts an imaging operation by the camera 111 of the imaging device 110 . The camera 111 continuously captures moving images and transmits them to the imaging control device 150 , which then displays the moving images on the display device 140 .

[0075] Next, in step S108 , the imaging control device 150 requests the robot control device 400 for the motion information of the robot arm 210 , and obtains the motion information from the robot control device 400 .

[0076] Next, in step S109, the imaging control device 150 determines a target position and target posture of the camera 111 that satisfies a predetermined relative relationship with the position and posture of the end effector 230. The imaging control device 150 calculates the position and posture of the end effector 230 using the motion information of the robot arm 210. The imaging control device 150 calculates the target position and target posture of the camera 111 using the position and posture of the end effector 230 and the information on the predetermined relative relationship stored in the storage unit 154.

[0077] Next, in step S110, the imaging control device 150 determines the target position and orientation of the UAV 120 and the target operational state of each component of the gimbal 112, for moving the camera 111 to the target position and orientation. The imaging control device 150 transmits an operational command to the UAV 120 and the imaging device 110 to move the UAV 120 to the target position and orientation, and an operational command to move the gimbal 112 to the target operational state.

[0078] Next, in step S111, the UAV 120 moves to the target position and target posture according to the motion command, and the camera 110 moves the gimbal 112 to the target motion state. As a result, the position and posture of the camera 111 and the position and posture of the end effector 230 satisfy a relative positional relationship.

[0079] Next, in step S112, the imaging control device 150 determines whether the user P has input a command to the imaging input device 130 to move the position and posture of the camera 111, i.e., a correction command. If the user P has input a command ("Yes" in step S112), the imaging control device 150 proceeds to step S113; if the user P has not input a command ("No" in step S112), the imaging control device 150 proceeds to step S117.

[0080] In step S113, the imaging control device 150 determines the target correction position and posture of the UAV 120, and the target correction operation states of each component of the gimbal 112, for moving the position and posture of the camera 111, in accordance with the correction command. The imaging control device 150 transmits the correction operation command, which is an operation command for moving the UAV 120 to the target correction position and posture, and the correction operation command, which is an operation command for moving the gimbal 112 to the target correction operation state, to the UAV 120 and the imaging device 110. At this time, the imaging control device 150 may also generate the operation command using information on the position and posture of the UAV 120 and the operation state of the gimbal 112 as feedback information.

[0081] Next, in step S114, the UAV 120 moves to the target correction position and posture according to the correction motion command, and the camera 110 moves the gimbal 112 to the target correction motion state. At this point, the UAV 120 and the camera 110 can also use information about the position and posture of the UAV 120 and the motion state of the gimbal 112 as feedback information to perform motion control.

[0082] Next, in step S115 , the imaging control device 150 requests the robot control device 400 for the motion information of the robot arm 210 , and obtains the motion information from the robot control device 400 .

[0083] Next, in step S116, the imaging control device 150 uses the motion information of the robot arm 210 to calculate the position and posture of the end effector 230. The imaging control device 150 uses the information on the position and posture of the end effector 230 and the corrected target position and corrected target posture of the camera 111 to determine a relational expression representing a new predetermined relative relationship. The imaging control device 150 updates the predetermined relative relationship information stored in the storage unit 154 with the new predetermined relative relationship information, thereby updating the predetermined relative relationship information. Consequently, the imaging control device 150 subsequently controls the position and posture of the camera 111 so that the position and posture of the camera 111 and the position and posture of the end effector 230 satisfy the updated relative positional relationship.

[0084] Then, in step S117, the shooting control device 150 determines whether the user P has input an instruction to end the action of the shooting system 100 to the shooting input device 130. If it has been input ("Yes" in step S117), the series of processing is ended; if it has not been input ("No" in step S117), the process returns to step S108.

[0085] Through the processing of steps S101 to S117, the imaging control device 150 determines the relative positional relationship desired by the user P. By controlling the operations of the unmanned aerial vehicle 120 and the gimbal 112, the imaging control device 150 controls the position and posture of the camera 111 so that the position and posture of the camera 111 and the position and posture of the end effector 230 satisfy the relative positional relationship. Furthermore, the imaging control device 150 corrects the relative positional relationship based on the operation input to the imaging input device 130. This allows the imaging control device 150 to determine a relative positional relationship that is more desirable for the user P.

[0086] Furthermore, in the exemplary embodiment, the imaging control device 150 is configured to correct the relative positional relationship in accordance with the correction instructions for the position and posture of the camera 111 when the imaging input device 130 is received during the operation control of the unmanned aerial vehicle 120 and the gimbal 112 based on the relative positional relationship. However, the present invention is not limited to this embodiment. For example, the imaging control device 150 may be configured to correct the position and posture of the camera 111 in accordance with the correction instructions but not to correct the relative positional relationship. In this case, the imaging control device 150 temporarily corrects the position and posture of the camera 111 in accordance with the correction instructions and then controls the position and posture of the camera 111 based on the existing relative positional relationship. Furthermore, the imaging control device 150 may also determine whether to correct the relative positional relationship based on instructions from the user P via the imaging input device 130.

[0087] (Variation 1)

[0088] Modification 1 of the exemplary embodiment differs from the exemplary embodiment in that the imaging control device 150A limits the flight area of the unmanned aerial vehicle 120. The following description of Modification 1 focuses on the differences from the exemplary embodiment, and descriptions of similarities with the exemplary embodiment are omitted as appropriate.

[0089] Figure 5 It is a perspective view showing an example of the configuration of a robot system 1 according to Modification 1 of the exemplary embodiment. Figure 6 1 is a block diagram showing an example of the functional structure of the imaging system 100 according to Modification 1. Figure 5 and Figure 6 As shown, the imaging control device 150A controls the movement of the UAV 120 so that the distance between the end effector 230 of the robot 200 and the UAV 120 is greater than the first predetermined distance L1. The imaging control device 150A controls the movement of the UAV 120 so that the distance between the end effector 230 of the robot 200 and the UAV 120 is less than the second predetermined distance L2. The second predetermined distance L2 is greater than the first predetermined distance L1.

[0090] Furthermore, when it is necessary to move the UAV 120 so that the distance between the end effector 230 and the UAV 120 is less than the first predetermined distance L1, the imaging control device 150A positions the UAV 120 at a position more than the first predetermined distance L1 away from the end effector 230, and causes the imaging device 110 to perform zoom-in photography. For example, the imaging control device 150A executes the above control when it receives an instruction or correction instruction from the imaging input device 130 that the distance between the end effector 230 and the UAV 120 is less than the first predetermined distance L1.

[0091] When it is necessary to move the UAV 120 so that the distance between the end effector 230 and the UAV 120 exceeds the second predetermined distance L2, the imaging control device 150A positions the UAV 120 at a distance less than the second predetermined distance L2 from the end effector 230, and causes the imaging device 110 to perform a zoom-out image. For example, the imaging control device 150A executes the above control when it receives an instruction or correction instruction from the imaging input device 130 that the distance between the end effector 230 and the UAV 120 exceeds the second predetermined distance L2.

[0092] Therefore, the imaging control device 150A controls the flight of the UAV 120 so that the position of the UAV 120 is maintained within an area A3 between an area A1 with a radius L1 centered on the end effector 230 and an area A2 with a radius L2 centered on the end effector 230. This can prevent the UAV 120 from approaching and interfering with the robot arm 210, and can also prevent the UAV 120 from interfering with objects such as the ceiling and walls surrounding the robot arm 210.

[0093] The position and posture calculation unit 152A of the imaging control device 150A includes a determination unit 1523 as a first determination unit and a second determination unit 1524. The first determination unit 1523 determines the target position and target posture of the camera 111 and outputs these to the second determination unit 1524. Hereinafter, the "target position" and "target posture" determined by the first determination unit 1523 will also be referred to as the "first target position" and "first target posture," respectively.

[0094] The second determination unit 1524 receives information on the position and posture of the end effector 230 from the first detection unit 1521. The second determination unit 1524 calculates the distance Lce between the first target position of the camera 111 and the position of the end effector 230, comparing the distance Lce with the first predetermined distance L1 and the distance Lce with the second predetermined distance L2. Information on the first predetermined distance L1 and the second predetermined distance L2 is pre-stored in the storage unit 154. In this variation, the reference point for the position of the end effector 230 is the origin of the second coordinate system formed by the X2, Y2, and Z2 axes. However, the reference point may be located at any position on the end effector 230.

[0095] When the distance Lce is greater than or equal to the first predetermined distance L1 and less than or equal to the second predetermined distance L2, the second decision unit 1524 executes the first decision. In the first decision, the second decision unit 1524 determines the first target position and the first target posture as the target position and the target posture of the camera 111 to be executed, respectively, and determines the second target position and the second target posture as the second target position and the second target posture.

[0096] If the distance Lce is less than the first predetermined distance L1, the second determination unit 1524 performs a second determination. In this second determination, the second determination unit 1524 determines a position on the line connecting the first target position of the camera 111 and the position of the end effector 230 that is at least the first predetermined distance L1 from the end effector 230 as the second target position of the camera 111. For example, the second determination unit 1524 determines a position at the first predetermined distance L1 as the second target position of the camera 111. Furthermore, the second determination unit 1524 determines the first target posture as the second target posture of the camera 111.

[0097] If the distance Lce exceeds the second predetermined distance L2, the second decision unit 1524 executes a third decision. In the third decision, the second decision unit 1524 determines a position on the line connecting the first target position of the camera 111 and the position of the end effector 230 that is less than the second predetermined distance L2 from the end effector 230 as the second target position of the camera 111. For example, the second decision unit 1524 determines a position at the second predetermined distance L2 as the second target position of the camera 111. Furthermore, the second decision unit 1524 determines the first target posture as the second target posture of the camera 111.

[0098] The second decision unit 1524 outputs information including the executed decision among the first to third decisions, the first target position and the first target posture, the second target position and the second target posture, and the distance Lce to the drive control unit 151 .

[0099] The vehicle control unit 1512 of the drive control unit 151 generates an operation command for the unmanned aerial vehicle 120 for moving the camera 111 according to the second target position and second target posture of the camera 111 received from the second determination unit 1524. The vehicle control unit 1512 outputs the operation command to the camera control unit 1511 and the vehicle control device 125 of the unmanned aerial vehicle 120.

[0100] The camera control unit 1511 generates an operation command for the gimbal 112 for moving the posture of the camera 111 according to the second target posture of the camera 111 received from the second determination unit 1524. Furthermore, the camera control unit 1511 determines the zoom rate or the zoom rate of the camera 111 using the first and second target positions of the camera 111 received from the second determination unit 1524.

[0101] Specifically, when receiving the first decision, the camera control unit 1511 decides not to change the zoom rate and the zoom rate in the current state.

[0102] Upon receiving the second decision, the camera control unit 1511 determines zoom-in shooting by the camera 111. The camera control unit 1511 calculates the distance LT12 between the first target position and the second target position. The camera control unit 1511 uses the distance LT12 and the distance Lce between the first target position and the end effector 230 to determine the zoom-in rate of the camera 111. The zoom-in rate is determined so that the size of the image of the end effector 230 displayed in an image captured by the camera 111 at the second target position using the zoom-in rate is equal to the size of the image of the end effector 230 displayed in an image captured by the camera 111 at the first target position without the zoom-in rate.

[0103] Upon receiving the third decision, the camera control unit 1511 determines that the camera 111 should zoom out. The camera control unit 1511 calculates the distance LT12 between the first target position and the second target position. Using the distance LT12 and the distance Lce between the first target position and the end effector 230, the camera control unit 1511 determines the zoom rate for the camera 111. This zoom rate is determined so that the size of the image of the end effector 230 displayed in an image captured by the camera 111 at the second target position using the zoom rate is equal to the size of the image of the end effector 230 displayed in an image captured by the camera 111 at the first target position without the zoom rate.

[0104] The camera control unit 1511 transmits an operation command for the camera 111 and an operation command for the gimbal 112 reflecting the determination results of the zoom rate and the zoom rate of the camera 111 to the camera control device 113 of the unmanned aerial vehicle 120 .

[0105] As described above, even when the UAV 120 is located at the second target position, the camera 111 can capture the same image as when the UAV 120 is located at the first target position and display it on the display device 140 .

[0106] In this modification, the imaging control device 150A is configured to use the first predetermined distance L1 and the second predetermined distance L2. However, the present invention is not limited thereto and may be configured to use only one of the first predetermined distance L1 and the second predetermined distance L2.

[0107] For example, when only the first predetermined distance L1 is used, the second determining unit 1524 executes the first determination if the distance Lce between the first target position of the camera 111 and the position of the end effector 230 is greater than or equal to the first predetermined distance L1. Furthermore, the second determining unit 1524 executes the second determination if the distance Lce is less than the first predetermined distance L1. The second determining unit 1524 does not execute the third determination.

[0108] For example, when only the second predetermined distance L2 is used, the second decision unit 1524 executes the first decision if the distance Lce is less than the second predetermined distance L2. Furthermore, the second decision unit 1524 executes the third decision if the distance Lce exceeds the second predetermined distance L2. The second decision unit 1524 does not execute the second decision.

[0109] The second determining unit 1524 may be configured to increase or decrease the first predetermined distance L1, the second predetermined distance L2, or both the first predetermined distance L1 and the second predetermined distance L2, based on the moving speed of the end effector 230. Specifically, the second determining unit 1524 may increase or decrease only the first predetermined distance L1 or only the second predetermined distance L2. The second determining unit 1524 may also increase or decrease both the first predetermined distance L1 and the second predetermined distance L2. In this case, the second determining unit 1524 may increase both the first predetermined distance L1 and the second predetermined distance L2, decrease both the first predetermined distance L1 and the second predetermined distance L2, or increase one of the first predetermined distance L1 and decrease the other. The second determining unit 1524 may also detect the moving speed of the end effector 230 based on information on the position and posture of the end effector 230 detected by the first detecting unit 1521. The first detection unit 1521 may detect the moving speed of the end effector 230 .

[0110] For example, if the moving speed of the end effector 230 increases, the second determining unit 1524 may increase the first predetermined distance L1 and decrease the second predetermined distance L2. This can more reliably prevent the UAV 120 from contacting the robot 200 and surrounding objects.

[0111] For example, if the moving speed of the end effector 230 decreases, the second determining unit 1524 may change the first predetermined distance L1 to decrease and the second predetermined distance L2 to increase. This allows the UAV 120 to approach the end effector 230 .

[0112] For example, the second determining unit 1524 may increase or decrease the first predetermined distance L1 and the second predetermined distance L2 in a manner that follows changes in the moving speed of the end effector 230. Alternatively, the second determining unit 1524 may determine whether to increase or decrease the first predetermined distance L1 and the second predetermined distance L2 based on a comparison result between the moving speed of the end effector 230 and a threshold value. In this case, the second determining unit 1524 may compare the moving speed of the end effector 230 with multiple threshold values and increase or decrease the first predetermined distance L1 and the second predetermined distance L2 in a stepwise manner.

[0113] In addition, in this modification, the imaging control device 150A limits the flight area of the UAV 120 based on the relationship between the target position of the camera 111 and the position of the end effector 230 , but the present invention is not limited to this.

[0114] For example, Figure 7 FIG is a perspective view showing another example of the flight restriction area of the unmanned aerial vehicle 120 in Modification 1. Figure 7As shown, the imaging control device 150A may also limit the flight area based on the relationship between the movable range A4 of the robot 200 and the target position of the camera 111 so that the UAV 120 does not intrude into the movable range A4. The movable range A4 is the range within which the robot arm 210 of the robot 200 can move. If the target position of the camera 111 is within the movable range A4, the imaging control device 150A may perform the same control as when the distance between the end effector 230 and the UAV 120 is less than the first predetermined distance L1.

[0115] The imaging control device 150A can also limit the flight area of the robot 200 based on the relationship between the robot 200's operating range A5 and the camera 111's target position so that the UAV 120 does not intrude into the operating range A5. The operating range A5 is the range within which the robot arm 210 can move when the robot 200 performs a predetermined task. If the camera 111's target position is within the operating range A5, the imaging control device 150A can perform the same control as when the distance between the end effector 230 and the UAV 120 is less than the first predetermined distance L1.

[0116] The imaging control device 150A may also limit the flight range based on the relationship between the target positions of the robot arm 210 and the camera 111 so that the UAV 120 does not intrude into the arm area A6, which is an area less than the third predetermined distance L3 from the robot arm 210. The imaging control device 150A may use feedback information from the joint drive device 212 of the robot arm 210 to detect the positions of the joints and other parts of the robot arm 210. Furthermore, the imaging control device 150A may also detect the position of the arm area A6 based on the detected position.

[0117] The arm region A6 is a region surrounding the robot arm 210 and changes in accordance with the movement of the robot arm 210. In this example, the arm region A6 is a region 100 meters from each joint 211 of the robot arm 210 (see Figure 1 ), but is not limited to this. For example, it may be an area based on any part of the robot arm 210, such as the surface of the robot arm 210. Furthermore, in this example, the arm area A6 is an area covering the entire robot arm 210, but is not limited to this. For example, it may be an area covering at least a part of the robot arm 210, such as a portion near the end effector 230. When the target position of the camera 111 is within the arm area A6, the imaging control device 150A may perform the same control as when the distance between the end effector 230 and the unmanned aerial vehicle 120 is less than the first predetermined distance L1.

[0118] In addition, the shooting control device 150A can be configured to respectively execute: control of limiting the flight area of the unmanned aerial vehicle 120 using the first specified distance L1 and the second specified distance L2; control of limiting the flight area of the unmanned aerial vehicle 120 using the movable range A4; control of limiting the flight area of the unmanned aerial vehicle 120 using the operating range A5; control of limiting the flight area of the unmanned aerial vehicle 120 using the arm area A6, and can also be configured to execute a combination of two or more of these controls.

[0119] (Variation 2)

[0120] Modification 2 of the exemplary embodiment differs from the exemplary embodiment and Modification 1 in that the imaging system 100 includes a motion detection device 160 for detecting the movement of the head H of the user P, and the position and posture of the camera 111 are changed in accordance with the movement of the head H detected by the motion detection device 160. The following description of Modification 2 will focus on the differences from the exemplary embodiment and Modification 1, and the description of the same points as the exemplary embodiment and Modification 1 will be omitted as appropriate.

[0121] Figure 8 It is a perspective view showing an example of the configuration of a robot system 1 according to Modification 2 of the exemplary embodiment. Figure 9 1 is a block diagram showing an example of the functional structure of the imaging system 100 according to Modification 2. Figure 8 and Figure 9 As shown, the imaging system 100 includes a display device 140B as a display device and a motion detection device 160. In addition, the imaging control device 150B of the imaging system 100 further includes a detection control unit 156.

[0122] Display device 140B is a head-mounted display mounted on the head H of a user P. In this variation, the head-mounted display has a goggle-like shape, with the lens portion of the head-mounted display forming a display surface for displaying images. Display device 140B can move along with the head H of the user P, thereby changing the position and orientation of the displayed image to follow the movement of the head H of the user P.

[0123] In addition, the display device 140B can also be configured not to be installed on the head H of the user P. In this case, the position of the display surface of the display device 140B, the posture of the display surface, or both the position and posture of the display surface can be changed, or the display surface of the display device 140B can be fixed.

[0124] The motion detection device 160 is an example of a detection device that detects the movement of the head H of the user P. While not particularly limited, in this variation, the motion detection device 160 includes at least one infrared sensor 161 and at least one infrared beacon 162 worn on the head H. In this variation, multiple infrared sensors 161, specifically three infrared sensors 161, are positioned around the user P and facing the user P. The three infrared sensors 161 are positioned away from the head H of the user P. Multiple infrared beacons 162, specifically four infrared beacons 162, are positioned at different locations on the head H. The head includes the portion of the human body above the neck, and can include, for example, the face, top of the head, temples, and occipital region.

[0125] The infrared beacon 162 emits infrared light. The infrared beacon 162 can be a light emitting body that emits infrared light by itself, such as an infrared LED (Light Emitting Diode), or a reflector that reflects the infrared light irradiated thereon, or can be configured to include both a light emitting body and a reflector. The infrared sensor 161 can receive infrared light and detect the direction, intensity, and intensity distribution of the received infrared light. The infrared sensor 161 can also be configured to receive only infrared light, or can be configured to emit infrared light by itself and receive infrared light such as reflected light of the infrared light. In the latter case, the infrared sensor 161 can be an infrared camera. By using three infrared sensors 161 to detect infrared light from four infrared beacons 162, the position and posture of the head H can be detected with high precision. The position of the head H can also be a three-dimensional position such as a predetermined reference point of the head H in a three-dimensional space, but is not limited thereto. The posture of the head H can be the posture of a specified part, surface or axis such as the front part of the head H, the plane that crosses the head H, and the axis of the head H that passes through the top of the head from the chin. Specifically, it can be the three-dimensional orientation of the above-mentioned specified part, surface or axis in three-dimensional space.

[0126] Furthermore, contrary to the above, the infrared sensor 161 may be worn on the head H of the user P, and the infrared beacon 162 may be arranged at a position away from the head H of the user P. The positions and numbers of the infrared sensor 161 and the infrared beacon 162 are not particularly limited as long as they can detect the position, posture, or both of the head H.

[0127] The detection control unit 156 controls the driving of the three infrared sensors 161, processes the results of the three infrared sensors 161 detecting infrared light from the four infrared beacons 162, and detects the three-dimensional positions and postures of the four infrared beacons 162. Specifically, the detection control unit 156 detects the position and posture of the head H of the user P by detecting the three-dimensional positions and postures of the infrared beacons 162. The detection control unit 156 outputs information on the position and posture of the head H to the position and posture calculation unit 152.

[0128] Specifically, the three infrared sensors 161 each receive infrared light emitted from the four infrared beacons 162. The infrared light emitted from each infrared beacon 162 is associated with identification information such as an ID assigned to that infrared beacon 162. Therefore, each infrared sensor 161 can detect the direction, intensity, and intensity distribution of the infrared light emitted by each of the four infrared beacons 162. The detection control unit 156 uses the information on the three-dimensional position and posture of each infrared sensor 161 pre-stored in the storage unit 154, as well as the detection results of the infrared light emitted by each infrared sensor 161, to detect the three-dimensional position of the four infrared beacons 162. For example, the detection control unit 156 detects the three-dimensional position of the four infrared beacons 162 according to the first coordinate system. Furthermore, the detection control unit 156 uses the three-dimensional position information of the four infrared beacons 162 to detect the three-dimensional position and posture of the head H of the user P. For example, the detection control unit 156 expresses the posture using posture angles such as a roll angle, a pitch angle, and a yaw angle.

[0129] Similar to the command for moving the position and posture of the camera 111 input to the imaging input device 130 in the exemplary embodiment, the position and posture calculation unit 152 uses the information on the position and posture of the head H as a command for moving the position and posture of the camera 111, and determines the target position and posture of the camera 111. The position and posture calculation unit 152 determines the target position and posture of the camera 111 for moving the camera 111 according to the amount of change in the position and posture of the camera 111 corresponding to the amount of change in the position and posture of the head H.

[0130] For example, horizontal and vertical movements of the head H correspond to horizontal and vertical movements of the camera 111. Movements of the head H in the roll, pitch, and yaw directions correspond to movements of the camera 111 in the roll, pitch, and yaw directions, respectively.

[0131] The determination unit 1523 of the position and posture calculation unit 152 calculates the target position and target posture of the camera 111 based on the information on the position and posture of the head H. For example, the storage unit 154 stores the relationship between various parameters used to move various devices according to the movement of the head H of the user P. For example, the relationship between the amount of change in the position and posture of the head H and the amount of change in the position and posture of the camera 111, that is, the change relationship, is stored in the storage unit 154. The determination unit 1523 uses the information on the position and posture of the head H and the change relationship to determine the target position and target posture of the camera 111 for moving the camera 111 to follow the movement of the head H. Here, the information on the position and posture of the head H includes information on the position and posture of the head H that the user P moves in order to visually confirm parts of the robot 200, such as the end effector 230, through the camera 111 while the robot 200 is in motion.

[0132] When the determination unit 1523 receives information indicating a change in the position and posture of the user P's head H (hereinafter also referred to as "head motion information") from the detection control unit 156 while executing control for determining the target position and posture of the camera 111 using a predetermined relative relationship, the determination unit 1523 may prioritize control for determining the target position and posture of the camera 111 based on the head motion information. Specifically, the determination unit 1523 receives the head motion information as a correction instruction to correct the position and orientation of the camera 111, thereby changing the target position and orientation of the camera 111 corresponding to the predetermined relative relationship to the target position and orientation of the camera 111 based on the head motion information. Thus, the target position and posture of the camera 111 determined using the predetermined relative relationship are corrected using the target position and posture of the camera 111 that has changed based on the position and posture of the head H. In other words, the user P can correct the target position and posture of the camera 111 to the intended position and posture by moving his head H.

[0133] Furthermore, the determination unit 1523 may determine the target position and posture of the camera 111 according to the head movement information as the corrected target position and posture of the camera 111, instead of the target position and posture of the camera 111 using the predetermined relative relationship. The determination unit 1523 may also determine the target position and posture of the camera 111 according to the head movement information as the corrected target position and posture of the camera 111 by adding the target position and posture of the camera 111 according to the head movement information to the target position and posture of the camera 111 using the predetermined relative relationship.

[0134] After correcting the target position and orientation of the camera 111, the determination unit 1523 may reflect the correction results in the determination of the target position and orientation of the camera 111 corresponding to the predetermined relative relationship. Alternatively, after correcting the target position and orientation of the camera 111, the determination unit 1523 may determine the target position and posture of the camera 111 without reflecting the correction results in the predetermined relative relationship.

[0135] The operation of the imaging system 100 according to the second modification will be described. Figure 10 1 is a flowchart showing an example of the operation of the imaging system 100 according to Modification 2. Figure 10 As shown, first, in step S201, the shooting control device 150B operates in an initial setting mode that determines the specified relative relationship between the end effector 230 of the robot 200 and the camera 111 of the shooting device 110, and the initial positions and initial postures of the camera 111 and the head H of the user P.

[0136] Next, the processing of steps S202 and S203 is the same as that of steps S102 and S103 in the exemplary embodiment.

[0137] Next, in step S204, the imaging control device 150B determines the initial position and initial posture of the user P's head H. Specifically, when the user P's head H reaches the desired position and posture, the imaging control device 150B inputs a command to determine the initial position and initial posture of the head H to the imaging input device 130. The imaging control device 150B causes the three infrared sensors 161 of the motion detection device 160 to detect infrared light. The imaging control device 150B processes the detection results of each infrared sensor 161 to detect the position and posture of the head H. The imaging control device 150B determines the detected position and posture of the head H as the initial position and initial posture of the head H.

[0138] Next, the processes of steps S205 to S212 are the same as the processes of steps S104 to S111 in the exemplary embodiment.

[0139] Next, in step S213, the imaging control device 150B determines whether the user P has input a correction command, i.e., an instruction to correct the position and posture of the camera 111 in accordance with the movement of the head H, to the imaging input device 130. If the correction command has been input ("Yes" in step S213), the imaging control device 150B proceeds to step S214; if the correction command has not been input ("No" in step S213), the imaging control device 150B proceeds to step S222.

[0140] In step S214, the imaging control device 150B causes the three infrared sensors 161 to continuously detect infrared light from the infrared beacon 162 of the head H. The imaging control device 150B may start the process of step S214 at any time between after the process of step S207 and after the process of step S213.

[0141] Next, in step S215, the user P visually checks the display device 140B and moves their head H to a desired position and posture. For example, the user P moves their head H so that the end effector 230 displayed on the display device 140B is at the desired position and posture. The imaging control device 150B processes the detection results of the infrared sensors 161 to detect the position and posture of the user P's head H relative to the initial position and posture. The imaging control device 150B detects the position and posture of the head H at predetermined time intervals.

[0142] Next, in step S216, the imaging control device 150B determines the target position and target posture of the camera 111, which are used to move the position and posture of the camera 111 according to the detected position and posture of the head H, as the corrected target position and corrected target posture of the camera 111. The imaging control device 150B calculates the corrected target position and corrected target posture of the camera 111 based on the information on the detected position and posture of the head H and the change relationship stored in the storage unit 154.

[0143] Next, in step S217, the imaging control device 150B determines the corrected target position and corrected target posture of the UAV 120 and the corrected target operation states of each component of the gimbal 112, for moving the position and posture of the camera 111 to the corrected target position and corrected target posture of the camera 111. The imaging control device 150B transmits a correction operation command to the UAV 120 and the imaging device 110 for moving the UAV 120 to the corrected target position and corrected target posture, and a correction operation command for operating the gimbal 112 to the corrected target operation state.

[0144] Next, in step S218 , the UAV 120 moves to the correction target position and correction target posture according to the correction operation command, and the camera 110 moves the gimbal 112 to the correction target operation state according to the correction operation command.

[0145] Then, in step S219, the shooting control device 150B determines whether the user P has input an end correction instruction to the shooting input device 130. If the input has been made ("Yes" in step S219), the process proceeds to step S220; if the input has not been made ("No" in step S219), the process returns to step S214.

[0146] The processes of steps S220 and S221 are the same as the processes of steps S115 and S116 in the exemplary embodiment.

[0147] Then, in step S222, the shooting control device 150B determines whether the user P has input an instruction to end the action of the shooting system 100 to the shooting input device 130. If it has been input ("Yes" in step S222), the series of processing is ended; if it has not been input ("No" in step S222), the process returns to step S209.

[0148] Through the processing of steps S201 to S222, the imaging control device 150B corrects the relative positional relationship based on the position and posture of the camera 111 that follows the movement of the head H of the user P. Therefore, the user P can easily determine a relative positional relationship that is more desirable to the user P by moving the head H to a desired position and orientation.

[0149] Furthermore, in Modification 2, upon receiving a command to correct the position and posture of camera 111 while controlling the movements of UAV 120 and gimbal 112 based on a relative positional relationship, imaging control device 150B corrects the relative positional relationship to correspond to the position and posture of camera 111 moving in accordance with the movement of user P's head H. However, this is not limiting. For example, imaging control device 150B may be configured to correct the position and posture of camera 111 in accordance with the movement of head H without correcting the relative positional relationship. In this case, imaging control device 150B temporarily corrects the position and posture of camera 111 in accordance with the movement of head H and then controls the position and posture of camera 111 in accordance with the existing relative positional relationship. Furthermore, imaging control device 150B may determine whether or not to correct the relative positional relationship based on a command from user P via imaging input device 130.

[0150] In Modification 2, the imaging control device 150B may be configured to use both control for moving the position and posture of the camera 111 in accordance with an operation using the imaging input device 130 and control for causing the position and posture of the camera 111 to follow the movement of the head H of the user P. The configuration of the imaging system 100 according to Modification 2 may also be applied to Modification 1.

[0151] (Other embodiments)

[0152] While the exemplary embodiments and variations of the present disclosure have been described above, the present disclosure is not limited to the aforementioned exemplary embodiments and variations. In other words, various modifications and improvements are possible within the scope of the present disclosure. For example, various modifications applied to the exemplary embodiments and variations, as well as configurations constructed by combining components from different exemplary embodiments and variations, are also within the scope of the present disclosure.

[0153] For example, in the exemplary embodiment and its variations, the imaging control devices 150, 150A, and 150B are configured to control the movement of the UAV 120 and the gimbal 112 based on a relative positional relationship, but the present invention is not limited thereto. For example, the imaging control devices 150, 150A, and 150B may also be configured to control the movement of the UAV 120 and the gimbal 112 based on positional and posture commands input by the user P to the imaging input device 130, and / or to control the position and posture of the camera 111 in a manner that tracks the movement of the user P's head H, rather than using a relative positional relationship.

[0154] In Modification 2, the motion detection device 160 includes the infrared sensor 161 and the infrared beacon 162 to detect the motion of the head H of the user P. However, the present invention is not limited thereto and may include any structure capable of detecting the motion of the head H.

[0155] For example, the motion detection device 160 may include an acceleration sensor and an angular velocity sensor worn on the head H to detect the acceleration and angular velocity of the head H in the six axes. In this case, the imaging control device 150B may be configured to receive the detection results from the acceleration sensor and the angular velocity sensor via wired or wireless communication. The imaging control device 150B may use the acceleration and angular velocity detection results to detect the position and posture of the head H.

[0156] Alternatively, the motion detection device 160 may include a three-dimensional camera positioned away from the head H to capture a three-dimensional image of the head H. The pixel value of each pixel in the three-dimensional image represents the distance to the subject reflected in that pixel. In this case, the imaging control device 150B may detect the image and posture of the head H reflected in the three-dimensional image through image processing such as pattern matching using a template of the head H, and detect the position of the head H based on the pixel value of each pixel in the three-dimensional image. Furthermore, the motion detection device 160 may include multiple three-dimensional cameras positioned at different positions and orientations. The imaging control device 150B may generate a three-dimensional model of the head H by processing the three-dimensional images from each of the three-dimensional cameras. The imaging control device 150B may use the three-dimensional model of the head H to detect the position and posture of the head H.

[0157] Alternatively, the motion detection device 160 may include a magnetic field generating device and a magnetic sensor worn on the head H to detect the position and posture of the magnetic sensor. In this case, the imaging control device 150B may be configured to receive detection results from the magnetic sensor via wired or wireless communication. The imaging control device 150B may use the detection results of the position and posture of the magnetic sensor to detect the position and posture of the head H.

[0158] In addition, various embodiments of the technology disclosed herein are listed below. One embodiment of the present disclosure relates to a camera system comprising: an unmanned aerial vehicle; a camera mounted on the unmanned aerial vehicle for capturing images of a robot operating an object; a display device located at a distance from the unmanned aerial vehicle for displaying images captured by the camera to a user operating the robot; and a control device for controlling the operation of the camera and the unmanned aerial vehicle, the control device acquiring information related to the operation of the robot, i.e., operation-related information, and moving the unmanned aerial vehicle by changing the position and orientation of the camera in accordance with the operation-related information.

[0159] According to the above aspect, the camera system utilizes information related to robot operation movements for controlling the movement of the unmanned aerial vehicle. This simplifies the user's operations for operating the unmanned aerial vehicle and reduces the amount of work required. Consequently, the camera system facilitates the operation of the unmanned aerial vehicle equipped with the camera.

[0160] In a shooting system according to one embodiment of the present disclosure, the control device may obtain motion information representing the motion of the robot as the motion-related information, and move the unmanned aerial vehicle in such a manner as to change the position and orientation of the shooting device according to the motion information.

[0161] According to the above embodiment, the camera system uses the robot's motion information to control the movement of the UAV. For example, the camera system can control the movement of the UAV so that it follows the robot's movements. This simplifies the user's operation of the UAV and reduces the amount of work required.

[0162] The shooting system involved in one form of the present disclosure may also be: it also has an input device for receiving command input from the above-mentioned user, and the above-mentioned control device receives an instruction to correct the position and orientation of the above-mentioned shooting device, that is, a correction instruction, through the above-mentioned input device, so that the position and orientation of the above-mentioned shooting device corresponding to the above-mentioned action-related information are changed to the position and orientation of the above-mentioned shooting device in accordance with the above-mentioned correction instruction, thereby moving the above-mentioned unmanned aerial vehicle.

[0163] According to the above aspect, the camera system can correct the movement of the unmanned aerial vehicle, which changes the position and orientation of the camera device in accordance with the action-related information, according to a correction command received via an input device. The camera system may use the position and orientation of the camera device according to the correction command instead of the position and orientation of the camera device corresponding to the action-related information, or it may use a position and orientation of the camera device generated by adding the position and orientation of the camera device according to the correction command to the position and orientation of the camera device corresponding to the action-related information. In either case, the user can correct the position and orientation of the camera device as intended.

[0164] In a shooting system involved in one form of the present disclosure, it may also be that: after the position and orientation of the above-mentioned shooting device are changed according to the above-mentioned correction instruction, the above-mentioned control device will reflect the change result of the position and orientation of the above-mentioned shooting device according to the above-mentioned correction instruction in the position and orientation of the above-mentioned shooting device corresponding to the above-mentioned action-related information.

[0165] According to the above aspect, the camera system can reflect the corrections made according to the correction instructions in the control of the movement of the UAV after the corrections. Therefore, the camera system can move the UAV in a manner that reflects the user's intention and changes the position and orientation of the camera device in accordance with the action-related information.

[0166] The shooting system involved in one form of the present disclosure may also be: further provided with a detection device for detecting the movement of the user's head, the control device obtains information on the movement of the user's head from the detection device, and moves the unmanned aerial vehicle in a manner that changes the position and orientation of the shooting device in accordance with the movement of the user's head.

[0167] According to the above embodiment, the camera system uses information about the user's head movement to control the movement of the unmanned aerial vehicle. Instead of using the position and orientation of the camera corresponding to the action-related information, the camera system can use the position and orientation of the camera that follows the movement of the user's head. Alternatively, the camera system can use the position and orientation of the camera generated by adding the position and orientation of the camera that follows the movement of the user's head to the position and orientation of the camera corresponding to the action-related information. In either case, the user's operation of the unmanned aerial vehicle can be simplified and the amount of operation can be reduced. For example, by moving their head in the direction and amount they intend to observe, the user can move the camera in the corresponding direction and amount of movement to observe the intended image.

[0168] In a shooting system involved in one form of the present disclosure, it may also be that: the above-mentioned robot includes an end effector and a robot arm that moves the above-mentioned end effector, and the above-mentioned control device obtains information on the position and orientation of the above-mentioned end effector as the above-mentioned action-related information, so as to move the above-mentioned unmanned aerial vehicle in a manner that changes the position and orientation of the above-mentioned shooting device in accordance with the position and orientation of the above-mentioned end effector.

[0169] According to the above embodiment, the user can manipulate the robot to change the position and orientation of the end effector, thereby causing the unmanned aerial vehicle to change the position and orientation of the imaging device. This simplifies user operation of the unmanned aerial vehicle and reduces the amount of manipulation required. For example, while the imaging device is capturing an image of the end effector, the imaging device moves in response to changes in the end effector's position and orientation, allowing the imaging device to continuously capture the end effector and display it on the display device. For example, when capturing images from a fixed position, even if the moving end effector is obscured by the robot arm, the imaging device can still capture the end effector without interference from the robot arm.

[0170] In a shooting system according to one embodiment of the present disclosure, the control device may control the movement of the UAV based on information on the position and orientation of the end effector so that the distance between the end effector and the UAV becomes greater than a first predetermined distance.

[0171] According to the above aspect, the imaging system can control the movement of the UAV so that the UAV does not enter an area less than the first predetermined distance from the end effector.

[0172] In a shooting system involved in one form of the present disclosure, it may also be that: when the above-mentioned control device receives an instruction that the distance between the above-mentioned end effector and the above-mentioned unmanned aerial vehicle is less than the above-mentioned first specified distance, the above-mentioned unmanned aerial vehicle is located at a position more than the above-mentioned first specified distance away from the above-mentioned end effector, and the above-mentioned shooting device performs push-shot shooting.

[0173] According to the above-mentioned form, when the unmanned aerial vehicle needs to move to a destination position that is less than the first specified distance from the end effector, the shooting system can prevent the unmanned aerial vehicle from entering an area that is less than the first specified distance from the end effector, and enable the shooting device to capture the same image as when shooting at the destination position.

[0174] In the imaging system according to one aspect of the present disclosure, the control device may increase or decrease the first predetermined distance according to a moving speed of the end effector based on information on the position and orientation of the end effector.

[0175] According to the above embodiment, the UAV moves in a manner that follows the position and orientation of the end effector, so the UAV's movement speed increases and decreases in the same manner as the end effector's movement speed. The imaging system increases or decreases the first predetermined distance based on the end effector's movement speed, thereby more reliably preventing the UAV from contacting the end effector and the robot arm. For example, if the end effector's movement speed increases, the first predetermined distance can be increased, and if the end effector's movement speed decreases, the first predetermined distance can be decreased. When the robot uses the end effector to perform dense operations, the end effector's movement speed tends to decrease, allowing the UAV to approach the end effector.

[0176] In a shooting system according to one embodiment of the present disclosure, the control device may control the movement of the UAV based on information on the position and orientation of the end effector so that the distance between the end effector and the UAV becomes less than a second predetermined distance.

[0177] According to the above embodiment, the imaging system controls the movement of the UAV so that the UAV does not enter an area beyond the second predetermined distance from the end effector. Therefore, the imaging system can prevent the UAV from coming into contact with surrounding objects. The second predetermined distance is greater than the first predetermined distance.

[0178] In a shooting system involved in one form of the present disclosure, it may also be that: when the above-mentioned control device receives an instruction that the distance between the above-mentioned end effector and the above-mentioned unmanned aerial vehicle exceeds the above-mentioned second specified distance, the above-mentioned unmanned aerial vehicle is located at a position less than the above-mentioned second specified distance away from the above-mentioned end effector, and the above-mentioned shooting device performs a pull-up shot.

[0179] According to the above-mentioned form, when the unmanned aerial vehicle needs to move to a destination position that is more than the second specified distance from the end effector, the shooting system can prevent the unmanned aerial vehicle from entering an area that is more than the second specified distance from the end effector, and enable the shooting device to capture the same image as when shooting at the destination position.

[0180] In the imaging system according to one aspect of the present disclosure, the control device may increase or decrease the second predetermined distance according to a moving speed of the end effector based on information on the position and orientation of the end effector.

[0181] According to the above aspect, the imaging system increases or decreases the second predetermined distance based on the end effector's moving speed, thereby more reliably preventing the UAV from contacting surrounding objects. For example, if the end effector's moving speed increases, the second predetermined distance may be decreased, and if the end effector's moving speed decreases, the second predetermined distance may be increased.

[0182] A robot system according to one aspect of the present disclosure includes the imaging system according to one aspect of the present disclosure and the above-mentioned robot. According to the above-mentioned aspect, the same effects as those of the imaging system according to one aspect of the present disclosure can be obtained.

[0183] In addition, the numbers such as ordinal numbers and quantities used above are all illustrative for the purpose of specifically describing the technology of the present disclosure, and the present disclosure is not limited to the illustrative numbers. In addition, the connection relationships between the constituent elements are illustrative for the purpose of specifically describing the technology of the present disclosure, and the connection relationships that realize the functions of the present disclosure are not limited to these.

[0184] In order to enable the present disclosure to be implemented in various forms without departing from the spirit of its essential characteristics, the scope of the present disclosure is defined by the appended claims rather than the description. Therefore, the illustrative embodiments and modifications are illustrative and not restrictive. All modifications within the scope of the claims and their equivalents are intended to be encompassed by the claims.

[0185] Description of Reference Numerals

[0186] 1…Robot system; 100…Photography system; 110…Photography device; 120…UAV; 130…Photography input device (input device); 140, 140B…Display device; 150, 150A, 150B…Photography control device (control device); 160…Motion detection device (detection device); 161…Infrared sensor; 162…Infrared beacon; 200…Robot; 210…Robot arm; 230…End effector; H…Head; P…User.

Claims

1. A shooting system, characterized in that: have: Unmanned aerial vehicles; a photographing device, mounted on the unmanned aerial vehicle, for photographing the robot operating on the object; a display device, disposed at a position away from the unmanned aerial vehicle, for displaying the image captured by the camera to a user operating the robot; as well as A control device for controlling the actions of the photographing device and the unmanned aerial vehicle, The control device obtains information related to the movement of the robot, that is, movement-related information, and moves the unmanned aerial vehicle in such a manner as to change the position and orientation of the imaging device in accordance with the movement-related information. The robot includes an end effector and a robot arm for moving the end effector. The control device obtains information on the position and orientation of the end effector as the motion-related information, and moves the UAV by changing the position and orientation of the imaging device in accordance with the position and orientation of the end effector. The control device controls the movement of the UAV based on the information on the position and orientation of the end effector so that the distance between the end effector and the UAV becomes greater than a first predetermined distance. The control device increases or decreases the first predetermined distance according to a moving speed of the end effector based on information on the position and orientation of the end effector.

2. The shooting system according to claim 1, wherein: The control device acquires motion information indicating the motion of the robot as the motion-related information, and moves the unmanned aerial vehicle so as to change the position and orientation of the imaging device according to the motion information.

3. The shooting system according to claim 1 or 2, characterized in that: further comprising an input device for receiving a command input by the user, The control device receives an instruction to correct the position and orientation of the shooting device, i.e., a correction instruction, via the input device, so that the position and orientation of the shooting device corresponding to the action-related information are changed to the position and orientation of the shooting device in accordance with the correction instruction, thereby moving the unmanned aerial vehicle.

4. The shooting system according to claim 3, wherein: After the position and orientation of the imaging device are changed according to the correction instruction, the control device reflects the result of the change in the position and orientation of the imaging device according to the correction instruction in the position and orientation of the imaging device corresponding to the action-related information.

5. The shooting system according to claim 1 or 2, characterized in that: further comprising a detection device for detecting movement of the user's head, The control device obtains information on the movement of the user's head from the detection device, and moves the unmanned aerial vehicle so as to change the position and orientation of the imaging device in accordance with the movement of the user's head.

6. The shooting system according to claim 1, wherein: When receiving a command indicating that the distance between the end effector and the UAV is less than the first predetermined distance, the control device positions the UAV at a position more than the first predetermined distance away from the end effector and causes the camera to perform zoom photography.

7. The shooting system according to any one of claims 1 or 6, characterized in that: The control device controls the operation of the unmanned aerial vehicle based on the information on the position and orientation of the end effector so that the distance between the end effector and the unmanned aerial vehicle becomes equal to or smaller than a second predetermined distance.

8. The shooting system according to claim 7, wherein: When receiving a command that the distance between the end effector and the UAV exceeds the second predetermined distance, the control device positions the UAV at a distance less than the second predetermined distance from the end effector and causes the camera to perform a zoom shot.

9. The shooting system according to claim 7, wherein: The control device increases or decreases the second predetermined distance according to a moving speed of the end effector based on information on the position and orientation of the end effector.

10. A robot system, characterized in that: have: The imaging system according to any one of claims 1 to 9; and The robot.

Images