Information processing device, information processing system, control method, and program

Abstract

To provide an information processing device, an information processing system, a control method, and a program capable of photographing by an imaging movable apparatus while responding instantly to situations where a subject is moving freely, and capable of improving the degree of freedom in photographing.SOLUTION: A control unit 111 that controls an imaging movable apparatus 100 equipped with an imaging apparatus 130 is configured to acquire maneuvering instruction information for a moving subject 150 that requires maneuvering, control photographing by the imaging apparatus 130, drive the imaging movable apparatus 100, and control its position and attitude. The control unit 111 controls the position and attitude of the imaging movable apparatus 100 on the basis of the maneuvering instruction information for the moving subject 150.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing system, a control method, and a program, and more particularly to an information processing apparatus, an information processing system, a control method, and a program for controlling the movement and shooting of an imaging-capable mobile device.

Background Art

[0002] Conventionally, a method is known in which an imaging-capable mobile device such as an aircraft equipped with an imaging device automatically shoots based on a predetermined flight route (see, for example, Patent Document 1). Further, a method is also known in which such an imaging-capable mobile device captures an image of a freely moving subject and performs tracking shooting by analyzing the captured image (see, for example, Patent Document 2).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional methods described in Patent Document 1 and the like that shoot along a predetermined flight route, there is a problem that it is difficult for the imaging-capable mobile device to shoot the subject at a desired angle of view when the subject moves freely.

[0005] In addition, in the conventional method described in Patent Document 2 or the like for analyzing and tracking the video of a freely moving subject, since it takes time to analyze the video, there is a problem that the tracking of the subject by the imaging-capable moving device is delayed and it may not be possible to take a picture at a desired angle of view. On the other hand, in order to solve this problem, for example, if the subject moves in a certain determined way and the imaging-capable moving body takes pictures in accordance with the determined way of movement, there arises a problem that the degree of freedom of shooting decreases.

[0006] An object of the present invention is to provide an information processing apparatus, an information processing system, a control method, and a program that can perform shooting by an imaging-capable moving device while immediately corresponding to the situation of a freely moving subject and can improve the degree of freedom of shooting.

Means for Solving the Problems

[0007] In order to solve the above problems, an information processing apparatus according to claim 1 of the present invention is an information processing apparatus that controls an imaging-capable moving device equipped with an imaging unit, and includes a steering instruction information acquisition means for acquiring steering instruction information of a moving subject accompanied by steering, an imaging control means for controlling shooting by the imaging unit, and a position and attitude control means for driving the imaging-capable moving device and controlling the position and attitude of the imaging-capable moving device, and the position and attitude control means controls the position and attitude of the imaging-capable moving device based on the steering instruction information.

[0008] In order to solve the above problems, an information processing system according to claim 17 of the present invention includes the imaging-capable moving device and the moving subject described above.

Effects of the Invention

[0009] According to the present invention, shooting by an imaging-capable moving device can be performed while immediately corresponding to the situation of a freely moving subject, and the degree of freedom of shooting can be improved.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Embodiments for Carrying Out the Invention

[0011] In the present invention, a user shoots a desired video by changing the position and orientation of an imaging-capable mobile device and shooting conditions according to the steering instruction information of a moving subject involving steering. In the present invention, an example will be described in which the imaging-capable mobile device moves by flying in the air by a propeller or the like, but the moving method is not limited to this. For example, it may move on the ground by wheels or move in water by a screw.

[0012] Hereinafter, an imaging-capable mobile device 100 including a control unit 111 as an information processing device according to an embodiment of the present invention, and a moving subject 150 which is a subject thereof and is wirelessly communicably connected to the control unit 111 will be described in detail with reference to the accompanying drawings.

[0013] FIG. 1 is a block diagram showing hardware configuration examples of the imaging-capable mobile device 100 and the moving subject 150, respectively.

[0014] In FIG. 1, the imaging-capable mobile device 100 is composed of an imaging device 130 and a mobile device 110 on which the imaging device 130 is mounted, which are communicably connected to each other. Further, the moving subject 150 is a subject of a video shot by the imaging device 130 (imaging unit) and moves in response to a steering instruction.

[0015] First, a configuration example of the mobile device 110 included in the imaging-capable mobile device 100 will be described with reference to FIG. 1.

[0016] The mobile device 110 includes a control unit 111, a ROM 112, a RAM 113, a position and orientation detection unit 114, an image processing unit 115, a shooting scenario list recording unit 116, a movement mechanism unit 117, and a communication unit 118. These blocks are connected to each other via a bus so as to be communicable.

[0017] The control unit 111 (imaging control means) is, for example, a CPU. It reads out a control program for each block included in the mobile device 110 and each block described later included in the imaging device 130 from the ROM 112, expands it in the RAM 113, and executes it. Thereby, the control unit 111 controls the operations of each block included in the imaging-capable mobile device 100. Further, the control unit 111 (manipulation instruction information acquisition means / position information acquisition means) receives manipulation instruction information and position information from the moving subject 150 via the communication unit 118.

[0018] The ROM 112 is an electrically erasable and recordable non-volatile memory, and stores operation programs for each block included in the imaging-capable mobile device 100, as well as parameters and the like necessary for the operations of each block.

[0019] The RAM 113 is a rewritable volatile memory, and is used for expanding programs executed by the control unit 111 and the like, and temporarily storing data generated by the operations of each block included in the imaging-capable mobile device 100.

[0020] The position and orientation detection unit 114 is a sensor that detects various information necessary for the position and orientation control of the mobile device 110, and outputs the detected position and orientation information to the control unit 111. Examples of such sensors include a GPS sensor for detecting position, a gyro sensor for detecting angular velocity, an acceleration sensor for detecting speed changes, a magnetic sensor for detecting direction, a pressure sensor for detecting altitude, and an ultrasonic sensor for detecting the distance to surrounding objects such as subjects. The control unit 111 (position and orientation control means) performs calculations according to the information detected by the position and orientation detection unit 114, and controls the position and orientation of the mobile device 110.

[0021] The image processing unit 115 performs various image processes related to the position and orientation of the mobile device 110 by analyzing the video captured by the imaging device 130 and the information detected by the position and orientation detection unit 114. For example, the image processing unit 115 performs image processing to recognize a subject (for example, the moving subject 150) included in the video captured by the imaging device 130, and the control unit 111 controls the moving mechanism unit 117 to track the subject using the recognition result. Regarding the subject recognition method and its tracking method, known methods are used and detailed descriptions are omitted.

[0022] The shooting scenario list recording unit 116 (shooting scenario list acquisition / holding means) records (holds) a shooting scenario list received (acquired) from an external information processing device (not shown) via the communication unit 118. Note that at the time of initial setting, a shooting scenario list may be pre-recorded in the shooting scenario list recording unit 116.

[0023] The moving mechanism unit 117 is composed of a buoyancy generation mechanism such as a motor and a propeller. The control unit 111 controls the position and orientation of the mobile device 110 by driving the moving mechanism unit 117 according to the information detected by the position and orientation detection unit 114 and the recognition result of the subject by the image processing unit 115. As a result, it is possible to perform tracking shooting to control the angle of view so that a subject (the moving subject 150) is included in the field of view captured by the imaging device 130.

[0024] The communication unit 118 communicates with the moving subject 150 based on a communication method defined by a standard such as a wireless LAN.

[0025] Next, a configuration example of the imaging device 130 included in the movable imaging device 100 will be described with reference to FIG. 1.

[0026] The imaging device 130 includes an optical system 131, an imaging unit 132, an A / D conversion unit 133, an image processing unit 134, a video recording unit 135, and a display unit 136. These blocks are connected to each other via a bus so as to be communicable.

[0027] The optical system 131 is composed of a lens group including a zoom lens and a focus lens and a diaphragm mechanism, and forms an object image on the imaging surface of the imaging unit 132.

[0028] The imaging unit 132 is an imaging element such as a CCD or a CMOS sensor, photoelectrically converts the optical image formed on the imaging surface of the imaging unit 132 by the optical system 131, and outputs the obtained analog image signal to the A / D conversion unit 133.

[0029] The A / D conversion unit 133 converts the input analog image signal into digital image data and outputs it. The digital image data output from the A / D conversion unit 133 is temporarily stored in the RAM 113 of the mobile device 110.

[0030] The image processing unit 134 performs various image processes on the digital image data stored in the RAM 113. Specifically, the image processing unit 134 performs various image processes for developing, displaying, and recording digital image data, such as demosaicking processing, noise reduction processing, white balance correction processing, and gamma processing. This image processing also includes a process of generating video data from the digital image data stored in the RAM 113 in time series.

[0031] The video recording unit 135 records data including the video data generated by the image processing unit 134 on a built-in recording medium.

[0032] The display unit 136 includes a display device such as an LCD, and displays digital image data stored in the RAM 113 and video data recorded in the video recording unit 135 on the display device.

[0033] Next, a configuration example of the moving subject 150 will be described with reference to FIG. 1. In the present invention, the imaging-capable moving device 100 photographs the moving subject 150 as a subject. In the present invention, an automobile will be described as an example of the moving subject 150, but the moving subject 150 is not limited thereto. For example, the moving subject 150 may be an object such as a motorcycle, a ship, an airplane, or a train that a passenger operates to move. Note that the moving subject 150 may be a device that does not require a passenger to operate and that recognizes the surrounding situation and performs automatic control. Further, the moving subject 150 may be a device that is not operated by a passenger but is remotely controlled like a radio-controlled car. The method of automatic control and the method of control by remote operation use known control methods, and detailed description thereof will be omitted.

[0034] The moving subject 150 includes a control unit 151, a ROM 152, a RAM 153, a position detection unit 154, a steering mechanism unit 155, an operation mechanism unit 156, and a steering communication unit 157. These blocks are connected to each other via a bus so as to be communicable.

[0035] The control unit 151 is, for example, a CPU, reads a control program for each block included in the moving subject 150 from the ROM 152, expands it in the RAM 153, and executes it. Thereby, the control unit 151 controls the operations of the respective blocks included in the moving subject 150. Further, the control unit 151 transmits steering instruction information and position information to the moving device 110 via the steering communication unit 157. The steering instruction information and the position information will be described later.

[0036] The ROM 152 is an electrically erasable and recordable non-volatile memory, and stores operation programs for the respective blocks included in the moving subject 150, as well as parameters and the like necessary for the operations of the respective blocks.

[0037] The RAM 153 is a rewritable volatile memory and is used for, for example, the deployment of programs executed by the control unit 151 and the like, and the temporary storage of data generated, etc. in the operations of each block included in the moving subject 150.

[0038] The position detection unit 154 has a plurality of sensors for acquiring the three-dimensional space position, and outputs the information on the three-dimensional space position calculated from the information detected by each sensor to the control unit 151. Examples of the plurality of sensors include, for example, a GPS sensor that detects a two-dimensional plane position and a barometric pressure sensor that detects altitude. The position detection unit 154 calculates the three-dimensional position information of the moving subject 150 by using these sensors. Note that a GPS sensor may be used as the sensor for detecting altitude.

[0039] The steering mechanism unit 155 is a mechanism for a passenger to give a steering instruction to the moving subject 150. Specifically, the steering mechanism unit 155 is composed of movable mechanisms such as a steering wheel, an accelerator pedal, a brake pedal, a turn signal lever, a wiper lever, and a light switch. Each time the passenger moves and gives an instruction to the steering mechanism unit 155, steering instruction information is output to the operation mechanism unit 156.

[0040] The operation mechanism unit 156 drives various mechanisms (such as movable mechanisms of the front wheels, engine, headlights, etc., and notification mechanisms such as turn signal lights, brake lights, and horns) (not shown) in the moving subject 150 according to the operation instruction information output from the operation mechanism unit 155. Specifically, when the steering angle of the steering wheel is output as operation instruction information from the operation mechanism unit 155, the operation mechanism unit 156 transmits this operation instruction information to an actuator that controls the traveling direction of the moving subject 150, and changes the direction of the front wheels. Also, when the degree of depression of the accelerator pedal is output as operation instruction information from the operation mechanism unit 155, the operation mechanism unit 156 transmits this operation instruction information to an actuator that controls the acceleration of the moving subject 150, and increases the engine speed. Also, there may be cases where operations such as light switch operation, turn signal lever operation, depression of the brake pedal, and horn operation are output as operation instruction information from the operation mechanism unit 155. In this case, the operation mechanism unit 156 lights the headlights, turn signal lights, and brake lights according to each operation instruction information to perform notification by light, and also sounds the horn to perform notification by sound.

[0041] The operation communication unit 157 communicates with the communication unit 118 based on a communication method defined by a standard such as wireless LAN.

[0042] Next, regarding the respective operation processes on the side of the imaging-capable mobile device 100 and the moving subject 150 when the imaging-capable mobile device 100 captures an image of the moving subject 150, a detailed explanation will be given with reference to the flowcharts of FIGS. 2 and 3. FIG. 2 shows a flowchart of the operation process on the side of the imaging-capable mobile device 100, and FIG. 3 shows a flowchart of the operation process on the side of the moving subject 150. That is, the operation process in FIG. 2 is executed by the control unit 111 provided in the mobile device 110 expanding the program stored in the ROM 112 into the RAM 113. On the other hand, the operation process in FIG. 3 is executed by the operation control unit 151 provided in the moving subject 150 expanding the program stored in the ROM 152 into the RAM 153.

[0043] In FIG. 2, first in step S201, the control unit 111 receives a shooting scenario list from an information processing device (not shown) via the communication unit 118 and records it in the shooting scenario list recording unit 116. If the shooting scenario list is already recorded in the shooting scenario list recording unit 116 in advance, step S201 is skipped and the process proceeds to step S202.

[0044] Here, the shooting scenario list will be described with reference to FIG. 4. The shooting scenario list is a list in which a plurality of shooting scenarios including the shooting conditions of the imaging device 130 and the position and orientation conditions of the moving device 110 are described. The movable imaging device 100 performs shooting according to the shooting conditions and the position and orientation conditions described in this shooting scenario list.

[0045] In the example of FIG. 4, as shooting conditions, the main subject, the shooting start condition, and the shooting end condition are described in the shooting scenario. Also, as the position and orientation conditions, the relative position between the movable imaging device 100 and the moving subject 150 and the camera work, which are conditions related to the shooting angle of view, are described in the shooting scenario. The relative position is a three-dimensional relative position, and in addition to two-dimensional planar relative positions such as 3 m to the right or 5 m to the rear with respect to the position of the moving subject 150, a relative position in the height direction such as 30 m above the height of the moving subject 150 is described. In the example of FIG. 4, the two-dimensional planar relative position is described on the left side of the slash ( / ), and the relative position in the height direction is described on the right side. Note that the condition that the relative positions in the height direction are the same indicates that the heights of the moving subject 150 and the movable imaging device 100 are the same. Note that the shooting scenario shown in FIG. 4 is an example and is not limited thereto. For example, the main subject included in the shooting conditions may not be included in the shooting conditions.

[0046] Next, with reference to FIG. 4, the specific contents of the shooting conditions and the position and orientation conditions described in the shooting scenario will be described. In the example of the shooting scenario where the shooting scenario name in FIG. 4 is "Curve-in Side", when the right turn signal of the moving subject 150 blinks, the imaging device 130 starts shooting an angle of view including the main subject (here, the front wheel of the moving subject 150). The imaging-capable moving device 100 shoots while traveling parallel at a relative position at the same altitude as the moving subject 150 at a position 3 m to the right of the moving subject 150. After that, when the right turn signal of the moving subject 150 turns off, the imaging device 130 ends the shooting. The above is the content of the shooting scenario with the shooting scenario name "Curve-in Side", and the imaging-capable moving device 100 operates with the goal of shooting a video with such content according to the steering instruction information.

[0047] In step S202 of FIG. 2, the control unit 111 transmits a request for steering instruction information and position information to the moving subject 150 via the communication unit 118.

[0048] Here, the operation process of the moving subject 150 will be described with reference to the flowchart of FIG. 3.

[0049] In step S301 of FIG. 3, the steering control unit 151 determines whether it has received a request for steering instruction information and position information from the imaging-capable moving device 100. The request for steering instruction information and position information is transmitted from the imaging-capable moving device 100 in step S202 of FIG. 2. When the request for steering instruction information and position information is received (YES in step S301), the process proceeds to step S302, and the steering control unit 151 transmits the steering instruction information and the position information to the moving device 110 via the steering communication unit 157, and ends this process. On the other hand, when the request for steering instruction information and position information has not been received (NO in step S301), the process of step S301 is repeated until this request is received.

[0050] Returning to FIG. 2, in step S203, the control unit 111 determines whether it has received the operation instruction information and the position information from the moving subject 150. If the operation instruction information and the position information are received (YES in step S203), the process proceeds to the process of step S204. On the other hand, if the operation instruction information and the position information are not received (NO in step S203), the process of step S203 is repeated until these information are received.

[0051] In step S204, after the control unit 111 instructs the imaging device 130 to start shooting, the control unit 111 performs tracking movement of the moving subject 150. For this tracking movement, since a known tracking movement method is used according to the position and orientation information by the position and orientation detection unit 114 and the subject recognition based on the above video data by the image processing unit 115, detailed description is omitted.

[0052] In step S205, the control unit 111 determines whether any shooting start condition of the shooting scenario list recorded in the shooting scenario list recording unit 116 matches the operation instruction information received in step S202. If the operation instruction information of one shooting scenario in the shooting scenario list matches the shooting start condition (YES in step S205), the process proceeds to step S206. On the other hand, if the operation instruction information does not match any shooting start condition of any shooting scenario in the shooting scenario list (NO in step S205), the process returns to step S202. Thereafter, the re-acquisition of the operation instruction information is repeated until the shooting start condition of any one shooting scenario in the shooting scenario list matches the operation instruction information.

[0053] In step S206, the control unit 111 (shooting scenario selection means) selects the matching shooting scenario as the shooting scenario to be used for the subsequent shooting.

[0054] In step S207, the control unit 111 controls the position and orientation of the moving device 110 by controlling the moving mechanism unit 117 so that the position and orientation of the moving device 110 conforms to the position and orientation conditions of the shooting scenario selected in step S206. Specifically, the control unit 111 drives the moving mechanism unit 117 so as to achieve the relative position described in the selected shooting scenario according to the position information of the moving subject 150, the information detected by the position and orientation detection unit 114, and the subject tracking result.

[0055] In step S208, when the control unit 111 determines that the imaging-capable moving device 100 has reached a position and orientation that conforms to the position and orientation conditions described in the shooting scenario selected in step S206, the control unit 111 instructs the imaging device 130 to start video recording. As a result, in the imaging device 130, the image processing unit 134 generates video data from the digital image data sequentially stored in the RAM 113 in time series, and the video recording unit 135 starts recording the video data.

[0056] In step S209, the control unit 111 starts controlling the position and orientation of the imaging-capable moving device 100 according to the camera work described in the shooting scenario selected in step S206. Note that the shooting (video recording) by the imaging device 130 started in step S208 continues until the process proceeds to step S211 described later.

[0057] In step S210, the control unit 111 determines whether or not the shooting end condition described in the shooting scenario selected in step S206 matches the operation instruction information most recently received from the moving subject 150. Here, the operation instruction information most recently received by the control unit 111 matches the shooting start condition and does not match the shooting end condition, so it is necessary to newly acquire the operation instruction information from the moving subject 150 again.

[0058] When the operation instruction information does not match the shooting end condition (NO in step S210), the control unit 111 proceeds to step S213, and the control unit 111 provided in the moving device 110 transmits a request for operation instruction information to the moving subject 150 via the communication unit 118.

[0059] After that, in step S214, the control unit 111 determines whether it has received the operation instruction information from the moving subject 150. If it has received the operation instruction information (YES in step S214 of FIG. 2), it returns to step S210 of FIG. 2. On the other hand, if it has not received the operation instruction information (NO in step S214 of FIG. 2), it returns to step S213 of FIG. 2.

[0060] On the other hand, when the latest operation instruction information obtained by the re-acquisition matches the shooting end condition (YES in step S210), the process proceeds to step S211.

[0061] In step S211, the control unit 111 instructs the imaging device 130 to end video recording. As a result, in the imaging device 130, the generation of video data by the image processing unit 134 and the recording of video data by the video recording unit 135 are ended.

[0062] In step S212, the control unit 111 determines whether all the shooting scenarios described in the shooting scenario list have been shot. If only some of the shooting scenarios described in the shooting scenario list have been shot (NO in step S212), it returns to step S204 to shoot the unshot shooting scenarios. On the other hand, if all the shooting scenarios described in the shooting scenario list have been shot (YES in step S212), this process ends.

[0063] As described above, the respective operation processes on the side of the imaging-capable mobile device 100 and the side of the moving subject 150 when the imaging-capable mobile device 100 of the present invention shoots the moving subject 150 have been described.

[0064] Next, with reference to FIG. 5, the positional relationship between the imaging-capable mobile device 100 and the moving subject 150 when the imaging-capable mobile device 100 performs shooting according to the operation process of FIG. 2 will be specifically described.

[0065] The position L401 indicates the position of the moving subject 150 at the start point, and the position L411 indicates the position of the imaging-capable moving device 100 at the start point. In the example of FIG. 5, at the start point, the imaging-capable moving device 100 is at the position L401 at the same height on the left side of the moving subject 150 at the position L411. Also, in the example of FIG. 5, the imaging-capable moving device 100 moves as positions L401, L402, L403, L404 over time, and the moving subject 150 moves as positions L411, L412, L413, L414 over time. Incidentally, in this specific example, the moving subject 150 is blinking the right turn signal at positions L412 and L413.

[0066] In the example of FIG. 5, the imaging-capable moving device 100 at the position L401 is tracking and moving while maintaining the same height on the left side of the moving subject 150 at the position L411 where the operation has started (step S204). This tracking movement continues until the operation instruction information transmitted from the moving subject 150 to the imaging-capable moving device 100 matches the shooting start condition of any of the shooting scenario lists in the shooting scenario list.

[0067] Next, when the moving subject 150 blinks the right turn signal, in response to the request for the operation instruction information from the imaging-capable moving device 100 (step S202), the moving subject 150 transmits the operation instruction information "right turn signal blinking" to the imaging-capable moving device 100.

[0068] When the control unit 111 receives this operation instruction information "right turn signal blinking" via the communication unit 118 (YES in step S203), the control unit 111 selects a shooting scenario in which the received operation instruction information matches the shooting start condition (steps S205, S206). Specifically, the control unit 111 selects the shooting scenario "curve in side" in which the operation instruction information "right turn signal blinking" matches the shooting start condition from the shooting scenario list shown in FIG. 4.

[0069] Next, the imaging movable device 100 moves to the relative position of the position and orientation conditions described in the selected shooting scenario "Curve In Side" (step S207). Specifically, since the relative position of the position and orientation conditions of the shooting scenario "Curve In Side" is "3 m to the right at the same altitude", the imaging movable device 100 moves from position L401 to position L402. That is, the imaging movable device 100 at position L402 is located 3 m to the right of the moving subject 150 at the same altitude. During this period, the moving subject 150 moves to the position indicated by positions L411 to L412 while blinking the right turn signal.

[0070] When the imaging movable device 100 assumes a position and orientation that matches the position and orientation conditions described in the selected shooting scenario "Curve In Side", the imaging device 130 starts recording the video (step S208). Specifically, when the imaging movable device 100 moves to position L402 and the relative position with the moving subject 150 at position L412 becomes "3 m to the right at the same altitude", the imaging device 130 starts recording the video.

[0071] The imaging movable device 100 performs shooting according to the camera work described in the selected shooting scenario "Curve In Side" (step S209). The camera work described in the selected shooting scenario "Curve In Side" is "parallel running". Therefore, the imaging movable device 100 performs tracking shooting while moving from position L402 to position L403 so as to run parallel with the moving subject 150 that turns right and moves from position L412 to position L413. This tracking shooting continues in step S210 of FIG. 2 until the operation instruction information matches the shooting end condition described in the shooting scenario list.

[0072] When the moving subject 150 turns off the right turn signal at position L414, the control unit 111 of the imaging-capable mobile device 100 provided at position L404 receives, via the communication unit 118, the operation instruction information "right turn signal off" from the moving subject 150 (step S214). Since the operation instruction information received at this time matches the shooting end condition of the selected shooting scenario "on the curve-in side" (YES in step S210), the imaging-capable mobile device 100 ends the recording of the video (step S211 in FIG. 2).

[0073] As described above, as shown in the example of FIG. 5, the imaging-capable mobile device 100 can move to the position described in the shooting scenario before the moving subject 150 starts to turn right by acquiring the operation instruction information "right turn signal blinking". Therefore, the imaging-capable mobile device 100 can record the video of the moving subject 150 without delay from the start of the right turn instead of during the right turn. In this way, the shooting by the imaging-capable mobile device 100 can be performed while immediately corresponding to the situation of a freely moving subject (the moving subject 150), and the degree of freedom of shooting can be improved.

[0074] In step S205 of FIG. 2, there may be a case where the shooting start condition described in the already shot shooting scenario matches the operation instruction information. In that case, it may be controlled not to select the already shot shooting scenario as the shooting scenario to be shot from now on. Thereby, it is possible to prevent excessive shooting of only a specific shooting scenario, and it is possible to increase the probability of selecting an unshot shooting scenario in which the same shooting start condition is described. This control is performed, for example, by including shooting status information indicating whether each shooting scenario has been shot or not, and executing it according to this shooting status information.

[0075] In the above embodiments, the blinking and extinguishing information of the "right turn signal" is taken as an example to explain the steering instruction information, but the steering instruction information is not limited to this. For example, when the shooting scenario is the shooting scenario "back shot" shown in FIG. 4, the steering instruction information of "depress the brake pedal" becomes the shooting start condition. In this case, when the received steering instruction information matches the shooting start condition, the imaging-capable mobile device 100 moves to the same height behind the moving subject 150, and the lit brake lamp is photographed. Also, for example, when the shooting scenario is the shooting scenario "aerial view" shown in FIG. 4, the steering instruction information of "the depression degree of the accelerator pedal is stable" becomes the shooting start condition. In this case, when the received steering instruction information matches the shooting start condition, the imaging-capable mobile device 100 moves above the moving subject 150, and the surrounding scenery and the moving subject 150 are photographed in an aerial view composition. When shooting such an aerial view composition, it is more aesthetically pleasing to shoot the moving subject 150 in a composition including a landmark such as a scenic landscape or a symbolic building. Therefore, the control unit 111 (object information acquisition means) may acquire the position information of the landmark from an information processing device (not shown), and start shooting when the landmark is included within the angle of view of the imaging device 130. Specifically, in addition to "the depression degree of the accelerator pedal is stable", "a landmark exists around the moving subject 150" is added to the shooting start condition of the shooting scenario "aerial view" in FIG. 4. That is, in this case, not only "the depression degree of the accelerator pedal is stable" but also "there are a moving subject and a landmark within the angle of view" are set as the shooting start conditions for the shooting scenario "aerial view". In step S205 of FIG. 2, when the control unit 111 determines that the received steering instruction information and the acquired position information of the landmark match the shooting start conditions of the shooting scenario "aerial view", the control unit 111 starts shooting the shooting scenario "aerial view".

[0076] In the above embodiments, it has been described that shooting is started when the shooting start condition described in the shooting scenario matches the operation instruction information in step S205 of FIG. 2. However, the shooting start condition is not limited to this. For example, even if the shooting start condition described in the shooting scenario matches the operation instruction information, if the moving subject 150 is moving fast and the imaging-capable moving device 100 cannot catch up, shooting may not be started. That is, when the shooting start condition described in the shooting scenario matches the operation instruction information, information on the traveling speed may also be acquired as the operation instruction information from the moving subject 150, and shooting may be started when the traveling speed is equal to or lower than a predetermined speed.

[0077] In the above embodiments, an example of tracking shooting while maintaining the relative position described in the selected shooting scenario has been described. However, the method of tracking shooting is not limited to this. For example, if there is an obstacle and it is impossible to maintain the relative position, the maintenance of the relative position may be temporarily stopped, and after avoiding the obstacle, tracking shooting may be performed again to maintain the relative position.

[0078] The detection of the obstacle is detected by the imaging-capable moving device 100 based on the information detected by the position and orientation detection unit 114 provided in the imaging-capable moving device 100. Further, the imaging-capable moving device 100 may also detect by receiving the position information of the obstacle from an information processing device (not shown) via the communication unit 118. In the case of a moving obstacle, since the position information changes moment by moment, the position information of the obstacle may be received at any time, or the predicted position information predicting the position at a future time may be received for detection.

[0079] In the above embodiments, an example has been described in which the control unit 111 provided in the mobile device 110 selects a shooting scenario and gives instructions to start and end video recording according to the operation instruction information. However, the selection of the shooting scenario and the instructions to start and end video recording are not limited to this. For example, a server (not shown) may manage the mobile device 110, the imaging device 130, and the moving subject 150, and the server may select a shooting scenario according to the operation instruction information and give instructions to start and end video recording. Further, in the above embodiments, an example has been described in which the control unit 111 provided in the mobile device 110 controls the imaging device 130. However, this is not limiting. For example, the imaging device 130 may also be provided with a control unit separately from the control unit 111 provided in the mobile device 110, and may control the operation of each block of the imaging device 130.

[0080] (Other Embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in a computer of the system or device read and execute the program. Further, it can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0081] The invention is not limited to the above embodiments, and various changes and modifications are possible without departing from the spirit and scope of the invention. Therefore, the claims are appended to disclose the scope of the invention.

[0082] As described above, the preferred embodiments of the present invention have been described. However, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist thereof.

[0083] The disclosure of this embodiment includes the following configurations, methods, and programs. An information processing apparatus for controlling an imaging-capable mobile device equipped with an imaging unit, comprising: a control instruction information acquisition means for acquiring control instruction information of a moving subject accompanied by operation; an imaging control means for controlling imaging by the imaging unit; and a position and orientation control means for driving the imaging-capable mobile device and controlling the position and orientation of the imaging-capable mobile device, wherein the position and orientation control means controls the position and orientation of the imaging-capable mobile device based on the control instruction information. The information processing apparatus according to Configuration 1, further comprising: a shooting scenario list acquisition and holding means for acquiring and holding a list of a plurality of shooting scenarios in which conditions related to shooting by the imaging unit are described; and a shooting scenario selection means for selecting one of the plurality of shooting scenarios according to the control instruction information. The information processing apparatus according to Configuration 2, wherein each of the plurality of shooting scenarios describes a shooting start condition of the imaging unit and a position and orientation condition of the imaging-capable mobile device. The information processing apparatus according to Configuration 3, wherein the shooting scenario selection means selects a shooting scenario among the plurality of shooting scenarios whose shooting start condition matches the control instruction information acquired by the control instruction information acquisition means. The information processing apparatus according to Configuration 4, wherein each of the plurality of shooting scenarios describes shooting status information indicating whether it has been shot or not, and the shooting scenario selection means does not select a shooting scenario among the plurality of shooting scenarios whose shooting start condition matches the control instruction information acquired by the control instruction information acquisition means but whose shooting status information indicates that it has been shot. The information processing apparatus according to Configuration 3, further comprising an object information acquisition means for acquiring position information of an object around the moving subject, wherein the shooting scenario selection means selects a shooting scenario among the plurality of shooting scenarios whose shooting start condition matches the control instruction information acquired by the control instruction information acquisition means and the position information acquired by the object information acquisition means. (Configuration 7) The position and orientation condition includes the relative position between the imaging-capable moving device and the moving subject. The imaging unit starts recording an image when the position and orientation of the imaging-capable moving device controlled by the position and orientation control means matches the relative position described in the imaging scenario selected by the imaging scenario selection means, which is an information processing apparatus according to any one of Configurations 3 to 6. (Configuration 8) The position and orientation condition further includes the camera work of the imaging unit. The position and orientation control means controls the position and orientation of the imaging-capable moving device according to the camera work described in the imaging scenario selected by the imaging scenario selection means while an image is being recorded by the imaging unit, which is an information processing apparatus according to Configuration 7. (Configuration 9) Each of the plurality of imaging scenarios further describes the imaging end condition of the imaging unit. The imaging unit ends the recording of the image when the operation instruction information matches the imaging end condition described in the imaging scenario selected by the imaging scenario selection means, which is an information processing apparatus according to Configuration 7 or 8. (Configuration 10) Each of the plurality of imaging scenarios further describes the main subject to be imaged by the imaging unit. The position and orientation control means controls the position and orientation of the imaging-capable moving device so that the main subject described in the imaging scenario selected by the imaging scenario selection means is included in the angle of view of the imaging unit while an image is being recorded by the imaging unit, which is an information processing apparatus according to any one of Configurations 7 to 9. (Configuration 11) The information processing apparatus further includes position information acquisition means for acquiring the position information of the moving subject. Until one of the imaging start conditions of the plurality of imaging scenarios matches the operation instruction information, the operation instruction information acquisition means repeatedly re-acquires the operation instruction information, and the position and orientation control means performs tracking movement of the moving subject based on the position information, which is an information processing apparatus according to any one of Configurations 3 to 10. (Configuration 12) The imaging unit images the moving subject, which is an information processing apparatus according to any one of Configurations 1 to 11. (Configuration 13) The information processing apparatus according to any one of Configurations 1 to 12, wherein the operation instruction information is information related to at least one operation instruction for the position and operation of the moving subject. (Configuration 14) The information processing apparatus according to Configuration 13, wherein the operation instruction information is information related to at least one operation instruction for the speed, direction of the moving subject, operation of a movable mechanism included in the moving subject, and notification by sound or light of a notification mechanism included in the moving subject. (Configuration 15) The information processing apparatus according to any one of Configurations 1 to 14, wherein the information processing apparatus is included in the movable imaging device. (Configuration 16) The information processing apparatus according to any one of Configurations 1 to 14, wherein the information processing apparatus is included in the moving subject. (Configuration 17) The information processing apparatus according to any one of Configurations 1 to 14, wherein the information processing apparatus is included in a server that can communicate with each of the movable imaging device and the moving subject. (Configuration 18) An information processing system including a movable imaging device and a moving subject according to any one of Configurations 1 to 17. (Method 1) A control method for an information processing apparatus that controls a movable imaging device equipped with an imaging unit, the control method including an operation instruction information acquisition step of acquiring operation instruction information for a moving subject accompanied by an operation, an imaging control step of controlling imaging by the imaging unit, and a position and orientation control step of driving the movable imaging device and controlling the position and orientation of the movable imaging device, wherein the position and orientation control step controls the position and orientation of the movable imaging device based on the operation instruction information. (Program 1) A program for causing a computer to function as each means included in the information processing apparatus according to any one of Configurations 1 to 17.

Explanation of Reference Numerals

[0084] 100 Movable imaging device 110 Moving device 111 Control unit 112 ROM 113 RAM 114 Position and orientation detection unit 115 Image processing unit 116 Shooting scenario list recording unit 117 Moving mechanism unit 118 Communication unit 130 Imaging device 131 Optical system 132 Imaging unit 133 A / D conversion unit 134 Image processing unit 135 Video recording unit 136 Display unit 150 Moving subject with operation 151 Operation control unit 152 ROM 153 RAM 154 Position detection unit 155 Operation mechanism unit 156 Motion mechanism unit 157 Operation communication unit

Claims

1. An information processing apparatus for controlling an image-capturable mobile device equipped with an imaging unit, comprising: a steering instruction information acquisition means for acquiring steering instruction information of a moving subject accompanied by steering; an imaging control means for controlling imaging by the imaging unit; a position and orientation control means for driving the image-capturable mobile device and controlling the position and orientation of the image-capturable mobile device and wherein the position and orientation control means controls the position and orientation of the image-capturable mobile device based on the steering instruction information.

2. a shooting scenario list acquisition and holding means for acquiring and holding a list of a plurality of shooting scenarios in which conditions related to shooting by the imaging unit are described; The information processing apparatus according to claim 1, further comprising a shooting scenario selection means for selecting one of the plurality of shooting scenarios according to the steering instruction information.

3. The information processing apparatus according to claim 2, wherein each of the plurality of shooting scenarios describes a shooting start condition of the imaging unit and a position and orientation condition of the image-capturable mobile device.

4. The information processing apparatus according to claim 3, wherein the shooting scenario selection means selects a shooting scenario among the plurality of shooting scenarios whose shooting start condition matches the steering instruction information acquired by the steering instruction information acquisition means.

5. Each of the plurality of shooting scenarios describes shooting status information indicating whether it has been shot or not, The information processing apparatus according to claim 4, wherein the shooting scenario selection means does not select a shooting scenario among the plurality of shooting scenarios whose shooting start condition matches the steering instruction information acquired by the steering instruction information acquisition means but whose shooting status information indicates that it has been shot.

6. further comprising an object information acquisition means for acquiring position information of an object around the moving subject, The information processing apparatus according to claim 3, wherein the shooting scenario selection means selects a shooting scenario among the plurality of shooting scenarios whose shooting start condition matches the steering instruction information acquired by the steering instruction information acquisition means and the position information acquired by the object information acquisition means.

7. The position and orientation condition includes the relative position between the image-capturable mobile device and the moving subject. The imaging unit starts recording an image when the position and orientation of the image-capturable mobile device controlled by the position and orientation control means match the relative position described in the imaging scenario selected by the imaging scenario selection means. The information processing apparatus according to claim 3, characterized in that.

8. The position and orientation condition further includes the camera work of the imaging unit. The position and orientation control means controls the position and orientation of the image-capturable mobile device according to the camera work described in the imaging scenario selected by the imaging scenario selection means while an image is being recorded by the imaging unit. The information processing apparatus according to claim 7, characterized in that.

9. Each of the plurality of imaging scenarios further describes an imaging end condition of the imaging unit. The imaging unit ends the recording of the image when the operation instruction information matches the imaging end condition described in the imaging scenario selected by the imaging scenario selection means. The information processing apparatus according to claim 7, characterized in that.

10. Each of the plurality of imaging scenarios further describes a main subject to be imaged by the imaging unit. The position and orientation control means controls the position and orientation of the image-capturable mobile device so that the main subject described in the imaging scenario selected by the imaging scenario selection means is included in the angle of view of the imaging unit while an image is being recorded by the imaging unit. The information processing apparatus according to claim 7, characterized in that.

11. The information processing apparatus further includes position information acquisition means for acquiring position information of the moving subject. Until one of the imaging start conditions of the plurality of imaging scenarios matches the operation instruction information, the operation instruction information acquisition means repeatedly re-acquires the operation instruction information, and the position and orientation control means performs tracking movement of the moving subject based on the position information. The information processing apparatus according to claim 3, characterized in that.

12. The imaging unit images the moving subject. The information processing apparatus according to claim 1, characterized in that.

13. The operation instruction information is information regarding at least one operation instruction of the position and movement of the moving subject. The information processing apparatus according to claim 1, characterized in that.

14. The operation instruction information is information regarding at least one operation instruction of the speed of the moving subject, the direction, the operation of the movable mechanism of the moving subject, and the notification by sound or light of the notification mechanism of the moving subject. The information processing apparatus according to claim 13, characterized in that.

15. The information processing apparatus according to claim 1, characterized by being included in the image-capturable mobile device.

16. The information processing apparatus according to claim 1, characterized by being included in the moving subject.

17. The information processing apparatus according to claim 1, characterized by being included in a server capable of communicating with each of the image-capturable mobile device and the moving subject.

18. An information processing system comprising the image-capturable mobile device and the moving subject according to claim 1.

19. A control method for an information processing apparatus that controls an image-capturable mobile device equipped with an imaging unit, comprising: a steering instruction information acquisition step of acquiring steering instruction information for a moving subject involving steering; an imaging control step of controlling imaging by the imaging unit; a position and orientation control step of driving the image-capturable mobile device and controlling the position and orientation of the image-capturable mobile device and having wherein the position and orientation control step controls the position and orientation of the image-capturable mobile device based on the steering instruction information.

20. A program for causing a computer to function as each means of the information processing apparatus according to claim 1.

Images