Information processing system, information processing device, control method, and program

Abstract

To provide an information processing system, an information processing device, a control method, and a program capable of making rapid and accurate changes to the imaging conditions of an imaging apparatus installed on an imaging movable body in response to changes in the movement of a moving subject involving maneuvering.SOLUTION: An information processing system has a moving subject 150 and an imaging movable apparatus 100 equipped with an imaging apparatus 130 for photographing the moving subject 150. When controlling the imaging by the imaging apparatus 130, the information processing system controls imaging conditions at a time of imaging of the imaging apparatus 130 on the basis of the maneuvering instruction information acquired from the moving subject 150.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing apparatus, a control method, and a program, and more particularly to an information processing system, an information processing apparatus, a control method, and a program for controlling video acquisition as a material used for creating a moving picture from an imaging-capable mobile device.

Background Art

[0002] Conventionally, a technique for dynamically controlling the imaging conditions of an imaging device mounted on an imaging-capable mobile device such as a drone according to surrounding information is known. For example, a method has been proposed in which when a drone detects an object from an image captured by an imaging device and approaches the object, the frame rate of the imaging device mounted on the drone is increased (see, for example, Patent Document 1). In addition, a method has been proposed in which the exposure of an imaging device mounted on a drone is changed based on peripheral information such as surrounding map information (see, for example, Patent Document 2).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional method, the imaging conditions of the imaging device mounted on the drone are changed based on the analysis result of the captured image and the surrounding information. For this reason, when the movement of the moving object to be photographed changes rapidly or greatly, there is a problem that the timing of changing the imaging conditions is delayed, the changed imaging conditions are not what the user desires, and the video desired by the user cannot be acquired.

[0005] Therefore, an object of the present invention is to provide an information processing system, an information processing apparatus, a control method, and a program that can quickly and accurately change imaging conditions of an imaging apparatus mounted on an imaging-capable moving body according to changes in the movement of a moving subject accompanied by operation.

Means for Solving the Problems

[0006] In order to solve the above problems, an information processing system according to claim 1 of the present invention is an information processing system including a moving subject accompanied by operation and an imaging-capable moving device mounted with an imaging device for photographing the moving subject, the information processing system including: acquisition means for acquiring operation instruction information of the moving subject; imaging control means for controlling imaging by the imaging device, wherein the imaging control means controls imaging conditions at the time of photographing by the imaging device based on the operation instruction information.

[0007] In order to solve the above problems, an information processing apparatus according to claim 35 of the present invention is an information processing apparatus communicably connected to an imaging-capable moving device mounted with a moving subject and an imaging device for photographing the moving subject, the information processing apparatus including: acquisition means for acquiring operation instruction information of the moving subject from the moving subject; imaging control means for controlling imaging by the imaging device, wherein the imaging control means transmits imaging conditions at the time of photographing to the imaging-capable moving device based on the operation instruction information.

Effects of the Invention

[0008] According to the present invention, it is possible to quickly and accurately change imaging conditions of an imaging device mounted on an imaging-capable moving body according to changes in the movement of a moving subject accompanied by operation.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Embodiment for Carrying Out the Invention

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are given the same reference numerals, and duplicate explanations are omitted.

[0011] (Example 1) Hereinafter, Example 1 which is a preferred embodiment of the present invention will be described with reference to the drawings.

[0012] In Example 1, a desired moving image is captured by changing the position and orientation of the image-capturable moving device and the imaging conditions according to the operation instruction information of the moving subject with operation. Also, in Example 1, an example will be described in which the image-capturable moving device is a drone that moves by flying in the air by a propeller or the like, but the moving method is not limited to this. For example, the image-capturable moving device may move on the ground by wheels or move in water by a screw.

[0013] First, the information processing system according to Example 1 will be described in detail using the accompanying drawings.

[0014] The information processing system of this embodiment includes an image-capturable moving device 100 and a moving subject 150 with operation that is wirelessly communicably connected to the image-capturable moving device 100, which is the object to be photographed.

[0015] FIG. 1 is a block diagram showing an example of the hardware configuration of each of the image-capturable mobile device 100 and the moving subject 150.

[0016] In FIG. 1, the image-capturable mobile device 100 is composed of an imaging device 130 that is communicably connected via a bus and a mobile device 110 on which the imaging device 130 is mounted. The moving subject 150 is a subject photographed by the imaging device 130 and moves in response to a control instruction.

[0017] First, an example of the configuration of the mobile device 110 included in the image-capturable mobile device 100 will be described with reference to FIG. 1.

[0018] The mobile device 110 includes a control unit 111, a ROM 112, a RAM 113, a position and attitude detection unit 114, an image processing unit 115, a shooting scenario list recording unit 116, a moving mechanism unit 117, and a communication unit 118. These blocks are communicably connected to each other via a bus.

[0019] The control unit 111 is, for example, a CPU. It reads a control program for each block included in the mobile device 110 and each block to be described later included in the imaging device 130 from the ROM 112, expands it in the RAM 113, and executes it. Thereby, the control unit 111 controls the operations of each block included in the image-capturable mobile device 100. The control unit 111 also receives control instruction information and position information from the moving subject 150 via the communication unit 118. The control instruction information and position information will be described later.

[0020] The ROM 112 is an electrically erasable and recordable non-volatile memory, and stores operation programs for each block included in the image-capturable mobile device 100, as well as parameters and the like necessary for the operations of each block.

[0021] The RAM 113 is a rewritable volatile memory, and is used for expanding programs executed by the control unit 111 and temporarily storing data generated in the operations of each block included in the image-capturable mobile device 100.

[0022] The position and orientation detection unit 114 is a sensor that detects various information necessary for position and orientation control, and outputs the detected position and orientation information to the control unit 111. Examples of such sensors include a GPS sensor for detecting position, a gyro sensor for detecting angular velocity, an acceleration sensor for detecting speed changes, a magnetic sensor for detecting direction, a pressure sensor for detecting altitude, and an ultrasonic sensor for detecting the distance to surrounding objects such as subjects. The control unit 111 performs calculations according to the information detected by the position and orientation detection unit 114, and controls the position and orientation of the mobile device 110.

[0023] The image processing unit 115 performs various image processes related to the position and orientation of the mobile device 110 by analyzing the video captured by the imaging device 130 and the information necessary for position and orientation control acquired by the position and orientation detection unit 114. For example, the image processing unit 115 performs image processing to recognize a subject (e.g., the moving subject 150) included in the video captured by the imaging device 130, and the control unit 111 controls the moving mechanism unit 117 to track the subject using the recognition result. Regarding the subject recognition method and tracking method, known methods are used and detailed descriptions are omitted.

[0024] The shooting scenario list recording unit 116 records a shooting scenario list received from an external information processing device (not shown) via the communication unit 118. Note that at the time of initial setting, a shooting scenario list may be pre-recorded in the shooting scenario list recording unit 116.

[0025] The moving mechanism unit 117 is composed of a buoyancy generation mechanism such as a motor and a propeller. The control unit 111 controls the position and orientation of the mobile device 110 by driving the moving mechanism unit 117 according to the information detected by the position and orientation detection unit 114 and the recognition result of the subject by the image processing unit 115. As a result, it is possible to perform tracking shooting to control the shooting angle so that the moving subject 150 is included in the field of view captured by the imaging device 130.

[0026] The communication unit 118 communicates based on a communication method defined by a standard such as wireless LAN.

[0027] Next, a configuration example of the imaging device 130 included in the imaging-capable mobile device 100 will be described with reference to FIG. 1.

[0028] The imaging device 130 includes an imaging condition control unit 131, an optical system 132, an imaging unit 133, an A / D conversion unit 134, an image processing unit 135, a display unit 136, and a video recording unit 137. These blocks are connected to each other via a bus so as to be communicable.

[0029] The imaging condition control unit 131 determines the imaging conditions during imaging performed in the optical system 132 and the imaging unit 133. The imaging conditions are, for example, aperture value, shutter speed, frame rate, ISO sensitivity, and the like.

[0030] The optical system 132 is composed of a lens group including a zoom lens and a focus lens and a diaphragm mechanism, and forms an object image on the imaging surface of the imaging unit 133.

[0031] The imaging unit 133 is an imaging element such as a CCD or a CMOS sensor, photoelectrically converts the optical image formed on the imaging surface of the imaging unit 133 by the optical system 132, and outputs the obtained analog image signal to the A / D conversion unit 134.

[0032] The A / D conversion unit 134 converts the input analog image signal into digital image data and outputs it. The digital image data output from the A / D conversion unit 134 is temporarily stored in the RAM 113.

[0033] The image processing unit 135 performs various image processes on the image data stored in the RAM 113. Specifically, various image processes for developing, displaying, and recording digital image data, such as demosaicking processing, noise reduction processing, white balance correction processing, and gamma processing, are applied. This image processing also includes a process of generating video data from the image data stored in the RAM 113 in time series.

[0034] The video recording unit 137 records data including the video data generated by the image processing unit 135 on a built-in recording medium.

[0035] The display unit 136 includes a display device such as an LCD, and displays the images stored in the RAM 113 and the images recorded in the video recording unit 137 on the display device.

[0036] Next, a configuration example of the moving subject 150 will be described with reference to FIG. 1. In the present invention, the imaging-capable moving device 100 captures the moving subject 150 with the imaging device 130 and generates image data. In the present invention, an automobile will be described as an example of the moving subject 150, but the moving subject 150 is not limited thereto. For example, the moving subject 150 may be an object such as a motorcycle, a ship, an airplane, or a train that a passenger operates and moves. Note that the moving subject 150 may be a device that does not require a passenger to operate and that recognizes the surrounding situation and performs automatic control. Further, the moving subject 150 may be a device that is not operated by a passenger and that is remotely controlled like a radio-controlled car. The method of automatic control and the method of control by remote operation use known control methods, and thus detailed description thereof will be omitted.

[0037] The moving subject 150 includes a control unit 151, a ROM 152, a RAM 153, a position detection unit 154, a steering mechanism unit 155, an operation mechanism unit 156, and a steering communication unit 157. These blocks are connected to each other via a bus so as to be communicable.

[0038] The control unit 151 is, for example, a CPU, reads a control program for each block included in the moving subject 150 from the ROM 152, expands it in the RAM 153, and executes it. Thereby, the control unit 151 controls the operations of the respective blocks included in the moving subject 150. Further, the control unit 151 transmits steering instruction information and position information to the moving device 110 via the steering communication unit 157. The steering instruction information and the position information will be described later.

[0039] The ROM 152 is an electrically erasable and recordable nonvolatile memory, and stores operation programs for the respective blocks included in the moving subject 150, as well as parameters and the like necessary for the operations of the respective blocks.

[0040] The RAM 153 is a rewritable volatile memory and is used for, for example, the deployment of programs executed by the control unit 151 and the like, and the temporary storage of data generated, etc. in the operations of each block included in the moving object 150.

[0041] The position detection unit 154 has a plurality of sensors for detecting the three-dimensional space position, and outputs the information on the three-dimensional space position calculated from the information detected by each sensor to the control unit 151. Examples of the plurality of sensors include a GPS sensor for detecting the two-dimensional plane position and a barometric pressure sensor for detecting the altitude. The position detection unit 154 calculates the three-dimensional position information of the moving object 150 by using these sensors. Note that a GPS sensor may be used as the sensor for detecting the altitude.

[0042] The steering mechanism unit 155 is a mechanism for a passenger to give a steering instruction to the moving object 150. Specifically, the steering mechanism unit 155 is composed of movable mechanisms such as a steering wheel, an accelerator pedal, a brake pedal, a turn signal lever, a wiper lever, and a light switch. Each time the passenger moves and gives an instruction to the steering mechanism unit 155, steering instruction information is output to the motion mechanism unit 156.

[0043] The operation mechanism unit 156 drives various mechanisms (such as movable mechanisms like the front wheels, engine, headlights, etc., and notification mechanisms like the turn signal light, brake lamp, horn, etc.) (not shown) in the moving subject 150 according to the operation instruction information output from the operation mechanism unit 155. Specifically, the operation instruction information includes information indicating the position and movement of the moving subject 150. More specifically, the operation instruction information includes information indicating the speed, direction, operation of the movable mechanism, notification by sound or light, and light emission of the light source of the moving subject 150. For example, when the steering angle of the steering wheel is output as operation instruction information from the operation mechanism unit 155, the operation mechanism unit 156 transmits this operation instruction information to the actuator that controls the traveling direction of the moving subject 150, and changes the direction of the front wheels. Also, when the degree of depression of the accelerator pedal is output as operation instruction information from the operation mechanism unit 155, the operation mechanism unit 156 transmits this operation instruction information to the actuator that controls the acceleration of the moving subject 150, and increases the engine speed. Also, operations such as window opening / closing operation, light switch operation, turn signal lever operation, depression of the brake pedal, horn operation, etc. may be output as operation instruction information from the operation mechanism unit 155. In this case, the operation mechanism unit 156 performs opening / closing of the window (movable mechanism), lighting of the headlights, turn signal lights, brake lamps, etc., and generation of a horn sound according to each operation instruction information.

[0044] The operation communication unit 157 communicates based on a communication method defined by a standard such as wireless LAN.

[0045] Next, the video recording process of the moving subject 150 according to this embodiment will be described in detail with reference to the flowcharts of FIGS. 2 and 3. The flowchart of FIG. 2 shows the process on the side of the imaging-capable mobile device 100 in this process, and the flowchart of FIG. 3 shows the process on the side of the moving subject 150 in this process.

[0046] First, the processing on the imaging-capable mobile device 100 side shown in FIG. 2 will be described. This processing is executed by the control unit 111 provided in the mobile device 110 by reading the program stored in the ROM 112 into the RAM 113. Also, for the steps with an execution subject other than the control unit 111 in the following steps, actually the control unit 111 controls the execution subject.

[0047] First, in step S201, the control unit 111 (shooting scenario selection means) acquires (selects) a shooting scenario from the shooting scenario list recorded in the shooting scenario list recording unit 116. The shooting scenario list may receive the shooting scenario list from an information processing device (not shown) via the communication unit 118 and record it in the shooting scenario list recording unit 116, or may be recorded in the shooting scenario list recording unit 116 in advance.

[0048] Here, the shooting scenario list will be described. The shooting scenario list is a list as shown in FIG. 4 in which shooting scenarios including the shooting conditions and imaging conditions of the imaging device 130 are described. The imaging-capable mobile device 100 performs shooting according to the shooting conditions and imaging conditions described in this shooting scenario. As shown in FIG. 4, in the shooting scenario list, as the shooting conditions for each shooting scenario, the main subject, the shooting start condition and shooting end condition determined based on the operation instruction information are described, and as the imaging conditions, the frame rate, shutter speed, and exposure settings are described. For example, in the "Curve-in side" scenario shown in FIG. 4, when the operation instruction information of the blinking of the right turn signal is received from the moving subject 150, the shooting is started. Note that the main subject may not be included in the shooting conditions of each shooting scenario.

[0049] Returning to FIG. 2, in step S202, the control unit 111 periodically receives the operation instruction information transmitted from the mobile device 110. Then, when the control unit 111 determines from the received operation instruction information that it meets the shooting start condition in the shooting scenario acquired in step S201, it starts the recording of the video under the imaging conditions described in that shooting scenario by the imaging device 130.

[0050] In step S203, the control unit 111 (acquisition means) transmits a request for steering instruction information to the moving object 150 via the communication unit 118.

[0051] Here, the processing on the side of the moving object 150 shown in FIG. 3 will be described. This processing is executed by the steering control unit 151 provided in the moving object 150 by reading the program stored in the ROM 152 into the RAM 153.

[0052] First, in step S301, the steering control unit 151 determines whether it has received a request for steering instruction information transmitted from the moving device 110 in step S203. If it has received the request for steering instruction information (YES in step S301), it proceeds to step S302, where the steering control unit 151 transmits the steering instruction information to the moving device 110 via the steering communication unit 157 and ends this processing. On the other hand, if it has not received the request for steering instruction information (NO in step S301), it repeats the determination processing in step S301 until it receives it.

[0053] The processing on the side of the moving object 150 has been described above.

[0054] Returning to FIG. 2, in step S204, the control unit 111 provided in the moving device 110 determines whether it has received the steering instruction information transmitted from the moving object 150 in step S302. If it has received the steering instruction information (YES in step S204), it transmits the received steering instruction information to the imaging condition control unit 131 provided in the imaging device 130 and proceeds to step S205. On the other hand, if it has not received the steering instruction information (NO in step S204), it returns to step S203, and again, the control unit 111 transmits a request for steering instruction information to the moving object 150.

[0055] In step S205, the imaging condition control unit 131 (scene attention degree determination means) provided in the imaging device 130 determines the scene attention degree based on the steering instruction information transmitted from the control unit 111.

[0056] Here, the scene attention degree will be described with reference to FIG. 5. The scene attention degree indicates the degree of attention for each shooting scene of the video (hereinafter referred to as a series of shooting videos) that the imaging device 130 started recording in step S202.

[0057] FIG. 5(a) is a diagram showing the relationship between the change in the scene attention degree in the shooting video and an example of video editing.

[0058] As shown in FIG. 5(a), generally in a series of shooting videos, the shooting scene where a sharp curve is turned has a particularly high scene attention degree, while the shooting scene where deceleration driving is performed has a low scene attention degree. Therefore, at the time of video editing, the playback speed is changed according to such a scene attention degree. Specifically, in a scene with a high scene attention degree, the playback speed is decreased to slow motion, and conversely, in a scene with a low scene attention degree, the playback speed is increased to fast forward. By performing such video editing, a highly artistic video in which the shooting scenes with high attention degree are emphasized can be created.

[0059] FIG. 5(b) is a diagram showing a table of the relationship between the scene attention degree and the steering instruction information that the imaging condition control unit 131 provided in the imaging device 130 holds in advance.

[0060] As shown in FIG. 5(b), a high scene attention degree is set for steering instruction information such as opening and closing of the window, turning the steering wheel sharply, using the blinker, accelerating, and drifting. On the other hand, a low scene attention degree is set for steering instruction information such as no steering instruction information for a certain period of time and decelerating. For other steering instruction information, a medium scene attention degree is set.

[0061] The above is the description regarding the scene attention degree.

[0062] Returning to FIG. 2, in step S206, the imaging condition control unit 131 provided in the imaging device 130 executes an imaging condition change necessity determination process for determining whether to change the imaging conditions based on the scene attention degree determined in step S205.

[0063] Here, the determination process for whether to change the imaging conditions executed in step S206 will be described using the flowchart of FIG. 6.

[0064] First, in step S601, it is determined whether the scene attention level determined in step S205 is high or low, or whether the scene attention level is medium. If the scene attention level is high or low (YES in step S601), the process proceeds to step S602. On the other hand, if the scene attention level is medium (NO in step S601), the process proceeds to step S605, and it is assumed that the imaging conditions are not changed (the determination result is no), and this process ends.

[0065] Next, in step S602, it is determined whether the conditions for changing the imaging conditions are satisfied. For example, if the imaging conditions have already been changed and shooting is in progress, if the same scene (or slow-motion scene) has been shot a predetermined number of times, if shooting of another scene with high urgency such as during landmark shooting is in progress, etc., the imaging conditions are not changed. If the conditions for changing the imaging conditions are not satisfied (NO in step S602), the process proceeds to step S604, and it is assumed that the imaging conditions are not changed (the determination result is no), and this process ends. On the other hand, if the conditions for changing the imaging conditions are satisfied (YES in step S602), the process proceeds to step S603, and it is assumed that the imaging conditions are changed (the determination result is yes), and this process ends.

[0066] The detailed processing flow of step S206 has been described above.

[0067] Returning to FIG. 2, if the determination result in step S206 is no, the process proceeds to step S209, while if it is yes, the process proceeds to step S207.

[0068] In step S207, an imaging condition control unit 131 (imaging control means) provided in the imaging device 130 controls the imaging unit 133 to change the imaging conditions based on the result determined in step S206. Specifically, the frame rate at the time of shooting is changed according to the scene attention degree. That is, after starting shooting at the frame rate described in the shooting scenario, if it is determined that the scene attention degree is high, assuming that slow motion will be used during editing, control is performed to increase the frame rate at the time of shooting so that it does not look unnatural even in slow motion. On the other hand, if it is determined that the scene attention degree is low after starting shooting, assuming that fast forward will be used during editing, control is performed to decrease the frame rate at the time of shooting. In addition, the imaging condition control unit 131 notifies the video recording unit 137 provided in the imaging device 130 of the start of imaging condition change.

[0069] In step S208, the video recording unit 137 provided in the imaging device 130 attaches an editing flag to the acquired frame data in accordance with the notification of the start of imaging condition change from the imaging condition control unit 131 in step S206. Here, when the changed imaging condition is control to increase the frame rate at the time of shooting, a flag is attached so that slow motion playback will be performed during editing. Also, when the changed imaging condition is control to decrease the frame rate at the time of shooting, a flag is attached so that fast forward playback will be performed during editing.

[0070] In step S209, the control unit 111 determines whether to end the video recording. If the most recently acquired operation instruction information matches the shooting end condition in the shooting scenario acquired in step S201 (YES in step S209), the process proceeds to step S210. After ending the video recording on the imaging-capable mobile device 100, this process ends. On the other hand, if the most recently acquired operation instruction information does not match the shooting end condition in the shooting scenario acquired in step S201 (NO in step S209), the process returns to step S203 to continue the video recording.

[0071] As described above, according to the video recording process of this embodiment, since the imaging conditions are changed according to the operation instruction information, the imaging conditions can be changed faster than the conventional method of changing the imaging conditions based on the image analysis information, and as a result, a video with high artistic quality can be obtained. Also, by using the editing flag, it becomes possible to set the playback speed at the optimal timing when playing back the video recorded by this process.

[0072] Note that in the example of FIG. 5, the scene attention level was determined in step S205 based on the operation instruction information, but information other than the operation instruction information may also be used in combination to determine the scene attention level. For example, information on an oncoming vehicle may be acquired, and the scene attention level in a scene where the moving subject 150 passes by the oncoming vehicle may be set according to the acceleration / deceleration information of the oncoming vehicle. Also, when the window of the moving subject 150 is open and the face of the operator who is operating the moving subject 150 is detected, the scene attention level may be set to "high". Further, landmark information may be detected from the map information, and the scene attention level may be set to "high" in a scene where the moving subject 150 passes by a landmark in the vicinity. Also, when the operation instruction information cannot be received, the scene attention level may be determined from existing image analysis information or map information.

[0073] Also, in this embodiment, control was performed to increase the frame rate during imaging when the scene attention level is high, but the control of the imaging conditions is not limited to this. For example, in a scene where the scene attention level is high, control may be performed to acquire video RAW so that detailed editing is possible, or control may be performed to open the aperture to blur the background and make the main subject stand out. Similarly, when the scene attention level is low, in addition to the control of lowering the frame rate during imaging described in this embodiment, control may be performed to increase the compression rate during recording, or control may be performed to end the shooting in anticipation of cutting in editing.

[0074] In this embodiment, the process shown in the flowchart of FIG. 2 is executed in the control unit 111 of the imaging-capable mobile device 100. However, it may be executed in a server (information processing device) that is communicably connected to the imaging-capable mobile device 100 and the mobile subject 150, and is different from them. In this case, the server pre-retains the shooting scenario list and the information in the table shown in FIG. 5(b), and a change instruction for the imaging conditions according to the operation instruction information from the mobile subject 150 is transmitted from the server to the imaging-capable mobile device 100.

[0075] Also, the process shown in the flowchart of FIG. 2 may be executed by the operation control unit 151 of the mobile subject 150. In this case, the ROM 152 pre-retains the shooting scenario list and the information in the table shown in FIG. 5(b), and a change instruction for the imaging conditions according to the operation instruction information at the mobile subject 150 is transmitted from the mobile subject 150 to the imaging-capable mobile device 100.

[0076] (Embodiment 2) Hereinafter, Embodiment 2, which is a preferred embodiment of the present invention, will be described with reference to the drawings.

[0077] First, the video recording process of the mobile subject 150 according to this embodiment will be described in detail with reference to the flowchart of FIG. 7.

[0078] This embodiment is characterized in that the imaging condition control unit 131 of the imaging device 130 controls the shutter speed among the imaging conditions at the time of shooting based on the operation instruction information received via the communication unit 118.

[0079] FIG. 7 is a flowchart of the process on the side of the imaging-capable mobile device 100 in the video recording process according to this embodiment. This process is executed by the control unit 111 of the mobile device 110 reading the program stored in the ROM 112 into the RAM 113. Also, in the following steps, the steps with an execution subject other than the control unit 111 are actually controlled by the control unit 111 as the execution subject.

[0080] First, in step S701, the control unit 111 acquires the shooting scenario in the shooting scenario list recorded in the shooting scenario list recording unit 116. The shooting scenario list may receive the shooting scenario list from an information processing device (not shown) via the communication unit 118 and record it in the shooting scenario list recording unit 116, or the shooting scenario list may be recorded in the shooting scenario list recording unit 116 in advance.

[0081] In this embodiment, a case where the shooting scenario list includes the "background scrolling" scenario shown in FIG. 8 will be described. The "background scrolling" scenario is a scenario in which the background is scrolled while the moving device 110 shoots while following the moving subject 150, and the ideal scrolling amount of the background is set.

[0082] FIG. 9 is a diagram showing an example of an image to be acquired in the "background scrolling" scenario. As shown in FIG. 9, the purpose of this embodiment is to acquire a highly artistic image in which the ideal scrolling amount of the background is N (pixels).

[0083] Regarding steps S702 to S704 in FIG. 7, since they are the same as steps S202 to S204 in FIG. 2 of the first embodiment, the description thereof will be omitted.

[0084] In step S705, the imaging condition control unit 131 (predicted flow amount calculation means) provided in the imaging device 130 calculates the predicted flow amount of the moving subject 150 based on the operation instruction information transmitted from the control unit 111. The predicted flow amount is an expected amount of the flow amount with respect to the speed of the moving subject 150 predicted from operation instruction information such as the depression amount of the accelerator pedal and the brake pedal. The predicted flow amount N_pred (pixels) is calculated by the following formula (1). N_pred = f(v,t) ···(1)

[0085] Here, v is the predicted velocity quantity of the moving subject 150 and is calculated based on the operation instruction information. Also, t is the shutter speed at the time of shooting. Other elements required for calculating the flow amount of the image plane, such as the distance from the imaging-capable moving device 100 to the moving subject 150, lens parameters, pixel pitch, etc., are expressed by a predetermined relational expression in this embodiment and are not included in the variables of the function f.

[0086] The above is the description regarding the predicted flow amount.

[0087] Returning to FIG. 7, in step S706, the imaging condition control unit 131 (ideal flow amount acquisition means) provided in the imaging device 130 first acquires the ideal flow amount N from the scenario of the "background flow" acquired in step S701. Next, based on the predicted flow amount N_pred calculated in step S705 and the ideal flow amount N, a process for determining whether to change the imaging conditions is performed.

[0088] Here, the process for determining whether to change the imaging conditions executed in step S706 will be described using the flowchart of FIG. 10.

[0089] First, in step S1001, it is determined whether the difference between the predicted flow amount N_pred and the ideal flow amount N is equal to or greater than a predetermined threshold value.

[0090] When the difference between the predicted flow amount N_pred and the ideal flow amount N is equal to or greater than a predetermined threshold value (YES in step S1001), it is determined that it is preferable to change the imaging conditions, particularly the shutter speed, and the process proceeds to step S1002. This is because if the flow amount of the captured video is smaller than the ideal flow amount by a predetermined threshold value or more, there is a high possibility of impairing the workability, and if the flow amount of the captured video is larger than the ideal flow amount by a predetermined threshold value or more, there is a high possibility of flowing even the minute movement of the photographed subject and impairing the workability.

[0091] On the other hand, when the difference between the predicted flow amount N_pred and the ideal flow amount N is less than a predetermined threshold value (NO in step S1001), the process proceeds to step S1005, where it is determined that there is no need to change the imaging conditions (the determination result is no), and this process ends.

[0092] Specifically, in this embodiment, when the predicted flow rate N_pred is smaller than the ideal flow rate N by a predetermined threshold or more, it is determined that the current shutter speed is too short compared to the shutter speed for obtaining the ideal flow rate, and control is performed to increase the shutter speed. On the other hand, when the predicted flow rate N_pred is larger than the ideal flow rate N by a predetermined threshold or more, it is determined that the current shutter speed is too long compared to the shutter speed for obtaining the ideal flow rate, and control is performed to decrease the shutter speed.

[0093] Next, in step S1002, it is determined whether or not the conditions for changing the imaging conditions are satisfied. For example, when the background video has already been shot a predetermined number of times or more, or when increasing the shutter speed causes the frame rate during shooting to exceed the limit, the imaging conditions are not changed. Also, when decreasing the shutter speed causes the non-exposure period to exceed a predetermined amount, or when the ISO sensitivity exceeds a predetermined value, the imaging conditions are not changed. If the conditions for changing the imaging conditions are not satisfied (NO in step S1002), the process proceeds to step S1004, where it is determined that the imaging conditions are not changed (the determination result is no), and this process ends. On the other hand, if the conditions for changing the imaging conditions are satisfied (YES in step S1002), the process proceeds to step S1003, where the imaging conditions are changed (the determination result is yes), and this process ends.

[0094] The detailed processing flow of step S706 has been described above.

[0095] Returning to FIG. 7, if the determination result in step S706 is no, the process proceeds to step S708, while if the result is yes, the process proceeds to step S707.

[0096] In step S707, the imaging condition control unit 131 provided in the imaging device 130 controls the imaging unit 133 to change the imaging conditions based on the result determined in step S706.

[0097] In step S708 of FIG. 7, the control unit 111 determines whether to end the video recording. Specifically, when the most recently acquired operation instruction information matches the shooting end condition in the shooting scenario acquired in step S701 (YES in step S708), it proceeds to step S709. After ending the video recording by the imaging-capable mobile device 100, this process ends. On the other hand, when the most recently acquired operation instruction information does not match the shooting end condition acquired in step S701 (NO in step S708), it returns to step S703 and continues the video recording.

[0098] As described above, according to the video recording process according to this embodiment, since the imaging conditions are changed according to the operation instruction information, the shutter speed can be changed faster than the conventional method of changing the imaging conditions based on the image analysis information, and as a result, a video with high work quality can be obtained.

[0099] In this embodiment, when the shutter speed is increased and the frame rate during shooting is exceeded, the control to change the shutter speed is not performed. However, the control of the imaging conditions is not limited to this. For example, control to also change the frame rate or control to change the aperture value may be performed. Also, a flag for performing editing to pseudo-longen the shutter speed by synthesizing a plurality of consecutive frames during editing may be added to the frame data acquired by the video recording unit 137.

[0100] In this embodiment, the control of the imaging conditions has been performed based only on the operation instruction information. However, the control of the imaging conditions is not limited to this. For example, in a situation where it is difficult to receive the operation instruction information, the predicted flow amount may be calculated from the motion vector or the like by image analysis in step S705.

[0101] (Embodiment 3) Hereinafter, Embodiment 3, which is a preferred embodiment of the present invention, will be described with reference to the drawings.

[0102] This embodiment is characterized in that, based on the operation instruction information received via the communication unit 118, the imaging condition control unit 131 of the imaging device 130 controls the exposure condition among the imaging conditions at the time of shooting.

[0103] FIG. 11 is a flowchart of the processing on the side of the imaging-capable mobile device 100 in the video recording process according to this embodiment. This processing is executed by the control unit 111 provided in the mobile device 110 reading the program stored in the ROM 112 into the RAM 113. Also, in the following steps, the steps with an execution entity other than the control unit 111 are actually controlled by the control unit 111 with respect to the execution entity.

[0104] Regarding steps S1101 to S1102, since they are the same as steps S201 to S202 in FIG. 2 of Embodiment 1, the description is omitted.

[0105] In step S1103, the control unit 111 provided in the mobile device 110 transmits a request for operation instruction information to the moving subject 150 via the communication unit 118.

[0106] In this embodiment, as the operation instruction information, for example, instruction information for controlling the on / off of the headlight and switching between high beam and low beam in the moving subject 150 is received. Note that the operation instruction information is not limited to the instruction information for the headlight, and may be instruction information for other items such as instruction information related to the indoor light that changes the exposure conditions of the moving subject 150 and its surroundings.

[0107] Regarding step S1104, since it is the same as step S204 in FIG. 2 of Embodiment 1, the description is omitted.

[0108] In step S1105, the imaging condition control unit 131 included in the imaging device 130 determines the amount of exposure change based on the operation instruction information transmitted from the control unit 111. The amount of exposure change is, for example, a predetermined amount of exposure change associated with predetermined operation instruction information, such as the turning on and off of the headlight or the switching between high beam and low beam in the moving subject 150. Note that the correspondence information indicating the correspondence between the operation instruction information and the amount of exposure change is stored in advance in the ROM 112 of the moving device 110, and the imaging condition control unit 131 determines the amount of exposure change using this correspondence information.

[0109] In step S1106, the imaging condition control unit 131 included in the imaging device 130 determines whether it is necessary to change the exposure condition of the imaging device 130 based on the amount of exposure change determined in step S1105. If the amount of exposure change is equal to or greater than the threshold value, it is determined that the imaging condition needs to be changed (the determination result is yes), and the process proceeds to step S1107. After executing the exposure condition change process for changing the imaging condition (exposure condition), the process proceeds to step S1108. On the other hand, if the amount of exposure change is less than the threshold value, it is determined that the imaging condition does not need to be changed (the determination result is no), and the process proceeds to step S1108.

[0110] Here, the exposure condition change process executed in step S1107 will be described in detail with reference to the drawings.

[0111] For example, as shown in Fig. 12(a), consider a scene where the moving subject 150 is photographed with the imaging device 130 under dark conditions such as at night. When an operation such as switching the headlight of the moving subject 150 from low beam to high beam is performed during the photographing, the moving device 110 acquires the operation instruction information. At this time, as soon as the operation instruction information is acquired, the imaging condition control unit 131 acquires the amount of exposure change corresponding to the operation instruction from the ROM 112, and if the acquired amount of exposure change is equal to or greater than the threshold value, the exposure condition change process of step S1107 is executed. Therefore, as shown in Fig. 12(b), the exposure tracking of the imaging unit 133 is promptly performed.

[0112] When the exposure condition of the imaging unit 133 is changed by analyzing and feeding back the video in the same scene, as shown in FIGS. 13(a) and 13(b), it takes some time for video analysis, and the exposure tracking of the imaging unit 133 lags behind the changes in the moving subject 150 and the surrounding luminance. As a result, many frames with inappropriate exposure are generated, and the quality of the video deteriorates. On the contrary, as in this embodiment, by changing the exposure condition based on the operation instruction information, it becomes possible to quickly track the exposure with respect to the sudden exposure changes of the moving subject 150 and the surroundings as shown in FIG. 12(b).

[0113] In this embodiment, the change of the exposure condition is performed by changing the ISO sensitivity of the imaging unit 133. At this time, for example, when the ISO sensitivity before the change is 100, etc., if it is the lower limit of the change and the ISO sensitivity cannot be lowered further, but the exposure needs to be lowered further, other exposure parameters such as the aperture value are controlled to lower the exposure. Also, for example, when the ISO sensitivity before the change is the normal ISO sensitivity of the imaging unit 133, etc., if it is the upper limit of the change and the ISO sensitivity cannot be raised further, but the exposure needs to be raised, similarly, other exposure parameters such as the aperture value are controlled to raise the exposure. When the imaging unit 133 has a built-in ND filter, the exposure condition may be changed by changing the density of the ND filter instead of changing the aperture value.

[0114] The above has described in detail the change of the exposure condition.

[0115] Returning to FIG. 11, in step S1108, the control unit 111 determines whether to end the video recording. Specifically, when the most recently acquired operation instruction information matches the shooting end condition in the shooting scenario acquired in step S1101 (YES in step S1108), it proceeds to step S1109, ends the video recording on the imaging-capable mobile device 100, and then ends this process. On the other hand, when the most recently acquired operation instruction information does not match the shooting end condition acquired in step S1101 (NO in step S1108), it returns to step S1103 to continue the video recording shooting.

[0116] As described above, according to the video recording process according to this embodiment, since the imaging conditions are changed according to the operation instruction information, exposure tracking can be performed faster than the conventional method of changing the imaging conditions based on the image analysis information, and as a result, a video with high work quality can be obtained.

[0117] In addition, in this embodiment, the exposure conditions have been controlled based only on the operation instruction information, but the control of the exposure conditions is not limited to this. For example, in addition to the operation instruction information, it may be performed based on time or position information. Specifically, in a tunnel or at a time when the sun has set, the exposure conditions are changed for a certain operation instruction, but no change in the exposure conditions is made even if the same operation instruction is given during the bright daytime. And the calculation of the exposure change amount is not limited to only the predetermined exposure change amount associated with a given operation instruction, and may be performed in consideration of the influence of ambient light.

[0118] (Embodiment 4) Hereinafter, Embodiment 4, which is a preferred embodiment of the present invention, will be described with reference to the drawings.

[0119] In the process on the side of the imaging-capable mobile device 100 in the video recording process of Embodiment 1 (FIG. 2), the operation instruction information was received from the moving subject 150 via the communication unit 118. In contrast, in this embodiment, in addition to the operation instruction information, environmental information is also received from the moving subject 150 via the communication unit 118. The environmental information referred to here means, for example, information including the position information of the moving subject 150, the weather information around it, the map, the structure information, and the sun-facing and shaded information. That is, this embodiment is characterized in that the imaging condition control unit 131 provided in the imaging device 130 controls the exposure condition among the imaging conditions at the time of shooting based on the received operation instruction information and environmental information.

[0120] FIG. 14 is a flowchart of the processing on the side of the imaging-capable mobile device 100 in the video recording process according to this embodiment. This processing is executed by the control unit 111 provided in the mobile device 110 by reading the program stored in the ROM 112 into the RAM 113. Also, for the steps in the following that are executed by entities other than the control unit 111, in reality, the control unit 111 controls the execution entities.

[0121] Regarding steps S1401 to S1402 in FIG. 14, since they are the same as steps S201 to S202 in FIG. 2 of Embodiment 1, the description is omitted.

[0122] In step S1403, the control unit 111 provided in the mobile device 110 transmits a request for steering instruction information and environmental information to the moving subject 150 via the communication unit 118.

[0123] Here, the steering instruction information refers to, for example, instruction information regarding the control of the steering wheel, brakes, and accelerator in the moving subject 150. Regarding the environmental information, as described above, it refers to information including the position information of the moving subject 150, the surrounding weather information, maps, structure information, sunlit and shaded area information.

[0124] In step S1404, the control unit 111 provided in the mobile device 110 determines whether it has received the steering instruction information and environmental information transmitted from the moving subject 150. If it has received the steering instruction information and environmental information (YES in step S1404), it transmits the received steering instruction information and environmental information to the imaging condition control unit 131 provided in the imaging device 130 and proceeds to step S1405. On the other hand, if it has not received at least one of the steering instruction information and environmental information (NO in step S1404), it returns to step S1403, and again, the control unit 111 transmits a request for steering instruction information and environmental information to the moving subject 150.

[0125] In step S1405, the imaging condition control unit 131 included in the imaging device 130 calculates a predicted exposure change amount based on the operation instruction information and the environmental information transmitted from the control unit 111. The predicted exposure change amount is a value obtained by predicting how much exposure change will occur after how much time has elapsed from the current time due to, for example, the switching between sunlight and shade. A detailed explanation of the predicted exposure change amount will be given later.

[0126] In step S1406, the imaging condition control unit 131 included in the imaging device 130 determines whether it is necessary to change the exposure condition of the imaging device 130 based on the predicted exposure change amount calculated in step S1405. If the predicted exposure change amount is equal to or greater than the threshold value, it is determined that a change in the imaging conditions is necessary (the determination result is yes), and the process proceeds to step S1407. After executing the exposure condition change process for changing the imaging condition (exposure condition), the process proceeds to step S1408. On the other hand, if the predicted exposure change amount is less than the threshold value, it is determined that there is no need to change the imaging conditions (the determination result is no), and the process proceeds to step S1408.

[0127] Here, the exposure condition change process executed in step S1407 will be described in detail with reference to the drawings.

[0128] For example, as shown in FIG. 15(a), consider a scene in which a moving subject 150 is being photographed by the imaging device 130 under the condition that the sun is out during the day. Each time the moving subject 150 alternately passes through sunlight and shade during the photographing, the brightness of the moving subject 150 and its surroundings changes. Therefore, in order to keep the photographed video at an appropriate exposure or an arbitrary exposure, it is necessary to change the exposure condition of the imaging unit 133 at any time.

[0129] Therefore, in this embodiment, first, environmental information such as position information, surrounding structure information, and weather information, and operation instruction information such as brakes and accelerators from the moving subject 150 are acquired. Next, based on the acquired environmental information and operation instruction information, a predicted exposure change amount is calculated from the predicted exposure change amount at the time of switching between sunny and shady conditions and the required time until the exposure change occurs. When the calculated predicted exposure change amount is equal to or greater than the threshold value, the exposure condition change process in step S1407 is executed. Specifically, as shown in FIG. 15(b), the imaging unit 133 gradually changes the exposure conditions in advance. As a result, it becomes possible to obtain a high-quality video without a sudden exposure change.

[0130] When the exposure conditions of the imaging unit 133 are changed by analyzing and feedbacking the video in a similar scene, as shown in FIGS. 16(a) and 16(b), it takes some time for video analysis, and the exposure follow-up of the imaging unit 133 lags behind the changes in the moving subject 150 and the surrounding brightness. As a result, many frames with inappropriate exposure are generated, and the quality of the video deteriorates.

[0131] Furthermore, with reference to another figure, the control of the timing for changing the exposure conditions in the exposure condition change process executed in step S1407 will be described in detail.

[0132] As shown in FIG. 17, for example, the difference in the exposure amount that becomes the appropriate exposure condition in each of the sunny and shady conditions is defined as the predicted exposure change amount. When this predicted exposure change amount is ΔEV, the exposure change of the captured video is gentle, and the exposure change amount per second (per unit time) that can maintain high quality is defined as Δa. Note that the value of Δa can be arbitrarily set by the user. The change of the exposure conditions is started t (seconds) before the time when the sunny and shady conditions are switched so that the exposure change amount per second becomes Δa. Here, t (seconds) is calculated by the following formula (2). t = ΔEV / 2a ···(2)

[0133] If the predicted cycle of the sunlit and shaded areas is 2t (seconds) or less, the exposure conditions are not changed. The predicted cycle of the sunlit and shaded areas is calculated, for example, from the actual distance between the sunlit and shaded areas and the estimated subject speed.

[0134] The above is a detailed description of the exposure condition change.

[0135] Returning to FIG. 14, in step S1408, the control unit 111 determines whether to end the video recording. Specifically, if the most recently acquired operation instruction information matches the shooting end condition in the shooting scenario acquired in step S1401 (YES in step S1408), the process proceeds to step S1409. After ending the video recording by the image-capturable mobile device 100, this process ends. On the other hand, if the most recently acquired operation instruction information does not match the shooting end condition acquired in step S1401 (NO in step S1408), the process returns to step S1403 to continue the video recording.

[0136] As described above, according to the video recording process according to the fourth embodiment, since the imaging conditions are changed according to the operation instruction information and the environmental information, exposure tracking can be performed faster than the conventional method of changing the imaging conditions based on the image analysis information, and as a result, a video with high quality can be obtained.

[0137] (Other Embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. Further, it can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0138] The invention is not limited to the above-described embodiments, and various changes and modifications are possible without departing from the spirit and scope of the invention. Therefore, the claims are attached to disclose the scope of the invention.

[0139] As described above, the preferred embodiments of the present invention have been explained. However, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist thereof.

[0140] The disclosure of this embodiment includes the following configurations, methods, and programs. (Configuration 1) An information processing system having a moving subject with operation and an imaging-capable moving device equipped with an imaging device for photographing the moving subject, the information processing system having an acquisition means for acquiring operation instruction information of the moving subject and an imaging control means for controlling imaging by the imaging device, wherein the imaging control means controls imaging conditions at the time of photographing by the imaging device based on the operation instruction information. (Configuration 2) Further having a shooting scenario selection means for selecting a shooting scenario from a shooting scenario list in which conditions related to shooting are described, wherein the shooting scenario list describes at least a shooting start condition and a shooting end condition determined based on the operation instruction information and imaging conditions of the imaging device. The information processing system according to Configuration 1. (Configuration 3) The information processing system according to Configuration 2, wherein the imaging conditions include at least one of a frame rate, a shutter speed, and an exposure condition. (Configuration 4) The imaging control means controls the imaging device to perform imaging of the moving subject with operation according to the shooting start condition, shooting end condition, and imaging conditions described in the shooting scenario selected by the shooting scenario selection means. The information processing system according to Configuration 3. (Configuration 5) The information processing system according to any one of Configurations 1 to 4, wherein the operation instruction information is information for instructing the position or operation of the moving subject. (Configuration 6) The information processing system according to Configuration 5, wherein the operation instruction information is information including at least one of information for instructing the speed, direction, operation of a movable mechanism, notification by sound or light, and light emission of a light source of the moving subject. Configuration 7: The information processing system according to any one of Configurations 4 to 6, further comprising a scene attention degree determination means for determining the scene attention degree of the shooting scene based on the operation instruction information. Configuration 8: The information processing system according to Configuration 7, wherein the imaging control means controls to increase the frame rate when it is determined that the scene attention degree is high, and controls to decrease the frame rate when it is determined that the scene attention degree is low. Configuration 9: The information processing system according to Configuration 8, wherein when the operation instruction information is any one of window opening / closing, a blinker, and drifting, the scene attention degree determination means determines that the scene attention degree is high. Configuration 10: The information processing system according to Configuration 8 or 9, wherein when there is no operation instruction information for a certain period of time, the scene attention degree determination means determines that the scene attention degree is low. Configuration 11: The information processing system according to Configuration 8 or 9, wherein when the operation instruction information is deceleration, the scene attention degree determination means determines that the scene attention degree is low. Configuration 12: The information processing system according to any one of Configurations 8 to 11, wherein the scene attention degree determination means acquires operation instruction information of another moving subject, and determines the scene attention degree by using the operation instruction information of the other moving subject in combination with the operation instruction information. Configuration 13: The information processing system according to any one of Configurations 8 to 11, wherein the scene attention degree determination means acquires image analysis information of the moving subject, and determines the scene attention degree by using the image analysis information in combination with the operation instruction information. Configuration 14: The information processing system according to Configuration 13, wherein the scene attention degree determination means acquires information of a landmark near the moving subject, and determines the scene attention degree by using the image analysis information in combination with the information of the landmark. Configuration 15: The information processing system according to any one of Configurations 8 to 14, wherein when the imaging control means does not satisfy the conditions for changing the imaging conditions, the frame rate is not changed according to the scene attention degree. (Configuration 16) When the imaging device changes the frame rate based on the scene attention degree, it is characterized in that it records by attaching a flag indicating that the playback speed is changed during editing, and the information processing system according to any one of Configurations 8 to 14. (Configuration 17) The information processing system according to any one of Configurations 4 to 6, characterized by having a predicted flow amount calculation means for calculating a predicted flow amount of the background based on the steering instruction information. (Configuration 18) It has an ideal flow amount acquisition means for acquiring an ideal flow amount of the background from the shooting scenario, and the imaging control means controls to change at least the shutter speed when the difference between the predicted flow amount of the background and the ideal flow amount is equal to or more than a predetermined value. The information processing system according to Configuration 17. (Configuration 19) The predicted flow amount of the background is calculated based on at least the predicted speed of the moving subject based on the steering instruction information and the shutter speed. The information processing system according to Configuration 17 or 18. (Configuration 20) When it is difficult to change the shutter speed based on the predicted flow amount of the background and the ideal flow amount, the imaging device is characterized in that it attaches a flag for synthesizing a plurality of frames during editing. The information processing system according to Configuration 18. (Configuration 21) When the imaging control means does not satisfy the conditions for changing the imaging conditions, it does not change the shutter speed based on the predicted flow amount of the background and the ideal flow amount. The information processing system according to Configuration 18. (Configuration 22) The information processing system according to any one of Configurations 4 to 6, characterized by acquiring an exposure change amount corresponding to the steering instruction information. (Configuration 23) The imaging control means changes the exposure conditions when the exposure change amount is equal to or more than a threshold value. The information processing system according to Configuration 22. (Configuration 24) The information processing system according to Configuration 22 or 23, characterized by changing the exposure conditions based on the ISO sensitivity. The information processing system according to Configuration 24, characterized in that when it is difficult to control exposure at the ISO sensitivity, the exposure conditions are changed based on at least one of the means of changing the aperture value, shutter speed, and ND filter. The information processing system according to any one of Configurations 1 to 25, characterized in that, in addition to the operation instruction information, environmental information around the subject involving operation is acquired together. The information processing system according to Configuration 26, characterized in that the environmental information is information including at least one of the position information of the subject involving operation, the surrounding weather, map, structure, sunny side and shady side, and time. The information processing system according to Configuration 27, characterized in that a predicted exposure change amount is calculated based on the operation instruction information and the environmental information. The information processing system according to Configuration 28, characterized in that the predicted exposure change amount is information including the predicted exposure change amount and the time required until the exposure change occurs. The information processing system according to Configuration 29, characterized by having means for inputting the exposure change amount per unit time, wherein the imaging control means calculates the time to start controlling exposure based on the predicted exposure change amount and the exposure change amount per unit time, and controls exposure based on the time. The information processing system according to any one of Configurations 1 to 30, characterized in that when it is difficult to acquire the operation instruction information, the imaging device controls the imaging conditions at the time of shooting based on at least the image analysis information of the subject involving operation. The information processing system according to any one of Configurations 1 to 31, characterized in that the imaging control means is mounted on the movable imaging device, and the acquisition means acquires the operation instruction information from the moving subject by requesting the operation instruction information to the moving subject. (Configuration 33) The imaging control means is mounted on the moving subject, and information for controlling the imaging conditions is transmitted from the moving subject to the imaging-capable moving device. The information processing system according to any one of Configurations 1 to 31, characterized in that. (Configuration 34) The imaging scenario list further describes a main subject. The information processing system according to any one of Configurations 2 to 4, characterized in that. (Configuration 35) An information processing device communicably connected to an imaging-capable moving device equipped with a moving subject and an imaging device for photographing the moving subject, the obtaining means for obtaining the steering instruction information of the moving subject from the moving subject, and the imaging control means for controlling the imaging by the imaging device. The imaging control means transmits imaging conditions at the time of shooting to the imaging-capable moving device based on the steering instruction information. An information processing device characterized by that. (Method 1) A control method for an information processing system having a moving subject and an imaging-capable moving device equipped with an imaging device for photographing the moving subject, the obtaining step of obtaining the steering instruction information of the subject accompanied by steering, and the imaging control step of controlling the imaging by the imaging device. The control method is characterized in that the imaging conditions at the time of shooting are controlled based on the steering instruction information in the imaging control step. (Method 2) A control method for an information processing device communicably connected to a moving subject and an imaging-capable moving device equipped with an imaging device for photographing the moving subject, the obtaining step of obtaining the steering instruction information of the moving subject from the moving subject, and the imaging control step of controlling the imaging by the imaging device. The control method is characterized in that the imaging conditions at the time of shooting are transmitted to the imaging-capable moving device based on the steering instruction information in the imaging control step. A control method characterized by that. (Program 1) A program for causing a computer to function as each means of the information processing system according to any one of Configurations 1 to 34. (Program 2) A program for causing a computer to function as each means of the information processing device according to Configuration 35.

Explanation of Signs

[0141] 100 Imaging-capable mobile device 110 Mobile device 111 Control unit 112 ROM 113 RAM 114 Position and attitude detection unit 115 Image processing unit 116 Shooting scenario list recording unit 117 Movement mechanism unit 118 Communication unit 130 Imaging device 131 Imaging condition control unit 132 Optical system 133 Imaging unit 134 A / D conversion unit 135 Image processing unit 136 Display unit 137 Video recording unit 150 Moving subject with operation 151 Operation control unit 152 ROM 153 RAM 154 Position detection unit 155 Operation mechanism unit 156 Movement mechanism unit 157 Operation communication unit

Claims

1. An information processing system comprising a moving subject that involves operation, and an imaging-capable mobile device equipped with an imaging device for photographing the moving subject, acquisition means for acquiring operation instruction information of the moving subject, imaging control means for controlling imaging by the imaging device, and the imaging control means controls imaging conditions at the time of photographing by the imaging device based on the operation instruction information. The information processing system is characterized by this.

2. The information processing system according to claim 1, further comprising photographing scenario selection means for selecting a photographing scenario from a photographing scenario list in which conditions related to photographing are described, wherein the photographing scenario list describes at least a photographing start condition and a photographing end condition determined based on the operation instruction information, and imaging conditions of the imaging device.

3. The information processing system according to claim 2, wherein the imaging conditions include at least one of a frame rate, a shutter speed, and an exposure condition.

4. The imaging control means controls the imaging device to perform imaging of the moving subject involving operation according to the photographing start condition, photographing end condition, and imaging conditions described in the photographing scenario selected by the photographing scenario selection means. The information processing system according to claim 3 is characterized by this.

5. The information processing system according to claim 1, wherein the operation instruction information is information for instructing the position and movement of the moving subject.

6. The information processing system according to claim 5, wherein the operation instruction information is information including at least one of information for instructing the speed, direction, movement of a movable mechanism, notification by sound or light, and light emission of a light source of the moving subject.

7. The information processing system according to claim 4, further comprising scene attention degree determination means for determining the scene attention degree of a photographing scene based on the operation instruction information.

8. The imaging control means performs control to increase the frame rate when it is determined that the scene attention degree is high, and performs control to decrease the frame rate when it is determined that the scene attention degree is low. The information processing system according to claim 7 is characterized by this.

9. The information processing system according to claim 8, wherein when the operation instruction information is any one of window opening / closing, a turn signal, and drifting, the scene attention degree determination means determines that the scene attention degree is high.

10. The information processing system according to claim 8, wherein when there is no operation instruction information for a certain period of time, the scene attention degree determination means determines that the scene attention degree is low.

11. The information processing system according to claim 8, wherein when the operation instruction information is deceleration, the scene attention degree determination means determines that the scene attention degree is low.

12. The scene attention degree determination means acquires operation instruction information of other moving subjects, and determines the scene attention degree by using the operation instruction information of the other moving subjects in combination with the operation instruction information. The information processing system according to claim 8 is characterized in that.

13. The scene attention degree determination means acquires image analysis information of the moving subject, and determines the scene attention degree by using the image analysis information in combination with the operation instruction information. The information processing system according to claim 8 is characterized in that.

14. The scene attention degree determination means acquires information on landmarks in the vicinity of the moving subject, and determines the scene attention degree by using the image analysis information in combination with the information on the landmarks. The information processing system according to claim 13 is characterized in that.

15. The imaging control means does not change the frame rate according to the scene attention degree when the conditions for changing the imaging conditions are not satisfied. The information processing system according to claim 8 is characterized in that.

16. When the imaging device changes the frame rate based on the scene attention degree, the information processing system according to claim 8 is characterized in that a flag indicating that the playback speed is changed during editing is attached and recorded.

17. The information processing system according to claim 4, further comprising a predicted flow amount calculation means for calculating a predicted flow amount of the background based on the operation instruction information.

18. having an ideal flow amount acquisition means for acquiring an ideal flow amount of the background from the shooting scenario, and when the difference between the predicted flow amount of the background and the ideal flow amount is equal to or greater than a predetermined value, the imaging control means controls to change at least the shutter speed. The information processing system according to claim 17 is characterized in that.

19. The predicted flow amount of the background is calculated based on at least the predicted speed of the moving subject based on the operation instruction information and the shutter speed, according to the information processing system described in claim 17.

20. The imaging device is characterized in that when it is difficult to change the shutter speed based on the predicted flow amount of the background and the ideal flow amount, a flag for synthesizing a plurality of frames is given at the time of editing, according to the information processing system described in claim 18.

21. The imaging control means is characterized in that when the conditions for changing the imaging conditions are not satisfied, the shutter speed is not changed based on the predicted flow amount of the background and the ideal flow amount, according to the information processing system described in claim 18.

22. The information processing system according to claim 4, characterized in that it acquires the amount of exposure change corresponding to the operation instruction information.

23. The imaging control means is characterized in that when the amount of exposure change is equal to or greater than a threshold value, it changes the exposure conditions, according to the information processing system described in claim 22.

24. The information processing system according to claim 22, characterized in that it changes the exposure conditions based on the ISO sensitivity.

25. When it is difficult to control the exposure with the ISO sensitivity, the exposure conditions are changed based on at least one of the means of changing the aperture value, the shutter speed, and the ND filter, according to the information processing system described in claim 24.

26. The information processing system according to claim 1, characterized in that in addition to the operation instruction information, it acquires the environmental information around the subject accompanied by the operation.

27. The environmental information is information including at least one of the position information of the subject accompanied by the operation, the surrounding weather, the map, the structure, the sunny side and the shady side, and the time, according to the information processing system described in claim 26.

28. The information processing system according to claim 27, characterized in that it calculates a predicted exposure change amount based on the operation instruction information and the environmental information.

29. The predicted exposure change amount is information including the predicted exposure change amount and the time required until the exposure change occurs, according to the information processing system described in claim 28.

30. It has means for inputting the amount of exposure change per unit time, The imaging control means calculates the time to start controlling the exposure based on the predicted exposure change amount and the exposure change amount per unit time, and controls the exposure based on the time, according to the information processing system described in claim 29.

31. When it is difficult to obtain the operation instruction information, the imaging device controls the imaging conditions at the time of shooting based on at least the image analysis information of the subject involving operation, according to the information processing system described in claim 1.

32. The imaging control means is mounted on the imaging-capable mobile device. The acquisition means acquires the operation instruction information from the moving subject by requesting the operation instruction information to the moving subject, according to the information processing system described in claim 1.

33. The imaging control means is mounted on the moving subject, and information for controlling the imaging conditions is transmitted from the moving subject to the imaging-capable mobile device, according to the information processing system described in claim 1.

34. The shooting scenario list further describes the main subject, according to the information processing system described in claim 2.

35. An information processing device communicably connected to an imaging-capable mobile device equipped with a moving subject and an imaging device for photographing the moving subject, an acquisition means for acquiring operation instruction information of the moving subject from the moving subject; an imaging control means for controlling imaging by the imaging device, and the imaging control means transmits imaging conditions at the time of shooting to the imaging-capable mobile device based on the operation instruction information, according to the information processing device.

36. A control method for an information processing system having a moving subject and an imaging-capable mobile device equipped with an imaging device for photographing the moving subject, an acquisition step of acquiring operation instruction information of a subject involving operation; an imaging control step of controlling imaging by the imaging device, and in the imaging control step, the imaging conditions at the time of shooting are controlled based on the operation instruction information, according to the control method.

37. A control method for an information processing device communicably connected to an imaging-capable mobile device equipped with a moving subject and an imaging device for photographing the moving subject, an acquisition step of acquiring operation instruction information of the moving subject from the moving subject; an imaging control step of controlling imaging by the imaging device, and In the imaging control step, an imaging condition at the time of imaging is transmitted to the imaging capable mobile device based on the operation instruction information, and a control method characterized by this is provided.

38. A program for causing a computer to function as each means included in the information processing system according to claim 1.

39. A program for causing a computer to function as each means included in the information processing apparatus according to claim 35.

Images