Information processing system, information processing apparatus, control method, program product, and storage medium

Abstract

The invention relates to an information processing system, an information processing apparatus, a control method, a program product, and a storage medium. The information processing system enables a mobile imaging apparatus to reliably capture an image of a moving subject being moved. The information processing system includes: moving a subject; a mobile image capturing apparatus for capturing an image of the mobile subject; a receiving unit configured to receive, from the mobile imaging apparatus, a detection result indicating whether the moving subject is included in an imaging field angle of the mobile imaging apparatus; and an instruction unit configured to issue an instruction related to manipulation of at least one of the moving subject and the moving image pickup apparatus according to the detection result.

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing device and a control method thereof, a computer program product, and a storage medium storing a program, and particularly relates to an information processing system, an information processing device and a control method thereof, a computer program product, and a storage medium capable of controlling a mobile imaging device and a mobile subject to be imaged. Background Art

[0002] When taking pictures of scenery while traveling by car during a journey or in daily life, the mainstream method is to take pictures of scenery using a camera attached to the inside or outside of a car. So far, it has been difficult to take an aerial view of a moving car together with the scenery during driving. However, in recent years, by using a mobile imaging device such as a drone to take images of a moving subject such as a car, such aerial view imaging has become possible.

[0003] When taking pictures using a drone, in many cases, the flight of the drone is automatically or manually operated so that the moving subject will be included in the imaging field angle of the drone. However, in such imaging, there are the following problems: when the speed of the moving subject is too high for the drone to catch up with the moving subject, or when the moving subject is blocked by an obstacle and the drone cannot see the moving subject, the drone cannot take an image of the moving subject.

[0004] As a proposal for solving such a problem, Japanese Unexamined Patent Application Publication No. 2017-56903 (JP 2017-56903A) proposes a technique for estimating the position of a moving subject based on information representing the movement of the moving subject and controlling the drone to face that position.

[0005] In addition, Japanese Unexamined Patent Application Publication No. 2018-201218 (JP 2018-201218, related to US20180131856A1) proposes a technique in which the drone is moved to a predetermined standby position designated by a control device to be in an imaging standby state, and after waiting for the imaging object (moving subject) to arrive, the drone is controlled to perform imaging.

[0006] However, the technique of JP 2017-56903A has the following problem: when the speed of the moving subject is too fast, the drone cannot catch up with the moving subject and the moving subject cannot be included in the field angle of view of the drone.

[0007] In addition, the technique of JP 2018-201218 has the following problem: when the movement of the moving subject is different from the predicted movement, the moving subject cannot be included in the imaging field angle of view of the drone. Summary of the Invention

[0008] The present invention provides an information processing system, an information processing device, a control method, a computer program product, and a storage medium storing a program, which enable a mobile imaging device to reliably capture an image of a moving object in motion.

[0009] Accordingly, an aspect of the present invention provides an information processing system including: a moving object; a mobile imaging device configured to capture an image of the moving object; a receiving unit configured to receive from the mobile imaging device a detection result indicating whether the moving object is included in a field of view angle of the mobile imaging device; and an instruction unit configured to issue an instruction related to the manipulation of at least one of the moving object and the mobile imaging device based on the detection result.

[0010] According to the present invention, a moving object in motion can be reliably captured by a mobile imaging device.

[0011] More features of the present invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram showing a hardware configuration of an information processing system including a mobile imaging device and a moving object according to a first embodiment.

[0013] Figure 2 shows Figure 1 the software configuration of the automatic driving ECU in

[0014] Figure 3 is a flowchart schematically showing an imaging process executed by the information processing system in the first embodiment.

[0015] Figure 4A , Figure 4B and Figure 4C are diagrams showing scenes of a moving object captured by a mobile imaging device in flight and images captured in each scene.

[0016] Figure 5A , Figure 5B and Figure 5C are diagrams showing examples of changes in a travel plan of a moving object according to the first embodiment.

[0017] Figure 6 is a block diagram showing a hardware configuration of an information processing system according to a second embodiment.

[0018] Figure 7 shows Figure 6 the software configuration of the notification information generation ECU in

[0019] Figure 8 is a flowchart schematically showing the imaging process performed by the information processing system in the second embodiment.

[0020] Figure 9 is a block diagram showing the software structure of the movement controller of the moving imaging device.

[0021] Figure 10 is a flowchart schematically showing the imaging process performed by the information processing system in the third embodiment.

[0022] Figure 11A 、 Figure 11B and Figure 11C is a diagram showing an example of a change in the travel plan of the moving subject according to the third embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the appended claims. Although multiple features are described in the embodiments, not all of the multiple features are necessarily required for the present invention, and these multiple features can be arbitrarily combined. In addition, in the drawings, the same or similar components are denoted by the same reference numerals, and redundant descriptions are omitted.

[0024] Hereinafter, an information processing system 100 according to the first embodiment of the present invention will be described.

[0025] The information processing system 100 includes a moving imaging device 10 (imaging device) and a moving subject 20, and it is assumed that the moving imaging device 10 is used to capture an image of the moving subject 20 equipped with an automatic driving system and accompanied by an operator. In the present embodiment, the moving imaging device 10 should be an unmanned aerial vehicle, such as a drone flying in the air through a thruster. However, the moving imaging device 10 is not limited thereto, as long as it can move. For example, the moving imaging device can be a wheeled vehicle moving on the ground or a submarine with a propeller moving in water. Although a car is assumed as the moving subject 20 in the present embodiment, the moving subject 20 is not limited thereto, as long as it moves according to the operation of the occupant or an instruction from the automatic driving system. For example, the moving subject 20 can be a motorcycle, a ship, an airplane, or a train. The moving subject 20 can be operated by remote control, such as a remote control car, without being operated by an occupant.

[0026] Hereinafter, the present embodiment will be described in detail with reference to the accompanying drawings.

[0027] Figure 1 is a block diagram showing the hardware structure of the information processing system 100 according to the present embodiment.

[0028] As Figure 1 shown, the information processing system 100 includes a mobile imaging device 10, a mobile subject 20, and an instruction device 30 (information processing device).

[0029] The mobile imaging device 10 includes a controller 11, a ROM 12, a RAM 13, a movement controller 14, a motor group 14a, a thruster group 14b, a sensor group 14c, an imaging device 15, a position and posture estimation unit 16, a mobile subject detection unit 17, a recording unit 18, and a communication unit 19. These blocks are communicably connected to each other via a bus. The thruster group 14b includes four thrusters, and the motor group 14a includes four motors. The four thrusters are respectively rotationally driven by the four motors.

[0030] The controller 11 is, for example, a CPU, reads the operation programs for the respective blocks included in the mobile imaging device 10 from the ROM 12, expands the operation programs onto the RAM 13, and executes the operation programs to control the operations of the respective blocks included in the mobile imaging device 10.

[0031] The ROM 12 is an electrically erasable and recordable non-volatile memory, and stores not only the operation programs for the respective blocks included in the mobile imaging device 10 but also the parameters required for the operations of the respective blocks.

[0032] The RAM 13 is a rewritable volatile memory and serves as a temporary storage area for the data output in the operations of the respective blocks included in the mobile imaging device 10.

[0033] The movement controller 14 controls the flight function of the mobile imaging device 10. In order for the mobile imaging device 10 to fly, it is necessary to control the motor group 14a that rotationally drives the thruster group 14b to generate the power required for flight. Therefore, the movement controller 14 controls the driving force of the motors constituting the motor group 14a to control the rpm of the thrusters constituting the thruster group 14b, thereby controlling the movement direction, position, and posture of the mobile imaging device 10. The movement controller 14 also uses the sensor group 14c to obtain information such as altitude and acceleration to identify the flight state of the mobile imaging device 10. There is no particular limitation on the sensor group 14c as long as it can obtain information such as altitude and acceleration. For example, the sensor group 14c may include a GPS sensor that detects GPS information, a gyro sensor that detects angular velocity, an acceleration sensor that detects acceleration, a magnetic sensor that detects the movement direction, an atmospheric pressure sensor that detects altitude, an ultrasonic sensor that detects the distance to surrounding objects, or LiDAR (light detection and ranging). In addition, the movement controller 14 transmits information related to the flight conditions from the communication unit 19 to the instruction device 30.

[0034] The imaging device 15 includes an optical system including a zoom lens group and a focusing lens group, and an image sensor that converts an image formed by the optical system into an electrical signal, and applies various image processes to obtain a captured image. The obtained captured image is sent to the moving object detection unit 17 and recorded in the recording unit 18.

[0035] The position and orientation estimation unit 16 detects the imaging position of the mobile imaging device 10. The imaging position is detected using, for example, the flight route of the mobile imaging device 10, map information, and information (GPS information, etc.) obtained from the sensor group 14c by the mobile controller 14. When the destination of the moving object 20 is set, a travel plan of the moving object 20 to the destination is determined, and accordingly, the flight route of the mobile imaging device 10 is determined.

[0036] The position and orientation estimation unit 16 can also estimate information related to a future imaging position based on information related to acceleration and moving direction obtained by the mobile controller 14 from the sensor group 14c. Information related to the estimated imaging position, the obtained acceleration and moving direction, and information related to a future flight route are transmitted as imaging-related information to the indicating device 30 via the communication unit 19.

[0037] The moving object detection unit 17 determines whether the entire or a part of the moving object 20 as an imaging object is captured based on the image captured by the imaging device 15. For example, a template image of the moving object 20 is prepared, and the moving object detection unit 17 determines whether the moving object 20 is included in the field of view angle of the imaging device 15 by performing pattern matching between the template image and the captured image. In addition, an indication determination unit 35 (indicating unit) described later in the indicating device 30 determines whether an indication related to the operation of the moving object needs to be issued based on the determination result.

[0038] The recording unit 18 is a detachable memory card or the like, and records the image processed by the imaging device 15 as a recorded image via the RAM 13.

[0039] The communication unit 19 exchanges information with the indicating device 30. Specifically, the communication unit 19 sends imaging-related information obtained from the position and orientation estimation unit 16 and information related to a determination result indicating whether the moving object 20 is included in the field of view angle of the captured image using the moving object detection unit 17 to the indicating device 30.

[0040] The structure and basic operations of the mobile imaging device 10 have been described above.

[0041] Next, with reference to Figure 1 an example of the structure of the moving object 20 will be described.

[0042] The moving object 20 is equipped with an automatic driving system that controls the moving object 20.

[0043] The moving object 20 includes a sensor group 21, a map database 22, a GPS receiver 23, an automatic driving ECU 24, a drive mechanism 25, and a communication unit 26. These blocks are communicably connected to each other via a bus.

[0044] The sensor group 21 detects the surrounding information and driving information of the moving object 20. The surrounding information obtained from the sensor group 21 includes information related to objects (such as pedestrians, guardrails, and other moving objects in front) around the moving object detected using, for example, LiDAR. Specifically, the LiDAR irradiates the surroundings of the moving object 20 with laser light and detects the reflected objects around the moving object 20 by receiving the reflected light. The LiDAR detects the relative distance between the object and the moving object 20 based on the time until the laser returns as reflected light from the detected object.

[0045] The driving information obtained from the sensor group 21 includes, for example, driving characteristics detected by a vehicle speed sensor and an acceleration sensor, and information related to operations detected by an accelerator pedal position sensor, a steering angle sensor, and a brake pedal position sensor.

[0046] The map database 22 pre - records map information and is stored in a recording device (not shown) of the moving object 20. The map information includes road position information, road inclination information, intersection position information, branch point position information, building position information, and traffic rule information at each position.

[0047] The GPS receiver 23 receives signals from multiple GPS satellites and determines the position of the moving object 20 based on the received signals. The position is represented by, for example, latitude and longitude. The GPS receiver 23 sends information related to the measured position of the moving object 20 to the automatic driving ECU 24.

[0048] The automatic driving ECU 24 (control unit) is an electronic controller including a CPU, a ROM, a RAM, and a CAN (Controller Area Network) communication circuit, and realizes automatic driving by controlling the hardware based on signals output by the CPU. Examples of specific operations are as follows. The CAN communication circuit is operated to store the data obtained from the sensor group 21, the map database 22, and the communication unit 26 as input data in the RAM. After that, the automatic driving ECU 24 determines a driving control signal by loading the program recorded in the ROM into the RAM and executing the program based on the input data, and sends the determined driving control signal to the drive mechanism 25. Details of this process will be described later.

[0049] The drive mechanism 25 includes actuators for an engine, a brake, and an electric power steering device and their controllers, and controls the drive of the moving object 20 based on the driving control signal received from the autonomous driving ECU 24.

[0050] The communication unit 26 exchanges information with the indicating device 30. Information related to driving characteristics and operations obtained from the sensor group 21 and information related to the position of the moving object 20 obtained from the GPS receiver 23 are sent to the indicating device 30 as driving information. In addition, indication information related to the operation of the moving object 20 is received from the indicating device 30. The communication unit 26 sends the received indication information to the autonomous driving ECU 24 and reflects it in the driving control signal.

[0051] The structure and basic operations of the moving object 20 have been described above.

[0052] Next, with reference to Figure 1 the structural example of the indicating device 30 will be described.

[0053] The indicating device 30 determines an instruction for operating the moving object 20 based on the information received from the mobile imaging device 10 and the moving object 20.

[0054] The indicating device 30 includes a control unit 31, a ROM 32, a RAM 33, an information acquisition unit 34, an instruction determination unit 35, and a communication unit 36. These blocks are communicably connected to each other via a bus.

[0055] The controller 31 reads the operation programs used by the blocks included in the indicating device 30 from the ROM 32, expands these operation programs onto the RAM 33, and executes these operation programs, thereby controlling the operations of the blocks included in the indicating device 30.

[0056] The ROM 32 is an electrically erasable and recordable non-volatile memory, and in addition to storing the operation programs used by the blocks included in the indicating device 30, it also stores the parameters required for the operations of these blocks.

[0057] The RAM 33 is a rewritable volatile memory and is used as a temporary storage area for the data output during the operations of the blocks included in the indicating device 30.

[0058] The information acquisition unit 34 stores the information obtained from the communication unit 36. The acquired information includes information related to imaging and flight obtained from the mobile imaging device 10, driving information obtained from the moving object 20, etc. The information acquired by the information acquisition unit 34 is sent to the instruction determination unit 35, and the instruction determination unit 35 determines an instruction related to the operation of the moving object 20.

[0059] When the instruction determination unit 35 receives the determination result of the moving subject detection unit 17 (the determination result indicating whether the moving subject 20 is included in the field of view angle of the image captured by the imaging device 15), the instruction determination unit 35 determines whether to issue an instruction related to manipulation to the moving subject 20 based on this determination result. When giving an instruction related to manipulation, the instruction determination unit 35 determines the instruction content based on the information obtained from the information acquisition unit 34. This process will be described in detail later. Then, the instruction determined by the instruction determination unit 35 is sent to the communication unit 36.

[0060] The communication unit 36 (reception unit) exchanges information between the mobile imaging device 10 and the moving subject 20. Specifically, the communication unit 36 receives information from the mobile imaging device 10 and the moving subject 20, sends this information to the instruction determination unit 35, and sends an instruction related to manipulation as the processing result of the instruction determination unit 35 to the moving subject 20.

[0061] The above has described the structure and basic operation of the instruction device 30.

[0062] Figure 2 is a block diagram showing the software structure of the autonomous driving ECU 24.

[0063] In this embodiment, the autonomous driving ECU 24 includes a driving information acquisition module 201, a surrounding information acquisition module 202, a map information acquisition module 203, a position information acquisition module 204, an instruction information acquisition module 205, and a driving control module 206.

[0064] In this embodiment, the driving control signal of the moving subject 20 is determined based on the information received from the sensor group 21, the map database 22, and the communication unit 26.

[0065] The driving information acquisition module 201 acquires information related to the driving of the moving subject 20 obtained from the sensor group 21. Specific examples of the acquired information include information related to the driving characteristics and operations of the moving subject 20.

[0066] The surrounding information acquisition module 202 acquires information related to the surroundings of the moving subject 20 obtained from the sensor group 21. Specific examples of the acquired information include the determination of the presence of surrounding objects using LiDAR, and information related to the relative distance between the surrounding objects and the moving subject 20 in the case of the presence of surrounding objects.

[0067] The map information acquisition module 203 acquires information related to the map from the map database 22.

[0068] The position information acquisition module 204 acquires information related to the position of the mobile object 20 from the GPS receiver 23.

[0069] The instruction information acquisition module 205 acquires information related to the operation of the mobile object 20 from the communication unit 26. Specific examples of the acquired information include the acceleration and deceleration of the mobile object 20 and the change of the route to the destination.

[0070] Then, the travel control module 206 determines a travel control signal for the mobile object 20 based on the information acquired from the sensor group 21, the map database 22, and the communication unit 26. The travel control signal determined by the travel control module 206 is sent to the drive mechanism 25.

[0071] Here, the operation of the automatic driving ECU 24 in the imaging process in the present embodiment, and the operations of the mobile imaging device 10 and the indicating device 30 required in association with the operation of the automatic driving ECU 24 will be described.

[0072] Figure 3 is a flowchart schematically showing the imaging process in the information processing system 100 in the present embodiment.

[0073] First, in step S301, the moving object detection unit 17 of the mobile imaging device 10 acquires the image captured by the imaging device 15.

[0074] Next, in step S302, the moving object detection unit 17 determines whether the entire mobile object 20 is included in the field of view angle of the imaging device 15 based on the image acquired in step S301. As a result of the determination, when the entire mobile object 20 is included in the field of view angle (yes in step S302), the process proceeds to step S307. Otherwise, the process proceeds to step S302a.

[0075] In step S307, the moving object detection unit 17 sends the information indicating that the entire mobile object 20 is included in the field of view angle to the indicating device 30 via the communication unit 19. When receiving this information, the indication determination unit 35 of the indicating device 30 determines that the imaging desired by the user has been performed, continues the imaging, and after a predetermined time has elapsed, returns the process to step S301.

[0076] On the other hand, in step S302a, the moving subject detection unit 17 transmits, via the communication unit 19, information indicating the degree to which the moving subject 20 is included in the field of view angle (whether the moving subject 20 is partially included in the field of view angle or not at all) to the indicating device 30. When receiving this information, the indication determination unit 35 of the indicating device 30 determines that the user has not performed the desired imaging and that an instruction related to the operation needs to be issued to the moving subject 20, instructs the moving imaging device 10 to stop imaging, and then causes the process to proceed to step S303.

[0077] Here, specific examples shown in Figure 4A 、 Figure 4B and Figure 4C will be used to illustrate the determination of the controller 31 in steps S307 and S302a.

[0078] Figure 4A 、 Figure 4B and Figure 4C are diagrams showing the imaging scenes of the moving subject 20 captured by the moving imaging device 10 during flight and the images captured in each scene.

[0079] Reference numerals 401, 402, and 403 represent the positions of the moving imaging device 10 in each imaging scene at the imaging times of t1 seconds, (t1 + Δ) seconds, and (t1 + 2Δ) seconds. Reference numerals 411 and 412 represent the positions of the moving subject 20 as the imaging object in the imaging scene, and reference numerals 421, 422, and 423 represent the positions of the moving subjects other than the imaging object in the imaging scene. Reference numerals 431, 432, and 433 represent the imaging field of view angles of the imaging device 15 in each imaging scene, and reference numerals 441, 442, and 443 represent the images captured by the imaging device 15 in each imaging scene.

[0080] In the case where the captured image 441 is obtained in step S301, the moving subject detection unit 17 detects that the entire moving subject 20 at position 411 is included in the imaging field of view angle 431 and sends the detection result to the indication determination unit 35 (in step S302, it is "yes"). When receiving the detection result, the indication determination unit 35 determines that the imaging desired by the user has been performed and that no instruction related to the operation needs to be issued to the moving subject 20, and maintains the current driving control of the moving subject 20 and continues the imaging using the imaging device 15 (step S307).

[0081] On the other hand, when the captured image 442 is obtained in step S301, the moving object detection unit 17 detects that the imaging field angle 432 includes only a part of the moving object 20 at the position 412 (being "no" in step S302), and sends the detection result to the instruction determination unit 35. When receiving the detection result, the instruction determination unit 35 determines that the imaging desired by the user has not been performed and an instruction related to operation needs to be issued to the moving object 20, and instructs the imaging device 15 to stop imaging (step S302a), and the process proceeds to step S303. Step S302a corresponds to a receiving step for receiving from the mobile imaging device 10 a detection result indicating whether the entire moving object 20 is included in the imaging field angle of the mobile imaging device 10.

[0082] In addition, when the captured image 443 is obtained in step S301, the moving object detection unit 17 detects that the imaging object (moving object 20) does not exist in the imaging field angle 433 (being "no" in step S302), and sends the detection result to the instruction determination unit 35. When receiving the detection result, the instruction determination unit 35 determines that the imaging desired by the user has not been performed and an instruction related to operation needs to be issued to the moving object 20, instructs the imaging device 15 to stop imaging (step S302a), and the process proceeds to step S303.

[0083] Return reference Figure 3 , in step S303, the instruction determination unit 35 obtains the travel information of the moving object 20 from the moving object 20. The travel information obtained here includes travel characteristics and operation-related information obtained from the travel information obtaining module 201 of the moving object 20, and position-related information obtained from the position information obtaining module 204 of the moving object 20, etc. This information is sent from the communication unit 26 of the moving object 20 to the communication unit 36 of the instruction device 30.

[0084] Next, in step S304, the instruction determination unit 35 obtains information related to the imaging and flight of the mobile imaging device 10 from the mobile imaging device 10. The information related to the imaging and flight obtained here includes information related to altitude and acceleration obtained by the movement controller 14, information related to the imaging position estimated by the position and posture estimation unit 16, and information related to the moving speed, moving direction, and future moving route. This information is sent from the communication unit 19 of the mobile imaging device 10 to the communication unit 36 of the instruction device 30.

[0085] Next, in step S305, the instruction determination unit 35 determines an instruction related to the operation of the moving subject 20 based on the travel information related to the moving subject 20 obtained in step S303 and the information related to the imaging and flight of the mobile imaging device 10 obtained in step S304. Specifically, the instruction determination unit 35 determines, based on the obtained information, an operation instruction required to place the entire moving subject 20 within the imaging field angle of the imaging device 15, and transmits the operation instruction to the moving subject 20 via the communication unit 36. Step S305 corresponds to an instruction step for giving an instruction related to the operation of at least one of the moving subject 20 and the mobile imaging device 10 according to the detection result.

[0086] Now, the examples shown in Figure 4A 、 Figure 4B and Figure 4C will be used to illustrate the determination of the instruction related to the operation of the moving subject 20 using the instruction determination unit 35 in step S305.

[0087] In a case where a part of the moving subject 20 is included in the imaging field angle 432 as in the captured image 442, the instruction determination unit 35 first calculates the speed difference between the mobile imaging device 10 and the moving subject 20 based on the information obtained in steps S303 and S304. Next, the instruction determination unit 35 calculates the distance in the horizontal direction between the mobile imaging device 10 and the moving subject 20 based on the position information related to the mobile imaging device 10 and the moving subject 20 obtained in steps S303 and S304. When the calculated distance falls within the first threshold, the instruction determination unit 35 determines that the entire moving subject 20 can be included in the imaging field angle by decelerating the moving subject 20, and calculates the deceleration amount of the moving subject 20 based on the calculated speed difference and distance. Thereafter, the instruction determination unit 35 transmits an operation instruction for decelerating the moving subject 20 together with the calculated deceleration amount to the moving subject 20.

[0088] On the other hand, in a case where the imaging object (the moving subject 20) does not exist within the imaging field angle 433 as in the captured image 443, the instruction determination unit 35 calculates the distance in the horizontal direction between the moving imaging device 10 and the moving subject 20 based on the position information obtained in steps S303 and S304. In a case where the calculated distance is equal to or greater than a second threshold that is larger than the first threshold, the instruction determination unit 35 determines that even if the moving subject 20 decelerates, the entire moving subject 20 cannot be included within the imaging field angle, and sends a manipulation instruction for changing the route to the destination of the moving subject 20 to the moving subject 20. According to this manipulation instruction, in step S306, which will be described later, the autonomous driving ECU 24 of the moving subject 20 determines a new route and speed. That is, a predetermined meeting point with the moving imaging device 10 is determined based on the current driving plan of the moving subject 20, and the route and speed are determined such that the moving subject 20 will reach the meeting point at the time when the moving imaging device 10 is expected to reach the meeting point. Reference will be made to Figure 5A 、 Figure 5B and Figure 5C to describe the specific processing.

[0089] Figure 5A Shows the driving plan 510 from the driving start position 500 to the destination 501 planned at the start of driving. In addition, Figure 5B shows the position 521 of the moving subject 20 when the moving imaging device 10 loses track of the moving subject 20. Figure 5C shows the determined predetermined meeting point 531 and the driving plan 530 after the route change.

[0090] First, when the user sets the destination 501 of the moving subject 20 at the driving start position 500, the autonomous driving ECU 24 determines a driving plan from the driving start position 500 to the destination 501 (the route indicated by the dotted line in the driving plan 510). After that, the moving subject 20 moves along the determined driving plan according to the manipulation instruction using the autonomous driving ECU 24.

[0091] During the movement, when the moving subject 20 at the position 521 deviates from the imaging field angle of the moving imaging device 10 at the imaging time (t1 + 2Δ) seconds, the instruction determination unit 35 calculates the distance in the horizontal direction between the moving imaging device 10 and the moving subject 20. In a case where the calculated distance is equal to or greater than the second threshold, the instruction determination unit 35 determines that it is necessary to determine a predetermined meeting point between the moving imaging device 10 and the moving subject 20 and change the driving plan of the moving subject 20.

[0092] Specifically, the indication determination unit 35 now defines a predetermined convergence point 531, determines a new travel plan (the route shown by the dotted line in the travel plan 530) of the moving subject 20 starting from the position 521, and causes the moving subject 20 to move along this new travel plan starting from the position 521. On the other hand, the moving imaging device 10 is caused to move along the route of the original travel plan. Thus, the moving subject 20 and the moving imaging device 10 can converge at the predetermined convergence point 531.

[0093] Return reference Figure 3 , in the next step S306, the automatic driving ECU 24 performs the following processing. First, the indication information acquisition module 205 acquires the information related to the operation instruction transmitted from the indication device 30 in step S305. Next, when the travel control module 206 acquires the information related to the operation instruction from the indication information acquisition module 205, the travel control module 206 changes the automatic driving control content in consideration of the information acquired from the travel information acquisition module 201, the surrounding information acquisition module 202, the map information acquisition module 203, and the position information acquisition module 204. When the travel control module 206 transmits the changed automatic driving control content to the drive mechanism 25, the drive mechanism 25 changes the travel control of the moving subject 20 according to this automatic driving control content. After that, the process proceeds to step S308.

[0094] For example, there is a case where the indication information acquisition module 205 acquires the information related to the operation instruction to decelerate the moving subject 20, and the surrounding information acquisition module 202 acquires the information indicating that there is no other moving subject in the surrounding or the relative distance from other moving subjects is long. In this case, the travel control module 206 determines that the speed of the moving subject 20 can be reduced, and determines the deceleration amount within the range not violating the traffic rules acquired from the map information acquisition module 203 based on the current speed information acquired from the travel information acquisition module 201. After that, the travel control module 206 transmits the determined deceleration amount as a travel control signal (automatic driving control content) to the drive mechanism 25, and notifies the indication device 30 that the changed travel control has started. When receiving the travel control signal, the drive mechanism 25 controls the travel of the moving subject 20. The travel control module 206 can notify the indication device 30 of the determined deceleration amount of the moving subject 20.

[0095] For example, when the instruction information acquisition unit 205 acquires information related to an operation instruction for changing the route to the destination, the travel control unit 206 determines that it is necessary to change the travel plan to the destination during the currently executing autonomous driving. In this case, the travel control module 206 re-searches the route to the destination based on the map information obtained from the map information acquisition module 203 and the current position information of the moving object 20 obtained from the position information acquisition module 204. In this re-search, a detour route for the moving object 20 is searched, and when the mobile imaging device 10 moves along the travel plan before the change, the detour route enables the mobile imaging device 10 to catch up with the moving object 20 at a predetermined point. Then, the travel control module 206 changes the travel plan to the searched detour route and determines operation instructions such as a steering wheel operation for traveling along the changed route. After that, the travel control module 206 sends the determined operation instructions as a travel control signal (autonomous driving control content) to the drive mechanism 25 and notifies the indicating device 30 that the changed travel control has started. When receiving the travel control signal, the drive mechanism 25 controls the travel of the moving object 20.

[0096] Next, when receiving the notification from the travel control module 206, the instruction determination unit 35 determines that the moving object 20 will be included in the imaging field angle of the mobile imaging device 10, instructs the mobile imaging device 10 to resume imaging in step S308, and then returns the process to step S301.

[0097] In the information processing system 100 according to the present embodiment, the indicating device 30 is described as a device different from the mobile imaging device 10 and the moving object 20, but it may be included in the mobile imaging device 10. Specifically, the indicating device 30 can be excluded from the information processing system 100, and instead, the instruction determination unit can be incorporated into the mobile imaging device 10. In this case, the instruction determination unit obtains the travel characteristics of the moving object 20, information related to operations, information related to positions, etc. from the moving object 20, and determines an instruction related to the operation of the moving object 20. Then, the mobile imaging device 10 sends the instruction related to the determined operation to the moving object 20 via the communication unit 19 of the mobile imaging device 10.

[0098] The indication determination unit may also be included in the moving object 20. Specifically, the indication device 30 may be excluded from the information processing system 100, and instead, the indication determination unit may be incorporated into the moving object 20. In this case, the indication determination unit obtains information related to imaging and flight from the mobile imaging device 10, and determines an operation instruction for the moving object 20. Then, the determined operation instruction is sent to the travel control module 206 via the indication information acquisition module 205.

[0099] Although the information processing system 100 according to the present embodiment controls the travel of the moving object 20 such that the moving object 20 is included in the imaging field angle of the imaging device 15 (frame-in), the travel of the moving object 20 may be controlled according to a predetermined imaging scenario. Specifically, the imaging scenario describes the imaging conditions of the mobile imaging device 10 and the position and posture conditions for obtaining a desired imaging clip determined based on written instructions related to imaging and editing regarding the composition and production of the moving image. If, during imaging by the imaging device 15 according to the imaging scenario, imaging according to the imaging scenario cannot be performed due to the movement of the moving object 20, the travel of the moving object 20 is controlled to perform the desired imaging.

[0100] For example, there is a case where, during imaging by the imaging device 15 according to an imaging scenario for obtaining an imaging clip in which the mobile imaging device 10 and the moving object 20 run in parallel, imaging according to the scenario cannot be performed, that is, the moving object 20 deviates from the imaging field angle ( "no" in step S302). In this case, the stop of imaging is indicated (step S302a), and the processing from step S303 is executed. Specifically, first, the relative speed in the horizontal direction between the mobile imaging device 10 and the moving object 20 is calculated based on the imaging position and moving speed obtained from the mobile imaging device 10 and the travel position of the moving object 20. Next, the travel speed of the moving object 20 is determined based on the calculated relative speed. Then, imaging according to the imaging scenario can be performed by controlling the travel of the moving object 20 to the determined speed.

[0101] As other examples, there are the following situations: During the period when the imaging device 15 performs imaging according to an imaging scenario for obtaining an imaging clip that causes the moving subject 20 to frame out of the field of view angle of the imaging device 15 to create a seam for editing, imaging according to this scenario cannot be performed (it is "No" in step S302). That is, the moving subject 20 does not frame out of the field of view angle. In this case, the stop of imaging is indicated (step S302a), and the processing from step S303 onwards is executed. Specifically, first, an acceleration amount for causing the moving subject 20 detected as being included in the field of view angle of the imaging device 15 to frame out is calculated based on the imaging position and moving speed obtained from the moving imaging device 10 and the traveling position of the moving subject 20. Then, imaging according to the imaging scenario can be performed by controlling the traveling of the moving subject 20 to the calculated acceleration amount.

[0102] Although the information processing system 100 in this embodiment has been described with respect to the instructions related to the manipulation of the moving subject 20 (control instructions related to traveling), these instructions may include other operation control instructions for the moving subject 20. Here, the other operation control instructions include lighting control instructions for the lights (such as turn signals and headlights, etc.) included in the moving subject 20, operation control instructions for the movable mechanism (such as opening and closing instructions for windows and driving instructions for windshield wipers, etc.). For example, in the case of shooting the flashing of the turn signal when turning at an intersection, it is necessary to make the turn signal of the moving subject 20 flash when the light of the turn signal of the moving subject 20 is included in the field of view angle of the imaging device 15. In this case, based on the imaging position, imaging direction, and captured image obtained from the moving imaging device 10, the timing immediately before the moving subject 20 reaches the intersection where the moving subject 20 plans to turn and whether the light of the turn signal of the moving subject 20 is included in the field of view angle of the imaging device 15 are judged. In the case where the light of the turn signal is included in the field of view angle at this timing, the moving subject 20 is controlled to make the light of the turn signal flash. Thus, a desired imaging clip of the turn signal of the moving subject 20 flashing when turning at an intersection can be obtained.

[0103] Hereinafter, the information processing system 100-2 according to the second embodiment of the present invention will be described. In this embodiment, components that are the same as those of the information processing system 100 according to the first embodiment are denoted by the same reference numerals, and repeated descriptions will be omitted.

[0104] The information processing system 100-2 of the second embodiment is different from the information processing system of the first embodiment in that it includes a manually operated moving subject 40 operated manually by an operator (user) through a steering wheel, a brake pedal, an accelerator pedal, etc. (hereinafter referred to as the operation unit 47), instead of the moving subject 20.

[0105] Figure 6 is a block diagram showing the hardware configuration of the information processing system 100-2 according to the present embodiment. In Figure 6 the manual moving object 40, the block that executes the same process as the block in Figure 1 the moving object 20 is represented by the same reference numeral with a suffix "a".

[0106] Compared with Figure 1 the moving object 20, the manual moving object 40 is different in that it has a notification information generation ECU 44 and a display unit 45. Therefore, the processing contents of the notification information generation ECU 44 and the display unit 45, which are features in the present embodiment, will be described in detail below.

[0107] The notification information generation ECU 44 (notification unit) is an electronic controller including a CPU, a ROM, a RAM, a CAN communication circuit, etc., and displays a signal according to the notification information on the display unit 45 by controlling the hardware based on the signal output by the CPU. As an example of a specific operation, the CAN communication circuit is operated to store the data obtained from the sensor group 21a, the map database 22a, and the communication unit 26a as input data in the RAM. After that, the notification information generation ECU 44 determines the notification information based on the input data by running the program loaded from the ROM to the RAM, and sends a signal according to the determined notification information to the display unit 45. The details of this processing will be described later.

[0108] The display unit 45 is built into the manual moving object 40 and displays a signal from the notification information generation ECU 44 to inform the operator. Here, the display unit 45 can use a known display (such as a liquid crystal display, a plasma display, an organic EL display, etc.) as the display screen.

[0109] Figure 7 is a block diagram showing the software configuration of the notification information generation ECU 44.

[0110] In Figure 7 the block that executes the same process as the block in the software configuration of the automatic driving ECU 24 in Figure 2 is represented by the same reference numeral with a suffix "a". Figure 7 The software configuration in Figure 2 is different from the software configuration in

[0111] In the notification information determination module 706, information related to notification to the operator manually moving the subject 40 is determined based on the information obtained from the sensor group 21a, the map database 22a, and the communication unit 26a. The signal related to the notification determined in the notification information determination module 706 is sent to the display unit 45. This process will be described in detail later.

[0112] Here, the operation of the notification information determination module 706 in the imaging process of the present embodiment, and the operations of the mobile imaging device 10 and the indicating device 30 required in association with the operation of the notification information determination module 706 will be described.

[0113] Figure 8 is a flowchart schematically showing the imaging process in the information processing system 100-2 of the present embodiment. In Figure 8 the same step numbers are assigned to the steps for performing the same processing as in the Figure 3 flowchart. Figure 8 The difference between the flowchart in Figure 3 and the flowchart in

[0114] is that instead of the processing in step S306, the processing in step S806 is performed. The content of the processing of step S806, which is a feature of the present embodiment, will be described in detail below. Figure 8 In step S806, the notification information generation ECU 44 performs the following processing. First, the instruction information acquisition module 205a acquires the information of the operation instruction sent from the indicating device 30 in

[0115] For example, when the instruction information acquisition module 205a acquires information related to an operation instruction for decelerating the manually moved subject 40, the following situation exists: the surrounding information acquisition module 202a acquires surrounding information indicating that there are no other moving subjects in the surrounding area or the relative distance from other moving subjects is long. In this case, the notification information determination module 706 determines that the speed can be reduced, and determines the deceleration amount within the range not violating the traffic rules acquired from the map information acquisition module 203a based on the current speed information acquired from the travel information acquisition module 201a. After that, the notification information determination module 706 sends the determined deceleration amount as notification information to the display unit 45, and notifies the indicating device 30 that the changed travel control has started. When receiving the notification information, the display unit 45 displays the notification information.

[0116] For example, in the case where the instruction information acquisition unit 205a acquires information related to an operation instruction for changing the route to the destination, the notification information determination module 706 determines that the currently set travel plan to the destination needs to be changed. In this case, the notification information determination module 706 re-searches the route to the destination based on the map information acquired from the map information acquisition module 203a and the current position information of the manually moved subject 40 acquired from the position information acquisition module 204a. In this re-search, a detour route of the manually moved subject 40 is searched, and when the mobile imaging device 10 moves along the travel plan before the change, the mobile imaging device 10 can catch up with the manually moved subject 40 at a predetermined point. Then, the notification information determination module 706 changes the travel plan to the searched detour route, sends the changed route as notification information to the display unit 45, and notifies the indicating device 30 that the changed travel control has started. When receiving the notification information, the display unit 45 displays the notification information.

[0117] Hereinafter, an information processing system according to a third embodiment of the present invention will be described. In this embodiment, components that are the same as those of the information processing system 100 according to the first embodiment are denoted by the same reference numerals, and repeated descriptions will be omitted.

[0118] The difference between the information processing system in the third embodiment and the information processing system in the first embodiment is that, in addition to controlling the movement of the moving subject 20, the movement of the mobile imaging device 10 is also controlled. More specifically, this embodiment is greatly different from the first embodiment in that the instruction determination unit 35 in the indicating device 30 determines an instruction related to the operation of the mobile imaging device 10, and the determined instruction is sent from the communication unit 36 to the communication unit 19 of the mobile imaging device 10.

[0119] Figure 92 is a block diagram showing a software structure of the mobile controller 14 of the mobile imaging device 10 according to the present embodiment.

[0120] The mobile controller 14 in this embodiment includes a flight status identification module 901 , a position information acquisition module 902 , an indication information acquisition module 903 and a mobile control determination module 904 .

[0121] In order to recognize the flight state of the mobile camera 10 , the flight state recognition module 901 detects information such as altitude and acceleration and objects around the mobile camera 10 using the sensor group 14 c (atmospheric pressure sensor, acceleration sensor, LiDAR, etc.).

[0122] The position information obtaining module 902 obtains the information related to imaging detected by the position and posture estimation unit 16. The information related to imaging includes the imaging position, movement speed, movement direction, and future movement route of the mobile imaging device 10.

[0123] The instruction information obtaining module 903 obtains information related to the manipulation of the mobile camera apparatus 10 from the communication unit 19. The information related to the manipulation of the mobile camera apparatus 10 includes acceleration, deceleration, and change of the route to the destination of the mobile camera apparatus 10.

[0124] Then, the movement control determination module 904 determines the movement control information for the mobile camera device 10 based on the information obtained from the sensor group 14c, the database (not shown) maintained in the ROM 12, and the communication unit 19. Based on the determined movement control information, the movement control determination module 904 controls the driving force of the four motors constituting the motor group 14a to control the rpm of the four propellers constituting the propeller group 14b, thereby controlling the movement direction, position, and posture of the mobile camera device 10.

[0125] Here, the operation of the movement control determination module 904 in the image capturing process of the present embodiment, and the operations of the mobile image capturing apparatus 10 and the pointing device 30 required in association with the operation of the movement control determination module 904 will be described.

[0126] Figure 10 FIG. 4 is a flowchart schematically showing the image pickup process in the information processing system of this embodiment. Figure 10 , assign the same step number to the steps used to perform the Figure 3 The steps of the same process are shown in the flowchart. Figure 10 The flowchart in Figure 3 The flowchart in FIG. 1 is different in that the processing in steps S1005 and S1006 is performed instead of the processing in steps S305 and S306. The processing contents of steps S1005 and S1006, which are the characteristics of the present embodiment, will be described in detail below.

[0127] In step S1005, the instruction determination unit 35 of the instruction device 30 determines one or more instructions related to operation based on the travel information of the moving subject 20 obtained in step S303 and the information related to imaging and flight of the mobile imaging device 10 obtained in step S304. Specifically, the instruction device 30 determines one or more operation instructions required to place the entire moving subject 20 within the imaging field angle of the imaging device 15 based on the obtained information, and sends the one or more operation instructions to at least one of the moving subject 20 and the mobile imaging device 10.

[0128] Here, Figure 4A 、 Figure 4B and Figure 4C a specific example shown will be used to illustrate the determination of the operation-related instruction using the instruction determination unit 35 in step S1005.

[0129] In the case where a part of the moving subject 20 is included in the imaging field angle 432 as in the captured image 442, the instruction determination unit 35 first calculates the speed difference between the mobile imaging device 10 and the moving subject 20 based on the information obtained in steps S303 and S304. Next, the instruction determination unit 35 calculates the distance in the horizontal direction between the mobile imaging device 10 and the moving subject 20 based on the position information of the mobile imaging device 10 and the moving subject 20 obtained in steps S303 and S304. When the calculated distance falls within the first threshold, the instruction determination unit 35 determines that the entire moving subject 20 can be included in the imaging field angle by reducing the relative speed between the moving subject 20 and the mobile imaging device 10. In addition, since there are other moving subjects that are not the imaging object at the position 422 in the vicinity, it is difficult to decelerate the moving subject 20. Therefore, the instruction determination unit 35 determines that the relative speed can be reduced by accelerating the mobile imaging device 10. Based on this determination, the instruction determination unit 35 calculates the acceleration amount of the mobile imaging device 10 based on the calculated distance and the speed difference. Then, the instruction determination unit 35 sends the operation instruction to accelerate the mobile imaging device 10 together with the calculated acceleration amount to the mobile imaging device 10.

[0130] On the other hand, in a case where the imaging object (the moving subject 20) does not exist in the imaging field angle 433 as in the captured image 443, the instruction determination unit 35 calculates the distance in the horizontal direction between the moving imaging device 10 and the moving subject 20 based on the respective position information obtained in steps S303 and S304. When the calculated distance is equal to or greater than a second threshold larger than the first threshold, the instruction determination unit 35 determines that even if the speed of the moving subject 20 is reduced, the entire moving subject 20 cannot be included in the imaging field angle. Further, when the calculated distance is equal to or greater than a third threshold larger than the second threshold, the instruction determination unit 35 determines that even if only the route of the moving subject 20 is changed, the entire moving subject 20 cannot be included in the imaging field angle. Based on this determination, the instruction determination unit 35 sends a manipulation instruction for changing the respective routes to the destination to both the moving subject 20 and the moving imaging device 10. According to the manipulation instruction, in step S1006 described later, the automatic driving ECU 24 of the moving subject 20 and the movement controller 14 of the moving imaging device 10 determine new routes and speeds. That is, based on the current travel plan of the moving subject 20, a predetermined meeting point with the moving imaging device 10 is determined, and the route and speed are determined such that the moving subject 20 also reaches the meeting point at the time when the moving imaging device 10 is expected to reach the meeting point. Reference will be made to Figure 11A , Figure 11B and Figure 11C to describe the specific processing.

[0131] Figure 11A shows the travel plan 1110 from the travel start position 1100 to the destination 1101 planned at the start of travel. Further, Figure 11B shows the position 1121 of the moving subject 20 and the position 1122 of the moving imaging device 10 when the moving imaging device 10 loses track of the moving subject 20. Further, Figure 11C shows the determined predetermined meeting point 1131 and the travel plan 1130 after the route change.

[0132] First, when the user sets the destination 1101 of the moving subject 20 at the travel start position 1100, the automatic driving ECU 24 determines a travel plan from the travel start position 1100 to the destination 1101 (the route indicated by the dotted line in the travel plan 1110). Thereafter, the moving subject 20 moves along the determined travel plan according to the manipulation instruction using the automatic driving ECU 24.

[0133] During movement, when the moving imaging device 10 at position 1122 can no longer detect the moving subject 20 at position 1121, the indication determination unit 35 calculates the distance in the horizontal direction between the moving imaging device 10 and the moving subject 20. When the calculated distance here is equal to or greater than the third threshold, the indication determination unit 35 determines a predetermined meeting point between the moving imaging device 10 and the moving subject 20, and determines that it is necessary to change the travel plans of the moving subject 20 and the moving imaging device 10.

[0134] Specifically, the indication determination unit 35 defines a predetermined meeting point 1131, determines a new travel plan (the route indicated by the dashed line in travel plan 1130) of the moving subject 20 starting from position 1121, and causes the moving subject 20 to move along this new travel plan starting from position 1121. On the other hand, when the moving imaging device 10 moves along the route of the original flight plan, the moving imaging device 10 cannot meet the moving subject 20 at the predetermined meeting point 1131. Therefore, the indication determination unit 35 determines a new travel plan (the route indicated by the alternately long and short dashed line in travel plan 1130) starting from the position 1122 of the moving imaging device 10, and causes the moving imaging device 10 to move along this new travel plan starting from position 1122. Thus, the moving subject 20 and the moving imaging device 10 can meet at the predetermined meeting point 1131.

[0135] Return reference Figure 10 , in the next step S1006, the automatic travel ECU 24 of the moving imaging device 10 performs processing similar to the processing in step S306. On the other hand, the movement controller 14 of the moving imaging device 10 performs the following processing.

[0136] First, the indication information acquisition module 903 acquires information related to the manipulation instruction sent from the indication device 30 in step S1005. Then, when acquiring information related to the manipulation instruction from the indication information acquisition module 903, the movement control determination module 904 changes the flight control content in consideration of the information acquired from the flight state recognition module 901 and the position information acquisition module 902. After that, the movement control determination module 904 controls the driving of the motor group 14a and the thruster group 14b according to the changed flight control content.

[0137] For example, there is the following situation: when the instruction information acquisition module 903 acquires information related to a manipulation instruction for accelerating the mobile imaging device 10, the flight state recognition module 901 acquires information indicating that there is no other moving subject in the vicinity or that the relative distance from other moving subjects is long. In this case, the movement control determination module 904 determines that the speed of the mobile imaging device 10 can be increased, and determines the acceleration amount based on the current speed information acquired from the position information acquisition module 902. After that, the movement control determination module 904 controls the driving of the motor group 14a and the thruster group 14b based on the determined acceleration amount, and notifies the indicating device 30 that the changed flight control has started. The movement control determination module 904 may notify the indicating device 30 of the determined acceleration amount of the mobile imaging device 10 together.

[0138] For example, in the case where the instruction information acquisition module 903 acquires information related to a manipulation instruction for changing the route to the destination, the movement control determination module 904 determines that the current flight plan to the destination needs to be changed. In this case, the movement control determination module 904 re-searches the route to the destination based on the current position information of the mobile imaging device 10 acquired from the position information acquisition module 902. In this re-search, the shortest route to the convergence point with the moving subject 20 is searched. After that, the movement control determination module 904 changes the flight plan to the searched shortest route, controls the driving of the motor group 14a and the thruster group 14b to move along the changed route, and notifies the indicating device 30 that the changed flight control has started.

[0139] In the information processing system according to the present embodiment, since there is a moving subject other than the imaging object within the close proximity of the moving subject 20, it is determined in step S1005 that it is difficult to decelerate the moving subject 20, and the mobile imaging device 10 is controlled to accelerate. However, other control methods can be considered in the case where the manipulation control of the moving subject 20 or the mobile imaging device 10 is restricted. Which control method is prioritized when considering multiple control methods is determined based on which control method resumes imaging earlier, or whether the control of the mobile imaging device 10 or the moving subject 20 is prohibited, and the like. For example, in the case of the moving subject 20, the following are considered: whether the traveling speed falls within the range where safe manipulation can be performed, whether there is a speed limit at the traveling position, and whether the change in traveling control violates traffic rules, and the like. In the case of the mobile imaging device 10, the following are considered: whether the changed flight speed is an object of flight restriction, whether there are no other subjects in the vicinity, and whether the changed flight speed exceeds the speed limit of the mobile imaging device 10, and the like. In the case where such restrictions are imposed, it is necessary to control the movement by assigning priorities to the control methods.

[0140] On the other hand, in the case where imaging according to the imaging scenario cannot be performed during imaging according to the imaging scenario (No in step S302), in step S1005, the instruction device 30 determines an instruction related to the operation of at least one of the moving subject 20 and the moving imaging device 10. However, there are cases where the operation control of at least one of the moving imaging device 10 and the moving subject 20 is restricted and cannot be performed as instructed in step S1006. Therefore, imaging according to the imaging scenario cannot be performed even after the instruction related to the operation. In this case, the instruction device 30 can change the imaging scenario and re-determine an instruction related to the operation of at least one of the moving subject 20 and the moving imaging device 10 based on the changed imaging scenario. Specifically, the direction of shooting the moving subject 20 can be changed. Alternatively, the focal length of the optical system of the imaging device 15 can be changed by changing the imaging conditions of the moving imaging device 10. In such a case, the captured image can be cropped in subsequent editing. It is determined based on the conditions (acceleration amount, deceleration amount, etc.) of the moving imaging device 10 and the moving subject 20 obtained in step S1006 whether imaging according to the imaging scenario can be performed even after the instruction related to the operation.

[0141] After changing the imaging scenario, the instruction device 30 (instruction determination unit 35) obtains the current states of the moving imaging device 10 and the moving subject 20 as current information. After that, in the case where the instruction device 30 determines based on the obtained current information that imaging according to the changed imaging scenario can be performed, the instruction device 30 causes the process to proceed to step S308, and the instruction determination unit 35 instructs the moving imaging device 10 to resume imaging. This shortens the time period until imaging in the moving imaging device 10 resumes.

[0142] Other embodiments

[0143] Embodiments of the present invention can also be implemented by the following method, that is, software (program) that executes the functions of the above-described embodiments is provided to a system or device through a network or various storage media, and a method in which a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0144] Although the present invention has been described with reference to exemplary embodiments, it should be understood that the present invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims is to be accorded the broadest interpretation so as to encompass all such modifications, equivalent structures and functions.

[0145] This application claims the benefit of Japanese Patent Application No. 2023-200673, filed on November 28, 2023, which is hereby incorporated herein by reference in its entirety.

Claims

1. An information processing system, comprising: Moving subjects; A mobile camera device, used for capturing an image of the moving subject; a receiving unit configured to receive, from the mobile camera device, a detection result indicating whether the mobile object is included in the camera field of view of the mobile camera device; as well as An instruction unit is configured to issue an instruction related to the manipulation of at least one of the moving object and the mobile imaging device according to the detection result.

2. The information processing system according to claim 1, wherein: The instruction unit issues an instruction related to manipulation of at least one of the moving object and the mobile imaging device according to the detection result so that the entire moving object will be included in the imaging field angle of the mobile imaging device.

3. The information processing system according to claim 1, wherein: The instruction unit issues an instruction related to manipulation of at least one of the moving object and the mobile imaging device according to the detection result, so that the moving object is out of the imaging field of the mobile imaging device.

4. The information processing system according to any one of claims 1 to 3, further comprising a controller configured to control the moving object, in, The controller controls movement of the moving object based on an instruction related to manipulation of the moving object.

5. The information processing system according to any one of claims 1 to 3, further comprising a notification unit configured to notify an operator of the moving object, in, The notification unit notifies the operator of information related to an instruction related to the manipulation of the moving object.

6. The information processing system according to any one of claims 1 to 3, wherein: The instruction unit is incorporated in the mobile imaging device, and the mobile imaging device issues an instruction related to manipulation to the mobile subject.

7. The information processing system according to any one of claims 1 to 3, wherein: The instruction unit is incorporated in the moving subject, and the moving subject issues an instruction related to manipulation to the mobile imaging device.

8. The information processing system according to any one of claims 1 to 3, wherein: The instruction related to the manipulation of the moving object is an instruction to change the speed of the moving object.

9. The information processing system according to any one of claims 1 to 3, wherein: The instruction related to the manipulation of the mobile object is an instruction to change the route of the mobile object.

10. The information processing system according to any one of claims 1 to 3, wherein: The instruction related to the manipulation of the mobile camera device is an instruction to change the route of the mobile camera device.

11. The information processing system according to any one of claims 1 to 3, wherein: The instructions related to the manipulation of the moving object include a lighting control instruction for a light of the moving object and an operation control instruction for a movable mechanism.

12. The information processing system according to any one of claims 1 to 3, wherein: When the mobile camera device is performing video recording according to a video recording scenario and it is determined that video recording according to the video recording scenario cannot be performed, the instruction unit issues instructions related to the operation, wherein the video recording scenario describes the video recording conditions and position and posture conditions of the mobile camera device for obtaining the desired video recording clip.

13. The information processing system according to any one of claims 1 to 3, wherein: When the mobile camera device is performing video recording according to a video recording scenario, even after an instruction related to the manipulation of at least one of the mobile subject and the mobile camera device is issued, video recording according to the video recording scenario cannot be performed because the manipulation control of at least one of the mobile subject and the mobile camera device is restricted. The instruction unit changes the video recording scenario and redetermines the instruction related to the manipulation of at least one of the mobile subject and the mobile camera device based on the changed video recording scenario, wherein the video recording scenario describes the video recording conditions and the position and posture conditions of the mobile camera device for obtaining the desired video clip.

14. The information processing system according to claim 13, wherein: When imaging according to the imaging scenario cannot be performed, the instruction unit instructs the mobile imaging device to stop imaging.

15. The information processing system according to claim 14, wherein: The instruction unit obtains the current status of the mobile camera device and the mobile subject as current information after the camera scene changes, and when it is determined based on the current information that camera shooting according to the changed camera scene is possible, the instruction unit instructs the mobile camera device to restart camera shooting.

16. An information processing device communicatively connected to a moving subject and a mobile camera device for capturing an image of the moving subject, the information processing device comprising: a receiving unit configured to receive, from the mobile camera device, a detection result indicating whether the mobile object is included in the camera field of view of the mobile camera device; as well as An instruction unit is configured to issue an instruction related to the manipulation of at least one of the moving object and the mobile imaging device according to the detection result.

17. A control method for an information processing device, the information processing device being communicatively connected to a moving subject and a mobile camera device for capturing an image of the moving subject, the control method comprising: A receiving step, for receiving, from the mobile camera device, a detection result indicating whether the mobile object is included in the camera field of view of the mobile camera device; as well as An indication step is used to issue an indication related to the manipulation of at least one of the moving object and the mobile camera device according to the detection result. 18 . A computer program product comprising a program for causing a computer to execute the control method of the information processing apparatus according to claim 17 .

19. A computer-readable storage medium storing a program for causing a computer to execute the control method of the information processing device according to claim 17.

Images