Information processing apparatus and system, control method, program product, and storage medium

Abstract

The invention discloses an information processing apparatus and system, a control method, a program product, and a storage medium. The information processing apparatus enables a mobile imaging apparatus to perform imaging while immediately responding to the state of a freely moving subject, and improves the degree of freedom of imaging. The information processing apparatus controls a mobile imaging apparatus equipped with an imaging apparatus. The information processing apparatus includes: a manipulation instruction information obtaining unit configured to obtain manipulation instruction information related to a moving subject accompanying manipulation; an imaging control unit configured to control imaging of the imaging apparatus; and a position and posture control unit configured to control a position and a posture of the mobile image pickup apparatus by driving the mobile image pickup apparatus based on the manipulation instruction information.

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing device, a control method, a computer program product, and a storage medium for controlling the movement and imaging of a mobile imaging device. Background Art

[0002] Conventionally, there have been known methods in which a mobile imaging device, such as a flying subject equipped with an imaging device, automatically performs imaging while flying along a predetermined flight route (for example, see Japanese Unexamined Patent Application Publication No. 2021-12721 (JP 2021-12721A)). In addition, there have also been known methods in which such a mobile imaging device captures an image of a freely moving subject and analyzes the captured image for tracking imaging (for example, see Japanese Unexamined Patent Application Publication No. 2018-78371 (JP 2018-78371A) (U.S. counterpart US20180131856 A1)).

[0003] However, the conventional method of performing imaging while flying along a predetermined flight route described in JP 2021-12721A has a problem that it is difficult for the mobile imaging device to capture a freely moving subject at a desired field of view angle.

[0004] In addition, the conventional method of tracking and capturing an image of a subject by analyzing an image of a freely moving subject described in JP 2018-78371A has a problem that since it takes time to analyze the captured image, the tracking of the subject by the mobile imaging device is delayed and imaging cannot be performed at a desired field of view angle. On the other hand, in order to solve this problem, when the movement of the subject is somewhat predetermined and the mobile imaging device performs imaging according to the predetermined movement of the subject, there arises a problem of reduced freedom of imaging. Summary of the Invention

[0005] The present invention provides an information processing system, an information processing device, a control method thereof, a computer program product, and a storage medium that enable a mobile imaging device to perform imaging while immediately responding to a freely moving subject and improve the freedom of imaging.

[0006] Therefore, an aspect of the present invention provides an information processing device for controlling a mobile imaging device equipped with an imaging device. The information processing device includes: a manipulation instruction information acquisition unit configured to acquire manipulation instruction information related to a moving subject accompanied by manipulation; an imaging control unit configured to control imaging of the imaging device; and a position and posture control unit configured to control the position and posture of the mobile imaging device by driving the mobile imaging device based on the manipulation instruction information.

[0007] According to the present invention, it is possible to perform imaging by moving an imaging device while immediately responding to the state of a freely moving subject, and the degree of freedom of imaging can be improved.

[0008] Other features of the present invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram showing an example of a hardware configuration of a mobile imaging device including a controller as an information processing device according to an embodiment of the present invention and a moving subject as an imaging target of the mobile imaging device.

[0010] Figure 2 is a flowchart showing the operation processing of the mobile imaging device.

[0011] Figure 3 is a flowchart showing the operation processing of the moving subject.

[0012] Figure 4 is a diagram showing a list of imaging scenarios.

[0013] Figure 5 is a diagram showing the positional relationship between the mobile imaging device and the moving subject that operate according to the imaging scenario. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0014] In the present embodiment, the position, posture, and imaging conditions of the mobile imaging device are changed according to the manipulation instruction information of the moving subject accompanying the manipulation, so as to capture a moving image desired by the user. In the present embodiment, an example in which the mobile imaging device moves by flying in the air using a thruster will be described, but the moving method is not limited thereto. For example, the mobile imaging device can move on the ground using wheels, or can move in water using a propeller.

[0015] Hereinafter, the mobile imaging device 100 including a controller 111 as an information processing device according to an embodiment of the present invention and a moving subject 150, which is an imaging target of the mobile imaging device 100 and is wirelessly connected to the mobile imaging device 100 for communication, will be described in detail with reference to the drawings.

[0016] Figure 1 is a block diagram showing an example of a hardware configuration of the mobile imaging device 100 and the moving subject 150.

[0017] As Figure 1As shown, the mobile imaging device 100 is composed of an imaging device 130 and a mobile device 110 on which the imaging device 130 is installed. The imaging device 130 and the mobile device 110 are communicably connected to each other. In addition, the mobile subject 150 is the subject of the moving image captured by the imaging device 130 and moves in response to a manipulation instruction.

[0018] First, a configuration example of the mobile device 110 included in the mobile imaging device 100 will be described with reference to Figure 1 the following.

[0019] The mobile device 110 includes a controller 111, a ROM 112, a RAM 113, a position and orientation detection unit 114, an image processor 115, a shooting scenario list storage unit 116, a moving mechanism 117, and a communication unit 118. These blocks are communicably connected to each other via a bus.

[0020] The controller 111 (imaging control unit) is, for example, a CPU that reads control programs for the blocks in the mobile device 110 and the blocks in the imaging device 130, which will be described later, from the ROM 112, expands the control programs onto the RAM 113, and runs the control programs. Therefore, the controller 111 controls the operations of the respective blocks in the mobile imaging device 100. In addition, the controller 111 (manipulation instruction information acquisition unit and position information acquisition unit) receives manipulation instruction information and position information from the mobile subject 150 via the communication unit 118.

[0021] The ROM 112 is an electrically erasable and recordable non-volatile memory, and stores parameters required for the operations of the respective blocks in addition to the operation programs for the respective blocks in the mobile imaging device 100.

[0022] The RAM 113 is a rewritable volatile memory and is used to expand the programs executed by the controller 111 and to temporarily store data generated by the operations of the blocks in the mobile imaging device 100.

[0023] The position and orientation detection unit 114 includes a plurality of sensors for detecting various information required for controlling the position and orientation of the mobile device 110 and outputting the detected position and orientation information to the controller 111. Examples of the sensors include a GPS sensor for detecting the position, a gyro sensor for detecting the angular velocity, an acceleration sensor for detecting the change in speed, a magnetic sensor for detecting the points of a compass, an atmospheric pressure sensor for detecting the altitude, and an ultrasonic sensor for detecting the distance to surrounding subjects such as the subject to be photographed. The controller 111 (position and orientation control unit) performs calculations based on the information detected by the position and orientation detection unit 114 and controls the position and orientation of the mobile device 110.

[0024] The image processor 115 performs various image processes related to the position and orientation of the mobile device 110 by analyzing the images captured by the imaging device 130 and the information detected by the position and orientation detection unit 114. For example, the image processor 115 performs image processes for identifying a subject (e.g., the moving subject 150) included in the video captured by the imaging device 130, and the controller 111 controls the moving mechanism 117 to track the subject using the identification result. Known methods are used to identify and track the subject, and a detailed description thereof will be omitted.

[0025] The imaging scene list storage unit 116 (imaging scene list acquisition and holding unit) records (holds) an imaging scene list received (acquired) via the communication unit 118 from an external information processing device (not shown). The imaging scene list may be pre-recorded in the imaging scene list storage unit 116 at the time of initial setting.

[0026] The moving mechanism 117 is constituted by a lift generation mechanism including a motor and a thruster. The controller 111 controls the position and orientation of the mobile device 110 by driving the moving mechanism 117 based on the information detected by the position and orientation detection unit 114 and the identification result of the subject by the image processor 115. This enables the tracking camera to control the field of view angle so that the subject (the moving subject 150) will be included in the field of view area captured by the imaging device 130.

[0027] The communication unit 118 communicates with the moving subject 150 based on a communication scheme defined by a standard such as Wireless LAN.

[0028] Next, a configuration example of the imaging device 130 included in the mobile imaging device 100 will be described with reference to Figure 1 an imaging device 130 included in the mobile imaging device 100 will be described.

[0029] The imaging device 130 includes an optical system 131, an imaging device 132, an A / D converter 133, an image processor 134, an image recording unit 135, and a display unit 136. These blocks are communicably connected to each other via a bus.

[0030] The optical system 131 is constituted by a diaphragm mechanism and a lens group including a zoom lens group and a focusing lens group, and forms a subject image on the imaging surface of the imaging device 132.

[0031] The imaging device 132 is an image sensor such as a CCD sensor or a CMOS sensor, for example, and performs photoelectric conversion on the optical image formed on the imaging surface of the imaging device 132 by the optical system 131, and outputs the obtained analog image signal to the A / D converter 133.

[0032] The A / D converter 133 converts the input analog image signal into digital image data and outputs the digital image data. The digital image data output from the A / D converter 133 is temporarily stored in the RAM 113 included in the mobile device 110.

[0033] The image processor 134 applies various image processes to the digital image data stored in the RAM 113. Specifically, the image processor 134 applies various image processes for developing, displaying, and recording the digital image data, such as demosaicing process, noise reduction process, white balance correction process, and gamma correction process. The image process includes a process of generating video data from the digital image data stored in the RAM 113 in time series.

[0034] The image recording unit 135 records the data including the video data generated by the image processor 134 into the built-in recording medium.

[0035] The display unit 136 includes a display panel such as an LCD panel and displays the digital image data stored in the RAM 113 and the video data recorded in the image recording unit 135 on the display panel.

[0036] Next, a configuration example of the moving object 150 will be described with reference to Figure 1 In the present embodiment, the mobile imaging device 100 captures an image of the moving object 150 as the object. Although in the present embodiment, the moving object 150 is an automobile, the moving object 150 is not limited thereto. For example, the moving object 150 may be an object that is moved by an occupant, such as a motorcycle, a ship, an airplane, or a train. The moving object 150 may be a device that is not operated by an occupant and automatically operates while recognizing the surrounding situation. The moving object 150 may be a device that is not operated by an occupant but is controlled by remote control, such as a radio-controlled car. The automatic operation and the remote operation are performed by known methods, and the detailed description thereof is omitted.

[0037] The moving object 150 includes an operation controller 151, a ROM 152, a RAM 153, a position detection unit 154, an operation mechanism 155, a drive mechanism 156, and an operation communication unit 157. These blocks are communicably connected to each other via a bus.

[0038] The operation controller 151 is, for example, a CPU that reads the control program for moving each frame in the moving object 150 from the ROM 152, expands the control program into the RAM 153, and executes the control program. Accordingly, the operation controller 151 controls the operations of each frame in the moving object 150. The operation controller 151 also transmits operation instruction information and position information to the mobile device 110 via the operation communication unit 157. The operation instruction information and the position information will be described later.

[0039] The ROM 152 is an electrically erasable and recordable nonvolatile memory, and stores parameters required for the operations of each frame in addition to the operation programs of each frame in the moving object 150.

[0040] The RAM 153 is a rewritable volatile memory and is used to expand the program executed by the operation controller 151 and to temporarily store data generated by the operations of the frames in the moving object 150.

[0041] The position detection unit 154 includes a plurality of sensors for obtaining the three-dimensional space position and outputs information related to the three-dimensional space position calculated based on the information detected by each sensor to the operation controller 151. For example, the sensors may include a GPS sensor for detecting a two-dimensional plane position and an atmospheric pressure sensor for detecting altitude. The position detection unit 154 calculates the three-dimensional space position of the moving object 150 by using these sensors. The GPS sensor may be used as a sensor for detecting altitude.

[0042] The operation mechanism 155 is a mechanism by which an occupant instructs the operation of the moving object 150. Specifically, the operation mechanism 155 is constituted by movable mechanisms such as a steering wheel, an accelerator pedal, a brake pedal, a turn signal lever, a wiper lever, and a light switch. Each time the occupant gives an instruction by operating the operation mechanism 155, operation instruction information is output to the drive mechanism 156.

[0043] Based on the operation instruction information output from the operation mechanism 155, the drive mechanism 156 drives mechanisms (such as movable mechanisms like front wheels, engines, and headlights, and notification mechanisms like turn signals, brake lights, and horns) not shown in the moving object 150. Specifically, when the steering angle of the steering wheel is output as operation instruction information from the operation mechanism 155, the drive mechanism 156 transmits the operation instruction information to the actuator for controlling the traveling direction of the moving object 150 and changes the direction of the front wheels. When the degree of depression of the accelerator pedal is output as operation instruction information from the operation mechanism 155, the drive mechanism 156 transmits the operation instruction information to the actuator for controlling the acceleration of the moving object 150 to increase the engine rpm. In addition, operations such as light switch operation, turn signal lever operation, brake pedal depression, and horn operation can be output as operation instruction information from the operation mechanism 155. In this case, the drive mechanism 156 notifies others by light by turning on the headlights, turn signals, or brake lights according to the operation instruction information, or notifies others by sound by sounding the horn.

[0044] The operation communication unit 157 communicates with the communication unit 118 based on a communication scheme defined by a standard such as Wireless LAN.

[0045] Next, with reference to Figure 2 and Figure 3 The operation processing of the mobile imaging device 100 and the moving object 150 when the mobile imaging device 100 captures an image of the moving object 150 will be described in detail with reference to the flowcharts in Figure 2 A flowchart showing the operation processing of the mobile imaging device 100 is shown, and Figure 3 A flowchart showing the operation processing of the moving object 150 is shown. That is, Figure 2 the operation processing in Figure 3 is executed by the controller 111 included in the mobile device 110 expanding the program stored in the ROM 112 to the RAM 113 and running the program. On the other hand,

[0046] In Figure 2 in step S201, the controller 111 first receives a list of imaging scenarios from an information processing device (not shown) via the communication unit 118 and records it in the imaging scenario list storage unit 116. If the imaging scenario list is pre-recorded in the imaging scenario list storage unit 116, step S201 is skipped and the process proceeds to step S202.

[0047] Now, with reference to Figure 4Describe a list of shooting scenarios. The list of shooting scenarios describes multiple shooting scenarios, and each shooting scenario includes the shooting conditions of the imaging device 130 and the position and pose conditions of the mobile device 110. The mobile imaging device 100 performs imaging according to the shooting conditions and the position and pose conditions described in the selected shooting scenario.

[0048] In Figure 4 's example, the main subject, shooting start conditions, and shooting end conditions are described as shooting conditions in the shooting scenario. As the position and pose conditions, the relative position between the mobile imaging device 100 and the mobile subject 150, which is a condition related to the imaging field of view angle, and the shooting method are described in the shooting scenario. The relative position is a three-dimensional relative position, which not only defines a two-dimensional planar relative position such as 3 m to the right or 5 m to the rear of the position relative to the mobile subject 150, but also defines a relative position in the vertical direction such as 30 m above the height relative to the mobile subject 150. In Figure 4 's example, the two-dimensional planar relative position is described on the left side of the slash ( / ), and the relative position in the vertical direction is described on the right side. The relative position in the vertical direction being the condition "SAME" indicates that the heights of the mobile subject 150 and the mobile imaging device 100 are the same. Figure 4 The shooting scenarios shown in

[0049] Next, the specific contents of the shooting conditions and the position and pose conditions described in the shooting scenario will be described with reference to Figure 4 In the example of the shooting scenario named "Inside the Curve" in Figure 4 , when the right turn signal of the mobile subject 150 flashes, the imaging device 130 starts imaging the field of view angle including the main subject (in this example, the front wheel of the mobile subject 150). The mobile imaging device 100 performs imaging while traveling parallel to the mobile subject 150 at a distance of 3 m to the right of the mobile subject 150 and at the same height as the mobile subject 150. Thereafter, when the right turn signal of the mobile subject 150 is turned off, the imaging device 130 ends the imaging. The above is the content of the shooting scenario named "Inside the Curve", and the mobile imaging device 100 operates for the purpose of shooting a moving image with such content according to the manipulation instruction information.

[0050] In Figure 2 In step S202 of

[0051] Now, with reference to Figure 3The flowchart in describes the operation process of moving the subject 150.

[0052] In Figure 3 step S301 of , the manipulation controller 151 determines whether a request for manipulation instruction information and position information is received from the mobile imaging device 100. In Figure 2 step S202 of , a request for manipulation instruction information and position information is transmitted from the mobile imaging device 100. When a request for manipulation instruction information and position information is received ( "Yes" in step S301), the process proceeds to step S302. In step S302, the manipulation controller 151 transmits the manipulation instruction information and the position information to the mobile device 110 via the manipulation communication unit 157, and ends the process. On the other hand, when a request for manipulation instruction information and position information is not received ( "No" in step S301), the process of step S301 is repeated until a request is received.

[0053] Return to reference Figure 2 , in step S203, the controller 111 determines whether manipulation instruction information and position information are received from the mobile subject 150. When manipulation instruction information and position information are received ( "Yes" in step S203), the process proceeds to step S204. On the other hand, when manipulation instruction information and position information are not received ( "No" in step S203), the process returns to step S202 until these information are obtained.

[0054] In step S204, the controller 111 instructs the imaging device 130 to start imaging, and then controls the movement of the movement mechanism 117 while tracking the mobile subject 150. For this tracking movement, a known tracking movement method using subject identification based on the position and pose information detected by the position and pose detection unit 114 and the moving image data generated by the image processor 115 is used, so its detailed description will be omitted.

[0055] In step S205, the controller 111 determines whether the manipulation instruction information received in step S202 matches any one of the imaging start conditions of the imaging scenarios recorded in the imaging scenario list storage unit 116. When the manipulation instruction information matches the imaging start condition of one of the imaging scenarios in the imaging scenario list ( "Yes" in step S205), the process proceeds to step S206. On the other hand, when the manipulation instruction information does not match the imaging start condition of any imaging scenario in the imaging scenario list ( "No" in step S205), the process returns to step S202. Thereafter, obtaining the manipulation instruction information is repeated until the manipulation instruction information matches the imaging start condition of any one of the imaging scenarios in the imaging scenario list.

[0056] In step S206, the controller 111 (camera scenario selection unit) selects a matching camera scenario as the camera scenario to be used for imaging from now on.

[0057] In step S207, the controller 111 controls the position and orientation of the mobile device 110 by controlling the movement mechanism 117 so that the position and orientation match the position and orientation conditions of the camera scenario selected in step S206. Specifically, the controller 111 drives the movement mechanism 117 to be at the relative position described in the selected camera scenario based on the position information of the moving subject 150, the information detected by the position and orientation detection unit 114, and the subject tracking result.

[0058] When it is determined that the mobile imaging device 100 has reached a position and orientation that match the position and orientation conditions described in the camera scenario selected in step S206, in step S208, the controller 111 instructs the imaging device 130 to start recording a moving image. Accordingly, in the imaging device 130, the image processor 134 generates moving image data from the digital image data sequentially stored in the RAM 113 in chronological order, and the image recording unit 135 starts recording the moving image data.

[0059] In step S209, the controller 111 starts controlling the position and orientation of the mobile imaging device 100 according to the imaging method described in the camera scenario selected in step S206. Note that the imaging (image recording) in the imaging device 130 started in step S208 continues until the process proceeds to step S211 described later.

[0060] In step S210, the controller 111 determines whether the imaging end condition described in the camera scenario selected in step S206 matches the manipulation instruction information recently received from the moving subject 150. Here, since the manipulation instruction information recently received by the controller 111 matches the imaging start condition and does not match the imaging end condition, it is necessary to newly obtain the manipulation instruction information from the moving subject 150.

[0061] When the manipulation instruction information does not match the imaging end condition ( "No" in step S210), the controller 111 in the mobile device 110 advances the process to step S213. In step S213, the controller 111 transmits a request for the manipulation instruction information to the moving subject 150 via the communication unit 118 and attempts to obtain the information.

[0062] Then, in step S214, the controller 111 determines whether manipulation instruction information has been received from the moving subject 150. When the manipulation instruction information is received ( Figure 2 "Yes" in step S214 in Figure 2Step S210 in. On the other hand, when no manipulation instruction information is received ( Figure 2 "No" in step S214 in), the process returns to Figure 2 Step S213 in.

[0063] On the other hand, when the newly obtained latest manipulation instruction information matches the imaging end condition ("Yes" in step S210), the process proceeds to step S211.

[0064] In step S211, the controller 111 instructs the imaging device 130 to end recording the moving image. Thus, in the imaging device 130, the image processor 134 ends generating the moving image data and the image recording unit 135 ends recording the moving image data.

[0065] In step S212, the controller 111 determines whether the imaging operations for all the imaging scenes listed in the imaging scene list are completed. When the imaging operations for not all the imaging scenes described in the imaging scene list are completed ("No" in step S212), the process returns to step S204, and the uncompleted imaging operations according to the imaging scenes are performed. On the other hand, when the imaging operations for all the imaging scenes described in the imaging scene list are completed ("Yes" in step S212), the process ends.

[0066] The operation processes of the mobile imaging device 100 and the mobile subject 150 when the mobile imaging device 100 of the present embodiment captures an image of the mobile subject 150 have been described above.

[0067] Next, with reference to Figure 5 Specifically describe the positional relationship between the mobile imaging device 100 and the mobile subject 150 when the mobile imaging device 100 performs an imaging operation according to the Figure 2 Operation process in.

[0068] At the start time, the mobile subject 150 is located at position L411, and the mobile imaging device 100 is located at position L401. In Figure 5 In the example of, at the start time, the position L401 of the mobile imaging device 100 is on the left side of the mobile subject 150 and at the same height as the position L411 of the mobile subject 150. In Figure 5 In the example of, the mobile imaging device 100 moves from position L401 to positions L402, L403, and L404 over time, and the mobile subject 150 moves from position L411 to positions L412, L413, and L414 over time. In this example, the mobile subject 150 flashes the right turn signal at positions L412 and L413.

[0069] In Figure 5In the example, the moving imaging device 100 at position L401 moves by tracking the moving subject 150 while maintaining the same height as the left side of the moving subject 150 that starts to be manipulated at position L411 (step S204). This tracking movement continues until the manipulation instruction information transmitted from the moving subject 150 to the moving imaging device 100 matches the imaging start condition of any one of the imaging scenarios in the imaging scenario list.

[0070] Next, when the moving subject 150 blinks its right turn signal, in response to a request for manipulation instruction information from the moving imaging device 100, the moving subject 150 transmits the manipulation instruction information "right turn signal blinks" to the moving imaging device 100 (step S202).

[0071] When the manipulation instruction information "right turn signal blinks" is received via the communication unit 118 ( "Yes" in step S203), the controller 111 selects an imaging scenario whose imaging start condition matches the received manipulation instruction information (steps S205 and S206). Specifically, the controller 111 selects from Figure 4 the imaging scenario list shown an imaging scenario "inside the curve" whose imaging start condition matches the manipulation instruction information "right turn signal blinks".

[0072] Next, the moving imaging device 100 moves to the relative position of the position and posture conditions described in the selected imaging scenario "inside the curve" (step S207). Specifically, since the relative position of the position and posture conditions of the imaging scenario "inside the curve" is "3 m to the right / same height", the moving imaging device 100 moves from position L401 to position L402. That is, the moving imaging device 100 at position L402 is located 3 m to the right at the same height as the moving subject 150. Note that the moving subject 150 moves from position L411 to position L412 while blinking its right turn signal during this period.

[0073] When the moving imaging device 100 reaches a position and posture that match the position and posture conditions described in the selected imaging scenario "inside the curve", the imaging device 130 starts recording a moving image (step S208). Specifically, when the moving imaging device 100 moves to position L402 and the relative position with respect to the moving subject 150 at position L412 becomes "3 m to the right / same height", the imaging device 130 starts recording a moving image.

[0074] The mobile imaging device 100 captures an image according to the imaging method described in the selected imaging scenario "Inner Side of Curve" (step S209). The imaging method described in the selected imaging scenario "Inner Side of Curve" is "Parallel Travel". Therefore, the mobile imaging device 100 moves from position L412 to position L413 in a manner parallel to the moving subject 150 turning right from position L402 to position L403, while tracking and capturing the image. This tracking imaging continues until the manipulation instruction information matches the imaging end condition described in the imaging scenario in Figure 2 step S210.

[0075] When the moving subject 150 turns off the right turn signal at position L414, the controller 111 included in the mobile imaging device 100 located at position L404 receives the manipulation instruction information "Right Turn Signal Turned Off" from the moving subject 150 via the communication unit 118 (step S214). Since the manipulation instruction information received at this time matches the imaging end condition of the selected imaging scenario "Inner Side of Curve" ( "Yes" in step S210), the mobile imaging device 100 ends recording the moving image ( Figure 2 step S211 in).

[0076] As shown in the example in Figure 5 , since the manipulation instruction information "Right Turn Signal Flashing" is obtained, the mobile imaging device 100 can move to the position described in the imaging scenario before the moving subject 150 starts to turn right. Therefore, the mobile imaging device 100 can record the moving image of the moving subject 150 without delay starting from the beginning of the right turn instead of starting midway through the right turn. In this way, the mobile imaging device 100 can capture images while immediately responding to the state of the freely moving subject (moving subject 150), and can improve the freedom of imaging.

[0077] In Figure 2 step S205, the imaging start condition described in the completed imaging scenario may match the manipulation instruction information. In this case, the completed imaging scenario can be controlled not to be selected as the imaging scenario to be performed. This can prevent excessive imaging of only specific imaging scenarios and increase the probability of selecting an uncompleted imaging scenario that describes the same imaging start condition. This control is performed according to, for example, the imaging status information included in each imaging scenario for indicating whether the imaging scenario has been completed.

[0078] In the above embodiment, the information related to the flashing and turning off of the right turn signal has been described as an example of the manipulation instruction information. However, the manipulation instruction information is not limited to this. For example, when selecting Figure 4In the case of the shooting scene "BACKSHOT" shown in the figure, the manipulation instruction information "step on the brake pedal" constitutes the shooting start condition. In this case, when the received manipulation instruction information matches the shooting start condition, the mobile imaging device 100 moves to the same height behind the moving subject 150 and shoots the illuminated brake light.

[0079] For example, when selecting Figure 4 the shooting scene "OVERLOOK" (bird's-eye view) shown in the figure, the manipulation instruction information "the depression degree of the accelerator pedal is stable" constitutes the shooting start condition. In this case, when the received manipulation instruction information matches the shooting start condition, the mobile imaging device 100 moves to the sky above the moving subject 150 and shoots the surrounding scenery and the moving subject 150 with an overhead composition. When shooting such an overhead composition, by shooting the moving subject 150 with a composition including beautiful scenery or landmarks such as symbolic buildings, a highly aesthetic moving image can be obtained.

[0080] Therefore, the controller 111 (subject information acquisition unit) can obtain position information related to subjects (such as landmarks) around the moving subject 150 from an information processing device (not shown), and start shooting when the landmark is included in the field of view angle of the imaging device 130. Specifically, Figure 4 the shooting start condition of the shooting scene "OVERLOOK" in also includes "there are landmarks around the moving subject 150" in addition to "the depression degree of the accelerator pedal is stable". That is to say, in this case, both "the depression degree of the accelerator pedal is stable" and "the landmark and the moving subject fall within the field of view angle" constitute the shooting start condition of the shooting scene "OVERLOOK". In Figure 2 step S205 of, when the controller 111 determines that the received manipulation instruction information and the obtained position information related to the landmark match the shooting start condition of the shooting scene "OVERLOOK", the controller 111 starts shooting according to the shooting scene "OVERLOOK".

[0081] Although in the above embodiment, shooting is started when it is determined in Figure 2 step S205 that the shooting start condition described in the shooting scene matches the manipulation instruction information, the shooting start condition is not limited to this. For example, even if the shooting start condition described in the shooting scene matches the manipulation instruction information, when the moving subject 150 moves rapidly and the mobile imaging device 100 cannot keep up, there is no need to start shooting. That is to say, when the shooting start condition described in the shooting scene matches the manipulation instruction information, information on the traveling speed can be obtained from the moving subject 150 as the manipulation instruction information, and shooting can be started when the traveling speed is equal to or lower than a predetermined speed.

[0082] In the above embodiments, an example of tracking shooting while maintaining the relative positions described in the selected shooting scene has been described, but the method of tracking shooting is not limited thereto. For example, if there is an obstacle and the relative position cannot be maintained, tracking shooting can be performed in such a way that the maintenance of the relative position is temporarily stopped, the obstacle is avoided, and then the relative position is maintained again.

[0083] The moving imaging device 100 detects an obstacle based on the information detected by the position and posture detection unit 114 included in the moving imaging device 100. The moving imaging device 100 can detect an obstacle by receiving the position information of the obstacle from an information processing device (not shown) via the communication unit 118. When the obstacle is moving, the position information changes continuously. Therefore, the obstacle can be detected as needed by receiving the position information related to the obstacle or receiving the predicted position information obtained by predicting the future position.

[0084] In the above embodiments, an example has been described in which the controller 111 included in the mobile device 110 indicates the selection of the shooting scene and the start and end of moving image recording according to the manipulation instruction information. However, the selection of the shooting scene or the indication of the start and end of moving image recording is not limited thereto. For example, a server (not shown) can coordinate the mobile device 110, the imaging device 130, and the moving subject 150, and the server can select a shooting scene corresponding to the manipulation instruction information, or can issue an instruction to start or end moving image recording. Although an example in which the controller 111 included in the mobile device 110 controls the imaging device 130 has been described in the above embodiments, this is not restrictive. For example, the imaging device 130 can also include a controller separate from the controller 111 included in the mobile device 110, and can control the operation of each block of the imaging device 130.

[0085] Other embodiments

[0086] Embodiments of the present invention can also be implemented by the following method, that is, by providing software (program) that executes the functions of the above embodiments to a system or device through a network or various storage media, and a method in which a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0087] Although the present invention has been described with reference to exemplary embodiments, it should be understood that the present invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be given the broadest interpretation so as to encompass all such modifications as well as equivalent structures and functions.

[0088] This application claims the benefit of Japanese Patent Application No. 2023-200674, filed on November 28, 2023, the entire contents of which are incorporated herein by reference.

Claims

1. An information processing device for controlling a mobile camera device equipped with a camera device, the information processing device comprising: a manipulation instruction information obtaining unit configured to obtain manipulation instruction information related to a moving object accompanied by manipulation; A camera control unit configured to control the camera of the camera device; as well as A position and posture control unit is configured to control the position and posture of the mobile camera device by driving the mobile camera device based on the manipulation instruction information.

2. The information processing device according to claim 1, further comprising: an imaging scenario list obtaining and holding unit configured to obtain and hold an imaging scenario list including a plurality of imaging scenarios describing conditions related to imaging by the imaging device; as well as A camera scene selection unit is configured to select one of the plurality of camera scenes according to the manipulation instruction information.

3. The information processing device according to claim 2, wherein: An imaging start condition of the imaging device and a position and posture condition of the mobile imaging device are described in each of the plurality of imaging scenarios.

4. The information processing device according to claim 3, wherein: The imaging scenario selection unit selects an imaging scenario whose imaging start condition matches the manipulation instruction information obtained by the manipulation instruction information obtaining unit from among the plurality of imaging scenarios.

5. The information processing device according to claim 4, wherein: The plurality of imaging scenarios each include imaging status information indicating whether an imaging operation according to the imaging scenario is completed or incomplete, and Among them, even if the shooting start condition of the shooting scene matches the manipulation instruction information obtained by the manipulation instruction information obtaining unit, the shooting scene selection unit does not select the following shooting scene from the multiple shooting scenes, wherein the shooting status information of the shooting scene indicates that the shooting operation corresponding to the shooting scene is completed.

6. The information processing device according to claim 3, further comprising a subject information obtaining unit configured to obtain position information related to subjects around the moving subject, in, The imaging scenario selection unit selects an imaging scenario whose imaging start condition matches the manipulation instruction information obtained by the manipulation instruction information obtaining unit and the position information obtained by the object information obtaining unit from among the plurality of imaging scenarios.

7. The information processing device according to any one of claims 3 to 6, wherein: The position and posture conditions include the relative position between the mobile camera device and the mobile subject, and When the position and posture of the mobile camera device controlled by the position and posture control unit match the relative position described in the camera scenario selected by the camera scenario selection unit, the camera device starts recording moving images.

8. The information processing device according to claim 7, wherein: The position and posture conditions also include the photographing method of the camera, and During the period when the camera device records the moving image, the position and posture control unit controls the position and posture of the mobile camera device according to the photographing method described in the camera scenario selected by the camera scenario selection unit.

9. The information processing device according to claim 7, wherein: Each of the plurality of imaging scenarios further includes an imaging end condition of the imaging device, and Wherein, when the manipulation instruction information matches the imaging end condition described in the imaging scenario selected by the imaging scenario selection unit, the imaging device ends recording the moving image.

10. The information processing device according to claim 7, wherein: The plurality of imaging scenarios each further include a main subject to be captured by the imaging device, and During the period when the camera device records the moving image, the position and posture control unit controls the position and posture of the mobile camera device so that the main subject described in the camera scene selected by the camera scene selection unit will be included in the field of view of the camera device.

11. The information processing device according to any one of claims 3 to 6 and 8 to 10, further comprising a position information obtaining unit configured to obtain position information related to the moving subject, in, The manipulation instruction information obtaining unit repeatedly obtains the manipulation instruction information until one imaging start condition among the plurality of imaging scenarios matches the manipulation instruction information, and the position and posture control unit performs tracking movement of the moving subject based on the position information.

12. The information processing device according to any one of claims 1 to 6 and 8 to 10, wherein: The imaging device captures an image of the moving object.

13. The information processing device according to any one of claims 1 to 6 and 8 to 10, wherein: The manipulation instruction information is related to a manipulation instruction for at least one of a position and a movement of the moving object.

14. The information processing device according to claim 13, wherein: The manipulation instruction information is related to at least one of the following manipulation instructions, wherein these manipulation instructions are related to the speed or direction of the moving object, the operation of a movable mechanism included in the moving object, and the notification by a notification mechanism included in the moving object through sound or light.

15. The information processing device according to any one of claims 1 to 6, 8 to 10 and 14, the information processing device being included in the mobile camera device, the mobile subject, or a server capable of communicating with the mobile camera device and the mobile subject. 16 . An information processing system comprising the mobile imaging device according to claim 1 and a moving object.

17. A control method of an information processing device, the information processing device being used to control a mobile camera device equipped with a camera device, the control method comprising: A manipulation instruction information obtaining step, for obtaining manipulation instruction information related to a moving object accompanied by manipulation; A camera control step, used to control the camera of the camera device; as well as A position and posture control step is used to control the position and posture of the mobile camera device by driving the mobile camera device based on the manipulation instruction information.

18. A computer program product comprising a program for causing a computer to execute the control method according to claim 17.

19. A computer-readable storage medium storing a program for causing a computer to execute the control method according to claim 17.

Images