Information processing apparatus, control method, program product, and storage medium

Abstract

The invention discloses an information processing apparatus, a control method, a program product, and a storage medium. The information processing apparatus is capable of accurately recording information related to a manipulation instruction of a moving subject in real time in association with photographing data photographed by a mobile imaging apparatus. The information processing apparatus includes: a generation section configured to obtain manipulation instruction information related to a moving subject accompanying manipulation, and generate manipulation instruction related information corresponding to the manipulation instruction information; and a recording means for recording, in association with each other, imaging data generated by an imaging apparatus mounted on a mobile imaging apparatus by capturing an image of the mobile subject, the imaging data and the manipulation instruction-related information generated by the generating means, whenever the imaging data is obtained from the imaging apparatus.

Description

Technical Field

[0001] The present invention relates to an information processing device, a control method, a program product, and a storage medium storing a program for controlling imaging and recording in a mobile imaging device. Background Art

[0002] In recent years, there has been a trend of taking images during driving and producing image works as driving VLOGs. Since various image edits are required in VLOGs, users who produce driving VLOGs using mobile imaging devices such as drones have started to appear.

[0003] To obtain higher-quality driving VLOGs, users may want edits for adding sounds to images according to the operation of a car and for inserting transitions between scenes of the captured images. For example, there are edits for adding engine sounds to the scene where the accelerator is depressed, edits for rotating the steering wheel at the end of a scene and adding a rotation transition between that scene and the next scene, and so on.

[0004] On the other hand, a technique for recording image data captured by a drone and information related to the image data is disclosed. For example, Japanese Unexamined Patent Application Publication No. 2022-104716 (JP 2022-104716A) discloses a technique for recording image data captured by a camera of a drone and generating identification information for identifying the recorded image data.

[0005] Japanese Unexamined Patent Application Publication No. 2012-216102 (JP 2012-216102A) discloses the following technique. That is, an in-vehicle imaging device captures an image with a built-in camera, obtains additional information characterizing the captured image, and transmits the additional information to an image management device. The image management device determines whether the image characterized by the additional information is necessary based on the additional information. When it is determined that the image is necessary, the image management device requests and receives image data from the imaging device and stores the image data in association with the additional information.

[0006] However, the identification information of JP 2022-104716A and the additional information of JP 2012-216102A are information related to the images captured by the camera (such as the imaging time, etc.), rather than information related to the subject captured by the camera.

[0007] For example, when the moving subject is a car, in theory, information related to the degree of depression of the accelerator of the user driving the car can be obtained by analyzing the acceleration of the car from the image data recorded in the camera, but it is difficult to obtain the degree of depression of the accelerator with high accuracy. In addition, when performing such an analysis, there is a problem that information related to the manipulation instruction of the moving subject cannot be recorded in real time in association with the captured image.

[0008] In VLOG production, a work is produced while combining various shooting data including image data and sound data obtained by shooting with a drone. Therefore, as in JP 2012-216102A, when only the image data determined to be necessary is recorded together with additional information, there is a problem that the shooting data required by the user during editing may not be recorded. Summary of the Invention

[0009] The present invention provides an information processing device, a control method, a computer program product, and a storage medium storing a control program that can accurately record information related to a manipulation instruction of a moving subject in real time in association with shooting data captured by a mobile imaging device.

[0010] Therefore, one aspect of the present invention provides an information processing device including: a generation component configured to obtain manipulation instruction information related to a moving subject accompanied by manipulation and generate manipulation instruction-related information corresponding to the manipulation instruction information; and a recording component configured to record the shooting data and the manipulation instruction-related information generated by the generation component in association with each other whenever the shooting data generated by the imaging device mounted on the mobile imaging device by shooting an image of the moving subject is obtained.

[0011] According to the present invention, information related to a manipulation instruction of a moving subject accompanied by movement can be recorded in real time with high accuracy in association with shooting data captured by a mobile imaging device.

[0012] Other features of the present invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. Brief Description of the Drawings

[0013] Figure 1 is a block diagram showing a hardware configuration example of a mobile imaging device including an image recording unit as an information processing device according to an embodiment of the present invention and a moving subject as an imaging target of the mobile imaging device.

[0014] Figure 2 is a block diagram showing a software configuration of the image recording unit.

[0015] Figure 3 is a flowchart showing a recording operation process performed by the image recording unit.

[0016] Figure 4A and Figure 4B are diagrams showing correspondence tables each for describing the relationship between manipulation instruction content and manipulation instruction information.

[0017] Figure 5This is a diagram showing a correspondence table for describing the relationship between manipulation instruction information and manipulation instruction-related information when the manipulation instruction-related information (hereinafter referred to as "manipulation instruction-related information") is an edit point flag.

[0018] Figure 6A and Figure 6B This is a diagram showing a correspondence table for describing the relationship between manipulation instruction information and manipulation instruction-related information when the manipulation instruction-related information is a flag related to a transition.

[0019] Figure 7 This is a diagram showing a correspondence table for describing the relationship between manipulation instruction information and manipulation instruction-related information when the manipulation instruction-related information is a captured transition flag.

[0020] Figure 8 This is a diagram showing a correspondence table for describing the relationship between manipulation instruction information and manipulation instruction-related information when the manipulation instruction-related information is a weighted result of the manipulation instruction information.

[0021] Figure 9 This is a diagram showing a correspondence table for describing the relationship between manipulation instruction information and manipulation instruction-related information when the manipulation instruction-related information is an edit point flag related to the surrounding space. Detailed Description of the Preferred Embodiments

[0022] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0023] In this embodiment, according to the manipulation instruction information of the moving subject accompanied by manipulation, a desired moving image is captured by changing the position, posture, and imaging conditions of the moving imaging device. In this embodiment, an example in which the moving imaging device moves in the air by a thruster will be described, but the moving method is not limited thereto. For example, the moving imaging device can move on the ground by using wheels, or can move in water by using a propeller.

[0024] Hereinafter, a moving imaging device 100 including an image recording unit 135 as an information processing device according to an embodiment of the present invention and a moving subject 150 accompanied by manipulation, which is an imaging target of the moving imaging device 100 and is communicably connected to the moving imaging device 100 by wireless communication, will be described in detail with reference to the accompanying drawings.

[0025] Figure 1 This is a block diagram showing an example of the hardware configuration of the moving imaging device 100 and the moving subject 150.

[0026] As Figure 1As shown, the mobile imaging device 100 is composed of an imaging device 130 (imaging unit) and a mobile device 110 on which the imaging device 130 is mounted. The imaging device 130 and the mobile device 110 are connected to each other via a bus so as to communicate image data, operation instruction information, imaging mode information, etc. described later. The moving subject 150 is a subject photographed by the imaging device 130 and moves in response to an operation instruction.

[0027] First, a configuration example of the mobile device 110 included in the mobile imaging device 100 will be described with reference to Figure 1 the following.

[0028] The mobile device 110 includes a controller 111, a ROM 112, a RAM 113, a position and orientation detection unit 114, an image processor 115, an imaging scenario list storage unit 116, a moving mechanism 117, and a communication unit 118. These blocks are communicably connected to each other via a bus.

[0029] The controller 111 is, for example, a CPU, reads control programs for the blocks in the mobile device 110 and the blocks in the imaging device 130 described later from the ROM 112, expands the control programs into the RAM 113, and runs the control programs. Therefore, the controller 111 controls the operations of the respective blocks in the mobile imaging device 100. In addition, the controller 111 receives operation instruction information and position information from the moving subject 150 via the communication unit 118. The operation instruction information and the position information will be described later.

[0030] The ROM 112 is an electrically erasable and recordable non-volatile memory, and stores parameters required for the operations of the respective blocks in addition to the operation programs of the respective blocks included in the mobile imaging device 100.

[0031] The RAM 113 is a rewritable volatile memory and is used to expand the programs executed by the controller 111 and to temporarily store the data generated by the operations of the blocks in the mobile imaging device 100.

[0032] The position and orientation detection unit 114 includes a plurality of sensors for detecting various information required for position and orientation control of the mobile device 110, and outputs the detected position and orientation information to the controller 111. Examples of the sensors include a GPS sensor for detecting position, a gyro sensor for detecting angular velocity, an acceleration sensor for detecting speed change, a magnetic sensor for detecting direction, an atmospheric pressure sensor for detecting altitude, and an ultrasonic sensor for detecting the distance to surrounding subjects such as a subject. The controller 111 performs calculations based on the information detected by the position and orientation detection unit 114 and controls the position and orientation of the mobile device 110.

[0033] The image processor 115 performs various image processes related to the position and orientation of the mobile device 110 by analyzing the images captured by the imaging device 130 and the information necessary for controlling the position and orientation detected by the position and orientation detection unit 114. For example, the image processor 115 performs image processing for identifying a subject (e.g., the moving subject 150) included in the images captured by the imaging device 130, and the controller 111 controls the moving mechanism 117 to track the subject using the identification result. Known methods are used to identify and track the subject, and the detailed description thereof will be omitted.

[0034] The imaging scene list storage unit 116 records an imaging scene list received from an external information processing device (not shown) via the communication unit 118. The imaging scene list may be pre-recorded in the imaging scene list storage unit 116 at the time of initial setting.

[0035] The moving mechanism 117 is composed of a lift generation mechanism including a motor and a thruster. The controller 111 controls the position and orientation of the mobile device 110 by driving the moving mechanism 117 according to the information detected by the position and orientation detection unit 114 and the identification result of the subject by the image processor 115. This enables the tracking camera to control the field of view angle so that the moving subject 150 will be included in the field of view area captured by the imaging device 130.

[0036] The communication unit 118 communicates with the moving subject 150 based on a communication scheme defined by a standard such as Wireless LAN.

[0037] Next, a configuration example of the imaging device 130 included in the mobile imaging device 100 will be described with reference to Figure 1 the following.

[0038] The imaging device 130 includes an optical system 131, an imaging device 132, an A / D converter 133, an image processor 134, an image recording unit 135, and a display unit 136. These blocks are communicably connected to each other via a bus.

[0039] The optical system 131 is composed of a diaphragm mechanism and a lens group including a zoom lens group and a focusing lens group, and forms a subject image on the imaging surface of the imaging device 132.

[0040] The imaging device 132 is an image sensor such as a CCD sensor or a CMOS sensor, for example, and performs photoelectric conversion on the optical image formed on the imaging surface of the imaging device 132 by the optical system 131, and outputs the obtained analog image signal to the A / D converter 133.

[0041] The A / D converter 133 converts the input analog image signal into digital image data and outputs the digital image data. The digital image data output from the A / D converter 133 is temporarily stored in the RAM 113.

[0042] The image processor 134 applies various image processes to the digital image data stored in the RAM 113. Specifically, the image processor 134 applies various image processes for developing, displaying, and recording the digital image data, such as demosaicing process, noise reduction process, white balance correction process, and gamma correction process. The image process includes a process of generating moving image data from the digital image data stored in the RAM 113 in time series.

[0043] The image recording unit 135 records the data including the moving image data generated by the image processor 134 into the built-in recording medium. The image recording unit 135 obtains the operation instruction information received via the communication unit 118 and the shooting mode information indicating the information related to the shooting mode, and generates and records the information related to the operation instruction, where the shooting mode is used to indicate the movement of the moving device 110 during the shooting of the imaging device 130, such as the rotation of the camera. The process of the image recording unit 135 will be described later.

[0044] The display unit 136 includes a display panel such as an LCD panel, and displays the digital image data stored in the RAM 113 and the video data recorded in the image recording unit 135 on the display panel.

[0045] Next, a configuration example of the moving subject 150 will be described with reference to Figure 1 In the present embodiment, the mobile imaging device 100 uses the imaging device 130 to shoot the moving subject 150 and generates image data. Although in the present embodiment, the moving subject 150 is a car, the moving subject 150 is not limited thereto. For example, the moving subject 150 may be a subject whose movement is controlled by an operator, such as a motorcycle, a boat, an airplane, or a train. The moving subject 150 may be a device that is not operated by an operator and automatically maneuvers while recognizing the surrounding situation. The moving subject 150 may be a device that is not operated by an occupant but is controlled by remote control, such as a radio-controlled car. The automatic operation and the remote operation are performed by known methods, and their detailed descriptions are omitted.

[0046] The moving subject 150 includes an operation controller 151, a ROM 152, a RAM 153, a position detection unit 154, an operation mechanism 155, a drive mechanism 156, and an operation communication unit 157. These blocks are communicably connected to each other via a bus.

[0047] The operation controller 151, which is a CPU for example, reads the control program for moving each frame in the subject 150 from the ROM 152, expands the control program into the RAM 153, and executes the control program. Therefore, the operation controller 151 controls the operations of each frame in the subject 150. The operation controller 151 also transmits operation instruction information and position information to the mobile device 110 via the operation communication unit 157. The operation instruction information and the position information will be described later.

[0048] The ROM 152 is an electrically erasable and recordable non-volatile memory, and stores parameters required for the operations of each frame in addition to the operation programs of each frame in the subject 150.

[0049] The RAM 153 is a rewritable volatile memory, and is used to expand the program executed by the operation controller 151 and to temporarily store data generated by the operations of the frames in the subject 150.

[0050] The position detection unit 154 includes a plurality of sensors for obtaining the three-dimensional space position, and outputs information related to the three-dimensional space position calculated based on the information detected by each sensor to the operation controller 151. For example, the sensors may include a GPS sensor for detecting the two-dimensional plane position and an atmospheric pressure sensor for detecting the altitude. The position detection unit 154 calculates the three-dimensional position information of the subject 150 by using these sensors. The GPS sensor can be used as a sensor for detecting the altitude.

[0051] The operation mechanism 155 is a mechanism by which the occupant instructs the operation of the subject 150. Specifically, the operation mechanism 155 is composed of movable mechanisms such as a steering wheel, an accelerator, a brake pedal, a turn signal lever, a wiper lever, and a light switch. Whenever the occupant gives an instruction by operating the operation mechanism, operation instruction information is output to the drive mechanism 156.

[0052] Based on the operation instruction information output from the operation mechanism 155, the drive mechanism 156 drives mechanisms (such as movable mechanisms such as front wheels, an engine, and headlights, and notification mechanisms such as turn signals, brake lights, and horns) in the subject 150 that are not shown.

[0053] Specifically, when the steering angle of the steering wheel is output as operation instruction information from the operation mechanism 155, the drive mechanism 156 transmits the operation instruction information to the actuator for controlling the traveling direction of the subject 150, and changes the direction of the front wheels. When the degree of depression of the accelerator is output as operation instruction information from the operation mechanism 155, the drive mechanism 156 transmits the operation instruction information to the actuator for controlling the acceleration of the subject 150 to increase the rpm of the engine.

[0054] In addition, operations such as lamp switch operation, turn signal lever operation, brake pedal depression, and horn operation can be output as operation instruction information from the operation mechanism 155. In this case, the drive mechanism 156 notifies others visually by lighting the headlamp, turn signal lamp, or brake lamp according to the operation instruction information, or notifies others audibly by sounding the horn.

[0055] The operation communication unit 157 communicates with the communication unit 118 based on a communication scheme defined by a standard such as Wireless LAN.

[0056] Next, the software configuration of the image recording unit 135 in Figure 2 will be described. Figure 1 is as follows.

[0057] The image recording unit 135 in this embodiment is characterized in that operation instruction-related information is generated based on operation instruction information obtained from the moving subject 150 via the communication unit 118, and the generated operation instruction-related information is recorded in association with the image data captured by the imaging device 130.

[0058] As Figure 2 shown, the image recording unit 135 in this embodiment includes an information generation module 201 and a recording module 202.

[0059] When the operation instruction information is obtained via the communication unit 118, the information generation module 201 (generation component) generates operation instruction-related information corresponding to the operation instruction information, and transmits the operation instruction-related information to the recording module 202.

[0060] Whenever the recording module 202 (recording component) obtains the image data captured by the imaging device 130 from the imaging device 130 via the bus, the recording module 202 associates the obtained image data with the operation instruction-related information transmitted from the information generation module 201. Thereafter, the recording module 202 transmits the associated image data and operation instruction-related information to the RAM 113, and records and holds them.

[0061] Here, the input data from the imaging device 130 to the information generation module 201 is the image data captured by the imaging device 132, but it is not limited thereto, as long as the input data is the captured data generated by the imaging device 130. For example, the captured data input from the imaging device 130 to the information generation module 201 may be sound data obtained by a sound acquisition device (not shown) of the imaging device 130 instead of image data. This makes it possible to easily obtain the material data used in the case where the image data and the sound data are captured separately and then these data are combined to create a VLOG.

[0062] For example, by associating and recording the engine sound obtained by the sound acquisition device with the degree of accelerator pedal depression (operation instruction information) at the time of acquisition, sound material data that emphasizes the volume according to the operation can be obtained. The shooting data input from the imaging device 130 to the information generation module 201 can be data including image data and sound data. Therefore, the image recording unit 135 can record the operation instruction related information generated for the sound in association with the image data.

[0063] Next, the recording operation process executed by the image recording unit 135 will be described using the Figure 3 flowchart in. This process is executed whenever the communication unit 118 obtains operation instruction information from the moving subject 150.

[0064] In step S301, when operation instruction information is input from the communication unit 118, the information generation module 201 generates operation instruction related information from the operation instruction information and outputs the operation instruction related information to the recording module 202. Here, the operation instruction information is a flag indicating operation instruction contents such as depressing the accelerator, turning the steering wheel, or opening the window, and is represented by a number indicating the operation instruction information set in advance as Figure 4A shown.

[0065] In addition, the operation instruction information can be a flag represented by multiple values distinguished according to the degree of accelerator pedal depression, for example, as Figure 4B shown. In addition, although the operation instruction related information is generated and output from the operation instruction information in this embodiment, the operation instruction information itself can also be output as the operation instruction related information. A method of generating operation instruction related information from the operation instruction information will be described in detail later.

[0066] In step S302, when the image data captured by the imaging device 130 is input, the recording module 202 records the input image data in association with the operation instruction related information generated in step S301. Then, this process ends.

[0067] The above is the processing content of the image recording unit 135 according to this embodiment.

[0068] Now, a method of generating operation instruction related information from the operation instruction information performed in step S301 will be described in detail.

[0069] In step S301, the information generation module 201 generates an edit point flag associated in advance with the input operation instruction information as the operation instruction related information. The edit point flag indicates the edit point (in-point or out-point) of the image and the edit point (in-point or out-point) of the sound. For example, when operation instruction information indicating the operation instruction content of depressing the accelerator is input (Figure 4A When there is a marker 1-1 in [ ] Figure 5 According to the table shown, an edit point marker 2-3 indicating the in-point of the corresponding sound is generated as manipulation instruction related information. Thereafter, in step S302, the edit point marker 2-3 generated as manipulation instruction related information is recorded in association with the image data. Therefore, the user can grasp the image data that is a candidate for the timing of the sound data for adding the engine sound when editing the image of the moving subject 150 during acceleration by stepping on the accelerator.

[0070] Although the case where an edit point marker pre-associated with the input manipulation instruction information exists as manipulation instruction related information is described in this embodiment, the edit point marker may not exist. If the edit point marker does not exist, at least one of the output marker 2-0 (a marker indicating the non-existence of the corresponding manipulation instruction related information) and the manipulation instruction information is output as manipulation instruction related information, or nothing is output.

[0071] In addition, in step S301, the information generation module 201 may generate a transition edit point marker or a recommended transition marker pre-associated with the input manipulation instruction information as manipulation instruction related information. As Figure 6A shown in the table in [ ], the transition edit point marker is a marker indicating an edit point (in-point or out-point, etc.) for image transition, a marker indicating an edit point (in-point or out-point, etc.) for sound transition. As Figure 6B shown in the table in [ ], the recommended transition marker is a marker indicating a recommended transition pre-associated with each manipulation instruction information.

[0072] Multiple manipulation instruction related information can be associated. For example, two manipulation instruction information ([ ] Figure 4A marker 1-2 in [ ]) indicating the content of the manipulation instruction of turning the steering wheel can be input. In this case, for one of the two manipulation instruction information, a transition edit point marker ([ ] Figure 6A marker 3-1 in [ ]) indicating the in-point for image transition is generated as manipulation instruction related information. In addition, for the other manipulation instruction information of the two manipulation instruction information, a recommended transition marker ([ ] Figure 6B marker 4-1 in [ ]) indicating a rotational transition is generated as manipulation instruction related information.

[0073] Thereafter, in step S302, the markers 3-1 and 4-1 generated as manipulation instruction related information are recorded in association with the image data. Therefore, even when not checking all images when editing the image of the moving subject 150 rotating by turning the steering wheel, the user can grasp, through image processing, the image data that is a candidate for the timing of inserting a rotational transition for rotating the image.

[0074] Although the case where a transition edit point flag and a recommended transition flag associated in advance with input manipulation instruction information exist as manipulation instruction related information is described in the present embodiment, the transition edit point flag and the recommended transition flag may not exist. If these pieces of information do not exist, at least one of an output flag 4-0 (a flag indicating the non-existence of corresponding manipulation instruction related information) and the manipulation instruction information is output as the manipulation instruction related information, or nothing is output.

[0075] In addition, in step S301, the information generation module 201 may generate a captured transition flag associated in advance with the manipulation instruction information and the photographing mode information obtained by the image recording unit 135 as the manipulation instruction related information.

[0076] Here, the photographing mode information is a flag indicating the content of the position and posture control (photographing mode) of the mobile device 110 by the controller 111 during the imaging of the imaging device 130, and is represented by a number indicating the content of the photographing mode set in advance as Figure 7 shown.

[0077] The captured transition flag here refers to a flag indicating a combination of the manipulation instruction content and the photographing mode type. For example, as shown in the table in Figure 7 , there is a case where a flag 1-4 indicating the manipulation instruction content of turning the steering wheel is input, and at the same time, a flag 5-1 indicating the photographing mode content of image rotation is input. In this case, a flag 6-1 corresponding to the combination of the flags 1-4 and 5-1 is generated as the manipulation instruction related information, and the flag 6-1 is a captured transition flag indicating that an image is captured while rotating when the steering wheel is turned. Thereafter, in step S302, the generated flag 6-1 as the manipulation instruction related information is recorded in association with the image data. This allows the user to grasp that a transition has been captured during imaging without checking all the images.

[0078] Although the case where a captured transition flag associated in advance with the input manipulation instruction information and the input photographing mode information exists as the manipulation instruction related information is described in the present embodiment, the captured transition flag may not exist. If these pieces of information do not exist, at least one of an output flag 6-0 (a flag indicating the non-existence of corresponding manipulation instruction related information), the manipulation instruction information, and the photographing mode information is output as the manipulation instruction related information, or nothing is output. When the input manipulation instruction information and the input photographing mode information are not associated with each other in advance, at least one of the output flag 6-0 and the manipulation instruction information is output as the manipulation instruction related information, or nothing is output.

[0079] In addition, in step S301, the information generation module 201 may weight the input manipulation instruction information and generate a weighted result as manipulation instruction related information. For example, as shown in the table in Figure 8 when the manipulation instruction information indicating the degree of accelerator pedal depression is input, flags 7-20, 7-40, and 7-60 indicating the weighted result are generated as manipulation instruction related information. Thereafter, in step S302, the flags 7-20, 7-40, and 7-60 generated as manipulation instruction related information are recorded in association with the corresponding image data. Therefore, when editing an image, the volume of the engine sound to be added can be changed according to the degree of accelerator pedal depression.

[0080] In addition, in step S301, the information generation module 201 may generate an edit point flag pre-associated with the combination of the manipulation instruction information and the surrounding space information obtained by the image recording unit 135 as manipulation instruction related information. The surrounding space information is not particularly limited as long as it is information related to the surrounding space of the moving subject 150 (vehicle body) accompanying the manipulation. For example, the information includes manipulation instruction related information of different moving subjects (other vehicle bodies) accompanying the manipulation in the surrounding space, information indicating the surrounding environment (such as a tunnel or an urban area, etc.) of the moving subject 150, and information indicating the weather (such as rain or snow, etc.) around the moving subject 150.

[0081] The surrounding space information is obtained by a surrounding space information obtaining device (not shown) and received by the information generation module 201 via the communication unit 118. For example, when another car is reflected in the captured image due to a lane change, as shown in the table in Figure 9 a flag 1-1 can be obtained from the moving subject 150, and the information generation module 201 can obtain the manipulation instruction information (flag 8-1) indicating the flashing instruction of the turn signal of the other car. In this case, an edit point flag 9-1 corresponding to the combination of the flags 1-1 and 8-1 is generated as manipulation instruction related information and recorded in association with the image data in step S302. This makes it easy for the user to find the scene where other vehicle bodies may be reflected in the captured image when editing the captured image.

[0082] Although the edit point flag associated in advance with the input manipulation instruction information and the input surrounding space information is described as existing as the manipulation instruction related information in the present embodiment, the edit point flag may not exist. If these information do not exist, at least one of the output flag 9-0 (the flag indicating the non-existence of the corresponding manipulation instruction related information), the manipulation instruction information, and the surrounding space information is output as the manipulation instruction related information, or nothing is output. When the input manipulation instruction information and the input surrounding space information are not associated with each other in advance, at least one of the output flag 9-0 and the manipulation instruction information is output as the manipulation instruction related information, or nothing is output.

[0083] In step S301, when a plurality of manipulation instruction information indicating the same manipulation instruction content are obtained during the imaging by the imaging device 132, the information generation module 201 may change the obtained plurality of identical manipulation instruction information into a combination of an associated set.

[0084] For example, when two different images are captured while a manipulation instruction to turn the steering wheel is issued in the moving object 150, the information generation module 201 obtains two identical manipulation instruction information (flag 1-4). In this case, one of the two identical manipulation instruction information (flag 1-4) is changed to flag 1-4-1, and the other is changed to flag 1-4-2, and these are set as a combination of the set. For flag 1-4-1, a flag indicating the out point for image transition is generated as the manipulation instruction related information, and this flag is the edit point flag associated in advance. For flag 1-4-2, a flag indicating the in point for image transition is generated as the manipulation instruction related information, and this flag is the edit point flag associated in advance.

[0085] Thereafter, in step S302, the flag generated as the manipulation instruction related information is recorded in association with the corresponding image data. This makes it easy for the user to find the timing of the transition and the candidates for the images to be connected by the transition when editing the image.

[0086] In addition, when three or more different images are captured while a manipulation instruction to turn the steering wheel is issued in the moving object 150, the information generation module 201 obtains three or more identical manipulation instruction information (flag 1-4). In this case, the flag first obtained among the three or more identical manipulation instruction information (flag 1-4) by the information generation module 201 is set as flag 1-4-11 (the first manipulation instruction information). On the other hand, the remaining flags are changed to flag 1-4-12 (the second manipulation instruction information), flag 1-4-13 (the third manipulation instruction information),... in the order of the flags obtained by the information generation module 201. Therefore, flag 1-4-11, 1-4-12, and 1-4-13 are set as a combination of the set.

[0087] For flags 1-4-11, a flag indicating the out point for image transition is generated as manipulation instruction related information, and this flag is a pre-associated edit point flag. Similarly, for flags 1-4-12, 1-4-13, ..., a flag indicating the in point of image transition is generated as manipulation instruction related information, and this flag is a pre-associated edit point flag. Thereafter, in step S302, each flag generated as manipulation instruction related information is recorded in association with the corresponding image data. This makes it easy for the user to find candidates for the images to be connected by the transition when editing the image.

[0088] In this embodiment, the case where when the information generation module 201 obtains a plurality of manipulation instruction information indicating the same manipulation instruction content, the plurality of identical manipulation instruction information is changed into a combined set has been described, but this is not restrictive. For example, when there is no manipulation instruction information to be changed into a combined set, the corresponding manipulation instruction related information is output without changing the manipulation instruction information.

[0089] The above is the content of the process of generating manipulation instruction related information. With the above configuration, the user can easily edit the image according to the manipulation instruction related information while referring to the image data.

[0090] In this embodiment, the case where the image recording unit 135 as the information processing device is located inside the mobile imaging device 100 has been described, but the image recording unit 135 may be located inside a server (not shown) that can communicate with the mobile imaging device 100 and the mobile subject 150. When the image recording unit 135 is inside the server, the manipulation instruction information from the mobile subject 150, the image data information captured by the imaging device 130, and the imaging mode information using the controller 111, etc. are transmitted to the image recording unit 135 inside the server.

[0091] Similarly in this embodiment, the image recording unit 135 as the information processing device may be located inside the mobile subject 150. In this case, the image data captured by the imaging device 130 and the imaging mode information using the controller 111, etc. are transmitted to the mobile subject 150.

[0092] Other embodiments

[0093] Embodiments of the present invention can also be implemented by the following method, that is, software (program) that executes the functions of the above embodiments is provided to a system or device through a network or various storage media, and the computer or central processing unit (CPU), microprocessing unit (MPU) of the system or device reads and executes the program.

[0094] Although the present invention has been described with reference to exemplary embodiments, it should be understood that the present invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be accorded the broadest interpretation so as to encompass all such modifications as well as equivalent structures and functions.

[0095] This application claims the benefit of Japanese Patent Application No. 2023-200675, filed on Nov. 28, 2023, the entire contents of which are incorporated herein by reference.

Claims

1. An information processing device, comprising: A generating component, used for obtaining manipulation instruction information related to a moving subject accompanied by manipulation, and generating manipulation instruction related information corresponding to the manipulation instruction information; as well as A recording means for recording, whenever photographing data generated by the photographing device by photographing the image of the moving subject is obtained from the photographing device mounted on the mobile photographing device, the photographing data and the manipulation instruction related information generated by the generating means are associated with each other.

2. The information processing device according to claim 1, wherein: The manipulation instruction related information is the manipulation instruction information.

3. The information processing device according to claim 1, wherein: The manipulation instruction related information is an edit point flag for indicating any one of an edit point for an image and an edit point for a sound.

4. The information processing device according to claim 1, wherein: The manipulation instruction related information is a transition edit point flag for indicating any one of an edit point for image transition and an edit point for sound transition.

5. The information processing device according to claim 1, wherein: The manipulation instruction related information is a recommended transition flag for indicating any one of a recommended transition for image transition and a recommended transition for sound transition.

6. The information processing device according to claim 1, wherein: The manipulation instruction information is a flag represented by multiple values.

7. The information processing device according to claim 1, wherein: The manipulation instruction related information is a flag corresponding to a combination of the manipulation instruction information and information related to a photographing method in the mobile camera device.

8. The information processing device according to claim 1, wherein: The manipulation instruction related information is a flag corresponding to a combination of the manipulation instruction information and surrounding space information related to the surrounding space of the moving subject.

9. The information processing device according to claim 8, wherein: The surrounding space information related to the surrounding space of the moving subject is manipulation instruction information of another subject different from the moving subject.

10. The information processing device according to claim 8, wherein: The surrounding space information related to the surrounding space of the moving subject is information indicating the surrounding environment of the moving subject.

11. The information processing device according to claim 8, wherein: The surrounding space information related to the surrounding space of the moving subject is information indicating the weather around the moving subject.

12. The information processing device according to any one of claims 1 to 11, wherein: The camera device includes a camera device and a sound acquisition device, and The photographing data is at least one of image data photographed by the imaging device and sound data acquired by the sound acquisition device.

13. The information processing device according to any one of claims 1 to 11, wherein: The generating means changes the obtained plurality of identical manipulation instruction information into information related to a combination of the associated sets.

14. The information processing device according to claim 13, wherein: The generating component changes one of the two identical manipulation instruction information obtained into manipulation instruction information associated with manipulation instruction related information indicating an entry point of an editing point, and changes the other of the two identical manipulation instruction information obtained into manipulation instruction information associated with manipulation instruction related information indicating an exit point of the editing point.

15. The information processing device according to claim 13, wherein: The generating component changes the earliest obtained manipulation instruction information among the three identical manipulation instruction information obtained into the first manipulation instruction information associated with the manipulation instruction related information indicating the entry point of the editing point, and changes the remaining two identical manipulation instruction information into the second manipulation instruction information and the third manipulation instruction information associated with the manipulation instruction related information indicating the exit point of the editing point in the order of obtaining.

16. An information processing device according to any one of claims 1 to 11, 14 and 15, wherein the information processing device is located in one of the following three locations: inside the mobile camera device; inside the mobile subject; and inside a server that can be communicatively connected to the mobile camera device and the mobile subject.

17. A control method for an information processing device, the control method comprising: a generating step for obtaining manipulation instruction information related to a moving subject accompanied by manipulation, and generating manipulation instruction related information corresponding to the manipulation instruction information; as well as A recording step for recording the shooting data and the manipulation instruction related information generated in the generating step in association with each other whenever shooting data generated by the shooting device by shooting the image of the moving subject is obtained from the shooting device mounted on the moving shooting device.

18. A computer program product comprising a computer program for causing a computer to execute the control method according to claim 17.

19. A computer-readable storage medium storing a computer program for causing a computer to execute the control method according to claim 17.

Images