Image file, information processing device, imaging device, and generation method

Abstract

An image file according to the present invention is provided with: moving picture data including a frame group; and accompanying information. The accompanying information includes first accompanying information and second accompanying information. The first accompanying information relates to all the frames included in the frame group, and the second accompanying information relates to a partial plurality of frames in the frame group.

Description

Image file, information processing device, imaging device, and generation method

[0001] The technology of the present disclosure relates to an image file, an information processing device, an imaging device, and a generation method.

[0002] Japanese Patent Publication No. 2021-177317 discloses a three-dimensional model construction system. The three-dimensional model construction system described in Japanese Patent Publication No. 2021-177317 includes a management means, a collection means, a generation means, and an assignment means. The management means associates and manages a plurality of moving objects with users. The collection means collects shooting data including incidental information related to the shooting, captured by a shooting device provided on each of the plurality of moving objects. The generation means generates a three-dimensional model using the shooting data collected by the collection means. The assignment means assigns a reward to the user associated with the moving object that captured the shooting data, depending on the shooting data used when the generation means generates the three-dimensional model. The generation means also uses the incidental information included in the new shooting data to identify a shooting area of ​​an existing three-dimensional model corresponding to the new shooting data, and updates the existing three-dimensional model using the new shooting data depending on the difference between the existing three-dimensional model and a three-dimensional model generated from the new shooting data in the shooting area.

[0003] One embodiment of the technique of the present disclosure provides an image file, an information processing device, an imaging device, and a generation method that allow the contents of the image file to be understood without playing the image file.

[0004] A first aspect of the technology of the present disclosure is an image file having the following configuration: The image file according to the first aspect comprises video data including a frame group and incidental information, the incidental information including first incidental information and second incidental information, the first incidental information being information relating to all frames included in the frame group, and the second incidental information being information relating to a portion of a plurality of frames in the frame group.

[0005] A second aspect of the technology of the present disclosure is a generation method for generating an image file including moving image data including a group of frames and additional information, the generation method comprising: an acquisition step for acquiring moving image data; a first assignment step for assigning first additional information, which is information about all frames included in the group of frames, to the image file as additional information; and a second assignment step for assigning second additional information, which is information about a portion of a plurality of frames included in the group of frames, to the image file as additional information.

[0006] 1 is a conceptual diagram showing an example of how an imaging device is used. FIG. 2 is a block diagram showing an example of the hardware configuration and main functions of the electrical system of an imaging device. FIG. 3 is a conceptual diagram showing an example of processing content for acquiring video image data, processing content for acquiring image recognition result information, and processing content for acquiring frame-related information. FIG. 4 is a conceptual diagram showing an example of processing content for acquiring a period-based complaint-related information group, processing content for acquiring scene information, and processing content for acquiring partial multiple-frame-related information. FIG. 5 is a conceptual diagram showing an example of processing content for acquiring summary information and processing content for acquiring overall-related information. FIG. 6 is a conceptual diagram showing an example of processing content for generating a video file including video image data and metadata. FIG. 7 is a conceptual diagram showing an example of how frame-related information, partial multiple-frame-related information, and overall-related information are saved (recorded) as meta-information in a video file. FIG. 8 is a conceptual diagram showing an example of processing content for adding file-related information to a video file. FIG. 9 is a conceptual diagram showing an example of the structure of image recognition result information. FIG. 10 is a conceptual diagram showing an example of processing content for adding partial multiple-frame-related information to a video file. FIG. 11 is a conceptual diagram showing an example of processing content for adding overall-related information to a video file.

[0033] FIG. 1 is a conceptual diagram showing an example of processing details for including subject change information in frame-related information, and processing details for including representative subject change information in some of the frame-related information, in an imaging device according to a first modified example.

[0034] FIG. 1 is a conceptual diagram showing an example of processing details for acquiring geometric information, in an imaging device according to a second modified example.

[0035] FIG. 1 is a conceptual diagram showing an example of processing details for including geometric change information in frame-related information, and processing details for including representative geometric change information in some of the frame-related information, in an imaging device according to a third modified example.

[0036] FIG. 1 is a conceptual diagram showing an example of processing details for including angle of view change information in frame-related information, and processing details for including representative angle of view change information in some of the frame-related information, in an imaging device according to a fourth modified example.

[0033] FIG. 10 is a conceptual diagram showing an example of processing for including pan / tilt information in frame-related information and for including representative pan / tilt information in some of the multiple-frame-related information in an imaging device according to a fifth modified example.

[0034] FIG. 11 is a conceptual diagram showing an example of processing for including inertial sensor signal difference information in frame-related information and for including representative inertial sensor signal difference information in some of the multiple-frame-related information in an imaging device according to a sixth modified example.

[0035] FIG. 12 is a conceptual diagram showing an example of processing for including recording time-related information in frame-related information and for including representative recording time-related information in some of the multiple-frame-related information in an imaging device according to a seventh modified example.

[0036] FIG. 13 is a conceptual diagram showing an example of processing for including operation-related information in frame-related information and for including representative operation-related information in some of the multiple-frame-related information in an imaging device according to an eighth modified example.

[0037] FIG. 14 is a conceptual diagram showing an example of processing for acquiring subject sensor information in an imaging device according to a ninth modified example.

[0038] FIG. 15 is a conceptual diagram showing an example of processing for including subject sensor information in frame-related information and for including representative subject sensor information in some of the multiple-frame-related information in an imaging device according to a ninth modified example.

[0039] FIG. 16 is a conceptual diagram showing an example of processing for including separate video file information in the overall-related information in an imaging device according to a tenth modified example. FIG. 10 is a conceptual diagram showing an example of a mode in which an external device is caused to execute an image file creation process.

[0007] Hereinafter, exemplary embodiments of an image file, an information processing device, an imaging device, and a generation method according to the techniques of the present disclosure will be described with reference to the accompanying drawings.

[0008] As an example, as shown in Fig. 1, an imaging device 10 captures an image of an imaging target area 12 designated as a subject. The range of the imaging target area 12 is determined by the angle of view designated by a user of the imaging device 10 (hereinafter referred to as "user"). In the example shown in Fig. 1, the imaging target area 12 includes people 12A to 12D and a road 12E. The imaging device 10 is an example of an "imaging device" according to the technology of the present disclosure.

[0009] 1, the imaging device 10 is a consumer-use digital camera with an interchangeable lens. The imaging device 10 includes an imaging device body 14 and an interchangeable lens 16. The interchangeable lens 16 is interchangeably attached to the imaging device body 14. Note that, although an interchangeable lens digital camera is given as an example of the imaging device 10 here, this is merely an example, and the imaging device 10 may also be a digital camera with a fixed lens.

[0010] Note that, although a consumer digital camera is illustrated here as an example of the imaging device 10, this is merely an example, and the technology of the present disclosure is also applicable when the imaging device 10 is an industrial digital camera. Furthermore, the technology of the present disclosure is also applicable when the imaging device 10 is an imaging device mounted on various electronic devices such as a drive recorder, a smart device, a wearable terminal, a cell observation device, an ophthalmic observation device, or a surgical microscope. Furthermore, the technology of the present disclosure is also applicable when the imaging device 10 is mounted on various modalities such as an endoscope device, an ultrasound diagnostic device, an X-ray imaging device, a CT (Computed Tomography) device, or an MRI (Magnetic Resonance Imaging) device.

[0011] This also applies to imaging devices mounted on vehicles.

[0012] The imaging device 10 is provided with a UI (User Interface) device 18. The UI device 18 is a device that accepts operations on the imaging device 10 and presents (e.g., displays) various information to the user. In the example shown in Fig. 1, the UI device 18 has a release button 20, a dial 22, a touch panel display 24, and instruction keys 26 that accept various instructions.

[0013] The release button 20 functions as an image capture preparation instruction unit and an image capture instruction unit, and issues image capture preparation instructions (for example, AF (Auto Focus) and AE (Auto Exposure) instructions) to the image capture device 10 and

[0014] This button is used to give an instruction to take a photograph.

[0015] The dial 22 is operated when setting an operation mode, etc. In the imaging device 10, various operation modes are selectively set by operating the dial 22. The operation modes include operation modes of the imaging system. Examples of operation modes of the imaging system include a still image imaging mode and a moving image imaging mode.

[0016] The touch panel display 24 includes a display 28 and a touch panel 30. Examples of the display 28 include a liquid crystal display or an EL (Electroluminescent Display) display. The display 28 displays a live view image, a still image, a moving image, a menu screen, and the like. In the example shown in FIG. 1 , a live view image is displayed on the display 28.

[0017] The imaging device 10 captures images of the imaging target area 12 at a moving image frame rate to generate moving image data 32 that show images of the imaging target area 12. The moving image data 32 includes a frame group 34. The frame group 34 is made up of a plurality of frames 36 (i.e., a plurality of image data arranged in chronological order). The type of frames 36 is, for example, visible light image data obtained by capturing images of the imaging target area 12 in the visible light range. However, the type of frames 36 is not limited to this, and may also be non-visible light image data obtained by capturing images in a wavelength range other than the visible light range.

[0018] The moving image data 32 is an example of "moving image data" according to the technology of the present disclosure. The frame group 34 is an example of "frame group" according to the technology of the present disclosure. The frame 36 is an example of "frame" according to the technology of the present disclosure.

[0019] 2, the imaging device 10 includes a UI device 18, an information processing device 38, an image sensor 40, and a communication I / F (Interface) 42. The information processing device 38 is an example of an "information processing device" according to the technology of the present disclosure.

[0020] The information processing device 38 includes a processor 44, a non-volatile memory (NVM) 46, and a random access memory (RAM) 48. The processor 44, the NVM 46, and the RAM 48 are connected to a bus 50. The processor 44 is an example of a "processor" according to the technology of the present disclosure.

[0021] The processor 44 is a processing device including a DSP (Digital Signal Processor), a CPU (Central Processing Unit), and a GPU (Graphics Processing Unit), and the DSP and GPU operate under the control of the CPU and are responsible for executing image processing. Here, a processing device including a DSP, a CPU, and a GPU is given as an example of the processor 44, but this is merely an example, and the processor 44 may be one or more CPUs and DSPs with integrated GPU functionality, one or more CPUs and DSPs without integrated GPU functionality, or may be equipped with a TPU (Tensor Processing Unit).

[0022] The NVM 46 is a non-volatile storage device that stores various programs, various parameters, etc. The NVM 46 may be, for example, a flash memory (for example, an EEPROM (Electrically Erasable and Programmable Read Only Memory)).

[0023] The RAM 48 is a memory that temporarily stores information and is used as a work memory by the processor 44. The RAM 48 may be, for example, a dynamic random access memory (DRAM) or a static random access memory (SRAM).

[0024] The image sensor 40 is connected to the bus 50. An example of the image sensor 40 is a complementary metal oxide semiconductor (CMOS) image sensor. Under the control of the processor 44, the image sensor 40 captures an image of the imaging target area 12 (see FIG. 1 ) to generate moving image data 32.

[0025] Here, a CMOS image sensor is given as an example of the image sensor 40, but this is merely an example, and the image sensor 40 may be other types of image sensors such as a CCD (Charge Coupled Device) image sensor.

[0026] is an example of a "sensor" according to the technology of the present disclosure.

[0027] The UI device 18 is connected to the bus 50. The UI device 18 receives instructions from the user and outputs a signal indicating the received instructions to the processor 44. The UI device 18 also presents various types of information to the user under the control of the processor 44. The presentation of the various types of information is realized, for example, by a display 28 (see FIG. 1 ) and a speaker (not shown).

[0028] The communication I / F 42 is an interface including a communication processor, an antenna, etc., and is connected to the bus 50. The communication standard applied to the communication I / F 42 is a wireless communication standard including, for example, 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0029] The imaging device 10 captures images in a video imaging mode to generate a video file containing video data 32. To confirm the contents of the video data 32 contained in the video file, the video file must be played back. In this case, for example, to understand the overall structure of the video data 32, most of the video data 32 must be played back, and the greater the number of frames 36 contained in the video data 32, the longer the playback time. Furthermore, to understand the structure of a portion of the frames 36 contained in the video data 32, for example, the corresponding portion of the frames 36 must be played back, which also requires playback time. Not only does this require playback time, but it also takes time to identify the corresponding portion of the frames 36 from the frame group 34.

[0030] In consideration of these circumstances, in the imaging device 10, an image file creation process is performed by the processor 44, as shown in FIG. 2 as an example. The image file creation process is a process for creating an image file. The image file created by the image file creation process has additional information (e.g., metadata), which will be described later. In the imaging device 10, an image file creation program 52 is stored in the NVM 46. The processor 44 reads the image file creation program 52 from the NVM 46 and executes the read image file creation program 52 on the RAM 48, thereby performing the image file creation process. The image file creation process is realized by the processor 44 operating as an acquisition unit 44A, a generation unit 44B, a first assignment unit 44C, a second assignment unit 44D, and a control unit 44E in accordance with the image file creation program 52 executed on the RAM 48.

[0031] The process performed by the acquisition unit 44A is an example of an "acquisition step" according to the technology of the present disclosure. The process performed by the first assignment unit 44C is an example of a "first assignment step" according to the technology of the present disclosure. The process performed by the second assignment unit 44D is an example of a "second assignment step" according to the technology of the present disclosure.

[0032] 3 as an example, the acquisition unit 44A acquires moving image data 32 from the image sensor 40. The acquisition unit 44A performs AI (Artificial Intelligence) image recognition processing on each frame 36 included in a frame group 34 of the moving image data 32, thereby acquiring image recognition result information 54 to be recorded as metadata. The image recognition result information 54 is information indicating the results obtained by performing the AI ​​image recognition processing.

[0033] 3, information about people 12A and 12B appearing in frame 36 and scene information 54A are included in image recognition result information 54. In image recognition result information 54, an identifier ("#1" to "#N" in the example shown in FIG. 3) is assigned to each subject recognized by the image recognition process. Subjects recognized by the image recognition process are not limited to a single subject, but also include combinations of multiple subjects.

[0034] In the example shown in Fig. 3, the information about the person 12A includes coordinates (i.e., two-dimensional coordinates) that can identify the position of the person 12A within the frame 36, the type of the person 12A (in the example shown in Fig. 3, "human", "young girl", and "name (in the example shown in Fig. 3, the name "Fuji Hanae")"), and the attribute of the person 12A (in the example shown in Fig. 3, "running"). In addition, in the example shown in Fig. 3, the information about the person 12B includes coordinates (i.e., two-dimensional coordinates) that can identify the position of the person 12B within the frame 36, the type of the person 12B (in the example shown in Fig. 3,

[0035] The following are shown: "Human", "Middle-aged woman", and "Name (in the example shown in FIG. 3, the name is "Fuji Masae")", as well as the attributes of person 12B (in the example shown in FIG. 3, "Crouching").

[0036] The scene information 54A is information about a scene identified from the frame 36. The scene identified from the frame 36 is, for example, a combination of the person 12A and the person 12B. In the example shown in FIG. 3, the scene information 54A indicates information such as "Hanae and Masae are playing." Here, the combination of the person 12A and the person 12B is illustrated as a scene identified from the frame 36, but this is merely an example. For example, the aspect of one subject (e.g., a combination of the person 12A, the person 12B, or the person 12A-12D with a landscape) captured in the frame 36 may be applied as a scene, and the scene obtained will vary depending on the specifications of the AI-based image recognition processing or the results obtained by performing the AI-based image recognition processing.

[0037] Although AI-based image recognition processing is exemplified here, the technology of the present disclosure is not limited thereto, and template matching-based image recognition processing may be applied, or AI-based image recognition processing and template matching-based image recognition processing may be used in combination. Furthermore, one or more labels may be assigned to a subject captured in the frame 36 in response to an external instruction given to the imaging device 10 (e.g., an instruction received by the UI device 18). Furthermore, the assigned one or more labels may be used as the image recognition result information 54. Examples of labels include coordinates that can identify the position of the subject within the frame 36, the type of subject, and attributes of the subject.

[0038] The acquisition unit 44A acquires frame-related information 56 for each frame 36 included in the frame group 34 of the moving image data 32. The frame-related information 56 includes image recognition result information 54 corresponding to the frame 36 and frame basic information 58 corresponding to the frame 36. The frame-related information 56 is an example of "information relating to one frame included in the frame group" according to the technology of the present disclosure.

[0039] The frame basic information 58 includes a frame identifier, model information, lens information, date and time information, and imaging condition information. The frame identifier is an identifier that can identify the frame 36. The model information is information that indicates the model of the imaging device body 14. The lens information is information that indicates the type and specifications of the interchangeable lens 16 (see FIG. 1). The date and time information is the date and time (e.g., time expressed in year, month, day, hour, minute, and second) that the frame 36 corresponding to the frame-related information 56 was obtained. The imaging condition information is information that indicates the imaging conditions set for the imaging device 10 (e.g., F-number, shutter speed, sensitivity of the image sensor 40, 35mm equivalent focal length, and on / off of image stabilization).

[0040] As an example, as shown in FIG. 4 , the acquisition unit 44A acquires all frame-related information 56 associated with a frame group 34 of the moving image data 32 by dividing it into multiple frame-related information groups 60. The multiple frame-related information groups 60 are obtained by dividing all frame-related information 56 for multiple specified periods (first period to Nth period in the example shown in FIG. 4 ). In other words, the frame-related information group 60 is a collection of multiple frame-related information 56. Note that the multiple periods may be determined according to an instruction received by the UI-based device 18. Alternatively, the multiple periods may be determined according to a predetermined algorithm. In this case, for example, all frame-related information 56 associated with the frame group 34 may be divided at time intervals that are uniquely determined depending on the number of all frames 36 included in the frame group 34.

[0041] The acquisition unit 44A acquires scene information 61 from the frame-related information group 60 and the corresponding frame group 34. The scene information 61 is information about a scene identified from the plurality of frames 36 with which the frame-related information group 60 is associated. Examples of scenes identified from the plurality of frames 36 include picnic lunch, picnic cooking, and picnic fishing. The scene information 61 is expressed, for example, as text that can identify the scene, a mark that can identify the scene (e.g., a pictogram), a thumbnail image that can identify the scene (e.g., a thumbnail still image or a thumbnail moving image), a combination of text and a mark, a combination of text and a thumbnail image, a combination of a thumbnail image and a mark, or a combination of text, a mark, and a thumbnail image.

[0042] The scene information 61 is obtained using a neural network (NN) 62, which is a type of mathematical model. The NN 62 is trained using a plurality of training data sets including information corresponding to a plurality of frame-related information groups 60 and correct answer data (here, as an example, information relating to a plurality of scenes), and scores relating to various scenes are output from the NN 62. The acquisition unit 44A inputs the frame-related information groups 60 to the NN 62. The acquisition unit 44A identifies a scene by referring to the score output from the NN 62 as a result of the frame-related information groups 60 being input to the NN 62, and acquires information relating to the identified scene as scene information 61.

[0043] Although the above description has been given with reference to an example in which a scene is identified from a plurality of frames 36 associated with the frame-related information group 60 using the NN 62, this is merely an example. For example, a scene determined for each period may be applied in accordance with an instruction given by a user to the imaging device 10 via the UI device 18 or a device directly or indirectly connected to the imaging device 10.

[0044] The acquisition unit 44A acquires partial multiple frame related information 64 for each frame related information group 60. The partial multiple frame related information 64 includes scene information 61 identified from the corresponding frame related information group 60 and period-specific basic information 66. The partial multiple frame related information 64 is an example of "information relating to a portion of a plurality of frames of the frame group" according to the technology of the present disclosure.

[0045] The period-specific basic information 66 includes a frame identifier, model information, lens information, time information, imaging condition information, etc. The frame identifier included in the period-specific basic information 66 is an identifier capable of identifying a frame 36 that contributed to obtaining the scene information 61 at a certain level or above. An example of a frame 36 that contributed to obtaining the scene information 61 at a certain level or above is a frame 36 associated with frame-related information 56 whose score output from the NN 62 is a certain value or above. The model information included in the period-specific basic information 66 is model information statistically obtained from the plurality of frame-related information 56 included in the frame-related information group 60 (e.g., the model information most frequently included in the plurality of frame-related information 56). The lens information included in the period-specific basic information 66 is lens information statistically obtained from the plurality of frame-related information 56 included in the frame-related information group 60 (e.g., the lens information most frequently included in the plurality of frame-related information 56).

[0046] The time information included in the period-based basic information 66 is information that includes two pieces of date and time information included in the first frame-related information 56 and the last frame-related information 56 of the frame-related information group 60. Furthermore, the time information included in the period-based basic information 66 may be two frame identifiers included in the first frame-related information 56 and the last frame-related information of the frame-related information group 60. Furthermore, the time information included in the period-based basic information 66 may be information that indicates a time zone calculated from the date and time information included in the first frame-related information 56 of the frame-related information group 60 and the date and time information included in the last frame-related information 56 of the frame-related information group 60.

[0047] The imaging condition information included in the period-specific basic information 66 is imaging condition information statistically obtained from the multiple frame-related information 56 included in the frame-related information group 60 (for example, the imaging condition information that is most frequently included in the multiple frame-related information 56).

[0048] 5 , the acquiring unit 44A acquires summary information indicating an outline of the video data 32. The summary refers to a title expressing the content shown in the video data 32, a sentence expressing the content shown in the video data 32, a mark expressing the content shown in the video data 32, a thumbnail image expressing the content shown in the video data 32, or the like.

[0049] In the example shown in FIG. 5 , time-slot summary information 68 and overall summary information 70 are given as examples of summary information. The time-slot summary information 68 is information indicating a summary of each time slot within the video data 32. The time-slot summary is, for example, a summary of each video data segment obtained by equally dividing the video data 32, or a summary of each video data segment obtained by dividing the video data 32 into the first to Nth periods shown in FIG. 4 . The overall summary information 70 is information indicating a summary of the entire video data 32. The time-slot summary information 68 and the overall summary information 70 are identified, for example, from multiple pieces of partial multi-frame related information 64. Hereinafter, when there is no need to distinguish between the time-slot summary information 68 and the overall summary information 70, they will be referred to as "summary information" without being assigned reference numerals.

[0050] The summary information is obtained using the NN 72. The NN 72 is trained using a plurality of training data sets including information corresponding to a plurality of pieces of partial multiple frame-related information 64 and correct answer data (here, as an example, information related to a plurality of summaries), and scores related to various summaries are output from the NN 72. The acquisition unit 44A inputs one or more pieces of partial multiple frame-related information 64 to the NN 72. The acquisition unit 44A identifies a summary by referring to the score output from the NN 72 in response to the input of one or more pieces of partial multiple frame-related information 64 to the NN 72, and acquires the information related to the identified summary as summary information. The summary information is an example of "information related to the configuration of video image data" according to the technology of the present disclosure.

[0051] Here, we have given an example of a form in which NN72 is trained using multiple training data including information corresponding to multiple partial frame-related information 64 and correct answer data, but this is merely one example, and the training data may also include information corresponding to multiple frame-related information 56.

[0052] Although the embodiment in which the overview is determined using the NN 62 has been described above, this is merely an example. For example, an overview determined in accordance with an instruction given by a user to the imaging device 10 via the UI device 18 or a device directly or indirectly connected to the imaging device 10 may be applied.

[0053] Furthermore, although summary information is exemplified here, this is merely one example. For example, instead of or together with the summary information, information indicating the types of all subjects appearing in the video data 32, information indicating geographical features identified from the scenery appearing in the video data 32, or information indicating common subjects appearing in the video data 32 may be applied. In this way, the information applied instead of or together with the summary information may be any information related to the configuration of the video data 32.

[0054] The acquisition unit 44A acquires the overall related information 74. The overall related information 74 includes the time period summary information 68, the overall summary information 70, and the overall basic information 76.

[0055] The overall basic information 76 includes a frame identifier, model information, lens information, time information, and imaging condition information.

[0056] It includes information, video identifiers, etc.

[0057] The frame identifiers included in the overall basic information 76 are identifiers that can identify frames 36 that contributed at a certain level or more to obtaining the summary information. An example of a frame 36 that contributed at a certain level or more to obtaining the summary information is a frame 36 associated with partial multiple frame-related information 64 whose score output from the NN 72 is equal to or greater than a certain value. The model information included in the overall basic information 76 is model information statistically obtained from the plurality of partial multiple frame-related information 64 (e.g., model information most frequently included in the plurality of partial multiple frame-related information 64). The lens information included in the overall basic information 76 is lens information statistically obtained from the plurality of partial multiple frame-related information 64 (e.g., lens information most frequently included in the plurality of partial multiple frame-related information 64). The time information included in the overall basic information 76 is the start time and end condition of the video data 32. Furthermore, the time information included in the overall basic information 76 may be the time required to play the video data 32.

[0058] The imaging condition information included in the overall basic information 76 is imaging condition information statistically obtained from a plurality of pieces of partial frame-related information 64 (for example, imaging condition information that is most frequently included in a plurality of pieces of partial frame-related information 64). The video identifier included in the overall basic information 76 is an identifier that is uniquely assigned to the entire video data 32. Note that the overall basic information 76 may also include other basic information related to the entire video data 32, such as the bit rate of the video data 32 or the codec used for encoding and decoding the video data 32.

[0059] As an example, as shown in Fig. 6, the generation unit 44B generates a video file 78. The video file 78 has metadata 80. The metadata 80 is data related to the video data 32 (i.e., data incidental to the video data 32). Here, the video file is an example of an "image file" according to the technology of the present disclosure. The metadata 80 is an example of "accompanying information" according to the technology of the present disclosure.

[0060] 7 , a plurality of pieces of frame-related information 56 are stored (recorded) as metadata 80 in a moving image file 78. In addition, a plurality of pieces of partial multi-frame-related information 64 (i.e., information including scene information 61 and period-specific basic information 66) extracted from the plurality of pieces of frame-related information 56 are also stored as metadata 80 in the moving image file 78. Furthermore, overall-related information 74 (i.e., information including time-zone summary information 68, overall summary information 70, and overall basic information 76) extracted from the plurality of pieces of partial multi-frame-related information 64 are also stored as metadata 80 in the moving image file 78.

[0061] As an example, as shown in FIG. 8 , the second assigning unit 44D assigns the frame-related information 56 to the video file 78 by including the frame-related information 56 acquired for each frame 36 by the acquiring unit 44A in the metadata 80. In this case, as shown in FIG. 9 , the image recognition result information 54 included in the frame-related information 56 assigned to the video file 78 is classified into multiple categories. For example, a subject identifier (in the example shown in FIG. 9 ), which is a unique identifier, is assigned to each subject (e.g., person 12A, person 12B, and each combination of person 12A and person 12B) appearing in the frame 36. Then, multiple categories, such as a type category, an attribute category, and a position category, are assigned to each subject identifier. In the example shown in FIG. 9 , multiple categories are arranged hierarchically for each type category and attribute category. Categories that are subordinate concepts or derivative concepts of the higher level are provided at lower levels. In the example shown in FIG. 9, information about the person 12A is assigned to "#1," and information about the person 12B is assigned to "#2."

[0062] The type category is a category that indicates the type of subject. In the example shown in FIG. 9 , "human" is assigned to the type category as the type of subject. A gender category and a name category are provided at a lower level than the type category. The gender category is a category that indicates gender, and the name category is a category that indicates the name of the subject (for example, a common noun or a proper noun, etc.).

[0063] The attribute category is a category that indicates the attribute of the subject. In the example shown in FIG. 9 , an action category, an expression category, and a clothing category are provided, and a color category is provided as a lower level of the clothing category. The action category is a category that indicates the action of the subject. The expression category is a category that indicates the expression of the subject. The clothing category is a category that indicates the type of clothing worn by the subject. The color category is a category that indicates the color of the clothing worn by the subject.

[0064] The position category is a category that indicates the position of the subject within the frame 36. In the example shown in Fig. 9, coordinate information that can identify the position of person 12A is assigned to "#1," and coordinates that can identify the position of person 12B are assigned to "#2." The coordinate information is, for example, the two-dimensional coordinates of the upper left corner and the two-dimensional coordinates of the lower right corner of a bounding box (i.e., a rectangular frame that surrounds the subject shown in the frame 36) obtained by AI-based image recognition processing.

[0065] In the moving image file 78, the various pieces of information contained in the frame basic information 58 may also be classified into a hierarchy by category in the same manner as the image recognition result information 54.

[0066] 10 , the second assigning unit 44D assigns the plurality of pieces of partial frame-related information 64 to the video file 78 by including the plurality of pieces of partial frame-related information 64 acquired by the acquiring unit 44A in the metadata 80. Note that the plurality of pieces of frame-related information 56 and the plurality of pieces of partial frame-related information 64 included in the metadata 80 are an example of “second incidental information” according to the technology of the present disclosure.

[0067] 11 , the first assigning unit 44C assigns the overall related information 74 acquired by the acquiring unit 44A to the video file 78 by including the overall related information 74 in the metadata 80. The overall related information 74 included in the metadata 80 is an example of the “first auxiliary information” and “information regarding all frames included in a frame group” according to the technology of the present disclosure.

[0068] As an example, as shown in FIG. 12 , the metadata 80 has a hierarchical structure. The hierarchical structure has an upper layer, a middle layer, and a lower layer. The upper layer stores information about the entire frame group 34. The middle layer stores information about a plurality of frames 36 obtained by dividing the frame group 34 into periods as shown in FIG. 4. The lower layer stores information about the frames 36. In FIG. 12 , the upper layer stores overall related information 74, the middle layer stores partial multi-frame related information 64, and the lower layer stores frame related information 56. That is, a plurality of pieces of partial multi-frame related information 64 exist in the layer immediately below the overall related information 74, and a plurality of pieces of frame related information 56 exist in the layer immediately below the partial multi-frame related information 64. This hierarchical structure of the metadata 80 makes it easier for users to identify information within the metadata 80.

[0069] 13, in the imaging device 10, the control unit 44E stores the moving image file 78 obtained as described above in the NVM 46. Note that, although the example shown in FIG. 13 shows an example in which the moving image file 78 is stored in the NVM 46, this is merely an example.

[0070] is merely an example, and the moving image file 78 may be stored in one or more storage media other than the NVM 46. The storage medium may be any medium that is connected to the imaging device 10 directly or indirectly via a wired or wireless method, etc. Examples of storage media include a DVD (Digital Versatile Disc), a USB (Universal Serial Bus) memory, an SSD (Solid State Drive), an HDD (Hard Disk Drive), and a magnetic tape drive.

[0071] Next, the operation of the imaging device 10 will be described with reference to Fig. 14. The processing flow shown in the flowchart of Fig. 14 is an example of a "generation method" according to the technique of the present disclosure.

[0072] 14, first, in step ST10, the acquisition unit 44A acquires moving image data 32 from the image sensor 40 (see FIG. 3). After the processing of step ST10 is executed, the image file creation processing proceeds to step ST12.

[0073] In step ST12, the acquisition unit 44A performs AI image recognition processing on each frame 36 included in the frame group 34 of the video data 32 acquired in step ST10, thereby acquiring image recognition result information 54 (see FIG. 3 ). After the processing of step ST12 is executed, the image file creation processing proceeds to step ST14.

[0074] In step ST14, the acquisition unit 44A acquires frame-related information 56 for each frame 36 included in the frame group 34 of the moving image data 32 acquired in step ST10 (see FIG. 3). The frame-related information 56 includes image recognition result information 54 and frame basic information 58 (see FIG. 3). After the processing of step ST14 is executed, the image file creation processing proceeds to step ST16.

[0075] In step ST16, the acquisition unit 44A acquires a plurality of frame-related information groups 60 from all of the frame-related information 56 associated with the frame groups 34 of the moving image data 32 acquired in step ST10 (see FIG. 4). After the processing of step ST16 is executed, the image file creation processing proceeds to step ST18.

[0076] In step ST18, the acquisition unit 44A acquires scene information 61 for each frame-related information group 60 acquired in step ST16 (see FIG. 4). The scene information 61 is obtained by inputting the frame-related information group 60 to the NN 62 and utilizing the score output from the NN 62. After the processing of step ST18 is executed, the image file creation processing proceeds to step ST20.

[0077] In step ST20, the acquisition unit 44A acquires partial multiple frame related information 64 for each frame related information group 60 acquired in step ST16 (see FIG. 4). The partial multiple frame related information 64 includes scene information 61 and basic information by period 66 (see FIG. 4). After the process of step ST20 is executed, the image file creation process proceeds to step ST22.

[0078] In step ST22, the acquisition unit 44A acquires summary information (see FIG. 5). The summary information is obtained by inputting the partial multi-frame related information 64 acquired in step ST20 into the NN 72 and utilizing the score output from the NN 72. After the process of step ST22 is executed, the image file creation process proceeds to step ST24.

[0079] In step ST24, the acquisition unit 44A acquires overall related information 74 (see FIG. 5). The overall related information 74 includes summary information and overall basic information 76 (see FIG. 5). After the process of step ST22 is executed, the image file creation process proceeds to step ST26.

[0080] In step ST26, the generation unit 44B generates a moving image file 78 including the moving image data 32 acquired in step ST10 and the metadata 80 (see FIG. 6). After the process of step ST26 is executed, the image file creation process proceeds to step ST28.

[0081] In step ST28, the second adding unit 44D adds the frame-related information 56 acquired in step ST14 to the video file 78 by including the frame-related information 56 in the metadata 80 (see FIG. 8 ). After the processing of step ST28 is executed, the image file creation processing proceeds to step ST30.

[0082] In step ST30, the second adding unit 44D adds the partial multiple frame related information 64 acquired in step ST20 to the video file 78 by including the partial multiple frame related information 64 in the metadata 80 (see FIG. 10 ). After the processing of step ST30 is executed, the image file creation processing proceeds to step ST32.

[0083] In step ST32, the first assigning unit 44C assigns the overall related information 74 acquired in step ST24 to the video file 78 by including the overall related information 74 in the metadata 80 (see FIG. 11 ). After the process of step ST32 is executed, the image file creation process proceeds to step ST34.

[0084] In step ST34, the control unit 44E stores the moving image file 78 obtained by executing the processes of steps ST10 to ST32 in the NVM 46 (see FIG. 13). After the process of step ST34 is executed, the image file creation process ends.

[0085] As described above, the imaging device 10 acquires the moving image data 32 and generates a moving image file 78 including the moving image data 32. Conventionally, a user would need to play the moving image data 32 to understand the contents of the moving image data 32. However, the imaging device 10 acquires the overall related information 74 as information about all frames 36 included in the frame group 34 of the moving image data 32 and adds it to the moving image file 78. Therefore, a user (e.g., a user or various devices) can understand the contents of all frames 36 included in the frame group 34 by referring to the overall related information 74 without playing the moving image data 32. Furthermore, the imaging device 10 acquires the partial multiple frame related information 64 as information about some of the frames 36 included in the frame group 34 of the moving image data 32 and adds it to the moving image file 78. Therefore, a user can understand the contents of some of the frames 36 included in the frame group 34 by referring to the partial multiple frame related information 64 without playing the moving image data 32. In this way, with the imaging device 10 according to this embodiment, the user or the like can understand the contents of the moving image data 32 without playing the moving image data 32 .

[0086] Furthermore, in the imaging device 10, frame-related information 56 is acquired as information about one frame 36 included in a frame group 34 of the moving image data 32, and is added to the moving image file 78. Therefore, a user or the like can understand the content of one frame 36 included in the frame group 34 by referring to the frame-related information 56, even without playing back the moving image data 32. This allows the user or the like to determine whether or not a frame 36 is important on a frame-by-frame basis.

[0087] Furthermore, in the imaging device 10, frame-related information 56 is assigned to the moving image file 78. The frame-related information 56 includes image recognition result information 54. The image recognition result information 54 includes scene information 54A. The scene information 54A is information relating to a scene identified from a frame 36. Therefore, a user or the like can understand the scene identified from a frame 36 by referring to the scene information 54A included in the image recognition result information 54 of the frame-related information 56, without having to play back the moving image data 32. This makes it possible to determine whether a frame 36 is important or not on a frame-by-frame basis.

[0088] Furthermore, in the imaging device 10, partial multiple frame related information 64 is assigned to the video file 78. The partial multiple frame related information 64 includes scene information 61. The scene information 61 is information related to a scene identified from a partial plurality of frames 36 included in the frame group 34 of the video data 32 (i.e., a plurality of frames 36 with which the frame related information group 60 is associated). Therefore, even without playing back the video data 32, a user or the like can understand a scene identified from a partial plurality of frames 36 included in the frame group 34 of the video data 32 by referring to the scene information 61 included in the partial multiple frame related information 64. This makes it possible to determine whether a frame 36 is important or not for each of the frames 36.

[0089] [First Modification] In the above embodiment, an example was described in which the frame-related information 56 includes the image recognition result information 54 and the frame basic information 58. However, the technology of the present disclosure is not limited to this. For example, information regarding changes in the subject in the video data 32 may be included in the frame-related information 56 and some of the multiple-frame-related information 64.

[0090] In this case, for example, as shown in FIG. 15 , the acquisition unit 44A acquires subject change information 82 and includes it in the frame-related information 56. The subject change information 82 is information related to changes in the person 12A captured in multiple frames 36 of the moving image data 32. An example of the information related to changes in the person 12A is the motion vector of the person 12A captured in multiple frames 36 that are successive in time (two frames 36 that are successive in time in the example shown in FIG. 15 ), or information based on the motion vector. Information based on the motion vector refers to, for example, information indicating the direction of movement of the person 12A or information indicating the amount of movement of the person 12A. Further, other examples of the information related to changes in the person 12A include information related to changes in the facial expression of the person 12A captured in multiple frames 36 that are successive in time, information related to changes in the optical elements (brightness or color) of the person 12A, or information related to changes in the size of the person 12A within the frames 36.

[0091] The acquisition unit 44A also acquires representative subject change information 84 and includes the acquired representative subject change information 84 in the partial multiple frame related information 64. The representative subject change information 84 included in the partial multiple frame related information 64 is information obtained from one or more subject change information 82 included in one or more frame related information 56 corresponding to the partial multiple frame related information 64. For example, of the one or more subject change information 82, the main subject change information 82 that satisfies a specified condition is set as the representative subject change information 84. Examples of subject change information 82 that satisfies the specified condition include the subject change information 82 of the largest person 12A, or the subject change information 82 that shows the greatest degree of change in person 12A (e.g., the subject change information 82 with the largest motion vector). In the example shown in FIG. 15 , examples of the representative subject change information 84 include information such as "the person is walking to the left."

[0092] In the first modified example, the subject change information 82 and the representative subject change information 84 are examples of "change information" according to the technology of the present disclosure.

[0093] As described above, in the imaging device 10 according to the first modification, the frame-related information 56 includes the subject change information 82. Therefore, the user or the like can grasp the change in the subject (here, as an example, the person 12A) for each frame 36 by referring to the subject change information 82 in the frame-related information 56, without having to play back the moving image data 32. This makes it possible to determine for each frame 36 whether or not a frame 36 is important.

[0094] Furthermore, in the imaging device 10 according to the first modification, the representative subject change information 84 is included in the partial multiple frame related information 64. Therefore, even without playing back the moving image data 32, the user or the like can grasp changes in the subject in units of a portion of the frames 36 included in the frame group 34 by referring to the representative subject change information 84 in the partial multiple frame related information 64. This makes it possible to determine whether or not a frame 36 is important in units of multiple frames 36.

[0095] [Second Modification] In the above-described first modification, an example in which subject change information 82 is included in frame-related information 56 and an example in which representative subject change information 84 is included in partial multiple-frame-related information 64 are described, but the technology of the present disclosure is not limited to this. For example, information indicating a change in the position or orientation of image capture device 10 may be included in frame-related information 56 and partial multiple-frame-related information 64.

[0096] In this case, for example, as shown in FIG. 16 , the acquisition unit 44A acquires geometric information 86 for each frame 36. The geometric information 86 is geometric information related to the imaging device 10. The geometric information refers to, for example, the position of the imaging device 10, the attitude of the imaging device 10, and the imaging orientation of the imaging device 10. Examples of the geometric information 86 include position information 86A, attitude information 86B, and imaging orientation information 86C. The position information 86A is information related to the position of the imaging device 10. The attitude information 86B is information related to the attitude of the imaging device 10. The imaging orientation information 86C is information in which the imaging direction of the imaging device 10 (i.e., the direction of the optical axis) is expressed in terms of an orientation.

[0097] The imaging device 10 is provided with a GNSS (Global Navigation Satellite System) receiver 88, an inertial sensor 90, and a geomagnetic sensor 92. The GNSS receiver 88, the inertial sensor 90, and the geomagnetic sensor 92 are connected to the processor 44. The GNSS receiver 88 receives radio waves transmitted from a plurality of satellites 94. The inertial sensor 90 measures physical quantities (e.g., angular velocity and acceleration) indicative of the three-dimensional inertial motion of the imaging device 10 and outputs an inertial sensor signal indicative of the measurement results. The geomagnetic sensor 92 detects geomagnetism and outputs a geomagnetic sensor signal indicative of the detection results.

[0098] The acquisition unit 44A calculates, as position information 86A, latitude, longitude, and altitude that can identify the current position of the imaging device 10 based on radio waves received by the GNSS receiver 88. The acquisition unit 44A also calculates attitude information 86B (e.g., information defined by a yaw angle, a roll angle, and a pitch angle) based on an inertial sensor signal input from the inertial sensor 90. The acquisition unit 44A also calculates imaging orientation information 86C based on the inertial sensor signal input from the inertial sensor 90 and a geomagnetic sensor signal input from the geomagnetic sensor 92. The acquisition unit 44A also calculates the imaging attitude of the imaging device 10 (whether the long side of the camera is oriented vertically or horizontally) from the information from the inertial sensor 90.

[0099] As an example, as shown in FIG. 17 , the acquisition unit 44A acquires geometric change information 96 and includes it in the frame-related information 56. The geometric change information 96 is information regarding changes in the geometric information 86 between multiple frames 36 (in the example shown in FIG. 17 , two frames 36 that are adjacent in time). For example, the geometric change information 96 includes the absolute value of the difference in the position information 86A between the multiple frames 36, the absolute value of the difference in the orientation information 86B between the multiple frames 36, and the absolute value of the difference in the imaging orientation information 86C. While the absolute value of the difference is exemplified here, a ratio may be used instead of the absolute value of the difference, and any index that can identify the degree of difference may be used. Note that, although the absolute value of the difference is exemplified below, the same applies.

[0100] The acquisition unit 44A acquires representative geometric change information 98 and includes the acquired representative geometric change information 98 in the partial multiple frame related information 64. The representative geometric change information 98 is information obtained from one or more pieces of geometric change information 96 included in one or more pieces of frame related information 56 corresponding to the partial multiple frame related information 64. For example, of the one or more pieces of geometric change information 96, geometric change information 96 that satisfies a specified condition is set as the representative geometric change information 98.

[0101] A first example of geometric change information 96 that satisfies the specified conditions is geometric change information 96 that has the greatest degree of change in geometric information 86, geometric change information 96 that has the smallest degree of change in geometric information 86, or geometric change information 96 that has information corresponding to the median value among multiple geometric change information 96.

[0102] A second example of geometric change information 96 that satisfies the specified conditions includes geometric change information 96 in which the absolute value of the difference in position information 86A between multiple frames 36 is the largest, geometric change information 96 in which the absolute value of the difference in position information 86A between multiple frames 36 is the smallest, or geometric change information 96 in which the absolute value of the difference in position information 86A between multiple frames 36 is the median.

[0103] A third example of geometric change information 96 that satisfies the specified conditions is geometric change information 96 in which the absolute value of the difference in posture information 86B between multiple frames 36 is the largest, geometric change information 96 in which the absolute value of the difference in posture information 86B between multiple frames 36 is the smallest, or geometric change information 96 in which the absolute value of the difference in posture information 86B between multiple frames 36 is the median.

[0104] A fourth example of geometric change information 96 that satisfies the specified conditions is geometric change information 96 in which the absolute value of the difference in imaging orientation information 86C between multiple frames 36 is the largest, geometric change information 96 in which the absolute value of the difference in imaging orientation information 86C between multiple frames 36 is the smallest, or geometric change information 96 in which the absolute value of the difference in imaging orientation information 86C between multiple frames 36 is the median.

[0105] In the second modified example, the geometric change information 96 and the representative geometric change information 98 are examples of "change information" according to the technology of the present disclosure.

[0106] As described above, in the imaging device 10 according to the second modification, the frame-related information 56 includes the geometric change information 96. Therefore, a user or the like can understand changes in the position or orientation of the imaging device 10 for each frame 36 by referring to the geometric change information 96 in the frame-related information 56, without having to play back the moving image data 32. This makes it possible to determine for each frame 36 whether or not a frame 36 is important.

[0107] Furthermore, in the imaging device 10 according to the second modified example, the representative geometric change information 98 is included in the partial multiple frame related information 64. Therefore, a user or the like can understand the change in the position or orientation of the imaging device 10 for each of a portion of the frames 36 included in the frame group 34 by referring to the representative geometric change information 98 in the partial multiple frame related information 64, without having to play back the moving image data 32. This makes it possible to determine for each of the frames 36 whether or not a frame 36 is important.

[0108] [Third Modification] In the above-described second modification, an example in which the geometric change information 96 is included in the frame-related information 56 and an example in which the representative geometric change information 98 is included in the partial frame-related information 64 are shown, but the technology of the present disclosure is not limited to this. For example, information indicating a change in the angle of view of the imaging device 10 may be included in the frame-related information 56 and the partial frame-related information 64.

[0109] In this case, for example, as shown in Fig. 18 , the acquisition unit 44A acquires angle-of-view change information 100 and includes it in the frame-related information 56. Angle-of-view information 102 is associated with a frame 36. The angle-of-view information 102 is information related to the angle of view set in the imaging device 10 (for example, the focal length associated with a change in zoom magnification). An example of the angle-of-view change information 100 is the absolute value of the difference in angle-of-view information 102 between multiple frames 36 (two frames 36 that are adjacent in time in the example shown in Fig. 18 ).

[0110] Furthermore, the acquisition unit 44A acquires representative angle of view change information 104 and includes the acquired representative angle of view change information 104 in the partial multiple frame related information 64. The representative angle of view change information 104 is information obtained from one or more pieces of angle of view change information 100 included in one or more pieces of frame related information 56 corresponding to the partial multiple frame related information 64. For example, of the one or more pieces of angle of view change information 100, the piece of angle of view change information 100 that satisfies a specified condition is set as the representative angle of view change information 104.

[0111] Examples of angle of view change information 100 that satisfies the specified conditions include angle of view change information 100 with the largest degree of change in angle of view information 102, angle of view change information 100 with the smallest degree of change in angle of view information 102, or angle of view change information 100 that corresponds to the median value among multiple angle of view change information 100.

[0112] In the third modified example, the angle-of-view change information 100 and the representative angle-of-view change information 104 are examples of “change information” according to the technology of the present disclosure. This makes it possible to determine whether a frame 36 is important or not for each frame 36.

[0113] As described above, in the imaging device 10 according to the third modification, the frame-related information 56 includes the angle-of-view change information 100. Therefore, a user or the like can understand the change in the angle of view of the imaging device 10 for each frame 36 by referring to the angle-of-view change information 100 in the frame-related information 56, even without playing back the moving image data 32.

[0114] Furthermore, in the imaging device 10 according to the third modification, the representative angle of view change information 104 is included in the partial multiple frame related information 64. Therefore, even without playing back the moving image data 32, the user or the like can understand the change in the angle of view of the imaging device 10 for each of a portion of the frames 36 included in the frame group 34 by referring to the representative angle of view change information 104 in the partial multiple frame related information 64. This makes it possible to determine for each of the frames 36 whether or not a frame 36 is important.

[0115] [Fourth Modification] In the above-described third modification, an example of including the angle of view change information 100 in the frame-related information 56 and an example of including the representative angle of view change information 104 in the partial multiple-frame-related information 64 are shown, but the technology of the present disclosure is not limited to this. For example, when a tracking mode for tracking a subject is set for the imaging device 10, information related to the tracking of the subject may be included in the frame-related information 56 and the partial multiple-frame-related information 64.

[0116] 19 , for example, the acquisition unit 44A acquires tracking-related information 106 for each frame 36, and includes the acquired tracking-related information 106 in the corresponding frame-related information 56. The tracking-related information 106 is, for example, coordinates that specify the position of a tracking frame 106A (e.g., an AF frame) that tracks the person 12A in the frame 36. Examples of the coordinates that specify the position of the tracking frame 106A include two-dimensional coordinates of the upper left corner of the tracking frame 106A and two-dimensional coordinates of the upper right corner of the tracking frame 106A when the upper left corner of the frame 36 is set as the origin.

[0117] Note that, although coordinates specifying the position of the tracking frame 106A are exemplified as the tracking-related information 106 here, the technology of the present disclosure is not limited to this. For example, the tracking-related information 106 may include information indicating that the tracking mode is set, or may include information indicating the characteristics of the subject being tracked (for example, the type of subject or attributes of the subject). In this way, the tracking-related information 106 may be any information related to the tracking of a subject.

[0118] The acquisition unit 44A acquires the representative tracking related information 108, and includes the acquired representative tracking related information 108 in the partial multiple frame related information 64. The representative tracking related information 108 is information obtained from one or more pieces of tracking related information 106 included in the one or more pieces of frame related information 56 corresponding to the partial multiple frame related information 64. For example, of the one or more pieces of tracking related information 106, the tracking related information 106 that satisfies a specified condition is set as the representative tracking related information 108.

[0119] An example of the tracking-related information 106 that satisfies the specified condition is the tracking-related information 106 corresponding to the frame 36 that has the tracking frame 106A that overlaps the most with the tracking frames 106A of other frames 36.

[0120] In the fourth modified example, the tracking-related information 106 and the representative tracking-related information 108 are examples of "information related to subject tracking" according to the technology of the present disclosure.

[0121] As described above, in the imaging device 10 according to the fourth modification, the frame-related information 56 includes the tracking-related information 106. Therefore, the user or the like can grasp information related to the tracking of the subject for each frame 36 by referring to the tracking-related information 106 in the frame-related information 56, even without playing back the moving image data 32. This makes it possible to determine for each frame 36 whether or not a frame 36 is important.

[0122] Furthermore, in the imaging device 10 according to the fourth modification, the representative tracking related information 108 is included in the partial multiple frame related information 64. Therefore, even without playing back the video data 32, the user or the like can grasp information related to the tracking of the subject in units of a partial plurality of frames 36 included in the frame group 34 by referring to the representative tracking related information 108 in the partial multiple frame related information 64. This makes it possible to determine whether or not a frame 36 is important in units of a plurality of frames 36.

[0123] [Fifth Modification] In the above-described fourth modification, an example in which the tracking-related information 106 is included in the frame-related information 56 and an example in which the representative tracking-related information 108 is included in the partial multiple-frame-related information 64 are shown, but the technology of the present disclosure is not limited to this. For example, information related to the movement of the imaging device 10 may be included in the frame-related information 56 and the partial multiple-frame-related information 64.

[0124] 20 , the acquisition unit 44A acquires pan / tilt information 110 for each frame 36, and includes the acquired pan / tilt information 110 in the corresponding frame-related information 56. The pan / tilt information 110 is information relating to panning and tilting under conditions in which the tracking mode is not set. For example, the pan / tilt information 110 includes information indicating the amount of panning and the amount of tilting between a plurality of frames 36 that are successive in time (e.g., two frames 36 that are adjacent in time).

[0125] The pan amount and tilt amount are calculated based on, for example, an inertial sensor signal output from the inertial sensor 90 (see FIG. 16 ). Here, information about panning and tilting when the tracking mode is not set is exemplified as the pan-tilt information 110, but the technology of the present disclosure is not limited to this. For example, information about panning and tilting when the tracking mode is set may be used as the pan-tilt information 110.

[0126] Furthermore, although the pan amount and tilt amount are exemplified here, the technology of the present disclosure is not limited to this, and other physical quantities related to pan and tilt, such as pan speed and tilt speed, may be applied instead of or together with the pan amount and tilt amount. Furthermore, pan and tilt are merely examples, and information regarding movement when the imaging device 10 moves in the horizontal direction (e.g., movement amount and movement speed) and information regarding movement when the imaging device 10 moves in the vertical direction may be applied instead of or together with the pan / tilt information 110.

[0127] The acquisition unit 44A acquires the representative pan / tilt information 112 and includes the acquired representative pan / tilt information 112 in the partial multiple-frame related information 64. The representative pan / tilt information 112 is information obtained from one or more pieces of pan / tilt information 110 included in one or more pieces of frame related information 56 corresponding to the partial multiple-frame related information 64. For example, among the one or more pieces of pan / tilt information 110, pan / tilt information 110 that satisfies a specified condition is set as the representative pan / tilt information 112. A first example of the pan / tilt information 110 that satisfies the specified condition is pan / tilt information 110 with a maximum or minimum pan amount. A second example of the pan / tilt information 110 that satisfies the specified condition is pan / tilt information 110 with a maximum or minimum tilt amount. A third example of the pan / tilt information 110 that satisfies the specified condition is pan / tilt information 110 with a maximum or minimum both pan and tilt amounts.

[0128] In the fifth modified example, the pan-tilt information 110 and the representative pan-tilt information 112 are examples of "information relating to the movement of the imaging device" according to the technique of the present disclosure.

[0129] As described above, in the imaging device 10 according to the fifth modification, the pan / tilt information 110 is included in the frame-related information 56. Therefore, the user or the like can grasp information about pan (e.g., the pan amount) and information about tilt (e.g., the tilt amount) for each frame 36 by referring to the pan / tilt information 110 in the frame-related information 56, without having to play back the moving image data 32. This makes it possible to determine for each frame 36 whether or not a frame 36 is important.

[0130] Furthermore, in the imaging device 10 according to the fifth modification, the representative pan / tilt information 112 is included in the partial multiple frame related information 64. Therefore, the user or the like can grasp information about pan (e.g., the pan amount) and information about tilt (e.g., the tilt amount) for each of a portion of the frames 36 included in the frame group 34 by referring to the representative pan / tilt information 112 in the partial multiple frame related information 64, without having to play back the moving image data 32. This makes it possible to determine for each of the frames 36 whether or not the frames 36 are important.

[0131] [Sixth Modification] In the fifth modification, the frame-related information 56 includes the pan-tilt information 110, and the partial multiple-frame-related information 64 includes the representative pan-tilt information 112. However, the technology of the present disclosure is not limited to this. For example, information about the sensors of the imaging device 10 may be included in the frame-related information 56 and the partial multiple-frame-related information 64.

[0132] In this case, for example, as shown in FIG. 21 , the acquisition unit 44A acquires inertial sensor signal difference information 114 for each frame 36 and includes the acquired inertial sensor signal difference information 114 in the corresponding frame-related information 56. The inertial sensor signal difference information 114 is the absolute value of the difference between inertial sensor signals 116 acquired one frame after the other. The inertial sensor signals 116 are acquired from the inertial sensor 90 (see FIG. 16 ) for each frame 36. An example of the inertial sensor signal difference information 114 is the absolute value of the difference between two inertial sensor signals 116 corresponding to two temporally adjacent frames 36.

[0133] The acquisition unit 44A acquires the representative inertial sensor signal difference information 118 and includes the acquired representative inertial sensor signal difference information 118 in the partial multiple frame related information 64. The representative inertial sensor signal difference information 118 is information obtained from one or more pieces of inertial sensor signal difference information 114 included in the one or more pieces of frame related information 56 corresponding to the partial multiple frame related information 64. For example, of the one or more pieces of inertial sensor signal difference information 114, inertial sensor signal difference information 114 that satisfies a specified condition is set as the representative inertial sensor signal difference information 118. An example of inertial sensor signal difference information 114 that satisfies the specified condition is inertial sensor signal difference information 114 in which the absolute value of the difference between two inertial sensor signals 116 is maximum or minimum.

[0134] In the sixth modification, the inertial sensor signal difference information 114 and the representative inertial sensor signal difference information 118 are an example of "information related to the first sensor" according to the technology of the present disclosure. Also, in the sixth modification, the inertial sensor 90 (see FIG. 16 ) is an example of the "first sensor" according to the technology of the present disclosure.

[0135] As described above, in the imaging device 10 according to the sixth modification, the frame-related information 56 includes the inertial sensor signal difference information 114. Therefore, a user or the like can understand the state of the imaging device 10 (e.g., the attitude or movement speed of the imaging device 10) for each frame 36 by referring to the inertial sensor signal difference information 114 in the frame-related information 56, without having to play back the moving image data 32. This makes it possible to determine for each frame 36 whether or not a frame 36 is important.

[0136] Furthermore, in the imaging device 10 according to the sixth modification, the representative inertial sensor signal difference information 118 is included in the partial multiple frame related information 64. Therefore, a user or the like can understand the state of the imaging device 10 (for example, the attitude or movement speed of the imaging device 10) for each of a portion of multiple frames 36 included in the frame group 34 by referring to the representative inertial sensor signal difference information 118 in the partial multiple frame related information 64, without having to play back the video data 32. This makes it possible to determine for each of multiple frames 36 whether or not a frame 36 is important.

[0137] Note that in the sixth modification, an example has been given in which the inertial sensor signal difference information 114 and the representative inertial sensor signal difference information 118 are acquired by the acquisition unit 44A as an example of "information related to the first sensor" according to the technology of the present disclosure, but this is merely one example. For example, the acquisition unit 44A may acquire geomagnetic sensor signal difference information and representative geomagnetic sensor signal difference information from the geomagnetic sensor signal output from the geomagnetic sensor 92 (see FIG. 16 ) in a similar manner to the inertial sensor signal difference information 114 and the representative inertial sensor signal difference information 118. In this case, the geomagnetic sensor signal difference information may be included in the frame-related information 56, and some of the representative geomagnetic sensor signal difference information may be included in the multiple-frame-related information 64.

[0138] Furthermore, in this sixth variant, the absolute value of the difference between the inertial sensor signals 116 is exemplified, but this is merely one example, and it may also be, for example, the ratio of one of the two inertial sensor signals 116 to the other, or any information that can identify the degree of difference between the inertial sensor signals 116.

[0139] Furthermore, in the sixth modification, the inertial sensor 90 is cited as an example of the “first sensor” according to the technology of the present disclosure, but this is merely an example. For example, the technology of the present disclosure can also be realized by installing various sensors, such as a distance measurement sensor or a temperature sensor, in the imaging device 10, and handling signals output from the various sensors in the same manner as the inertial sensor signal difference information 114. For example, if the imaging device 10 is equipped with a distance measurement sensor, information based on the distance measurement results from the distance measurement sensor (e.g., information indicating the distances between several locations, or a scaled-down image of a distance image) may be handled in the same manner as the inertial sensor signal difference information 114. Furthermore, if the imaging device 10 is equipped with a temperature sensor, information based on the measurement results from the temperature sensor (e.g., the outside air temperature, or a scaled-down image of a thermal image) may be handled in the same manner as the inertial sensor signal difference information 114.

[0140] [Seventh Modification] In the sixth modification, the frame-related information 56 includes the inertial sensor signal difference information 114, and the partial multiple-frame-related information 64 includes the representative inertial sensor signal difference information 118. However, the technology of the present disclosure is not limited to this. For example, the frame-related information 56 and the partial multiple-frame-related information 64 may include information corresponding to the recording time at which the subject in the video data 32 was recorded.

[0141] 22 , for example, the acquisition unit 44A identifies a plurality of frames 36 in which the person 12A appears consecutively within the moving image data 32. The acquisition unit 44A acquires two image capture times 120. The image capture times 120 are the times at which one frame's worth of image data is captured. One of the two image capture times 120 is the image capture time 120 corresponding to the first frame 36 of the plurality of frames 36 identified by the acquisition unit 44A, and the other is the image capture time 120 corresponding to the last frame 36 of the plurality of frames 36 identified by the acquisition unit 44A.

[0142] The acquisition unit 44A calculates the recording time related information 122 based on the two image capture times 120. The recording time related information 122 refers to, for example, information indicating the elapsed time between the two image capture times 120 (i.e., the time during which the person 12A is continuously captured in the moving image data 32). Note that, although the elapsed time between the two image capture times 120 is given as the recording time related information 122 here, this is merely an example, and the recording time related information 122 may be the two image capture times 120 themselves, or may be a frame identifier capable of identifying the two frames 36 corresponding to the two image capture times 120.

[0143] The acquisition unit 44A includes the recording time related information 122 in the plurality of frame related information 56 corresponding to the plurality of frames 36 in which the person 12A appears consecutively in the moving image data 32. Note that the recording time related information 122 may be included in one or more specified frames of the plurality of frame related information 56 corresponding to the plurality of frames 36 in which the person 12A appears consecutively in the moving image data 32. In this case, for example, the recording time related information 122 may be included in the frame related information 56 corresponding to the first frame 36 or the last frame 36 of the plurality of frames 36 in which the person 12A appears consecutively in the moving image data 32.

[0144] The acquisition unit 44A acquires the representative recording time related information 124 and includes the acquired representative recording time related information 124 in the partial multiple frame related information 64. The representative recording time related information 124 is information obtained from one or more pieces of recording time related information 122 included in the one or more pieces of frame related information 56 corresponding to the partial multiple frame related information 64. For example, of the one or more pieces of recording time related information 122, recording time related information 122 that satisfies a specified condition is set as the representative recording time related information 124. An example of recording time related information 122 that satisfies the specified condition is recording time related information 122 with the longest or shortest elapsed time between two image capture times 120 (i.e., the time during which the person 12A is continuously captured in the moving image data 32).

[0145] In this seventh variant, the recording time related information 122 and the representative recording time related information 124 are examples of "information corresponding to the recording time at which a subject in video image data is recorded" according to the technology of the present disclosure.

[0146] As described above, in the imaging device 10 according to the seventh modification, the frame-related information 56 includes the recording time-related information 122. Therefore, by referring to the recording time-related information 122 in the frame-related information 56, a user or the like can ascertain, on a frame 36-by-frame basis, the length of time that a particular subject (e.g., person 12A) is continuously captured in the moving image data 32, without having to play back the moving image data 32. This makes it possible to determine, on a frame 36-by-frame basis, whether a frame 36 is important or not.

[0147] Furthermore, in the imaging device 10 according to the seventh modification, the partial multiple frame related information 64 includes the representative recording time related information 124. Therefore, even without playing back the moving image data 32, the user or the like can refer to the representative recording time related information 124 in the partial multiple frame related information 64 to ascertain the time during which a specific subject (e.g., person 12A) is continuously captured in the moving image data 32 for each of a portion of the frames 36 included in the frame group 34. This makes it possible to determine for each of the frames 36 whether or not a frame 36 is important.

[0148] [Eighth Modification] In the seventh modification, the frame-related information 56 includes the recording time-related information 122, and the partial frame-related information 64 includes the representative recording time-related information 124. However, the technology of the present disclosure is not limited to this. For example, information related to operations on the imaging device 10 may be included in the frame-related information 56 and the partial frame-related information 64.

[0149] 23 , the acquisition unit 44A acquires operation-related information 126 for each frame 36, and includes the acquired operation-related information 126 in the corresponding frame-related information 56. The operation-related information 126 is information indicating an operation performed on the imaging device 10 (for example, an operation accepted by the UI-based device 18).

[0150] The acquisition unit 44A acquires the representative operation related information 127 and includes the acquired representative operation related information 127 in the partial multiple frame related information 64. The representative operation related information 127 is information obtained from one or more pieces of operation related information 126 included in the one or more pieces of frame related information 56 corresponding to the partial multiple frame related information 64. For example, of the one or more pieces of operation related information 126, the operation related information 126 that satisfies a specified condition is set as the representative operation related information 127. An example of the operation related information 126 that satisfies the specified condition is the operation related information 126 that indicates the most or least number of operations out of the plurality of pieces of operation related information 126 included in all of the frame related information 56 corresponding to the partial multiple frame related information 64.

[0151] In the eighth modified example, the operation-related information 126 and the representative operation-related information 127 are examples of "information relating to operations on an imaging device used in imaging to obtain moving image data" according to the technology of the present disclosure.

[0152] As described above, in the imaging device 10 according to the eighth modification, the operation-related information 126 is included in the frame-related information 56. Therefore, the user, etc., can understand the operations performed on the imaging device 10 on a frame 36-by-frame basis by referring to the operation-related information 126 in the frame-related information 56, without having to play back the moving image data 32. For example, a frame 36 obtained while a menu setting operation is being performed on the imaging device 10 is unlikely to have the user paying attention to the subject, and is therefore likely not to be an important frame for the user, etc. By referring to the operation-related information 126, the user, etc., can identify whether a frame 36 was obtained while a menu setting operation was being performed on the imaging device 10. This allows the user, etc., to determine whether a frame 36 is important on a frame 36-by-frame basis.

[0153] Furthermore, in the imaging device 10 according to the eighth modification, the representative operation related information 127 is included in the partial multiple frame related information 64. Therefore, even without playing back the moving image data 32, the user or the like can understand the operations performed on the imaging device 10 in units of a portion of multiple frames 36 included in the frame group 34 by referring to the representative operation related information 127 in the partial multiple frame related information 64. This makes it possible to determine whether or not a frame 36 is important in units of multiple frames 36.

[0154] [Ninth Modification] In the above-described eighth modification, an example was shown in which the operation-related information 126 was included in the frame-related information 56 and an example was shown in which the representative operation-related information 127 was included in the partial multiple-frame-related information 64, but the technology of the present disclosure is not limited to this. For example, information related to a sensor that measures a subject in the video data 32 (for example, a physical quantity indicating the current state of the subject, or a characteristic possessed by the subject, etc.) may be included in the frame-related information 56 and the partial multiple-frame-related information 64.

[0155] In this case, for example, as shown in Fig. 24 , the acquisition unit 44A acquires subject sensor information 128 for each frame 36. The subject sensor information 128 is information obtained from a sensor attached to the subject, person 12A. In the example shown in Fig. 24 , a wearable terminal 130 is shown as the sensor attached to the person 12A. The wearable terminal 130 is a wristband-type wearable terminal and is attached to the wrist of the person 12A. The wearable terminal 130 has a communication function and communicates with the imaging device 10.

[0156] The wearable terminal 130 is equipped with a function for measuring a pulse. The wearable terminal 130 measures the pulse of the person 12A and transmits information indicating the measured pulse to the imaging device 10 as subject sensor information 128. In the imaging device 10, the subject sensor information 128 transmitted from the wearable terminal 130 is received by the communication I / F 42, and the received subject sensor information 128 is acquired by the acquisition unit 44A.

[0157] As an example, as shown in FIG. 25, the acquisition unit 44A includes the subject sensor information 128 acquired from the wearable terminal 130 in the frame-related information 56 of the corresponding frame 36 (i.e., the frame 36 containing the person 12A shown in FIG. 24).

[0158] Furthermore, the acquisition unit 44A acquires representative subject sensor information 132 and includes the acquired representative subject sensor information 132 in the partial multiple frame related information 64. The representative subject sensor information 132 is information obtained from one or more subject sensor information 128 included in the one or more frame related information 56 corresponding to the partial multiple frame related information 64. For example, of the one or more subject sensor information 128, subject sensor information 128 that satisfies a specified condition is set as the representative subject sensor information 132. An example of the representative subject sensor information 132 that satisfies the specified condition is the subject sensor information 128 that indicates the highest pulse rate among the plurality of subject sensor information 128 included in all of the frame related information 56 corresponding to the partial multiple frame related information 64.

[0159] In the ninth modified example, the subject sensor information 128 and the representative subject sensor information 132 are examples of "information related to a second sensor that measures a subject in video data" according to the technology of the present disclosure.

[0160] As described above, in the imaging device 10 according to the ninth modification, the subject sensor information 128 is included in the frame-related information 56. Therefore, by referring to the subject sensor information 128 in the frame-related information 56, a user or the like can ascertain the physical condition (e.g., pulse rate, blood pressure, or blood glucose level) of a specific subject (e.g., person 12A) for each frame 36, without having to play back the video data 32. For example, if the pulse rate of person 12A is high, the person 12A is likely to be excited. That is, a frame 36 obtained by capturing an image of person 12A in an excited state is likely to be an important frame 36. Furthermore, for example, if the blood pressure of person 12A is low, the person 12A is likely to be in poor physical condition. Furthermore, for example, if the ninth modification is applied to a scene in which person 12A is undergoing surgery at a hospital, it is possible to ascertain the medical procedures that significantly affected the subject's physical condition. Therefore, by referring to the subject sensor information 128, a user or the like can determine whether a frame 36 is important for each frame 36.

[0161] Furthermore, in the imaging device 10 according to the ninth modification, the representative subject sensor information 132 is included in the partial multiple frame related information 64. Therefore, by referring to the representative subject sensor information 132 in the partial multiple frame related information 64, a user or the like can ascertain the physical condition (e.g., pulse rate, blood pressure, or blood sugar level) of a specific subject (e.g., person 12A) for each of a plurality of frames 36 included in the frame group 34, without having to play back the moving image data 32. This makes it possible to determine for each of a plurality of frames 36 whether or not a frame 36 is important.

[0162] In this ninth variant, a wearable terminal 130 is exemplified, but this is merely one example, and the terminal may be a smartphone or a tablet terminal, or any electronic device equipped with one or more sensors such as a thermometer or a blood pressure monitor (i.e., a sensor capable of measuring the subject in the video data 32).

[0163] Furthermore, in the ninth modification, an example in which the pulse of person 12A is measured by wearable terminal 130 has been described, but this is merely one example. For example, the measured information may be biometric information such as the blood pressure, breathing volume, number of breaths, or blood sugar level of the subject captured in video image data 32. Furthermore, in the case where the imaging target area is a production line in a factory or the like, the measured information is not limited to biometric information and may be information measured by a sensor installed in the production line.

[0164] [Tenth Modification] In the above embodiment, the summary information and the overall basic information 76 are exemplified as information to be included in the overall related information 74, but the technology of the present disclosure is not limited to this. For example, the video data 32 may be information related to a portion of other video data, and the overall related information 74 may include information related to the other video data.

[0165] 26 , when moving image data 32 incorporates a portion of moving image data 134A contained in a moving image file 134 different from moving image file 78, first adding unit 44C includes separate moving image file information 136 in overall related information 74. Separate moving image file information 136 is information that identifies the location of moving image file 134.

[0166] If the video file 134 is stored in a memory such as the NVM 46, the separate video file information 136 is, for example, an address in the memory. If the video file 134 exists on the Internet, the separate video file information 136 is, for example, a uniform resource locator (URL).

[0167] For example, the separate video file information 136 is included in the metadata 134B of the video file 134, and the first assigning unit 44C acquires the separate video file information 136 from the metadata 134B when a portion of the video data 134A is incorporated into the video data 32. The first assigning unit 44C then assigns the separate video file information 136 to the video file 78 by including the separate video file information 136 acquired from the metadata 134B in the overall related information 74.

[0168] In this way, when a portion of video data 134A is incorporated into video data 32, separate video file information 136 is included in overall related information 74. Separate video file information 136 is information that identifies the source location of a portion of video data 134A. Therefore, by referring to separate video file information 136 in overall related information 74, a user or the like can identify which video file a portion of video data 32 originates from, without having to play video data 32.

[0169] Note that, in the tenth modified example, an example has been described in which part of the video data 134A contained in the video file 134 is incorporated into the video data 32, but the technology of the present disclosure is not limited to this. For example, even if part of the video data contained in one or more video files other than the video file 134 is incorporated into the video data 32 together with part of the video data 134A, the separate video file information 136 can be included in the overall related information 74 in a similar manner to the tenth modified example.

[0170] Furthermore, in the tenth modification, information specifying the location of another video file 134 is exemplified as the separate video file information 136, but the technology of the present disclosure is not limited to this. The separate video file information 136 may include a thumbnail image (e.g., a thumbnail still image or a thumbnail video) relating to a portion of the video data 134A, or may include an identifier identifying a frame included in the portion of the video data 134A, the number of frames, or the like. In this way, the separate video file information 136 only needs to include information relating to a portion of the video data 134A.

[0171] [Other Modifications] In the above embodiment, an example in which the image file creation process is executed by the information processing device 38 in the imaging device 10 has been described, but the technology of the present disclosure is not limited to this. For example, as shown in FIG. 27 , the image file creation process may be executed by a computer 140 in an external device 138 that is communicatively connected to the imaging device 10 via a network 137 such as a local area network (LAN) or a wide area network (WAN). An example of the computer 140 is a server computer for a cloud service. The computer 140 is an example of an "information processing device" according to the technology of the present disclosure.

[0172] 27, the computer 140 includes a processor 142, a storage 144, and a memory 146. The storage 144 stores an image file creation program 52.

[0173] The imaging device 10 requests the external device 138 to execute an image file creation process via the network 137. In response to this, the processor 142 of the external device 138 reads the image file creation program 52 from the storage 144 and executes the image file creation program 52 on the memory 146. The processor 142 performs the image file creation process in accordance with the image file creation program 52 executed on the memory 146. The processor 142 then provides the processing result obtained by executing the image file creation process to the imaging device 10 via the network 137.

[0174] 27 shows an example of an embodiment in which the external device 138 executes the image file creation process, but this is merely one example. For example, the image file creation process may be distributed between the imaging device 10 and the external device 138, or may be distributed among a plurality of devices including the imaging device 10 and the external device 138.

[0175] In the above embodiment, an example was described in which the frame-related information 56, the partial multiple-frame-related information 64, and the overall-related information 74 are recorded in the metadata 80 of the moving image file 78, but the technology of the present disclosure is not limited to this. For example, a frame 36 may be extracted from the moving image data 32 of the moving image file 78, an image file may be generated for the extracted frame 36, and the frame-related information 56, the partial multiple-frame-related information 64, and the overall-related information 74 may be added to the generated image file. In this case, an image file in a format that allows additional information to be added to an area separate from the image data (i.e., a recordable format) is used.

[0176] The format of the moving image file may be any file format that allows additional information to be added to an area separate from the image data. The file format may be any of MPEG (Moving Picture Experts Group)-4, H.264, MJPEG (Motion JPEG), HEIF (High Efficiency Image File Format), AVI (Audio Video Interleave), MOV (QuickTime file format), WMV (Windows Media Video), and FLV (Flash Video).

[0177] From the viewpoint of adding the metadata (additional information) described in the above embodiment, it is preferable that the video data is in HEIF format.

[0178] The items and number of additional information that can be added to an image file vary depending on the file format, and additional information for new items can be added by updating the version information of the image file. The items of additional information refer to the viewpoints for adding additional information (i.e., the categories into which the information is classified).

[0179] In the above embodiment, an example has been described in which the image file creation program 52 is stored in the NVM 46, but the technology of the present disclosure is not limited to this. For example, the image file creation program 52 may be stored in a portable, computer-readable, non-transitory storage medium such as an SSD, a USB memory, or a magnetic tape. The image file creation program 52 stored in the non-transitory storage medium is installed in the imaging device 10. The processor 44 executes the image file creation process in accordance with the image file creation program 52.

[0180] In addition, the image file creation program 52 may be stored in a storage device such as another computer or server device connected to the imaging device 10 via a network, and the image file creation program 52 may be downloaded and installed in the imaging device 10 in response to a request from the imaging device 10.

[0181] It should be noted that it is not necessary to store the entire image file creation program 52 in a storage device such as another computer or server device connected to the imaging device 10, or in the NVM 46; only a portion of the image file creation program 52 may be stored therein.

[0182] Furthermore, although the imaging device 10 shown in FIG. 2 has an information processing device 38 built in, the technology of the present disclosure is not limited to this, and for example, the information processing device 38 may be provided outside the imaging device 10.

[0183] In the above embodiment, an example in which the technology of the present disclosure is realized by a software configuration has been described, but the technology of the present disclosure is not limited to this, and devices including an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a PLD (Programmable Logic Device) may also be applied.

[0184] A combination of hardware and software configurations may also be used.

[0185] The hardware resources that execute the image file creation process described in the above embodiments can be various processors, as listed below. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource that executes the image file creation process by executing software, i.e., a program. Examples of processors include dedicated electronic circuits, such as FPGAs, PLDs, or ASICs, which are processors with a circuit configuration designed specifically for executing specific processes. Each processor has built-in or connected memory, and each processor executes the image file creation process by using the memory.

[0186] The hardware resource that executes the image file creation process may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the image file creation process may be a single processor.

[0187] Examples of a system using a single processor include: a first configuration in which one processor is configured using a combination of one or more CPUs and software, and this processor functions as a hardware resource that executes the image file creation process; a second configuration in which a processor is used to implement the functions of an entire system, including multiple hardware resources that execute the image file creation process, on a single IC (Integrated Circuit) chip, as typified by SoCs (System-on-a-Chips); and in this way, the image file creation process is implemented using one or more of the above-mentioned various processors as hardware resources.

[0188] Furthermore, the hardware structure of these various processors can be, more specifically, electronic circuits that combine circuit elements such as semiconductor devices. The image file creation process described above is merely an example. It goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the process.

[0189] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0190] In this specification, the grammatical concept of "A or B" includes not only the concept of "either one of A or B," but also a concept synonymous with "at least one of A and B." In other words, "A or B" includes the meaning that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A or B" is also applied when three or more things are expressed by connecting them with "or."

[0191] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. video data including a group of frames; and additional information, The video data includes a part of another video data included in a different image file, the different image file being different from the image file, The auxiliary information includes first auxiliary information and second auxiliary information, the first auxiliary information includes information on all frames included in the frame group and information on the other video data; The second auxiliary information is information about a part of a plurality of frames of the frame group. Image file.

2. The information relating to the other video data includes information for identifying the location of the other image file. The image file of claim 1 .

3. The separate image file has metadata, the metadata includes information regarding the other video data; The first auxiliary information includes information related to the other video data obtained from the metadata.

3. The image file according to claim 1 or 2.

4. The first auxiliary information includes information regarding the configuration of the video data. The image file according to any one of claims 1 to 3.

5. The second auxiliary information further includes information regarding one frame included in the frame group. The image file according to any one of claims 1 to 4.

6. The second auxiliary information includes scene information regarding a scene identified from the plurality of frames, or a scene identified from the one frame. The image file according to claim 5.

7. The second auxiliary information includes change information regarding a change in a subject in the video data, or a change in a position, a direction, or a field of view of an imaging device used in imaging to obtain the video data. The image file according to any one of claims 1 to 6.

8. The change information includes information regarding tracking of the subject or information regarding movement of the imaging device. The image file according to claim 7.

9. The second auxiliary information includes information about a first sensor of an imaging device used in imaging to obtain the moving image data. The image file according to any one of claims 1 to 8.

10. The second auxiliary information includes information regarding a second sensor that measures a subject in the video image data. The image file according to any one of claims 1 to 9.

11. The second auxiliary information includes information regarding an operation of an image capturing device used in capturing an image to obtain the moving image data. The image file according to any one of claims 1 to 10.

12. The second auxiliary information includes information corresponding to a recording time during which the subject in the video image data is recorded. The image file according to any one of claims 1 to 11.

13. An information processing device comprising a processor that generates the image file according to any one of claims 1 to 12.

14. a sensor used in capturing the moving image data included in the image file according to any one of claims 1 to 12; a processor for generating the image file based on signals output by the sensor; An imaging device comprising:

15. A method for generating an image file including video data including a frame group and supplementary information, comprising the steps of: The video data includes a part of another video data included in a different image file, the different image file being different from the image file, acquiring the video data; a first adding step of adding first auxiliary information to the image file as the auxiliary information, the first auxiliary information including information on all frames included in the frame group and information on the other moving image data; and a second adding step of adding second auxiliary information, which is information about a part of a plurality of frames among the frame group, to the image file as the auxiliary information. Generation method.

Images