Image file, information processing apparatus, imaging apparatus, and generation method
By embedding relevant information in the dynamic image file, the problem of needing to play dynamic image files in the prior art to understand the content is solved, and the content of dynamic image files is quickly queried and processed, and the information processing efficiency is improved.
Patent Information
- Application Number
- CN202380030567.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-30
- Filing Date
- 2023-02-15
- Publication Date
- 2024-11-15
AI Technical Summary
In the prior art, the dynamic image files generated by the camera device need to be played to grasp their content, resulting in high time consumption, especially when it is necessary to view the structural information of multiple frames, which is inefficient.
By embedding frame-related information, multi-frame-related information and overall-related information as metadata in the dynamic image file, users can understand the contents of the frame group, multiple frames and the entire image file through this information even if they do not play the dynamic image data.
It improves the accessibility and query efficiency of dynamic image file contents, so that users can understand important frames and scene information without playing the entire file, and improves the efficiency of information processing.
Smart Images

Figure CN118974719A_ABST
Abstract
Description
Technical Field
[0001] The technology of the present invention relates to an image file, an information processing device, a camera device and a generation method. Background Art
[0002] A three-dimensional model construction system is disclosed in Japanese Patent Publication No. 2021-177317. The three-dimensional model construction system described in Japanese Patent Publication No. 2021-177317 has a management mechanism, a collection mechanism, a generation mechanism and an assignment mechanism. The management mechanism manages multiple mobile bodies by establishing corresponding associations with users respectively. The collection mechanism collects photographic data including incidental information related to photography taken by photographic devices respectively possessed by multiple mobile bodies. The generation mechanism generates a three-dimensional model using the photographic data collected by the collection mechanism. The assignment mechanism assigns a reward to a user who has established a corresponding association with the mobile body that photographed the photographic data based on the photographic data used by the generation mechanism when generating the three-dimensional model. In addition, the generation mechanism uses the incidental information included in the new photographic data to determine the photographic area of the existing three-dimensional model corresponding to the new photographic data, and uses the new photographic data to update the existing three-dimensional model based on the difference between the three-dimensional model generated from the new photographic data in the photographic area and the existing three-dimensional model. Summary of the invention
[0003] One embodiment of the technology according to the present invention provides an image file, an information processing device, an imaging device, and a generation method that allow the content of the image file to be understood without playing the image file.
[0004] Means for solving technical problems
[0005] The first mode involved in the technology of the present invention is an image file having the following structure. That is, the image file involved in the first mode has dynamic image data including a frame group and supplementary information, the supplementary information includes first supplementary information and second supplementary information, the first supplementary information is information related to all frames included in the frame group, and the second supplementary information is information related to a part of a plurality of frames in the frame group.
[0006] The second method involved in the technology of the present invention is a generation method, which generates an image file containing dynamic image data including a frame group and accompanying information. The generation method includes the following steps: an acquisition step, acquiring dynamic image data; a first assigning step, assigning information related to all frames included in the frame group, i.e., first accompanying information, as accompanying information to the image file; and a second assigning step, assigning information related to a part of multiple frames in the frame group, i.e., second accompanying information, as accompanying information to the image file. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] Figure 1This is a conceptual diagram showing an example of a method of using the imaging device.
[0008] Figure 2 This is a block diagram showing an example of the hardware configuration of the electrical system of the imaging device and the functions of the main parts.
[0009] Figure 3 This is a conceptual diagram showing an example of the processing contents of acquiring dynamic image data, the processing contents of acquiring image recognition result information, and the processing contents of acquiring frame-related information.
[0010] Figure 4 This is a conceptual diagram showing an example of the processing contents of acquiring a claim-related information group divided by period, the processing contents of acquiring scene information, and the processing contents of acquiring a part of multi-frame related information.
[0011] Figure 5 This is a conceptual diagram showing an example of the processing content of acquiring summary information and the processing content of acquiring overall related information.
[0012] Figure 6 This is a conceptual diagram showing an example of the processing content of generating a moving image file including moving image data and metadata.
[0013] Figure 7 This is a conceptual diagram showing an example of how frame-related information, part of multi-frame-related information, and overall related information are stored (recorded) as meta-information in a moving image file.
[0014] Figure 8 This is a conceptual diagram showing an example of the processing content of adding file-related information to a moving image file.
[0015] Fig. 9 This is a conceptual diagram showing an example of the structure of image recognition result information.
[0016] Fig.10 This is a conceptual diagram showing an example of the processing content of adding a part of multi-frame related information to a moving image file.
[0017] Fig.11 This is a conceptual diagram showing an example of the content of processing for adding overall related information to a moving image file.
[0018] Fig.12 This is a conceptual diagram showing an example of the structure of metadata.
[0019] Fig.13 This is a conceptual diagram showing an example of the processing content of the control unit.
[0020] Fig.14 This is a flowchart showing an example of the flow of image file creation processing.
[0021] Fig.15 This is a conceptual diagram showing an example of processing contents in which frame-related information includes subject change information and processing contents in which part of multi-frame-related information includes representative subject change information in the imaging device according to the first modification.
[0022] Fig.16 This is a conceptual diagram showing an example of the processing content of acquiring geometric information in the camera device involved in the second modification.
[0023] Fig.17 This is a conceptual diagram showing an example of the processing content of frame-related information including geometric change information and the processing content of a part of multi-frame related information including representative geometric change information in the camera device involved in the second variant.
[0024] Fig.18 This is a conceptual diagram showing an example of processing contents in which frame-related information includes view angle change information and processing contents in which a part of multi-frame-related information includes representative view angle change information in the camera device involved in the third modified example.
[0025] Fig.19 This is a conceptual diagram showing an example of processing contents of frame-related information including tracking-related information and processing contents of a part of multi-frame-related information including representative tracking-related information in the imaging device involved in the fourth modified example.
[0026] Fig. 20 This is a conceptual diagram showing an example of processing contents in which frame-related information includes pan-tilt information and processing contents in which part of multi-frame-related information includes representative pan-tilt information in the imaging device according to the fifth modification.
[0027] Fig.21 This is a conceptual diagram showing an example of processing contents in which frame-related information includes inertial sensor signal difference information and processing contents in which part of multi-frame-related information includes representative inertial sensor signal difference information in the imaging device according to the sixth modification.
[0028] Fig. 22 This is a conceptual diagram showing an example of the processing content in which frame-related information includes recording time-related information and the processing content in which a part of multi-frame related information includes representative recording time-related information in the camera device involved in the seventh variant.
[0029] Fig.23 This is a conceptual diagram showing an example of processing contents in which frame-related information includes operation-related information and processing contents in which a portion of multi-frame-related information includes representative operation-related information in a camera device involved in the eighth variant.
[0030] Fig.24This is a conceptual diagram showing an example of the content of processing for acquiring subject sensor information in the imaging device according to the ninth modification.
[0031] Fig.25 This is a conceptual diagram showing an example of processing contents in which frame-related information includes subject sensor information and processing contents in which part of multi-frame-related information includes representative subject sensor information in the imaging device involved in the ninth modification.
[0032] Fig.26 This is a conceptual diagram showing an example of processing content in which the entire related information includes another moving image file information in the camera device involved in the 10th modification.
[0033] Fig. 27 This is a conceptual diagram showing an example of a method of causing an external device to execute image file creation processing. DETAILED DESCRIPTION
[0034] Hereinafter, an example of an embodiment of an image file, an information processing device, an imaging device, and a generation method according to the technology of the present invention will be described with reference to the drawings.
[0035] As an example, Figure 1 As shown in FIG. 1 , the imaging device 10 captures an imaging target area 12 designated as a subject. The range of the imaging target area 12 is set according to the angle of view designated by a user of the imaging device 10 (hereinafter referred to as a “user”). Figure 1 In the example shown, the imaging target area 12 includes persons 12A to 12D and a road 12E. The imaging device 10 is an example of an “imaging device” according to the technique of the present invention.
[0036] exist Figure 1 In the example shown, the imaging device 10 is a consumer-use interchangeable lens digital camera. The imaging device 10 includes an imaging device body 14 and an interchangeable lens 16. The interchangeable lens 16 is replaceably mounted on the imaging device body 14. In addition, here, as an example of the imaging device 10, an interchangeable lens digital camera is cited, but this is only an example, and a fixed lens digital camera may also be used.
[0037] In addition, here, a civilian digital camera is illustrated as an example of the imaging device 10, but this is only an example, and the technology of the present invention is also applicable when the imaging device 10 is an industrial digital camera. Furthermore, the technology of the present invention is also applicable when the imaging device 10 is an imaging device mounted on various electronic devices such as a driving recorder, a smart device, a wearable terminal, a cell observation device, an ophthalmic observation device, or a surgical microscope.
[0038] Furthermore, the technology of the present invention is also applicable when the imaging device 10 is an imaging device mounted on various medical imaging devices such as an endoscope device, an ultrasonic diagnostic device, an X-ray imaging device, a CT (Computed Tomography) device, or an MRI (Magnetic Resonance Imaging) device.
[0039] The imaging device 10 is provided with a UI (User Interface) device 18. The UI device 18 is a device that receives an operation on the imaging device 10 or presents (for example, displays) various information to the user.
[0040] exist Figure 1 In the example shown, the UI system device 18 has a release button 20, a dial 22, a touch panel display 24, and instruction keys 26 for receiving various instructions.
[0041] The release button 20 functions as an imaging preparation instruction unit and an imaging instruction unit, and is a button for instructing the imaging device 10 to perform imaging preparation (for example, AF (Auto Focus) and AE (Auto Exposure) instructions) and to instruct imaging.
[0042] The dial 22 is operated when setting the operation mode, etc. In the imaging device 10, various operation modes are selectively set by operating the dial 22. The operation mode includes the operation mode of the imaging system. As an example of the operation mode of the imaging system, a still image shooting mode and a moving image shooting mode can be cited.
[0043] The touch panel display 24 includes a display 28 and a touch panel 30. An example of the display 28 is a liquid crystal display or an EL (Electroluminescent) display. The display 28 displays a live preview image, a still image, a moving image, a menu screen, and the like. Figure 1 In the example shown, a live preview image is displayed on the display 28 .
[0044] The camera device 10 generates dynamic image data 32 representing an image showing the imaging target area 12 by capturing the imaging target area 12 at a dynamic image frame rate. The dynamic image data 32 includes a frame group 34. The frame group 34 is composed of a plurality of frames 36 (i.e., a plurality of image data arranged in a time series). The type of the frame 36 is, for example, visible light image data obtained by capturing the imaging target area 12 in the visible light region. However, the type of the frame 36 is not limited to this, and may also be invisible light image data obtained by capturing in a wavelength region other than the visible light region.
[0045] In addition, the dynamic image data 32 is an example of the "dynamic image data" involved in the technology of the present invention. Furthermore, the frame group 34 is an example of the "frame group" involved in the technology of the present invention. Furthermore, the frame 36 is an example of the "frame" involved in the technology of the present invention.
[0046] As an example, Figure 2 As shown, the imaging device 10 includes a UI device 18, an information processing device 38, an image sensor 40, and a communication I / F (Interface) 42. The information processing device 38 is an example of an "information processing device" according to the technique of the present invention.
[0047] The information processing device 38 includes a processor 44, a NVM (Non-volatile memory) 46, and a RAM (Random Access Memory) 48. The processor 44, the NVM 46, and the RAM 48 are connected to a bus 50. The processor 44 is an example of a "processor" according to the technique of the present invention.
[0048] The processor 44 is a processing device including a DSP (Digital Signal Processor), a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The DSP and the GPU work under the control of the CPU and are responsible for performing image-related processing. Here, as an example of the processor 44, a processing device including a DSP, a CPU and a GPU is cited, but this is only an example. The processor 44 may also be one or more CPUs and DSPs with integrated GPU functions, one or more CPUs and DSPs without integrated GPU functions, or may be equipped with a TPU (Tensor Processing Unit).
[0049] The NVM 46 is a nonvolatile storage device that stores various programs, various parameters, etc. Examples of the NVM 46 include a flash memory (for example, an EEPROM (Electrically Erasable and Programmable Read Only Memory)).
[0050] The RAM 48 is a memory for temporarily storing information, and is used as a working memory by the processor 44 .
[0051] Examples of the RAM 48 include DRAM (Dynamic Random Access Memory) and SRAM (Static Random Access Memory).
[0052] The image sensor 40 is connected to the bus 50. An example of the image sensor 40 is a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The image sensor 40 captures the image of the imaging target area 12 (see FIG. 1 ). Figure 1 ) to generate dynamic image data 32.
[0053] Here, a CMOS image sensor is cited as an example of the image sensor 40 , but this is just an example, and the image sensor 40 may be another type of image sensor such as a CCD (Charge Coupled Device) image sensor.
[0054] The image sensor 40 is an example of a “sensor” according to the technique of the present invention.
[0055] The UI system device 18 is connected to the bus 50. The UI system device 18 receives instructions from the user and outputs a signal indicating the received instructions to the processor 44. In addition, the UI system device 18 prompts the user with various information under the control of the processor 44. For example, the prompt of various information is displayed through the display 28 (refer to Figure 1 ) and speakers (not shown).
[0056] The communication I / F 42 is an interface including a communication processor and an antenna, etc., and is connected to the bus 50. Communication standards applicable to the communication I / F 42 include, for example, wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), and Bluetooth (registered trademark).
[0057] The camera device 10 generates a dynamic image file including dynamic image data 32 by shooting in a dynamic image shooting mode. In order to confirm the content of the dynamic image data 32 included in the dynamic image file, it is necessary to play the dynamic image file. At this time, for example, in order to understand the overall structure of the dynamic image data 32, it is necessary to play most of the dynamic image data 32, and the more frames 36 included in the dynamic image data 32, the more playback time is required. And, for example, in order to understand the structure of a part of the multiple frames 36 included in the dynamic image data 32, it is necessary to play the corresponding part of the multiple frames 36, which still requires playback time. Not only is the playback time required, but it also takes time to determine the corresponding part of the multiple frames 36 from the frame group 34.
[0058] Therefore, in view of this situation, in the imaging device 10, as an example, Figure 2 As shown, the image file creation process is performed by the processor 44. The image file creation process is a process of creating an image file. The image file created by performing the image file creation process has the accompanying information (for example, metadata) described later. In the camera device 10, the NVM 46 stores an image file creation program 52. The processor 44 performs the image file creation process by reading the image file creation program 52 from the NVM 46 and executing the read image file creation program 52 on the RAM 48. The image file creation process is realized by the acquisition unit 44A, the generation unit 44B, the first granting unit 44C, the second granting unit 44D and the control unit 44E operating according to the image file creation program 52 executed by the processor 44 on the RAM 48.
[0059] In addition, the processing performed by the acquisition unit 44A is an example of the "acquisition process" involved in the technology of the present invention. Furthermore, the processing performed by the first granting unit 44C is an example of the "first granting process" involved in the technology of the present invention. Furthermore, the processing performed by the second granting unit 44D is an example of the "second granting process" involved in the technology of the present invention.
[0060] As an example, Figure 3 As shown, the acquisition unit 44A acquires the dynamic image data 32 from the image sensor 40. The acquisition unit 44A acquires the image recognition result information 54 recorded as metadata by performing an AI (Artificial Intelligence) method image recognition process on each frame 36 included in the frame group 34 of the dynamic image data 32. The image recognition result information 54 is information indicating the result obtained by performing the AI method image recognition process.
[0061] exist Figure 3In the example shown, information about the persons 12A and 12B shown in the frame 36 and scene information 54A are included in the image recognition result information 54. In the image recognition result information 54, an identifier is assigned to each subject recognized by the image recognition process (in Figure 3 In the example shown, they are "#1" to "#N"). The subject recognized by the image recognition process is not limited to a single subject but includes a combination of a plurality of subjects.
[0062] exist Figure 3 In the example shown, as information related to the person 12A, coordinates (i.e., two-dimensional coordinates) that can identify the position of the person 12A in the frame 36, the type of the person 12A (in Figure 3 In the example shown, "person", "young girl" and "name (in Figure 3 In the example shown, the name "Fuji Flower Branch" and the attributes of the character 12A (in Figure 3 "Walk" in the example shown).
[0063] And, in Figure 3 In the example shown, as information related to the person 12B, coordinates (i.e., two-dimensional coordinates) that can identify the position of the person 12B in the frame 36, the type of the person 12B (in Figure 3 The examples shown are "person", "middle-aged woman" and "name (in Figure 3 In the example shown, the name "Fuji Masahei" is shown)") and the attributes of the character 12B (in Figure 3 "Squatting" in the example shown).
[0064] Scene information 54A is information related to the scene determined based on frame 36. The scene determined based on frame 36 is, for example, the combination of person 12A and person 12B. Figure 3 In the example shown, as scene information 54A, information "Huazhi and Yahui are playing" is shown. Here, as a scene determined based on frame 36, the combination of person 12A and person 12B is shown, but this is just an example. For example, the combination of one subject (for example, person 12A, person 12B, or person 12A to 12D and scenery) shown in frame 36 may be applied as a scene, and the scene may change according to the method of AI-based image recognition processing or the result obtained by performing AI-based image recognition processing.
[0065] In addition, although an AI-based image recognition process is exemplified here, the technology of the present invention is not limited thereto, and a template matching-based image recognition process may also be applied, or an AI-based image recognition process and a template matching-based image recognition process may be used in combination. Furthermore, one or more labels may be assigned to the subject shown in the frame 36 based on an instruction from the outside to the camera device 10 (for example, an instruction received by the UI system device 18). Furthermore, the one or more assigned labels may be used as the image recognition result information 54. As labels, for example, coordinates that can determine the position of the subject in the frame 36, the type of the subject, and the attributes of the subject may be used.
[0066] The acquisition unit 44A acquires frame-related information 56 for each frame 36 included in the frame group 34 of the dynamic image data 32. The frame-related information 56 includes the image recognition result information 54 corresponding to the frame 36 and the frame basic information 58 corresponding to the frame 36. The frame-related information 56 is an example of "information related to one frame included in the frame group" involved in the technology of the present invention.
[0067] The frame basic information 58 includes a frame identifier, model information, lens information, date and time information, and image capture condition information. The frame identifier is an identifier that can identify the frame 36. The model information is information indicating the model of the image capture device body 14. The lens information is information indicating the interchangeable lens 16 (see Figure 1 The date and time information is information on the type and specification of the frame 36 corresponding to the frame-related information 56 (for example, the time represented by year, month, day, hour, minute, and second). The imaging condition information is information indicating the imaging conditions set for the imaging device 10 (for example, F value, shutter speed, sensitivity of the image sensor 40, 35 mm conversion focal length, on / off of hand shake correction, etc.).
[0068] As an example, Figure 4 As shown, the acquisition unit 44A divides all frame-related information 56 associated with the frame group 34 of the dynamic image data 32 into a plurality of frame-related information groups 60 for acquisition. The plurality of frame-related information groups 60 are obtained by dividing the plurality of specified periods (in Figure 4 In the example shown, all frame-related information 56 of the 1st period to the Nth period) is obtained. That is, the frame-related information group 60 is a collection of multiple frame-related information 56. In addition, multiple periods can be determined according to instructions received by the UI system device 18. In addition, multiple periods can also be determined according to a pre-set algorithm. At this time, for example, all frame-related information 56 associated with the frame group 34 can also be divided at a time interval determined as a unique time interval based on the number of all frames 36 included in the frame group 34.
[0069] The acquisition unit 44A acquires scene information 61 from the frame-related information group 60 and the corresponding frame group 34. The scene information 61 is information related to the scene determined by the plurality of frames 36 associated with the frame-related information group 60. As the scene determined by the plurality of frames 36, for example, lunch on a picnic, cooking on a picnic, or fishing on a picnic can be cited. The scene information 61 is expressed, for example, by text that can determine the scene, a mark (for example, a pictogram) that can determine the scene, a thumbnail image (for example, a thumbnail still image or a thumbnail dynamic image) that can determine the scene, a combination of text and a mark, a combination of text and a thumbnail image, a combination of a thumbnail image and a mark, a combination of text, a mark, and a thumbnail image, and the like.
[0070] The scene information 61 is obtained using a NN (Neural Network) 62 which is a kind of mathematical model. The NN 62 is subjected to learning using a plurality of training data including information corresponding to a plurality of frame-related information groups 60 and reference truth data (here, as an example, information related to a plurality of scenes), and scores related to various scenes are output from the NN 62. The acquisition unit 44A inputs the frame-related information group 60 into the NN 62. The acquisition unit 44A determines the scene by referring to the score output from the NN 62 by inputting the frame-related information group 60 into the NN 62, and acquires information related to the determined scene as the scene information 61.
[0071] In addition, here, the example of determining the scene determined by the plurality of frames 36 associated with the frame-related information group 60 using the NN 62 is described, but this is only an example. For example, the user can apply the determined scene to each period according to the instruction given to the camera 10 via the UI system device 18 or a device directly or indirectly connected to the camera 10.
[0072] The acquisition unit 44A acquires a portion of multi-frame related information 64 in units of the frame related information group 60. The portion of multi-frame related information 64 includes scene information 61 determined from the corresponding frame related information group 60 and basic information 66 divided by period. The portion of multi-frame related information 64 is an example of "information related to a portion of multiple frames in the frame group" involved in the technology of the present invention.
[0073] The basic information 66 classified by period includes a frame identifier, model information, lens information, time information, and camera condition information. The frame identifier included in the basic information 66 classified by period is an identifier that can identify the frame 36 that contributes to a certain level or more when acquiring the scene information 61. As an example of the frame 36 that contributes to a certain level or more when acquiring the scene information 61, there can be cited a frame 36 associated with the frame-related information 56 whose score output from the NN 62 is greater than a certain value. The model information included in the basic information 66 classified by period is the model information obtained by statistics based on the multiple frame-related information 56 included in the frame-related information group 60 (for example, the model information that includes the most in the multiple frame-related information 56). The lens information included in the basic information 66 classified by period is the lens information obtained by statistics based on the multiple frame-related information 56 included in the frame-related information group 60 (for example, the lens information that includes the most in the multiple frame-related information 56).
[0074] The time information included in the basic information 66 divided by period is information including two date and time information included in the frame-related information 56 at the beginning and the frame-related information 56 at the end of the frame-related information group 60. Furthermore, the time information included in the basic information 66 divided by period may be two frame identifiers included in the frame-related information 56 at the beginning and the frame-related information at the end of the frame-related information group 60. Furthermore, the time information included in the basic information 66 divided by period may be information indicating a time band calculated based on the date and time information included in the frame-related information 56 at the beginning of the frame-related information group 60 and the date and time information included in the frame-related information 56 at the end of the frame-related information group 60.
[0075] The imaging condition information included in the basic information 66 divided by period is imaging condition information obtained by statistics based on the plurality of frame-related information 56 included in the frame-related information group 60 (for example, the imaging condition information that includes the most among the plurality of frame-related information 56 ).
[0076] As an example, Figure 5 As shown, the acquisition unit 44A acquires summary information indicating a summary of the dynamic image data 32. The summary refers to a title representing the content displayed in the dynamic image data 32, a text representing the content displayed in the dynamic image data 32, a tag representing the content displayed in the dynamic image data 32, or a thumbnail image representing the content displayed in the dynamic image data 32.
[0077] exist Figure 5In the example shown, as an example of summary information, summary information divided by time band 68 and overall summary information 70 can be cited. The summary information divided by time band 68 is information indicating a summary divided by time band in the moving image data 32. The summary divided by time band is, for example, a summary of each moving image data obtained by dividing the moving image data 32 into equal parts or a summary of each moving image data divided into equal parts. Figure 4 The summary of each dynamic image data obtained by dividing the dynamic image data 32 into the first period to the Nth period shown. The overall summary information 70 is information indicating the summary of the dynamic image data 32 as a whole. The summary information 68 divided by time band and the overall summary information 70 are determined, for example, based on a plurality of partial multi-frame related information 64. Hereinafter, when it is not necessary to distinguish and explain the summary information 68 divided by time band and the overall summary information 70, they are referred to as "summary information" without being marked.
[0078] Summary information is obtained using NN72. Learning is performed on NN72 using multiple training data including information equivalent to multiple parts of multi-frame related information 64 and reference truth data (here, as an example, information related to multiple summaries), and scores related to various summaries are output from NN72. The acquisition unit 44A inputs one or more parts of multi-frame related information 64 into NN72. The acquisition unit 44A determines the summary with reference to the score output from NN72 by inputting one or more parts of multi-frame related information 64 into NN72, and obtains information related to the determined summary as summary information. Summary information is an example of "information related to the structure of dynamic image data" involved in the technology of the present invention.
[0079] In addition, here, an example of a method of learning NN72 using multiple training data including information equivalent to multiple parts of multi-frame related information 64 and benchmark truth data is given, but this is only an example, and the training data can also include information equivalent to multiple frame related information 56.
[0080] Here, the method of determining the profile using the NN 62 is described as an example, but this is only an example. For example, the user can apply the determined profile according to the instruction given to the camera 10 via the UI system device 18 or a device directly or indirectly connected to the camera 10.
[0081] Here, although summary information is illustrated, this is only an example. For example, information indicating the types of all subjects shown in the dynamic image data 32, information indicating geographical features determined from the scenery shown in the dynamic image data 32, or information indicating common subjects shown in the dynamic image data 32 may be used instead of or together with the summary information. In this way, the information used instead of or together with the summary information may be any information as long as it is information related to the structure of the dynamic image data 32.
[0082] The acquisition unit 44A acquires the overall related information 74. The overall related information 74 includes the summary information 68 divided by time zone, the overall summary information 70, and the overall basic information 76.
[0083] The overall basic information 76 includes a frame identifier, model information, lens information, time information, shooting condition information, and a moving image identifier.
[0084] The frame identifier included in the overall basic information 76 is an identifier capable of identifying the frame 36 that contributed a certain level or more when acquiring the summary information.
[0085] As an example of a frame 36 that contributes a certain level or more when obtaining summary information, there can be cited a frame 36 associated with a portion of multi-frame related information 64 whose score output from NN72 becomes a certain value or more. The model information included in the overall basic information 76 is the model information obtained by statistics based on multiple portions of multi-frame related information 64 (for example, the model information that is most included in the multiple portions of multi-frame related information 64). The lens information included in the overall basic information 76 is the lens information obtained by statistics based on multiple portions of multi-frame related information 64 (for example, the lens information that is most included in the multiple portions of multi-frame related information 64). The time information included in the overall basic information 76 is the start time and end condition of the dynamic image data 32. In addition, the time information included in the overall basic information 76 may also be the time required to play the dynamic image data 32.
[0086] The image capturing condition information included in the overall basic information 76 is image capturing condition information obtained by statistics based on the plurality of partial multi-frame related information 64 (for example, the image capturing condition information that is most included in the plurality of partial multi-frame related information 64). The dynamic image identifier included in the overall basic information 76 is a unique identifier that is attached to the entire dynamic image data 32. In addition, the overall basic information 76 may include the bit rate of the dynamic image data 32 or the codec used for encoding and decoding the dynamic image data 32 as other basic information related to the entire dynamic image data 32.
[0087] As an example, Figure 6As shown, the generating unit 44B generates a moving image file 78. The moving image file 78 has metadata 80. The metadata 80 is data related to the moving image data 32 (that is, data accompanying the moving image data 32). Here, the moving image file is an example of an "image file" involved in the technology of the present invention. The metadata 80 is an example of "accompanying information" involved in the technology of the present invention.
[0088] To summarize the contents described so far, for example, Figure 7 As shown, the plurality of frame-related information 56 is stored (recorded) in the moving image file 78 as metadata 80. Furthermore, the plurality of partial multi-frame related information 64 (i.e., information including the scene information 61 and the basic information 66 divided by period) extracted from the plurality of frame-related information 56 is also stored in the moving image file 78 as metadata 80. Furthermore, the overall related information 74 (i.e., information including the summary information 68 divided by time band, the overall summary information 70, and the overall basic information 76) extracted from the plurality of partial multi-frame related information 64 is also stored in the moving image file 78 as metadata 80.
[0089] As an example, Figure 8 As shown, the second adding unit 44D adds the frame related information 56 acquired by the acquiring unit 44A for each frame 36 to the moving image file 78. At this time, for example, Fig. 9 As shown, the image recognition result information 54 included in the frame-related information 56 assigned to the dynamic image file 78 is classified into a plurality of categories. For example, a subject identifier (in the example of FIG. 1 ) as a unique identifier is attached to each subject (e.g., the person 12A, the person 12B, and each combination of the person 12A and the person 12B) that appears in the frame 36. Fig. 9 In the example shown, they are "#1" and "#2"). Then, the subject identifier is assigned multiple categories such as type category, attribute category, and position category. Fig. 9 In the example shown, multiple categories are hierarchically arranged for the type category and the attribute category. Categories of subordinate concepts or derived concepts of the upper hierarchy are arranged in the lower hierarchy. Fig. 9 In the example shown, information about the person 12A is allocated to "#1", and information about the person 12B is allocated to "#2".
[0090] The category type is a category indicating the type of the subject. Fig. 9 In the example shown, "person" is assigned as the type of the subject in the category category. A gender category and a name category are provided in the lower level of the category category. The gender category is a category indicating the gender, and the name category is a category indicating the name of the subject (for example, a common noun or a proper noun, etc.).
[0091] The attribute category is a category indicating the attribute of the subject. Fig. 9 In the example shown, action categories, expression categories, and clothing categories are provided, and color categories are provided as a lower level of clothing categories. The action category is a category that indicates the action of the subject. The expression category is a category that indicates the expression of the subject. The clothing category is a category that indicates the type of clothing worn by the subject. The color category is a category that indicates the color of the clothing worn by the subject.
[0092] The position category is a category indicating the position of the subject within the frame 36. Fig. 9 In the example shown, coordinate information that can identify the position of person 12A is assigned to "#1", and coordinate information that can identify the position of person 12B is assigned to "#2". The coordinate information is, for example, the two-dimensional coordinates of the upper left corner of a bounding box (i.e., a quadrilateral box surrounding the subject shown in frame 36) obtained by AI-based image recognition processing and the two-dimensional coordinates of the lower right corner of the bounding box.
[0093] In the moving image file 78 , various information included in the frame basic information 58 may be hierarchically classified by category in the same manner as the image recognition result information 54 .
[0094] As an example, Fig.10 As shown, the second assigning unit 44D assigns a plurality of partial multi-frame related information 64 to the moving image file 78 by including the plurality of partial multi-frame related information 64 acquired by the acquiring unit 44A in the metadata 80. The plurality of frame related information 56 and the plurality of partial multi-frame related information 64 included in the metadata 80 are examples of the "second supplementary information" involved in the technology of the present invention.
[0095] As an example, Fig.11 As shown, the first assigning unit 44C assigns the overall related information 74 to the moving image file 78 by including the overall related information 74 acquired by the acquiring unit 44A in the metadata 80. The overall related information 74 included in the metadata 80 is an example of the "first supplementary information" and "information related to all frames included in the frame group" involved in the technology of the present invention.
[0096] As an example, Fig.12 As shown in FIG. 1 , metadata 80 has a hierarchical structure. The hierarchical structure has an upper layer, a middle layer, and a lower layer. The upper layer stores information related to the entire frame group 34. The middle layer stores information related to the frame group 34. Figure 4 The information about the plurality of frames 36 obtained by dividing the frame group 34 during the period shown is stored in the lower layer. Fig.12In the example shown, the overall related information 74 is stored in the upper layer, the partial multi-frame related information 64 is stored in the middle layer, and the frame related information 56 is stored in the lower layer. That is, there are a plurality of partial multi-frame related information 64 in the layer below the overall related information 74, and there are a plurality of frame related information 56 in the layer below the partial multi-frame related information 64. In this way, the metadata 80 is hierarchical, and the user can easily identify the information in the metadata 80.
[0097] As an example, Fig.13 As shown, in the imaging device 10 , the control unit 44E stores the moving image file 78 obtained as described above in the NVM 46 .
[0098] In addition, Fig.13 In the illustrated example, the moving image file 78 is stored in the NVM 46 , but this is merely an example, and the moving image file 78 may be stored in one or more storage media other than the NVM 46 .
[0099] The storage medium may be any medium that is directly or indirectly connected to the camera device 10 by wired or wireless means. Examples of the storage medium include DVD (Digital Versatile Disc), USB (Universal Serial Bus) memory, SSD (Solid State Drive), HDD (Hard Disk Drive), and tape drive.
[0100] Next, refer to Fig.14 , the function of the camera device 10 is described. Fig.14 The processing flow represented by the flowchart shown is an example of the “generation method” involved in the technology of the present invention.
[0101] exist Fig.14 In the image file creation process shown in FIG. 1 , first, in step ST10, the acquisition unit 44A acquires the dynamic image data 32 (see FIG. 1 ) from the image sensor 40. Figure 3 ). After executing the process of step ST10, the image file creation process proceeds to step ST12.
[0102] In step ST12, the acquisition unit 44A acquires image recognition result information 54 (see FIG. 5 ) by performing an AI-based image recognition process on each frame 36 included in the frame group 34 of the dynamic image data 32 acquired in step ST10. Figure 3 ). After executing the process of step ST12, the image file creation process proceeds to step ST14.
[0103] In step ST14, the acquisition unit 44A acquires frame-related information 56 (see FIG. 1 ) for each frame 36 included in the frame group 34 of the dynamic image data 32 acquired in step ST10. Figure 3 ). The frame related information 56 includes the image recognition result information 54 and the frame basic information 58 (refer to Figure 3 ). After executing the process of step ST14, the image file creation process proceeds to step ST16.
[0104] In step ST16, the acquisition unit 44A acquires a plurality of frame-related information groups 60 (see FIG. 1 ) from all frame-related information 56 associated with the frame group 34 of the moving image data 32 acquired in step ST10. Figure 4 ). After executing the process of step ST16, the image file creation process proceeds to step ST18.
[0105] In step ST18, the acquisition unit 44A acquires scene information 61 (see Figure 4 ). The scene information 61 is obtained by using the score output from the NN 62 by inputting the frame related information group 60 to the NN 62. After executing the process of step ST18, the image file creation process proceeds to step ST20.
[0106] In step ST20, the acquisition unit 44A acquires a portion of the multi-frame related information 64 (see Figure 4 ). A portion of the multi-frame related information 64 includes scene information 61 and basic information 66 divided by period (refer to Figure 4 ). After executing the process of step ST20, the image file creation process proceeds to step ST22.
[0107] In step ST22, the acquisition unit 44A acquires the summary information (refer to Figure 5 ). The summary information is obtained by using the score output from the NN 72 by inputting a part of the multi-frame related information 64 acquired in step ST20 into the NN 72. After the process of step ST22 is performed, the image file creation process proceeds to step ST24.
[0108] In step ST24, the acquisition unit 44A acquires the overall related information 74 (see Figure 5 The overall related information 74 includes summary information and overall basic information 76 (refer to Figure 5 ). After executing the process of step ST22, the image file creation process proceeds to step ST26.
[0109] In step ST26, the generating unit 44B generates a moving image file 78 (see FIG. 1 ) including the moving image data 32 and the metadata 80 acquired in step ST10. Figure 6 ). After executing the process of step ST26, the image file creation process proceeds to step ST28.
[0110] In step ST28, the second adding unit 44D adds the frame related information 56 acquired in step ST14 to the metadata 80, thereby adding the frame related information 56 to the moving image file 78 (see Figure 8 ). After executing the process of step ST28, the image file creation process proceeds to step ST30.
[0111] In step ST30, the second adding unit 44D adds a portion of the multi-frame related information 64 acquired in step ST20 to the moving image file 78 (see Fig.10 ). After executing the process of step ST30, the image file creation process proceeds to step ST32.
[0112] In step ST32, the first adding unit 44C adds the overall related information 74 acquired in step ST24 to the moving image file 78 (see Fig.11 ). After executing the process of step ST32, the image file creation process proceeds to step ST34.
[0113] In step ST34, the control unit 44E stores the moving image file 78 obtained by executing the processing of steps ST10 to ST32 in the NVM 46 (see Fig.13 ). After executing the process of step ST34, the image file creation process ends.
[0114] As described above, in the camera device 10, the moving image data 32 is acquired to generate the moving image file 78 including the moving image data 32. Here, in the past, in order to understand the content of the moving image data 32, the user needs to play the moving image data 32. However, in the camera device 10, the overall related information 74 is acquired as information about all the frames 36 included in the frame group 34 of the moving image data 32, and is added to the moving image file 78. Therefore, even if the user (for example, the user or various devices) does not play the moving image data 32, by referring to the overall related information 74, it is possible to understand the content of all the frames 36 included in the frame group 34. In addition, in the camera device 10, as information about a part of the plurality of frames 36 included in the frame group 34 of the moving image data 32, the part of the multi-frame related information 64 is acquired and added to the moving image file 78. Therefore, even if the user does not play the moving image data 32, by referring to the part of the multi-frame related information 64, it is possible to understand the content of a part of the plurality of frames 36 included in the frame group 34. As described above, according to the imaging device 10 according to the present embodiment, the user or the like can understand the content of the moving image data 32 without playing the moving image data 32 .
[0115] Furthermore, in the imaging device 10, the frame-related information 56 is acquired as information about one frame 36 included in the frame group 34 of the moving image data 32, and is added to the moving image file 78. Therefore, even if the user does not play the moving image data 32, by referring to the frame-related information 56, the user can understand the content of one frame 36 included in the frame group 34. Thus, whether or not a frame 36 is important can be determined in units of frames 36.
[0116] Furthermore, in the imaging device 10, the frame-related information 56 is added to the moving image file 78. The frame-related information 56 includes the image recognition result information 54. The image recognition result information 54 includes the scene information 54A. The scene information 54A is information related to the scene determined from the frame 36. Therefore, even if the user does not play the moving image data 32, by referring to the scene information 54A included in the image recognition result information 54 of the frame-related information 56, it is possible to grasp the scene determined from the frame 36. Thus, whether or not a frame 36 is important can be determined in units of frames 36.
[0117] Furthermore, in the camera device 10, a part of the multi-frame related information 64 is given to the dynamic image file 78. The part of the multi-frame related information 64 includes scene information 61. The scene information 61 is information related to the scene determined based on a part of the plurality of frames 36 included in the frame group 34 of the dynamic image data 32 (that is, a plurality of frames 36 associated with the frame related information group 60). Therefore, even if the user does not play the dynamic image data 32, by referring to the scene information 61 included in the part of the multi-frame related information 64, it is possible to grasp the scene determined based on a part of the plurality of frames 36 included in the frame group 34 of the dynamic image data 32. In this way, it is possible to determine whether a frame 36 is important in units of a plurality of frames 36.
[0118] [First Modification] In the above embodiment, an example is given in which the image recognition result information 54 and the frame basic information 58 are included in the frame related information 56, but the technology of the present invention is not limited to this. For example, the frame related information 56 and a part of the multi-frame related information 64 may include information related to the change of the subject in the dynamic image data 32.
[0119] At this time, for example, Fig.15 As shown, the acquisition unit 44A acquires the subject change information 82 and includes it in the frame related information 56. The subject change information 82 is information related to the change of the person 12A shown in the plurality of frames 36 of the dynamic image data 32. As an example of the information related to the change of the person 12A, a plurality of frames 36 that are temporally preceding and following each other (in the frame 36 of the dynamic image data 32) can be cited. Fig.15 In the example shown, the movement vector of the person 12A shown in two temporally adjacent frames 36 or information based on the movement vector. The information based on the movement vector refers to, for example, information indicating the movement direction of the person 12A or information indicating the movement amount of the person 12A. In addition, as another example of information related to the change of the person 12A, there can be cited information related to the change of the expression of the person 12A shown in multiple frames 36 that are temporally adjacent, information related to the change of the optical element (brightness or hue) of the person 12A, or information related to the change of the size of the person 12A within the frame 36.
[0120] Furthermore, the acquisition unit 44A acquires representative subject change information 84, and includes the acquired representative subject change information 84 in a portion of multi-frame related information 64. The representative subject change information 84 included in a portion of multi-frame related information 64 is information obtained based on one or more subject change information 82 included in one or more frame related information 56 corresponding to the portion of multi-frame related information 64. For example, main subject change information 82 that satisfies specified conditions among the one or more subject change information 82 is taken as the representative subject change information 84. As the subject change information 82 that satisfies the specified conditions, for example, the subject change information 82 of the person 12A with the largest size or the subject change information 82 of the person 12A with the greatest degree of change (for example, the subject change information 82 with the largest motion vector) can be cited. Furthermore, in Fig.15 In the example shown, as representative subject change information 84, for example, information “a person is moving to the left” can be cited.
[0121] In the first modification, the subject change information 82 and the representative subject change information 84 are examples of “change information” according to the technique of the present invention.
[0122] As described above, in the imaging device 10 according to the first modification, the frame-related information 56 includes the subject change information 82. Therefore, even if the user does not play the dynamic image data 32, by referring to the subject change information 82 in the frame-related information 56, the user can grasp the change of the subject (in this case, the person 12A, for example) in units of frames 36. Thus, whether or not a frame 36 is important can be determined in units of frames 36.
[0123] Furthermore, in the imaging device 10 according to the first modification, the part of the multi-frame related information 64 includes the representative subject change information 84. Therefore, even if the user does not play the dynamic image data 32, by referring to the representative subject change information 84 in the part of the multi-frame related information 64, the user can grasp the change of the subject in units of the part of the plurality of frames 36 included in the frame group 34. Thus, whether or not the frame 36 is important can be determined in units of the plurality of frames 36.
[0124] [Second Modification] In the first modification, the frame-related information 56 includes the subject change information 82 and the part of the multi-frame related information 64 includes the representative subject change information 84. However, the technology of the present invention is not limited to this. For example, the frame-related information 56 and the part of the multi-frame related information 64 may include information indicating a change in the position or orientation of the imaging device 10.
[0125] At this time, for example, Fig.16As shown, the acquisition unit 44A acquires geometric information 86 in units of frames 36. The geometric information 86 is geometric information related to the camera device 10. The geometric information refers to, for example, the position of the camera device 10, the posture of the camera device 10, and the shooting orientation of the camera device 10. As the geometric information 86, for example, there can be cited position information 86A, posture information 86B, and shooting orientation information 86C. The position information 86A is information related to the position of the camera device 10. The posture information 86B is information related to the posture of the camera device 10. The shooting orientation information 86C is information that expresses the shooting direction (that is, the direction of the optical axis) of the camera device 10 in terms of orientation.
[0126] The camera device 10 is provided with a GNSS (Global Navigation Satellite System) receiver 88, an inertial sensor 90, and a geomagnetic sensor 92. The GNSS receiver 88, the inertial sensor 90, and the geomagnetic sensor 92 are connected to the processor 44. The GNSS receiver 88 receives radio waves transmitted from a plurality of satellites 94. The inertial sensor 90 measures a physical quantity (e.g., angular velocity and acceleration) representing the three-dimensional inertial motion of the camera device 10, and outputs an inertial sensor signal representing the measurement result. The geomagnetic sensor 92 detects geomagnetism, and outputs a geomagnetic sensor signal representing the detection result.
[0127] The acquisition unit 44A calculates the latitude, longitude, and altitude that can identify the current position of the camera device 10 as the position information 86A based on the radio waves received by the GNSS receiver 88. In addition, the acquisition unit 44A calculates the posture information 86B (for example, information defined by the yaw angle, the roll angle, and the pitch angle) based on the inertial sensor signal input from the inertial sensor 90. In addition, the acquisition unit 44A calculates the camera orientation information 86C based on the inertial sensor signal input from the inertial sensor 90 and the geomagnetic sensor signal input from the geomagnetic sensor 92. In addition, the acquisition unit 44A calculates the shooting posture of the camera device 10 (whether the long side direction of the camera is vertical or horizontal) based on the information of the inertial sensor 90.
[0128] As an example, Fig.17 As shown, the acquisition unit 44A acquires the geometric change information 96 and includes it in the frame related information 56. The geometric change information 96 is related to the plurality of frames 36 (in Fig.17The example shown is information about the change of geometric information 86 between two temporally adjacent frames 36). For example, the geometric change information 96 includes the absolute value of the difference between the position information 86A of the multiple frames 36, the absolute value of the difference between the posture information 86B of the multiple frames 36, and the absolute value of the difference between the camera orientation information 86C. Here, the absolute value of the difference is illustrated, but a ratio can also be used instead of the absolute value of the difference. As long as the difference can be determined, it can be any indicator. In addition, the absolute value of the difference is also illustrated below, which can be said to be the same.
[0129] The acquisition unit 44A acquires representative geometric change information 98, and includes the acquired representative geometric change information 98 in a portion of the multi-frame related information 64. The representative geometric change information 98 is information obtained based on one or more geometric change information 96 included in one or more frame related information 56 corresponding to the portion of the multi-frame related information 64. For example, the geometric change information 96 that satisfies the specified condition among the one or more geometric change information 96 is used as the representative geometric change information 98.
[0130] As a first example of geometric change information 96 that satisfies the specified conditions, there may be cited geometric change information 96 in which the degree of change of geometric information 86 is the greatest, geometric change information 96 in which the degree of change of geometric information 86 is the smallest, or geometric change information 96 having information equivalent to a central value among a plurality of geometric change information 96.
[0131] As a second example of geometric change information 96 that satisfies the specified conditions, there may be cited geometric change information 96 in which the absolute value of the difference between the position information 86A between multiple frames 36 is the largest, geometric change information 96 in which the absolute value of the difference between the position information 86A between multiple frames 36 is the smallest, or geometric change information 96 in which the absolute value of the difference between the position information 86A between multiple frames 36 is the central value.
[0132] As a third example of geometric change information 96 that satisfies the specified conditions, there may be cited geometric change information 96 in which the absolute value of the difference between posture information 86B between multiple frames 36 is the largest, geometric change information 96 in which the absolute value of the difference between posture information 86B between multiple frames 36 is the smallest, or geometric change information 96 in which the absolute value of the difference between posture information 86B between multiple frames 36 is the central value.
[0133] As a fourth example of geometric change information 96 that satisfies the specified conditions, there may be cited geometric change information 96 in which the absolute value of the difference between the camera orientation information 86C between multiple frames 36 is the largest, geometric change information 96 in which the absolute value of the difference between the camera orientation information 86C between multiple frames 36 is the smallest, or geometric change information 96 in which the absolute value of the difference between the camera orientation information 86C between multiple frames 36 is the central value.
[0134] In addition, in the second modification, the geometric change information 96 and the representative geometric change information 98 are examples of the “change information” involved in the technology of the present invention.
[0135] As described above, in the imaging device 10 according to the second modification, the frame-related information 56 includes the geometric change information 96. Therefore, even if the user does not play the dynamic image data 32, by referring to the geometric change information 96 in the frame-related information 56, the user can grasp the change in the position or orientation of the imaging device 10 in units of frames 36. Thus, whether or not a frame 36 is important can be determined in units of frames 36.
[0136] Furthermore, in the imaging device 10 according to the second modification, the part of the multi-frame related information 64 includes the representative geometric change information 98. Therefore, even if the user does not play the dynamic image data 32, by referring to the representative geometric change information 98 in the part of the multi-frame related information 64, the user can grasp the change in the position or orientation of the imaging device 10 in units of a part of the plurality of frames 36 included in the frame group 34. Thus, whether or not a frame 36 is important can be determined in units of the plurality of frames 36.
[0137] [Third Modification] In the second modification, the frame-related information 56 includes the geometric change information 96 and the part of the multi-frame related information 64 includes the representative geometric change information 98, but the technology of the present invention is not limited to this. For example, the frame-related information 56 and the part of the multi-frame related information 64 may include information indicating the change of the viewing angle of the camera device 10.
[0138] At this time, for example, Fig.18 As shown in FIG. 1 , the acquisition unit 44A acquires the view angle change information 100 and includes it in the frame related information 56. The frame 36 is associated with the view angle information 102. The view angle information 102 is information related to the view angle set in the camera device 10 (for example, the focal length formed by the change of the zoom factor). As an example of the view angle change information 100, a plurality of frames 36 (in Fig.18 In the example shown, it is the absolute value of the difference in viewing angle information 102 between two temporally adjacent frames 36).
[0139] Furthermore, the acquisition unit 44A acquires representative viewpoint change information 104, and includes the acquired representative viewpoint change information 104 in a portion of the multi-frame related information 64. The representative viewpoint change information 104 is information obtained based on one or more viewpoint change information 100 included in one or more frame related information 56 corresponding to the portion of the multi-frame related information 64. For example, the viewpoint change information 100 that satisfies the specified condition among the one or more viewpoint change information 100 is used as the representative viewpoint change information 104.
[0140] As the perspective change information 100 that satisfies the specified conditions, for example, there can be cited the perspective change information 100 with the largest degree of change in the perspective information 102, the perspective change information 100 with the smallest degree of change in the perspective information 102, or the perspective change information 100 corresponding to the central value among multiple perspective change information 100.
[0141] In the third modification, the view angle change information 100 and the representative view angle change information 104 are examples of the “change information” involved in the technique of the present invention. This makes it possible to determine whether each frame 36 is an important frame 36 .
[0142] As described above, in the imaging device 10 according to the third modification, the frame-related information 56 includes the angle-of-view change information 100. Therefore, even if the user does not play the dynamic image data 32, by referring to the angle-of-view change information 100 in the frame-related information 56, the user can understand the change in the angle of view of the imaging device 10 in units of frames 36.
[0143] Furthermore, in the imaging device 10 according to the third modification, the representative angle of view change information 104 is included in the part of the multi-frame related information 64. Therefore, even if the user does not play the dynamic image data 32, by referring to the representative angle of view change information 104 in the part of the multi-frame related information 64, the user can grasp the change of the angle of view of the imaging device 10 in units of a part of the plurality of frames 36 included in the frame group 34. Thus, whether or not a frame 36 is important can be determined in units of the plurality of frames 36.
[0144] [Fourth Modification] In the third modification, the frame-related information 56 includes the view angle change information 100 and the part of the multi-frame related information 64 includes the representative view angle change information 104, but the technology of the present invention is not limited to this. For example, when the tracking mode for tracking the subject is set for the camera 10, the frame-related information 56 and the part of the multi-frame related information 64 may include information related to tracking the subject.
[0145] At this time, for example, Fig.19 As shown, the acquisition unit 44A acquires the tracking-related information 106 in units of the frame 36, and includes the acquired tracking-related information 106 in the corresponding frame-related information 56. The tracking-related information 106 is, for example, coordinates for determining the position of a tracking frame 106A (for example, an AF frame) for tracking the person 12A in the frame 36. As coordinates for determining the position of the tracking frame 106A, for example, the two-dimensional coordinates of the upper left corner of the tracking frame 106A and the two-dimensional coordinates of the upper right corner of the tracking frame 106A when the upper left corner of the frame 36 is used as the origin can be cited.
[0146] In addition, here, the coordinates for determining the position of the tracking frame 106A are exemplified as the tracking related information 106, but the technology of the present invention is not limited to this. For example, the tracking related information 106 may include information indicating a state in which a tracking mode is set, or may include information indicating features of a subject set as a tracking target (e.g., the type of subject or the attribute of the subject, etc.). In this way, the tracking related information 106 may be any information as long as it is information related to the tracking of the subject.
[0147] The acquisition unit 44A acquires a representative tracking-related information 108, and includes the acquired representative tracking-related information 108 in a portion of the multi-frame related information 64. The representative tracking-related information 108 is information obtained based on one or more tracking-related information 106 included in one or more frame related information 56 corresponding to the portion of the multi-frame related information 64. For example, the tracking-related information 106 that satisfies a specified condition among the one or more tracking-related information 106 is used as the representative tracking-related information 108.
[0148] The tracking-related information 106 satisfying the specified condition may be, for example, the tracking-related information 106 corresponding to the frame 36 having the tracking frame 106A that overlaps most with the tracking frames 106A of other frames 36 .
[0149] In the fourth modification, the tracking-related information 106 and the representative tracking-related information 108 are examples of “information related to tracking of a subject” according to the technique of the present invention.
[0150] As described above, in the imaging device 10 according to the fourth modification, the frame-related information 56 includes the tracking-related information 106. Therefore, even if the user does not play the dynamic image data 32, by referring to the tracking-related information 106 in the frame-related information 56, the user can grasp the information related to the tracking of the subject in units of the frames 36. Thus, whether or not a frame 36 is important can be determined in units of the frames 36.
[0151] Furthermore, in the imaging device 10 according to the fourth modification, the part of the multi-frame related information 64 includes the representative tracking related information 108. Therefore, even if the user does not play the dynamic image data 32, by referring to the representative tracking related information 108 in the part of the multi-frame related information 64, the user can grasp the information related to the tracking of the subject in units of the part of the multi-frame 36 included in the frame group 34. Thus, it is possible to determine whether the frame 36 is important in units of the multi-frame 36.
[0152] [Fifth Modification] In the fourth modification, the frame-related information 56 includes the tracking-related information 106 and the part of the multi-frame-related information 64 includes the representative tracking-related information 108, but the technology of the present invention is not limited to this. For example, the frame-related information 56 and the part of the multi-frame-related information 64 may include information related to the movement of the camera device 10.
[0153] At this time, for example, Fig. 20 As shown, the acquisition unit 44A acquires the pan-tilt information 110 in units of frames 36, and includes the acquired pan-tilt information 110 in the corresponding frame-related information 56. The pan-tilt information 110 is information related to pan and information related to pitch when the tracking mode is not set. For example, the pan-tilt information 110 includes information indicating the amount of pan between a plurality of frames 36 that are temporally adjacent (for example, two frames 36 that are temporally adjacent) and information indicating the amount of pitch.
[0154] In addition, regarding the translation amount and the pitch amount, for example, the inertial sensor 90 (refer to Fig.16 ) outputted by the inertial sensor signal. Here, as the pan-pitch information 110, information related to pan and information related to pitch when the tracking mode is not set are exemplified, but the technology of the present invention is not limited to this. For example, information related to pan and information related to pitch when the tracking mode is set may be used as the pan-pitch information 110.
[0155] Furthermore, although the pan amount and the pitch amount are exemplified here, the technology of the present invention is not limited thereto, and other physical quantities related to pan and pitch, such as pan speed and pitch speed, may be used instead of or together with the pan amount and the pitch amount. Furthermore, pan and pitch are merely examples, and information related to movement when the camera device 10 moves in the horizontal direction (for example, movement amount and movement speed, etc.) and information related to movement when the camera device 10 moves in the vertical direction may be used instead of or together with the pan-pitch information 110.
[0156] The acquisition unit 44A acquires representative pan-pitch information 112, and includes the acquired representative pan-pitch information 112 in a part of the multi-frame related information 64. The representative pan-pitch information 112 is information obtained based on one or more pan-pitch information 110 included in one or more frame related information 56 corresponding to the part of the multi-frame related information 64. For example, among the one or more pan-pitch information 110, the pan-pitch information 110 that satisfies the specified condition is taken as the representative pan-pitch information 112. As a first example of the pan-pitch information 110 that satisfies the specified condition, the pan-pitch information 110 with the largest or smallest pan amount can be cited. As a second example of the pan-pitch information 110 that satisfies the specified condition, the pan-pitch information 110 with the largest or smallest pitch amount can be cited. As a third example of the pan-pitch information 110 that satisfies the specified condition, the pan-pitch information 110 with the largest or smallest pan amount and the pitch amount can be cited.
[0157] In the fifth modification, the pan-tilt information 110 and the representative pan-tilt information 112 are examples of “information related to the movement of the imaging device” according to the technique of the present invention.
[0158] As described above, in the imaging device 10 according to the fifth modification, the frame-related information 56 includes the pan-tilt information 110. Therefore, even if the user does not play the dynamic image data 32, by referring to the pan-tilt information 110 in the frame-related information 56, the user can grasp the information related to pan (e.g., the pan amount) and the information related to pitch (e.g., the pitch amount) for each frame 36. Thus, whether or not a frame 36 is important can be determined for each frame 36.
[0159] Furthermore, in the imaging device 10 according to the fifth modification, the part of the multi-frame related information 64 includes the representative pan-tilt information 112. Therefore, even if the user does not play the dynamic image data 32, by referring to the representative pan-tilt information 112 in the part of the multi-frame related information 64, the user can grasp the information related to pan (for example, the pan amount) and the information related to pitch (for example, the pitch amount) for each of the part of the plurality of frames 36 included in the frame group 34. Thus, whether or not a frame 36 is important can be determined for each of the plurality of frames 36.
[0160] [Sixth Modification] In the fifth modification, the frame-related information 56 includes the pan-tilt information 110 and the part of the multi-frame related information 64 includes the representative pan-tilt information 112, but the technology of the present invention is not limited to this. For example, the frame-related information 56 and the part of the multi-frame related information 64 may include information related to the sensor included in the imaging device 10.
[0161] At this time, for example, Fig.21As shown, the acquisition unit 44A acquires the inertial sensor signal difference information 114 in units of frames 36, and includes the acquired inertial sensor signal difference information 114 in the corresponding frame related information 56. The inertial sensor signal difference information 114 is the absolute value of the difference between the inertial sensor signals 116 acquired successively in time. The inertial sensor signal 116 is acquired from the inertial sensor 90 (refer to FIG. 1 ) in units of frames 36. Fig.16 ) is obtained. An example of the inertial sensor signal difference information 114 is the absolute value of the difference between two inertial sensor signals 116 corresponding to two temporally adjacent frames 36.
[0162] The acquisition unit 44A acquires representative inertial sensor signal difference information 118, and includes the acquired representative inertial sensor signal difference information 118 in a part of the multi-frame related information 64. The representative inertial sensor signal difference information 118 is information obtained based on one or more inertial sensor signal difference information 114 included in one or more frame related information 56 corresponding to the part of the multi-frame related information 64. For example, inertial sensor signal difference information 114 that satisfies a specified condition among the one or more inertial sensor signal difference information 114 is used as the representative inertial sensor signal difference information 118. As an example of the inertial sensor signal difference information 114 that satisfies the specified condition, there can be cited the inertial sensor signal difference information 114 that has the largest or smallest absolute value of the difference between the two inertial sensor signals 116.
[0163] In the sixth modification, the inertial sensor signal difference information 114 and the representative inertial sensor signal difference information 118 are examples of “information related to the first sensor” involved in the technology of the present invention. Fig.16 ) is an example of the "first sensor" involved in the technology of the present invention.
[0164] As described above, in the imaging device 10 according to the sixth modification, the inertial sensor signal difference information 114 is included in the frame-related information 56. Therefore, even if the user does not play the dynamic image data 32, by referring to the inertial sensor signal difference information 114 in the frame-related information 56, the user can grasp the state of the imaging device 10 (for example, the posture or movement speed of the imaging device 10) in units of frames 36. Thus, whether or not a frame 36 is important can be determined in units of frames 36.
[0165] Furthermore, in the imaging device 10 according to the sixth modification, the representative inertial sensor signal difference information 118 is included in the part of the multi-frame related information 64. Therefore, even if the user does not play the dynamic image data 32, by referring to the representative inertial sensor signal difference information 118 in the part of the multi-frame related information 64, the user can grasp the state of the imaging device 10 (for example, the posture or movement speed of the imaging device 10) in units of a part of the plurality of frames 36 included in the frame group 34. Thus, whether or not a frame 36 is important can be determined in units of the plurality of frames 36.
[0166] In addition, in the sixth modification, as an example of the "information related to the first sensor" involved in the technology of the present invention, the example of the method in which the acquisition unit 44A acquires the inertial sensor signal difference information 114 and the representative inertial sensor signal difference information 118 is cited, but this is only an example. For example, the acquisition unit 44A may also acquire the inertial sensor signal difference information 114 and the representative inertial sensor signal difference information 118 from the geomagnetic sensor 92 (reference Fig.16 ) to obtain the geomagnetic sensor signal difference information and the representative geomagnetic sensor signal difference information from the geomagnetic sensor signal output by the frame. At this time, it is sufficient to include the geomagnetic sensor signal difference information in the frame related information 56 and include the representative geomagnetic sensor signal difference information in a part of the multi-frame related information 64.
[0167] Furthermore, in this sixth variant, the absolute value of the difference between the inertial sensor signals 116 is illustrated, but this is merely an example, and it may also be, for example, the ratio of one of the two inertial sensor signals 116 to the other, as long as it is information that can determine the degree of difference between the inertial sensor signals 116.
[0168] Furthermore, in the sixth variant, the inertial sensor 90 is cited as an example of the "first sensor" involved in the technology of the present invention, but this is only an example. For example, when the camera device 10 is equipped with various sensors such as a distance sensor or a temperature sensor, and the signals output from the various sensors are processed in the same manner as the inertial sensor signal difference information 114, the technology of the present invention can also be established. For example, when the camera device 10 is equipped with a distance sensor, information based on the distance measurement result obtained by using the distance sensor (for example, information indicating the distance of several places or an image with a reduced distance image, etc.) can also be processed in the same manner as the inertial sensor signal difference information 114. Furthermore, when the camera device 10 is equipped with a temperature sensor, information based on the measurement result obtained by using the temperature sensor (for example, the external temperature or an image with a reduced thermal image, etc.) can also be processed in the same manner as the inertial sensor signal difference information 114.
[0169] [Seventh Modification] In the sixth modification, the frame-related information 56 includes the inertial sensor signal difference information 114 and the part of the multi-frame related information 64 includes the representative inertial sensor signal difference information 118, but the technology of the present invention is not limited to this. For example, the frame-related information 56 and the part of the multi-frame related information 64 may include information corresponding to the recording time of the subject in the dynamic image data 32.
[0170] At this time, for example, Fig. 22 As shown, the acquisition unit 44A determines a plurality of frames 36 in which the person 12A is continuously shown in the dynamic image data 32. The acquisition unit 44A acquires two shooting times 120. The shooting time 120 refers to the time when one frame of image capture is performed. One of the two shooting times 120 is the shooting time 120 corresponding to the first frame 36 among the plurality of frames 36 determined by the acquisition unit 44A, and the other is the shooting time 120 corresponding to the last frame 36 among the plurality of frames 36 determined by the acquisition unit 44A.
[0171] The acquisition unit 44A calculates the recording time related information 122 based on the two shooting times 120. The recording time related information 122 is, for example, information indicating the elapsed time between the two shooting times 120 (i.e., the time during which the person 12A is continuously shown in the dynamic image data 32). In addition, here, as the recording time related information 122, the elapsed time between the two shooting times 120 is cited, but this is only an example, and the recording time related information 122 may also be the two shooting times 120 themselves, or may be a frame identifier that can identify the two frames 36 corresponding to the two shooting times 120.
[0172] The acquisition unit 44A includes the recording time related information 122 in the plurality of frame related information 56 corresponding to the plurality of frames 36 in which the person 12A appears continuously in the dynamic image data 32. In addition, the recording time related information 122 may be included in one or more frame related information 56 designated among the plurality of frame related information 56 corresponding to the plurality of frames 36 in which the person 12A appears continuously in the dynamic image data 32. In this case, for example, the recording time related information 122 may be included in the frame related information 56 corresponding to the first frame 36 or the last frame 36 among the plurality of frames 36 in which the person 12A appears continuously in the dynamic image data 32.
[0173] The acquisition unit 44A acquires the representative recording time related information 124, and includes the acquired representative recording time related information 124 in a part of the multi-frame related information 64. The representative recording time related information 124 is information obtained based on one or more recording time related information 122 included in one or more frame related information 56 corresponding to the part of the multi-frame related information 64. For example, the recording time related information 122 that satisfies the specified condition among the one or more recording time related information 122 is taken as the representative recording time related information 124. As an example of the recording time related information 122 that satisfies the specified condition, the recording time related information 122 with the longest or shortest elapsed time between two shooting moments 120 (that is, the time during which the person 12A is continuously shown in the dynamic image data 32) can be cited.
[0174] In the seventh modification, the recording time related information 122 and the representative recording time related information 124 are examples of “information corresponding to the recording time at which the subject in the moving image data is recorded” according to the technique of the present invention.
[0175] As described above, in the imaging device 10 according to the seventh modification, the frame-related information 56 includes the recording time-related information 122. Therefore, even if the user does not play the moving image data 32, by referring to the recording time-related information 122 in the frame-related information 56, the user can grasp the time during which a specific subject (for example, the person 12A) continues to appear in the moving image data 32 in units of frames 36. Thus, whether or not a frame 36 is important can be determined in units of frames 36.
[0176] Furthermore, in the imaging device 10 according to the seventh modification, the part of the multi-frame related information 64 includes the representative recording time related information 124. Therefore, even if the user does not play the dynamic image data 32, by referring to the representative recording time related information 124 in the part of the multi-frame related information 64, the user can grasp the time during which the specific subject (for example, the person 12A) continues to appear in the dynamic image data 32 in units of a part of the plurality of frames 36 included in the frame group 34. Thus, it is possible to determine whether the frame 36 is important in units of the plurality of frames 36.
[0177] [Eighth Modification] In the seventh modification, the frame-related information 56 includes the recording time-related information 122 and the part of the multi-frame-related information 64 includes the representative recording time-related information 124, but the technology of the present invention is not limited to this. For example, the frame-related information 56 and the part of the multi-frame-related information 64 may include information related to the operation of the camera device 10.
[0178] At this time, for example, Fig.23As shown, the acquisition unit 44A acquires the operation-related information 126 in units of frames 36, and includes the acquired operation-related information 126 in the corresponding frame-related information 56. The operation-related information 126 is information indicating an operation performed on the imaging device 10 (for example, an operation received by the UI system device 18).
[0179] The acquisition unit 44A acquires the representative operation-related information 127, and includes the acquired representative operation-related information 127 in a portion of the multi-frame related information 64. The representative operation-related information 127 is information obtained based on one or more operation-related information 126 included in one or more frame related information 56 corresponding to the portion of the multi-frame related information 64. For example, the operation-related information 126 that satisfies the specified condition among the one or more operation-related information 126 is taken as the representative operation-related information 127. As an example of the operation-related information 126 that satisfies the specified condition, the operation-related information 126 that indicates the most operations or the least operations among the plurality of operation-related information 126 included in all the frame related information 56 corresponding to the portion of the multi-frame related information 64 can be cited.
[0180] In the eighth modification, the operation-related information 126 and the representative operation-related information 127 are examples of “information related to the operation on the imaging device used for capturing and obtaining moving image data” involved in the technology of the present invention.
[0181] As described above, in the camera device 10 involved in the eighth modification, the frame-related information 56 includes the operation-related information 126. Therefore, even if the user does not play the dynamic image data 32, by referring to the operation-related information 126 in the frame-related information 56, it is possible to grasp the operation performed on the camera device 10 in units of frames 36. For example, regarding the frame 36 obtained during the operation of setting a menu on the camera device 10, the possibility that the user pays attention to the subject is low, and the possibility that the frame is not important to the user is high. Whether the frame 36 is obtained during the operation of setting a menu on the camera device 10, the user can determine it by referring to the operation-related information 126. Therefore, it is possible to determine whether it is an important frame 36 in units of frames 36.
[0182] Furthermore, in the imaging device 10 according to the eighth modification, the part of the multi-frame related information 64 includes the representative operation related information 127. Therefore, even if the user does not play the dynamic image data 32, by referring to the representative operation related information 127 in the part of the multi-frame related information 64, the user can grasp the operation performed on the imaging device 10 in units of the part of the multi-frame 36 included in the frame group 34. Thus, whether or not a frame 36 is important can be determined in units of the multi-frame 36.
[0183] [9th Modification] In the above-mentioned 8th Modification, an example of a method in which the frame-related information 56 includes the operation-related information 126 and an example of a method in which a part of the multi-frame related information 64 includes the representative operation-related information 127 is shown, but the technology of the present invention is not limited to this. For example, the frame-related information 56 and a part of the multi-frame related information 64 may include information related to a sensor that measures a subject in the dynamic image data 32 (for example, a physical quantity representing the current state of the subject or a feature of the subject, etc.).
[0184] At this time, for example, Fig.24 As shown, the acquisition unit 44A acquires the subject sensor information 128 in units of frames 36. The subject sensor information 128 is information obtained from a sensor mounted on the subject, that is, the person 12A. Fig.24 In the example shown, a wearable terminal 130 is shown as a sensor mounted on the person 12A. The wearable terminal 130 is a wristband-type wearable terminal, and is mounted on the wrist of the person 12A. The wearable terminal 130 has a communication function, and communicates with the camera 10.
[0185] The wearable terminal 130 is equipped with a pulse measuring function. The wearable terminal 130 measures the pulse of the person 12A and transmits information indicating the measured pulse to the imaging device 10 as the subject sensor information 128. In the imaging device 10, the communication I / F 42 receives the subject sensor information 128 transmitted from the wearable terminal 130, and the acquisition unit 44A acquires the received subject sensor information 128.
[0186] As an example, Fig.25 As shown, the acquisition unit 44A in the corresponding frame 36 (ie, Fig.24 The frame related information 56 of the frame 36 showing the person 12A shown includes the subject sensor information 128 acquired from the wearable terminal 130 .
[0187] Furthermore, the acquisition unit 44A acquires representative subject sensor information 132, and includes the acquired representative subject sensor information 132 in a portion of the multi-frame related information 64. The representative subject sensor information 132 is information obtained based on one or more subject sensor information 128 included in one or more frame related information 56 corresponding to the portion of the multi-frame related information 64. For example, the subject sensor information 128 that satisfies the specified condition among the one or more subject sensor information 128 is taken as the representative subject sensor information 132. As an example of the representative subject sensor information 132 that satisfies the specified condition, the subject sensor information 128 that shows the highest pulse among the plurality of subject sensor information 128 included in all the frame related information 56 corresponding to the portion of the multi-frame related information 64 can be cited.
[0188] In the ninth modification, the subject sensor information 128 and the representative subject sensor information 132 are examples of “information related to the second sensor for measuring the subject in the moving image data” according to the technique of the present invention.
[0189] As described above, in the camera device 10 involved in this 9th modification, the frame-related information 56 includes the subject sensor information 128. Therefore, even if the user does not play the dynamic image data 32, as long as the subject sensor information 128 in the frame-related information 56 is referred to, the physical condition (for example, pulse, blood pressure or blood sugar level) of a specific subject (for example, person 12A) can be grasped in units of frames 36. For example, when the pulse of person 12A is high, the possibility that person 12A is excited is high. That is, the frame 36 obtained by photographing the person 12A in an excited state is likely to be an important frame 36. And, for example, when the blood pressure of person 12A is low, the possibility that person 12A is physically unwell is high. And, for example, when this 9th modification is applied to the scene where person 12A undergoes surgery in a hospital, it is possible to grasp the medical behavior that has a great impact on the physical condition of the subject. Therefore, as long as the user refers to the subject sensor information 128, it is possible to determine whether it is an important frame 36 in units of frames 36.
[0190] Furthermore, in the imaging device 10 according to the ninth modification, the part of the multi-frame related information 64 includes the representative subject sensor information 132. Therefore, even if the user does not play the dynamic image data 32, by referring to the representative subject sensor information 132 in the part of the multi-frame related information 64, the user can grasp the physical condition (for example, pulse, blood pressure, or blood sugar level) of the specific subject (for example, the person 12A) in units of the part of the plurality of frames 36 included in the frame group 34. Thus, whether or not the frame 36 is important can be determined in units of the plurality of frames 36.
[0191] In addition, in this 9th variant, a wearable terminal 130 is illustrated, but this is merely an example and it may also be a smart phone or a tablet terminal, as long as it is an electronic device equipped with one or more sensors such as a thermometer or a blood pressure meter (i.e., a sensor that can measure the subject within the dynamic image data 32).
[0192] Furthermore, in the ninth modification, the example of measuring the pulse of the person 12A by the wearable terminal 130 is given, but this is only an example. For example, the measured information may be biological information such as blood pressure, breathing size, number of breathings, or blood sugar level of the subject shown in the dynamic image data 32. Furthermore, it is not limited to biological information. When the production line of a factory or the like is the imaging target area, it may also be information measured by a sensor installed on the production line.
[0193] [10th Modification] In the above embodiment, the summary information and the overall basic information 76 are exemplified as the information included in the overall related information 74, but the technology of the present invention is not limited to this. For example, the dynamic image data 32 is information related to a part of other dynamic image data, and the overall related information 74 may include information related to the other dynamic image data.
[0194] For example, Fig.26 As shown, when a part of the dynamic image data 134A included in the dynamic image file 134 different from the dynamic image file 78 is combined with the dynamic image data 32, the first adding unit 44C includes the other dynamic image file information 136 in the overall related information 74. The other dynamic image file information 136 is information for specifying the location of the dynamic image file 134.
[0195] If the video file 134 is stored in a memory such as the NVM 46 , the other video file information 136 is, for example, an address in the memory. Also, if the video file 134 exists on the Internet, the other video file information 136 is, for example, a URL (Uniform Resource Locator).
[0196] For example, the other moving image file information 136 is included in the metadata 134B of the moving image file 134, and the first assigning unit 44C acquires the other moving image file information 136 from the metadata 134B when a part of the moving image data 134A is combined with the moving image data 32. Then, the first assigning unit 44C includes the other moving image file information 136 acquired from the metadata 134B in the overall related information 74, thereby assigning the other moving image file information 136 to the moving image file 78.
[0197] In this way, when a part of the dynamic image data 134A is combined with the dynamic image data 32, the overall related information 74 includes the other dynamic image file information 136. The other dynamic image file information 136 is information for identifying the location where the part of the dynamic image data 134A is derived from. Therefore, even if the user does not play the dynamic image data 32, by referring to the other dynamic image file information 136 in the overall related information 74, it is possible to identify which dynamic image file the part of the dynamic image data 32 is derived from.
[0198] In addition, in the present tenth modification, a method in which a part of the dynamic image data 134A included in the dynamic image file 134 is combined with the dynamic image data 32 is described, but the technology of the present invention is not limited to this. For example, even when a part of the dynamic image data included in one or more dynamic image files other than the dynamic image file 134 is combined with a part of the dynamic image data 134A into the dynamic image data 32, another dynamic image file information 136 may be included in the overall related information 74 in the same manner as in the tenth modification.
[0199] Furthermore, in the tenth modification, information for identifying the location of another dynamic image file 134 is exemplified as another dynamic image file information 136, but the technology of the present invention is not limited thereto. Another dynamic image file information 136 may include a thumbnail image (e.g., a thumbnail still image or a thumbnail dynamic image) related to a portion of the dynamic image data 134A, or may include an identifier or the number of frames for identifying a portion of the dynamic image data 134A. In this way, it is sufficient as long as the information related to a portion of the dynamic image data 134A is included in the another dynamic image file information 136.
[0200] [Other Variations] In the above embodiment, an example of the image file creation process being executed by the information processing device 38 in the camera device 10 is given for explanation, but the technology of the present invention is not limited to this. Fig. 27 As shown, the image file creation process may be performed by a computer 140 in an external device 138 that is communicatively connected to the camera device 10 via a network 137 such as a LAN (Local Area Network) or a WAN (Wide Area Network). As an example of the computer 140, a server computer for a cloud service can be cited. The computer 140 is an example of an "information processing device" involved in the technology of the present invention.
[0201] exist Fig. 27 In the illustrated example, the computer 140 includes a processor 142, a storage space (storage) 144, and a memory 146. The storage space 144 stores the image file creation program 52.
[0202] The camera 10 requests the external device 138 to execute the image file creation process via the network 137. In response, the processor 142 of the external device 138 reads the image file creation program 52 from the storage space 144 and executes the image file creation program 52 on the memory 146. The processor 142 performs the image file creation process in accordance with the image file creation program 52 executed on the memory 146. Then, the processor 142 provides the processing result obtained by executing the image file creation process to the camera 10 via the network 137.
[0203] exist Fig. 27 , an example of a method of causing the external device 138 to execute the image file creation process is shown, but this is only an example. For example, the image capture device 10 and the external device 138 may execute the image file creation process in a distributed manner, or the image capture device 10 and a plurality of devices including the external device 138 may execute the image file creation process in a distributed manner.
[0204] In the above embodiment, an example of recording the frame-related information 56, the part of the multi-frame-related information 64, and the whole-related information 74 in the metadata 80 of the moving image file 78 is given for explanation, but the technology of the present invention is not limited to this. For example, the frame 36 may be extracted from the moving image data 32 of the moving image file 78, an image file may be generated for the extracted frame 36, and the frame-related information 56, the part of the multi-frame-related information 64, and the whole-related information 74 may be added to the generated image file. As the image file in this case, an image file in a format in which additional information can be added to a region different from the image data (i.e., a recordable format) is used.
[0205] The moving image file may be in any file format as long as additional information can be added to a region different from the image data.
[0206] The file format can be any one of MPEG (Moving Picture Experts Group)-4, H.264, MJPEG (Motion JPEG), HEIF (High Efficiency Image File Format), AVI (Audio Video Interleave), MOV (QuickTime fileformat), WMV (Windows Media Video), and FLV (Flash Video).
[0207] In addition, from the viewpoint of providing metadata (additional information) described in the above embodiment, HEIF moving image data is preferred.
[0208] The items and number of additional information that can be added to an image file vary depending on the file format, and by updating the version information of the image file, additional information can be added to new items. The items of additional information represent the viewpoint when adding the additional information (ie, the category into which the information is classified).
[0209] In the above embodiment, an example in which the image file creation program 52 is stored in the NVM 46 is described, but the technology of the present invention is not limited to this. For example, the image file creation program 52 can be stored in a non-transitory storage medium readable by a mobile computer such as an SSD, a USB memory, or a magnetic tape. The image file creation program 52 stored in the non-transitory storage medium is installed in the camera device 10. The processor 44 executes the image file creation process according to the image file creation program 52.
[0210] Alternatively, the image file creation program 52 may be stored in advance in a storage device such as another computer or server device connected to the imaging device 10 via a network, and downloaded in response to a request from the imaging device 10 and installed in the imaging device 10 .
[0211] In addition, the storage device such as another computer or server device connected to the imaging device 10 or the NVM 46 does not need to store all of the image file creation program 52 in advance, and a part of the image file creation program 52 may be stored in advance.
[0212] And, in Figure 2 The image pickup device 10 shown has the information processing device 38 built therein, but the technology of the present invention is not limited thereto, and for example, the information processing device 38 may be provided outside the image pickup device 10 .
[0213] In the above-mentioned embodiment, an example of how the technology of the present invention is implemented through a software structure is given for illustration, but the technology of the present invention is not limited to this, and devices including ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array) or PLD (Programmable Logic Device) can also be applied.
[0214] Furthermore, a combination of a hardware structure and a software structure may also be used.
[0215] As hardware resources for executing the image file creation process described in the above embodiment, various processors shown below can be used. As a processor, for example, a general-purpose processor CPU that functions as a hardware resource for executing the image file creation process by executing software, that is, a program can be cited. Also, as a processor, for example, a processor having a circuit structure specially designed for executing a specific process, that is, a dedicated circuit, such as FPGA, PLD, or ASIC can be cited. Each processor has a built-in or connected memory, and each processor executes the image file creation process by using the memory.
[0216] The hardware resource for executing the image file creation process may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA). Furthermore, the hardware resource for executing the image file creation process may also be one processor.
[0217] As an example of a system composed of one processor, there is first a method in which one processor is composed of a combination of one or more CPUs and software and the processor functions as a hardware resource for executing image file creation processing. There is second a method in which a processor that realizes the overall function of a system including multiple hardware resources for executing image file creation processing is implemented by using one IC (Integrated Circuit) chip represented by SoC (System-on-a-chip) or the like. In this way, the image file creation processing is implemented using one or more of the above-mentioned various processors as hardware resources.
[0218] Furthermore, as the hardware structure of these various processors, more specifically, a circuit composed of a combination of circuit elements such as semiconductor elements can be used. Furthermore, the above-mentioned image file creation process is only an example. Therefore, unnecessary steps can be deleted, new steps can be added, or the processing order can be changed within the scope of the main purpose.
[0219] The recorded contents and illustrated contents shown above are detailed descriptions of the parts involved in the technology of the present invention, and are only an example of the technology of the present invention. For example, the descriptions related to the above-mentioned structure, function, action and effect are descriptions related to an example of the structure, function, action and effect of the parts involved in the technology of the present invention. Therefore, without departing from the scope of the main purpose of the technology of the present invention, the recorded contents and illustrated contents shown above can be processed as follows: delete unnecessary parts, add new elements or replace them. In addition, in order to avoid complexity and make it easy to understand the parts involved in the technology of the present invention, in the recorded contents and illustrated contents shown above, descriptions related to technical common sense, etc. that do not need to be specifically explained are omitted on the basis of being able to implement the technology of the present invention.
[0220] In this specification, the grammatical concept of "A or B" includes not only the concept of "any one of A and B", but also the concept with the same meaning as "at least one of A and B". That is, "A or B" includes the following meanings: it can be only A, it can be only B, or it can be a combination of A and B. In addition, in this specification, when three or more things are connected and expressed with "or", the same way of thinking as "A or B" is also applied.
[0221] All documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as if each individual document, patent application, or technical standard was specifically and individually described as being incorporated by reference.
Claims
1. An image file, comprising: Dynamic image data, including groups of frames; and Additional information, The supplementary information includes first supplementary information and second supplementary information. The first supplementary information is information related to all frames included in the frame group. The second supplementary information is information related to a part of a plurality of frames in the frame group.
2. The image file according to claim 1, wherein: The dynamic image data is information related to a part of other dynamic image data, The first supplementary information includes information related to the other moving image data.
3. The image file according to claim 1 or 2, wherein: The first supplementary information includes information on the structure of the moving image data.
4. The image file according to any one of claims 1 to 3, wherein: The second supplementary information further includes information related to one frame included in the frame group.
5. The image file according to claim 4, wherein: The second supplementary information includes scene information related to the scene determined based on the part of the plurality of frames or the scene determined based on the one frame.
6. The image file according to any one of claims 1 to 5, wherein: The second supplementary information includes change information regarding a change in a subject in the moving image data or a change in a position, orientation, or angle of view of an imaging device used in capturing the moving image data.
7. The image file according to claim 6, wherein: The change information includes information related to tracking of the subject or information related to movement of the imaging device.
8. The image file according to any one of claims 1 to 7, wherein: The second supplementary information includes information on a first sensor included in an imaging device used for capturing the moving image data.
9. The image file according to any one of claims 1 to 8, wherein: The second supplementary information includes information related to a second sensor that measures a subject in the moving image data.
10. The image file according to any one of claims 1 to 9, wherein: The second supplementary information includes information related to an operation of an imaging device used in capturing images for obtaining the moving image data.
11. The image file according to any one of claims 1 to 10, wherein: The second supplementary information includes information corresponding to a recording time at which a subject in the moving image data is recorded. 12 . An information processing device comprising a processor for generating the image file according to claim 1 .
13. A camera device comprising: A sensor for capturing the dynamic image data included in the image file according to any one of claims 1 to 11; and A processor generates the image file according to the signal output from the sensor.
14. A method for generating an image file including dynamic image data including a frame group and accompanying information, the method comprising the following steps: An acquisition step of acquiring the dynamic image data; a first adding step of adding first additional information, which is information related to all frames included in the frame group, to the image file as the additional information; and The second adding step is to add, to the image file, second additional information which is information on a part of the plurality of frames in the frame group as the additional information.
