Information processing device and information processing method

Abstract

This information processing method comprises: an association step for associating first imaging processing for generating a first image file including first image data obtained by imaging a first subject with second imaging processing for generating a second image file including second image data obtained by imaging a second subject; an acquisition step for acquiring first subject information pertaining to the first subject; and an imparting step for imparting the first subject information to a second image file by including the first subject information in second incidental information recorded in the second image file.

Description

Information processing device and information processing method

[0001] The technology of the present disclosure relates to an information processing device and an information processing method.

[0002] International Publication No. 2004-061387 discloses a video capture system that acquires video information of an object from multiple viewpoints. The video capture system described in International Publication No. 2004-061387 includes cameras, a detection means, a synchronization means, a data addition means, and a calibration means. The cameras are multiple three-dimensionally movable cameras that acquire video data of moving images. The detection means acquires camera parameters for each camera. The synchronization means synchronizes the multiple cameras. The data addition means adds correspondence information that associates the video data of the synchronized moving images from each camera and between the video data of the moving images and the camera parameters. The calibration means calibrates the video data of each moving image with the corresponding camera parameters based on the correspondence information, thereby obtaining information for analyzing the movement and posture of the object.

[0003] The video capture system described in International Publication No. 2004-061387 includes a video data storage means and a camera parameter storage means. The video data storage means stores video data with corresponding information added for each frame. The camera parameter storage means stores camera parameters with corresponding information added. The corresponding information is a frame count of video data of moving images acquired from one of multiple cameras.

[0004] Japanese Patent Laid-Open Publication No. 2004-072349 discloses an imaging device including a first imaging means, a second imaging means, a first field of view control means, and a field of view control means. In the imaging device described in Patent Literature 2, the first imaging means captures an image in a first direction, and the second imaging means captures an image in a second direction. The first field of view control means controls the field of view of the first imaging means to a different first field of view. The second field of view control means controls the field of view of the second imaging means to a second field of view adjacent to the first field of view in the horizontal plane. In the imaging device described in Japanese Patent Laid-Open Publication No. 2004-072349, the first field of view control means and the second field of view control means do not share a ridge line, and the lens center of the virtual imaging means having the first field of view and the lens center of the virtual imaging means having the second field of view approximately coincide with each other.

[0005] Furthermore, Japanese Patent Application Laid-Open Publication No. 2014-011633 discloses a wireless synchronization system using multiple imaging devices, and Japanese Patent Application Laid-Open Publication No. 2017-135754 discloses an imaging system using multiple cameras.

[0006] An embodiment of the technique of the present disclosure provides an information processing device and an information processing method that can improve the convenience of image files.

[0007] A first aspect of the technology of the present disclosure is an information processing method comprising: a linking step of linking a first imaging process that generates a first image file including first image data obtained by imaging a first subject, and a second imaging process that generates a second image file including second image data obtained by imaging a second subject; an acquisition step of acquiring first subject information related to the first subject; and an assignment step of assigning the first subject information to the second image file by including the first subject information in second incidental information recorded in the second image file.

[0008] A second aspect of the technology of the present disclosure is an information processing device that includes a processor, and that links a first imaging process that generates a first image file including first image data obtained by imaging a first subject, and a second imaging process that generates a second image file including second image data obtained by imaging a second subject, thereby obtaining first subject information regarding the first subject, and adding the first subject information to the second image file by including the first subject information in second ancillary information recorded in the second image file.

[0009] 1 is a conceptual diagram showing an example of how an imaging system is used. FIG. 1 is a block diagram showing an example of the hardware configuration of the electrical system of the imaging system. FIG. 2 is a block diagram showing an example of the functions of a processor of a first imaging device and the functions of a processor of a second imaging device. FIG. 3 is a conceptual diagram showing an example of the processing content of a first cooperation unit and a first generation unit. FIG. 4 is a conceptual diagram showing an example of the processing content of a first generation unit and a first acquisition unit. FIG. 5 is a conceptual diagram showing an example of the processing content of a first assignment unit. FIG. 6 is a conceptual diagram showing an example of a hierarchical structure of first subject information recorded in first metadata 60 of a first moving image file. FIG. 7 is a conceptual diagram showing an example of the processing content of a first cooperation unit and a first assignment unit. FIG. 8 is a conceptual diagram showing an example of the processing content of a second cooperation unit and a second generation unit. FIG. 9 is a conceptual diagram showing an example of the processing content of a second generation unit and a second acquisition unit. FIG. 10 is a conceptual diagram showing an example of the processing content of a second assignment unit. FIG. 11 is a conceptual diagram showing an example of the processing content of a second cooperation unit, a second acquisition unit, and a second assignment unit. FIG. 12 is a conceptual diagram showing an example of the processing content of a second cooperation unit and a second assignment unit. 10 is a flowchart showing an example of the flow of a second image file creation process. FIG. 11 is a conceptual diagram showing an example of processing content when identification information is transmitted from a first imaging device of the first and second imaging devices to a second imaging device according to a first modified example. FIG. 12 is a conceptual diagram showing an example of processing content when first time information is transmitted from a first imaging device of the first and second imaging devices to a second imaging device according to a second modified example. FIG. 13 is a conceptual diagram showing an example of processing content when second time information is transmitted from a second imaging device of the first and second imaging devices to the first imaging device according to the second modified example. FIG. 14 is a conceptual diagram showing an example of processing content of a first imaging device of a third modified example. FIG. 15 is a conceptual diagram showing an example of processing content when first imaging device-related information is transmitted from a first imaging device of the first and second imaging devices to a second imaging device according to the third modified example. FIG. 16 is a conceptual diagram showing an example of processing content when second imaging device-related information is transmitted from a second imaging device of the first and second imaging devices to the first imaging device according to the third modified example.10 is a conceptual diagram showing an example of an aspect in which an imaging system according to a fourth modified example is used. FIG. 11 is a conceptual diagram showing an example of processing content when thermal image-related information is transmitted from the first imaging device of the first and second imaging devices to the second imaging device according to the fourth modified example. FIG. 12 is a conceptual diagram showing an example of processing content when visible light-related information is transmitted from the second imaging device of the first and second imaging devices to the first imaging device according to the fourth modified example. FIG. 13 is a conceptual diagram showing an example of an aspect in which a distance image is generated by the first imaging device according to the fourth modified example. FIG. 14 is a conceptual diagram showing an example of processing content when distance image-related information is transmitted from the first imaging device of the first and second imaging devices to the second imaging device according to the fourth modified example. FIG. 15 is a conceptual diagram showing an example of an aspect in which the first imaging device is used as a front camera of an automobile and the second imaging device is used as a rear camera of the automobile. FIG. 16 is a conceptual diagram showing an example of an aspect in which an external device is caused to execute an image file creation process. FIG. 17 is a conceptual diagram showing an example of a structure of a still image file.

[0010] Hereinafter, an example of an embodiment of an information processing method and an information processing device according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0011] As an example, as shown in FIG. 1 , an imaging system 2 includes a first imaging device 10 and a second imaging device 12. The imaging system 2 is a system that performs processing by linking the first imaging device 10 and the second imaging device 12. In the imaging system 2, the first imaging device 10 and the second imaging device 12 capture an image of a common subject, a subject 14. Here, the first imaging device 10 is an example of a "first imaging device" according to the technology of the present disclosure. The second imaging device 12 is an example of a "second imaging device" according to the technology of the present disclosure. The subject 14 is an example of a "first subject," a "second subject," and a "common subject" according to the technology of the present disclosure.

[0012] In the example shown in FIG. 1 , the first imaging device 10 and the second imaging device 12 are consumer digital cameras. Examples of consumer digital cameras include interchangeable lens digital cameras and fixed lens digital cameras. The first imaging device 10 and the second imaging device 12 may also be industrial digital cameras. The first imaging device 10 and the second imaging device 12 may also be imaging devices mounted on various electronic devices such as smart devices, wearable devices, cell observation devices, ophthalmologic observation devices, or surgical microscopes. The first imaging device 10 and the second imaging device 12 may also be imaging devices mounted on various electronic devices such as endoscope devices, ultrasound diagnostic devices, X-ray imaging devices, CT (Computed Tomography) devices, or MRI (Magnetic Resonance Imaging) devices.

[0013] The imaging device may be an imaging device mounted on various modalities such as a device.

[0014] 1, the subjects 14 include human subjects 14A and 14B. The human subjects 14A and 14B face the first imaging device 10 and have their backs to the second imaging device 12. In the example shown in Fig. 1, the first imaging device 10 captures images of the human subjects 14A and 14B from the front, and the second imaging device 12 captures images of the human subjects 14A and 14B from the rear.

[0015] The first imaging device 10 captures an image of the subject 14, thereby generating image data 16 representing an image of the subject 14. The image data 16 is image data obtained by capturing images of the human subjects 14A and 14B from the front side by the first imaging device 10. The image represented by the image data 16 is an image representing the front side of the human subjects 14A and 14B.

[0016] The second imaging device 12 captures an image of the subject 14, thereby generating image data 18 representing an image of the subject 14. The image data 18 is image data obtained by capturing images of the human subjects 14A and 14B from their rear sides by the second imaging device 12. The image represented by the image data 18 is an image representing the rear side of the human subjects 14A and 14B.

[0017] As an example, as shown in FIG. 2, the first imaging device 10 includes a first information processing device 20, a communication I / F (Interface) 21, an image sensor 22, and a UI device 24.

[0018] The first information processing device 20 includes a processor 26, a non-volatile memory (NVM) 28, and a random access memory (RAM) 30. The processor 26, the NVM 28, and the RAM 30 are connected to a bus 34.

[0019] The processor 26 is a processing device including a DSP (Digital Signal Processor), a CPU (Central Processing Unit), and a GPU (Graphics Processing Unit).

[0020] The DSP and GPU operate under the control of the CPU and are responsible for executing image processing. Here, a processing device including a DSP, a CPU, and a GPU is given as an example of the processor 26, but this is merely an example, and the processor 26 may be one or more CPUs and DSPs with integrated GPU functionality, one or more CPUs and DSPs without integrated GPU functionality, or may be equipped with a TPU (Tensor Processing Unit).

[0021] The NVM 28 is a non-volatile storage device that stores various programs, various parameters, etc. The NVM 28 may be, for example, a flash memory (for example, an EEPROM (Electrically Erasable and Programmable Read Only Memory)).

[0022] The RAM 30 is a memory that temporarily stores information and is used as a work memory by the processor 26. The RAM 30 may be, for example, a dynamic random access memory (DRAM) or a static random access memory (SRAM).

[0023] The communication I / F 21 is an interface including a communication processor, an antenna, etc., and is connected to the bus 34. The communication standard applied to the communication I / F 21 is a wireless communication standard including, for example, 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0024] The image sensor 22 is connected to the bus 34. An example of the image sensor 22 is a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0025] The image sensor 22 generates image data 16 by capturing an image of the subject 14 (see FIG. 1 ) under the control of the processor 26. The type of image data 16 is, for example, visible light image data obtained by capturing an image of the subject 14 in the visible light range. However, the type of image data 16 is not limited to this, and may be non-visible light image data obtained by capturing an image in a wavelength range other than the visible light range.

[0026] The image sensor 22 incorporates an A / D converter (not shown), and generates image data 16 by digitizing analog image data obtained by capturing an image of the subject 14. The image data 16 generated by the image sensor 22 is acquired and processed by the processor 26.

[0027] Here, a CMOS image sensor is used as an example of the image sensor 22, but this is merely an example, and the image sensor 22 may be other types of image sensors such as a CCD (Charge Coupled Device) image sensor. Also, here, an example is described in which the image sensor 22 captures an image of the subject 14 in the visible light range.

[0028] However, this is merely an example, and the subject 14 may be captured in a wavelength range other than the visible light range.

[0029] The UI (User Interface) device 24 has a reception function for receiving instructions from a user and a presentation function for presenting information to the user. The reception function is realized by, for example, a touch panel and hard keys (e.g., a release button and menu selection keys), etc. The presentation function is realized by, for example, a display and a speaker, etc.

[0030] The second imaging device 12 includes a second information processing device 36 corresponding to the first information processing device 20, a communication I / F 38 corresponding to the communication I / F 21, an image sensor 40 corresponding to the image sensor 22, and a UI device 42 corresponding to the UI device 24. The second information processing device 36 includes a processor 44 corresponding to the processor 26, an NVM 46 corresponding to the NVM 28, and a RAM 48 corresponding to the RAM 30. As such, the second imaging device 12 includes the same multiple hardware resources as the first imaging device 10. Therefore, a description of the multiple hardware resources included in the second imaging device 12 will be omitted here. Note that the first information processing device 20 and the second information processing device 36 are examples of "information processing devices" according to the technology of the present disclosure. The processors 26 and 44 are examples of "processors" according to the technology of the present disclosure.

[0031] The first imaging device 10 and the second imaging device 12 generate video files containing video data by capturing images in a video capturing mode, which is an operating mode in which images are captured at a predetermined frame rate (e.g., several tens of frames per second). Information obtained by the first imaging device 10 is recorded as metadata in the video file generated by the first imaging device 10, and information obtained by the second imaging device 12 is recorded as metadata in the video file generated by the second imaging device 12. In other words, there is no correlation between the information contained in the metadata of the video file generated by the first imaging device 10 and the information contained in the metadata of the video file generated by the second imaging device 12. Therefore, for example, if a user performing processing on one video file wants to refer to information contained in the other video file, it is time-consuming to play the other video file or find the necessary information from the metadata in the video file.

[0032] In view of these circumstances, in the imaging system 2, as shown in Fig. 3 as an example, the first image file creation process is performed by the processor 26 of the first imaging device 10, and the second image file creation process is performed by the processor 44 of the second imaging device 12. Furthermore, communication is performed between the first imaging device 10 and the second imaging device 12 via the communication I / Fs 21 and 38, so that the first image file creation process and the second image file creation process are performed in cooperation with each other.

[0033] In the first imaging device 10, a first image file creation program 52 is stored in the NVM 28. The processor 26 performs a first image file creation process by reading the first image file creation program 52 from the NVM 28 and executing the read first image file creation program 52 on the RAM 30. The first image file creation process is realized by the processor 26 operating as a first linking unit 26A, a first generating unit 26B, a first acquiring unit 26C, a first assigning unit 26D, and a first control unit 26E in accordance with the first image file creation program 52 executed on the RAM 30.

[0034] In the second imaging device 12, a second image file creation program 54 is stored in the NVM 48. The processor 44 performs a second image file creation process by reading the second image file creation program 54 from the NVM 46 and executing the read second image file creation program 54 on the RAM 48. The second image file creation process is realized by the processor 44 operating as a second linking unit 44A, a second generating unit 44B, a second acquiring unit 44C, a second assigning unit 44D, and a second control unit 44E in accordance with the second image file creation program 54 executed on the RAM 48.

[0035] In this embodiment, the first image file creation process is an example of a "first imaging process" according to the technology of the present disclosure. The second image file creation process is an example of a "second imaging process" according to the technology of the present disclosure. The process performed by the first linking unit 26A and the process performed by the second linking unit 44A are an example of a "linking process" according to the technology of the present disclosure. The process performed by the first acquisition unit 26C and the process performed by the second acquisition unit 44C are an example of an "acquisition process" according to the technology of the present disclosure. The process performed by the first assigning unit 26D and the process performed by the second assigning unit 44D are an example of an "assigning process" according to the technology of the present disclosure.

[0036] 4, the first linkage unit 26A of the first imaging device 10 establishes communication with the second linkage unit 44A of the second imaging device 12 via the communication I / Fs 21 and 38 (see FIGS. 2 and 3). By communicating with the second linkage unit 44A, the first linkage unit 26A links the first image file creation process (see FIG. 3) performed by the first imaging device 10 with the second image file creation process (see FIG. 3) performed by the second imaging device 12.

[0037] The first generation unit 26B acquires multiple frames of image data 16 from the image sensor 22, and generates a first moving image file 56 based on the acquired multiple frames of image data 16. The first moving image file 56 is a moving image file that includes first moving image data 58 and first metadata 60. The first moving image data 58 is moving image data that includes multiple frames of image data 16.

[0038] In this embodiment, the image data 16 is an example of a "first frame" according to the technology of the present disclosure. The image data 16 of multiple frames is an example of "multiple first frames" according to the technology of the present disclosure. The first moving image data 58 is an example of "moving image data composed of multiple first frames," "first image data," and "first moving image data" according to the technology of the present disclosure. The first moving image file 56 is an example of a "first image file" and "first moving image file" according to the technology of the present disclosure.

[0039] The first metadata 60 is data related to the first moving image file 56 (i.e., data incidental to the first moving image data 58) and is recorded in the first moving image file 56. The first metadata 60 is an example of “first incidental information” according to the technology of the present disclosure.

[0040] The first metadata 60 includes overall related data 60A and a plurality of frame related data 60B. The overall related data 60A is data relating to the entire first video file 56. The overall related data 60A includes, for example, an identifier uniquely assigned to the first video file 56, the time the first video file 56 was created, the time required to play the first video file 56, the bit rate and codec of the first video data 58, etc.

[0041] The plurality of frame-related data 60B correspond one-to-one to the plurality of frames of image data 16 included in the first moving image data 58. The frame-related data 60B includes data related to the corresponding image data 16. The frame-related data 60B includes, for example, a frame identifier 60B1, a date and time 60B2, imaging conditions 60B3, first subject information 62, and second subject information 74. The frame identifier 60B1 is an identifier that can identify a frame. The date and time 60B2 is the date and time when the frame (i.e., image data 16) corresponding to the frame-related data 60B was obtained. The imaging conditions 60B3 are imaging conditions set for the first imaging device 10 (e.g., aperture, shutter speed, sensitivity of the image sensor 22, 35 mm equivalent focal length, and on / off of image stabilization, etc.). The first subject information 62 is information relating to the subject included in each frame constituting the first moving image data 58, and the second subject information 74 is information transmitted from the second linkage unit 44A and received by the first imaging device 10 via the first linkage unit 26A. Details of the first subject information 62 and the second subject information 74 will be described later.

[0042] 5 as an example, the first acquisition unit 26C acquires image data 16 frame by frame in chronological order from the moving image data 58 of the first moving image file 56. The first acquisition unit 26C then performs AI (Artificial Intelligence) image recognition processing on the acquired image data 16 to acquire first subject information 62, which is information about the subject 14. The first subject information 62 is various information obtained by performing AI image recognition processing on the image data 16. While AI image recognition processing is illustrated here, this is merely an example, and other types of image recognition processing, such as template matching image recognition processing, may be performed instead of or in addition to the AI ​​image recognition processing.

[0043] 5, first subject information 62A related to human subject 14A and first subject information 62B related to human subject 14B are exemplified as first subject information 62. First subject information 62A includes coordinate information, subject type information, and subject attribute information. Note that when AI-based image recognition processing is performed using a convolutional neural network (CNN), first subject information 62A may include, as information related to human subject 14B (e.g., subject attribute information), a gradient-weighted class activation mapping (CAM) image, a feature map obtained from a convolutional layer, or a confidence level (i.e., score) output from the CNN.

[0044] The coordinate information included in the first subject information 62A is information relating to coordinates that can identify the position within the image of the human subject 14A appearing in the image represented by the image data 16 (for example, the position within a two-dimensional coordinate plane with the upper left corner of the image represented by the image data 16 as the origin). Examples of coordinates included in the first subject information 62A include the coordinates of the upper left corner 64A in a front view of a bounding box 64 obtained from AI-based image recognition processing of the human subject 14A, and the coordinates of the lower right corner 64B of the bounding box 64 in a front view.

[0045] The subject type information included in the first subject information 62A is information indicating the type of human subject 14A within the bounding box 64. The first subject information 62A includes, as subject type information, a creature category (in the example shown in FIG. 5 , human), a gender category (in the example shown in FIG. 5 , "male"), and a name category (in the example shown in FIG. 5 , the name "Fuji Taro"), etc. Furthermore, the first subject information 62A includes, as subject attribute information, a direction category (in the example shown in FIG. 5 , "front"), etc. Note that, although an example in which the gender category and the name category belong to the subject type information is given here, this is merely an example, and the gender category and the name category may also belong to the subject attribute information.

[0046] The second subject information 62B is configured in the same manner as the first subject information 62B. In the example shown in Fig. 5, examples of coordinates included in the second subject information 62B include the coordinates of the upper left corner 66A in a front view and the coordinates of the lower right corner 66B in a front view of a bounding box 66 obtained by AI image recognition processing for the person 14B. In addition, the name "Fujiichiro" is assigned to the name category of the second subject information 62B.

[0047] 6 , the first assigning unit 26D assigns the first subject information 62 to the first moving image file 56 by including the first subject information 62 in the first metadata 60. For example, the first assigning unit 26D assigns the first subject information 62 to the first moving image file 56 by including the first subject information 62 corresponding to the image data 16 in the frame-related data 60B corresponding to the image data 16. The first subject information 62 is assigned to the first moving image file 56 for each piece of image data 16 included in the first moving image data 58.

[0048] As an example, as shown in FIG. 7 , the plurality of pieces of information included in the first subject information 62 assigned to the first moving image file 56 are classified into a plurality of categories. For example, a subject identifier ("#1" and "#2" in the example shown in FIG. 7 ), which is a unique identifier, is assigned to each subject included in the subject 14. A plurality of categories, such as a type category, an attribute category, and a position category, are then assigned to the subject identifier. In the example shown in FIG. 7 , a plurality of categories are hierarchically arranged for each type category and attribute category. Categories that are subordinate concepts or derivative concepts of the higher level are provided at lower levels. In the example shown in FIG. 7 , first subject information 62A is assigned to "#1," and first subject information 62B is assigned to "#2."

[0049] The type category is a category that indicates the type of subject. The subject type information included in the first subject information 62A is classified into the type category. In the example shown in FIG. 7, "human" is assigned to the type category as the type of subject. A gender category and a name category are provided at a lower level than the type category. The gender category is a category that indicates gender, and the name category is a category that indicates the name of the subject (for example, a common noun or a proper noun, etc.).

[0050] The attribute category is a category that indicates the attribute of the subject. The subject attribute information included in the first subject information 62A is classified into the attribute categories. In the example shown in FIG. 7, a direction category, an expression category, and a clothing category are provided, and a color category is provided as a lower level of the clothing category. The direction category is a category that indicates the direction of the subject. The expression category is a category that indicates the expression of the subject. The clothing category is a category that indicates the type of clothing the subject is wearing. The color category is a category that indicates the color of the clothing the subject is wearing.

[0051] The position category indicates the position of the subject within the image. Coordinates included in the first subject information 62A are classified into the position category. In the example shown in Fig. 7, coordinates included in the first subject information 62A are assigned to "#1", and coordinates included in the first subject information 62B are assigned to "#2".

[0052] 8 , every time the first assigning unit 26D assigns first subject information 62 to the first moving image file 56 for each image data 16, the first assigning unit 26D transmits the same first subject information 62 to the second associate unit 44A of the second imaging device 12 via the first associate unit 26A. That is, every time the first subject information 62 for each image data 16 is included in the frame-related data 60B by the first assigning unit 26D, the same first subject information 62 as the first subject information 62 included in the frame-related data 60B is transmitted from the first imaging device 10 to the second imaging device 12. Note that the transmission of the first subject information 62 to the second associate unit 44A may be performed after the recording of the first moving image file 56 is completed.

[0053] 9, the second linkage unit 44A of the second imaging device 12 establishes communication with the first linkage unit 26A of the first imaging device 10 via the communication I / Fs 21 and 38 (see FIGS. 2 and 3). By communicating with the first linkage unit 26A, the second linkage unit 44A links the second image file creation process (see FIG. 3) performed by the second imaging device 12 with the first image file creation process (see FIG. 3) performed by the first imaging device 10.

[0054] The second generation unit 44B acquires a plurality of frames of image data 18 from the image sensor 40, and generates a second moving image file 68 based on the acquired plurality of frames of image data 18.

[0055] The second video file 68 is a video file that includes second video data 70 and second metadata 72. The second video data 70 is video data that includes image data 18 of a plurality of frames.

[0056] In this embodiment, the image data 18 is an example of a "second frame" according to the technology of the present disclosure. The image data 18 of multiple frames is an example of a "multiple second frames" according to the technology of the present disclosure. The second moving image data 70 is an example of "moving image data composed of multiple second frames," "second image data," and "second moving image data" according to the technology of the present disclosure. The second moving image file 68 is an example of a "second image file" and "second moving image file" according to the technology of the present disclosure.

[0057] The second metadata 72 is data related to the second moving image file 68 (i.e., data incidental to the second moving image data 70) and is recorded in the second moving image file 68. The second metadata 72 is an example of “second incidental information” according to the technology of the present disclosure.

[0058] The second metadata 72 includes overall related data 72A and multiple frame related data 72B. The overall related data 72A is data relating to the entire second video file 68. The overall related data 72A includes, for example, an identifier uniquely assigned to the second video file 68, the time the second video file 68 was created, the time required to play the second video file 68, the bit rate of the second video data 70, the codec, etc.

[0059] The plurality of frame-related data 72B correspond one-to-one to the plurality of frames of image data 18 included in the second moving image data 70. The frame-related data 72B includes data related to the corresponding image data 18. Like the first frame-related data 60B, the frame-related data 72B includes, for example, a frame identifier 72B1, a date and time 72B2, and image capturing conditions 72B3. The frame-related data 72B also includes first subject information 62 and second subject information 74, as will be described later.

[0060] 10 , the second acquisition unit 44C acquires image data 18 frame by frame in chronological order from moving image data 70 in the second moving image file 68. The second acquisition unit 44C then performs AI-based image recognition processing on the acquired image data 18 to acquire second subject information 74, which is information about the subject 14. The second subject information 74 is various information obtained by performing AI-based image recognition processing on the image data 18. While AI-based image recognition processing is illustrated here, this is merely an example, and other types of image recognition processing, such as template matching-based image recognition processing, may be performed instead of or in addition to the AI-based image recognition processing.

[0061] 10 , second subject information 74A related to human subject 14A and second subject information 74B related to human subject 14B are exemplified as second subject information 74. Similar to first subject information 62, second subject information 74A includes coordinate information, subject type information, subject attribute information, etc.

[0062] The subject type information included in the second subject information 74A is information that indicates the type of human subject 14A within the bounding box 76. The second subject information 74A includes, as subject type information, a living thing category (human in the example shown in FIG. 10 ), etc. The second subject information 74A also includes, as subject attribute information, an orientation category ("rear face" in the example shown in FIG. 10 ).

[0063] The coordinate information included in the second subject information 74A is information relating to coordinates that can identify the position within the image of human subject 14A appearing in the image represented by image data 18 (for example, the position within a two-dimensional coordinate plane with the upper left corner of the image represented by image data 18 as the origin). Examples of coordinates included in the second subject information 74A include the coordinates of an upper left corner 76A in a front view of a bounding box 76 obtained from AI-based image recognition processing of human subject 14A, and the coordinates of a lower right corner 76B of the bounding box 76 in a front view.

[0064] The subject type information included in the second subject information 74B is information that indicates the type of human subject 14B within the bounding box 78. The second subject information 74B includes, as subject type information, a living thing category (human in the example shown in FIG. 10 ), etc. The second subject information 74B also includes, as subject attribute information, an orientation category ("rear face" in the example shown in FIG. 10 ).

[0065] The coordinate information included in the second subject information 74B is information relating to coordinates that can identify the position within the image of human subject 14B appearing in the image represented by image data 18 (for example, the position within a two-dimensional coordinate plane with the upper left corner of the image represented by image data 18 as the origin). Examples of coordinates included in the second subject information 74B include the coordinates of the upper left corner 78A in a front view of a bounding box 78 obtained from AI-based image recognition processing of human subject 14B, and the coordinates of the lower right corner 78B of the bounding box 78 in a front view.

[0066] 11 , the second assigning unit 44D assigns the second subject information 74 to the second moving image file 68 by including the second subject information 74 in the second metadata 72. For example, the second assigning unit 44D assigns the second subject information 74 to the second moving image file 68 by including the second subject information 74 corresponding to image data 18 in the frame-related data 72B corresponding to the image data 18. The second subject information 74 is assigned to the second moving image file 68 for each piece of image data 18 included in the second moving image data 70. Note that the multiple pieces of information included in the second subject information 74 assigned to the second moving image file 68 are classified into multiple categories in the same manner as the example shown in FIG.

[0067] 12, when the first subject information 62 is transmitted from the first linkage unit 26A of the first imaging device 10 to the second imaging device 12 (see FIG. 8), the first subject information 62 is received by the second linkage unit 44A of the second imaging device 12. The first subject information 62 received by the second linkage unit 44A is acquired by the second acquisition unit 44C.

[0068] The second assigning unit 44D assigns the first subject information 62 to the second moving image file 68 by including the first subject information 62 acquired by the second acquiring unit 44C in the second metadata 72. For example, the second assigning unit 44D assigns the first subject information 62 to the second moving image file 68 by including the first subject information 62 acquired by the second acquiring unit 44C in the frame-related data 72B corresponding to the latest image data 18. As a result, the first frame-related data 72B includes the first subject information 62 in addition to the second subject information 74, and therefore the user or the like can obtain the first subject information 62, which is information also included in the first moving image file 56, from the second moving image file 68.

[0069] 13 , every time the second assigning unit 44D assigns second subject information 74 to the second moving image file 68 for each image data 18, the second assigning unit 44D transmits the same second subject information 74 to the first linking unit 26A of the first imaging device 10 via the second linking unit 44A. That is, every time the second assigning unit 44D includes the second subject information 74 in the frame-related data 72B for each image data 18, the same second subject information 74 as the second subject information 74 included in the frame-related data 72B is transmitted from the second imaging device 12 to the first imaging device 10. Note that the transmission of the second subject information 74 to the first linking unit 26A may be performed after the recording of the second moving image file 68 is completed.

[0070] 14 , when second subject information 74 is transmitted from the second linkage unit 44A of the second imaging device 12 to the first imaging device 10 (see FIG. 13 ), the second subject information 74 is received by the first linkage unit 26A of the first imaging device 10. The second subject information 74 received by the first linkage unit 26A is acquired by the first acquisition unit 26C.

[0071] The first assigning unit 26D assigns the second subject information 74 acquired by the first acquiring unit 26C to the first moving image file 56 by including the second subject information 74 in the first metadata 60. For example, the first assigning unit 26D assigns the second subject information 74 to the first moving image file 56 by including the second subject information 74 acquired by the first acquiring unit 26C in the frame-related data 60B corresponding to the latest image data 16. As a result, the first frame-related data 60B includes the second subject information 74 in addition to the first subject information 62, and therefore the user or the like can obtain the second subject information 74, which is information also included in the second moving image file 68, from the first moving image file 56.

[0072] 15 , in the first imaging device 10, the first control unit 26E stores the first moving image file 56 obtained as described above in the NVM 28. In the second imaging device 12, the second control unit 44E stores the second moving image file 68 obtained as described above in the NVM 28.

[0073] 15 shows an example in which the first moving image file 56 is stored in the NVM 28 and the second moving image file 68 is stored in the NVM 46, but this is merely one example, and the first moving image files 56 and 68 may be stored in one or more storage media other than the NVMs 28 and 46. The storage medium may be any medium that is connected to the first imaging device 10 and the second imaging device 12 directly or indirectly via a wired or wireless method, etc. Examples of storage media include a DVD (Digital Versatile Disc), a USB (Universal Serial Bus) memory, an SSD (Solid State Drive), an HDD (Hard Disk Drive), and a magnetic tape drive.

[0074] Next, the operation of the imaging system 2 will be described with reference to Fig. 16 and Fig. 17. Note that the processing flow shown in the flowcharts of Fig. 16 and Fig. 17 is an example of an "imaging processing method" according to the technique of the present disclosure.

[0075] First, with reference to Figure 16, we will explain an example of the flow of the first image file creation process performed by the processor 26 when an instruction to start executing the first image file creation process in video image capture mode is received by the UI device 24 of the first imaging device 10.

[0076] 16 , first, in step ST10, the first linkage unit 26A establishes communication with the second linkage unit 44A of the second imaging device 12 via the communication I / Fs 21 and 38, thereby linking the first image file creation process with the second image file creation process (see FIGS. 4 and 9 ). After the process of step ST10 is executed, the first image file creation process proceeds to step ST12.

[0077] In step ST12, the first generation unit 26B determines whether or not one frame of image has been captured by the image sensor 22. If one frame of image has not been captured by the image sensor 22 in step ST12, the determination is negative, and the first image file creation process proceeds to step ST24. If one frame of image has been captured by the image sensor 22 in step ST12, the determination is positive, and the first image file creation process proceeds to step ST14.

[0078] In step ST14, the first generation unit 26B acquires the image data 16 from the image sensor 22 (see FIG. 4). After the process of step ST14 is executed, the first image file creation process proceeds to step ST16.

[0079] In step ST16, the first generation unit 26B generates a first moving image file 56 including the image data 16 acquired in step ST14 (see FIG. 4). If the image data 16 acquired in step ST14 is image data 16 for the second frame or later, the first generation unit 26B updates the contents of the first moving image file 56 by including the image data 16 acquired in step ST14 in the first moving image file 56 as image data 16 for one frame. After the processing of step ST16 is executed, the first image file creation processing proceeds to step ST18.

[0080] In step ST18, the first acquisition unit 26C acquires first subject information 62 by performing AI image recognition processing on the image data 16 acquired in step ST14 (see FIG. 5 ). After the processing of step ST18 is executed, the first image file creation processing proceeds to step ST20.

[0081] In step ST20, the first assigning unit 26D assigns the first subject information 62 acquired in step ST18 to the first moving image file 56 by including the first subject information 62 in the first metadata 60 of the first moving image file 56 generated in step ST16 (see FIG. 6 ). After the process of step ST20 is executed, the first image file creation process proceeds to step ST22.

[0082] In step ST22, the first assigning unit 26D transmits the first subject information 62, which is the same as the first subject information 62 assigned to the first moving image file 56 in step ST20, to the second assigning unit 44A of the second imaging device 12 via the first assigning unit 26A (see FIG. 8 ). After the processing of step ST22 is executed, the first image file creation processing proceeds to step ST24.

[0083] In step ST24, the first assigning unit 26D determines whether the second subject information 74 transmitted from the second linkage unit 44A of the second imaging device 12 (see step ST62 in FIGS. 13 , 14 , and 17 ) has been acquired by the first acquiring unit 26C via the first linkage unit 26A. If the second subject information 74 transmitted from the second linkage unit 44A of the second imaging device 12 has not been acquired by the first acquiring unit 26C via the first linkage unit 26A in step ST24, the determination is negative, and the first image file creation process proceeds to step ST30. If the second subject information 74 transmitted from the second linkage unit 44A of the second imaging device 12 has been acquired by the first acquiring unit 26C via the first linkage unit 26A in step ST24, the determination is positive, and the first image file creation process proceeds to step ST26.

[0084] In step ST26, the first assigning unit 26D determines whether the first moving image file 56 has already been generated by the first generating unit 26B in step ST16. If the first moving image file 56 has not been generated by the first generating unit 26B in step ST26, the determination is negative, and the first image file creation process proceeds to step ST32. If the first moving image file 56 has already been generated by the first generating unit 26B in step ST26, the determination is positive, and the first image file creation process proceeds to step ST28.

[0085] In step ST28, the first assigning unit 26D assigns the second subject information 74 acquired in step ST24 to the first moving image file 56 by including the second subject information 74 in the first metadata 60 (see FIG. 14 ). After the processing of step ST28 is executed, the first image file creation processing proceeds to step ST32.

[0086] In step ST30, the first assigning unit 26D determines whether a predetermined time (e.g., several seconds) has elapsed since the execution of the process of step ST24 was started. If the predetermined time has not elapsed since the execution of the process of step ST24 was started in step ST30, the determination is negative, and the first image file creation process proceeds to step ST24. If the predetermined time has elapsed since the execution of the process of step ST24 was started in step ST30, the determination is positive, and the first image file creation process proceeds to step ST32.

[0087] In step ST32, the first control unit 26E determines whether a condition for terminating the first image file creation process (hereinafter referred to as the "first image file creation process termination condition") has been satisfied. A first example of the first image file creation process termination condition is that an instruction to terminate the first image file creation process has been received by the UI device 24. A second example of the first image file creation process termination condition is that the amount of data of the first moving image data 58 has reached an upper limit. If the first image file creation process termination condition has not been satisfied in step ST32, the determination is negative, and the first image file creation process proceeds to step ST12. If the first image file creation process termination condition has been satisfied in step ST32, the determination is positive, and the first image file creation process proceeds to step ST34.

[0088] In step ST34, the first control unit 26E stores the first moving image file 56 obtained by executing the processes of steps ST10 to ST32 in the NVM 28 (see FIG. 15). After the process of step ST34 is executed, the first image file creation process ends.

[0089] Figure 17 shows an example of the flow of the second image file creation process performed by the processor 44 when an instruction to start executing the second image file creation process under video image capture mode is received by the UI device 42 of the second imaging device 12.

[0090] Here, differences between the second image file creation process and the first image file creation process described in FIG. 16 will be described. First, the first image file creation process in FIG. 16 is a process mainly performed by the first imaging device 10, whereas the second image file creation process in FIG. 17 is a process mainly performed by the second imaging device 12. Thus, while the first moving image file 56 is created by the first imaging device 10 in FIG. 16, the second moving image file 68 is created by the second imaging device 12 in FIG. 17. Also, in FIG. 16, the first imaging device 10 acquires first subject information 62 from image data 16 acquired by the first imaging device 10 and acquires second subject information 74 by cooperating with the second imaging device 12. Meanwhile, in FIG. 17, the second imaging device 12 acquires second subject information 74 from image data 18 acquired by the second imaging device 12 and acquires the first subject information 62 by cooperating with the first imaging device 10.

[0091] Other than the above differences, the description of each step (ST50, ST52, ST54, etc.) in FIG. 17 is substantially the same as the description of each step (ST10, ST12, ST14, etc.) in FIG.

[0092] As described above, in the imaging system 2, by establishing communication between the first imaging device 10 and the second imaging device 12, the first image file creation process performed by the first imaging device 10 and the second image file creation process performed by the second imaging device 12 are linked. In the first imaging device 10, first subject information 62 is acquired as information about the subject 14 (see FIG. 5), and the first subject information 62 is included in the first metadata 60, thereby forming the first subject information 62.

[0093] The information 62 is added to the first moving image file 56 (see FIG. 6).

[0094] Meanwhile, the second imaging device 12 also acquires second subject information 74 as information relating to the subject 14 (see Figure 10), and the second subject information 74 is included in the second metadata 72, thereby assigning the second subject information 74 to the second moving image file 68 (see Figure 11).

[0095] Here, the second imaging device 12 acquires from the first imaging device 10 the first subject information 62, which is the same as the first subject information 62 assigned to the first moving image file 56 (see FIG. 12 ). Then, the second imaging device 12 adds the first subject information 62 to the second moving image file 68 by including the first subject information 62 in the second metadata 72 (see FIG. 12 ). Therefore, for example, a user or device performing processing on the second moving image file 68 can obtain, from the second moving image file 68, the first subject information 62 that is the same as the first subject information 62 included in the first moving image file 56. That is, a user or device performing processing on the second moving image file 68 can understand what information the first subject information 62 included in the first moving image file 56 is (e.g., characteristics of the subject 14 when captured from the first imaging device 10) without having to play back the first moving image file 56. As a result, the usability of the second moving image file 68 is improved.

[0096] Meanwhile, the first imaging device 10 acquires, from the second imaging device 12, second subject information 74 identical to the second subject information 74 assigned to the second video file 68 (see FIG. 14 ). Then, in the first imaging device 10, the second subject information 74 is included in the first metadata 60, thereby assigning the second subject information 74 to the first video file 56 (see FIG. 14 ). Therefore, for example, a user or device performing processing on the first video file 56 can obtain, from the first video file 56, second subject information 74 identical to the second subject information 74 included in the second video file 68. That is, a user or device performing processing on the first video file 56 can understand what information the second subject information 74 included in the second video file 68 is (e.g., characteristics of the subject 14 when captured from the second imaging device 12) without having to play back the second video file 68. As a result, the usability of the first video file 56 is improved.

[0097] Furthermore, in the imaging system 2, the first imaging device 10 and the second imaging device 12 capture images of a common subject, that is, a subject 14. The first metadata 60 of the first video file 56 and the second metadata 72 of the second video file 68 include first subject information 62 and second subject information 74 regarding the subject 14. Therefore, a user or device performing processing on the first video file 56 can obtain the first subject information 62 and the second subject information 74 regarding the common subject, that is, the subject 14, from the first video file 56. As a result, the usability of the first video file 56 is improved. Furthermore, a user or device performing processing on the second video file 68 can obtain the first subject information 62 and the second subject information 74 regarding the common subject, that is, the subject 14, from the second video file 68. As a result, the usability of the second video file 68 is improved.

[0098] In the above embodiment, an example was described in which the first image file creation process was performed by the first imaging device 10 and the second image file creation process was performed by the second imaging device 12. However, the technology of the present disclosure is not limited to this. For example, the first image file creation process and the second image file creation process may be performed by the first imaging device 10 or the second imaging device 12 at different time periods. In this case, the first subject information 62 and the second subject information 74 can be shared between the first moving image file 56 and the second moving image file 68 obtained at different time periods by the first imaging device 10 or the second imaging device 12, thereby improving usability. Furthermore, the first subject information 62 and the second subject information 74 may be shared after the first moving image file 56 and the second moving image file 68 have been recorded.

[0099] In the above embodiment, an example has been described in which the first subject information 62 and the second subject information 74 are generated for each frame and assigned to the first moving image file 56 and the second moving image file 68, but the technology of the present disclosure is not limited to this. For example, the first subject information 62 and the second subject information 74 may be generated and assigned to the first moving image file 56 and the second moving image file 68 when certain conditions are met. Examples of the certain condition include a condition that a specific instruction is received by the UI device 42, a condition that imaging is performed within a specified imaging period, a condition that specific imaging conditions are set, a condition that the first imaging device 10 or the second imaging device 12 reaches a specific position, a condition that the distance between the first imaging device 10 and the second imaging device 12 falls within a specific range, a condition that the orientation of the first imaging device 10 or the second imaging device 12 assumes a specific orientation, a condition that the first imaging device 10 or the second imaging device 12 reaches a specific position, a condition that a certain amount of time has passed, a condition that a certain number of frames have been captured, a condition that imaging has been performed under specified imaging conditions, or a condition that imaging has been performed under specified environments. Furthermore, for example, the first subject information 62 and the second subject information 74 may be generated every two or more predetermined number of frames (e.g., several frames to several tens of frames) and assigned to the first moving image file 56 and the second moving image file 68. The above description also applies to each of the modified examples described below.

[0100] Hereinafter, for the sake of convenience, when there is no need to distinguish between the first subject information 62 and the second subject information 74, they will be referred to as "subject information" without a reference numeral. Also, hereinafter, for the sake of convenience, when there is no need to distinguish between the first moving image file 56 and the second moving image file 68, they will be referred to as "moving image files." Also, hereinafter, for the sake of convenience, when there is no need to distinguish between the first metadata 60 and the second metadata 72, they will be referred to as "metadata" without a reference numeral. Also, hereinafter, for the sake of convenience, when there is no need to distinguish between the first imaging device 10 and the second imaging device 12, they will be referred to as "imaging devices" without a reference numeral. Also, hereinafter, for the sake of convenience, when there is no need to distinguish between the first information processing device 20 and the second information processing device 36, they will be referred to as "information processing devices" without a reference numeral. In the following, for the sake of convenience, when there is no need to distinguish between the processor 26 and the processor 44, they will be referred to as "processors" without a reference numeral. In the following, for the sake of convenience, when there is no need to distinguish between the NVM 28 and the NVM 46, they will be referred to as "NVMs" without a reference numeral. In the following, for the sake of convenience, when there is no need to distinguish between the RAM 30 and the RAM 48, they will be referred to as "RAMs" without a reference numeral. In the following, for the sake of convenience, when there is no need to distinguish between the first image file creation program 52 and the second image file creation program 54, they will be referred to as "image file creation programs" without a reference numeral. In the following, for the sake of convenience, when there is no need to distinguish between the first image file creation process and the second image file creation process, they will be referred to as "image file creation processes" without a reference numeral.

[0101] [First Modification] In the above embodiment, an example was described in which the metadata of a video file includes first subject information 62 and second subject information 74. However, the technology of the present disclosure is not limited to this. For example, the first metadata 60 and second metadata 72 recorded in a video file may include information that is common to both.

[0102] 18 , in the first imaging device 10, the first acquisition unit 26C generates identification information 80 on the condition that the first generation unit 26B has generated the first moving image file 56. The identification information 80 is information (e.g., a code) common to the first metadata 60 and the second metadata 72. The first assignment unit 26D assigns the identification information 80 to the first moving image file 56 by including the identification information 80 generated by the first acquisition unit 26C in the first metadata 60 in the same manner as the first subject information 62. The first acquisition unit 26C also transmits the identification information 80 to the second imaging device 12 via the first linkage unit 26A in the same manner as the first subject information 62.

[0103] In the second imaging device 12, the second acquisition unit 44C acquires the identification information 80 transmitted from the first imaging device 10 via the second linkage unit 44A. The second assignment unit 44D assigns the identification information 80 to the second moving image file 68 by including the identification information 80 acquired by the second acquisition unit 44C in the second metadata 72 in the same manner as the first subject information 62.

[0104] As a result, since the identification information 80, which is information common to the first metadata 60 and the second metadata 72, is included in both the first metadata 60 and the second metadata 72, a user or device performing processing on a video file can identify which of multiple video files is related.

[0105] Furthermore, the identification information 80 may be generated when a specific condition is satisfied. Examples of the specific conditions that may be satisfied include when a specific instruction is received by the UI device 42, when imaging is performed within a specified imaging period, when specific imaging conditions are set for the imaging device, when the imaging device reaches a specific position, when the distance between the first imaging device 10 and the second imaging device 12 falls within a specific range, when the orientation of the imaging device assumes a specific orientation, when the imaging device reaches a specific position, when a certain amount of time has passed, when imaging is performed for a certain number of frames, when imaging is performed under specified imaging conditions, or when imaging is performed under a specified environment.

[0106] For example, identification information 80 generated when a specific condition is satisfied is included in frame-related data 60B corresponding to image data 16 obtained at the timing corresponding to the timing at which identification information 80 is generated. Frame-related data 72B of second video file 68 also includes identification information 80 in the same manner as first subject information 62. This allows a user or device performing processing on video files to identify highly related information between frames of first video file 56 and second video file 68 (for example, information obtained when a specific condition is satisfied).

[0107] 18 has been described with reference to an example in which the identification information 80 is generated by the first imaging device 10 and provided to the second imaging device 12, but this is merely one example, and the identification information 80 may be generated by the second imaging device 12 and provided to the first imaging device 10. Furthermore, the identification information 80 may be assigned to the first imaging device 10 and the second imaging device 12 from an external source (for example, a user or a device).

[0108] [Second Variant] In the above first variant, an example was given in which the first metadata 60 and second metadata 72 of the video file have identification information 80, but the technology of the present disclosure is not limited to this. For example, the first metadata 60 and second metadata 72 of the video file may have time information related to the frame.

[0109] 19 , in the first imaging device 10, the first acquisition unit 26C acquires first time information 82 on a frame-by-frame basis (i.e., each time one frame's worth of image is captured). The first time information 82 is, for example, information indicating the time (e.g., the time corresponding to the image capture time) at which the first generation unit 26B acquired the image data 16 obtained by capturing one frame's worth of image data by the image sensor 22. While a time is illustrated here, this is merely an example, and the first time information 82 may be an identifier capable of identifying frames acquired in chronological order since the start of imaging, or the elapsed time since the start of imaging.

[0110] The first assigning unit 26D assigns the first time information 82 to the first moving image file 56 by including the first time information 82 generated by the first acquiring unit 26C in the corresponding frame-related data 60B in the same manner as the first subject information 62. As a result, the frame-related data 60B of the first moving image file 56 includes the first time information 82. Therefore, a user or a device performing processing on the first moving image file 56 can identify, from the first moving image file 56, information corresponding to image data 16 obtained at a specific timing.

[0111] On the other hand, the first acquisition unit 26C transmits the first time information 82 to the second imaging device 12 via the first link unit 26A in the same manner as the first subject information 62.

[0112] In the second imaging device 12, the second acquisition unit 44C acquires the first time information 82 transmitted from the first imaging device 10 via the second linkage unit 44A. The second assignment unit 44D assigns the first time information 82 to the second moving image file 68 by including the first time information 82 acquired by the second acquisition unit 44C in the frame-related data 72B in the same manner as the first subject information 62. As a result, the frame-related data 72B of the second moving image file 68 includes the first time information 82. Therefore, a user or device performing processing on the second moving image file 68 can identify information corresponding to image data 16 obtained at a specific timing from the second moving image file 68.

[0113] As an example, as shown in FIG. 20 , in the second imaging device 12, the second acquisition unit 44C acquires second time information 84 on a frame-by-frame basis (i.e., each time one frame is captured). The second time information 82 is, for example, information indicating the time (e.g., the time corresponding to the image capture time) at which the second generation unit 44B acquired the image data 18 obtained by capturing one frame of image data by the image sensor 40. This makes it easy to identify the image data 16 (frame) in the first moving image file 56 and the image data 18 (frame) in the second moving image file 68, which were captured at the same time. While time is used as an example here, this is merely an example, and the second time information 82 may be an identifier capable of chronologically identifying frames captured since the start of imaging, or the elapsed time since the start of imaging.

[0114] The second assigning unit 44D assigns the second time information 84 to the second moving image file 68 by including the second time information 84 generated by the second acquiring unit 44C in the corresponding frame-related data 72B in the same manner as the second object information 74. As a result, the frame-related data 72B of the second moving image file 68 includes the second time information 84. Therefore, a user or a device performing processing on the second moving image file 68 can identify, from the second moving image file 68, information corresponding to image data 18 obtained at a specific timing.

[0115] On the other hand, the second acquisition unit 44C transmits the second time information 84 to the second imaging device 12 via the first link unit 26A in the same manner as the second subject information 74.

[0116] In the first imaging device 10, the first acquisition unit 26C acquires the second time information 84 transmitted from the second imaging device 12 via the first linkage unit 26A. The first assignment unit 26D assigns the second time information 84 to the first moving image file 56 by including the second time information 84 acquired by the first acquisition unit 26C in the frame-related data 60B in the same manner as the second subject information 74. As a result, the frame-related data 60B of the first moving image file 56 includes the second time information 84. Therefore, a user or device performing processing on the first moving image file 56 can identify information corresponding to image data 18 obtained at a specific timing from the first moving image file 56.

[0117] [Third Modification] In the above-mentioned second modification, an example was given in which the first metadata 60 and second metadata 72 of the video file have first time information 82 and second time information 84, but the technology of the present disclosure is not limited to this, and for example, the first metadata 60 and second metadata 72 of the video file may have information about the imaging device.

[0118] 21 , for example, the first acquisition unit 26C acquires first imaging device-related information 86 in frame units (i.e., for each image data 16) obtained by capturing an image of the subject 14 by the image sensor 22. The first imaging device-related information 86 is information related to the first imaging device 10. Examples of the first imaging device-related information 86 include first position information 86A, first attitude information 86B, and first imaging orientation information 86C. The first position information 86A is information related to the position of the first imaging device 10. The first attitude information 86B is information related to the attitude of the first imaging device 10. The first imaging orientation information 86C is information in which the imaging direction (i.e., the direction of the optical axis) of the first imaging device 10 is expressed in terms of an orientation.

[0119] The first imaging device related information 86 is an example of "information related to the first imaging device" according to the technology of the present disclosure. The first position information 86A is an example of "first position information" according to the technology of the present disclosure. The first attitude information 86B is an example of "first direction information" according to the technology of the present disclosure. The first imaging orientation information 86C is an example of "first direction information" according to the technology of the present disclosure.

[0120] The first imaging device 10 is provided with a GNSS (Global Navigation Satellite System) receiver 88, an inertial sensor 90, and a geomagnetic sensor 92. The GNSS receiver 88, the inertial sensor 90, and the geomagnetic sensor 92 are connected to the processor 26. The GNSS receiver 88 receives radio waves transmitted from a plurality of satellites 94. The inertial sensor 90 measures physical quantities (e.g., angular velocity and acceleration) indicative of the three-dimensional inertial motion of the first imaging device 10 and outputs an inertial sensor signal indicative of the measurement results. The geomagnetic sensor 92 detects geomagnetism and outputs a geomagnetic sensor signal indicative of the detection results.

[0121] The first acquisition unit 26C calculates, as first position information 86A, latitude, longitude, and altitude that can identify the current position of the first imaging device 10 based on radio waves received by the GNSS receiver 88. The first acquisition unit 26C also calculates first attitude information 86B (e.g., information defined by a yaw angle, a roll angle, and a pitch angle) based on an inertial sensor signal input from the inertial sensor 90. The first acquisition unit 26C also calculates first imaging orientation information 86C based on the inertial sensor signal input from the inertial sensor 90 and a geomagnetic sensor signal input from the geomagnetic sensor 92. The first acquisition unit 26C also calculates the imaging attitude of the first imaging device 10 (whether the long side of the camera is oriented vertically or horizontally) from the information from the inertial sensor 90.

[0122] As an example, as shown in FIG. 22, in the first imaging device 10, the first acquisition section 26C acquires first imaging device related information 86 on a frame-by-frame basis (that is, each time one frame's worth of imaging is performed).

[0123] The first assigning unit 26D assigns the first imaging device-related information 86 to the first moving image file 56 by including the first imaging device-related information 86 generated by the first acquiring unit 26C in the corresponding frame-related data 60B in the same manner as the first subject information 62. As a result, the frame-related data 60B of the first moving image file 56 includes the first imaging device-related information 86. Therefore, a user or a device performing processing on the first moving image file 56 can identify information about the first imaging device 10 (here, as an example, the position of the first imaging device 10, the imaging attitude of the first imaging device 10, and the imaging direction of the first imaging device 10) from the first moving image file 56.

[0124] On the other hand, the first acquisition unit 26C transmits the first imaging device related information 86 to the second imaging device 12 via the first link unit 26A in the same manner as the first subject information 62.

[0125] In the second image capture device 12, the second acquisition unit 44C acquires the first image capture device-related information 86 transmitted from the first image capture device 10 via the second linkage unit 44A. The second assignment unit 44D assigns the first image capture device-related information 86 to the second moving image file 68 by including the first image capture device-related information 86 acquired by the second acquisition unit 44C in the frame-related data 72B in the same manner as the first subject information 62. As a result, the frame-related data 72B of the second moving image file 68 includes the first image capture device-related information 86, so that a user or device performing processing on the second moving image file 68 can identify information about the first image capture device 10 (here, as an example, the position of the first image capture device 10, the imaging attitude of the first image capture device 10, and the imaging direction of the first image capture device 10) from the second moving image file 68.

[0126] 23 as an example, the second acquisition unit 44C acquires second imaging device-related information 96 for each frame (i.e., for each image data 18) obtained by capturing an image of the subject 14 by the image sensor 40. The second imaging device-related information 96 is information related to the second imaging device 12. The second imaging device-related information 96 is an example of "information related to the second imaging device" according to the technology of the present disclosure.

[0127] The second imaging device-related information 96 includes, for example, second position information 96A, second attitude information 96B, and second imaging orientation information 96C. The second position information 96A is information relating to the position of the second imaging device 12. The second attitude information 96B is information relating to the attitude of the second imaging device 12. The second imaging orientation information 96C is information in which the imaging direction (i.e., the direction of the optical axis) of the second imaging device 12 is expressed in terms of an orientation.

[0128] The second imaging device 12 is provided with a GNSS receiver 98 similar to the GNSS receiver 88 , an inertial sensor 100 similar to the inertial sensor 90 , and a geomagnetic sensor 102 similar to the geomagnetic sensor 92 .

[0129] The second acquisition unit 44C calculates, as second position information 96A, latitude, longitude, and altitude that can identify the current position of the second image capture device 12 based on radio waves received by the GNSS receiver 98. The second acquisition unit 44C also calculates second attitude information 96B (e.g., information defined by a yaw angle, a roll angle, and a pitch angle) based on an inertial sensor signal input from the inertial sensor 100. The second acquisition unit 44C also calculates second image capture orientation information 96C based on the inertial sensor signal input from the inertial sensor 100 and a geomagnetic sensor signal input from the geomagnetic sensor 102.

[0130] As an example, as shown in FIG. 24, in the first imaging device 10, the first acquisition section 26C acquires first imaging device related information 86 on a frame-by-frame basis (that is, each time one frame's worth of imaging is performed).

[0131] The second assigning unit 44D assigns the second imaging device related information 96 to the second moving image file 68 by including the second imaging device related information 96 generated by the second acquiring unit 44C in the corresponding frame related data 72B in the same manner as the second subject information 74. As a result, the frame related data 72B of the second moving image file 68 includes the second imaging device related information 96, and therefore a user or device performing processing on the second moving image file 68 can identify information about the second imaging device 12 (here, as an example, the position of the second imaging device 12, the imaging attitude of the second imaging device 12, and the imaging direction of the second imaging device 12) from the second moving image file 68.

[0132] On the other hand, the second acquisition unit 44C transmits the second imaging device related information 96 to the first imaging device 10 via the second linkage unit 44A in the same manner as the second subject information 74.

[0133] In the first imaging device 10, the first acquisition unit 26C acquires second imaging device-related information 96 transmitted from the second imaging device 12 via the first linkage unit 26A. The first assignment unit 26D assigns the second imaging device-related information 96 to the first moving image file 56 by including the second imaging device-related information 96 acquired by the first acquisition unit 26C in the frame-related data 60B in a manner similar to that of the second subject information 74. As a result, the frame-related data 60B of the first moving image file 56 includes the second imaging device-related information 96, so that a user or device performing processing on the first moving image file 56 can identify information about the second imaging device 12 (here, as an example, the position of the second imaging device 12, the imaging attitude of the first imaging device 10, and the imaging direction of the second imaging device 12) from the first moving image file 56.

[0134] In the third modified example, the first imaging device 10 has the second imaging device related information 96 in addition to the first imaging device related information 86, and therefore it is possible to identify the relationship, such as the position, between the subject imaged by the first imaging device 10 and the subject imaged by the second imaging device 12. This makes it possible to determine whether the first subject information 62 acquired by the first imaging device 10 through imaging and the second subject information 74 acquired by cooperation with the second imaging device 12 are information about a common subject.

[0135] In the third modified example, first position information 86A, first attitude information 86B, and first imaging orientation information 86C are exemplified as the first imaging device-related information 86, and second position information 96A, second attitude information 96B, and second imaging orientation information 96C are exemplified as the second imaging device-related information 96. However, the technology of the present disclosure is not limited to this. For example, the first imaging device-related information 86 and the second imaging device-related information 96 may include distance information. The distance information is information indicating the distance between the first imaging device 10 and the second imaging device 12. The distance information is calculated using, for example, the first position information 86A and the second position information 96A. Alternatively, the distance information may be information indicating the distance (i.e., the distance between the first imaging device 10 and the second imaging device 12) obtained by performing distance measurement using phase difference pixels or laser distance measurement between the first imaging device 10 and the second imaging device 12. In this way, since the first imaging device related information 86 and the second imaging device related information 96 contain distance information, a user or device performing processing on a moving image file can determine the distance between the first imaging device 10 and the second imaging device 12 from the moving image file.

[0136] The first imaging orientation information 86C or the second imaging orientation information 96C may also include information indicating the direction (for example, orientation) of the first imaging device 10 or the second imaging device 12 from one to the other.

[0137] In addition, in the third modified example, an example in which the processor calculates the first position information 86A and the second position information 96A using GNSS has been given, but this is merely one example. For example, information (e.g., latitude, longitude, and altitude) identified from a position specified in map data by a user or the like may be used as the first position information 86A or the second position information 96A.

[0138] Furthermore, the first position information 86A and the second position information 96A do not have to be information defined by latitude, longitude, and altitude; the first position information 86A or the second position information 96A may be information defined by latitude and longitude, or may be information defined by two-dimensional coordinates or three-dimensional coordinates.

[0139] When the first position information 86A or the second position information 96A is defined by two-dimensional coordinates or three-dimensional coordinates, for example, the current position of the imaging device is defined by the two-dimensional coordinates or three-dimensional coordinates on a two-dimensional plane or in a three-dimensional space applied to real space with a position designated by a user or the like as the origin. In this case, the current position of the imaging device is calculated based on, for example, an inertial sensor signal and a geomagnetic sensor signal.

[0140] Furthermore, in the third modified example, information defined by a yaw angle, a roll angle, and a pitch angle has been exemplified as the first attitude information 86B and the second attitude information 96B. However, this is merely an example, and information indicating an attitude identified from the yaw angle, the roll angle, and the pitch angle among multiple attitudes of the imaging device (for example, upward, downward, diagonally downward, diagonally upward, etc.) may be used as the first attitude information 86B or the second attitude information 96B.

[0141] [Fourth Modification] In the above example, an example was described in which the first imaging device 10 includes an image sensor 22 and the second imaging device 12 includes an image sensor 40. However, in the fourth modification, an example will be described in which the first imaging device 10 includes an infrared light sensor 104 and the second imaging device 12 includes a visible light sensor 106, as shown in FIG. 25.

[0142] 25 shows an example in which a human subject 108, which is an example of a "subject" according to the technology of the present disclosure, is imaged from approximately the same position and from approximately the same direction by the first imaging device 10 and the second imaging device 12. The infrared light sensor 104 provided in the first imaging device 10 is a sensor that captures light (infrared light in this case) in a wavelength range higher than the wavelength range of visible light, and the visible light sensor 106 provided in the second imaging device 12 is a sensor that captures visible light.

[0143] The signals output from the infrared light sensor 104 and the visible light sensor 106 are of different types. That is, the signals output from the infrared light sensor 104 are signals obtained by capturing an image of infrared light, and the signals output from the visible light sensor 106 are signals obtained by capturing an image of visible light.

[0144] The first imaging device 10 uses the infrared light sensor 104 to capture an image of the human subject 108, thereby generating thermal imaging data 110 that represents a thermal image. The first imaging device 10 also generates a legend 110A that indicates a standard of temperature distribution within the thermal imaging data 110. The legend 110A is associated with the thermal imaging data 110. The second imaging device 12 uses the visible light sensor 106 to capture an image of the human subject 108, thereby generating visible light image data 112 that represents a visible light image.

[0145] The infrared light sensor 104 is an example of a "first sensor" according to the technology of the present disclosure. The visible light sensor 106 is an example of a "second sensor" according to the technology of the present disclosure. The thermal imaging data 110 is an example of a "first output result" and "non-visible light image data" according to the technology of the present disclosure. The visible light image data 112 is an example of a "second output result" and "visible light image data" according to the technology of the present disclosure.

[0146] As an example, as shown in FIG. 26 , in the first imaging device 10, the first acquisition unit 26C acquires thermal image-related information 114 on a frame-by-frame basis (i.e., each time one frame's worth of image is captured). The thermal image-related information 114 is information related to the thermal imaging data 110. An example of the thermal image-related information 114 is information including specified temperature range data. The specified temperature range data is data indicating an image area within the thermal imaging data 110 that falls within a specified temperature range (e.g., 37 degrees or higher). The thermal image-related information 114 may include temperature text information and a legend 110A. The thermal image-related information 114 may also include data obtained by reducing the thermal imaging data 110.

[0147] The first assigning unit 26D assigns the thermal image related information 114 to the first moving image file 56 by including the thermal image related information 114 generated by the first acquiring unit 26C in the corresponding frame related data 60B in the same manner as the first object information 62. As a result, the frame related data 60B of the first moving image file 56 contains the thermal image related information 114, so that a user or device performing processing on the first moving image file 56 can identify information related to the thermal image data 110 from the first moving image file 56.

[0148] On the other hand, the first acquisition unit 26C transmits the thermal image related information 114 to the second imaging device 12 via the first link unit 26A in the same manner as the first object information 62.

[0149] In the second imaging device 12, the second acquisition unit 44C acquires the thermal image-related information 114 transmitted from the first imaging device 10 via the second link unit 44A. The second assignment unit 44D assigns the thermal image-related information 114 to the second moving image file 68 by including the thermal image-related information 114 acquired by the second acquisition unit 44C in the frame-related data 72B in the same manner as the first object information 62. Since the frame-related data 72B of the second moving image file 68 contains the thermal image-related information 114, a user or device performing processing on the second moving image file 68 can identify information related to the thermal image data 110 from the second moving image file 68. Therefore, a user who obtains the second moving image file 68 can refer to the visible light image data 112 and the thermal image-related information 114 solely from the second moving image file 68. In addition, a user or a device can create a composite image, for example, by adding information about the thermal imaging data 110 to the visible light image represented by the visible light image data 112 included in the second moving image file 68.

[0150] 27 , in the second imaging device 12, the second acquisition unit 44C acquires visible light-related information 116 on a frame-by-frame basis (i.e., each time one frame's worth of image is captured). The visible light-related information 116 is information related to the visible light image data 112. Examples of the visible light-related information 116 include information corresponding to the type or attribute information of the subject, such as age, sex, and facial expression, as well as data obtained by reducing the visible light image data 112 (e.g., thumbnail image data).

[0151] The second assigning unit 44D assigns the visible light-related information 116 to the second moving image file 68 by including the visible light-related information 116 generated by the second acquiring unit 44C in the corresponding frame-related data 72B in the same manner as the second object information 74. As a result, the frame-related data 72B of the second moving image file 68 includes the visible light-related information 116, and therefore a user or device performing processing on the second moving image file 68 can identify information related to the visible light image data 112 from the second moving image file 68.

[0152] On the other hand, the second acquisition unit 44C transmits the visible light-related information 116 to the first imaging device 10 via the second link unit 44A in the same manner as the second subject information 74.

[0153] In the first imaging device 10, the first acquisition unit 26C acquires the visible light-related information 116 transmitted from the first imaging device 10 via the first link unit 26A. The first assignment unit 26D assigns the visible light-related information 116 to the first moving image file 56 by including the visible light-related information 116 acquired by the first acquisition unit 26C in the frame-related data 60B in the same manner as the second object information 74. As a result, the frame-related data 60B of the first moving image file 56 contains the visible light-related information 116, allowing a user or device performing processing on the first moving image file 56 to identify information related to the visible light image data 112 from the first moving image file 56. Therefore, a user who has obtained the first moving image file 56 can refer to the thermal imaging data 110 and the visible light-related information 116 using only the first moving image file 56. Furthermore, a user or a device can create a composite image by adding information about the visible light-related information 116 to the thermal image represented by the thermal imaging data 110 included in the first moving image file 56, for example.

[0154] In the example shown in FIG. 25 , thermal imaging data 110 is shown, but this is merely an example. For example, as shown in FIG. 28 , distance image data 118 representing a distance image may be used instead of the thermal imaging data 110. In this case, the first imaging device 10 is provided with a distance measurement sensor 120, which measures the distance to the subject 122. The distance measurement sensor 120 includes multiple IR (Infrared Rays) pixels arranged two-dimensionally. Each of the multiple IR pixels receives IR reflected light from the subject 122, thereby measuring the distance for each IR pixel. The distance measurement result for each IR pixel is a distance image. A distance image refers to an image in which the distance to the target measured for each IR pixel is represented by color and / or shade.

[0155] When using distance image data 118 instead of thermal image data 110 in this way, as shown in Fig. 29 as an example, in the first imaging device 10, the first acquisition unit 26C acquires distance image related information 124 on a frame-by-frame basis (i.e., each time one frame's worth of image data is captured). The distance image related information 124 is information related to the distance image data 118. An example of the distance image related information 124 is data indicating one or more specified image areas within the distance image data 118. The distance image related information 124 may also include image data obtained by reducing the distance image data 118 (e.g., thumbnail image data).

[0156] First assigning unit 26D assigns distance image related information 124 to first moving image file 56 by including distance image related information 124 generated by first acquiring unit 26C in corresponding frame related data 60B in the same manner as first subject information 62. As a result, frame related data 60B of first moving image file 56 includes distance image related information 124, and therefore a user or device performing processing on first moving image file 56 can identify information related to distance image data 118 from first moving image file 56.

[0157] Meanwhile, the first acquisition unit 26C transmits the distance image related information 124 to the second imaging device 12 via the first link unit 26A in the same manner as the first subject information 62.

[0158] In the second imaging device 12, the second acquisition unit 44C acquires the distance image related information 124 transmitted from the first imaging device 10 via the second linkage unit 44A. The second assignment unit 44D assigns the distance image related information 124 to the second moving image file 68 by including the distance image related information 124 acquired by the second acquisition unit 44C in the frame related data 72B in the same manner as the first object information 62. As a result, the frame related data 72B of the second moving image file 68 includes the distance image related information 124, allowing a user or device performing processing on the second moving image file 68 to identify information related to the distance image data 118 from the second moving image file 68. Therefore, a user who has obtained the second moving image file 68 can refer to the distance image data 118 and visible light data using only the second moving image file 68. In addition, a user or a device can create a composite image, for example, by adding information about the distance image data 118 to the visible light image represented by the visible light image data 112 included in the second moving image file 68.

[0159] Although the fourth modification has been described with reference to an example in which the human subject 108 is imaged by the infrared light sensor 104, the technology of the present disclosure is not limited to this. For example, the technology of the present disclosure is also applicable when the subject 198 is imaged in a wavelength range lower than visible light.

[0160] [Other Modifications] In the above embodiment, an example was described in which the first imaging device 10 and the second imaging device 12 capture images of the subject 14, which is a common subject. However, different subjects may be captured by the first imaging device 10 and the second imaging device 12. In this case, information about one of the different subjects and information about the other of the different subjects can be identified from a single video file (e.g., the first video file 56 or the second video file 68).

[0161] A possible scenario in which different subjects are captured by the first imaging device 10 and the second imaging device 12 is when the first imaging device and the second imaging device 12 are used as part of a drive recorder mounted in a vehicle. For example, as shown in FIG. 30 , a first imaging device 10 is attached to a vehicle 126 as a front camera mounted in a two-camera drive recorder, and a second imaging device 12 is attached as a rear camera. The first imaging device 10 captures an image of a subject 128 (a person in the example shown in FIG. 30 ) in front of the vehicle, and the second imaging device 12 captures an image of a subject 130 (a vehicle in the example shown in FIG. 30 ) behind the vehicle. This allows a user or device performing processing on the first moving image file 56 and the second moving image file 68 to identify information about the subject 130 from the first moving image file 56 and to identify information about the subject 128 from the second moving image file 68. Therefore, for example, it is possible to improve the efficiency of the task of checking which image data 16 contained in the first moving image file 56 corresponds to which image data 18 contained in the second moving image file 68.

[0162] Note that the automobile 126 is merely an example, and the first and second image capturing devices 10 and 12 may be attached to other types of vehicles such as trains or motorcycles at positions where different subjects can be captured. Also, the example of capturing images of the front and rear of the automobile 126 is merely an example, and images may be captured of the right and left diagonally forward sides of the vehicle, the left and right sides of the vehicle, or the outside and inside of the vehicle, as long as the first and second image capturing devices 10 and 12 are attached to the vehicle so as to capture different subjects.

[0163] In the above embodiment, an example has been described in which the first image file creation process is executed by the first information processing device 20 in the first imaging device 10, and the second image file creation process is executed by the second information processing device 36 in the second imaging device 12, but the technology of the present disclosure is not limited to this. For example, as shown in FIG. 31 ,

[0164] An external device connected to the imaging device so as to be able to communicate with the imaging device via a network 132 such as a Small Area Network (Small Area Network)

[0165] The image file creation process may be executed by a computer 136 in the device 134. An example of the computer 136 is a server computer for a cloud service.

[0166] 31, the computer 136 includes a processor 138, a storage 140, and a memory 142. The storage 140 stores an image file creation program.

[0167] The imaging device requests the external device 134 to execute an image file creation process via the network 132. In response to this, the processor 138 of the external device 134 reads an image file creation program from the storage 140 and executes the image file creation program on the memory 142. The processor 138 performs the image file creation process in accordance with the image file creation program executed on the memory 142. The processor 138 then provides the processing result obtained by executing the image file creation process to the imaging device via the network 132.

[0168] 31 shows an example of an embodiment in which the external device 134 executes the image file creation process, but this is merely one example. For example, the image file creation process may be distributed and executed between the imaging device and the external device 134, or the image file creation process may be distributed and executed between a plurality of devices including the imaging device and the external device 134.

[0169] In the above embodiment, an example of a form in which a moving image file is generated has been described, but the format of the moving image file may be any of MPEG (Moving Picture Experts Group)-4, H.264, MJPEG (Motion JPEG), HEIF (High Efficiency Image File Format), AVI (Audio Video Interleave), MOV (QuickTime file format), WMV (Windows Media Video), and FLV (Flash Video). Note that, from the viewpoint of adding the metadata (additional information) described in the above embodiment, HEIF moving image data is preferable. Furthermore, the technology of the present disclosure is also applicable when a still image file is generated. In this case, an image file in a format that allows additional information to be added to an area separate from the image data (i.e., a recordable format) is used as the still image file.

[0170] An example of the structure of an image file in a format in which additional information can be added to an area separate from the image data is the data structure of a JPEG (Joint Photographic Experts Group) file conforming to the Exif (Exchangeable Image File Format) standard, as shown in Fig. 32. Although a JPEG file is shown here as an example, this is merely an example, and the image file is not limited to a JPEG file.

[0171] JPEG XT Part 3, which is a type of JPEG, provides marker segments "APP1" and "APP11" as areas where additional information can be added. "APP1" stores tag information related to the date and time, location, and conditions of image data capture. "APP11" includes JUMBF (JPEG Universal Metadata box format) boxes (specifically, for example, JUMBF1 and JUMBF2 boxes) which are storage areas for metadata. The JUMBF1 box contains Content Header 1 (Content Header 2) where metadata is stored.

[0172] There is a Type box in which information can be written in JSON (JavaScript (registered trademark) Object Notation) format. The metadata description method is not limited to JSON format, and XML (Extensible Markup Language) format can also be used. Furthermore, in the JUMBF2 box, information different from that in the JUMBF1 box can be written in the Content Type box. Approximately 60,000 such JUMBF boxes can be created in a JPEG file.

[0173] Furthermore, the data structure of Exif version 3.0 (Exif 3.0) expands the area where additional information can be added compared to the previous version, Exif 2.32, by adding a box area conforming to JUMBF. This box area may have multiple levels, and in this case, additional information can be stored (i.e., written) with different information content or abstraction levels depending on the level of hierarchy. For example, the type of subject captured in the image data may be written in a higher level, and the state or attributes of the subject may be written in a lower level.

[0174] The items and number of additional information that can be added to an image file vary depending on the file format, and additional information for new items can be added by updating the version information of the image file. The items of additional information refer to the viewpoints for adding additional information (i.e., the categories into which the information is classified).

[0175] In the above embodiment, an example in which the image file creation program is stored in the NVM has been described, but the technology of the present disclosure is not limited to this. For example, the image file creation program may be stored in a portable, computer-readable, non-transitory storage medium such as an SSD (Solid State Drive), a USB memory, or a magnetic tape. The image file creation program stored in the non-transitory storage medium is installed in the imaging device. The processor executes an image file creation process in accordance with the image file creation program.

[0176] In addition, the image file creation program may be stored in a storage device such as another computer or server device connected to the imaging device via a network, and the image file creation program may be downloaded and installed in the imaging device in response to a request from the imaging device.

[0177] It is not necessary to store the entire image file creation program in the storage device of another computer or server device connected to the imaging device, or in the NVM; only a part of the image file creation program may be stored therein.

[0178] Furthermore, although the imaging device shown in FIG. 2 has a built-in information processing device, the technology of the present disclosure is not limited to this, and for example, the information processing device may be provided outside the imaging device.

[0179] In the above embodiment, an example in which the technology of the present disclosure is realized by a software configuration has been described, but the technology of the present disclosure is not limited to this, and devices including an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a PLD (Programmable Logic Device) may also be applied.

[0180] A combination of hardware and software configurations may also be used.

[0181] The hardware resources that execute the image file creation process described in the above embodiments can be various processors, as listed below. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource that executes the image file creation process by executing software, i.e., a program. Examples of processors include dedicated electronic circuits, such as FPGAs, PLDs, or ASICs, which are processors with a circuit configuration designed specifically for executing specific processes. Each processor has built-in or connected memory, and each processor executes the image file creation process by using the memory.

[0182] The hardware resource that executes the image file creation process may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the image file creation process may be a single processor.

[0183] Examples of a system using a single processor include: a first configuration in which one processor is configured using a combination of one or more CPUs and software, and this processor functions as a hardware resource that executes the image file creation process; a second configuration in which a processor is used to implement the functions of an entire system, including multiple hardware resources that execute the image file creation process, on a single IC (Integrated Circuit) chip, as typified by SoCs (System-on-a-Chips); and in this way, the image file creation process is implemented using one or more of the above-mentioned various processors as hardware resources.

[0184] Furthermore, the hardware structure of these various processors can be, more specifically, electronic circuits that combine circuit elements such as semiconductor devices. The image file creation process described above is merely an example. It goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the process.

[0185] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0186] In this specification, the grammatical concept of "A or B" includes not only the concept of "either one of A or B," but also a concept synonymous with "at least one of A and B." In other words, "A or B" includes the meaning that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A or B" is also applied when three or more things are expressed by connecting them with "or."

[0187] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. a linking step of linking a first imaging process for generating a first image file including first image data obtained by imaging a first subject with a second imaging process for generating a second image file including second image data obtained by imaging a second subject; acquiring first object information relating to the first object; an adding step of adding the first subject information to the second image file by including the first subject information in second incidental information recorded in the second image file, The acquiring step acquires second subject information related to the second subject as information to be included in the second auxiliary information, The adding step adds the second subject information to the first image file by including the second subject information in first incidental information recorded in the first image file. Information processing methods.

2. The first and second auxiliary information recorded in the first image file have common information. The information processing method according to claim 1 .

3. the first image data is moving image data composed of a plurality of first frames; The first auxiliary information recorded in the first image file includes first time information related to the corresponding first frame.

3. The information processing method according to claim 1.

4. The adding step includes adding the first time information to the second image file by including the first time information in the second auxiliary information. The information processing method according to claim 3.

5. the second image data is moving image data composed of a plurality of second frames, The second annex information includes second time information regarding the corresponding second frame. The information processing method according to any one of claims 1 to 4.

6. The adding step includes adding the second time information to the first image file by including the second time information in first auxiliary information recorded in the first image file. The information processing method according to claim 5.

7. The first incidental information recorded in the first image file includes information about a first image capturing device that performs the first image capturing process. The information processing method according to any one of claims 1 to 6.

8. The adding step includes adding information about the first imaging device to the second image file by including the information about the first imaging device in the second incidental information. The information processing method according to claim 7.

9. The information about the first imaging device includes first position information about a position of the first imaging device, first direction information about an imaging direction of the first imaging device, or distance information about a distance between the first imaging device and a second imaging device that performs the second imaging process.

9. The information processing method according to claim 7 or 8.

10. The second incidental information includes information regarding a second imaging device that performs the second imaging process. The information processing method according to any one of claims 7 to 9.

11. The adding step includes adding information about the second imaging device to the first image file by including the information about the second imaging device in the first incidental information. The information processing method according to claim 10.

12. The first subject and the second subject are a common subject. The information processing method according to any one of claims 1 to 11.

13. the first imaging process uses a first sensor that images the first subject; the second imaging process uses a second sensor that images the second subject; The first output result output from the first sensor and the second output result output from the second sensor are different in type from each other. The information processing method according to claim 12.

14. one of the first output result and the second output result is visible light image data obtained by capturing an image of visible light; The other of the first output result and the second output result is non-visible light image data obtained by capturing an image of light in a wavelength range higher or lower than the wavelength range of the visible light. The information processing method according to claim 13.

15. one of the first output result and the second output result is visible light image data obtained by capturing an image of visible light; The other of the first output result and the second output result is distance image data obtained by performing distance measurement. The information processing method according to claim 13.

16. The first object and the second object are different objects. The information processing method according to any one of claims 1 to 12.

17. The first imaging device performing the first imaging process and the second imaging device performing the second imaging process are mounted on a vehicle. The information processing method according to claim 16.

18. the first image file is a first moving image file including first moving image data as the first image data, The second image file is a second moving image file including second moving image data as the second image data.

18. The information processing method according to any one of claims 1 to 17.

19. A processor is provided. The processor, a first imaging process for generating a first image file including first image data obtained by imaging a first subject, and a second imaging process for generating a second image file including second image data obtained by imaging a second subject, acquiring first object information relating to the first object; adding the first object information to the second image file by including the first object information in second incidental information recorded in the second image file; acquiring second subject information related to the second subject as information to be included in the second auxiliary information; The second subject information is added to the first image file by including the second subject information in first incidental information recorded in the first image file. Information processing device.

Images