Imaging apparatus, recording device and display control device

The imaging device segments and encodes HDR images for optimal recording and playback across devices with varying compatibility, ensuring high-quality image reproduction through HEIF file format and metadata management.

JP2024163156AActive Publication Date: 2024-11-21CANON KK
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2024148279
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-11-21
Estimated Expiration
2038-02-16

AI Technical Summary

Technical Problem

Existing image processing devices do not provide an optimal method for recording and playback of High Dynamic Range (HDR) images, particularly when transitioning between devices with varying compatibility.

Method used

An imaging device that divides HDR image data into multiple segments, encodes each segment, and records them in a HEIF file format, along with metadata for playback compatibility, using encoding methods like H.265, and a display control device for combining and displaying these segments as a cohesive HDR image.

Benefits of technology

Enables efficient recording and playback of HDR images across devices with varying capabilities, maintaining image quality and compatibility through segment-based encoding and metadata management.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024163156000001_ABST
    Figure 2024163156000001_ABST
Patent Text Reader

Abstract

To record an image in a recording form suitable for recording and reproduction when recording an image that has a large amount of data as of an HDR image.SOLUTION: An imaging apparatus records HDR (High Dynamic Range) image data obtained by photographing. When encoding HDR image data obtained by photographing with an imaging sensor, the imaging apparatus divides the HDR image data in an encoding area, serving as an encoding target, into a plurality of separate HDR image data, encodes each of the plurality of separate HDR image data by means of encoding means, and exerts control such that the plurality of encoded separate HDR image data are recorded on a recording medium in a predetermined recording format.SELECTED DRAWING: Figure 3
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an imaging device, a recording device, and a display control device. [Background technology]

[0002] An imaging device is known as an image processing device that compresses and codes image data. Such an image processing device acquires a moving image signal by an imaging unit, compresses and codes the acquired moving image signal, and records the compressed and coded image file on a recording medium. Conventionally, image data before compression and coding was expressed in SDR (Standard Dynamic Range) with a maximum brightness level of 100 nits. However, in recent years, image data is expressed in HDR (High Dynamic Range) with an upper limit of the brightness level extended to about 10,000 nits, and image data having a brightness range close to the brightness range that humans can perceive is provided.

[0003] Patent Document 1 describes an image data recording device that, when capturing and recording an HDR image, generates and records image data that allows the HDR image to be viewed in detail even on a device that does not support HDR. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-139618 A Summary of the Invention [Problem to be solved by the invention]

[0005] Patent Document 1 describes recording an HDR image, but does not consider the optimal recording method for recording an HDR image.

[0006] Therefore, an object of the present invention is to provide a device that records images in a recording format suitable for recording and playback when recording images with a large amount of data, such as HDR images, and a display control device for playing back images recorded in that recording format. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, an imaging device according to the present invention comprises: The imaging device records HDR (high dynamic range) image data obtained by shooting, and is characterized by having an imaging sensor, an encoding means for encoding the HDR image data obtained by shooting using the imaging sensor, and a recording control means for dividing HDR image data of an encoding area to be encoded, among the HDR image data obtained by shooting using the imaging sensor, into a plurality of divided HDR image data, encoding each of the plurality of divided HDR image data by the encoding means, and controlling the plurality of encoded divided HDR image data to be recorded on a recording medium in a predetermined recording format.

[0008] The recording device of the present invention further comprises: The image data capturing system has an encoding means for encoding image data obtained by photographing, and a recording control means for dividing the image data captured by the imaging sensor into a plurality of split image data, encoding each of the plurality of split image data by the encoding means, and controlling the recording of the plurality of encoded split image data in one image file of a predetermined format on a recording medium, wherein the recording control means controls the recording of overlay information for combining the plurality of split image data back to the state before the division as an image item in the image file.

[0009] Further, the display control device of the present invention comprises: The image processing apparatus includes a read means for reading an HDR image file from a recording medium on which the HDR image data of a coding area to be coded is divided into a plurality of divided HDR image data, the plurality of divided HDR image data is coded by an encoding means, and the coded divided HDR image data is recorded as an HDR image file in a predetermined recording format, a combine means for combines each of the plurality of divided HDR image data included in the read HDR image file, and a display control means for combining the plurality of HDR image data decoded by the decoding means and controlling the display means to display the combined data as one HDR image data. Effect of the Invention

[0010] According to the present invention, it is possible to provide an imaging device that records images in a recording format suitable for recording and playback when recording images with a large amount of data and high resolution, and a display control device for playing back images recorded in that recording format. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing the configuration of an imaging device 100. [Diagram 2] A diagram showing the structure of a HEIF file. [Diagram 3] 11 is a flowchart showing a process in HDR shooting mode. [Figure 4] 11 is a flowchart showing a process for determining a division method for image data in HDR shooting mode. [Diagram 5] 1A and 1B are diagrams showing an encoding area and a division method for image data when recording HDR image data. [Figure 6] 1 is a flowchart showing a process for constructing a HEIF file. [Figure 7] 11 is a flowchart showing a display process when playing back HDR image data recorded as a HEIF file. [Figure 8] 11 is a flowchart showing an overlay image creation process. [Figure 9]11 is a flowchart showing a property acquisition process of an image item. [Figure 10] 11 is a flowchart showing a process of acquiring data of an image item. [Figure 11] 11 is a flowchart showing the process of obtaining the item ID of an image that constitutes a main image. [Figure 12] 11 is a flowchart showing an image creation process for one image item. [Figure 13] FIG. 4 is a diagram showing the positional relationship between an overlay image and divided images. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described in detail below with reference to the accompanying drawings, taking an image pickup device 100 as an example. However, the present invention is not limited to the following embodiment.

[0013] <Configuration of imaging device> FIG. 1 is a block diagram showing an imaging device 100. As shown in FIG. 1, the imaging device 100 includes a CPU 101, a memory 102, a non-volatile memory 103, an operation unit 104, an imaging unit 112, an image processing unit 113, an encoding processing unit 114, a display control unit 115, and a display unit 116. Furthermore, the imaging device 100 includes a communication control unit 117, a communication unit 118, a recording medium control unit 119, and an internal bus 130. The imaging device 100 forms an optical image of a subject on a pixel array of the imaging unit 112 using a photographing lens 111, and the photographing lens 111 may be either non-detachable or detachable from the body (housing, main body) of the imaging device 100. Furthermore, the imaging device 100 writes and reads image data to and from a recording medium 120 via a recording medium control unit 119. The recording medium 120 may be either detachable or detachable from the imaging device 100.

[0014] The CPU 101 executes a computer program stored in the non-volatile memory 103 to control the operation of each unit (each functional block) of the imaging device 100 via an internal bus .

[0015] The memory 102 is a rewritable volatile memory. The memory 102 temporarily records a computer program for controlling the operation of each part of the imaging device 100, information such as parameters related to the operation of each part of the imaging device 100, information received by a communication control unit 117, etc. The memory 102 also temporarily records an image acquired by the imaging unit 112, and images and information processed by the image processing unit 113, the encoding processing unit 114, etc. The memory 102 has a sufficient storage capacity for temporarily recording these.

[0016] The non-volatile memory 103 is an electrically erasable and recordable memory, and may be, for example, an EEPROM. The non-volatile memory 103 stores computer programs that control the operation of each part of the imaging device 100 and information such as parameters related to the operation of each part of the imaging device 100. The computer programs realize various operations performed by the imaging device 100.

[0017] The operation unit 104 provides a user interface for operating the imaging device 100. The operation unit 104 includes various buttons such as a power button, a menu button, and a shooting button, and the various buttons are configured by switches, a touch panel, or the like. The CPU 101 controls the imaging device 100 according to a user's instruction input via the operation unit 104. Note that, although the case where the CPU 101 controls the imaging device 100 based on an operation input via the operation unit 104 has been described here as an example, the present invention is not limited to this. For example, the CPU 101 may control the imaging device 100 based on a request input via the communication unit 118 from a remote controller (not shown), a mobile terminal (not shown), or the like.

[0018] The photographing lens (lens unit) 111 is composed of a group of lenses (not shown) including a zoom lens, a focus lens, etc., a lens control unit (not shown), an aperture (not shown), etc. The photographing lens 111 can function as a zoom unit that changes the angle of view. The lens control unit adjusts the focus and controls the aperture value (F value) by a control signal transmitted from the CPU 101. The imaging unit 112 can function as an acquisition unit that sequentially acquires a plurality of images that constitute a moving image. For example, an area image sensor such as a CCD (charge-coupled device) or a CMOS (complementary metal-oxide semiconductor) element is used as the imaging unit 112. The imaging unit 112 has a pixel array (not shown) in which photoelectric conversion units (not shown) that convert an optical image of a subject into an electric signal are arranged in a matrix, that is, two-dimensionally. An optical image of the subject is formed on the pixel array by the photographing lens 111. The imaging unit 112 outputs the captured image to the image processing unit 113 or the memory 102. The imaging unit 112 can also acquire a still image.

[0019] The image processing unit 113 performs a predetermined image processing on the image data output from the imaging unit 112 or the image data read from the memory 102. Examples of the image processing include an interpolation process, a reduction process (resizing process), and a color conversion process. The image processing unit 113 also performs a predetermined calculation process for exposure control, distance measurement control, and the like, using the image data acquired by the imaging unit 112. The exposure control, distance measurement control, and the like are performed by the CPU 101 based on the calculation results obtained by the calculation process by the image processing unit 113. Specifically, the CPU 101 performs an AE (automatic exposure) process, an AWB (auto white balance) process, an AF (auto focus) process, and the like.

[0020] The encoding processing unit 114 compresses the size of the image data by performing intraframe predictive coding (intra-screen predictive coding), inter-frame predictive coding (inter-screen predictive coding), or the like on the image data. The encoding processing unit 114 is, for example, a coding device configured with a semiconductor element or the like. The encoding processing unit 114 may be a coding device provided outside the imaging device 100. The encoding processing unit 114 performs encoding processing by, for example, the H.265 (ITU H.265 or ISO / IEC23008-2) method.

[0021] The display control unit 115 controls the display unit 116. The display unit 116 has a display screen (not shown). The display control unit 115 performs resizing, color conversion, and the like on the image data to generate an image that can be displayed on the display screen of the display unit 116, and outputs the image, that is, an image signal, to the display unit 116. The display unit 116 displays an image on the display screen based on an image signal sent from the display control unit 115. The display unit 116 has an OSD (On Screen Display) function that is a function of displaying a setting screen such as a menu on the display screen. The display control unit 115 can superimpose an OSD image on the image signal and output the image signal to the display unit 116. The display unit 116 is composed of a liquid crystal display, an organic EL display, or the like, and displays the image signal sent from the display control unit 115. The display unit 116 may be, for example, a touch panel. When the display unit 116 is a touch panel, the display unit 116 can also function as the operation unit 104.

[0022] The communication control unit 117 is controlled by the CPU 101. The communication control unit 117 generates a modulated signal conforming to a wireless communication standard previously determined by IEEE802.11 or the like, and outputs the modulated signal to the communication unit 118. The communication control unit 117 also receives a modulated signal conforming to the wireless communication standard via the communication unit 118, decodes the received modulated signal, and outputs a signal corresponding to the decoded signal to the CPU 101. The communication control unit 117 includes a register for storing communication settings. The communication control unit 117 can adjust the transmission and reception sensitivity during communication under the control of the CPU 101. The communication control unit 117 can transmit and receive using a predetermined modulation method. The communication unit 118 outputs a modulated signal supplied from the communication control unit 117 to an external device 127 such as an information communication device outside the imaging device 100, and also includes an antenna for receiving a modulated signal from the external device 127. The communication unit 118 also includes a communication circuit and the like. Although the case where wireless communication is performed by the communication unit 118 has been described as an example here, the communication performed by the communication unit 118 is not limited to wireless communication. For example, the communication unit 118 and the external device 127 may be connected by an electrical connection using wiring or the like.

[0023] The recording medium control unit 119 controls the recording medium 120. The recording medium control unit 119 outputs a control signal for controlling the recording medium 120 to the recording medium 120 based on a request from the CPU 101. For example, a non-volatile memory or a magnetic disk is used as the recording medium 120. As described above, the recording medium 120 may be removable or non-removable. The recording medium 120 records encoded image data and the like. The image data and the like are saved as files in a format compatible with the file system of the recording medium 120. Examples of the files include MP4 files (ISO / IEC 14496-14:2003) and MXF (Material eXchange Format) files. The respective functional blocks 101 to 104, 112 to 115, 117, and 119 are accessible to each other via an internal bus 130.

[0024] Here, a normal operation of the imaging device 100 of this embodiment will be described.

[0025] When a user operates a power button on the operation unit 104, a start-up instruction is issued from the operation unit 104 to the CPU 101 of the imaging device 100. In response to this instruction, the CPU 101 controls a power supply unit (not shown) to supply power to each block of the imaging device 100. When power is supplied, the CPU 101 checks, for example, which mode the mode changeover switch of the operation unit 104 is in, for example, a still image capture mode, a playback mode, etc., based on an instruction signal from the operation unit 102.

[0026] In the normal still image shooting mode, the imaging device 100 performs shooting processing when the user operates the still image recording button of the operation unit 104 in a shooting standby state. In the shooting processing, image data of a still image shot by the imaging unit 112 is subjected to image processing by the image processing unit 113, and is then encoded by the encoding processing unit 114, and the encoded image data is recorded as an image file by the recording medium control unit 119 on the recording medium 120. Note that in the shooting standby state, an image is shot by the imaging unit 112 at a predetermined frame rate, and the image processing unit performs image processing for display and the display control unit 115 displays the image on the display unit 116, thereby displaying a live view image.

[0027] In the playback mode, an image file recorded on the recording medium 120 is read by the recording medium control unit 119, and the image data of the read image file is decoded by the encoding processing unit 114. In other words, the encoding processing unit 114 also has a decoder function. Then, the image processing unit 113 performs processing for display, and the display control unit 115 causes the display unit 116 to display the image.

[0028] Normal still image capture and playback are performed as described above, but the imaging device of this embodiment has an HDR shooting mode for capturing not only normal still images but also HDR still images, and can also play back captured HDR still images.

[0029] The process of capturing and playing back HDR still images will be described below.

[0030] <File structure> First, we will explain the file structure when recording HDR still images.

[0031] Recently, a still image file format called High Efficiency Image File Format (hereinafter referred to as HEIF) was established (ISO / IEC 23008-12:2017).

[0032] Compared to conventional still image file formats such as JPEG, it has the following features: This file format conforms to the ISO Base Media File Format (hereinafter referred to as ISOBMFF). (ISO / IEC 14496-14:2003) -Can store multiple still images, not just a single one. -Can store still images compressed in a compression format used to compress video, such as HEVC / H.265 or AVC / H.264.

[0033] In this embodiment, HEIF is used as a recording file for HDR still images.

[0034] First, we will explain the data stored in HEIF.

[0035] HEIF manages each piece of stored data in units called items.

[0036] In addition to the data itself, each item has an item ID (item_ID) that is an integer value that is unique within the file, and an item type (item_type) that indicates the type of item.

[0037] Items can be divided into image items, whose data represents an image, and metadata items, whose data is metadata.

[0038] Image items include coded image items, which are image data that has been coded, and derived image items, which represent an image that is the result of manipulating one or more other image items.

[0039] An example of a derived image item is an overlay image, which is an overlay type derived image item. This is an overlay image that is the result of arranging any number of image items at any positions and compositing them using an ImageOverlay structure (overlay information).

[0040] An example of a metadata item is Exif data.

[0041] As mentioned above, HEIF can store multiple image items.

[0042] If there are relationships between multiple images, those relationships can be described.

[0043] Examples of relationships between multiple images include the relationship between a derived image item and its constituent image items, the relationship between a main image and a thumbnail image, and so on.

[0044] Similarly, the relationship between image items and metadata items can be described.

[0045] The HEIF format is based on the ISOBMFF format, so we will first provide a brief explanation of ISOBMFF.

[0046] The ISOBMFF format manages data in a structure called a box.

[0047] A box is a data structure that begins with a 4-byte data length field and a 4-byte data type field, followed by data of arbitrary length.

[0048] The structure of the data section is determined by the data type. The ISOBMFF specification and the HEIF specification specify several data types and the structures of their data sections.

[0049] A box may also contain other boxes in its data. In other words, boxes can be nested. A box nested in the data part of a box is called a subbox.

[0050] A box that is not a subbox is called a file-level box, which is a box that can be accessed sequentially from the beginning of the file.

[0051] We will use Figure 2 to explain HEIF format files.

[0052] First, file-level boxes are described.

[0053] The file type box, whose data type is 'ftyp', stores information about file compatibility. A file specification conforming to ISOBMFF declares the file structure and data stored in the file in a 4-byte code called a brand, and stores them in the file type box. By placing a file type box at the beginning of a file, a file reader can find out the file structure by checking the contents of the file type box, without having to further read and interpret the file contents.

[0054] In the HEIF specification, the file structure is represented by the brand 'mif1'. If the encoded image data is HEVC compressed data, it is represented by the brand 'heic' or 'heix' according to the HEVC compression profile.

[0055] Metadata boxes, with a data type of 'meta', contain various sub-boxes that store data about each item, as detailed below.

[0056] The media data box with data type 'mdat' stores data for each item, for example, coded image data for coded image items and Exif data for metadata items.

[0057] Next, the sub-boxes of the metadata box will be described.

[0058] A handler box with a data type of 'hdlr' stores information that indicates the type of data that the metadata box manages. In the HEIF specification, the handler_type of the handler box is 'pict'.

[0059] A data information box whose data type is 'dinf' specifies the file in which the data targeted by this file exists. In ISOBMFF, it is possible for the data targeted by a file to be stored in a file other than the file itself. In that case, the data reference in the data information box stores a data entry URL box that describes the URL of the file in which the data exists. If the targeted data exists in the same file, a data entry URL box that only has a flag indicating this is stored.

[0060] The primary item box, whose data type is 'pitm', stores the item ID of the image item that represents the main image.

[0061] The item information box, whose data type is 'iinf', is a box for storing the following item information entries.

[0062] An item information entry with a data type of 'infe' stores the item ID, item type, and flags for each item.

[0063] The item type of an image item whose encoded image data is HEVC compressed data is 'hvc1', and the image item is an image item.

[0064] The item type of a derived image item of overlay type, that is, the ImageOverlay structure (overlay information), is 'iovl' and is classified as an image item.

[0065] The item type of an Exif format metadata item is 'Exif' and is a metadata item.

[0066] In addition, when the least significant bit of the flag field of an image item is set, it is possible to specify that the image item is a hidden image. When this flag is set, the image item is not treated as a display target during playback and is hidden.

[0067] An item reference box with a data type of 'iref' stores the reference relationship between items, including the type of reference, the item ID of the referencing item, and the item IDs of one or more referenced items.

[0068] In the case of a derived image item in overlay format, the reference type is set to 'dimg', the item ID of the derived image item is set to the item ID of the referencing item, and the item IDs of each image item that makes up the overlay are set to the item ID of the referenced item.

[0069] In the case of thumbnail images, the reference type is set to 'thmb', the item ID of the thumbnail image is set to the item ID of the referencing item, and the item ID of the original image is set to the item ID of the referenced item.

[0070] An item property box with a data type of 'iprp' is a box for storing the following item property container box and item property association boxes.

[0071] An item property container box, whose data type is 'ipco', is a box that contains individual property data boxes.

[0072] Each image item can have property data that represents the characteristics and attributes of the image.

[0073] The property data box includes the following:

[0074] Decoder configuration and initialization data (for HEVC, the type is 'hvcC') is data used to initialize the decoder. HEVC parameter set data (VideoParameterSet, SequenceParameterSet, PictureParameterSet) is stored here.

[0075] Image spatial extents (type 'ispe') are the dimensions of the image (width, height).

[0076] Colour information (type 'colr') is the colour space information of an image.

[0077] Image rotation information (type 'irot') specifies the direction of rotation when the image is rotated for display.

[0078] The pixel information of an image (type 'pixi') is information that indicates the number of bits of data that make up the image.

[0079] In addition, in this embodiment, as HDR metadata, mastering display colour volume information (type 'MDCV') and contents light level information (type 'CLLI') are stored as property data boxes.

[0080] There are other properties besides those mentioned above, but they are not listed here.

[0081] The item property association box, of data type 'ipma', stores each item-property association in the form of an item ID and an array of indexes in 'ipco' of the associated properties.

[0082] An item data box whose data type is 'idat' is a box that stores item data whose data size is small.

[0083] The data of the derived image item in the overlay format can store the ImageOverlay structure (overlay information) that stores the position information of the constituent images in 'idat'. The overlay information includes the canvas_fill_value parameter, which is background color information, and the output_width and output_height parameters, which are the final playback image size after overlay synthesis. In addition, for each image that is a synthesis element, there are horizontal_offset and vertical_offset parameters that indicate the horizontal and vertical position coordinates of the placement position in the synthesis image. By using these parameters included in the overlay information, a synthesis image can be reproduced in which multiple images are placed at any position within a single image with a specified background color and size. The item location box, whose data type is 'iloc', stores the position information of the data of each item in the form of an offset reference (construction_method), an offset value from the offset reference, and a length.

[0084] The offset reference is the beginning of the file, or 'idat'.

[0085] There are other metadata boxes in addition to those mentioned above, but they will not be mentioned here.

[0086] Figure 2 illustrates a structure in which two encoded image items constitute an overlay-formatted image. The number of encoded image items constituting the overlay-formatted image is not limited to two. When the number of encoded image items constituting the overlay-formatted image increases, the following boxes and items increase accordingly. · An item information entry of the encoded image item is added to the item information box. · The item ID of the encoded image item increases in the destination item ID of the reference relation type 'dimg' in the item reference box. · In the item property container box, decoder configuration, initialization data, image spatial range, etc. · An item of the index of the encoded image item and its related properties is added to the item property related box. · An item of the position information of the encoded image item is added to the item location box. · Encoded image data is added to the media data box.

[0087] <Shooting of HDR Image> Next, the processing in the imaging device 100 when shooting and recording an HDR image will be described.

[0088] The width and height of the image to be recorded have become larger in recent years, but if the image is encoded at a size that is too large, compatibility may be lost when the image is decoded on other models, or the system size may become large. Specifically, in this embodiment, an example in which H.265 is used for encoding and decoding will be described. H.265 has parameters such as Profile and Level as a standard, and these parameters change when the image encoding method or image size increases. These values ​​are parameters that determine whether the device can play the image at the time of decoding, and there are cases where the device judges these parameters and rejects the image as not playable. In this embodiment, when a large image size is recorded, the image is divided into multiple image display areas, and the overlay method touted in the HEIF standard is used, and the encoded data is stored and managed in a HEIF file. By dividing the image into multiple image display areas in this way, the size of each image display area is reduced, and playback compatibility with other models is improved. In the following, the size of one side of the encoding area is set to 4096 or less for explanation. The encoding format may be other than H.265, and the size of one side may be set to a size other than 4096.

[0089] Next, alignment constraints on the start position of the coding area, the width and height of the coding area, and the start position of the playback area, the width and height of the playback area will be described. When encoding a divided image, an imaging device generally has hardware constraints such as vertical and horizontal coding start alignment of the coding area and playback width and height alignment during playback. In other words, it is necessary to calculate and encode the coding start position of one or more coding areas, the start position of the playback area, and the width and height. Since the imaging device switches between multiple image sizes for recording according to instructions from the user, it is necessary to switch between them for each image size.

[0090] The encoding processing unit 114 described in this embodiment has alignment constraints on the start position in the horizontal direction (hereinafter, x direction) of 64 pixels and in the vertical direction (hereinafter, y direction) of 1 pixel during encoding. In addition, the alignment constraints on the width and height to be encoded are 32 pixel units and 16 pixel units, respectively. The start position of the playback area is aligned in 2 pixel units in the x direction and 1 pixel units in the y direction. In addition, the width and height of the playback area are aligned in 2 pixel units and 1 pixel units, respectively. Note that although examples of alignment constraints have been described here, other alignment constraints may generally be used.

[0091] Next, the flow from capturing HDR image data in HDR recording mode (HDR shooting mode) to recording the captured HDR image data as a HEIF file will be described with reference to Fig. 3. In HDR mode, the imaging unit 112 and image processing unit 113 perform processing to express the HDR color space using the BT.2020 color gamut indicating HDR and a PQ gamma curve. Recording for HDR will be omitted in this embodiment.

[0092] In S101, CPU 101 determines whether or not the user has pressed SW2. When the user captures a subject and presses SW2 (YES in S101), CPU 101 detects that SW2 has been pressed and transitions to S102. CPU 101 continues this detection until SW2 is pressed (NO in S101).

[0093] In S102, CPU 101 determines how many parts to divide the coding area vertically and horizontally (number of divisions N) and how to divide it in order to save the image with the angle of view captured by the user in a HEIF file. The result is stored in memory 102. S102 will be described later with reference to Figure 4. Proceed to S103.

[0094] In S103, the CPU 101 initializes a variable M in the memory 102 to M=1. Then, the process proceeds to S104.

[0095] In S104, the CPU 101 monitors whether the loop has been performed the number of times equal to the division number N determined in S102. If the number of loops has reached the division number N (YES in S104), the process proceeds to S109, and if not (NO in S104), the process proceeds to S105.

[0096] In S105, the CPU 101 reads out information on the coding start area and the playback area of ​​the divided image M from the memory 102. The process proceeds to S106.

[0097] In S106, the CPU 101 informs the encoding processing unit 114 which area of ​​the entire image arranged in the memory 102 is to be encoded, and causes the encoding processing unit 114 to encode the area.The process proceeds to S107.

[0098] In S107, the CPU 101 temporarily stores in the memory 102 the coded data resulting from the coding performed by the coding processing unit 114 and associated information generated during coding. In the specific example of H.265, this refers to H.265 standard information such as VPS, SPS, and PPS that must be stored in the HEIF file later. Although not directly related to this embodiment and therefore not described here, the VPS, SPS, and PPS of H.265 are various pieces of information necessary for decoding, such as the coded data size, bit depth, designation of display / non-display areas, and frame information. The process then proceeds to S108.

[0099] In S108, the CPU 101 increments the variable M and returns to S104.

[0100] In S109, the CPU 101 generates metadata to be stored in the HEIF file. The CPU 101 extracts information required for playback from the imaging unit 112, image processing unit 113, encoding processing unit 114, etc., and temporarily stores it in the memory 102 as metadata. Specifically, this is data formatted in Exif format. Details of Exif data are already common, so a description of them will be omitted in this embodiment. Proceed to S110.

[0101] In S110, the CPU 101 encodes the thumbnail. The image processing unit 113 reduces the entire image in the memory 102 to the thumbnail size and temporarily places it in the memory 102. The CPU 101 instructs the encoding processing unit 114 to encode this thumbnail image in the memory 102. Here, the thumbnail is encoded at a size that satisfies the alignment constraint described above, but a method in which the encoding area and the playback area are different may be used. This thumbnail is also encoded in H.265. Proceed to S111.

[0102] In S111, the CPU 101 temporarily stores in the memory 102 the coded data resulting from the coding of the thumbnail by the coding processing unit 114 and associated information generated during coding. As in the explanation of S107 above, this refers to H.265 standard information such as VPS, SPS, PPS, etc. Next, the process proceeds to S112.

[0103] In S112, in the steps up to this point, the CPU 101 has stored various data in the memory 102. The CPU 101 sequentially combines the data stored in this memory 102 to complete a HEIF file, and stores it in the memory 102. The flow for completing this HEIF file will be explained later using Figure 6. Proceed to S113.

[0104] In S113, the CPU 101 instructs the recording medium control unit 119 to write the HEIF file in the memory 102 to the recording medium 120. Then, the process returns to S101.

[0105] After shooting is performed in these steps, the HEIF file is recorded on the recording medium 120.

[0106] Next, the division method decision in S102 explained above will be explained using Figures 4 and 5(a). In the following, the upper left corner of the sensor image area is taken as the origin (0, 0), the coordinates are expressed as (x, y), and the width and height are expressed as w and h. The area is expressed as [x, y, w, h], combining the starting coordinates (x, y) and the width and height w and h.

[0107] In S121, the CPU 101 obtains the recording mode for shooting from the memory 102. The recording mode is a recording method that determines, for example, the image size, image aspect ratio, image compression rate, etc., and the user selects one of these recording modes to shoot. For example, this is a setting such as L size or 3:2 aspect. Since the shooting angle of view differs depending on the recording mode, the shooting angle of view is determined. Proceed to S122.

[0108] In S122, the start coordinate position (x0, y0) of the playback image is obtained from within the sensor image area [0, 0, H, V]. Next, the process proceeds to S123.

[0109] In S123, the area [x0, y0, Hp, Vp] of the reproduced image is obtained from the memory 102. Next, the process proceeds to S124.

[0110] In S124, it is necessary to encode so as to include the area of ​​the reproduced image. As mentioned earlier, the alignment of the coding start position is 64 pixels in the x direction, so if encoding is to be done so as to include the area of ​​the reproduced image, encoding must start from the position (Ex0, Ey0). In this way, the size of Lex is found. If the offset of this Lex is 0 (NO in S124), proceed to S126, and if necessary (YES in S124), proceed to S125.

[0111] In S125, Lex is calculated. As described above, Lex is determined by the alignment of the playback image area and the encoding start position. CPU 101 determines Lex as follows: x0 is divided by 64 pixels, which is the pixel alignment of the encoding start position in the horizontal direction, and the quotient is multiplied by the previous 64 pixels. Since the calculation does not include the remainder, the coordinates obtained are located to the left of the start position of the playback image area. Therefore, Lex is calculated as the difference between the x coordinate of the playback image area and the x coordinate of the encoding start position calculated above. Proceed to S126.

[0112] In S126, the coordinates of the final position of the right end of the top line of the reproduced image are calculated. These are set as (xN, y0). xN is calculated by adding Hp to x0. Then, the process proceeds to S127.

[0113] In S127, the right edge of the coding area is found so as to include the right edge of the reproduced image. To find the right edge w of the coding area, an alignment constraint on the coding width is necessary. As mentioned earlier, the coding width and height are each restricted to multiples of 32 pixels. From this constraint and the relationship with (xN, y0), CPU 101 calculates whether or not a Rex offset is required beyond the right edge of the reproduced image. If Rex is required (YES in S127), proceed to S128; if not (NO in S127), proceed to S129.

[0114] In S128, the CPU 101 adds Rex to the right of the right edge coordinates (xN, y0) of the reproduced image so as to align with 32 pixels, and obtains the end position (ExN, Ey0) of the encoding.The process proceeds to S129.

[0115] In S129, the CPU 101 obtains the coordinates of the bottom right corner of the display area. Since the size of the reproduced image is self-evident, these coordinates are (xN, yN). The process proceeds to S130.

[0116] In S130, the bottom edge of the coding area is found so that it includes the bottom edge of the reproduced image. To find the bottom edge of the coding area, an alignment constraint on the coding height is necessary. As mentioned earlier, the coding height is restricted to a multiple of 16 pixels. From the relationship between this constraint and the bottom right coordinates (xN, yN) of the reproduced image, the CPU 101 calculates whether a Vex offset is required beyond the bottom edge of the reproduced image. If Vex is required (YES in S130), proceed to S131; if not (NO in S130), proceed to S132.

[0117] In S131, the CPU 101 calculates Vex based on the relationship between the above constraint and the bottom right coordinate (xN, yN) of the reproduced image, and obtains the bottom right coordinate (ExN, EyN) of the encoding area. Vex is calculated by setting it to a position that is a multiple of 16 pixels, which is the encoding height alignment, so that it includes yN from y0, which is the vertical encoding start position. Specifically, assuming that y0=0, Vp is divided by 16 pixels, which is the encoding height alignment, and the quotient and remainder are obtained. Since the encoding area must be taken so as to include the reproduced area, if there is a remainder, 1 is added to the quotient and multiplied by the previous 16 pixels. This obtains EyN, which is the y coordinate of the bottom end of the encoding area that includes Vp, even though it is a multiple of 16 pixels. Vex is obtained by the difference between EyN and yN. Note that EyN is a value obtained by offsetting yN downward by Vex, and ExN is a value already obtained in S128. Proceed to S132.

[0118] In S132, CPU 101 calculates the size of the coding area, Hp'xVp', as follows: Hp' is calculated by adding Lex and Rex calculated from the alignment constraint to the horizontal size Hp. Vp' is also calculated by adding Vex calculated from the alignment constraint to the vertical size Vp. The size of the coding area, Hp'xVp', is calculated from Hp' and Vp'. The process proceeds to S133.

[0119] In S133, the CPU 101 determines whether the horizontal size Hp' to be encoded exceeds 4096 pixels to be divided, and if it does (YES in S133), the process proceeds to S134, and if it does not (NO in S133), the process proceeds to S135.

[0120] In S134, the horizontal size Hp' to be coded is divided into two or more regions so that the size is 4096 pixels or less. For example, if it is 6000 pixels, it is divided into two, and if it is 9000 pixels, it is divided into three.

[0121] When dividing into two, divide it at an approximate center position that satisfies the previous 32 pixel alignment for the encoding width. If the divisions are not even, make the left divided image larger. When dividing into three, if the divisions are not even, divide the divided areas into A, B, and C while paying attention to the previous alignment for the encoding width, so that divided areas A and B are equal in size, and divided area C is slightly smaller than A. Even when dividing into three or more parts, the division positions and sizes are determined using the same algorithm. Proceed to S135.

[0122] In S135, the CPU 101 determines whether the vertical size Vp' to be encoded exceeds 4096 pixels into which to divide, and if it does (YES in S135), proceeds to S136, and if it does not (NO in S135), ends the process.

[0123] In S136, Vp' is divided into a plurality of divided regions in the same manner as in S134, and the process ends.

[0124] An example of the division in S102 will be described with specific numerical values ​​using Figures 5(b), (c), (d), and (e). Figure 5 shows not only the division method of the image data, but also the relationship between the coding area and the reproduction area.

[0125] Let us now consider FIG. 5(b). FIG. 5(b) shows a case where an image is divided into two regions, left and right, and Lex, Rex, and Vex are assigned to the left, right, and bottom edges of the playback region. First, the playback image region is [256+48, 0, 4864, 3242], which is the region of the final image to be recorded by the imaging device. The coordinates of the upper left corner of this playback image region are (256+48, 0), which is not a multiple of 64 pixels, which is the start alignment of the encoding mentioned above. Therefore, Lex is set 48 pixels to the left of this, and the left edge of the encoding start position is at x=256. Next, the width of the playback region is w=48+4864, which is not a multiple of 32 pixels, which is the alignment of the width of the encoding, and 16 pixels of Rex are assigned to the right edge of the playback image region in consideration of the width constraint alignment during encoding. Similarly, when calculating the vertical direction, Vex=4 pixels. Because the horizontal size of this reconstructed image exceeds 4096, it is divided horizontally into two approximately at the center, taking into account the alignment of the encoding width. As a result, the encoding area of ​​divided image 1 is [256, 0, 2496, 3242+8], and the encoding image area of ​​divided image 2 is [256+2496, 0, 2432, 3242+8].

[0126] Now let us consider Figure 5(c). Figure 5(c) shows a case where an image is divided into two regions, left and right, and Rex and Vex are assigned to the right and bottom of the playback region. Since the x position of the playback image region is x=0, it coincides with the 64 pixels which is the start alignment of the encoding region, so Lex=0. After this, the calculation method is the same as that described above, and as a result, the encoding region of divided image 1 is calculated as [0, 0, 2240, 32456+8], and the encoding image region of divided image 2 is calculated as [2240, 0, 2144, 2456+8].

[0127] Now let us consider Figure 5(d). Figure 5(d) shows a case where an image is divided into two regions, left and right, and Lex and Rex are added to the left and right of the playback region. The vertical size of the playback image region is h=3648, which is a multiple of 16 pixels, which is the image height constraint during encoding, so Vex=0. The rest of the calculation method is the same as that described above, and as a result, the coding region of divided image 1 is calculated as [256+48, 0, 2240, 3648], and the coding image region of divided image 2 is calculated as [256+2240, 0, 2144, 3648].

[0128] FIG. 5(d) will be explained. FIG. 5(d) shows a case where an image is divided into four regions vertically and horizontally, and Lex and Rex are added to the left and right of the reproduction region. In this case, the coding region exceeds 4096 in both the vertical and horizontal directions, so the region is divided into a square shape. Thereafter, the calculation method is the same as that described above, and as a result, the coding region of divided image 1 is [0, 0, 4032, 3008], the coding image region of divided image 2 is [4032, 0, 4032, 3008], the coding image region of divided image 3 is [0, 3008, 4032, 2902], and the coding image region of divided image 4 is [4032, 3008, 4032, 2902].

[0129] In this way, the CPU 101 must determine the area of ​​the reproduced image according to the recording mode and calculate the divided areas for each. As an alternative method, these calculations may be performed in advance and the results may be stored in the memory 102. The CPU 101 may read information on the divided areas and the reproduced area according to the recording mode from the memory 102 and set it in the encoding processing unit 114.

[0130] When a captured image is divided into a plurality of divided images and recorded as in this embodiment, in order to restore the captured image from the recorded image, it is necessary to synthesize a plurality of display areas. When generating a composite image, it is more efficient if there are no overlapping areas in each divided image, since there is no need to encode or record extra data. Therefore, in this embodiment, each playback area and each encoding area are set so that there are no overlapping areas (overlapping areas) at the boundaries where the divided images meet. However, depending on conditions such as hardware alignment constraints, overlapping areas may be provided between the divided images.

[0131] Next, the method of constructing a HEIF file mentioned above in S112 will be described with reference to Fig. 6. Since a HEIF file has the structure shown in Fig. 2, the CPU 101 constructs the file in memory 102 in order from the beginning.

[0132] In S141, an 'ftyp' box is generated. This box is used to know the compatibility of the file, and is placed at the beginning of the file. CPU 101 stores 'heic' in the major brand of this box, and 'mif1' and 'heic' in the compatible brands. In this embodiment, these values ​​are used, but other values ​​may be used. Proceed to S142.

[0133] In S142, a 'meta' box is generated. This box is a box for storing the multiple boxes described below. After generating this box, the CPU 101 proceeds to S143.

[0134] In S143, an 'hdlr' box is generated to be stored in the 'iinf' box. This box indicates the attributes of the 'meta' box described above. The CPU 101 stores 'pict' in this type and proceeds to S144. Note that 'pict' is information indicating the type of data managed by the metadata box, and in the HEIF specifications, the handler_type of the handler box is defined as 'pict'.

[0135] In S144, a 'dinf' box is generated to be stored in the 'iinf' box. This box indicates the location of the data that is the subject of this file. After generating this box, CPU 101 proceeds to S145.

[0136] In S145, a 'pitm' box is generated to be stored in the 'iinf' box. This box stores the item ID of the image item representing the main image. Since the main image is an overlay image composited with an overlay, CPU 101 stores the overlay information item ID as the item ID of the main image. Proceed to S146.

[0137] In S146, an 'iinf' box is generated to be stored in the 'meta' box. This box is used to manage a list of items. Here, an initial value is entered into the data length field, 'iinf' is saved in the data type field, and the process proceeds to S147.

[0138] In S147, an 'infe' box is generated to be stored in the 'iinf' box. This 'infe' box is a box for registering item information for each item stored in the file, and an 'infe' box is generated for each item. For divided images, each divided image is registered in this box as one item. In addition, overlay information for constructing a main image from multiple divided images, Exif information, and thumbnail images are each registered as items. At this time, as described above, the overlay information, divided images, and thumbnail images are registered as image items. For image items, it is possible to add a flag indicating a hidden image, and by adding this flag, the image will not be displayed during playback. In other words, by adding this hidden image flag to an image item, it is possible to specify an image to be hidden during playback. Therefore, for divided images, a flag indicating a hidden image is set, and the flag is not set for the overlay information item. As a result, each divided image itself is not displayed, and only the image resulting from the overlay synthesis of multiple divided images using the overlay information is displayed. Create an 'infe' box for each item individually and store them in the previous 'iinf'. Proceed to S148.

[0139] In S148, an 'iref' box is generated to be stored in the 'iinf' box. This box stores information indicating the relationship between the image (main image) configured as an overlay and the divided images that configure that image. Since the division method was determined in S102 described above, CPU 101 generates this box based on the determined division method. Proceed to S149.

[0140] In S149, an 'iprp' box is generated to be stored in the 'iinf' box. This box is a box for storing item properties, and is a box for storing the 'ipco' box generated in S150 and the 'ipma' box generated in S151. Proceed to S150.

[0141] In S150, an 'ipco' box is generated to be stored in the 'iprp' box. This box is a property container box for an item, and stores various properties. There are multiple properties, and CPU 101 generates the following property container box and stores them in the 'ipco' box. Some properties are generated for each image item, and some are generated commonly to multiple image items. A 'colr' box is generated as information common to the overlay image composed of overlay information that becomes the main image, and the divided images. In the 'colr' box, color information such as HDR gamma curve is stored as color space information of the main image (overlay image) and the divided images. For the divided images, an 'hvcC' box and an 'ispe' box are generated, respectively. In S107 and S111 described above, CPU 101 reads the associated information at the time of encoding that is stored in memory 102, and generates a property container box for 'hvcC'. The associated information stored in this 'hvcC' includes not only information when the coding area was coded and the size (width, height) of the coding area, but also the size (width, height) of the playback area and the position information of the playback area in the coding area. These associated information are Golomb compressed and recorded in the property container box of 'hvcC'. Golomb compression is a common compression method, so a description is omitted here. In addition, the 'ispe' box stores size (width, height) information of the playback area of ​​the divided image (for example, 2240x2450, etc.). An 'irot' box that stores information indicating the rotation of the overlay image that will be the main image, and a 'pixi' box that indicates the number of bits of the image data are generated. The 'pixi' box may be generated separately for the main image (overlay image) and the divided image, but in this embodiment, the overlay image and the divided image have the same number of bits, i.e., 10 bits, so only one is generated. In addition, 'CLLI' and 'MDCV' boxes in which supplementary information for HDR is stored are also generated.Furthermore, as properties of the thumbnail image, a 'colr' box storing color space information, an 'hvcC' box storing encoding information, an 'ispe' box storing image size, a 'pixi' box storing information on the number of bits of image data, and a 'CLLI' box storing supplementary information on HDR are generated separately from the main image (overlay image). These property container boxes are generated and stored in the 'ipco' box. Proceed to S151.

[0142] In S151, an 'ipma' box is generated. This box indicates the relationship between items and properties, and indicates which of the above-mentioned properties each item is related to. The CPU 101 determines the relationship between items and properties from various data stored in the memory 102, and generates this box. The process proceeds to S152.

[0143] In S152, an 'idat' box is generated. This box stores overlay information indicating how to arrange the playback areas of the divided images to generate an overlay image. The overlay information includes a canvas_fill_value parameter that is background color information, and output_width and output_height parameters that are the size of the entire overlay image. In addition, for each divided image that is a composition element, there are horizontal_offset and vertical_offset parameters that indicate the horizontal and vertical position coordinates for combining the divided images. CPU 101 writes this information into these parameters based on the division method determined in S102. Specifically, the size of the entire playback area is written in the output_width and output_height parameters that indicate the size of the overlay image. Then, in the horizontal_offset and vertical_offset parameters that indicate the position information of each divided image, the offset values ​​in the width direction and height direction from the start coordinate position (x0, y0) of the upper left corner of the playback area to the upper left corner position of the divided image are written, respectively. By generating the overlay information in this manner, when playing back an image, the divided images are arranged based on the horizontal_offset and vertical_offset parameters and then combined, making it possible to play back the image in its pre-split state. In this embodiment, since no overlapping areas are provided in the divided images, position information is written so that the divided images are arranged so that they do not overlap. Then, by specifying the area to be displayed by the output_width and output_height parameters out of the image obtained by combining the divided images, it is possible to play back the image with only the playback area as the display target. An 'idat' box is generated in which the overlay information generated in this manner is stored. Proceed to S153.

[0144] In S153, an 'iloc' box is generated. This box indicates where in the file various data are to be placed. As various information is stored in memory 102, this box is generated from the size of this information. Specifically, information indicating the overlay is stored in the aforementioned 'idat', and this is position and size information within this 'idat'. Also, thumbnail data and code data 12 are stored in the 'mdat' box, and this position and size information is stored. Proceed to S154.

[0145] In S154, an 'mdat' box is generated. This box contains multiple boxes described below. After generating the 'mdat' box, the CPU 101 proceeds to S155.

[0146] In S155, the Exif data is stored in the 'mdat' box. Since the Exif metadata was saved in the memory 102 in the previous S109, the CPU 101 reads it out of the memory 102 and adds it to the 'mdat' box. Then the process proceeds to S156.

[0147] In S156, the thumbnail data is stored in the 'mdat' box. Since the thumbnail data was saved in the memory 102 in the previous S110, the CPU 101 reads it out of the memory 102 and adds it to the 'mdat' box. Then, the process proceeds to S157.

[0148] In S157, the data of encoded image 1 is stored in the 'mdat' box. Since the data of divided image 1 was saved in memory 102 in the first loop in S106 above, CPU 101 reads it from memory 102 and adds it to the 'mdat' box. This is repeated up to encoded image N, and all encoded images 1-N are added to the 'mdat' box. Through these steps, CPU 101 builds a HEIF file.

[0149] Through the above processing, an HDR image can be recorded on the recording medium 120 in the HEIF format as shown in FIG.

[0150] <Playback of HDR Images> Next, the processing in the imaging device 100 when playing back an HDR image file recorded in the HEIF format on the recording medium 120 will be described. In this embodiment, the case of playing back on the imaging device 100 will be described. However, in an image processing device without an imaging unit, when playing back an HDR image recorded in the HEIF format recorded on the recording medium 120, similar processing may be realized.

[0151] Using FIG. 7, the playback (display) processing of HEIF when the main image is an overlay image will be described.

[0152] In step S701, the CPU 101 uses the recording medium control unit 119 to read the head portion of the specified file existing on the recording medium 120 into the memory 102. Then, it checks whether a file type box with a correct structure exists in the head portion of the read file, and further checks whether'mif1' representing HEIF exists in the brand therein.

[0153] Also, when a brand corresponding to a unique file structure is recorded, the existence of that brand is checked. As long as that brand guarantees a specific structure, this brand check can omit some subsequent structure checks, for example, step S703 and step S704.

[0154] In step S702, the CPU 101 reads the metadata box of the specified file from the recording medium 120 into the memory 102.

[0155] In step S703, the CPU 101 searches for a handler box from the metadata box read in step S702 and checks the structure. In the case of HEIF, the handler type must be 'pict'.

[0156] In step S704, CPU 101 searches for a data information box from the metadata box read in step S702 and checks the structure. In this embodiment, since it is assumed that data exists in the same file, it is checked whether a flag indicating this is set in the data entry URL box.

[0157] In step S705, the CPU 101 searches for a primary item box from the metadata box read in step S702, and acquires the item ID of the main image.

[0158] In step S706, CPU 101 searches for an item information box from the metadata box read in step S702, and acquires an item information entry corresponding to the item ID of the main image acquired in step S705. In the case of an HEIF format image file recorded by the above-mentioned HDR image shooting process, overlay information is specified as the main image. In other words, the item information entry corresponding to the item ID of the main image has an item type of 'iovl', which indicates overlay.

[0159] In step S707, the CPU 101 executes an overlay image creation process, which will be described later with reference to FIG.

[0160] In step S708, the CPU 101 displays the overlay image created in step S707 on the display unit 116 via the display control unit 115.

[0161] When displaying the created overlay image, it may be necessary to specify the color space information of the image. The color space information of the image is specified by the color_primaries of the color space property 'colr', where the color gamut is specified, and the transfer characteristics (equivalent to gamma) are specified by transfer_characteristics, so these values ​​are used. For example, an HDR image might have a color gamut of Rec.ITU-R BT.2020 and a transfer characteristic of Rec.ITU-R BT.2100(PQ).

[0162] Also, if HDR metadata such as 'MDCV' and 'CLLI' exists in the item property container box, it is possible to use it.

[0163] Next, the overlay image creation process will be described with reference to FIG.

[0164] In step S801, the CPU 101 executes a process for acquiring properties of an overlay image, which will be described later with reference to FIG.

[0165] In step S802, CPU 101 executes an overlay information acquisition process for the overlay image. Since the overlay information is recorded in the 'idat' box, the data stored in the 'idat' box is acquired.

[0166] The overlay information includes the image size of the overlay image (output_width, output_height parameters), position information (horizontal_offset, vertical_offset) of each image item (divided image item) that constitutes the overlay image, and the like.

[0167] Overlay information is treated as image item data for an image item whose item type is overlay 'iovl'.

[0168] This acquisition process will be described later with reference to FIG.

[0169] In step S803, the CPU 101 executes a process for obtaining item IDs of all divided image items, which will be described later with reference to FIG.

[0170] In step S804, CPU 101 uses the overlay image size in the overlay information acquired in step S802 to secure memory for storing image data of that size. This memory area will be called a canvas.

[0171] In step S805, the CPU 101 initializes a divided image counter n to 0.

[0172] In step S806, the CPU 101 checks whether the divided image counter n is equal to the number of divided image items. If it is equal, the process proceeds to step S2210. If it is not equal, the process proceeds to step S807.

[0173] In step S807, the CPU 101 executes image creation processing for one image item for the n-th divided image item, as will be described later with reference to FIG.

[0174] In step S808, CPU 101 places the image (divided image) of the n-th divided image item created in step S807 on the canvas secured in step S804, in accordance with the overlay information acquired in step S802.

[0175] In an overlay, the divided images that make up the overlay can be placed at any position on the canvas, and the position is described in the overlay information. However, the area outside the overlay image is not displayed. Therefore, when placing a divided image on the canvas, only the area of ​​the coding area of ​​the divided image that overlaps with the area of ​​the overlay image, i.e., the playback area of ​​the divided image, is placed.

[0176] The following description will be given with reference to FIG.

[0177] The coordinate system has the upper left corner of the overlay image as the origin (0,0), with the X coordinate increasing to the right and the Y coordinate increasing downward.

[0178] Let the size of the overlay image be width Wo and height Ho (0 < Wo, 0 < Ho).

[0179] Therefore, the overlay image has its upper left coordinate at (0, 0) and its lower right coordinate at (Wo - 1, Ho - 1).

[0180] Also, let the size of the divided image be width Wn and height Hn (0 < Wn, 0 < Hn).

[0181] Let the upper left position of the divided image be (Xn, Yn).

[0182] Therefore, the divided image has its upper left coordinate at (Xn, Yn) and its lower right coordinate at (Xn + Wn - 1, Yn + Hn - 1).

[0183] The overlapping area between the divided image and the overlay image (canvas) can be obtained by the following method.

[0184] In the following cases, there is no overlapping area. Wo - 1 < Xn (the left end of the divided image is to the right of the right end of the overlay image) Xn + Wn - 1 < 0 (the right end of the divided image is to the left of the left end of the overlay image) Ho - 1 < Yn (the upper end of the divided image is below the lower end of the overlay image) Yn + Hn - 1 < 0 (the lower end of the divided image is above the upper end of the overlay image) In this case, this image is not a processing target.

[0185] Also, in the following case, the entire divided image becomes the overlapping area. 0 <= Xn and (Xn + Wn - 1) <= (Wo - 1) and 0 <= Yn and (Yn + Hn - 1) <= (Ho - 1) In this case, place the entire divided image at the specified position (Xn, Yn) on the canvas.

[0186] In cases other than the above, a part of the divided image is the placement target. The upper left coordinate of the overlapping area is (Xl, Yt), and the lower right coordinate is (Xr, Yb). The leftmost Xl is as follows. if(0<=Xn) Xl = Xn; else Xl=0; The right end Xr is as follows. if(Xn+Wn-1<=Wo-1) Xr = Xn + Wn - 1; else Xr=Wo-1; The upper limit Yt is as follows. if(0<=Yn) Yt = Yn; else Yt=0; The lower limit Yb is as follows. if(Yn+Hn-1<=Ho-1) Yb = Yn + Hn - 1; else Yb=Ho-1; The size of this overlapping area is width Xr-Xl+1 and height Yb-Yt+1.

[0187] As described above, the upper left coordinate (Xl, Yt) and the lower right coordinate (Xr, Yb) of this overlapping area are in a coordinate system with the upper left corner of the canvas as the origin (0,0).

[0188] The upper left coordinates (Xl, Yt) of the overlapping area are expressed as follows in a coordinate system with the upper left corner of the divided images as the origin. Xl' = Xl - Xn; Yt' = Yt - Yn;

[0189] To summarize, the following can be said:

[0190] A rectangle with width (Xr-Xl+1) and height (Yb-Yt+1) is cut out from the top left of the divided image at a position (Xl', Yt') away, and placed at position (Xl, Yt) on the canvas.

[0191] This makes it possible to position the playback area of ​​the divided image at an appropriate position on the canvas.

[0192] In step S809, the CPU 101 increments the divided image counter n by 1, and the process returns to step S806.

[0193] In step S810, CPU 101 checks whether a rotation property 'irot' exists in the properties of the overlay image acquired in step S801, and if it exists, checks the rotation angle. If the rotation property does not exist, or if the rotation property exists but the rotation angle is 0, the process ends. If the rotation property exists and the rotation angle is other than 0, the process proceeds to step S811.

[0194] In step S811, CPU 101 rotates the image of the canvas created in steps S806 to S809 by the angle acquired in step S810, and sets the rotated image as a created image.

[0195] This makes it possible to create an overlay image.

[0196] In the above description, the processes in steps S806 to S809 are repeated in a loop for the number of divided images. However, in an environment where parallel processing is possible, the processes in steps S806 to S809 may be processed in parallel for the number of divided images.

[0197] Next, the property acquisition process for an image item will be described with reference to FIG.

[0198] In step S901, CPU 101 searches for an entry of a specified image item ID from an item property related box in an item property box in a metadata box, and obtains an array of property indexes stored therein.

[0199] In step S902, the CPU 101 initializes an array counter n to zero.

[0200] In step S903, CPU 101 checks whether array counter n is equal to the number of array elements. If they are equal, the process ends. If they are not equal, the process proceeds to step S904.

[0201] In step S904, CPU 101 obtains the property of the index of the nth element in the array from the item property container box in the item property box in the metadata box.

[0202] In step S905, the CPU 101 increments the array counter n by 1, and the process returns to step S903.

[0203] Next, the image item data acquisition process will be described with reference to FIG.

[0204] In step S1001, the CPU 101 searches for an entry of a specified image item ID from an item location box in a metadata box, and obtains an offset reference (construction_method), an offset, and a length.

[0205] In step S1002, CPU 101 checks the offset reference acquired in step S1001. A value of 0 for the offset reference indicates an offset from the beginning of the file, and a value of 1 indicates an offset within the item data box. If the offset reference is 0, the process proceeds to step S1003. If the offset reference is 1, the process proceeds to step S1004.

[0206] In step S1003, CPU 101 reads length bytes from the offset byte position from the beginning of the file into memory 102.

[0207] In step S1004, CPU 101 reads out length bytes from the offset byte position from the beginning of the data portion of the item data box in the metadata box into memory 102.

[0208] Next, the process of acquiring the item ID of the image that constitutes the main image will be described with reference to FIG.

[0209] In step S1101, CPU 101 searches the item reference box in the metadata box for an entry whose reference type is 'ding' and whose referencing item ID is the item ID of the main image.

[0210] In step S1102, CPU 101 acquires the array of reference destination item IDs of the entry acquired in step S1101.

[0211] Next, the image creation process for one encoded image item will be described with reference to FIG.

[0212] In step S1201, CPU 101 acquires the properties of the image item, as described above with reference to FIG.

[0213] In step S1202, CPU 101 obtains the data of the image item, which has already been explained with reference to Fig. 10. The image item data is encoded image data.

[0214] In step S1203, the CPU 101 initializes the decoder using the decoder configuration and initialization data from the properties acquired in step S1201.

[0215] In step S1204, the CPU 101 decodes the encoded image data acquired in step S1202 using a decoder, and acquires the decoded result.

[0216] In step S1205, CPU 101 checks whether pixel format conversion is required.

[0217] If not required, the process ends. If required, the process proceeds to step S1206.

[0218] When the pixel format of the decoder's output data differs from the image pixel format supported by the display device, pixel format conversion is required.

[0219] For example, if the pixel format of the decoder output data is YCbCr (luminance-chrominance) format and the pixel format of the image on the display device is RGB format, pixel format conversion from YCbCr format to RGB format is required. Also, pixel format conversion is required when the same YCbCr format has different bit depths (8 bits, 10 bits, etc.) or chrominance samples (4:2:0, 4:2:2, etc.).

[0220] Note that the coefficients used for conversion from RGB format to YCbCr format are specified in the matrix_coefficients of the color space information property 'colr', so that the inverse of the coefficients can be used to convert from YCbCr format to RGB format.

[0221] In step S1206, the CPU 101 converts the decoder output data acquired in step S1204 into a desired pixel format.

[0222] This allows for the creation of an image of the specified encoded image item.

[0223] Through the above processing, it is possible to play back HDR images recorded on the recording medium 120 in the HEIF format as shown in FIG.

[0224] <Other embodiments> The present invention has been described above based on the embodiments, but the present invention is not limited to these embodiments, and various forms that do not deviate from the gist of the present invention are also included in the present invention.

[0225] The functions of the above-described embodiments may be treated as a control method, and the control method may be executed by an image processing device. A program having the functions of the above-described embodiments may be treated as a control program, and the control program may be executed by a computer included in the image processing device. The control program may be recorded, for example, in a computer-readable storage medium.

[0226] The present invention can be realized by supplying a program that realizes one or more functions of the above-mentioned embodiments to a system or device via a network or a storage medium. Then, one or more processors in the computer of the system or device can read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

Claims

1. An imaging device that records HDR (high dynamic range) image data obtained by shooting, An imaging sensor; an encoding means for encoding HDR image data obtained by capturing an image using the imaging sensor; and a recording control means for dividing the HDR image data of a coding area to be encoded from the HDR image data obtained by capturing an image using the imaging sensor into a plurality of divided HDR image data, encoding each of the plurality of divided HDR image data by the encoding means, and controlling the plurality of encoded divided HDR image data to be recorded on a recording medium in a predetermined recording format.

2. 2. The imaging device according to claim 1, wherein the encoding means compresses and encodes the plurality of divided HDR image data in accordance with a High Efficiency Video Coding (HEVC) standard.

3. 2. The image pickup apparatus according to claim 1, wherein the predetermined recording format is HEIF (High Efficiency Image File Format).

4. 4. The imaging device according to claim 3, wherein the predetermined recording format is a High Efficiency Image File Format (HEIF), and the plurality of divided HDR image data are recorded in an overlay format.

5. 5. The imaging device according to claim 4, wherein the recording control means controls to record position information of each of the plurality of divided HDR image data as overlay information so that, in the overlay format, the plurality of divided HDR image data are arranged in a positional relationship corresponding to the encoding area.

6. The imaging device described in claim 5, characterized in that the recording control means records the overlay information and the plurality of divided HDR image data as image items, records the plurality of divided HDR image data with non-display information indicating that they are non-display image items, and does not add the non-display information to the overlay information.

7. 2. The imaging device according to claim 1, wherein the recording control means controls so that the encoded divided HDR image data is recorded on the recording medium in a predetermined recording format as a single image file.

8. An encoding means for encoding image data obtained by photographing; a recording control means for controlling the recording of the encoded divided image data in a single image file of a predetermined format on a recording medium, the image data being divided into a plurality of divided image data, the plurality of divided image data being encoded by the encoding means, The recording device according to claim 1, wherein the recording control means controls so that overlay information for combining the plurality of divided image data into a state before the division is recorded as an image item in the image file.

9. 9. The recording apparatus according to claim 8, wherein the predetermined recording format is HEIF (High Efficiency Image File Format).

10. A recording device as described in claim 8 or 9, characterized in that the recording control means records the overlay information and the multiple split image data as image items, records the multiple split image data by adding information for hiding images, and does not add information for hiding images to the overlay information.

11. a readout means for dividing HDR image data of a coding area to be coded into a plurality of divided HDR image data, coding each of the plurality of divided HDR image data by the coding means, and reading out the HDR image file from a recording medium on which the plurality of coded divided HDR image data is recorded as an HDR image file in a predetermined recording format; a combining means for combining each of the plurality of divided HDR image data included in the read HDR image file; and display control means for combining the plurality of HDR image data decoded by the decoding means and controlling the combination to be displayed on a display means as one HDR image data.

Citation Information

Patent Citations

  • Image pickup device

    JP2013141074A

  • Reception device and reception method

    JP2017073761A

  • Information processing device and method

    WO2015008775A1

  • Information processing device, information recording medium, information processing method, and program

    WO2016103968A1

  • Image processing device, reproduction device, image processing method, and reproduction method

    WO2018021261A1