Image processing device and image processing method

Abstract

The present feature pertains to an image processing device and an image processing method with which it is possible to obtain a tile image having an appropriate image size. An image processing unit sets the image size of a tile image in accordance with the intended image size or intended bit rate of a tile image for forming a grid image, the tile image being stored in a high-efficiency image file format (HEIF) file, and generates a tile image having the set image size. The present feature can be applied to, for example, a digital camera, etc., that generates a HEIF file in which a tile image is stored.

Description

[Technical field]

[0001] The present technology relates to an image processing device and an image processing method, and more particularly to an image processing device and an image processing method that make it possible to obtain, for example, a tile image with an appropriate image size. [Background technology]

[0002] High Efficiency Image File Format (HEIF) is a file format for efficiently storing images (see Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] ISO / IEC 23008-12:2017, Information technology -- High efficiency coding and media delivery in heterogeneous environments -- Part 12: Image File Format Summary of the Invention [Problem to be solved by the invention]

[0004] HEIF defines grid as one of the item types. An image (item) with an item type of grid is formed by tiling one or more input images. An image with an item type of grid is also called a grid image, and the input images that form a grid image are also called tile images.

[0005] HEIF specifies that the image sizes of the tile images that form a grid image must be the same, but does not specify what those image sizes should be.

[0006] The present technology has been made in consideration of these circumstances, and makes it possible to obtain tile images with an appropriate image size. [Means for solving the problem]

[0007] A first image processing device of the present technology is an image processing device that includes an image processing unit that sets the image size of a tile image that forms a grid image, and is stored in a High Efficiency Image File Format (HEIF) file, in accordance with a target image size or a target bit rate of the tile image, and generates the tile image of that image size.

[0008] A first image processing method of the present technology is an image processing method that includes setting an image size of a tile image that forms a grid image, which is stored in a High Efficiency Image File Format (HEIF) file, in accordance with a target image size or a target bit rate of the tile image, and generating the tile image of that image size.

[0009] In the first image processing device and image processing method of the present technology, the image size of the tile images forming a grid image, which are stored in a High Efficiency Image File Format (HEIF) file, is set according to a target image size or target bit rate of the tile images, and the tile images of that image size are generated.

[0010] A second image processing device of the present technology is an image processing device that has an image processing unit that sets the image size of tile images that form a grid image and are stored in a High Efficiency Image File Format (HEIF) file in accordance with external information, and generates the tile images of that image size.

[0011] A second image processing method of the present technology is an image processing method that includes setting the image size of tile images that form a grid image and are stored in a High Efficiency Image File Format (HEIF) file in accordance with external information, and generating the tile images of that image size.

[0012] In the second image processing device and image processing method of the present technology, the image size of tile images forming a grid image stored in a HEIF (High Efficiency Image File Format) file is set according to external information, and the tile images of that image size are generated.

[0013] The image processing device may be an independent device or an internal block constituting a single device.

[0014] The image processing device can be realized by causing a computer to execute a program. The program can be provided by being recorded on a recording medium or transmitted via a transmission medium. [Brief description of the drawings]

[0015] [Figure 1] 1 is a block diagram showing an example of the configuration of an embodiment of a digital camera to which the present technology is applied. [Diagram 2] FIG. 1 is a diagram showing an example of a JPEG file format that complies with JPEG (Joint Photographic Experts Group). [Diagram 3] FIG. 1 is a diagram illustrating an example of an ISO base media file format. [Figure 4] FIG. 2 is a diagram showing an example of a format of a HEIF file that complies with HEIF. [Diagram 5] FIG. 1 is a diagram showing an example of the format of a HEIF file in image item format. [Figure 6] FIG. 13 is a diagram showing an example of an iprp box. [Figure 7] FIG. 1 is a diagram showing an example of the format of a HEIF file in image sequence format. [Figure 8] FIG. 1 is a diagram showing an example of a trak box. [Figure 9] FIG. 13 is a diagram showing an example of a collection file in which main images and thumbnail images are stored. [Figure 10] FIG. 13 is a diagram showing an example of a sequence file in which a track of a main image and a track of thumbnail images of the main image are stored. [Figure 11] FIG. 1 is a diagram showing an example of a HEIF file in which tile images are stored. [Figure 12] FIG. 11 is a diagram showing an example of a use case in which the image size or bit rate of a tile image is restricted. [Figure 13] FIG. 13 is a diagram illustrating an example of setting the image size of a tile image in the first use case. [Figure 14] FIG. 13 is a diagram illustrating an example of setting the image size of a tile image in the second use case. [Figure 15] FIG. 13 is a diagram illustrating an example of setting the image size of a tile image in the third use case. [Figure 16] FIG. 11 is a diagram showing a first example of four unedited HEIF files generated by four cameras in the third use case, and a new HEIF file generated from the four unedited HEIF files in editing software. [Figure 17] FIG. 11 is a diagram showing a second example of four unedited HEIF files generated by four cameras in the third use case, and a new HEIF file generated from the four unedited HEIF files in editing software. [Figure 18] FIG. 13 is a diagram showing a third example of four unedited HEIF files generated by four cameras in a third use case, and a new HEIF file generated from the four unedited HEIF files in editing software. [Figure 19]A figure showing a fourth example of four unedited HEIF files generated by four cameras in the third use case and a new HEIF file generated from the four unedited HEIF files in editing software. [Figure 20] FIG. 13 is a diagram illustrating an example of setting the image size of a tile image in the fourth use case. [Figure 21] FIG. 13 is a diagram illustrating an example of setting the image size of a tile image in the fifth use case. [Figure 22] FIG. 13 is a diagram illustrating an example of setting the image size of a tile image in the sixth use case. [Diagram 23] 2 is a block diagram showing an example of the configuration of an encoding control unit 42. FIG. [Figure 24] 11 is a flowchart illustrating an example of processing by the image processing unit 110. [Diagram 25] 13A to 13C are diagrams illustrating an example of an image size setting method for setting the image size of a tile image so that the image size of the tile image is equal to or smaller than a target image size. [Figure 26] 13 is a flowchart illustrating an example of a process for setting the image size of a tile image so that the image size of the tile image is equal to or smaller than a target image size. [Figure 27] 13A to 13C are diagrams illustrating an example of an image size setting method for setting the image size of a tile image so that the bit rate of the tile image is equal to or lower than a target bit rate. [Figure 28] 13A to 13C are diagrams illustrating an example of an image size setting method for setting the image size of a tile image so that the image size and bit rate of the tile image are equal to or smaller than a target image size and target bit rate, respectively. [Figure 29] 1 is a block diagram showing an example of the configuration of an embodiment of a computer to which the present technology is applied. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] <One embodiment of a digital camera to which the present technology is applied>

[0017] FIG. 1 is a block diagram showing an example of the configuration of an embodiment of a digital camera to which the present technology is applied.

[0018] Digital camera 10 has an optical system 11, an image sensor 12, a signal processing unit 13, a media 14, I / Fs (Interfaces) 15 and 16, buttons / keys 17, a touch panel 18, an LCD panel 19, a viewfinder 20, and an I / F 21, etc.

[0019] The optical system 11 collects light from a subject onto the image sensor 12 .

[0020] The image sensor 12 receives light from the optical system 11 and performs photoelectric conversion to capture an image, thereby generating image data as an electrical signal, and supplies the data to a signal processing unit 13 .

[0021] The signal processing unit 13 has an optical system / image sensor control unit 41, an encoding control unit 42, a file control unit 43, a media control unit 44, an operation control unit 45, a display control unit 46, and a UI control unit 47.

[0022] The optical system / image sensor control unit 41 controls the optical system 11 and the image sensor 12, and supplies the image (data) obtained by imaging performed under that control to the encoding control unit .

[0023] The encoding control unit 42 supplies the image from the optical system / image sensor control unit 41 to the display control unit 46, and also encodes the image as necessary and supplies it to the file control unit 43. The encoding control unit 42 also decodes the image supplied from the file control unit 43 as necessary and supplies it to the display control unit 46.

[0024] The file control unit 43 generates a file storing the image supplied from the encoding control unit 42, and supplies the file to the media control unit 44. The file control unit 43 also plays the file supplied from the media control unit 44, i.e., reads data such as images stored in the file. For example, an image read from a file is supplied from the file control unit 43 to the encoding control unit 42.

[0025] The media control unit 44 controls the exchange of files between the medium 14 and the I / Fs 15 and 16. For example, the media control unit 44 causes a file from the file control unit 43 to be recorded on the medium 14 or transmitted from the I / Fs 15 and 16. The media control unit 44 also reads a file from the medium 14 or causes the I / Fs 15 and 16 to receive the file and supply it to the file control unit 43.

[0026] In response to a user's operation of the buttons / keys 17 or the touch panel 18, the operation control unit 45 supplies an operation signal corresponding to the operation to a necessary block.

[0027] The display control unit 46 performs display control and the like to supply the images and the like supplied from the encoding control unit 42 to the liquid crystal panel 19, the viewfinder 20, and the I / F 21 for display.

[0028] The UI control unit 47 is in charge of UI (User Interface) control.

[0029] The medium 14 is, for example, a storage medium such as an SD card. The I / F 15 is, for example, an I / F for a LAN (Local Area Network) such as WiFi (registered trademark) or Ethernet (registered trademark). The I / F 16 is, for example, an I / F for a USB (Universal Serial Bus). The buttons / keys 17 and the touch panel 18 are operated by a user when inputting commands or other information to the digital camera 10. The touch panel 18 can be configured integrally with a liquid crystal panel 19. The liquid crystal panel 19 and the viewfinder 20 display images and the like supplied from the display control unit 46. The I / F 21 is an I / F that transmits at least images, such as an HDMI (High-Definition Multimedia Interface) (registered trademark) or a DP (Display Port).

[0030] In the digital camera 10 configured as above, the optical system / image sensor control unit 41 generates, for example, a YUV image having the same resolution (number of pixels) (size) as the RAW image from an image of RAW data obtained by imaging with the image sensor 12 (hereinafter also referred to as a RAW image), and supplies this together with the RAW image to the encoding control unit 42. The encoding control unit 42 generates a main image (master image) of the HEIF file from the YUV image from the optical system / image sensor control unit 41. For example, the YUV image from the optical system / image sensor control unit 41 can be used as the main image of the HEIF file as is.

[0031] The encoding control unit 42 generates, from the YUV main image, a YUV image (hereinafter also referred to as a screen nail image) having a lower resolution than the main image as a first other image based on the main image for display on the liquid crystal panel 19 or an external display, and generates, for example, a YUV image (hereinafter also referred to as a thumbnail image) having a lower resolution than the screen nail image as a second other image based on the main image for index display (list display). The encoding control unit 42 supplies, for example, the screen nail image to the liquid crystal panel 19 via the display control unit 46, and displays it as a so-called through image. As the thumbnail image, for example, an image with a long side of 320 pixels or less can be used. The ratio of the size (number of pixels) between the main image and the screen nail image as the first other image based on the main image, or the thumbnail image as the second other image based on the main image can be, for example, 200 times or less. Similarly, the ratio of the size between the screen nail image as the first other image based on the main image and the thumbnail image as the second other image based on the main image can be 200 times or less. As the screen nail image, for example, an image with a resolution of 4K or more can be adopted. Furthermore, as the screen nail image, for example, an image with a resolution of 4K (QFHD) or FHD can be adopted depending on the user's selection. Furthermore, images with the same resolution can be adopted as the main image and the screen nail image. When images with the same resolution are adopted as the main image and the screen nail image, the HEIF file can store both the main image and the screen nail image, or the main image can be stored without storing the screen nail image. When the main image is stored without storing the screen nail image in the HEIF file, the main image can be resized and used as the screen nail image.

[0032] In addition, the encoding control unit 42 encodes the main image, screennail image, and thumbnail image corresponding to the RAW image (the main image, screennail image, and thumbnail image generated from the same RAW image) as necessary, and supplies them to the file control unit 43 together with the RAW image.

[0033] The file control unit 43 generates, as necessary, a RAW file in which a RAW image is stored, a HEIF file in which the corresponding main image, screennail image, and thumbnail image (the main image, screennail image, and thumbnail image generated from the same RAW image) are stored, and / or a JPEG file, and supplies them to the media control unit 44. A HEIF file is a file that complies with HEIF (High Efficiency Image File Format), and a JPEG file is a file that complies with JPEG (Joint Photographic Experts Group).

[0034] The media control unit 44 records the RAW file, HEIF file, or JPEG file from the file control unit 43 on the media 14 or transmits it from the I / F 15 or 16.

[0035] The type of file (e.g., RAW file, HEIF file, JPEG file, etc.) generated by the file control unit 43 can be selected, for example, according to a user operation (designation). Also, as described below, HEIF files come in an image item format and an image sequence format, and whether the image item format or the image sequence format is to be adopted can be selected, for example, according to a user operation. Furthermore, the file control unit 43 can convert between HEIF files and JPEG files according to a user operation.

[0036] Furthermore, the file control unit 43 can generate multiple files of the same image content, but with different codecs, image sizes (resolutions), color formats, and bit depths.

[0037] When the file control unit 43 generates multiple files with the same image content, the encoding control unit 42 generates a stream of images (file stream) to be stored in each of the multiple files from the YUV image from the optical system / image sensor control unit 41.

[0038] The encoding control unit 42 can generate image streams with different codecs, image sizes (resolutions), color formats, and bit depths.

[0039] For example, the encoding control unit 42 can generate an image of a predetermined size, a predetermined color format, and a predetermined bit depth from a YUV image supplied from the optical system / image sensor control unit 41, and generate a first stream by encoding the image with a predetermined codec (encoding method). Furthermore, the encoding control unit 42 can generate an image of a different size, a different color format, and a different bit depth from the same YUV image supplied from the optical system / image sensor control unit 41, and generate a second stream by encoding the image with a different codec.

[0040] Then, the file control unit 43 can generate a file storing the first stream and a file storing the second stream.

[0041] <JPEGファイル>

[0042] FIG. 2 is a diagram showing an example of a JPEG file format that complies with JPEG (Joint Photographic Experts Group).

[0043] A JPEG file is configured by storing, for example, Exif metadata, a thumbnail image, XMP (Extensible Metadata Platform) (registered trademark) metadata, an MPF ​​indicating the storage location (position) of the main image and the simplified display image, etc., a main image, and an image for simplified display. For example, a screen nail image can be used as the image for simplified display.

[0044] <ISOベースメディアファイルフォーマット>

[0045] FIG. 3 is a diagram showing an example of the ISO base media file format.

[0046] HEIF (ISO / IEC 23008-12) is a file format that conforms to the ISO Base Media File Format (ISO / IEC 14496-12), and therefore HEIF files conform to the ISO Base Media File Format.

[0047] The ISO base media file format is composed of units called boxes, which serve as containers for storing data, and has a structure called a box structure.

[0048] A box has a type (box type) and actual data (data). The type indicates the type of actual data in the box. Actual data can be playable media data such as images (still images, videos), audio, subtitles, etc., attribute names (field names) and attribute values ​​(field values) of the attribute names (variables represented by the attribute names), and various other data.

[0049] Furthermore, a box can be used as actual data, that is, a box can have another box as actual data, which allows a hierarchical structure to be created.

[0050] A base media file that complies with the ISO base media file format can have an ftyp box, a moov box (MovieBox), a meta box (MetaBox), an mdat box (MediaDataBox), and the like. The ftyp box stores identification information that identifies the file format. The moov box can store a trak box, and the like. The meta box can store an iinf box, an iprp box, an iref box, an iloc box, and the like. The mdat box can store media data (AV data) and any other data.

[0051] HEIF complies with the above ISO Base Media File Formats.

[0052] <HEIFファイル>

[0053] FIG. 4 is a diagram showing an example of a format of a HEIF file that complies with HEIF.

[0054] HEIF files are broadly divided into an image item format and an image sequence format, and the image item format is further divided into a single image format that has only one item, which will be described later, and an image collection format that has multiple items.

[0055] A HEIF file in the image item format has an ftyp box, a meta box, and an mdat box.

[0056] A HEIF file in image sequence format has an ftyp box, a moov box, and an mdat box.

[0057] Note that a HEIF file can have both a meta box and a moov box, not just one of them.

[0058] The ftyp box stores identification information for identifying the file format, for example, that the file is a HEIF file in image item format or image sequence format.

[0059] The meta box and moov box store metadata necessary for playback and management of the media data stored in the mdat box, such as the storage location of the media data.

[0060] The mdat box stores media data (AV data) and the like.

[0061] In the digital camera 10, which HEIF file to generate, the image item format or the image sequence format, can be selected according to, for example, a user's operation. Also, when encoding and storing an image in the mdat box of the HEIF file, only intra-encoding is permitted for the image item format, and intra-encoding and inter-encoding are permitted for the image sequence format. Therefore, for example, when high-speed access to data stored in the HEIF file is prioritized, generation of a HEIF file in the image item format can be selected, and when reducing the size (amount of data) of the HEIF file is prioritized, generation of a HEIF file in the image sequence format can be selected.

[0062] FIG. 5 is a diagram showing an example of the format of a HEIF file in image item format.

[0063] In a HEIF file in the image item format, information indicating that it is a HEIF file in the image item format, such as mif1, is stored (as an attribute value) in the ftyp box.

[0064] The meta box stores an iinf box, an iref box, an iprp box, and an iloc box.

[0065] The iinf box stores the number of items (attribute names and attribute values ​​representing) that are media data (AV data) stored in the mdat box. An item is a piece of data stored in the mdat box of a HEIF file in image item format; for example, one image (screen) is an item. In this specification, one image, whether a still image or a video, is also called a frame. One frame is one item.

[0066] The iref box stores information indicating the relationship between items. For example, the mdat box can store the corresponding main image, screennail image, and thumbnail image as items. When the mdat box stores item I1 as the main image, item I2 as the screennail image, and item I3 as the thumbnail image, the iref box stores information indicating that item I2 is the screennail image of the main image as item I1, and information indicating that item I3 is the thumbnail image of the main image as item I1.

[0067] The iprp box stores information about the properties of an item.

[0068] The iloc box stores information about the storage location of the items stored in the mdat box.

[0069] In the mdat box of the image item format (HEIF file), for example, a frame of an image is stored as an item. One or more items can be stored in the mdat box. Also, a frame as an item can be encoded and stored in the mdat box. However, encoding of a frame as an item stored in the mdat box of the image item format is limited to intra encoding. As an encoding method (codec) for encoding a frame as an item, for example, HEVC or the like can be adopted.

[0070] FIG. 6 is a diagram showing an example of the iprp box of FIG.

[0071] The iprp box stores an ipco box and an ipma box related to item properties. The ipco box stores the properties of the item stored in the mdat box, such as codec information related to the codec of the image as the item and image size information related to the size. The ipma box stores an index (pointer) to the properties stored in the ipco box of the item stored in the mdat box.

[0072] FIG. 7 is a diagram showing an example of the format of a HEIF file in image sequence format.

[0073] In a HEIF file in the image sequence format, information indicating that it is a HEIF file in the image sequence format, such as msf1, is stored in the ftyp box.

[0074] The moov box stores a trak box, which stores information about the track stored in the mdat box.

[0075] A track consists of one independent piece of media data, such as images and audio, that is played according to a timeline. For example, a track consists of one or more frames of images that form an elementary stream. For tracks stored in an mdat box, multiple tracks, for example, image and audio tracks recorded at the same time, can be played simultaneously.

[0076] The media data of a track is composed of units called samples. A sample is the smallest unit (access unit) for accessing media data in a HEIF file. Therefore, it is not possible to access media data in a HEIF file in units smaller than a sample.

[0077] For image media data, one sample is, for example, one frame, etc. For audio media data, one sample is, for example, one audio frame or the like defined by the standard of the audio media data.

[0078] In the mdat box of the image sequence format (HEIF file), the media data of a track is arranged in units called chunks. A chunk is a collection of one or more samples arranged at logically consecutive addresses.

[0079] When multiple tracks of media data are stored in the mdat box, the multiple tracks are arranged in an interleaved manner in chunk units.

[0080] As described above, an mdat box in the image sequence format stores one or more tracks made up of media data such as images and audio.

[0081] In the mdat box, the frames of the images that make up the track can be encoded and stored. To encode the frames that make up the track stored in the mdat box in the image sequence format, long GOP can be used as the GOP (Group of Picture), and both intra-encoding and inter-encoding can be used. As a codec for encoding the frames that make up the track, for example, HEVC can be used.

[0082] FIG. 8 is a diagram showing an example of a trak box.

[0083] The trak box can store a tkhd box and an mdia box. The tkhd box stores header information of the track managed by the trak box, such as the creation date and time of the track. The mdia box stores the minf box and the like. The minf box stores the stbl box. The stbl box stores the stsd box, stsc box, stsz box, and stco box, which store the track samples and, in turn, information for accessing the chunks. The stsd box stores codec information related to the codec of the track. The stsc box stores the chunk size (the number of samples in one chunk). The stsz box stores the sample size. The stco box stores the chunk offset, i.e., the offset of the placement position of each chunk of the track stored in the mdat box.

[0084] Here, a HEIF file in the image item format is also called a collection file, and a HEIF file in the image sequence format is also called a sequence file.

[0085] The digital camera 10 can generate a HEIF file that stores the main image and also any required screennail and / or thumbnail images.

[0086] <Collection file>

[0087] FIG. 9 is a diagram showing an example of a collection file in which main images and thumbnail images are stored.

[0088] Now, it is assumed that frames (items) are encoded in HEVC and stored in the mdat box of the collection file.

[0089] In the ftyp box, heic, which indicates that the file format is an image item format and that the codec is HEVC, is stored as identification information for identifying the file format.

[0090] The number of items (item count) stored in the mdat box is stored in the iinf box. In Fig. 9, a total of four items (frames) are stored in the mdat box: the main image (hereinafter also referred to as main image Item#1) identified by item ID#1, the main image Item#2, the thumbnail image (hereinafter also referred to as thumbnail image Item#101) identified by item ID#101, and the thumbnail image Item#102. Therefore, the number of items is four. Note that the thumbnail image Item#101 is the thumbnail image of the main image Item#1, and the thumbnail image Item#102 is the thumbnail image of the main image Item#2.

[0091] The iinf box further stores an infe box for each item stored in the mdat box. The infe box stores an item ID that identifies the item and an item type. In FIG. 9, there are infe boxes for the main images Item#1 and Item#2, and the thumbnail images Item#101 and Item#102.

[0092] In the iref box, for example, a thmb box is stored as information relating items stored in the mdat box. In the thmb box, a reference source and a reference destination are stored in association with each other as information relating a main image and a thumbnail image of the main image. In the thmb box, the reference source represents the item ID of the main image, and the reference destination represents the item ID of the thumbnail image of the main image specified by the item ID of the reference source. Therefore, according to the reference destination associated with the reference source, it is possible to recognize the item ID of the thumbnail image of the main image specified by the item ID represented by the reference source. Also, according to the reference source associated with the reference destination, it is possible to recognize the item ID of the main image of the thumbnail image specified by the item ID represented by the reference destination.

[0093] The iprp box stores an ipco box and an ipma box, as described in Fig. 6. The ipco box stores properties of a frame as an item stored in the mdat box, such as codec information related to the codec and image size information related to the size, as described in Fig. 6. The ipma box stores an index to the properties stored in the ipco box of the item stored in the mdat box, as described in Fig. 6.

[0094] The iloc box stores information about the storage location of an item in the mdat box, as described in Fig. 6. In Fig. 9, the iloc box stores that the number of items is 4. Furthermore, the iloc box stores the offsets to the storage locations and sizes of the main images Item#1 and Item#2 and the thumbnail images Item#101 and Item#102 stored in the mdat box, in association with the item IDs.

[0095] <Sequence file>

[0096] FIG. 10 is a diagram showing an example of a sequence file in which a track of a main image and a track of thumbnail images of the main image are stored.

[0097] Now, let us assume that frames are encoded in HEVC and stored in the mdat box of the sequence file.

[0098] In the ftyp box, hevc, which indicates that the file format is an image sequence format and that the codec is HEVC, is stored as identification information for identifying the file format.

[0099] As described in Fig. 7, the moov box stores a trak box that manages each track stored in the mdat box. In Fig. 10, the track of the main image identified by track ID #1 (hereinafter also referred to as track #1) and track #2 of the thumbnail image of the main image in track #1 are stored in the mdat box. Therefore, the moov box stores a trak box that manages track #1 and a trak box that manages track #2. The (frame of) n-th thumbnail image (from the beginning) of track #2 is the thumbnail image of the n-th main image in track #1.

[0100] A sequence file is useful, for example, when continuous shooting is performed with the digital camera 10, in the case where a plurality of frames of main images and thumbnail images obtained by the continuous shooting are each recorded as one track.

[0101] The tkhd box of the trak box that manages track #1 of the main images stores track ID #1 that identifies track #1, the image size of the main images that make up track #1, rotation information that indicates the orientation of the digital camera 10 when the main images were captured, and the creation date and time of track #1. The tkhd box of the trak box that manages track #2 of the thumbnail images stores track ID #2 that identifies track #2 and the creation date and time of track #2.

[0102] In addition to the tkhd box and mdia box described in Fig. 7, the trak box can store a tref box. The tref box stores a track ID that identifies other tracks related to the track managed by the trak box in which the tref box is stored, and information that indicates the contents of the track. In Fig. 10, a tref box is provided in a trak box that manages track #2. The tref box stores information that indicates that the other track related to track #2 is track #1 (track_ID=1) and that the data that constitutes track #2 is a thumbnail image (track #2 is a thumbnail image track) (type=thmb).

[0103] In addition to the MINF box described in Fig. 8, the mdia box of the trak box can store an hdlr box. The hdlr box stores information indicating the type of data that constitutes the track managed by the trak box in which the hdlr box is stored. The hdlr box stored in the trak box that manages track #1 of the main image (in the mdia box in which it is stored) stores information (pict) indicating that the data that constitutes track #1 is a picture (frame), and the hdlr box stored in the trak box that manages track #2 of the thumbnail images stores information indicating that the data that constitutes track #2 is a picture.

[0104] The minf box is as explained in FIG.

[0105] <HEIF file containing tiled images>

[0106] FIG. 11 is a diagram showing an example of a HEIF file in which tile images are stored.

[0107] As described above, one of the item types in HEIF is grid. A grid image, which is an image (item) whose item type is grid, is formed by tiling one or more tile images.

[0108] As the tile images, for example, divided images obtained by dividing an image captured by the digital camera 10 or the like can be used.

[0109] Furthermore, the images themselves captured by the digital camera 10 or the like can be used as the tile images, and a grid image can be formed from one or more of such tile images. For example, a plurality of images captured by a plurality of cameras can be used as tile images, and a grid image can be formed from such a plurality of tile images.

[0110] In the following, for ease of understanding, unless otherwise specified, divided images obtained by dividing an image captured by digital camera 10 or the like will be adopted as tile images.

[0111] For grid images, where the item type is grid, each tile (one or more) that makes up the grid image is stored as an item in the HEIF file.

[0112] Note that, with respect to a grid image, the grid image itself is not stored in the HEIF file, but the tile images that form the grid image are stored in the HEIF file. However, for convenience, in this specification, a HEIF file that stores a tile image is also referred to as a HEIF file that stores a grid image formed from the tile image.

[0113] FIG. 11 shows an example of a HEIF file that stores (tile images that form) a grid image.

[0114] In the HEIF file of Figure 11, for example, nine images obtained by dividing an image captured by digital camera 10 (hereinafter also referred to as the captured image) into 3 x 3 (horizontal x vertical) pieces are stored as tile images Item#1 to Item#9 in the mdat box as items.

[0115] In Fig. 11, the items stored in the mdat box are nine items, tile images Item#1 to Item#9, but the number of items in the iinf box and iloc box is 10. This is because, in addition to the nine items, tile images Item#1 to Item#9, a grid image (reconstructed image) formed from the tile images Item#1 to Item#9 is also counted as an item. In Fig. 11, the item ID of the grid image is 10, and for ease of explanation, this grid image will also be referred to as grid image Item#10.

[0116] For grid image Item#10, the media data is not stored in the mdat box, but instead, an idat box is stored in the meta box. The idat box stores metadata of the grid image, such as the number of tiles horizontally, the number of tiles vertically, output_width, and output_height.

[0117] The number of tiles horizontally and the number of tiles vertically indicate the number of tile images #1 to #9 horizontally and vertically, respectively, that make up the grid image Item#10.

[0118] output_width and output_height represent the horizontal and vertical sizes (number of pixels) of the canvas, which is the image area in which the tile images that form the grid image are tiled (arranged). If the horizontal and vertical pixel counts of a tile image are represented as tile_width and tile_height, respectively, then tile_width × number of tiles horizontal must be greater than or equal to output_width, and tile_height × number of tiles vertical must be greater than or equal to output_height.

[0119] Furthermore, for grid image Item#10, information (offset and size) regarding the storage location of grid image Item#10 is stored in the iloc box, and this information indicates the storage location of the idat box of grid image Item#10.

[0120] The HEIF file in Fig. 11 has 10 infe boxes because 10 items are stored: tile images Item#1 to Item#9 and tile image Item#10 formed from tile images Item#1 to Item#9. As described in Fig. 9, an item ID that identifies an item and an item type are stored (registered) in the infe box, and grid, which represents a grid item, is stored in the infe box for grid image Item#10 as the item type of grid image Item#10. An item with an item type of grid, here grid image Item#10, is called a grid item.

[0121] In a HEIF file in which a grid item is stored, a dimg box is stored in the iref box. The dimg box stores information that associates a grid item with a tile image that constitutes the grid item. For example, the dimg box stores the item ID of a tile image as a reference destination, and stores the item ID of a grid image formed from the tile image as a reference source. In FIG. 11, item IDs #1 to #9 of tile images Item#1 to Item#9 are stored as reference destinations, and item ID #10 of grid image Item#10 is stored as a reference source. In addition, the dimg box stores a reference counter that indicates the number of tile images that form the grid image.

[0122] For such a HEIF file, the file control unit 43 can recognize that the grid image Item#10 is a grid item (reconstructed image) formed from one or more tile images, from the item type grid stored in the infe box of the grid image Item#10. Furthermore, the file control unit 43 can identify the item ID#10 of the grid image Item#10 formed from the tile images and the item IDs#1 to #9 of the tile images Item#1 to Item#9 used to form the grid image Item#10, from the reference source and reference destination stored in the dimg box. Furthermore, the file control unit 43 can identify the horizontal and vertical numbers of the tile images #1 to #9 forming the grid image #10, and the horizontal and vertical sizes of the canvas on which the tile images #1 to #9 are arranged when forming the grid image #10, from the tile number horizontal and tile number vertical stored in the idat box, as well as the output_width and output_height.

[0123] The file control unit 43 can form a grid image Item#10 of item ID #10 identified from the dimg box by arranging the number of tile images, Item#1 to Item#9, of item IDs #1 to #9 identified from the dimg box in the horizontal and vertical directions identified from the idat box, on a canvas of the size identified from the idat box.

[0124] Below, we will explain use cases in which the image size and bitrate (data volume) of tile images stored in HEIF files are restricted.

[0125] FIG. 12 is a diagram showing an example of a use case in which the image size or bit rate of a tile image is restricted.

[0126] A first use case in which the image size or bit rate of a tile image is limited is, for example, when a tile image stored in a HEIF file is played on a device other than the device that generated the HEIF file. In this case, the image size or bit rate of the tile image is limited so that the tile image can be decoded or played back in software (SW) or hardware (HW) within the range of performance (playback capability) of the other device. For example, Intel's CPUs have different performances, such as the range of image sizes that can be hardware decoded, depending on the generation. When playing back tile images on such a CPU, the image size of the tile image is limited depending on the performance of the CPU.

[0127] A second use case in which the image size or bit rate of a tile image is limited is, for example, when a HEIF file in which a tile image is stored is transferred from a device that generated the HEIF file to another device. In this case, the image size or bit rate of the tile image is limited to comply with, for example, a communication standard (connection standard) for a connection method between the device that generated the HEIF file and the other device.

[0128] A third use case in which the image size or bit rate of a tile image is limited is, for example, when generating a single HEIF file that collectively stores tile images stored in multiple HEIF files and can form a grid image from those tile images. In this case, the image size or bit rate of the tile image is limited so that the image size of all the tile images in each of the multiple HEIF files is the same.

[0129] A fourth use case in which the image size or bit rate of a tile image is limited may be, for example, a case in which multiple nodes serving as processing blocks for processing tile images encode one tile image in parallel for each node, and generate an HEIF file in which the encoded tile image is stored in each of the multiple nodes, within a predetermined time. In this case, the image size or bit rate of the tile image is limited so that the generation of the HEIF file, including the encoding of the tile image in the node, can be completed within a predetermined time. For example, this may be the case in which images captured by continuous shooting with a large-sized imager are used as a grid image, and multiple tile images forming the grid image are processed by (semiconductor) chips serving as multiple nodes.

[0130] A fifth use case in which the image size or bit rate of a tile image is limited is, for example, when multiple nodes acting as processing blocks for processing tile images decode one tile image in parallel for each node, and perform slideshow playback or time lapse playback, etc., on a grid image formed from the multiple tile images decoded by the multiple nodes. In this case, the playback time for one frame is limited, so the image size or bit rate of the tile image is limited so that the playback time constraint can be guaranteed (so that a grid image can be formed within the playback time).

[0131] A sixth use case in which the image size or bit rate of a tile image is limited is, for example, a case in which each of a plurality of nodes serving as a processing block for processing tile images encodes each tile image in parallel, transfers the tile image under limited transmission bandwidth conditions, and generates a HEIF file at the transfer destination within a predetermined time. In this case, the image size or bit rate of the tile image is limited so that the tile image can be transferred under limited transmission bandwidth conditions and the generation of the HEIF file can be completed within a predetermined time. For example, this applies to a case in which a tile image stored in a HEIF file is transferred from a camera to storage in response to the operation of a so-called learning remote commander (smart remote commander) (WiFi remote commander).

[0132] The image size or bit rate of a tile image stored in a HEIF file is also limited by, for example, the profile, level, or tier of the codec supported by the device that handles the tile image. For example, the image size of the tile image is limited to an image size defined by the level of the codec supported by the device that handles the tile image. Also, for example, the bit rate of the tile image is limited to a bit rate defined by the profile, level, or tier of the codec supported by the device that handles the tile image.

[0133] In addition, if the device handling the tile images has limitations on the chroma sampling (chroma format) or encoder tools supported by the device, the bit rate of the tile images will be limited to the bit rate specified in the profile that specifies the chroma sampling and encoder tools supported by the device handling the tile images.

[0134] In the digital camera 10, if the image size of the tile images is set arbitrarily, problems may arise in use cases where the image size or bit rate of the tile images is restricted as described above.

[0135] Therefore, the present technology makes it possible to appropriately set the image size of a tile image and generate a tile image with such appropriately set image size (hereinafter also referred to as set image size).

[0136] With this technology, by setting the image size appropriately, Ensuring playback compatibility when playing back tile images forming a grid image as content captured by the digital camera 10 on other devices Complying with the communication standard (connection standard) for communication between the digital camera 10 and other devices, which is required when transferring tile images that form a grid image from the digital camera 10 to other devices Editing can be performed without transcoding to collect tile images that form a grid image stored in multiple HEIF files and generate a HEIF file that contains tile images that form a new grid image. When multiple nodes encode and decode multiple tile images that form a grid image in parallel, the time it takes to generate a HEIF file containing encoded tile images and the time it takes to play back a grid image formed from the decoded tile images must be within a specified time frame, depending on the performance of the nodes, for example, the throughput at which the nodes encode tile images and the throughput at which the nodes decode (play back) the tile images. - Guarantee that the transfer time of tile images from a node is within a specified time according to the transfer bandwidth (transfer throughput). etc. to be realized.

[0137] The image size (set image size) of the tile image can be set according to information from outside the digital camera 10, such as user operations on the menu screen of the digital camera 10, the performance of the external device obtained through negotiation between the digital camera 10 and the external device as another device with which the digital camera 10 communicates, and the communication standard as the connection method followed when the digital camera 10 and the external device communicate with each other.

[0138] FIG. 13 is a diagram illustrating an example of setting the image size of a tile image in the first use case.

[0139] Figure 13 shows a case in which a HEIF file containing tile images generated by digital camera 10 is transferred to an external device such as a smartphone or a PC (Personal Computer), and the tile images stored in the HEIF file are played on the external device.

[0140] The manufacturer of digital camera 10 can set the image size of tile images by, for example, taking into account playback environments around the world and rewriting the firmware of digital camera 10 with new firmware as external information.

[0141] For example, if it is recognized that compatibility of external devices such as smartphones and PCs with so-called 4K images has become a global standard, the manufacturer of digital camera 10 can rewrite the firmware of digital camera 10 so as to set the image size of tile images to the image size of 4K images. Rewriting of the firmware of digital camera 10 can be performed by communication via I / F 15, etc. Thereafter, if it is recognized that compatibility of external devices with so-called 8K images has become a global standard, the manufacturer of digital camera 10 can rewrite the firmware of digital camera 10 so as to set the image size of tile images to the image size of 8K images.

[0142] In digital camera 10, the image size of the tiled images can be set in response to a user's operation on the menu screen of digital camera 10 as information from outside.

[0143] For example, when transferring a HEIF file to a smartphone that supports playback of up to 4K images, the user can operate the menu screen to set the image size of the tile images stored in the HEIF file to an image size equal to or smaller than the image size of the 4K image. Also, for example, when transferring a HEIF file to a PC that supports playback of up to 8K images, the user can operate the menu screen to set the image size of the tile images stored in the HEIF file to an image size equal to or smaller than the image size of the 8K image.

[0144] In digital camera 10, the image size of the tile images can be set according to the performance (playback capability) of a smartphone or PC serving as an external device connected to digital camera 10.

[0145] For example, when the digital camera 10 is connected to a smartphone that supports playback of up to 4K images and a HEIF file is transferred, the digital camera 10 negotiates with the smartphone, and depending on whether the smartphone supports playback of up to 4K images, acquired through the negotiation, the image size of the tile image stored in the HEIF file can be set to an image size equal to or smaller than the image size of the 4K image (e.g., 4K). Also, for example, when the digital camera 10 is connected to a PC that supports playback of up to 8K images and a HEIF file is transferred, the digital camera 10 negotiates with the PC, and depending on whether the PC supports playback of up to 8K images, acquired through the negotiation, the image size of the tile image stored in the HEIF file can be set to an image size equal to or smaller than the image size of the 8K image (e.g., 8K).

[0146] FIG. 14 is a diagram illustrating an example of setting the image size of a tile image in the second use case.

[0147] Figure 14 shows a case in which a HEIF file containing tile images generated by digital camera 10 is transferred to an external device such as PC (Personal Computer)-A or PC-B, and the tile images stored in the HEIF file are played back on that external device.

[0148] In digital camera 10, the image size of the tile images can be set according to the communication standard supported by the external device, which is information from the outside.

[0149] For example, when transferring a HEIF file to PC-A that complies with communication standard A supporting codec levels up to 5.2, digital camera 10 can set the image size of the tile images stored in the HEIF file to an image size equal to or smaller than the image size supported by level 5.2 in accordance with communication standard A supported by PC-A. Also, for example, when transferring a HEIF file to PC-B that complies with communication standard B supporting codec levels up to 6.2, digital camera 10 can set the image size of the tile images stored in the HEIF file to an image size equal to or smaller than the image size supported by level 6.2 in accordance with communication standard B supported by PC-B. In digital camera 10, the communication standards supported by PC-A and PC-B can be acquired, for example, by negotiation between digital camera 10 and each of PC-A and PC-B.

[0150] In addition, if the external device supports multiple communication standards, the digital camera 10 can negotiate with the external device to determine the communication standard to be used when transferring the HEIF file to the external device, and set the image size of the tile image stored in the HEIF file according to that communication standard.

[0151] FIG. 15 is a diagram illustrating an example of setting the image size of a tile image in the third use case.

[0152] Figure 15 shows a case in which four HEIF files are combined into one HEIF file in editing software (a new HEIF file is generated by combining the four HEIF files) so that a new grid image can be formed by combining the tile images stored in each of the four HEIF files generated by four cameras.

[0153] The four cameras that generate the unedited HEIF files that are input to the editing software are configured, for example, in the same manner as the digital camera 10.

[0154] The HEIF standard requires that all tile images forming a new grid image stored in a new HEIF file generated by editing software have the same image size. Therefore, the four cameras that generate the unedited HEIF file generate the unedited HEIF file using the same parameter set for the image size, chroma sampling, etc. of the tile images that form the grid image.

[0155] In Fig. 15, each of the four cameras captures an image of the same image size (an image that will become a grid image), and a HEIF file is generated that stores tile images obtained by dividing the image into 2 x 2 (horizontal x vertical) pieces. Then, in the editing software, new HEIF files are generated that store the tile images stored in the HEIF files generated by the four cameras so that a new grid image can be formed by combining the tile images stored in the HEIF files generated by the four cameras.

[0156] Since the four cameras use the same parameter set to generate unedited HEIF files, for example, one of the four cameras is set as the master camera, and the remaining three cameras are set as slave cameras. For example, a user operates one of the four cameras to set it as the master camera. The master camera communicates with the remaining three cameras and sets the three cameras as slave cameras.

[0157] The master camera sets the image size of the tile images and instructs the three slave cameras on the image size of the tile images through communication. The other cameras set the image size of the tile images to the same image size as the master camera in response to instructions from the master camera as information from an external device.

[0158] For example, the master camera can set the image size of the tile image in response to a user operation. Also, for example, the master camera can collect the codec performance of the three slave cameras as information from an external device, such as one or more of the codec profile, level, and tier, by communicating with the three slave cameras, and set the image size of the tile image to the image size corresponding to the lowest performance.

[0159] Of the four cameras, one is set as the master camera and the remaining three are set as slave cameras, and if the four cameras are capable of communicating with an external device such as the cloud, they can set the image size of the tile images according to instructions from the cloud. The cloud can set the image size instructed to the four cameras in the same way as the master camera, for example.

[0160] In addition, if the four cameras are able to communicate with the editing software (or the hardware, such as a PC on which the software is implemented), the four cameras can set the image size of the tile images according to the functions and performance of the editing software.

[0161] FIG. 16 shows a first example of four unedited HEIF files generated by four cameras in the third use case, and a new HEIF file generated from the four unedited HEIF files in editing software.

[0162] In Figure 16, images G1, G2, G3, and G4 of the same image size are captured by four cameras, and HEIF files #1, #2, #3, and #4, which store tile images obtained by dividing the images G1, G2, G3, and G4 into 2 x 2 pieces, are generated as pre-edited HEIF files.

[0163] Then, in Figure 16, HEIF file #0, which stores the tile images stored in HEIF files #1 to #4 before editing, is generated as a new HEIF file so that a new grid image is formed in which the 2 x 2 tile images stored in HEIF files #1 to #4 are arranged in the upper left, upper right, lower left, and lower right, respectively.

[0164] FIG. 17 is a diagram showing a second example of four unedited HEIF files generated by four cameras in the third use case, and a new HEIF file generated from the four unedited HEIF files in editing software.

[0165] In Figure 17, as in Figure 16, images G1 to G4 of the same image size are captured by four cameras, and pre-edited HEIF files #1 to #4 are generated, containing tile images obtained by dividing the images G1 to G4 into 2 x 2 pieces.

[0166] Here, the top left, top right, bottom left, and bottom right tile images of the 2×2 tile images stored in HEIF file #i will be referred to as tile images Ti1, Ti2, Ti3, and Ti4, respectively.

[0167] In Figure 17, a new HEIF file #0 is generated so that a new grid image is formed in which tile images T11, T21, T31, and T41 are arranged in the upper left, tile images T12, T22, T32, and T42 are arranged in the upper right, tile images T13, T23, T33, and T43 are arranged in the lower left, and tile images T14, T24, T34, and T44 are arranged in the lower right.

[0168] FIG. 18 is a diagram showing a third example of four unedited HEIF files generated by four cameras in a third use case, and a new HEIF file generated from the four unedited HEIF files in editing software.

[0169] In Figure 18, images G1, G3, and G4 of the same image size are captured by three of the four cameras, as in Figure 16, and HEIF files #1, #3, and #4 containing tile images T11 to T14, T31 to T34, and T41 to T44 obtained by dividing the images G1, G3, and G4 into 2 x 2 pieces are generated as pre-edited HEIF files.

[0170] Furthermore, in Figure 18, the remaining camera captures image G2, which has an image size that is 3 / 2 times larger in both width and height than images G1, G3, and G4, and HEIF file #2, which stores tile images T21 to T29 obtained by dividing image G2 into 3 x 3 pieces, is generated as the pre-edited HEIF file.

[0171] The tile images stored in HEIF files #1 to #4 have the same parameter sets, such as image size.

[0172] In Figure 18, a new HEIF file #0 is generated in which tile images T11 to T14, T21 to T24, T31 to T34, and T41 to T44 stored in HEIF files #1 to #4 are stored, so that a new grid image is formed in which the 2 x 2 tile images T11 to T14 stored in HEIF file #1, the 2 x 2 tile images T21 to T24 (the area surrounded by a thick frame in the figure) selected from the 3 x 3 tile images T21 to T29 stored in HEIF file #2, the 2 x 2 tile images T31 to T34 stored in HEIF file #3, and the 2 x 2 tile images T41 to T44 stored in HEIF file #4 are arranged in the upper left, upper right, lower left, and lower right, respectively.

[0173] Selection of the 2 x 2 tile images to be stored in new HEIF file #0 from the 3 x 3 tile images T21 to T29 stored in HEIF file #2 can be performed, for example, in editing software, in response to user operations, etc.

[0174] FIG. 19 is a diagram showing a fourth example of four unedited HEIF files generated by four cameras in the third use case, and a new HEIF file generated from the four unedited HEIF files in editing software.

[0175] In FIG. 19, pre-edited HEIF files #1 to #4 have been generated, similarly to FIG.

[0176] In Figure 19, 2 x 2 tile images T21 to T24 selected from the 3 x 3 tile images T21 to T29 stored in HEIF file #2 in the same manner as in Figure 18, 2 x 2 tile images T31 to T34 stored in HEIF file #3, and 2 x 2 tile images T41 to T44 stored in HEIF file #4 are arranged to the right of, below, and diagonally below the right of the 2 x 2 tile images T11 to T14 stored in HEIF file #1, and a new HEIF file #0 has been generated in which tile images T11 to T14, T21 to T28, T31 to T34, and T41 to T44 stored in HEIF files #1 to #4 are stored so that a new grid image is formed in which four tile images T25 to T28 of the remaining tile images T25 to T29 of the 3 x 3 tile images T21 to T29 stored in HEIF file #2 are arranged on the right edge.

[0177] As shown in Figure 18, a new grid image can be formed by arranging the same number of tile images (four tile images in Figure 18) from the tile images stored in each of HEIF files #1 to #4, or as shown in Figure 19, it can be formed by arranging a unequal number of tile images (four tile images and eight tile images in Figure 19) from the tile images stored in each of HEIF files #1 to #4.

[0178] FIG. 20 is a diagram illustrating an example of setting the image size of a tile image in the fourth use case.

[0179] Figure 20 shows a case in which multiple nodes, which act as processing blocks that process tile images, encode each tile image in parallel and generate a single HEIF file within a specified period of time.

[0180] That is, in FIG. 20, for example, an image captured by digital camera 10 is divided into four tile images, each of which is encoded at each node, and a single HEIF file containing the encoded tile images is generated within a specified time.

[0181] The encoding of the tile images at the node and the generation of a single HEIF file storing the encoded tile images may be performed by the digital camera 10 or may be performed externally to the digital camera 10.

[0182] In Figure 20, the total processing time required to encode tile images at each node and generate HEIF files is affected by the performance of the node (encoding throughput). In order to keep this total processing time within a specified period, it is necessary to limit the image size and bit rate (amount of data) of the tile images processed by the node according to the performance of the node.

[0183] Therefore, for the image size or bit rate of the tile image, a target image size or target bit rate that is a target (limit) can be set according to the performance of the node so that the total processing time falls within a predetermined time. Then, according to the target image size or target bit rate, the image size of the tile image can be set so that, for example, the image size or bit rate of the tile image is equal to or less than the target image size or target bit rate.

[0184] The target image size or the target bit rate can be set according to external information.

[0185] For example, in digital camera 10, a target image size or a target bit rate can be set so that the total processing time falls within a predetermined time period, according to the node performance input by a user operation or a predetermined time period (according to a user operation). If the node is an external device outside digital camera 10, a target image size or a target bit rate can be set so that the total processing time falls within a predetermined time period, according to the node performance as information obtained from the node as the external device.

[0186] In addition, in digital camera 10, for example, a target image size or a target bit rate can be set according to one or more of the profile, level, and tier of the codec supported by the node as an external device, so as not to violate the profile, level, and tier.

[0187] In addition, in digital camera 10, a target image size or target bit rate can be set according to the connection method for connecting to a node as an external device, for example, the communication standard followed when digital camera 10 and the node communicate with each other, so as not to violate the codec profile, level, and tier supported by the communication standard.

[0188] The image size of the tile image can be set so that the image size or bit rate of the tile image is equal to or less than the target image size or target bit rate, or can be set so that the image size and bit rate of the tile image fall within a specified range based on the target image size and target bit rate, respectively.

[0189] In FIG. 20, the target image size or target bit rate is set according to external information such as node performance, but the image size of the tile images can also be set so that the total processing time falls within a specified time period according to external information such as node performance.

[0190] Also, in FIG. 20, a tile image is obtained by dividing an image captured by digital camera 10, but it is also possible to input multiple images captured by multiple cameras, for example, to be used as tile images to a node.

[0191] FIG. 21 is a diagram illustrating an example of setting the image size of a tile image in the fifth use case.

[0192] Figure 21 shows a case in which multiple nodes, which act as processing blocks that process tile images, decode multiple tile images stored in a HEIF file in parallel, and perform playback with restrictions on the playback time per image, such as slideshow playback and timelapse playback, on a grid image formed from the multiple tile images decoded by the multiple nodes.

[0193] That is, in FIG. 21, for example, an image captured by digital camera 10 is divided into four tile images, each of which is decoded by each node, and a grid image is formed from the four tile images within a specified time period to satisfy the playback time constraints.

[0194] The decoding of the tile images at the nodes and the formation of a grid image in which the tile images are arranged may be performed by digital camera 10 or may be performed externally to digital camera 10 .

[0195] In Figure 21, the total processing time required to decode tile images and form grid images at each node is affected by the performance of the node (decoding throughput). In order to keep the total processing time within a specified time, it is necessary to limit the image size and bit rate of the tile images processed by the node according to the performance of the node.

[0196] Therefore, a target image size or bit rate for the tile images can be set according to the performance of the node so that the total processing time falls within a predetermined time.Then, the image size of the tile images can be set so that the image size or bit rate of the tile images is equal to or less than the target image size or target bit rate.

[0197] The target image size or target bit rate can be set according to external information, as in the case of FIG.

[0198] In FIG. 21, the target image size or target bit rate is set according to external information such as node performance, but the image size of the tile images can also be set so that the total processing time falls within a specified time period according to external information such as node performance.

[0199] Also, in FIG. 21, a tile image is obtained by dividing an image captured by digital camera 10. However, for example, as in the case of FIG. 20, multiple images captured by multiple cameras can be used as tile images and input to a node.

[0200] FIG. 22 is a diagram illustrating an example of setting the image size of a tile image in the sixth use case.

[0201] Figure 22 shows a case in which multiple nodes, which act as processing blocks for processing tile images, encode and transfer each tile image in parallel, and generate an HEIF file containing the tile image at the destination, all of which is completed within a specified time.

[0202] That is, in FIG. 22, for example, an image captured by digital camera 10 is divided into four tile images, each of which is encoded and transferred at each node, and a single HEIF file containing the encoded tile images is generated at the transfer destination within a specified time.

[0203] The encoding of the tile images at the nodes may be performed in digital camera 10 or external to digital camera 10 .

[0204] 22, the total processing time required for encoding tile images at each node, transferring the tile images, and generating HEIF files is affected by the transmission bandwidth available for transferring tile images in the network used for transferring the tile images. Therefore, in order to keep the total processing time within a specified time, it is necessary to limit the image size and bit rate of the tile images processed at the node according to the transmission bandwidth of the network.

[0205] Therefore, a target image size or bit rate for the tile images can be set according to the network transmission bandwidth so that the total processing time falls within a predetermined time. The image size of the tile images can be set so that the image size or bit rate is equal to or less than the target image size or target bit rate.

[0206] The target image size or the target bit rate can be set according to external information.

[0207] For example, in digital camera 10, a target image size or a target bit rate can be set so that the total processing time falls within a predetermined time period according to a predetermined time period input by a user operation or according to a network transfer bandwidth (according to a user operation). Also, in the case where digital camera 10 can obtain a transfer bandwidth from an external network, a target image size or a target bit rate can be set so that the total processing time falls within a predetermined time period according to the transfer bandwidth obtained from the network.

[0208] In FIG. 22, the target image size or target bit rate is set in accordance with external information such as network bandwidth, but it is also possible to set the image size of the tile images so that the total processing time falls within a specified time period in accordance with external information such as network bandwidth.

[0209] Also, in FIG. 22, a tile image is obtained by dividing an image captured by digital camera 10, but it is also possible to input multiple images captured by multiple cameras, for example, to be used as tile images, into a node.

[0210] As described above, the image size of tile images can be set statically, so to speak, by rewriting the firmware of digital camera 10 with new firmware as external information, in accordance with the playback environment around the world, as explained in FIG. 13.

[0211] Furthermore, the image size of the tile images can be set dynamically, so to speak, as described with reference to FIGS.

[0212] For example, the image size of a tile image can be dynamically set in response to external information, such as a user operation or information from an external device such as the cloud.

[0213] In addition, the image size of the tile image can be dynamically set depending on the performance of the node or other external device connected to digital camera 10 (encoding and decoding throughput, corresponding codec profile, level, and tier, etc.), the communication standard (connection standard) for connecting digital camera 10 to the external device, and the network transfer bandwidth (communication conditions) used when transferring the tile image.

[0214] Furthermore, the image size of the tile images can be dynamically set depending on the configuration of the system that processes the tile images, such as a system that encodes the tile images (encode system) or a system that decodes the tile images (decode system).

[0215] As a system configuration for processing tile images, for example, as shown in FIG. 20, when a configuration is adopted in which multiple tile images obtained by dividing an image captured by a single digital camera 10 are encoded by multiple nodes, the image size of the tile images can be dynamically set according to the performance of the nodes.

[0216] For example, when a system for processing tile images is configured to process HEIF files containing tile images generated by multiple cameras, as shown in Figure 15, the image size of the tile images can be dynamically set based on the performance of the multiple cameras that generate the HEIF files containing the tile images, for example, the codec profiles supported by the multiple cameras.

[0217] For example, when adopting a system configuration for processing tile images in which tile images are decoded by nodes as shown in FIG. 21, the image size of the tile images can be dynamically set according to the performance (constraints) of the hardware and software that constitute the node, such as the profile of the codec supported by the node.

[0218] <Example of the configuration of the encoding control unit 42> FIG. 23 is a block diagram showing an example of the configuration of the encoding control unit 42. As shown in FIG.

[0219] Note that FIG. 23 illustrates only an image processing unit 110, which is a part of the encoding control unit 42 that generates tile images.

[0220] The image processing unit 110 includes a setting unit 111 and a generation unit 112 .

[0221] Setting unit 111 sets the image size of the tile image in accordance with the target image size or target bit rate of the tile image so that, for example, the image size or bit rate of the tile image is equal to or smaller than the target image size or target bit rate, or so that the image size or bit rate of the tile image falls within a predetermined range for the target image size or target bit rate, and supplies the image size to generation unit 112. Alternatively, setting unit 111 sets the image size of the tile image in accordance with external information, and supplies the image size to generation unit 112.

[0222] The generation unit 112 generates tile images of the image size (set image size) from the setting unit 111 by dividing the image supplied from the optical system / image sensor control unit 41 to the encoding control unit 42.

[0223] FIG. 24 is a flowchart illustrating an example of processing by the image processing unit 110.

[0224] In step S101, setting unit 111 sets the image size of the tile image according to one or both of the target image size and the target bit rate of the tile image. Alternatively, setting unit 111 sets the image size of the tile image according to external information. Then, setting unit 111 supplies the image size of the tile image (set image size) to generation unit 112, and the process proceeds from step S101 to step S102.

[0225] In step S102, the generation unit 112 generates tile images of the image size supplied from the setting unit 111 by dividing the image supplied from the optical system / image sensor control unit 41 to the encoding control unit 42, and then the process ends.

[0226] Thereafter, the tile images generated by the generation unit 112 are encoded by the encoding control unit 42, and a HEIF file in which the encoded tile images are stored is generated by the file control unit 43.

[0227] As described above, the image size of the tile image is set according to either or both of the target image size and target bit rate of the tile image, or according to external information, so that a tile image with an appropriate image size can be obtained.

[0228] FIG. 25 shows an example of an image size setting method for setting the image size of a tile image so that the image size of the tile image is equal to or smaller than the target image size.

[0229] In FIG. 25, whole image size horizontal input_width indicates the number of horizontal pixels of a captured image captured by digital camera 10 before being divided into tile images, and whole image size vertical input_height indicates the number of vertical pixels of the captured image.

[0230] The align size horizontal align_width and align size vertical align_height represent a predetermined number of pixels (hereinafter also referred to as the align pixel number) when the image size of the image to be processed is required to be a multiple of a predetermined number of pixels in a codec that encodes / decodes tile images. The align size horizontal align_width represents the align pixel number in the horizontal direction, and the align size vertical align_height represents the align pixel number in the vertical direction.

[0231] The target image size width "target_width" and the target image size height "target_height" respectively represent the target number of pixels in the width and height of the tile image (target image size).

[0232] The horizontal image size (tile_width) and vertical image size (tile_height) of a tile image represent the number of pixels in the width and height of the tile image, respectively.

[0233] The setting unit 111 receives as input the whole image size (horizontal input_width) and the whole image size (vertical input_height), as well as the align size (horizontal align_width) and the align size (vertical align_height).

[0234] In the digital camera 10, the overall image size horizontal input_width and the overall image size vertical input_height can be set, for example, in response to a user operation. For example, a plurality of pairs of the overall image size horizontal input_width and the overall image size vertical input_height are prepared, and a pair corresponding to a user operation can be selected from the plurality of pairs and set as the overall image size horizontal input_width and the overall image size vertical input_height. In addition, the overall image size horizontal input_width and the overall image size vertical input_height can be set in response to an instruction (information) from an external source such as a cloud. In the digital camera 10, an image having an image size represented by the overall image size horizontal input_width and the overall image size vertical input_height is captured.

[0235] The horizontal align size align_width and the vertical align size align_height can be set according to the codec used by the encoding control unit 42 for encoding and decoding the tile images. Additionally, the horizontal align size align_width and the vertical align size align_height can be set in the same way as the horizontal input_width of the overall image size and the vertical input_height of the overall image size, for example.

[0236] When the overall image size horizontal input_width and overall image size vertical input_height, as well as the align size horizontal align_width and align size vertical align_height, are set in response to user operation, it can be said that the overall image size horizontal input_width and overall image size vertical input_height, as well as the align size horizontal align_width and align size vertical align_height are parameters that are set in digital camera 10 in response to user operation.

[0237] The target image size (horizontal target_width) and the target image size (vertical target_height) can be set according to a user operation or external information such as information from an external device.

[0238] The setting unit 111 sets the horizontal image size tile_width and vertical image size tile_height (image size) of the tile image so that the number of horizontal pixels and the number of vertical pixels of the tile image are less than the target image size horizontal target_width and the target image size vertical target_height, respectively, in accordance with the overall image size horizontal input_width and the overall image size vertical input_height, as well as the target image size horizontal target_width and the target image size vertical target_height.

[0239] That is, the setting unit 111 obtains the maximum tile_width and tile_height that are integer multiples of the horizontal align size align_width and vertical align size align_height that satisfy formula (1), and sets these as the horizontal image size tile_width and vertical image size tile_height of the tile image, respectively.

[0240] tile_width = nx * align_width tile_height = ny * align_height tile_width <= target_width tile_height <= target_height tile_width <= input_width tile_height <= input_height (1)

[0241] nx and ny represent positive integers.

[0242] FIG. 26 is a flowchart illustrating an example of a process for setting the image size of a tile image so that the image size of the tile image is equal to or smaller than the target image size.

[0243] Note that in Figure 26, the process of setting the image size of a tile image is described by focusing on the setting of the horizontal image size (tile_width) out of the horizontal image size (tile_width) and vertical image size (tile_height). However, the vertical image size (tile_height) is also set in the same way as the horizontal image size (tile_width).

[0244] In step S111, the setting unit 111 sets a variable tile_num_tmp, which indicates the number of tile images in the horizontal direction when dividing a captured image into tile images, to CEIL(input_width / target_width, 1), and the process proceeds to step S112. The function CEIL(A,B) indicates the minimum value of integer multiples of B that are equal to or greater than A.

[0245] In step S112, the setting unit 111 sets the variable tile_width_temp, which indicates a candidate value for the horizontal image size tile_width of the tile image, to CEIL(input_width / tile_num_tmp, align_width), and the processing proceeds to step S113. As a result, the variable tile_width_temp is set to a value that is an integer multiple of the horizontal align size align_width, as a candidate for the horizontal image size tile_width of the tile image.

[0246] In step S113, the setting unit 111 determines whether the variable tile_width_temp is equal to or smaller than the horizontal target image size target_width.

[0247] If it is determined in step S113 that the variable tile_width_temp is not equal to or smaller than the target image horizontal size target_width, the process proceeds to step S114.

[0248] In step S114, the setting unit 111 increments the variable tile_num_tmp by 1, thereby increasing the number of tile images in the horizontal direction when dividing the captured image into tile images by 1, and the process returns to step S112, where the same process is repeated.

[0249] Also, if it is determined in step S113 that the variable tile_width_temp is equal to or smaller than the horizontal target image size target_width, the process proceeds to step S115.

[0250] In step S115, the setting unit 111 sets the horizontal image size tile_width to the variable tile_width_temp, and ends the process.

[0251] Note that the above explanation of FIG. 26 becomes an explanation of the setting of the vertical image size tile_height by replacing "horizontal" with "vertical".

[0252] FIG. 27 shows an example of an image size setting method for setting the image size of a tile image so that the bit rate of the tile image is equal to or lower than the target bit rate.

[0253] In FIG. 27, to set the image size of the tile images, the overall image size horizontal input_width and overall image size vertical input_height, as well as the align size horizontal align_width and align size vertical align_height are used, as in the case of FIG.

[0254] 27, the image size of the tile image is set using the overall image size (horizontal input_width, vertical input_height), align size (horizontal align_width, vertical align_height), and image quality (bit_per_pixel). The image quality (bit_per_pixel) indicates the number of bits as the amount of data (code amount) allocated per pixel of the tile image, and corresponds to (affects) the image quality of the tile image.

[0255] Furthermore, in Fig. 27, to set the image size of a tile image, a target amount of bits target_bit_per_tile is used as a target bit rate, instead of the target image size (horizontal target_width) and the target image size (vertical target_height) described in Fig. 25. The target amount of bits target_bit_per_tile as a target bit rate represents the target amount of data (amount of code) for the tile image.

[0256] The image quality bit_per_pixel is supplied to the setting unit 111 as an input.

[0257] The image quality bit_per_pixel can be set according to user operation, for example, in the same way as the overall image size horizontal input_width and overall image size vertical input_height in Fig. 25. When the image quality bit_per_pixel is set according to user operation, it can be said that the image quality bit_per_pixel is a parameter that is set in digital camera 10 according to user operation.

[0258] The target bit amount target_bit_per_tile (target bit rate) can be set based on external information such as user operation or information from an external device, for example, the network transfer bandwidth when transferring tile images over the network, as described in Figure 22.

[0259] The setting unit 111 sets the horizontal image size tile_width and vertical image size tile_height (image size) of the tile image according to the horizontal image size input_width and vertical image size input_height, image quality bit_per_pixel, and target bit amount target_bit_per_tile so that the bit rate (data amount) of the tile image tile_width * tile_height * bit_per_pixel is equal to or less than the target bit amount target_bit_per_tile as the target bit rate.

[0260] That is, the setting unit 111 finds the maximum tile_width and tile_height that are integer multiples of the horizontal align size align_width and vertical align size align_height that satisfy formula (2), and sets these as the horizontal image size tile_width and vertical image size tile_height of the tile image, respectively.

[0261] tile_width = align_width * nx tile_height = align_height * ny tile_width * tile_height * bit_per_pixel <= target_bit_per_tile tile_width <= input_width tile_height <= input_height (2)

[0262] FIG. 28 shows an example of an image size setting method for setting the image size of a tile image so that the image size and bit rate of the tile image are equal to or smaller than the target image size and target bit rate, respectively.

[0263] In FIG. 28, the image size of the tile images is set using the overall image size horizontal input_width and overall image size vertical input_height, as well as the align size horizontal align_width and align size vertical align_height, in the same manner as in FIG.

[0264] Furthermore, in FIG. 28, the image size of the tile images is set using the overall image size (horizontal input_width and vertical input_height), as well as the align size (horizontal align_width and vertical align_height), and the image quality (bit_per_pixel) described in FIG. 27.

[0265] In addition, in FIG. 28, the image size of the tile images is set using the target image size (horizontal target_width) and target image size (vertical target_height) described in FIG. 25, and the target bit amount (target_bit_per_tile) as the target bit rate described in FIG. 27.

[0266] The setting unit 111 sets the horizontal image size tile_width and vertical image size tile_height (image size) of the tile image according to the horizontal image size input_width and vertical image size input_height, the horizontal target image size target_width and vertical target image size target_height, image quality bit_per_pixel, and target bit amount target_bit_per_tile so that the number of horizontal pixels and the number of vertical pixels of the tile image are less than the horizontal target image size target_width and vertical target image size target_height, respectively, and so that the bit rate (data amount) of the tile image tile_width * tile_height * bit_per_pixel is less than the target bit amount target_bit_per_tile as the target bit rate.

[0267] That is, the setting unit 111 obtains the maximum tile_width and tile_height that are integer multiples of the horizontal align size align_width and vertical align size align_height that satisfy formula (3), and sets these as the horizontal image size tile_width and vertical image size tile_height of the tile image, respectively.

[0268] tile_width = nx * align_width tile_height = ny * align_height tile_width * tile_height * bit_per_pixel <= target_bit_per_tile tile_width <= target_width tile_height <= target_height tile_width <= input_width tile_height <= input_height (3)

[0269] The target image size (horizontal target_width) and (vertical target_height) as well as the target bit amount (target_bit_per_tile) as the target bit rate may be restricted by the profile, level, and tier of the codec supported by the node, etc. When the target image size (horizontal target_width) and (vertical target_height) as well as the target bit amount (target_bit_per_tile) as the target bit rate are restricted to predetermined values ​​by the codec profile, etc., the horizontal image size (tile_width) and vertical image size (tile_height) of the tile image are set according to the target image size (horizontal target_width) and the target image size (vertical target_height) restricted to the predetermined values ​​as well as the target bit amount (target_bit_per_tile) as the target bit rate, according to formula (3).

[0270] <Description of the computer to which this technology is applied>

[0271] Next, the above-mentioned series of processes can be performed by hardware or software. When the series of processes is performed by software, the programs constituting the software are installed in a general-purpose computer or the like.

[0272] FIG. 29 is a block diagram showing an example of the configuration of an embodiment of a computer in which a program for executing the series of processes described above is installed.

[0273] The program can be recorded in advance on the hard disk 905 or ROM 903 as a recording medium built into the computer.

[0274] Alternatively, the program can be stored (recorded) in a removable recording medium 911 driven by the drive 909. Such a removable recording medium 911 can be provided as a so-called package software. Here, examples of the removable recording medium 911 include a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, and a semiconductor memory.

[0275] The program can be installed in the computer from the removable recording medium 911 as described above, or can be downloaded to the computer via a communication network or a broadcasting network and installed in the built-in hard disk 905. That is, the program can be transferred to the computer wirelessly from a download site via an artificial satellite for digital satellite broadcasting, or transferred to the computer by wire via a network such as a LAN (Local Area Network) or the Internet.

[0276] The computer includes a CPU (Central Processing Unit) 902 , to which an input / output interface 910 is connected via a bus 901 .

[0277] When a user inputs a command by operating an input unit 907 via an input / output interface 910, the CPU 902 executes a program stored in a read only memory (ROM) 903 in accordance with the command. Alternatively, the CPU 902 loads a program stored in a hard disk 905 into a random access memory (RAM) 904 and executes the program.

[0278] As a result, the CPU 902 performs processing according to the above-mentioned flowchart or processing performed by the configuration of the above-mentioned block diagram. Then, the CPU 902 outputs the processing result from the output unit 906 via the input / output interface 910, or transmits it from the communication unit 908, or further records it on the hard disk 905, as necessary.

[0279] The input unit 907 is composed of a keyboard, a mouse, a microphone, etc. The output unit 906 is composed of an LCD (Liquid Crystal Display), a speaker, etc.

[0280] Here, in this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing executed in parallel or individually (for example, parallel processing or processing by objects).

[0281] The program may be processed by one computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to a remote computer for execution.

[0282] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.

[0283] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0284] For example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0285] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0286] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices.

[0287] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0288] The present technology can have the following configurations.

[0289] <1> an image processing unit that sets an image size of a tile image forming a grid image according to a target image size or a target bit rate of the tile image stored in a High Efficiency Image File Format (HEIF) file, and generates the tile image of that image size; Image processing device. <2> The image processing unit sets an image size of the tile image so that the image size or the bit rate of the tile image is equal to or smaller than a target image size or a target bit rate. <1> The image processing device according to claim 1 . <3> The image processing unit sets an image size of the tile image in accordance with a parameter set in response to a user operation. <1> or <2> The image processing device according to claim 1 . <4> The image processing unit sets an image size of the tile image in accordance with an image size of the grid image that is set in response to a user operation. <1> or <2> The image processing device according to claim 1 . <5> The image processing unit sets an image size of the tile image in accordance with an image quality of the tile image that is set in response to a user operation. <3> The image processing device according to claim 1 . <6> The image processing unit sets the target image size or the target bit rate in response to external information. <1> Or <5> 13. The image processing device according to claim 12, <7> The image processing unit sets the target image size or the target bit rate in response to a user operation. <6> The image processing device according to claim 1 . <8> The image processing unit sets the target image size or the target bit rate in response to information from an external device. <6> The image processing device according to claim 1 . <9> The image processing unit sets the target image size or the target bit rate according to one or more of a profile, a level, and a tier of a codec supported by the external device. <8> The image processing device according to claim 1 . <10> The image processing unit sets the target image size or the target bit rate in accordance with a connection method with the external device. <8> The image processing device according to claim 1 . <11> The image processing unit sets the target image size or the target bit rate in accordance with the performance of the external device. <8> The image processing device according to claim 1 . <12> The image processing unit sets an image size of the tile image in accordance with the target image size and the target bit rate. <1> Or <11> The image processing device according to claim 1 . <13> and setting an image size of the tile images forming a grid image according to a target image size or a target bit rate of the tile images stored in a High Efficiency Image File Format (HEIF) file, and generating the tile images of that image size. Image processing methods. <14> and an image processing unit that sets the image size of tile images that form a grid image and are stored in a High Efficiency Image File Format (HEIF) file in accordance with external information and generates the tile images with that image size. Image processing device. <15> The image processing unit sets an image size of the tile image in response to a user operation. <14> The image processing device according to claim 1 . <16> The image processing unit sets an image size of the tile image in accordance with information from an external device. <14> The image processing device according to claim 1 . <17> The image processing unit sets an image size of the tile image in accordance with one or more of a profile, a level, and a tier of a codec supported by the external device. <16> The image processing device according to claim 1 . <18> The image processing unit sets an image size of the tile image in accordance with a connection method with the external device. <16> The image processing device according to claim 1 . <19> The image processing unit sets an image size of the tile image in accordance with the performance of the external device. <16> The image processing device according to claim 1 . <20> The method includes setting the image size of tile images that form a grid image and are stored in a High Efficiency Image File Format (HEIF) file in accordance with external information, and generating the tile images with the image size. Image processing methods. [Explanation of symbols]

[0290] 10 digital camera, 11 optical system, 13 signal processing section, 14 media, 15,16 I / F, 17 button / key, 18 touch panel, 19 liquid crystal panel, 20 viewfinder, 21 I / F, 41 optical system / image sensor control section, 42 encoding control section, 43 file control section, 44 media control section, 45 operation control section, 46 display control section, 47 UI control section, 110 image processing section, 111 setting section, 112 generation section, 901 bus, 902 CPU, 903 ROM, 904 RAM, 905 hard disk, 906 output section, 907 input section, 908 communication section, 909 drive, 910 input / output interface, 911 removable recording medium

Claims

1. an image processing unit that sets the image size of the tile images forming the grid image to be equal to or smaller than the target image size based on a target image size and a setting unit of the image size stored in a HEIF (High Efficiency Image File Format) file, and generates the tile images of that image size; An image processing device comprising:

2. The set unit is set in response to a user operation. The image processing device according to claim 1 .

3. The image processing unit sets an image size of the tile image based on an image size of the grid image. The image processing device according to claim 1 .

4. The image processing unit sets the target image size in response to external information. The image processing device according to claim 1 .

5. The image processing unit sets the target image size in response to a user operation. The image processing device according to claim 4.

6. The image processing unit sets the target image size in response to information from an external device. The image processing device according to claim 4.

7. The image processing unit sets the target image size according to one or more of a profile, a level, and a tier of a codec supported by the external device. The image processing device according to claim 6.

8. The image processing unit sets the target image size in accordance with a connection method with the external device. The image processing device according to claim 6.

9. The image processing unit sets the target image size in accordance with the performance of the external device. The image processing device according to claim 6.

10. The image processing unit sets the image size of the tile image based on the target bit rate and image quality of the tile image such that the image size of the tile image is equal to or smaller than the target image size and the bit rate of the tile image is equal to or smaller than the target bit rate. The image processing device according to claim 1 .

11. Based on a target image size and a setting unit of image size of a tile image forming a grid image stored in a HEIF (High Efficiency Image File Format) file, the image size of the tile image is set to be equal to or smaller than the target image size, and the tile image of the image size is generated. Image processing methods.

12. an image processing unit that sets an image size of the tile image forming the grid image based on a target bit rate and image quality of the tile image stored in a High Efficiency Image File Format (HEIF) file so that the bit rate of the tile image is equal to or less than the target bit rate, and generates the tile image with that image size An image processing device comprising:

13. The image quality is set in response to a user operation. The image processing device according to claim 12.

14. The image processing unit sets an image size of the tile image based on an image size of the grid image as well. The image processing device according to claim 12.

15. The image processing unit sets the target bit rate in response to a user operation. The image processing device according to claim 12.

16. The image processing unit sets the target bit rate in response to information from an external device. The image processing device according to claim 12.

17. The image processing unit sets the target bit rate in accordance with one or more of a profile, a level, and a tier of a codec supported by the external device. The image processing device according to claim 16.

18. The image processing unit sets the target bit rate in accordance with a connection method with the external device. The image processing device according to claim 16.

19. The image processing unit sets the target bit rate in accordance with the performance of the external device. The image processing device according to claim 16.

20. and setting an image size of the tile image forming the grid image based on a target bit rate and image quality of the tile image stored in a High Efficiency Image File Format (HEIF) file, so that the bit rate of the tile image is equal to or less than the target bit rate, and generating the tile image of that image size. Image processing methods.

Images