Information processing device, information processing method, and program

Abstract

This makes it possible to identify whether the media data stored in a media file is content generated or modified by AI. [Solution] Obtain first media data output by a machine learning model. Generate first descriptive information describing the first media data. Generate second descriptive information describing information about the machine learning model used to output the first media data. Generate association information indicating the relationship between the first media data, the first descriptive information, and the second descriptive information. Generate a media file containing the first media data, the first descriptive information, the second descriptive information, and the association information.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] In recent years, with the development of AI processing technology, a technology called generative AI has been developed that learns various data to generate a machine learning model and generates or modifies various contents by providing input data.

[0003] For example, image generation AI that generates a new image by inputting text information to a model that has learned many images, and a technology that enables chatting as if having a conversation by inputting text have also been developed. In addition, a machine learning model can generate a summary of a long text, generate a new sentence, or generate a completely new image from a plurality of images. According to such technology, it is possible to generate an image representing a person who does not exist from a plurality of images, or to generate a color image from a black-and-white image. In addition, it is possible to modify an input image by inputting an image and text to a machine learning model. In addition, it is possible to generate various contents by utilizing generative AI, such as generating a video or audio, or generating program code. Furthermore, a technology has been developed that generates a rendering result from an arbitrary free viewpoint by a machine learning model trained based on images taken from a plurality of viewpoints. Technologies called NeRF (Neural Radiance Field) or Gaussian Plating enable rendering of images from a three-dimensional free viewpoint from a plurality of two-dimensional images.

[0004] Images taken with a camera or smartphone, as well as images processed by image analysis services, are typically stored on memory cards or other storage devices. Media data such as images and videos generated by generative AI are also stored on memory cards or other storage devices, similar to images taken with a camera or smartphone, when stored as media content.

[0005] Images are typically encoded to reduce their data size on storage devices. Many codec standards can be used for encoding, including JPEG, H.264 (AVC), H.265 (HEVC), H.266 (VVC), and AV1. Similarly, codec standards such as NNR, which define a compressed neural network representation with interchangeable parameters and weights for machine learning-trained neural networks used in multimedia analysis and processing, media encoding, data analysis, and data generation and modification, can also be used for encoding. Compression encoding of 3D data, such as point cloud data and mesh data, can also be used in a similar manner.

[0006] To store this encoded, compressed data in a file, a normative file structure including metadata is defined. This structure specifies how the stored data is associated and the format of the metadata. The ISO-based media file format (ISOBMFF, ISO / IEC 14496-12) is one such file format.

[0007] ISOBMFF is used for transmission over local storage, networks, or other bitstream distribution mechanisms. ISOBMFF is a well-known, flexible, and extensible file format that encapsulates and describes encoded timed or untimed media data or bitstreams. Several extensions exist for this file format. For example, ISO / IEC 14496-15 specifies encapsulation tools for various NAL (Network Abstraction Layer) unit-based video encoding formats. Examples of such encoding formats include AVC (Advanced Video Coding), SVC (Scalable Video Coding), HEVC (High Efficiency Video Coding), L-HEVC (Layered HEVC), and VVC (Versatile Video Coding).

[0008] Another example of file format extensions is ISO / IEC 23090-2, which defines the Omnidirectional Media Application Format (OMAF). Yet another example of file format extensions is ISO / IEC 23090-10 and ISO / IEC 23090-18, which define the transmission of Visual Volumetric Video-based Coding (V3C) media data and Geometry-based Point Cloud Compression (G-PCC) media data.

[0009] Another example of file format extensions is the High Efficiency Image File Format (ISO / IEC 23008-12, HEIF), which specifies tools for encapsulating still images or sequences of still images, such as HEVC still images, into files.

[0010] These file formats are standards developed by the Moving Picture Experts Group (MPEG) for storing and sharing images and image sequences, and they define the file structure in an object-oriented manner.

[0011] Patent Document 1 discloses a method for identifying regions within an image stored in a HEIF file as Region items and associating them with the stored image, thereby enabling the identification of regions within an image, and for adding annotation information to the identified regions within the image.

[0012] Furthermore, Patent Document 2 discloses a method for storing information used in AI-based detection of content elements within an image as metadata in a media file. This makes it possible to identify information related to the inference process by recording the results of the inference process that detects regions within an image using AI technology. [Prior art documents] [Patent Documents]

[0013] [Patent Document 1] International Publication No. 2021 / 204526 [Patent Document 2] U.S. Patent Application No. 2021-0349943 [Overview of the project] [Problems that the invention aims to solve]

[0014] The methods described in Patent Documents 1 and 2 allow for recording the results of inferences made using AI and identifying how the inferences were performed, but they do not anticipate handling information related to the generation of the media content itself using AI. In other words, the methods described in Patent Documents 1 and 2 cannot identify that the data that constitutes the media content within a file is data generated by the inference process of an AI called a generative AI, nor can they grasp the process by which such AI-generated media data was generated. Furthermore, since the copyright of such content data cannot be identified, it is impossible to determine whether the use of the media data constitutes copyright infringement. In addition, if the conditions used by the AI ​​when generating the media content could be identified, it would be possible to regenerate the media content by changing those conditions, but it is not possible to identify such conditions.

[0015] The present invention aims to make it possible to identify media data stored in a media file as content generated or modified by AI. [Means for solving the problem]

[0016] To achieve the object of the present invention, for example, an information processing apparatus according to one embodiment has the following configuration: a first acquisition means for acquiring first media data output by a machine learning model; a first generation means for generating first descriptive information describing the first media data; a second generation means for generating second descriptive information describing information about the machine learning model used when outputting the first media data; a third generation means for generating association information indicating the association between the first media data, the first descriptive information, and the second descriptive information; and a fourth generation means for generating a media file storing the first media data, the first descriptive information, the second descriptive information, and the association information. [Effects of the Invention]

[0017] It becomes possible to identify that the media data stored in the media file is content generated or modified by AI.

Brief Description of the Drawings

[0018] [Figure 1] A block diagram showing an example of the hardware configuration of the storage device. [Figure 2] A diagram for explaining the file structure of the HEIF file. [Figure 3] A diagram for explaining the structure of the AIGenerationInformationProperty. [Figure 4] A diagram for explaining the structure of the EntityToGroupBox. [Figure 5] A diagram for explaining the structure of the CopyrightProperty. [Figure 6] A diagram for explaining the structure of the UUIDBox. [Figure 7] A diagram for explaining the structure of the UUIDPropertyCopyrightProperty. [Figure 8] A flowchart showing an example of the media file generation process by the storage device. [Figure 9] A flowchart showing an example of the playback process of the media file by the storage device or the like. [Figure 10] A diagram for explaining the structure of the DeepLearningInformationEntityGroupBox. [Figure 11] A diagram for explaining the structure of the AIGenerationInformationEntityGroupBox. [Figure 12] A diagram showing an example of the configuration of the media file generated by the storage device. [Figure 13] A diagram showing another example of the configuration of the media file generated by the storage device.

Embodiments for Carrying Out the Invention

[0019] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0020] [Embodiment 1] The information processing device according to this embodiment acquires media data output by a machine learning model and generates first descriptive information (metadata) describing the media data. Next, the information processing device generates second metadata describing information about the machine learning model used when outputting the media data. Furthermore, the information processing device generates association information indicating the relationship between the media data, the first metadata, and the second metadata, and generates a media file storing this data and the association information. The following description of such an information processing device will be made with reference to Figures 1 to 13.

[0021] First, an example of the hardware configuration of the media file storage device 100 (hereinafter simply referred to as the storage device 100), which functions as an information processing device according to this embodiment, will be explained using the block diagram in Figure 1. As shown in Figure 1, each functional unit of the storage device 100 is connected to each other so as to be able to communicate via the system bus 109. In this embodiment, the case in which each functional unit shown in Figure 1 is implemented in hardware will be explained, but the storage device 100 may be configured so that some or all of the functional units are implemented in software (computer programs). In that case, this computer program is executed by the CPU 101, and as a result, the functions corresponding to each functional unit are realized.

[0022] The CPU 101 executes various processes using computer programs or data stored in the RAM 103 and ROM 102. In doing so, the CPU 101 controls the operation of the entire storage device 100 and executes or controls the various processes described as being performed by the storage device 100.

[0023] ROM 102 is an example of a non-volatile storage device capable of permanent information storage. ROM 102 stores configuration data for the storage device 100, computer programs and data related to the startup of the storage device 100, and computer programs and data related to the basic operation of the storage device 100. The data stored in ROM 102 includes, for example, parameters and display data necessary for the operation of each functional unit.

[0024] RAM 103 is an example of a volatile memory device capable of temporary information storage. RAM 103 has an area for storing computer programs and data loaded from ROM 102 or non-volatile memory 110, and an area for storing captured images input from imaging unit 104. Furthermore, RAM 103 has an area used by image processing unit 105 when performing various processes, an area for storing data received from external sources by communication unit 108, and a work area used by CPU 101 when performing various processes. In this way, RAM 103 can provide various areas as appropriate.

[0025] For example, RAM103 is used not only as an area for deploying computer programs, but also as a storage area (output buffer) for temporarily storing data output during the operation of various functional units.

[0026] The imaging unit 104 converts the optical image formed on the imaging surface of an image sensor (e.g., a CMOS sensor or CCD) via an optical system (not shown) into an optical system, and performs various image processing (noise reduction processing, gain processing, etc.) on the analog signal obtained by the photoelectric conversion. The imaging unit 104 then converts the processed analog signal into a digital signal using A / D conversion and outputs the digital signal as the captured image.

[0027] The image processing unit 105 performs various image processing operations on the image. The image processing operations according to this embodiment include, for example, development-related operations such as gamma conversion, color space conversion, white balance processing, or exposure correction. The image processing unit 105 may also be capable of performing image analysis operations or image synthesis operations that combine two or more images.

[0028] The image processing unit 105 includes an encoding / decoding unit 111, a metadata processing unit 112, an inference processing unit 113, and a learning processing unit 114. In this embodiment, for ease of understanding, it is described that the processing by these functional units (encoding / decoding unit 111, metadata processing unit 112, inference processing unit 113, and learning processing unit 114) is performed on a single piece of hardware, the image processing unit 105. However, the processing by these functional units may be performed on multiple pieces of hardware, and the configuration is not particularly limited as long as similar functions can be performed.

[0029] The encoding / decoding unit 111 is a video or still image codec conforming to H.265 (HEVC), H.264 (AVC), H.266 (VVC), AV1, or JPEG, etc. The encoding / decoding unit 111 performs encoding or decoding of images (still images or moving images (image sequences)) handled by the storage device 100. The encoding / decoding unit 111 may also perform encoding or decoding of data such as parameters and weights of a machine learning model generated by the learning processing unit 114, or media data such as audio data. Hereinafter, the machine learning model may be referred to as "AI". Furthermore, it may perform encoding or decoding of 3D data such as point cloud data, mesh data, or Gaussian splatting.

[0030] The metadata processing unit 112 acquires data encoded by the image encoding / decoding unit 111 (encoded data). The metadata processing unit 112 then generates a media file conforming to a predetermined file format (e.g., HEIF) that includes the encoded data and metadata related to the encoded data. In the following description, it will be assumed that an HEIF file conforming to the ISOBMFF specification is used as the media file, but the system is not limited to such media files as long as it can store similar information. Specifically, the metadata processing unit 112 performs analysis processing of the encoded data stored in an image file containing still images or image sequences, and generates information related to the still images or image sequences, or acquires parameters related to the encoded data. The metadata processing unit 112 then stores this information as metadata along with the encoded data in the image file. The metadata processing unit 112 can also generate image files conforming to other video file formats specified in MPEG or formats such as JPEG, not limited to HEIF. The encoded data to be acquired may be data previously stored in the ROM 102 or non-volatile memory 110, or data acquired via the communication unit 108 and stored in the RAM 103. Furthermore, the metadata processing unit 112 generates and stores metadata related to the media data generated and modified by the inference processing unit 113, as well as the input data input to the machine learning model when generating and modifying the media data. In the following, we will not particularly distinguish between generating media data by the machine learning model and generating media data as a result of modifying media data by the machine learning model, and will sometimes refer to "generating and modifying" media data.

[0031] In addition, the metadata processing unit 112 generates and stores data representing various inference results using machine learning models, as well as related metadata. For example, the metadata processing unit 112 generates and stores data representing the recognition results of regions within an image obtained by analyzing the image, as well as metadata indicating that data. Furthermore, the metadata processing unit 112 also generates and stores metadata related to the input data and algorithms used for training the learning processing unit 114. Moreover, the metadata processing unit 112 analyzes the metadata stored in the image file and performs metadata processing during the playback of still images or image sequences.

[0032] The inference processing unit 113 performs inference processing on input data using a learning model generated by the learning processing unit 114, or a learning model learned by an external device. The input data used in this inference processing is the input data corresponding to the learning model used. For example, if the inference processing unit 113 uses a learning model that detects regions within an image, it inputs an image as input data to the learning model, and as a result, it can detect people or subject regions within the image. The metadata processing unit 112 generates and stores the data that identifies the detected object or region. Also, if the inference processing unit 113 uses a learning model that generates an image from text data as input, it can input text data to the learning model and generate an image that is the result of the inference. Furthermore, if the inference processing unit 113 uses a learning model called NeRF that can reconstruct a 3D scene, it can input coordinates and viewing angle as input data to the learning model, infer the position, transparency, and color in 3D space, and use the inferred information to generate a rendered image from that viewpoint. This is then converted into image data to generate and store the image.

[0033] In addition, the inference processing unit 113 enables various inference processes related to media data by performing inference processing on various learning models and input data corresponding to such learning models. Here, the inference processing unit 113 can employ various learning models that are used in line with the development of AI technology in its inference processing. For example, the inference processing unit 113 can use learning models that produce various outputs, such as generating text by summarizing a large amount of text data, generating audio data from text data, and generating color image data from monochrome image data. The information input as input data may be data acquired in advance, or it may be information specified by the user by operating the operation input unit 107. The inference processing performed by the inference processing unit 113 may be a detection process such as object recognition from images stored in an image file, or it may be a generation process that generates or modifies the images themselves stored in the image file. Such processing may be performed, for example, by an external device or service that can communicate via the storage device 100 and the communication unit 108. In that case, the storage device 100 will acquire inference result data such as the detection results of subject objects and generated images from the external device. The inference processing performed by the inference processing unit 113 may be performed by a single learning model, or by various processes combining multiple learning models.

[0034] The learning processing unit 114 performs a learning process called machine learning using the dataset to be learned. The dataset to be learned can be image data acquired from the imaging unit 104, data acquired from an external device that can communicate via the storage device 100 and the communication unit 108, or information specified by the user by operating the operation input unit 107. The algorithm used for learning may be based on a program pre-stored in the ROM 102 of the storage device 100, program data acquired from an external device that can communicate via the storage device 100 and the communication unit 108, or information specified by the user by operating the operation input unit 107.

[0035] Furthermore, the learning process described as being performed by the learning processing unit 114 may be executed by, for example, an external device or service that can communicate via the storage device 100 and the communication unit 108. In that case, the storage device 100 may acquire the trained model data from the external device, or it may store the results on the external device and acquire them as referenced information. Also, there is no limit to the algorithm used for learning; learning can be performed using various algorithms with the same training data.

[0036] The display unit 106 is a display device having a liquid crystal display (LCD) or a touch panel screen. The display unit 106 may be a display device detachably connected to the storage device 100, or it may be a display device integrated with the storage device 100. The display unit 106 performs various display processing, such as displaying images (still images or moving images) being captured by the imaging unit 104 (live view display), or displaying information such as various settings / GUI (Graphical User Interface). The display unit 106 also displays images when the generated image file is played back. Furthermore, the display unit 106 may also display data generated and analyzed by the metadata processing unit 112 as identifiable information along with the image.

[0037] The operation input unit 107 is a user interface such as an operation button, switch, mouse, or keyboard, and can input various instructions to the CPU 101 by receiving user input. In the case where the display unit 106 is a touch panel screen, the operation input unit 107 may include a touch panel sensor.

[0038] The communication unit 108 is a communication interface for data communication with external devices. The communication unit 108 may be, for example, a network interface that connects to a network and transmits and receives transmission frames. In this case, the communication unit 108 may be, for example, a PHY and MAC (transmission media control processing) capable of wired LAN connection via Ethernet®. Furthermore, if the communication unit 108 is capable of connecting to a wireless LAN, the communication unit 108 may include a controller that performs wireless LAN control such as IEEE802.11a / b / g / n / ac / ax, an RF circuit, and an antenna.

[0039] The non-volatile memory 110 is a non-volatile information storage device with a large storage capacity, such as an SD card, CompactFlash®, or flash memory. For example, the non-volatile memory 110 may store image files generated in this embodiment, or it may store image files acquired from an external device via the communication unit 108.

[0040] The hardware configuration shown in Figure 1 is merely one example of a configuration that can realize the operation of the storage device 100 described below, and can be modified or altered as appropriate. For example, the imaging unit 104 in Figure 1 is integrated with the storage device 100, but it may be detachably connected to the storage device 100. Also, the image processing unit 105 may be a device that can be detachably attached to the storage device 100, or it may be an external device that can communicate with the storage device 100 via the communication unit 108.

[0041] Next, the generation of image files by the storage device 100 will be described. The image files generated by the storage device 100 can store multiple images and can include information associated with the stored images. In the following, HEIF is adopted as the file format for the image files, and a method for generating and storing metadata by deriving the necessary information in order to generate an image file (HEIF file) compliant with HEIF will be described. However, the file format of the media file used in this embodiment is not limited in this way, and may be other video file formats defined in MPEG, omnidirectional media application file formats, file formats that handle 3D data such as point cloud data, or formats such as JPEG. Furthermore, the media file in this embodiment is not limited to image files, and may be any form of media file that can store information about media data generated by AI processing. For example, media files such as text data files or audio data files may be used.

[0042] Next, the file structure of an HEIF file will be explained using Figure 2. As shown in Figure 2, an HEIF file 200 generally contains the following three boxes (storage areas).

[0043] The first box, 201, is FileTypeBox('ftyp'). This box stores the brand name that the HEIF file reader uses to identify the specification of the HEIF file.

[0044] The second box, 202, is MetaBox('meta'). As shown in Figure 2, this box 202 stores multiple types of descriptive information about the image, each separated into different boxes. The information stored in box 202 will be described later.

[0045] The third box, 203, is MediaDataBox('mdat'). Encoded data (images) 241-242 are stored in this box 203 as an encoded bitstream. In this embodiment, image data is generated by the inference processing unit 113, and the encoded image data is stored in box 203 as media data. Here, we will assume that image data is used as the dataset for which the learning processing unit 114 learns using a learning algorithm, and that such image data is stored in box 203. However, the media data used is not limited to images; if other media data is learned and generated, a bitstream of the corresponding media data is stored in box 203. This bitstream may be data compressed using a compression algorithm. Box 203 may also store a bitstream of compressed image data acquired from the imaging unit 104 and store data generated separately by the inference processing unit 113. In such a case, box 203 can store, for example, region data that can identify detected regions as data generated by the inference processing unit 113.

[0046] Box 203 stores the training model data 243 generated or acquired by the training processing unit 114. The training model data 243 may be a bitstream compressed by a compression algorithm such as NNR, or it may be data that can represent uncompressed parameters and weights.

[0047] Box 203 stores the inference input data 244-245 used by the inference processing unit 113 when performing inference processing using the learned model data 243. The inference input data 244-245 are stored as encoded bitstreams if the media data to be processed is image data or audio data. If the media data is text data or data that can be represented as metadata, the data compressed using a general-purpose compression algorithm may be stored as the inference input data 244-245.

[0048] Box 203 stores training data 247-248, which will be used as training datasets by the learning processing unit 114 for machine learning, and training algorithm data 246, which will be used during training. If the media data to be processed is image data or audio data, the training data 247-248 will be stored as encoded bitstreams. If the media data is text data or data that can be represented as metadata, data compressed using a general-purpose compression algorithm may be stored as training data 247-248. The training algorithm data 246 may be, for example, identification information that can refer to the training algorithm, program code data, or a pre-compiled executable program.

[0049] Furthermore, box 203 stores an Exif data block 249 containing information such as imaging data captured by the imaging unit 104. In the example shown in Figure 2, a configuration in which box 203 is used as the area for storing encoded data 241-242, learning model data 243, inference input data 244-245, learning data 247-248, learning algorithm data 246, and Exif data block 249 will be described. However, instead of box 203, a box structure such as 'idat' or 'imda' may be used as the area for storing such data. In the following description, the encoded data 241-242 stored in box 203, and possibly the learning data 247-248 or the inference input data 244-245, will be referred to as "image" or "encoded data," with appropriate variations in terminology.

[0050] Furthermore, if media data such as video or audio, image sequences, timed metadata, or timed text data is stored, it may be stored separately in a MovieBox ('moov') not shown. This box can store metadata to describe various information related to presentations such as video or audio stored in image files. If the data to be stored is an image sequence, metadata is stored using a mechanism that describes various information related to video, but unlike video, timed information is optional.

[0051] Box 211 is a HandlerReferenceBox('hdlr') and stores a declaration of the handler type for analyzing the structure of box 202. In the HEIF file 200 generated by the storage device 100 according to this embodiment, the metadata describing the untimed data stored in box 202 targets still images. Therefore, box 211 is set to the handler type name 'pict' to identify that it targets still images.

[0052] Box 212 is PrimaryItemBox('pitm'), which specifies the identifier (item ID) of the image data that will be designated as the representative item among the image items stored in the HEIF file 200. In this embodiment, the image item designated as the first priority item in Box 212 is displayed as the image that is normally displayed.

[0053] Box 213 is an ItemLocationBox('iloc') and stores information indicating the storage location of each information item, including image items, in the HEIF file 200. Typically, Box 213 describes the storage location of an image item as a byte offset from the beginning of the HEIF file 200 and the data length from that beginning. In other words, Box 213 can store information that identifies the location of the encoded data, training model data, inference input data, training algorithm data, and Exif data block stored in Box 203. For derived items, the information stored in Box 213 indicates that no data exists in Box 203. If no data exists in Box 203, it means that there is no box data structure in Box 203, or the data for the derived item is stored in Box 217 within Box 202.

[0054] Box 214 is an ItemInfoBox('iinf'). Box 214 stores information that defines basic information (item information) for all items contained in the HEIF file 200, such as the item ID or item type indicating the item type. The item information can include not only image items such as encoded image items or derived image items, but also items that indicate various data related to AI processing, such as learning model items, inference input items, learning algorithm items, or Exif information items indicating Exif data blocks. The inference input data stored in box 214 only needs to be information that specifies an item according to the type of data; for example, for image data, it will be information that defines it as an image item. Similarly, for text data, information that defines it as a text item will be stored in box 214, and for metadata, information that defines it as a metadata item will be stored in box 214. Furthermore, information regarding learning model items may be stored in box 214 as a deductive information item of an item type URI defined for the purpose of detecting content elements.

[0055] Box 215 is an ItemReferenceBox('iref') and stores information describing the relationships between items contained in the HEIF file 200 (relationship information). In the case where an image item is an captured image, Box 215 stores relationship information describing the relationship between the image item and its shooting information (such as Exif data). In the case where multiple image items are related as derived images, Box 215 stores relationship information describing the relationships between those image items. An item reference type is specified for each item relationship, making the type of item reference identifiable. In Box 215, the reference relationships between each item are described by writing the item ID specified in Box 214 in the from_item_ID and to_item_ID fields, respectively. Box 215 also stores relationship information describing the relationships between items related to AI processing. Note that the relationships between items related to AI processing may be described in Box 215 or in the EntityToGroupBox described later, and there are no particular limitations on the description location as long as it is done in a predetermined way.

[0056] Box 216 is an ItemPropertiesBox('iprp') that stores various attribute information (item properties) about the information items contained in the HEIF file 200. More specifically, box 216 includes an ItemPropertyContainerBox('ipco'), which is box 221 that describes the attribute information, and an ItemPropertyAssociation('ipma'), which is box 222 that describes information indicating the association between the attribute information and each item. Box 221 may store attribute information such as entry data indicating the HEVC parameter set required for decoding an HEVC image item, or entry data indicating the width and height of an image item in pixels. Here, the item properties may be attribute information that can specify user-unique information.

[0057] The UUIDProperty('uuid') shown in Figure 7 is an example of attribute information that can store user-defined properties. User-defined information may include, for example, vendor-specific information or information defined in industry standards that have been uniquely extended using the standards defined in MPEG. The UUIDProperty shown in Figure 7 includes the 4-character code 'uuid' shown in Definition 701, and this 4-character code is used to identify the UUIDProperty. Furthermore, the UUIDProperty includes extended_type, which allows identification of user-specific extension types as shown in Definition 702. The specification of the 16-byte code to be specified for the extension type is done in the manner specified in IETF RFC4122 and ISO / IEC 9834-8. User-defined properties identified by the 4-character code in Definition 701 and the extension type in Definition 702 may include attribute information that can be arbitrarily specified by the user in field 703. Property information stored as a uuid property can be associated with an item or entity group, just like other property definitions. Since the uuid property is user-defined attribute information, the attribute information specified here will normally be ignored by file processing devices that cannot identify the specified extension type.

[0058] The UUIDProperty shown in Figure 7 is stored in a file as attribute information that can be directly associated with an item or entity group. On the other hand, the UUIDBox shown in Figure 6 is an example of user-defined metadata that can be stored in any metadata hierarchy. User-defined information in a UUIDBox may include, for example, vendor-specific information or information defined in industry standards that have been uniquely extended using the standards defined in MPEG, similar to the information in a UUIDProperty. The UUIDBox shown in Figure 6 includes the four-character code 'uuid' as shown in Definition 601, and this four-character code is used to identify the UUIDBox. Furthermore, the UUIDBox includes extended_type, which allows identification of a user-specific extension type as shown in Definition 602. The specification of the 16-byte code to be specified for the extension type is done in the manner specified in IETF RFC4122 and ISO / IEC 9834-8. The user-defined metadata identified by the four-character code in Definition 601 and the extension type in Definition 602 may include metadata that can be arbitrarily specified by the user in field 603. Note that since the UUID box is user-defined metadata, the attribute information specified here will normally be ignored by file processing devices that cannot identify the specified extension type. Unlike UUIDProperty, UUIDBox can be specified in any Box hierarchy, allowing for user-specific definitions, including its scope of application. On the other hand, UUIDProperty differs from UUIDBox in that it can be specified as information confined to property association.

[0059] Using a UUID box to define metadata allows for the storage of application-specific metadata, which can be uniquely defined, within a file. Furthermore, the Content Authentication Initiative (CAI), an organization dedicated to verifying the authenticity of content, has established the C2PA standard as a way to embed information such as editing content or rights information into digital data like images or videos. In this standard as well, the UUID box is applied to define how metadata defined by C2PA can be stored in media files defined by MPEG.

[0060] Additionally, TransformativeProperty may be stored as attribute information that can be specified as an item property, intended to transform and display the image when it is output. TransformativeProperty may be used to store, for example, data indicating rotation information for displaying an image rotated, or data indicating cropping information for displaying an image cropped (cut out).

[0061] Next, box 222 ('ipma') uses the information item ID (item ID) to store entry data for each item indicating its association with the attribute information stored in box 221. Note that for items that do not have attribute information to associate with other items, such as Exif information items, no entry data indicating the association will be stored.

[0062] Box 217 is an ItemDataBox('idat') and stores data about items contained in the HEIF file 200. Box 217 stores a data structure for describing, for example, a derived image item. For example, for an item of type 'grid' as shown in Box 214, the data structure for a grid derived image item, which is defined by reconstructing the input image in a predetermined grid order, is specified in Box 217. For the input image of the derived image item, an item reference of type dimg reference is specified using Box 215. Note that if a derived item does not have a data structure, for example, for the identity derived image item 'iden', no data structure is stored in Box 217.

[0063] Box 218 is a GroupListBox('grpl'). Box 218 stores metadata for grouping and storing entities such as items or tracks contained in the HEIF file 200. Box 218 stores boxes that are extended definitions of EntityToGroupBox as shown in Figure 4 for each grouping type parameter. EntityToGroupBox includes the grouping_type shown in definition 401. grouping_type includes a 4-character code defined for each grouping type, and this 4-character code is used to identify the grouping type of EntityToGroupBox. The grouping type is a concept that defines the relationships between multiple entities included in a group. EntityToGroupBox includes group_id402 to uniquely identify the entity group itself, and num_entities_in_group403 indicating the number of entities included in the entity group. EntityToGroupBox also includes entity_id404 the number specified in num_entities_in_group. In entity_id404, it is possible to specify an item ID that identifies an item defined in box 214, or a track ID that identifies a single track of a presentation contained in a MovieBox (not shown). Furthermore, for entity groups of a specific group type, it is possible to specify a group_id to identify other entity groups. EntityToGroupBox is also configured to allow extension definitions for each grouping type, and is used as a structure in part 405 to define extension parameters according to the grouping type. By identifying the grouping type in EntityToGroupBox in this way, it becomes possible to treat multiple image items or entities such as tracks included in a group as a meaningful group unit.

[0064] In the box configuration shown in Figure 2, box 231 is stored as an extension of EntityToGroupBox with grouping type 'dlif (DeepLearningInformationEntityGroupBox)', which is a group type for grouping information used in machine learning processing. Additionally, box 232 is stored as an extension of EntityToGroupBox with grouping type 'aigi (AIGenerationInformationEntityGroupBox)', which is a group type for grouping information used in generating and modifying media data using a machine learning model.

[0065] Box 231 is the box where the extension definition for EntityToGroupBox is performed, as mentioned above. Here, box 231 contains the definitions 402 to 404 included in EntityToGroupBox, and grouping_type401 contains 'dlif' as the 4-character code (4CC) that identifies DeepLearningInformationEntityGroupBox.

[0066] Furthermore, the entity_id specifies an item ID indicating the learning model generated as a result of learning, an item ID indicating the learning algorithm, and an item ID indicating the learning data that constitutes the learning dataset. These specifications allow for the identification of information related to a series of machine learning processes. In addition, the entity_id can also specify the group ID of the 'dlif' entity group, which is a group of separately stored machine learning process information. This allows for the identification of information related to differential learning as a group.

[0067] Box 232 is the box where the extension definition for EntityToGroupBox is performed, as mentioned earlier. Here, box 232 contains the definitions 402 to 404 included in EntityToGroupBox, and grouping_type401 contains 'aigi' as the 4-character code (4CC) that identifies AIGenerationInformationEntityGroupBox.

[0068] Furthermore, the entity_id specifies an item ID indicating media data such as images generated as a result of inference, an item ID indicating the learning model used in the inference process when generating media data, and an item ID indicating the inference input data which is the input dataset used for inference. These specifications allow for the identification of information related to the inference process that generates and modifies a series of media data. Note that the item ID indicating the learning model may be an item ID indicating the learning model generated as a result of the learning specified in the aforementioned 'dlif' entity group. The detailed definitions of the 'dlif' entity group and the 'aigi' entity group will be described later.

[0069] Next, we will explain the definition of an item property that allows identification of whether or not media data is AI-generated media data that can be stored in the HEIF file 200. Figure 3 shows the data structure of AIGeneratedInformationPropety, which is an item property that can be stored in box 221 of the HEIF file 200. This AIGeneratedInformationPropety is an extension of ItemFullProperty and has property_type301('aign'). AIGeneratedInformationPropety also includes parameters generation_type302, generation_media_type303, input_data_type304, and learning_data_type305. Note that the definition described as being written in AIGeneratedInformationPropety may be configured to be defined as AIGeneratedInformationBox rather than as a Property, similar to the definitions in UUIDPropety and UUIDBox. In this case, for media data with time-sensitive information, such as video or audio, the definition can also be stored as an optional Box specified as SampleEntry within SampleDescriptionBox, which specifies the configuration of the sample contained within TrackBox('trak') which is included in MovieBox('moov') (not shown in the diagram).

[0070] Such an AIGeneratedInformationPropety may be defined as follows: An AIGeneratedInformationPropety is a Descriptive item property identified by property_type301('aign'). An AIGeneratedInformationPropety identifies that the media data corresponding to the associated item is content generated and modified using AI. generation_type302 is an unsigned integer that identifies the type of content generated and modified using AI. Here, a value of 0 is undefined. Note that if the type of content is unknown, a value of 0 may be specified for generation_type302. Here, a value of 1 indicates that the media data is AI-generated media data, and a value of 2 indicates that the media data is partially modified by AI. Also, here, a value of 3 indicates that the media data has been processed using AI. Values ​​from 4 onwards are reserved.

[0071] A value of 1 indicates that the media data is a (new) image generated based on text information, or a (new) document generated based on an image. A value of 2 indicates that the media data is partially modified (partially non-existent) content, such as a fake image. A value of 3 indicates that the media data is a corrected version of the original media data (refined through AI-based correction processing, such as improved accuracy or noise reduction).

[0072] `generation_media_type303` is an unsigned integer that identifies the media data type of the item to which this property is associated. The media type specified in this parameter should be the same as the information specified as the content type for the item defined as an entry in `ItemInformationBox`. If time-limited media data is used, the same information as the media data type specified as the media handler should be specified in `generation_media_type303`. Note that the media data box value 0 is not defined. If the media type is unknown, the value 0 may be specified in `generation_media_type303`. Here, a value of 1 indicates that the media data is a still image, a value of 2 indicates that the media data is a video, a value of 3 indicates that the media data is audio, a value of 4 indicates that the media data is text data, a value of 5 indicates that the media data is metadata, a value of 6 indicates that the media data is 3D still image data, and a value of 7 indicates that the media data is 3D video data. Values ​​8 and above are reserved.

[0073] `input_data_type304` is information that identifies the type of data input when generating or modifying the media data associated with this property. The values ​​that can be defined in this parameter are the same as the values ​​defined in `generation_media_type303`. If inference processing is performed using input data containing multiple types of data, this parameter may include a number of parameters equal to the number of types. In that case, `generation_media_type303` must have a data structure that allows multiple parameters to be specified. Also, if the input data is associated, information that matches the media type of the associated data should be specified.

[0074] `learning_data_type305` is a value that identifies what type of data the model used to train the media data to which this property is associated was trained. The values ​​that can be defined in this parameter are the same as the values ​​defined in `generation_media_type303`. For models trained using data containing multiple data types, this parameter may be configured to allow specifying a number of parameters equal to the number of data types. In that case, `learning_data_type305` must have a data structure that allows specifying multiple parameters.

[0075] In this embodiment, to identify whether an item was generated or modified by AI, such information is stored in a property that can be associated with the item. However, this identification may also be performed by associating information that identifies whether an item was generated or modified by AI with the item (without using a property). Details on how to perform this association will be described later. By performing such an association using a property, it becomes easy to identify whether an item was generated or modified by AI simply by checking the property associated with the item. Note that the AIGeneratedInformationPropety described in this embodiment is just an example, and it does not need to include all the parameters described above, and may also include additional parameters. Furthermore, similar information may be described in an identifiable way using a different 4CC.

[0076] Next, we will explain the definition of an item property that allows for the identification of copyright information for media data that can be stored in the HEIF file 200. Figure 5 shows an example of the data structure of CopyrightProperty, an item property that can be stored in box 221 of the HEIF file 200. This CopyrightProperty is an extension of ItemFullProperty and has property_type501('cprt'). CopyrightProperty also includes parameters pad502, language503, and notice504. The CopyrightProperty according to this embodiment is a definition that allows CopyrightBox, as defined in ISO / IEC 14496-12 (ISOBMFF), to be treated as a property.

[0077] Such a CopyrightProperty can be defined as follows: A CopyrightProperty is a Descriptive item property identified by property_type501. A CopyrightProperty contains a copyright declaration applicable to the media data corresponding to the associated item. In this embodiment, the copyright declaration associated with media data is information regarding the copyright of the media data (copyright information), and includes copyright information for the media data itself, or copyright information for data used as training data for a machine learning model used to generate or modify the media data. Here, multiple CopyrightProperties using different language codes may be associated with the same item. Furthermore, it is possible to associate CopyrightProperties with different copyright declarations for each item.

[0078] In this embodiment, the copyright information stored in association with media data includes information indicating that the media data is a copyrighted work, information indicating that the training data of the machine learning model that output the media data includes the copyrighted work, or information indicating that the input data to the machine learning data when the media data was output includes the copyrighted work. Here, the copyright information may be, for example, information indicating the year the copyrighted work was published and the copyright holder of the copyrighted work, or it may be information indicating only whether or not the associated media data was output by a learning model that uses the copyrighted work as training data, or it may be information that allows access to the copyrighted work (e.g., a URL). Furthermore, such copyright information may include information shown in text, or it may include flag information indicating that the associated media data was output by a learning model that uses the copyrighted work as training data, and its form is not particularly limited.

[0079] Pad502 is a parameter whose value is always specified as 0, and it is a 1-bit field included for byte alignment. Language503 declares the language code for the following text in the form of a 3-character code as defined in ISO 639-2. Each character is specified as the difference between its ASCII value and 0x60. Since this language code is limited to three lowercase letters, these values ​​are strictly positive. Notice504 specifies the copyright notice.

[0080] As shown in this embodiment, by defining CopyrightBox as CopyrightProperty, it becomes possible to describe copyright information for each item contained in a file. In other words, it becomes possible to specify copyright information for each item, such as still images, contained in a single file.

[0081] Next, we will describe the definition for grouping and identifying the information used in machine learning processing, which can be stored in the HEIF file 200. Figure 10 shows an example of the data structure of DeepLearningInformationEntityGroupBox, which is an entity group that groups the information used in machine learning processing, and which can be stored in box 218 of the HEIF file. This DeepLearningInformationEntityGroupBox is an extension of EntityToGroupBox and has grouping_type1001('dlif'). No additional parameters specific to this entity group type are defined here.

[0082] Such a DeepLearningInformationEntityGroupBox can be defined as follows. A DeepLearningInformationEntityGroupBox is identified by grouping_type'dlif'. A DeepLearningInformationEntityGroupBox is a machine learning information group that associates a learning model with a learning algorithm and a learning dataset. When a unique ID is used in a DeepLearningInformationEntityGroupBox, a machine learning information group can specify separately grouped entity groups as entities of the machine learning information group. For example, in a DeepLearningInformationEntityGroupBox, an entity group for grouping learning algorithms may be defined separately and its group ID may be specified, or an entity group for grouping learning models may be defined separately and multiple learning models, including learning models using differential learning, may be grouped and specified as a single learning model group.

[0083] The number of entities within a machine learning information group must be three or more. One entity_id value represents an item or entity group indicating the learning model generated as a result of learning. Another entity_id value represents an item or entity group indicating the algorithm information used for learning. The remaining entity_id values ​​represent items or tracks of data that constitute the dataset used for learning. If it is necessary to associate multiple types of data for learning, the entity_id values ​​may be made to correspond to the associated data. Such data associations may be made using separately defined group or item references, and the entity_id value may specify only one type of data (for which such associations are made). Flags may also be used to identify when multiple entity_id values ​​are specified as training data.

[0084] Furthermore, entities included in a machine learning information group do not need to specify all of the information about the machine learning model using entity_id. For example, the entity_id may be configured to indicate only the learning algorithm information and the learning dataset. Alternatively, for example, flags may be used to identify entities included in this entity group, and the information specified within the group may switch depending on the value of flags (for example, depending on the value of flags, the configuration may switch between one in which the entity_id indicates only the learning algorithm information and the learning dataset, and another in which the entity_id indicates different data).

[0085] Furthermore, some or all of the entities included in this entity group may be associated using item references. In that case, the associated entities can be specified, for example, by defining a reference type that associates training algorithm information with a training model, or by defining a reference type that associates a training dataset with a training model. In this way, as long as each item is associated in the same manner, there are no particular limitations on where such association information is described.

[0086] Next, we will explain the definition for grouping and identifying information generated and modified using a learning model that can be stored in the HEIF file 200. Figure 11 shows an example of the data structure of AIGenerationInformationEntityGroupBox, which is an entity group that groups information generated and modified using a learning model that can be stored in box 218 of the HEIF file. This AIGenerationInformationEntityGroupBox is an extension of EntityToGroupBox and has grouping_type1101('aigi'). No additional parameters specific to this entity group type are defined here.

[0087] Such an AIGenerationInformationEntityGroupBox can be defined as follows: The AIGenerationInformationEntityGroupBox is identified by grouping_type'aigi'. The AIGenerationInformationEntityGroupBox is an AI generation / modification information group that stores association information indicating the relationship between generated / modified media data, the learning model used for said generation / modification, and the input dataset used for said generation / modification. In the following, "information when media data is generated / modified" refers to information associated with such generated / modified media data, indicating the learning model used for said generation / modification or the input dataset used for said generation / modification.

[0088] When a unique ID is used in AIGenerationInformationEntityGroupBox, the AI ​​Generation / Modification Information Group can specify a separately grouped entity group as an entity of the AI ​​Generation / Modification Information Group. For example, in AIGenerationInformationEntityGroupBox, an entity group for grouping learning models may be defined separately, and multiple learning models, including learning models using differential learning, may be grouped and specified as a single learning model group.

[0089] The number of entities within an AI-generated / modified information group must be three or more. One entity_id value represents an item, track, or entity group indicating media data generated or modified as a result of inference processing using a learning model. Another entity_id value represents an item or entity group indicating the learning model used for generation or modification. The remaining entity_id values ​​represent items or tracks of data that serve as the input dataset used for the inference processing for generation and modification. If it is necessary to associate multiple types of input data for inference processing, the entity_id values ​​may be made to correspond to the associated input data. Such data associations may be made using separately defined group or item references, and the entity_id value may specify only one type of data (for which such associations are made). Flags may also be used to identify when a set of multiple entity_id values ​​is specified as input data.

[0090] Furthermore, entities included in the AI-generated / modified information group do not need to specify all of the information generated or modified from media data using entity_id. For example, entities included in this entity group can be identified using flags, and the information specified within the group can be switched depending on the value of flags (for example, depending on the value of flags, the configuration can switch between one in which only the learning model information (information indicating the learning model) and input dataset are shown by entity_id, and another in which only the generated / modified media data and input dataset are shown by entity_id).

[0091] Furthermore, some or all of the entities included in this entity group may be associated using item references. In that case, the associated entities can be specified, for example, by defining a reference type that associates an input dataset with a learning model, or by defining a reference type that associates media data with the input dataset used to generate that media data. In this way, as long as each item is associated in a similar manner, there are no particular limitations on where such association information is described.

[0092] Next, we will describe a definition for associating and storing information about AI generation and modification with any media item stored in the HEIF file 200, using information about AI processing configured according to the above definition. If the media data generated and modified by the AI ​​is a still image, the media data consists of data that encodes the still image and an image item that identifies it. The item ID of the still image item generated and modified by the AI, the item ID indicating the learning model information used when generating and modifying it, and the item ID indicating the data input when generating and modifying it are specified in the entity of AIGenerationInformationEntityGroupBox and grouped together. This allows the information used for generation and modification to be identified as a group by grouping the learning model, the data input to the learning model, and the image data generated and modified as a result of inputting data to the learning model.

[0093] In this embodiment, the storage device 100 groups items representing image encoding data 241-242, learning model data 243, and inference input data 244-245 in the following data in box 203, thereby enabling the identification of information generated and modified from still images as a group (associated within the group). Furthermore, by associating AIGeneratedInformationPropety with the group ID representing this group or the item ID representing the generated and modified still image, the grouped information can be identified as a property (associated within the property). Additionally, by associating CopyrightProperty with the group ID representing this group or the item ID representing the generated and modified still image, copyright information for items included in the group specified by the group ID, or for items specified by the item ID, can be specified.

[0094] Furthermore, the storage device 100 according to this embodiment groups the item ID of the learning model information and the item ID that identifies the learning algorithm information and learning dataset used when learning the learning model in the DeepLearningInformationEntityGroupBox entity, in order to identify the history of model generation of the learning model specified as an entity in the AIGenerationInformationEntityGroupBox. This makes it possible to identify the learning dataset and its learning algorithm used when generating the learning model as a group associated with the learning model. In the storage device 100 according to this embodiment, the items representing the learning model data 243, learning algorithm data 246, and learning data 247-248 are grouped in the following data of box 203, making it possible to identify the information when the learning model was generated as a group. In addition, by associating CopyrightProperty with the group ID representing this group, the item ID representing the learning algorithm data, or the item ID representing the learning data, it becomes possible to specify the copyright information of the items included in the group specified by the group ID, or the items specified by the item ID.

[0095] Here, with reference to Figure 12, an example of an output file produced by the storage device 100 according to this embodiment will be described. The image file according to this embodiment is configured to store the image generation process of the AI ​​generated by two Entity Groups, the AI ​​Generation Information Entity Group and the Deep Learning Information Entity Group, in a file that can be identified by referring to the file data structure. Furthermore, the storage device 100 according to this embodiment can use this image file to generate a learning model by learning using the image and text information related to the image as training data. When text information is input to the learning model to generate an image, the text information is stored in the file together with the generated image (associated with the information used when generating the image). This can be used, for example, when storing information about a learning model called an image generation AI, such as Stable Diffusion, which outputs a 2D image using media data such as text information as input data, in a file together with the output image from the learning model.

[0096] In the example in Figure 12, HEVC encoded data (HEVC Image Data) is stored, as shown in description 1204 corresponding to the 'mdat' box 203, and indicated by descriptions 1230-1231 corresponding to the encoded data 241-242. Description 1230 indicates the image generated by the AI, and description 1231 indicates a thumbnail image of the image generated by the AI. Also in the example in Figure 12, the generator data block, indicated by description 1232 corresponding to the learning model data 243, is stored as data for the learning model obtained through machine learning. In addition, the plain text item data, indicated by description 1233 corresponding to the inference input data 244-245, is stored as input text data that is input to the learning model when generating images. Furthermore, the execution program data, indicated by description 1236 corresponding to the learning algorithm data 246, is stored as the execution program data for the learning algorithm. Furthermore, HEVC encoded data (HEVC Image Data) indicated by description 1234, corresponding to training data 247 to 248, is stored as images used as training data, and text item data indicated by description 1235 is stored as text description data used as training data. Note that Exif data block 249 is not stored in this file.

[0097] Description 1201 corresponds to box 201 of 'ftyp'. In description 1201, 'mif1' is stored as the type value major-brand for brand definitions compliant with HEIF files, and 'heic' is stored as the type value compatible-brands for compatible brand definitions.

[0098] Description 1202 corresponds to the 'etyp' box, which is not shown in Figure 2. In description 1202, 'unif' is stored as the type value compatible-brands for the extended brand definition compliant with the HEIF file. This indicates that the ID value at the file level is a uniquely identifiable value.

[0099] Next, in description 1203, which corresponds to the 'meta' box 202, various metadata information describing the untimed data stored in the example output file is shown. Description 1210 corresponds to the hdlr box 211, and the handler type of MetaDataBox(meta) specified in description 1210 is 'pict'. Description 1211 corresponds to the pitm box 212. In description 1211, 1 is stored as item_ID, and the ID of the image to be displayed as the first priority image is specified.

[0100] Description 1212 corresponds to iinf box 214. Description 1212 shows item information (item ID and item type) for each item. Each item is identifiable by its item ID, which indicates what type of item the item identified by that item ID is. In the example in Figure 12, 10 items are stored in description 1212, so entry_count is 10, and description 1212 contains 10 types of information, with the item ID and item type specified for each item.

[0101] In the illustrated image file, the first piece of information, indicated by description 1240, corresponds to an HEVC encoded image item of type hvc1, which represents an image generated by the AI. The fifth piece of information, indicated by description 1244, corresponds to an HEVC encoded image item of item type hvc1, which is a thumbnail image. The sixth and seventh pieces of information, indicated by descriptions 1245 and 1246, correspond to HEVC encoded image items of item type hvc1, which are the training dataset images. The second piece of information, indicated by description 1241, corresponds to a deductive information item of type uri, which represents a learning model that generates images using text information as input data. The third and fourth pieces of information, indicated by descriptions 1242 and 1243, correspond to text items of type mime, which represent the text information input to the learning model when generating images. The eighth and ninth pieces of information, indicated by descriptions 1247 and 1248, correspond to text items of type mime, which represent the text information that will become the training dataset corresponding to the training dataset images. The tenth piece of information shown in description 1249 corresponds to the item that indicates the learning algorithm information of type uri.

[0102] Description 1213 corresponds to iloc box 213. Description 1213 specifies the storage location and data size information for each item within the HEIF file. For example, in the example in Figure 12, description 1213 indicates that the encoded image item with item_ID 1 is stored at offset 01 in the file, and that the size of the item is L1 byte. Such descriptions identify the location of each data within mdatBox.

[0103] Description 1214 corresponds to iref box 215, which indicates the reference relationship (association) between each item. The item reference shown in description 1250 has thmb specified as the reference type, indicating a thumbnail relationship. In the example in Figure 12, description 1214 shows that the HEVC encoded image item of item_ID5, specified in from_item_ID, references the HEVC encoded image item of item_ID1, specified in to_item_ID. This indicates that the HEVC encoded image item of item_ID5 is a thumbnail image of the HEVC encoded image item of item_ID1. The item references shown in descriptions 1251 and 1252 have cdsc specified as the reference type, indicating a content description relationship. In the example in Figure 12, description 1251 shows that the text information item of item_ID8, specified in from_item_ID, references the HEVC encoded image item of item_ID6, specified in to_item_ID. This indicates that the text information item with item_ID8 describes the content information of the HEVC encoded image item with item_ID6. Similarly, description 1252 indicates that the text information item with item_ID9, specified in from_item_ID, references the HEVC encoded image item with item_ID7, specified in to_item_ID. This indicates that the text information item with item_ID9 describes the content information of the HEVC encoded image item with item_ID7.

[0104] Descriptions 1215 and 1216 correspond to the grpl box 218, which specifies the entity group. In the HEIF file according to this embodiment, two entity groups are specified: AI Generation Information Entity Group and Deep Learning Information Entity Group.

[0105] Description 1215 corresponds to aigi box 232, and description 1216 corresponds to dlif box 231. Description 1215 specifies group_id as 100 and entity_id as item_id1, 2, 3, and 4. Here, item_id1, listed first, is identified as an item (image item in this file example) representing media data generated and modified by AI (learning model and input data). Also, item_id2, listed second, is identified as an item representing the learning model data generated and modified by AI, and item_id3 and 4, listed third and subsequent times, are identified as items representing the input data generated and modified by AI. Description 1216 specifies group_id as 101 and entity_id as item_id2, 10, 6, and 7. Here, item_id2, listed first, is identified as an item representing the learning model data generated as a result of machine learning. Furthermore, item_id10, listed second, is identified as an item representing the program data for executing the learning algorithm to generate the learning model, while item_id6 and 7, listed third and subsequent times, are identified as items representing the data that will become the learning dataset. Note that item_id6 and 7, as shown in descriptions 1251 and 1252 respectively, are further associated with item data representing text information as part of the learning dataset, and are identified together as learning data.

[0106] Description 1217 corresponds to the 'iprp' box 216, which includes Description 1220 corresponding to the 'ipco' box 221 and Description 1221 corresponding to the 'ipma' box 222. Description 1220 lists attribute information that may be used for each item or entity group as entry data. As shown in the figure, Description 1220 includes the first and second entries indicating encoding parameters and the third and fourth entries indicating the display pixel size of the item. Description 1220 also includes a fifth entry indicating that it is AI-generated media data, sixth and seventh entries providing detailed parameters for the learning model and learning algorithm execution program, and an eighth entry indicating a copyright declaration.

[0107] The attribute information listed in description 1220 is associated with each item or entity group stored in the HEIF file in the entry data of description 1221, which corresponds to the 'ipma' box 222. In the example in Figure 12, the image item with item_ID 1 is associated with 'hvcC' (property_index 1), indicating that it is an encoding parameter. Similarly, the image item with item_ID 1 is associated with 'ispe' (property_index 3), indicating that the image size is 4032 pixels × 3024 pixels. Furthermore, the image item with item_ID 1 is associated with 'aign' (property_index 5) and 'cprt' (property_index 8), indicating that it is AI-generated and modified media data and copyright declaration information. The learning model item with item_ID 2 is associated with 'uuid' (property_index 6), indicating detailed parameters specific to the learning model, etc. Image items with item_ID 5 are associated with 'ispe' (property_index 4), indicating that they are 768 pixels × 576 pixels in size. Similarly, image items with item_ID 5 are associated with 'hvcC' (property_index 2), indicating the encoding parameters. Image items with item_ID 6 and 7 are associated with the common 'ispe' (property_index 3), indicating that they are identical images with the same size of 4032 pixels × 3024 pixels. Similarly, image items with item_ID 6 and 7 are associated with the common 'hvcC' (property_index 1), indicating the identical encoding parameters. The learning algorithm item with item_ID 10 is associated with 'uuid' (property_index 7), indicating detailed parameters specific to the learning algorithm execution program.

[0108] Additionally, the AI-generated information entity group with item_id (group_id) 100 is associated with 'cprt' and displays a copyright declaration.

[0109] Note that since items with item_ID 3, 4, 8, and 9, and the entity group with group_id 101, do not have any associated item properties, the corresponding entry information is not stored in the file.

[0110] In this example HEIF file, the images and associated text information that form the training dataset are defined as items. The training dataset can be identified by associating these data using irefBox. However, the training dataset can also be identified by defining the images as items, storing the 'udes' property specified in ISO / IEC 23008-12 (HEIF) in an ipcoBox for the text information, and associating the images and text information using an ipma box. Alternatively, the data can be identified by listing entity IDs in a dlif entity group without associating these data, allowing them to be treated as a dataset.

[0111] Next, with reference to Figure 13, another example of an output file output by the storage device 100 according to this embodiment will be described. In this embodiment, instead of using the entity group method shown in Figure 12 as the file data structure for the image file, the item reference type is defined and items are associated, thereby storing information that can identify the generation process of the image generated by the AI ​​within the file. In this example of a file, a learning model is generated by learning the image and metadata regarding the camera spatial coordinates and viewpoint direction at the time the image was taken as training data. Furthermore, an example will be described in which an image from a free viewpoint is generated by inputting metadata indicating the spatial coordinates and viewpoint direction of a virtual viewpoint as input data to the learning model, and the information at the time the image was generated is stored in the file along with the image. The file shown in Figure 13 is an example of a file that stores information when outputting a 2D image from a virtual viewpoint generated using a neural network called NeRF, which reconstructs a 3D scene from a series of multiple 2D images, along with the output image. The image stored as the output result here is assumed to be an image generated using NeRF, where image data is generated from volume density and radiance.

[0112] In the example in Figure 13, HEVC encoded data (HEVC Image Data) is stored, as shown in description 1303 corresponding to 'mdat' box 203, and indicated by descriptions 1330-1331 corresponding to encoded data 241-242. Descriptions 1330 and 1331 show images output from different virtual viewpoints, generated by the AI ​​(NeRF). Additionally, the generator data block, indicated by description 1332 corresponding to the learning model data 243, is stored as data for the NeRF learning model (neural network) obtained through machine learning. Furthermore, the metadata item data, indicated by descriptions 1332-1333 corresponding to the inference input data 244-245, is stored as input metadata (virtual viewpoint and gaze direction) when generating the image. Finally, the execution program data, indicated by description 1337 corresponding to the learning algorithm data 246, is stored as the execution program data for the (NeRF) learning algorithm. Furthermore, the HEVC encoded data (HEVC Image Data) indicated by description 1335, corresponding to training data 247 to 248, is stored as training data (a series of 2D) images, and the metadata item data indicated by description 1336 is stored as training data indicating the viewpoint and gaze direction corresponding to the training data images in description 1335. Note that Exif data block 249 is not stored in this file.

[0113] Description 1301 corresponds to box 201 of 'ftyp'. In description 1301, 'mif1' is stored as the type value major-brand for brand definitions compliant with HEIF files, and 'heic' is stored as the type value compatible-brands for compatible brand definitions.

[0114] Next, in description 1302, which corresponds to the 'meta' box 202, various metadata information describing the untimed data stored in the example output file is shown. Description 1310 corresponds to the hdlr box 211, and the handler type of MetaDataBox(meta) specified in description 1310 is 'pict'. Description 1311 corresponds to the pitm box 212. In description 1311, 1 is stored as item_ID, and the ID of the image to be displayed as the first priority image is specified.

[0115] Description 1312 corresponds to iinf box 214. Description 1312 shows item information (item ID and item type) for each item. Each item is identifiable by its item ID, which indicates what type of item the item identified by that item ID is. In the example in Figure 13, 14 items are stored in description 1312, so entry_count is 14, and description 1312 contains 14 types of information, with each item having its own item ID and item type specified.

[0116] In the illustrated image file, the first and second pieces of information corresponding to descriptions 1340 and 1341 correspond to HEVC encoded image items of type hvc1, and these items represent images generated by AI (Neural Network). The third piece of information corresponding to description 1342 corresponds to a deductive information item of type uri, and this item represents a neural network model using NeRF. The fourth and fifth pieces of information corresponding to descriptions 1343 and 1344 correspond to metadata items of type meta, and these items represent metadata describing the 3D spatial position x, y, z and line of sight directions θ, φ, which are input into the learning model when generating the image.

[0117] Furthermore, the 6th to 9th pieces of information corresponding to descriptions 1345 to 1348 correspond to HEVC encoded image items of item type hvc1, which serve as the training dataset images. The 10th to 13th pieces of information corresponding to descriptions 1349 to 1352 correspond to metadata items of type meta. These items, corresponding to the 10th to 13th pieces of information, are metadata items used as part of the training dataset along with the images. In this case, they describe the 3D spatial position x, y, z and line-of-sight directions θ, φ, which indicate the camera position and orientation at the time the images corresponding to the training dataset images were captured.

[0118] The 14th piece of information corresponding to description 1349 corresponds to an item that shows the learning algorithm information of type uri.

[0119] Furthermore, the metadata items described in descriptions 1343, 1344, and 1349 through 1352 may be described as properties instead of being defined as items, and associated with the corresponding images as item properties. In this case, the property data structure can be described using CameraExtrinsicMatrixProperty (cmex), which is being considered for standardization as ISO / IEC 23008-12 (HEIF).

[0120] Description 1313 corresponds to iloc box 213. Description 1313 specifies the storage location and data size information for each item within the HEIF file. For example, in the example in Figure 13, description 1313 indicates that the encoded image item with item_ID 1 is stored at offset 01 in the file, and that the size of the item is L1 byte. Such descriptions identify the location of data within mdatBox.

[0121] Description 1314 corresponds to iref box 215, which indicates the reference relationship (association) between each item. The item reference shown in description 1360 has genr specified as the reference type, indicating an association of items related to AI generation and modification. In the example in Figure 13, description 1314 shows that the HEVC encoded image item item_ID1 specified in from_item_ID references the item item_ID3, which represents a neural network model using NeRF, and the metadata item item_ID4, which describes the 3D spatial position x, y, z and the gaze direction θ, φ. This indicates that the HEVC encoded image item item_ID1 is an AI-generated image generated and modified by inputting the metadata item item_ID4, which describes the 3D spatial position x, y, z and the gaze direction θ, φ, into the neural network model shown by NeRF in item_ID3. Similarly, the item reference shown in description 1361 has genr specified as the reference type, indicating an association of items related to AI generation and modification. Similarly, in Figure 13, description 1361 indicates that the HEVC encoded image item of item_ID2 specified in from_item_ID references the item representing the NeRF neural network model of item_ID3 specified in to_item_ID, and the metadata item of item_ID5 that describes the 3D spatial position x, y, z and the line of sight direction θ, φ. This indicates that the HEVC encoded image item of item_ID2 is an AI-generated image generated and modified by inputting the metadata item of item_ID5 that describes the 3D spatial position x, y, z and the line of sight direction θ, φ into the NeRF neural network model of item_ID3.

[0122] The reference type `genr` is an item reference that enables the identification of information equivalent to that of the aigi entity group shown in Figure 12. In the aigi entity group, the item ID representing the generated / modified media data, specified as the first entity ID, is specified as `from_item_ID`. In the aigi entity group, the item ID representing the learning model, specified as the second entity ID, is specified as the first item ID in `to_item_ID`. Furthermore, the item IDs representing the input data, specified as the third and subsequent entity IDs, are specified consecutively in `to_item_ID` from the second onward. Through this description, regardless of the entity group, it becomes possible to associate media data generated / modified by item references with the learning model data and its input data used to generate / modify the media data.

[0123] Furthermore, the item reference shown in description 1362 specifies `lern` as the reference type, indicating an association of items related to the generation of a machine learning model. In the example in Figure 13, description 1362 shows that it references an item showing the execution program data of a learning algorithm and HEVC encoded image items of item_ID6,7,8, and9, for generating the learning model of item_ID14 specified in `to_item_ID` from the item showing the NeRF neural network model of item_ID3 specified in `from_item_ID`. This indicates that the item showing the NeRF neural network model of item_ID3 is a learning model generated as a result of training with the HEVC encoded image items of item_ID6,7,8,9 as the training dataset, using the item showing the execution program data of a learning algorithm to generate the learning model of item_ID14.

[0124] The reference type `lern` is an item reference that enables the identification of information equivalent to that of the `dlif` entity group shown in Figure 12. In the `dlif` entity group, the item ID that represents the learning model data generated as a result of machine learning, specified as the first entity ID, is specified as `from_item_ID`. In the `dlif` entity group, the item that represents the execution program data of the learning algorithm used to generate the learning model, specified as the second entity ID, is specified as the first item ID of `to_item_ID`. Furthermore, item IDs that represent the data that will become the learning dataset, specified as the third and subsequent entity IDs, are specified consecutively as `to_item_ID`. Through this description, regardless of the entity group, it becomes possible to associate the learning model data generated as a result of machine learning with the learning algorithm data and its learning dataset used in the learning of that learning model, through item references.

[0125] The item references shown in descriptions 1363 to 1366 specify `lnds` as the reference type, indicating an association of training datasets. In the example in Figure 13, description 1363 shows that the item representing the HEVC encoded image item with item_ID6 specified in `from_item_ID` references the metadata item with item_ID10 specified in `to_item_ID`. This indicates that the HEVC encoded image item with item_ID6 and the metadata item with item_ID10 are associated as a set for training. Similarly, in descriptions 1364, 1365, and 1366, the HEVC encoded image item and the metadata item are associated as a set of training data. Note that when describing using item properties instead of metadata items, such associations are described in ipma box 222.

[0126] Description 1315 corresponds to the 'iprp' box 216 and includes Description 1320, which corresponds to the 'ipco' box 221, and Description 1321, which corresponds to the 'ipma' box 222. Description 1320 lists attribute information that may be used for each item or entity group as entry data. As shown in the figure, Description 1320 includes a first entry indicating encoding parameters and a second entry indicating the display pixel size of the item. Description 1320 also includes a third entry indicating that it is AI-generated media data, fourth and fifth entries providing detailed parameters of the learning model and learning algorithm execution program, and a sixth entry indicating a copyright declaration. The attribute information listed in Description 1320 is associated with each item or entity group stored in the HEIF file in the entry data of Description 1321, which corresponds to the 'ipma' box 222. In the example in Figure 13, the association between items and properties is described in the same way as in Figure 12.

[0127] Note that not all of the data described in this HEIF file example needs to be acquired and stored in the file. For example, metadata used when acquiring some or all of this data from an external device may be included in the file.

[0128] Next, we will explain the generation process for generating media files with a file structure that can store media data in association with the fact that the content was generated and modified by AI, and the conditions under which it was generated and modified, referring to the flowchart in Figure 8.

[0129] The process shown in the flowchart of Figure 8 is performed by the CPU 101 using computer programs and data read from the ROM 102 or non-volatile memory 110 into the RAM 103 to perform various control processing. The generation process according to the flowchart of Figure 8 is started, for example, when the CPU 101 detects that the user has entered instructions related to shooting by operating the operation input unit 107, or that instructions related to AI processing have been entered. However, the event that triggers the start of the process according to the flowchart of Figure 8 is not limited to a specific event. The data and metadata generated in each step are temporarily stored in the output buffer while the processing is performed.

[0130] In S801, the CPU 101 controls the imaging unit 104 or the image processing unit 105 to acquire a training dataset. The method of acquiring the training dataset is not particularly limited; for example, the training dataset may be acquired from the non-volatile memory 110, or from an external device via the communication unit 108. Alternatively, the training dataset may be acquired from the imaging unit 104 as a series of captured image data.

[0131] In S802, the CPU 101 acquires learning algorithm data from an external device via non-volatile memory 110 or communication unit 108. If the acquired learning algorithm data is program code, the CPU 101 generates executable data.

[0132] In S803, the learning processing unit 114 uses the program execution code of the learning algorithm acquired by the CPU 101 to perform machine learning processing using the learning dataset acquired by the CPU 101 in S801. In S804, the learning processing unit 114 generates learning model data as a result of the learning in S803.

[0133] In S805, the metadata processing unit 112 generates metadata related to the training dataset. For example, if the training dataset is image data, descriptive information such as encoding parameters for encoding the image, image size information, or item information to identify these is generated as metadata related to the training dataset.

[0134] In S806, the metadata processing unit 112 generates metadata related to the learning algorithm data. The metadata generated for the learning algorithm data includes, for example, descriptive information such as information about the detailed parameters of the learning algorithm, or item information that identifies the learning algorithm as an item.

[0135] In S807, the metadata processing unit 112 generates metadata describing information about the learning model generated as a result of the learning process in S803. This metadata may include, for example, information about the detailed parameters of the learning model, or item information that identifies the learning model data as items. Furthermore, in S807, the metadata processing unit 112 generates metadata (association information) to associate the learning model data with the learning algorithm data and the learning dataset. Note that S807 may be executed by obtaining the trained model data corresponding to the association information from an external device. Also, in S807, metadata used to reference the learning model data containing the association information, which is held by the external device, may be obtained. If information regarding the creation history or copyright of such a learning model is publicly available, such information may be associated and recorded as metadata.

[0136] In S808, the CPU 101 acquires input data for a learning model used to execute inference processing that generates and modifies media data (for example, by receiving user input from the operation input unit 107). The method of acquiring this input data is not particularly limited; for example, the input data may be acquired from data that has been acquired in advance and stored in the non-volatile memory 110, or it may be acquired from an external device via the communication unit 108.

[0137] In S809, the metadata processing unit 112 generates metadata related to the input data. For example, if the input data is image data, descriptive information such as encoding parameters for encoding the image, image size information, or item information to identify these is generated as metadata related to the input data.

[0138] In steps S810-S811, the CPU 101 acquires the media data output by the learning model. Here, in step S810, the inference processing unit 113 performs media data generation and modification processing using the learning model and input data. Then, in step S811, the CPU 101 acquires the media data obtained as a result of step S810. At this time, the metadata processing unit 112 generates and records descriptive information describing the media data acquired in step S811. If the media data acquired here is data that can be compressed, the encoding / decoding unit 111 may perform compression encoding processing on the media data.

[0139] In S812, the metadata processing unit 112 generates metadata describing information about the learning model used when outputting the media data. Here, metadata describing information about the learning model is generated, which includes association information between the learning model used when outputting the media data, or its input data, and the media data. The metadata processing unit 112 also generates property information that indicates (makes identifiable) that the media data acquired in S811 is data that has been generated or modified by the AI ​​(learning model). Furthermore, if the media data acquired in S811 is data for which a copyright declaration regarding the generated or modified media data can be specified (i.e., data for which corresponding copyright information exists), the metadata processing unit 112 also generates metadata regarding such copyright declaration.

[0140] In S813, the CPU 101 outputs a media file containing the generated metadata and data, and terminates the process shown in Figure 8. More specifically, the metadata processing unit 112 configures the final metadata to be stored in the media file based on the information stored in the output buffer. Next, the metadata processing unit 112 combines the information in the 'ftyp' box 201 related to the media file, the information in the 'meta' box 202 containing the final metadata, and the information in the 'mdat' box 203 containing media data and various AI-related data. Furthermore, the CPU 101 writes the media file generated by this combination process from the RAM 103 to the non-volatile memory 110 and saves it.

[0141] As described above, the storage device 100 according to this embodiment acquires the dataset used for learning, the algorithm data used for learning, and the learning model data generated as a result of learning, and associates them as metadata. Next, the storage device 100 acquires the learning model data and the input data used for inference processing, and associates and identifies the resulting generated and modified media data. The storage device 100 also stores information that identifies that the media data was generated and modified by AI in a file. Furthermore, the storage device 100 also stores a copyright declaration related to the series of AI generation processes in a file as metadata. For example, as a copyright declaration for machine learning algorithm information related to the AI ​​generation process, license information in open source code may be stored.

[0142] As mentioned above, the media data according to this embodiment is not limited to image data. For example, the media data may include moving images, audio data, voice data, text data such as documents, or metadata media data. Furthermore, the series of learning datasets and input data, learning algorithm data, and learning model data may be data pre-stored in the ROM 102 or non-volatile memory 110, or data received via the communication unit 108, and there are no restrictions on the means of acquisition and the format of the data as long as it is similarly available to the storage device 100.

[0143] Furthermore, the input data fed into the learning model when outputting media data is not limited to still images; it may also include moving images, audio data, voice data, text data such as documents, or metadata obtained by analyzing content. Any data in a format that can be stored in a media file is acceptable. In addition, the metadata described in this embodiment may be stored as Exif tag information. In this case, it is desirable that the metadata be data defined as an Exif tag, but it is also acceptable to describe that the metadata is Exif tag information using manufacturer notes or similar.

[0144] Furthermore, it is desirable that media data stored in files in this manner be recorded along with information that can prove that the data itself has not been tampered with or fraudulently generated. From this perspective, for example, a mechanism to guarantee authenticity as stipulated in C2PA may be used, and this guarantee of authenticity may be associated with the media data as metadata before being stored in the file.

[0145] [Playback process flow] Next, we will explain the process for playing media files. Here, the media file playback process may be performed by the storage device 100 that generated the media file, or it may be performed by a playback device, which is an information processing device (not shown) different from the storage device 100. Here, it is assumed that the processor (e.g., CPU 101) of the device that performs the media file playback process reads the metadata of the media file to be processed, thereby enabling playback and modification of the media data stored in the media file.

[0146] In the following, with reference to Figure 9, the playback process of a media file (here, a HEIF file storing still images as media data) executed by the storage device 100 according to this embodiment will be described. The process shown in the flowchart of Figure 9 can be realized by operating each block, for example, by the CPU 101 reading the corresponding processing program stored in the ROM 102, expanding it into the RAM 103, and executing it. This playback process will be described as starting, for example, when the storage device 100 is set to playback mode and a user input that instructs playback of the media file to be processed is detected.

[0147] In S901, the CPU 101 acquires the HEIF file (target file) to be played back that has been instructed for playback. In S902, the CPU 102 acquires metadata and image data from the HEIF file, and the metadata processing unit 112 analyzes the acquired metadata to understand the structure of the target file. In S903, the CPU 101 identifies a representative item based on the information in the 'pitm' box 212 of the metadata and causes the encoding / decoding unit 111 to decode the encoded data 241 indicated by that representative item. Next, the encoding / decoding unit 111 acquires the encoded data corresponding to the metadata of the image item designated as the representative item, performs the decoding process, and saves the decoded data to a buffer on RAM 103. In the following explanation, it is assumed that the image data designated as the representative item is used as the target of playback processing, but if playback processing is performed on multiple image data, it is possible to perform the same processing on each of those image data.

[0148] In S904, the metadata processing unit 112 retrieves metadata associated with the image to be played, which is designated as the representative item and stored in the target file. It determines whether the metadata associated with the representative item contains information indicating that the item is media data generated or modified by AI. If it does, the process proceeds to S905; otherwise, the process proceeds to S908.

[0149] In S905, the metadata processing unit 112 stores information indicating that the image to be played back is media data generated and modified by AI in a buffer on RAM 103.

[0150] In S906, the metadata processing unit 112 determines whether the generation process (by AI) of the representative item can be identified. Here, the metadata processing unit 112 can determine if the generation process of the representative item can be identified if, for example, the representative item is associated with the learning model data that generated or modified the representative item, the input data to the learning model when the representative item was generated, the algorithm information used when the learning model that generated the representative item was trained, or the training dataset, or a property indicating the generation process of the representative item. If it is determined that the generation process can be identified, the process proceeds to S907; otherwise, the process proceeds to S908. In S907, the metadata processing unit 112 saves the generation process of the representative item to a buffer on RAM 103 and proceeds to S908.

[0151] In S908, the CPU 101 determines whether copyright information is associated with the representative item. If it is determined that it is associated, the process proceeds to S909; otherwise, the process proceeds to S910. In S909, the metadata processing unit 112 saves the copyright information to a buffer on RAM 103 and proceeds to S910.

[0152] In the S910, the CPU 101 displays an image of a representative item on the display unit 106. Here, the CPU 102 displays the image in a configuration that allows it to refer to information stored in a buffer on RAM 103, which indicates that the image to be played is media data generated and modified by AI, or information related to AI generation including the history of AI generation, and copyright information. This information may always be displayed with the image, or it may be possible to enable selective display, such as turning the display ON / OFF for each item in response to user input to a selection menu. It may also be possible to select whether or not to display this information as an option. The decision to display or not can be made, for example, in response to user operation via the UI.

[0153] According to the embodiments described above, by storing metadata indicating that the media data stored in the media file is data generated or modified using AI, it becomes possible to identify that the media data was generated or modified using AI. It also becomes possible to identify the conditions used when generating the media data using AI, as well as the copyright of the media data generated using AI. Furthermore, it becomes possible to change the training dataset and training algorithm for generating the AI ​​learning model, retrain the model, and then generate media data under the same conditions. It also becomes possible to regenerate media data by changing the generation conditions without changing the learning model. In addition, it becomes possible to track the history of the generation of media data generated or modified using AI. Furthermore, it becomes possible to identify whether the media data generated or modified using AI infringes on copyright.

[0154] More specifically, it becomes possible to identify whether media data is AI-generated, modified, or processed using AI without altering the content. Therefore, the risk of copyright infringement when using media data stored in media files can be reduced. Furthermore, by associating the media data with the learning model or input data used to output the media data, or the algorithm or learning data used to generate that learning model, it becomes possible to identify the details of how the media data was output. Moreover, by associating and storing such details of how the media data was output with the media data, it becomes possible to modify some of the data included in that history and output the media data again. This makes it possible, for example, to re-output media data by changing some of the input data for the same learning model, or to re-output media data using a different learning model while keeping the input data the same. It also becomes possible to regenerate the learning model by changing the learning algorithm or the learning data, and then re-output media data using the same input data for the regenerated learning model.

[0155] Furthermore, by storing copyright information together with media data, it becomes possible to identify the copyright of media data generated and modified by AI, as well as the copyright of each data used in the process of its generation and modification, thereby enabling the identification of copyright when using such content. It is desirable that such data be used in conjunction with a mechanism that can guarantee that it has not been tampered with. It is also desirable that such data be stored compressed and encoded, but this is not strictly necessary. In addition, metadata regarding copyright declarations may be stored as separately referable information without specifying the copyright declaration itself.

[0156] Furthermore, by making this copyright information and other related information accessible to users when using media data stored in media files, end users of the media data can easily identify this information. In particular, it becomes possible to track the history of how AI-generated and modified media data was created, and to identify whether AI-generated and modified media data consistently does not infringe copyright.

[0157] The disclosures herein include the following information processing devices, information processing methods, and programs. (Item 1) A first acquisition method for acquiring the first media data output by the machine learning model, A first generation means for generating first descriptive information describing the first media data, A second generation means for generating second descriptive information that describes information about the machine learning model used when outputting the first media data, A third generation means for generating association information indicating the association between the first media data, the first descriptive information, and the second descriptive information, A fourth generation means for generating a media file storing the first media data, the first description information, the second description information, and the association information, An information processing device characterized by comprising: (Item 2) The information processing apparatus according to item 1, characterized in that the first media data includes still images, moving images, audio data, voice data, text data, or metadata. (Item 3) Furthermore, the system includes a second acquisition means for acquiring the input data that was input to the machine learning model when outputting the first media data. The information processing apparatus according to item 1 or 2, wherein the fourth generation means is characterized by storing the input data in the media file. (Item 4) The information processing device according to any one of items 1 to 3, characterized in that the second acquisition means acquires the input data as second media data input to the machine learning model and metadata identifying the second media data. (Item 5) The information processing device according to any one of items 1 to 4, characterized in that the second descriptive information includes data of the learning algorithm used when training the machine learning model and the training dataset used when training the machine learning model. (Item 6) The system further comprises a fifth generation means for generating a third descriptive information indicating that the first media data is media data output by a machine learning model, The information processing apparatus according to item 5, characterized in that the fourth generation means generates the media file to further store the third description information. (Item 7) The system further comprises a sixth generation means for generating copyright information for the first media data, The information processing device according to any one of items 1 to 6, characterized in that the fourth generation means generates the media file to further store the copyright information. (Item 8) The information processing device according to item 7, characterized in that the copyright information includes information indicating that the first media data is a copyrighted work, information indicating that the training data of the machine learning model contains a copyrighted work, or information indicating that the input data input to the machine learning model when outputting the first media data contains a copyrighted work. (Item 9) The information processing device according to any one of items 1 to 8, characterized in that the machine learning model outputs a two-dimensional image as the first media data based on the input data input to the machine learning model when outputting the first media data. (Item 10) The information processing device according to item 9, characterized in that the input data is a fourth descriptive information indicating the spatial coordinates and viewpoint direction of a virtual viewpoint. (Item 11) The information processing device described in any one of items 1 to 10, characterized in that the media file is a media file conforming to the provisions of ISOBMFF. (Item 12) Acquisition means for acquiring a media file containing first media data output by a machine learning model, first descriptive information describing the first media data, second descriptive information describing information about the machine learning model used when outputting the first media data, and association information indicating the association between the first media data, the first descriptive information, and the second descriptive information. A control means that performs playback processing of the first media data based on the media file, An information processing device characterized by comprising: (Item 13) The process of acquiring the first media data output by the machine learning model, A step of generating first descriptive information that describes the first media data, A step of generating second descriptive information that describes information about the machine learning model used when outputting the first media data, A step of generating association information that indicates the association between the first media data, the first descriptive information, and the second descriptive information, A step of generating a media file that stores the first media data, the first description information, the second description information, and the association information, An information processing method characterized by comprising: (Item 14) A step of obtaining a media file that stores first media data output by a machine learning model, first descriptive information describing the first media data, second descriptive information describing information about the machine learning model used when outputting the first media data, and association information indicating the association between the first media data, the first descriptive information, and the second descriptive information. A step of performing playback processing of the first media data based on the media file, An information processing method characterized by comprising: (Item 15) A program to cause a computer to function as one of the means of an information processing device described in any one of items 1 through 12.

[0158] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0159] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0160] 100: Storage device, 101: CPU, 102: ROM, 103: RAM, 104: Imaging unit, 105: Image processing unit, 106: Display unit, 107: Operation input unit, 108: Communication unit, 109: System bus, 110: Non-volatile memory

Claims

1. A first acquisition means for acquiring the first media data output by the machine learning model, A first generation means for generating first descriptive information describing the first media data, A second generation means for generating second descriptive information that describes information about the machine learning model used when outputting the first media data, A third generation means for generating association information indicating the association between the first media data, the first descriptive information, and the second descriptive information, A fourth generation means for generating a media file storing the first media data, the first description information, the second description information, and the association information, An information processing device characterized by comprising:

2. The information processing apparatus according to claim 1, characterized in that the first media data includes still images, moving images, audio data, voice data, text data, or metadata.

3. Furthermore, the system includes a second acquisition means for acquiring the input data input to the machine learning model when outputting the first media data, The information processing apparatus according to claim 1, wherein the fourth generation means is characterized by storing the input data in the media file.

4. The information processing apparatus according to claim 3, characterized in that the second acquisition means acquires the input data as second media data input to the machine learning model and metadata identifying the second media data.

5. The information processing apparatus according to claim 1, characterized in that the second descriptive information includes data of the learning algorithm used when training the machine learning model and a training dataset used when training the machine learning model.

6. The system further comprises a fifth generation means for generating a third descriptive information indicating that the first media data is media data output by a machine learning model, The information processing apparatus according to claim 5, characterized in that the fourth generation means generates the media file to further store the third description information.

7. The system further comprises a sixth generation means for generating copyright information for the first media data, The information processing apparatus according to claim 1, characterized in that the fourth generation means generates the media file to further store the copyright information.

8. The information processing apparatus according to claim 7, characterized in that the copyright information includes information indicating that the first media data is a copyrighted work, information indicating that the training data of the machine learning model contains a copyrighted work, or information indicating that the input data input to the machine learning model when outputting the first media data contains a copyrighted work.

9. The information processing apparatus according to claim 1, characterized in that the machine learning model outputs a two-dimensional image as the first media data based on the input data input to the machine learning model when outputting the first media data.

10. The information processing apparatus according to claim 9, characterized in that the input data is a fourth descriptive information indicating the spatial coordinates and viewpoint direction of a virtual viewpoint.

11. The information processing apparatus according to claim 1, characterized in that the media file is a media file conforming to the ISOBMFF specifications.

12. Acquisition means for acquiring a media file containing first media data output by a machine learning model, first descriptive information describing the first media data, second descriptive information describing information about the machine learning model used when outputting the first media data, and association information indicating the association between the first media data, the first descriptive information, and the second descriptive information. A control means that performs playback processing of the first media data based on the media file, An information processing device characterized by comprising:

13. The process of acquiring the first media data output by the machine learning model, A step of generating first descriptive information that describes the first media data, A step of generating second descriptive information that describes information about the machine learning model used when outputting the first media data, A step of generating association information that indicates the association between the first media data, the first descriptive information, and the second descriptive information, A step of generating a media file that stores the first media data, the first description information, the second description information, and the association information, An information processing method characterized by comprising:

14. A step of obtaining a media file that stores first media data output by a machine learning model, first descriptive information describing the first media data, second descriptive information describing information about the machine learning model used when outputting the first media data, and association information indicating the association between the first media data, the first descriptive information, and the second descriptive information. A step of performing playback processing of the first media data based on the media file, An information processing method characterized by comprising:

15. A program for causing a computer to function as one of the means of an information processing device according to any one of claims 1 to 12.

Images