Information processing apparatus, information processing method, and program

Abstract

To solve the problem in which: pieces of material data used for creation of a 3D model and a virtual viewpoint image are appropriately edited from the created 3D model and virtual viewpoint image and a material data set.SOLUTION: An information processing apparatus specifies the editing histories of a plurality of pieces of material data used for creating a virtual viewpoint image, and when the specified editing histories satisfy predetermined conditions for the plurality of pieces of material data, provides authentication information to a virtual viewpoint image created on the basis of the plurality of pieces of material data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a technique for handling data used to generate a virtual viewpoint image. [Background technology]

[0002] There is a technology that generates a 3D model showing the three-dimensional shape of a subject from video captured by a camera. A virtual viewpoint image can be generated by using the generated 3D model and a virtual camera virtually installed in a virtual space. In addition to generating a 3D model from captured data, it can also be generated by adding 3D objects drawn using CG or by combining objects edited from existing 3D models generated from the captured data or CG. Patent Document 1 discloses a method for managing edits to a 3D model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6791530 Public Relations Summary of the Invention [Problem to be solved by the invention]

[0004] However, there is a risk that it may not be easy to determine from the generated 3D model, virtual viewpoint image, and material dataset whether the material data used for generation has been appropriately edited. For example, virtual viewpoint images used as sports learning materials must be generated using a 3D model that accurately reproduces the movements of a teacher's arms and legs. In this case, if the position of some body parts of the teacher's 3D model is changed, the teacher's movements cannot be accurately reproduced, and virtual viewpoint images that are inappropriate for learning materials are generated. Learners who view the virtual viewpoint images will be unaware that the positions of some body parts of the teacher's 3D model have been changed, resulting in incorrect learning.

[0005] Therefore, an object of the present disclosure is to provide a 3D model, a virtual viewpoint image, and a material data set that show that the material data used for generation has been appropriately edited. [Means for solving the problem]

[0006] In order to achieve the above object, the information processing apparatus of the present disclosure has the following configuration: That is, a specifying unit that specifies an editing history of a plurality of pieces of material data used to generate a virtual viewpoint image; an assigning means for assigning authentication information to a virtual viewpoint image generated based on the plurality of material data when the editing histories of the plurality of material data identified by the identifying means each satisfy a predetermined condition; It has. [Effects of the Invention]

[0007] According to the present disclosure, it is possible to provide a 3D model, a virtual viewpoint image, and a material data set that show that the material data used for generation has been appropriately edited. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a configuration diagram of an image processing device. [Figure 2] FIG. 10 is a diagram illustrating authentication conditions for material data. [Figure 3] FIG. 1 is a diagram illustrating a hardware configuration of an image processing device. [Figure 4] FIG. 4 is a diagram illustrating authentication information of a virtual viewpoint image according to the first embodiment. [Figure 5] FIG. 2 is a diagram illustrating a process of editing material data according to the first embodiment. [Figure 6] 10A to 10C are diagrams illustrating a process of adding authentication information to a generated virtual viewpoint image and outputting the image according to the first embodiment. [Figure 7] 10A and 10B are diagrams illustrating a mechanism for determining whether or not an authentication condition is satisfied when a virtual viewpoint image is generated according to the first embodiment. [Figure 8] FIG. 10 is a diagram illustrating authentication conditions and utilization conditions for material data according to the second embodiment. [Figure 9] 10A and 10B are diagrams illustrating a process of adding authentication information to a generated virtual viewpoint image and outputting the image according to the second embodiment. [Figure 10] FIG. 13 is a diagram illustrating authentication information for a material data set according to the third embodiment. [Figure 11] FIG. 11 is a diagram illustrating a process of adding authentication information to a material data set and outputting the same, according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Preferred embodiments of the present disclosure will be described in detail below with reference to the drawings. However, the present disclosure is not limited to the following embodiments. In each drawing, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] <Embodiment 1> In this embodiment, authentication conditions are set in advance to indicate the range of editing that can be performed on the material data, and if the editing history of each piece of material data specified by the user satisfies the authentication conditions, authentication information is assigned to the virtual viewpoint image generated using each piece of material data.

[0011] The virtual viewpoint image in this embodiment is also called a virtual viewpoint video, but is not limited to an image corresponding to a viewpoint freely (arbitrarily) designated by the user, and also includes, for example, an image corresponding to a viewpoint selected by the user from a plurality of candidates. Furthermore, in this embodiment, the description will be centered on a case where the virtual viewpoint is designated by a user operation, but the virtual viewpoint may also be designated automatically based on the results of image analysis, etc. Furthermore, in this embodiment, the description will be centered on a case where the virtual viewpoint image is a moving image, but the virtual viewpoint image may also be a still image.

[0012] The viewpoint information used to generate a virtual viewpoint image is information indicating the position and orientation (line of sight direction) of the virtual viewpoint. Specifically, the viewpoint information is a parameter set including a parameter indicating the three-dimensional position of the virtual viewpoint and a parameter indicating the orientation of the virtual viewpoint in the pan, tilt, and roll directions. Note that the content of the viewpoint information is not limited to the above. For example, the parameter set serving as viewpoint information may include a parameter indicating the size of the field of view (angle of view) of the virtual viewpoint. Furthermore, the viewpoint information may have multiple parameter sets. For example, the viewpoint information may have multiple parameter sets corresponding to multiple frames constituting a moving image of the virtual viewpoint image, and may be information indicating the position and orientation of the virtual viewpoint at each of multiple consecutive time points.

[0013] A virtual viewpoint image is generated, for example, by the following method. First, multiple images (multiple viewpoint images) are acquired by capturing images from different directions using multiple imaging devices. Next, a foreground image in which a foreground region corresponding to a predetermined object, such as a person or a ball, is extracted, and a background image in which a background region other than the foreground region is extracted are acquired from the multiple viewpoint images. Furthermore, a foreground model representing the three-dimensional shape of the predetermined object and texture data for coloring the foreground model are generated based on the foreground image, and texture data for coloring a background model representing the three-dimensional shape of a background, such as a stadium, is generated based on the background image. The texture data is then mapped to the foreground model and background model, and rendering is performed according to the virtual viewpoint indicated by the viewpoint information, thereby generating a virtual viewpoint image. However, the method for generating a virtual viewpoint image is not limited to this, and various methods can be used, such as a method of generating a virtual viewpoint image by projective transformation of captured images without using a three-dimensional model.

[0014] A foreground image is an image in which an object region (foreground region) is extracted from an image captured by an imaging device. An object extracted as a foreground region is a dynamic object (moving body) that moves (its absolute position and shape can change) when images are captured from the same direction in a time series. Examples of objects include players, referees, and other people on the field where a sport is being played, such as a ball in a ball game, or singers, musicians, performers, and presenters in a concert or entertainment event.

[0015] A background image is an image of at least a region (background region) different from the foreground object. Specifically, a background image is an image in which the foreground object has been removed from the captured image. Furthermore, the background refers to an imaged object that remains stationary or nearly stationary when images are captured from the same direction in chronological order. Examples of such imaged objects include a stage for a concert, a stadium where an event such as a sport is held, a structure such as a goal used in a ball game, or a field. However, the background is at least a region different from the foreground object, and the imaged object may include other objects in addition to the object and background.

[0016] In this embodiment, data for generating a virtual viewpoint image, such as a captured image captured by an imaging device and data generated based on the captured image, is referred to as material data. The material data generated based on the captured image includes, for example, data on foreground and background images extracted from the captured image, 3D model data representing the shape of an object in three-dimensional space, and texture data for coloring the 3D model. The 3D model includes a foreground model, which is a 3D model of an object in the foreground region, and a background model, which is a 3D model of an object in the background region.

[0017] The type of material data is not limited as long as it is data for generating a virtual viewpoint image. For example, the material data may include camera parameters that indicate the imaging conditions of an imaging device that acquires a captured image. Furthermore, although the above describes an example of material data when a virtual viewpoint image is generated using a 3D model, the material data may differ from the above when a virtual viewpoint image is generated using image-based rendering that does not use a 3D model. In this way, the material data may differ depending on the method for generating the virtual viewpoint image.

[0018] FIG. 1 is a configuration diagram of an information processing device 100.

[0019] The information processing device 100 is composed of an editing unit 113, an input unit 114, an output unit 115 capable of outputting a virtual viewpoint image with authentication information, an authentication unit 116 that manages the authentication information, an operation unit 117, an instruction control unit 118, and a data storage means 104 that stores data used in these processes.

[0020] FIG. 3 is a diagram showing a hardware configuration for realizing the information processing device 100. As shown in FIG.

[0021] The information processing device 100 includes a CPU 311, a ROM 312, a RAM 313, an auxiliary storage device 314, an output unit 315, an operation unit 316, a communication I / F 317, and a bus 318. The CPU 311 controls each unit of the information processing device 100 and performs arithmetic processing on material data input from the communication I / F 317, using computer programs and data stored in the ROM 312 and RAM 313. The information processing device 100 may be configured to include one or more dedicated hardware components different from the CPU 311 and to execute at least a portion of the processing performed by the CPU 311. Examples of dedicated hardware components include processors used for image processing and control, ASICs (application-specific integrated circuits), field programmable gate arrays, GPUs, and DSPs (digital signal processors). The ROM 312 stores programs that do not require modification. The RAM 313 temporarily stores programs supplied from the auxiliary storage device 314, data used when each functional block of the information processing device 100 operates, and images and sounds output via the communication I / F 317. Hereinafter, data such as images and audio used to generate virtual viewpoint images will be referred to as material data. The auxiliary storage device 314 is configured, for example, as a hard disk or solid-state drive. The auxiliary storage device stores programs used to edit material data and generate virtual viewpoint images, as well as material data, its editing history, and authentication information. The output unit 315 is configured as an LCD display or an I / F for video streaming connection, and displays a GUI (Graphical User Interface) for operating the information processing device 100 and the processing results of the information processing device 100. The operation unit 316 is used to edit material data, input a camera path for virtual viewpoint images, and input instructions for generating virtual viewpoint images. The operation unit 316 is configured, for example, as a keyboard, mouse, joystick, touch panel, etc. The communication I / F 317 in FIG. 3 is used for communication between the information processing device 100 and an external device. The external device may be, for example, a data server that stores external data such as music and CG to be included in the virtual viewpoint images, or that uploads generated virtual viewpoint images. When the information processing device 100 is connected to an external device via a wired connection, a communication cable is connected to the communication I / F 317. If the information processing device 100 has a function for wireless communication with an external device, the communication I / F 317 includes an antenna. A bus 318 connects the various components of the information processing device 100 to transmit information.

[0022] The type of authentication information to be added to the virtual viewpoint image generated by the information processing device 100, the authentication server used to manage the authentication information, and the like use values ​​defined in advance within the information processing device 100. Hereinafter, data to be assigned to material data for determining whether or not authentication information can be assigned to a virtual viewpoint image will be referred to as authentication conditions. The authentication conditions indicate conditions for editing content that is permitted for editing of material data. The output unit 115 assigns authentication information to virtual viewpoint images and sets each block in the information processing device 100 used to generate and update authentication conditions for each material data.

[0023] The functional blocks of the information processing device 100 will be described below.

[0024] The input unit 114 includes an authentication data generating means 105 and a data input means 106 .

[0025] The authentication data generation means 105 initializes the editing history and authentication conditions of the material data by assigning initial values ​​for the editing history and authentication conditions to the material data based on instructions received by the data input means 106. The material data with the initialized editing history and authentication conditions is sent to the data storage means 104. When multiple material data are handled as one combination (set), an authentication condition may be set that combines the authentication conditions corresponding to each piece of material data, or a common authentication condition may be set.

[0026] The data input means 106 acquires material data from a camera or an external data server. In this embodiment, it is assumed that authentication conditions for the material data are also acquired when the material data is acquired. The authentication conditions may be set by the creator of the material data, or by a user who issues an instruction to the data input means 106 using the operation unit 117. Whether editing conditions are required for the acquired material data may be defined in advance and set in the data input means 106. For example, the method for determining whether data editing has occurred may be changed depending on the input device or data source, or editing determination details may be defined for each data format. The acquired material data and authentication conditions are sent to the authentication data generation means 105. If material data with an editing history and authentication conditions set in advance is acquired, the data may be sent directly to the data storage means 104 without going through the authentication data generation means 105.

[0027] The editing unit 113 includes an editing instruction input unit 101 , an editing control unit 102 , and an image processing unit 103 .

[0028] The editing instruction input means 101 accepts editing instructions from the user for the material data via the operation unit 117 and the instruction control unit 118. Editing is performed on material data used to generate a virtual viewpoint image, such as video and audio captured by a camera, existing 3D model data and music files, 2D images, and acquired sensor data. Editing instructions may include editing individual pieces of material data, as well as combining or compositing two or more pieces of material data. Examples of processing that handles multiple pieces of material data include compositing audio and video, combining and changing the position of background and foreground, and replacing textures applied to 3D objects.

[0029] The editing control means 102 generates an editing processing command to be executed by the image processing means 103, targeting the material data to be edited based on the editing instruction from the editing instruction input means 101. The editing processing command includes at least an editing operation that defines the content of correction to the material data, and data necessary for calculating the time code in the virtual viewpoint image of the edited image frame. The editing control means 102 saves the editing processing command issued to the material data in the editing history of the material data to be edited. If the calculation results of the editing processing by the image processing means 103 are to be included in the editing history of the editing processing command, the editing control means 102 instructs the image processing means 103 to output a log necessary to update the editing history, obtains the result, and updates the history information.

[0030] The image processing means 103 acquires material data from the data storage means 104, executes an editing processing command, and transmits the edited material data to the data storage means 104. If it is necessary to divide or combine authentication conditions as a result of editing the material data, the editing control means 102 updates the authentication conditions. For example, when editing by combining two or more pieces of material data, the authentication conditions for each piece of material data are combined. Furthermore, if the material data used as input when registering the 3D model data has usage restrictions or an editing history, that information must be carried over to the edited 3D model data. In this case, the editing control means 102 generates the authentication conditions to be assigned to the edited 3D model data.

[0031] The output unit 115 includes an output means 107 , a video output instruction means 108 , an output control means 109 , and a video generation means 110 .

[0032] The output means 107 outputs the virtual viewpoint image generated by the video generation means 110 to an external device (not shown).

[0033] The video output instruction means 108 receives an instruction to generate a virtual viewpoint image from the instruction control unit 118. The instruction to generate a virtual viewpoint image in the present disclosure includes a time code of the video and a camera path of the virtual viewpoint image to be output, as well as a setting of whether authentication information is to be assigned to the generated virtual viewpoint image and the type of authentication information to be assigned. The authentication information to be assigned to the virtual viewpoint image uses authentication information designated by the user, such as an NFT (Non-Fungible Token) that manages ownership and the number of times it is issued, copyright, access rights for viewing the virtual viewpoint image, and an electronic certificate indicating authenticity. It is also possible to provide authentication information that can be assigned in advance and allow the user to select one of the authentication information. The received instruction to generate a virtual viewpoint image is sent to the output control means 109.

[0034] Based on the generation instruction received from the video output instruction means 108, the output control means 109 specifies the camera path, the material data used to generate the virtual viewpoint image, and the start and end time codes, and instructs the video generation means 110 to generate the virtual viewpoint image. At this time, the output control means 109 acquires the material data used to generate the virtual viewpoint image, the authentication conditions of the material data, and the editing history from the data storage means 104. Upon receiving a notification from the video generation means 110 that the generation of the virtual viewpoint image has been completed, the output control means 109 instructs the data evaluation means 111 to determine the authentication conditions of the generated virtual viewpoint image and the material data, determine whether or not to assign authentication information to the generated virtual viewpoint image, and register the authentication information.

[0035] The image generating means 110 generates a virtual viewpoint image of a specified camera path from the raw data stored in the data storage means 104 .

[0036] The authentication unit 116 includes a data evaluation means 111 and an authentication data management means 112 .

[0037] The data evaluation means 111 determines whether or not authentication information can be assigned to the virtual viewpoint image instructed by the output control means 109. The determination of whether or not authentication information can be assigned is made by acquiring the virtual viewpoint image instructed by the output control means 109 and its authentication conditions from the data storage means 104, and analyzing the editing history and the generated virtual viewpoint image. When generating a virtual viewpoint image, if the authentication conditions of the data used as input include a condition that requires reconfirmation of the authentication information, the authentication data management means 112 is used to inquire about the authentication information of the material data.

[0038] Based on instructions obtained from the output control means 109, the authentication data management means 112 queries the authentication information of the material data used to generate the virtual viewpoint image, and registers and manages the authentication information of the generated virtual viewpoint image.

[0039] The operation unit 117 accepts inputs that specify the camera path of the virtual camera, inputs that select material data to be used in generating a virtual viewpoint image, and inputs that specify start and end time codes. For example, the operation unit 117 has input members such as a joystick, a jog dial, a touch panel, a keyboard, and a mouse. The client specifies the camera path and material data by operating the input members. The accepted inputs are sent to the instruction control unit 118.

[0040] In response to an input acquired from the operation unit 117, the instruction control unit 118 transmits instructions to each means of each unit based on the input content. For example, if the input is to register material data in the data storage means 104, the instruction is transmitted to the data input means 106. If the input is to edit material data, the instruction is transmitted to the editing instruction input means 101. If the input is to generate a virtual viewpoint video, the instruction is transmitted to the video output instruction means 108.

[0041] The data storage means 104 stores the material data and the metadata attached to the material data. The metadata will be described with reference to FIG. 2. Note that the data storage means 104 may store the virtual viewpoint image generated by the video generation means and the metadata attached to the virtual viewpoint image. Furthermore, although the present embodiment assumes that the data storage means 104 is built into the information processing device 100, the present invention is not limited to this. For example, an external device may be provided as the storage device.

[0042] Fig. 2 is a diagram illustrating authentication conditions for material data. Fig. 2(a) and (b) show authentication conditions for each piece of material data, where the material data are camera 1 images 206 to camera 3 images 208 and camera 3 audio 209 captured by a camera array connected to the information processing device 100. Fig. 2(c) shows a 3D model generated based on camera 1 images 206 to camera 3 images 208 and camera 3 audio 209, where the material data is a 3D model generated based on camera 1 images 206 to camera 3 images 208 and camera 3 audio 209.

[0043] FIG. 2(a) shows an example in which camera 1 images to camera 3 images and camera 3 audio are treated as a single combination of material data (hereinafter referred to as material data set 1). Metadata 201 is set for a pointer table 210 of material data set 1, and the metadata 201 includes a data format 254, authentication conditions 252, and a data address 253. The pointer table 210 corresponds to camera image 1 data 202 to camera image 3 data 204, camera 1 image editing history 211 to camera 3 image editing history 213, camera 3 audio data 205, and camera 3 audio editing history 214. The pointer table 210 includes camera 1 image pointer 206 to camera 3 image pointer 209. The data format 254 indicates the data structure of each material data in the pointer table 210. The authentication conditions 252 store the authentication conditions for each material data. The data address 253 stores a pointer used to access the material data and the editing history. In this embodiment, a pointer refers to an address in data storage means where the editing history and material data are stored. In Figure 2(a), authentication condition 252 is set to "editing history == unedited," which indicates that authentication information can be assigned to a virtual viewpoint image or 3D model generated using material data set 1 if it is unedited. In other words, if the material data has been edited when generating a virtual viewpoint image or 3D model using material data set 1, authentication information cannot be assigned to the generated virtual viewpoint image or 3D model. The process of assigning authentication information will be described later.

[0044] FIG. 2(b) shows an example in which metadata is added to a combination of camera 1 images and camera 2 images as one piece of material data (hereinafter referred to as material data set 2), camera 3 images, and camera 3 audio.

[0045] Metadata 215 is set for a pointer table 220 of material dataset 2, and the metadata 215 includes a data format 217, authentication conditions 218, and a data address 219. The pointer table 220 corresponds to camera 1 image data 202 to camera 2 image data 203, camera 1 image editing history 211 to camera 2 image editing history 212, and camera 1 image pointer 206 to camera 2 image pointer 207. Authentication conditions 218 are set to "no restriction," indicating that authentication information can be assigned to a virtual viewpoint image or 3D model generated using material dataset 2 even if it has been edited. In other words, when a virtual viewpoint image or 3D model is generated using only material dataset 2, authentication information is assigned to the generated virtual viewpoint image or 3D model even if the camera 1 image and camera 2 image have been edited. However, when a virtual viewpoint image is generated using other material data in addition to material dataset 2, whether authentication information is assigned to the generated virtual viewpoint image and 3D model is determined based on the authentication conditions of that material data.

[0046] Metadata 221 is set for a pointer 208 for the camera 3 image, and the metadata 221 includes a data format 222, authentication conditions 223, and a data address 224. The pointer 208 corresponds to camera 3 image data 204 and camera 3 image editing history 213. The authentication condition 223 is set to "editing history ≠ foreground retouching," which indicates that authentication information can be assigned to a virtual viewpoint image or 3D model generated using the camera 3 image as long as the subject (foreground) in the image is not edited. In this case, if the subject in the camera 3 image is edited, authentication information is not assigned to the virtual viewpoint image or 3D model generated using the camera 3 image. Note that, for an image containing multiple subjects, information specifying the subject for which editing is restricted may be set as an authentication condition. For example, if an image contains player A in the center of the field of view and player B at the edge, editing restrictions are enabled for player A, but editing restrictions are not set for player B. This makes it possible to clearly identify the subjects for which editing restrictions are valid, even when multiple subjects are included in the image.

[0047] Metadata 225 is set for a pointer 209 for camera 3 audio, and the metadata 225 includes a data format 226, authentication conditions 227, and a data address 228. Pointer 209 corresponds to camera 3 audio data 205 and camera 3 audio editing history 214. Authentication condition 227 is set to "editing history ≠ audio editing," which indicates that if there is no editing of the audio, authentication information can be added to a virtual viewpoint image or a 3D model generated using the camera 3 audio.

[0048] When generating a virtual viewpoint image or a 3D model using the material dataset 2, camera 3 images, and camera 3 audio shown in Figure 2(b), the authentication conditions are the combination of metadata corresponding to each pointer. Specifically, the authentication conditions are "no restrictions" for material dataset 2, "editing history ≠ foreground retouching" for camera 3 images, and "editing history ≠ audio editing" for camera 3 audio.

[0049] FIG. 2(c) shows an example of authentication conditions for a 3D model generated using images from cameras 1 to 3 and audio from camera 3.

[0050] Metadata 240 is set for 3D model pointer 243, and metadata 240 includes data format 237, authentication conditions 238, and data address 239. 3D model pointer 243 is associated with 3D model data 241, editing histories 211-214 of camera 1 images to camera 3 images and camera 3 audio used to generate the 3D model data, and 3D model editing history 242. Note that camera 1 image editing history 211 to camera 3 audio editing history 214 record the editing history when 3D model data 241 was generated. Since authentication condition 238 is "editing history == unedited," authentication information can be assigned to a virtual viewpoint image generated using unedited 3D model data 241. Note that authentication conditions for a 3D model generated using camera 1 images to camera 3 images and camera 3 audio are assumed to be set separately when the 3D model is generated.

[0051] FIG. 4 is a diagram illustrating authentication information of a virtual viewpoint image.

[0052] FIG. 4A illustrates an authentication condition that certifies the high reliability of a virtual viewpoint image. Metadata 401 of a virtual viewpoint image 405 includes an identifier 402 for identifying material data used to generate the virtual viewpoint image 405, authentication information 403, and a data address 404. The authentication information 403 is information indicating whether authentication information has been assigned to the virtual viewpoint image 405. In this embodiment, the authentication information includes information indicating that authentication information has been assigned, or if authentication information has been assigned, the authentication information 401 is included in the metadata 401. If authentication information has not been assigned, the authentication information 401 may not be included in the metadata 401. It is sufficient to determine whether authentication information has been assigned. If authentication information has been assigned, details of the authentication information are included in addition to the fact that authentication information has been assigned. In this embodiment, a digital certificate indicating that the virtual viewpoint image 405 was created satisfying the authentication condition is included. In addition to the digital certificate, an NFT may be issued and managed using a blockchain.

[0053] 4(b) shows an example in which authentication information including the specification of usage conditions for the virtual viewpoint image is further attached. The authentication information 406 includes authentication information 407 that the authentication data management means 112 references when requesting access to the virtual viewpoint image 408 or when secondary use is made. The authentication information 407 is used by setting usage conditions for use of material data inherited from material data and usage conditions for use of the virtual viewpoint image 408. For example, when distributing the virtual viewpoint image 408, the virtual viewpoint image 408 must be distributed with an icon indicating the digital certificate included in the authentication information 407 superimposed on it. Note that usage conditions may also be set to include a time limit for viewing the virtual viewpoint image and restrictions on secondary use.

[0054] 4(c) shows an example of distributing a 3D model generated using material data instead of a virtual viewpoint image. Metadata 409 is set for the 3D model 413, and the metadata 409 includes an identifier 410 of the material data used to generate the 3D model 413, authentication information 411, and a data address 412. The authentication information 410 indicates whether authentication information has been assigned to the 3D model 411.

[0055] By doing the above, it is possible to clarify whether the material data used to generate a virtual viewpoint image or a 3D model meets the authentication conditions, and highly reliable virtual viewpoint images can be generated.

[0056] FIG. 5 is a diagram illustrating the editing process of material data.

[0057] In S501, the editing instruction input means 101 acquires an editing instruction for material data owned by the user via the operation unit 117 and the instruction control unit 118. The editing instruction includes information for identifying the material data to be edited and the user's account information. The editing instruction input means 101 generates an editing processing command based on the acquired editing instruction and transmits it to the editing control means 102.

[0058] In S502, the editing control means 102 acquires the material data to be edited, the authentication conditions for the material data, and the editing history of the material data from the data storage means 104. Note that if an NFT has been issued for the material data, the material data to be edited may be identified by referencing transaction data in a blockchain (not shown) based on the account information acquired in S501.

[0059] In S503, the image processing means 103 generates an image showing the authentication conditions of the acquired material data, and notifies the user of the image using a display device (not shown).

[0060] In S504, the editing instruction input means 101 acquires editing content of the material data based on a user operation via the operation unit 117 and the instruction control unit 118. The editing content is, for example, content to change the texture of the 3D model. The acquired editing content is transmitted to the editing control means 102.

[0061] In S505, the editing control means 102 edits the material data in accordance with the acquired editing content, and then outputs an instruction to the image processing means 103 to generate an image showing the edited material data, and notifies the user of this using a display device (not shown).

[0062] In S506, the editing control means 102 updates the editing history of the material data by adding the edit content performed in S505 to the editing history of the material data acquired in S501. The editing history of the updated material data is transmitted to the data storage means 104, which stores it as the latest editing history. Note that the data storage means 104 may store only the latest editing history. In this case, the editing history also stores the date and time of each edit and account information.

[0063] In S507, the editing control means 102 waits until the next instruction is received. If the received instruction is to end editing, the editing process ends. Note that in S506, the editing history of the material data is updated each time editing is performed, but this is not limited to this, and the editing history may be updated all at once when an instruction to end editing is received. If the received instruction is not to end editing, the process proceeds to S504.

[0064] By the above process, the editing history of the material data can be updated.

[0065] In this embodiment, it is assumed that editing content that does not satisfy the authentication conditions for the material data is also reflected in the material data, but this is not limited to this. If the editing content acquired in S504 does not satisfy the authentication conditions for the material data, it may be restricted so that editing cannot be performed.

[0066] In this embodiment, it is assumed that the data storage means 104 is located inside the information processing device 100. If the data storage means 104 is configured as an external device, when updating the editing history in S506, the information to be transmitted, for example, the editing history, may be encrypted to prevent alteration of the editing history. In this case, an encryption key is shared with the external device in advance.

[0067] FIG. 6 is a diagram illustrating a process of adding authentication information to a generated virtual viewpoint image and outputting the image.

[0068] In S601, the video output instruction means 108 acquires an instruction to generate a virtual viewpoint image from the user via the operation unit 117 and the instruction control unit 118. The instruction to generate includes an instruction to specify the camera path of the virtual viewpoint, and the material data and time code used to generate the virtual viewpoint image. The acquired instruction to generate is sent to the output control means 109.

[0069] In S602, based on the acquired generation instruction, the output control means 109 transmits an instruction to the data storage means 104 to transmit material data to be used for generation to the video generation means 110. Furthermore, the output control means 109 transmits an instruction to the data storage means 104 to transmit editing history information and authentication conditions of the material data to the output control means 109. As a result, the video generation means 110 acquires the material data specified by the generation instruction, and the output control means 109 acquires the editing history and authentication conditions of the specified material data. The output control means 109 transmits the camera path and time code acquired in S601 to the video generation means 110.

[0070] In S603, the video generation means 110 generates a virtual viewpoint image based on the material data, camera path, and time code acquired in S602. The generated virtual viewpoint image is transmitted to the output means 107 and the data storage means 104.

[0071] In S604, the data evaluation means 111 sets an initial value for the evaluation result. In this embodiment, the evaluation result is set to 0.

[0072] In S605, the data evaluation means 111 determines whether the editing history and authentication information of all the material data used in the virtual viewpoint image generated in S603 have been checked. If all have been checked, the process proceeds to S609. If not all have been checked, the process proceeds to S606.

[0073] In S606, the data evaluation means 111 sets the unconfirmed material data as an evaluation target.

[0074] In S607, the data evaluation means 111 determines whether the editing history of the material data set as the evaluation target satisfies the authentication conditions for the material data. For example, if the material data is a 3D model, has no editing history ("editing history == unedited"), and the authentication condition is that the three-dimensional shape of the 3D model is not changed, it is determined that the authentication conditions are met. If the authentication conditions are met, proceed to S605. If the authentication conditions are not met, proceed to S608.

[0075] In S608, the data evaluation means 111 sets the evaluation result to 1. The same evaluation result is applied to all material data used to generate the virtual viewpoint image. In other words, if there is even one piece of material data that does not satisfy the authentication conditions among the material data used to generate the virtual viewpoint image, the evaluation result will be 1, and if there is no material data that does not satisfy the authentication conditions, the evaluation result will be 0.

[0076] In S609, the data evaluation means 111 checks all the material data used to generate the virtual viewpoint image and determines whether the evaluation result is 0. If the evaluation result is 0, the process proceeds to S610. If the evaluation result is not 0, the process proceeds to S611.

[0077] In S610, the data evaluation means 111 assigns authentication information to the virtual viewpoint image. Specifically, the data evaluation means 111 outputs an instruction to assign authentication information to the authentication data management means 112, and the authentication data management means 112 registers the authentication information to be assigned to the virtual viewpoint image. When the registration of the authentication information is completed, the authentication data management means 112 notifies the data evaluation means 111 that the registration is complete. The data evaluation means 111 outputs the registered authentication information to the data storage means 104, and the data storage means 104 stores the authentication information as metadata of the virtual viewpoint image. Note that when an NFT is used for the authentication information, a process of issuing an NFT is performed.

[0078] In S611, the output means 107 outputs the generated virtual viewpoint image to an external device. In addition, in S610, if authentication information is assigned, information or an image indicating that the authentication information has been assigned may be output. For example, an icon or an image indicating that the authentication information has been assigned may be provided and output by superimposing it on the generated virtual viewpoint image.

[0079] By the above processing, authentication information can be assigned only to virtual viewpoint images generated using material data that meets the authentication conditions, making it clear that the material data used to generate the virtual viewpoint images is being used appropriately.

[0080] In the present embodiment, the process of adding authentication information when generating a virtual viewpoint image has been described, but the present invention is not limited to this. Even in the process of generating a 3D model instead of generating a virtual viewpoint image, the processes of S605 to S610 are performed to consider whether to add authentication information.

[0081] Fig. 7 is a diagram showing the processing of an edit processing command for material data used to generate a virtual viewpoint image. Fig. 7(a) and (b) show an example of the processing of an edit processing command for the material data of Fig. 2(a), and Fig. 7(c) and (d) show an example of the processing of an edit processing command for adding material data of external audio to the material data of Fig. 2(a).

[0082] Editing process commands 701 to 703 in FIG. 7(a) are editing process commands targeted at the camera 1 image. Editing process command 701 adds object obj_a of a shape defined by shape1 to coordinate coord1 of frame 150 of the camera 1 image. Editing process command 702 moves obj_a, added to coordinate coord1 by edit process command 701, to coordinate coord2 by frame 2100. Editing process command 703 deletes obj_a in frame 2200. For the editing process commands 701 to 703, the editing control means 102 generates an editing history 710. It also assigns an identification ID to each editing process command and saves the editing process commands 701 to 705. The editing history 710 includes IDs 0 to 2 of the editing commands targeted at the camera 1 image, and information on frames 150 to 2200 affected by the editing. After the editing process commands are saved, the camera 1 image editing history 211 in FIG. 2 is recorded based on the editing process commands 701 to 703.

[0083] The edit process command 704 is an edit process command for the camera 2 image. The edit process command 704 adds obj_b, defined by shape2, to the coordinates of coord2 for frame 170. Because object obj_b is not deleted, the image processing means 103 adds obj_b up to the final frame of the camera 2 image. In response to the edit process command 704, the edit control means 102 generates an edit history 711. The edit history 711 includes the ID 3 of the edit command for the camera 2 image and information about frames 170 to the final frame image of camera 2 that are affected by the editing. No editing operations have been performed on the camera 3 image. The edit history 709 remains unchanged at -1, indicating no editing. After the edit process command is saved, the camera 2 image edit history 212 in Figure 2 is recorded based on the edit process command 704.

[0084] Editing process command 705 is an editing process command for camera 3 audio. Editing process command 705 clips the recording audio that is input in synchronization with frames 1000 to 2300 of the camera 3 image. Editing control means 102 generates editing history 713 for editing process command 705. Editing history 713 includes ID 4 of the editing command for camera 3 audio and information on frames 1000 to 2300 that will be affected by the editing. After the editing process command is saved, camera 3 audio editing history 214 in FIG. 2 is recorded based on editing process command 705.

[0085] FIG. 7(b) is a diagram for explaining the editing history of each piece of material data.

[0086] The video generation means 110 generates a virtual viewpoint image using the material data stored in the data storage means 104. When the generation of the virtual viewpoint image is completed, the output control means 109 outputs the frame number referenced when the virtual viewpoint image was generated and the editing history 710 to the data evaluation means 111. The data evaluation means 111 checks the editing history of the material data used to generate the virtual viewpoint image to determine whether or not authentication information should be assigned.

[0087] Consider generating a virtual viewpoint image using material data having metadata 201 in FIG. 2(a). Authentication condition 252 permits the assignment of authentication information provided that the material data is not edited. When generating a virtual viewpoint image using camera 1 images, camera 2 images, camera 3 images, and camera 3 audio, the video generation means 110 references editing history 710 to editing history 713 shown in FIG. 7(b). Editing history 710 indicates that editing has occurred for frames 150 to 2200 of camera 1 image, editing history 711 indicates that editing has occurred for frames 170 and onward of camera 2 image, and editing history 713 indicates that editing has occurred for frames 1000 to 2000 of camera 3 audio. The data evaluation means 111 determines that authentication information can be assigned to virtual viewpoint images of frames 0 to 149 where no image editing has occurred, but that authentication information cannot be assigned to frames 150 and onward. As described above, the editing history of each material data is referenced, and if even one piece of material data does not satisfy the authentication condition, it is determined that authentication information cannot be assigned. If it is desired to assign a plurality of pieces of authentication information, the authentication condition may be divided into a plurality of pieces, and whether or not to assign the authentication information may be determined according to the editing processing command for the material data.

[0088] The requirements for assigning authentication information to a virtual viewpoint image can be changed by instructions from the video output instruction means 108. For example, consider a case where authentication information is assigned when a virtual viewpoint image is generated with reference to camera 1 image, camera 2 image, camera 3 image, and camera 3 audio, and there is no editing of the image or audio. In this case, if the editing history shown in FIG. 7(b) is input to the data evaluation means 111, there is audio editing in frames 1000 to 2300 of the audio during recording, so authentication information can only be assigned to frames 0 to 149 and from frame 2301 onwards. The editing process commands for adding external audio to Fig. 7(a) are shown in Fig. 7(c), and the editing history is shown in Fig. 7(d). Editing process commands 706-707 read the audio data specified in file_1 and add it to the audio track of frames 120-1500. The editing control means 102 records frames 120-2000 to be synthesized in the editing history 714. However, since there is no editing of the audio data itself in file_1, the audio data in the editing history 714 is marked as unedited. In this case, since the editing history 713 shows that editing was performed on the audio recorded in frames 1000 to 2300, it is determined that authentication information can be added if the virtual viewpoint image is generated from frames 0 to 119, but that authentication information cannot be added from frame 120 onwards.

[0089] As another example, consider a case where a virtual viewpoint image is created by adding externally acquired audio data to the shooting data using the editing process command 707, and authentication information is assigned to the virtual viewpoint image without any image or audio editing. In this case, as a result of the editing process command 707, audio is added to frames 120 to 1500, as shown in the editing history 714. While the externally acquired audio data itself remains unchanged, superimposing external audio on frames that already contain audio during recording changes the audio data of the virtual viewpoint image in frames 120 to 149. For this reason, the period during which authentication information can be assigned to the generated virtual viewpoint image is limited to frames 0 to 119.

[0090] Although the example of generating a virtual viewpoint image has been described, the present invention is not limited to this, and a 3D model may be generated. In this case, it is determined whether or not to assign authentication information to the 3D model.

[0091] Consider a case where the data evaluation means 111 determines whether or not to grant authentication information using material data having authentication information (b) in Figure 2(b), the editing processing command in Figure 7(c), and the editing history in Figure 7(d).

[0092] Authentication condition 223 of authentication information (b) does not permit editing of the foreground model of the camera 3 image, and authentication condition 227 does not permit editing of the camera 3 audio, but does permit editing of the camera 1 image and camera 2 image. When editing history 710-714 is evaluated by data evaluation means 111, authentication information cannot be assigned to frames 120-2300, where audio has been edited due to the addition of external audio in editing processing commands 706-707. However, it is determined that authentication information can be assigned to frames 0-119 and virtual viewpoint images from frame 2301 onwards, where the camera 3 image and camera 3 audio are unchanged.

[0093] The above processing makes it possible to manage the editing content of the material data used to generate the virtual viewpoint image, and to determine whether the editing history and the generated video satisfy the conditions for assigning authentication information to the virtual viewpoint image.

[0094] <Embodiment 2> In the first embodiment, the modification history of the material data used for the virtual viewpoint image is referenced, and if the authentication conditions for each material data are met, authentication information is assigned to the generated virtual viewpoint image. However, there is a risk that an inappropriate virtual viewpoint image will be generated depending on the combination of material data. For example, in a virtual viewpoint image used for sports judging, the relative positions of multiple players, and the relative positions and shapes of players and competition lines, goals, equipment, etc., must be maintained as they were when the images were recorded. In this case, if the relative positions of the players are changed, a so-called fake image will be generated. In the second embodiment, in addition to the authentication conditions for each material data, utilization conditions such as the combination and arrangement of each material data are set, and authentication information is assigned to the virtual viewpoint image if the authentication and utilization conditions for each material data are met.

[0095] In this embodiment, the utilization conditions are set for each piece of material data. However, the utilization conditions are not limited to this and may be set according to the characteristics of the virtual viewpoint image to be generated. For example, if a single subject occupies 80% or more of the virtual viewpoint image, authentication information may not be assigned.

[0096] FIG. 8 shows authentication conditions and utilization conditions for material data. In this embodiment, the material data is described as a 3D model generated based on camera 1 images to camera 3 images and camera 3 audio. Metadata 802 is set for a pointer table 801 for the 3D model, and the metadata 802 includes a data format 803, authentication conditions and utilization conditions 804, and a data address 805. The pointer table 801 corresponds to camera 1 image editing history 211 to camera 3 image editing history 213, camera 3 audio editing history 248, 3D model data 806, and 3D model editing history 807. In the authentication conditions and utilization conditions 804, the utilization condition is set to "object margin > distance," which indicates a condition regarding the placement of objects (3D models of subjects) in the virtual space. This condition indicates that authentication information can be assigned when the distance between multiple placed 3D models is greater than a predetermined distance. Note that the utilization conditions are not limited to the above. Examples of utilization conditions determined by the data evaluation means 111 are shown below.

[0097] As a utilization condition, a specific user can be permitted to view or edit a virtual viewpoint image with authentication data. In this case, information on users permitted to use the material data is used when determining the authentication information. If this utilization condition is set in the authentication information of the input material data, the data evaluation means 111 inquires about permitted users using the authentication data management means 112. When a virtual viewpoint image is generated using this material data, it is determined that authentication information can be assigned if the user of the service application 203 that performs editing and virtual viewpoint image generation is a user who is permitted to use the material data. Furthermore, the output control means 109 may include a pointer to the authentication information assigned to the referenced material data in the authentication information of the generated virtual viewpoint image.

[0098] As a utilization condition, the assignment of authentication information can be permitted when a virtual viewpoint image is output in a format defined in the utilization condition. This condition is used, for example, to guarantee the quality of the generated virtual viewpoint image or to restrict secondary use. The condition for assigning authentication information is defined and used so that the output virtual viewpoint image is used limited to a predetermined resolution, frame rate, size, bit rate, video format including encryption, or storage area on the output device. The data evaluation means 111 permits the assignment of authentication information when the generated virtual viewpoint image is saved in a format defined in the utilization condition.

[0099] As a utilization condition, a condition prohibiting editing of audio or video can be set. When the editing control means 102 edits material data including this utilization condition, the editing control means 102 assigns an edited history to the data and stores it in the data storage means 104. The video generation means 110 stores in the data storage means 104 the editing history of the material data used when generating the virtual viewpoint image, and authentication information. When there is no history of deletion of authentication information in the editing history of the material data referenced when generating the virtual viewpoint image, the data evaluation means 111 permits the assignment of authentication information to the generated virtual viewpoint image.

[0100] As a utilization condition, restrictions can be placed on color changes to material data for which utilization conditions are specified. These restrictions are used to manage differences between the original color space of the material data and the color space and color depth saved by the image processing means 103 and the image generation means 110, as well as changes to the color of the textures of images and 3D models included in the material data and the color of the entire screen. Changes to specific objects or color gamuts within the material data, such as the color of a person's clothing or skin, may also be specified as targets. When editing material data with authentication information that restricts color changes, the editing control means 102 records the changes in the editing history if it detects an edit involving a color change to a restricted object or the entire screen. The data evaluation means 111 checks whether the editing history of the material data stored in the data storage means 104 contains any restricted editing operations. It also uses transparent compositing to check that there are no history of operations affecting the color of the material data in layers above it. If there are no color changes, the authentication information is permitted to be added to the generated virtual viewpoint image.

[0101] As a utilization condition, a condition that imposes restrictions on changing the position of a specified 3D model can be set. This utilization condition applies to data for which the shape information of a generated 3D model and the world coordinates on the 3D model are predetermined. When editing material data for which this utilization condition is set, the editing control means 102 records the change in the editing history when it detects an editing command that includes a change in the position of the 3D model. The data evaluation means 111 permits the assignment of authentication information if there is no history of a change in position in the editing history of this material data stored in the data storage means 104.

[0102] As a condition for use, editing of the background is prohibited in virtual viewpoint images that have a distinction between background and foreground. The data evaluation means 111 permits the assignment of authentication information to the material data if there is no change to the material data that constitutes the background. If the background is made up of a combination of multiple material data, it confirms that none of the material data has been changed. It also confirms that there are no changes to the position or scale of background and foreground objects.

[0103] A condition that places restrictions on the addition of foreground models can be added as a utilization condition. This condition prohibits the addition of a new model to the foreground in a virtual viewpoint image where there is a distinction between the background and the foreground. If a foreground object is added, the data evaluation means 111 determines that the material data cannot be authenticated. This condition may be designed to have area information in the 3D model space that restricts the addition of foreground models as needed, and to restrict the addition of new objects to that area.

[0104] As a utilization condition, a condition that imposes restrictions on editing of a foreground model can be set. This condition prohibits editing of material data used to generate a foreground object in a virtual viewpoint image in which the background and foreground are distinguished. If the material data referenced when generating the foreground has an editing history, the data evaluation means 111 determines that the material data cannot be authenticated.

[0105] As a utilization condition, it is possible to set whether a specific object is included in the generated virtual viewpoint image. This condition can be used to confirm that the generated virtual viewpoint image contains a virtual advertisement or logo, or that it does not contain any object that is not permitted for secondary distribution. The data evaluation means 111 performs pattern matching or object detection using AI to determine whether a specific object is included in the generated virtual viewpoint image, and if the desired detection result is obtained, it allows the assignment of authentication information to the material data.

[0106] As a utilization condition, a condition that imposes restrictions on editing of objects in a specified area can be set. In this condition, if editing occurs within a specific area of ​​a 2D or 3D video, the data evaluation means 111 determines that the material data cannot be authenticated.

[0107] As a utilization condition, restrictions can be set on the area in which the virtual camera is placed. The data evaluation means 111 permits the assignment of authentication information to the material data if the trajectory of the virtual camera path of the virtual viewpoint image does not include a prohibited area.

[0108] 9 is a diagram illustrating the process of adding authentication information to a generated virtual viewpoint image and outputting it. Note that steps S601 to S611 are the same as those in FIG. 6, and therefore the description thereof will be omitted.

[0109] In S901, the data evaluation means 111 determines whether the material data set as the evaluation target satisfies the utilization conditions. For example, suppose the material data is a 3D model of athlete A. Furthermore, the utilization conditions are "When the material data and the 3D model of athlete B are placed on the background model of stadium C, athlete A will be closer to the finish line than athlete B within 2 meters of the finish line of stadium C, and will reach the finish line first." The generated virtual viewpoint image is checked, and if the utilization conditions are met, the process proceeds to S605; if the utilization conditions are not met, the process proceeds to S608. Note that the utilization conditions may be determined simultaneously with the authentication conditions. In this case, S901 is incorporated into S607.

[0110] By the above process, authentication information can be assigned to virtual viewpoint images that satisfy the authentication conditions and utilization conditions of the material data, thereby enabling more detailed setting of targets to which authentication information is assigned.

[0111] <Embodiment 3> In the first embodiment, the information processing device 100 refers to the editing history of each piece of material data specified by the user, and if the authentication conditions for each piece of material data are met, authentication information is assigned to the generated virtual viewpoint image. However, it is also possible that each piece of material data specified by the user is transmitted to the user device, and the virtual viewpoint image is generated on the user device side. Therefore, in the second embodiment, authentication information is assigned to a combination of multiple pieces of material data specified by the user.

[0112] Fig. 10 is a diagram illustrating authentication information for a material data set. In this embodiment, explanation will be given using material data set 1 in Fig. 2(a). Metadata 1001 for material data set 11005 includes an identifier 1002 for identifying the contents of material data set 1, authentication information 1003, and a data address 1004. Identifier 1002 stores identifiers for identifying the contents of material data set 1, namely, images from cameras 1 to 3 and audio from camera 3.

[0113] FIG. 11 is a diagram for explaining the process of adding authentication information to a material data set and outputting it. Note that S602 and S604 to S610 are the same processes as those in FIG. 6, and therefore the description thereof will be omitted.

[0114] In S1101, video output instruction means 108 acquires an instruction to generate a material data set from the user via operation unit 117 and instruction control unit 118. Note that the generation instruction includes an instruction to specify multiple pieces of material data.

[0115] In S1102, the output control means 109 generates a material data set.

[0116] In S1103, the data evaluation means 111 assigns authentication information to the material data set.

[0117] By the above process, authentication information can be assigned to the material data set. Note that, although the present embodiment has been described using material data set 1 as an example, the present invention is not limited to this and may also be applied to a material data set that combines multiple 3D models.

[0118] In the above embodiment, authentication information is assigned to a virtual viewpoint image or a 3D model generated using material data, but this is not limiting. For example, authentication information can be assigned to material data output by inputting material data into a specific learning model.

[0119] Note that a computer program that realizes part or all of the control in this embodiment and the functions of the above-described embodiment may be supplied to an image processing system or the like via a network or various storage media. A computer (or a CPU, MPU, or the like) in the image processing system or the like may then read and execute the program. In this case, the program and the storage medium storing the program constitute the present disclosure.

[0120] The disclosure of this embodiment includes the following configurations, methods, and programs.

[0121] (Configuration 1) a specifying means for specifying an editing history of a plurality of pieces of material data used to generate a virtual viewpoint image; an assigning means for assigning authentication information to a virtual viewpoint image generated based on the plurality of material data when the editing histories of the plurality of material data identified by the identifying means each satisfy a predetermined condition; An apparatus comprising:

[0122] (Configuration 2) The apparatus according to configuration 1, wherein the adding means does not add the authentication information to the material data if at least one of the editing histories of the plurality of material data does not satisfy the predetermined condition.

[0123] (Configuration 3) 3. The device according to configuration 1 or 2, wherein the predetermined conditions are set by a creator who has created the material data.

[0124] (Configuration 4) 4. The apparatus according to any one of configurations 1 to 3, wherein the predetermined condition is a condition indicating editing content that is permitted for editing the material data.

[0125] (Configuration 5) 5. The device according to any one of configurations 1 to 4, wherein the authentication information is information that specifies an access right for viewing the virtual viewpoint image.

[0126] (Configuration 6) The device described in any one of configurations 1 to 5, characterized in that the authentication information is represented using an NFT.

[0127] (Configuration 7) 7. The device according to any one of configurations 1 to 6, wherein the material data is at least one of 3D model data, audio data, and image data.

[0128] (Configuration 8) The material data is 3D model data, The device according to configuration 7, wherein the predetermined condition is a condition indicating editing content that changes the color of the 3D model.

[0129] (Configuration 9) The material data is 3D model data, 8. The apparatus according to configuration 7, wherein the predetermined condition is a condition that restricts the position of the 3D model in a virtual space in which the 3D model is placed.

[0130] (Configuration 10) The device according to configuration 9, wherein the predetermined condition is a condition that restricts the position of the 3D model based on a positional relationship between the 3D model and a different 3D model.

[0131] (Configuration 11) 11. The device according to any one of configurations 1 to 10, wherein the specifying means specifies the plurality of material data based on a user operation.

[0132] (Configuration 12) The material data is material data for which an NFT indicating ownership has been issued, 12. The apparatus according to claim 11, wherein the specifying means specifies the material data to which the user has ownership.

[0133] (Configuration 13) 13. The device according to any one of configurations 1 to 12, wherein the virtual viewpoint image is generated based on a plurality of captured images obtained by capturing images using a plurality of imaging devices.

[0134] (Configuration 14) 14. The device according to any one of configurations 1 to 13, further comprising output means for outputting the virtual viewpoint image to another device.

[0135] (Configuration 15) a specifying means for specifying an editing history of material data used to generate a virtual viewpoint image; an assigning means for assigning authentication information to the material data when the editing history of the material data identified by the identifying means satisfies a predetermined condition; An apparatus comprising:

[0136] (Configuration 16) a specifying means for specifying an editing history of a plurality of pieces of material data used to generate a virtual viewpoint image; an assigning means for assigning authentication information to a combination of the plurality of material data when the editing histories of the plurality of material data identified by the identifying means each satisfy a predetermined condition; An apparatus comprising:

[0137] (Method 1) a specifying step of specifying an editing history of a plurality of pieces of material data used to generate a virtual viewpoint image; an assigning step of assigning authentication information to a virtual viewpoint image generated based on the plurality of material data when the editing histories of the plurality of material data identified by the identifying step each satisfy a predetermined condition; A method comprising:

[0138] (Method 2) a specifying step of specifying an editing history of material data used to generate a virtual viewpoint image; an assigning step of assigning authentication information to the material data when the editing history of the material data identified by the identifying step satisfies a predetermined condition; A method comprising:

[0139] (Method 3) a specifying step of specifying an editing history of a plurality of pieces of material data used to generate a virtual viewpoint image; an assigning step of assigning authentication information to a combination of the plurality of material data when the editing histories of the plurality of material data identified by the identifying step each satisfy a predetermined condition; A method comprising:

[0140] (program) A program for causing a computer to function as each means of the device according to any one of configurations 1 to 16. [Explanation of symbols]

[0141] 102 Editing Control Means 104 Data storage means 105 Authentication data generation means 111 Data Evaluation Tools 112 Authentication data management measures

Claims

1. A specifying means for specifying an editing history of material data used to generate a virtual viewpoint image based on a plurality of captured images; an assigning means for assigning authentication information to a virtual viewpoint image generated based on the material data when the editing history of the material data identified by the identifying means satisfies a predetermined condition; An information processing device comprising:

2. The information processing device described in Claim 1, characterized in that the granting means does not grant the authentication information to the virtual viewpoint image if the editing history of the material data does not satisfy the specified conditions.

3. An information processing device as described in Claim 1, characterized in that the specified conditions are set by the creator who created the material data.

4. The information processing device according to claim 1, characterized in that the specified conditions are conditions indicating the editing content that is permitted for editing the material data.

5. The information processing device described in Claim 1, characterized in that the authentication information is information that specifies access rights for viewing the virtual viewpoint image.

6. An information processing device as described in Claim 1, characterized in that the authentication information is represented using NFT.

7. The information processing device described in Claim 1, characterized in that the material data is 3D model data indicating a 3D model of an object.

8. The information processing device described in Claim 7, characterized in that the specified condition is a condition indicating editing content that changes the color of the 3D model.

9. The information processing device according to claim 7, characterized in that the specified condition is a condition that restricts the position of the 3D model in the virtual space in which the 3D model is placed.

10. The information processing device according to claim 9, characterized in that the specified condition is a condition that restricts the position of the 3D model based on a positional relationship between the 3D model and a different 3D model.

11. The information processing device described in Claim 1, characterized in that the identification means identifies the material data based on user operation.

12. The material data is material data for which an NFT indicating ownership has been issued, 12. The information processing apparatus according to claim 11, wherein said specifying means specifies said material data whose ownership is owned by said user.

13. The information processing device according to claim 1, further comprising an output means for outputting the virtual viewpoint image to another device.

14. A specifying step of specifying an editing history of material data generated based on a plurality of captured images and used to generate a virtual viewpoint image; an assigning step of assigning authentication information to a virtual viewpoint image generated based on the material data when the editing history of the material data identified by the identifying step satisfies a predetermined condition; An information processing method comprising:

15. A program for causing a computer to function as each means of an information processing device described in any one of claims 1 to 13.

Images