Information processing system, information processing method, and program

Abstract

To solve the problem that there is no mechanism for easily managing a generated 3D model for each imaging matter when a plurality of imaging matters are simultaneously imaged.SOLUTION: An information processing system acquires a plurality of three-dimensional models generated based on a plurality of captured images obtained by simultaneously capturing a subject corresponding to each of a plurality of imaging matters, associates one of the plurality of imaging matters with one of the plurality of three-dimensional models, and outputs information indicating a combination of the associated imaging matter and three-dimensional model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a 3D model of a subject that is generated based on a plurality of captured images. [Background technology]

[0002] There is a technology that generates a virtual viewpoint image, which is an image captured from a virtual viewpoint, using a 3D model of a subject generated using multiple captured images captured by an imaging system consisting of multiple imaging devices. This technology can provide virtual viewpoint images captured from positions where a real imaging device cannot be placed, for example, in sports such as soccer or basketball.

[0003] Patent Document 1 describes a system in which photographs are taken for each customer in a photo studio and a folder is created and managed for each customer. When applied to a system that generates virtual viewpoint images, it is conceivable to take photographs for each shooting project and manage the 3D models of the subjects by the time of shooting. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-203598 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in order to efficiently shoot in a studio that generates virtual viewpoint images, it is conceivable to shoot multiple projects simultaneously. In this case, it is necessary to manage the generated 3D models for each shoot, but no mechanism for doing so had been considered.

[0006] The present disclosure aims to provide a system that allows for easy management of generated 3D models for each photography project. [Means for solving the problem]

[0007] In order to solve the above problems, the information processing system according to the present disclosure has the following configuration: an acquisition means for acquiring a plurality of 3D models generated based on a plurality of captured images obtained by simultaneously capturing subjects corresponding to a plurality of photographic subjects, a correlation means for correlating one of the plurality of photographic subjects with one of the plurality of 3D models, and an output means for outputting information indicating the combination of the photographic subjects and the 3D models correlated by the correlation means. [Effects of the Invention]

[0008] According to the present disclosure, generated 3D models can be easily managed for each photography project. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram illustrating an example of an entire system according to a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the hardware configuration of an information processing device. [Figure 3] FIG. 10 is a diagram illustrating an example of case identification information according to the first embodiment. [Figure 4] 10A to 10C are diagrams illustrating a process of identifying a 3D model of a subject and a corresponding photography subject according to the first embodiment. [Figure 5] 10 is a flowchart illustrating a process of a case specifying unit 6 according to the first embodiment. [Figure 6] 10 is a diagram showing information indicating a combination of a 3D model and a photographing subject generated by a subject specifying unit 6 according to the first embodiment. FIG. [Figure 7] FIG. 10 is a diagram showing a confirmation screen for information indicating a combination of a 3D model and a photography subject displayed on the input / output unit 7 according to the first embodiment. [Figure 8] FIG. 2 is a diagram showing a photographing case file including data for each photographing case according to the first embodiment. [Figure 9] FIG. 10 is a sequence diagram illustrating a process flow for displaying an AR image of a 3D model of a subject corresponding to a photography case according to the first embodiment. [Figure 10] FIG. 2 is a diagram illustrating an example of an AR image displayed on the user terminal 10 according to the first embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example of an entire system according to a second embodiment. [Figure 12] FIG. 11 is a diagram illustrating an example of case identification information according to the second embodiment. [Figure 13] FIG. 10 is a diagram illustrating an imaging system according to a second embodiment. [Figure 14] 10 is a flowchart illustrating a process of a case specifying unit 6 according to the second embodiment. [Figure 15] FIG. 11 is a diagram showing a shooting time file including data for each shooting time according to the third embodiment. [Figure 16] FIG. 11 is a sequence diagram showing the flow of a process for displaying an AR image of a 3D model of a subject corresponding to a photography case according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] <Embodiment> According to a preferred embodiment of the present disclosure, an information processing system includes an acquisition unit that acquires a plurality of 3D models generated based on a plurality of captured images obtained by simultaneously capturing a subject corresponding to each of a plurality of photographic subjects. The information processing system also includes a correlation unit that correlates one of the plurality of photographic subjects with one of the plurality of 3D models. The information processing system also includes an output unit that outputs information indicating the combination of the photographic subjects and the 3D models correlated by the correlation unit.

[0011] This aspect allows for easy management of the 3D models of the subjects for each shooting project, even when multiple shooting projects are simultaneously photographed and 3D models of multiple subjects are generated. There are several management methods. For example, a file for each shooting project may be generated using information indicating the combination of the shooting project and the 3D model. The file may include 3D models of the subjects corresponding to the shooting project, but not 3D models of subjects not corresponding to the shooting project. Alternatively, data including 3D models of multiple subjects corresponding to multiple shooting projects may be associated with information indicating the combination of the shooting projects and the 3D models for management. In this case, when generating a virtual viewpoint image using the 3D models, 3D models of subjects corresponding to shooting projects other than the desired shooting project are made transparent. This allows for the generation of a virtual viewpoint image that includes only 3D models of subjects corresponding to the desired shooting project.

[0012] The associating means associates one of the plurality of photographic subjects with one of the plurality of 3D models based on the position information of one of the plurality of 3D models. Multiple associating processes are possible. For example, the plurality of photographic subjects may be associated with different regions. In this case, a photographic subject associated with a region containing one of the plurality of 3D models among the plurality of photographic subjects is associated with one of the plurality of 3D models. Another associating process may involve using a tracking device. For example, the plurality of photographic subjects may be associated with different tracking devices, and a photographic subject associated with the tracking device located closest to the position of one of the plurality of 3D models among the plurality of photographic subjects is associated with the one of the plurality of 3D models.

[0013] The information processing system also includes a generation unit that generates multiple files containing 3D models of subjects corresponding to the photography project based on information indicating a combination of the photography project and the 3D model. These multiple generated files are recorded in a recording unit. Note that this recording unit is assumed to be a database accessible by users via a management server. The information processing system also includes an identification unit that identifies, among the multiple files, a file corresponding to a photography project identified by a first user operation. Note that the first user operation is an operation in which a user photographs a QR code (registered trademark) or a marker using a camera mounted on a smartphone or tablet device. This operation allows information identifying the photography project indicated in the QR code or marker to be obtained, and the database can be accessed via the management server.

[0014] This aspect allows the user to obtain a 3D model of the subject that corresponds to the desired shooting time.

[0015] The information processing system also includes a generating unit that generates a virtual viewpoint image using the 3D model included in the identified file.

[0016] This aspect allows the user to view a virtual viewpoint image generated using a 3D model of a subject corresponding to a desired photographic subject.

[0017] The information processing system also includes a display control unit that controls the display of a composite image generated based on the captured image acquired by a second user operation and the virtual viewpoint image. The second user operation is an operation of capturing a picture of a landscape or an object using a camera mounted on a smartphone or tablet device. The first user operation and the second user operation may be the same.

[0018] This aspect allows the user to view AR images using virtual viewpoint images.

[0019] The information processing system also includes a correction unit that corrects the information indicating the combination of the photography subject and the 3D model based on a third user operation. The third user operation is an operation performed by an administrator who manages photography to determine whether the combination of the photography subject and the 3D model is appropriate and correct it if it is inappropriate. Therefore, the first user operation and the third user operation are performed by different users.

[0020] Example 1 Preferred embodiments of the present disclosure will be described in detail below with reference to the drawings. Note that the following embodiments do not limit the present disclosure, and not all of the combinations of features described in the embodiments are necessarily essential to the solutions of the present disclosure. Furthermore, in the drawings, the same reference numerals are used to designate the same or similar components, and redundant descriptions will be omitted.

[0021] In this embodiment, multiple photography projects are simultaneously photographed in a single studio, and multiple 3D models containing multiple subjects corresponding to the multiple photography projects are generated. Then, the photography projects to which each of the multiple 3D models corresponds are identified, and information indicating the combinations of the 3D models and the photography projects is generated. The administrator in charge of managing the photography associates the 3D models with the photography projects using the information indicating the combinations as a reference. This makes it possible to easily manage the 3D models for each photography project, even when multiple photography projects are simultaneously photographed.

[0022] As an example of how the results of associating 3D models with photography projects can be used, a file containing a 3D model of the corresponding subject is generated for each photography project and uploaded to the cloud. Users who request photography projects can access the cloud through the management server, obtain virtual viewpoint images generated by the management server using the 3D model corresponding to the photography project, and view them on their devices.

[0023] The information processing system is a system that generates a virtual viewpoint image representing a scene from a specified virtual viewpoint based on multiple images captured by multiple imaging devices and a specified virtual viewpoint. The virtual viewpoint image in this embodiment is also called a free viewpoint video, but is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by a user. For example, the virtual viewpoint image also includes an image corresponding to a viewpoint selected by a user from multiple candidates. Furthermore, in this embodiment, the description will focus on a case where the virtual viewpoint is specified by a user operation, but the virtual viewpoint may also be specified automatically based on the results of image analysis, etc. Furthermore, in this embodiment, the description will focus on a case where the virtual viewpoint image is a video, but the virtual viewpoint image may also be a still image.

[0024] The viewpoint information used to generate a virtual viewpoint image is information indicating the position and orientation (line of sight direction) of the virtual viewpoint. Specifically, the viewpoint information is a parameter set including a parameter indicating the three-dimensional position of the virtual viewpoint and a parameter indicating the orientation of the virtual viewpoint in the pan, tilt, and roll directions. Note that the content of the viewpoint information is not limited to the above. For example, the parameter set serving as viewpoint information may include a parameter indicating the size of the field of view (angle of view) of the virtual viewpoint. Furthermore, the viewpoint information may have multiple parameter sets. For example, the viewpoint information may have multiple parameter sets corresponding to multiple frames constituting a moving image of the virtual viewpoint image, and may be information indicating the position and orientation of the virtual viewpoint at each of multiple consecutive time points.

[0025] The information processing system has multiple imaging devices that capture images of an imaging area from multiple directions. The imaging area may be, for example, a stadium where sports such as soccer or karate are held, or a stage where a concert or a play is held. The multiple imaging devices are installed at different positions surrounding the imaging area and capture images synchronously. Note that the multiple imaging devices do not need to be installed around the entire periphery of the imaging area; depending on installation space restrictions, they may be installed only around a portion of the periphery of the imaging area. Furthermore, the number of imaging devices is not limited to the example shown in the figure. For example, if the imaging area is a soccer stadium, approximately 30 imaging devices may be installed around the stadium. Furthermore, imaging devices with different functions, such as telephoto cameras and wide-angle cameras, may be installed.

[0026] In this embodiment, the multiple image capturing devices are cameras each having an independent housing and capable of capturing images from a single viewpoint. However, this is not limiting, and two or more image capturing devices may be configured in the same housing. For example, a single camera equipped with multiple lens groups and multiple sensors and capable of capturing images from multiple viewpoints may be installed as the multiple image capturing devices.

[0027] (System Configuration) FIG. 1 is a block diagram illustrating an example of an overall system according to a first embodiment.

[0028] The overall system includes an information processing system, a recording unit 8, a management server 9, and a user terminal 10. The information processing system includes a plurality of imaging units 1, a synchronization unit 2, a three-dimensional shape estimation unit 3, a matching unit 4, and an input / output unit 7. The matching unit 4 includes a subject separation unit 5 and a case identification unit 6. Note that each unit of the information processing system may be a system made up of multiple devices, or a system in which multiple units are made up of a single device. In the following explanation, the matching unit 4 is assumed to be one information processing device, and the remaining units are each assumed to be made up of individual devices.

[0029] The multiple imaging units 1 capture images in synchronization with one another based on a synchronization signal from the synchronization unit 2. The imaging units 1 output the captured images to the three-dimensional shape estimation unit 3. The multiple imaging units 1 are installed to surround a capture area including the subject, so that the subject can be captured from multiple directions.

[0030] The synchronization unit 2 outputs a synchronization signal to the plurality of image capture units 1 .

[0031] The three-dimensional shape estimation unit 3 uses the input captured images to generate, for example, a silhouette image of the subject, and then generates the three-dimensional shape of the subject using a volume intersection method or the like. In this embodiment, the three-dimensional shape of the subject is referred to as a 3D model of the subject. The 3D model of the subject may also include color information of the three-dimensional shape of the subject. For example, if the three-dimensional shape of the subject is represented by a point cloud, the color information is the RGB value of each point in the point cloud. The color information is not limited to RGB values, and may also be captured images for determining the color. For example, if the three-dimensional shape of the subject is represented by a mesh model, the color information may also be a texture map.

[0032] The three-dimensional shape estimation unit 3 also associates the generated 3D model of the subject with the shooting time of the captured image and outputs it to the association unit 4. That is, a 3D model of the subject is generated for each shooting time of images captured by the multiple image capture units 1, and the generated 3D model of the subject is associated with the shooting time and output. At this time, multiple captured images corresponding to the same shooting time may be output together. In this embodiment, it is assumed that the data is output to the association unit 4, but it may also be output to a recording unit (not shown). In this case, the association unit 4 acquires 3D models organized by shooting time from the recording unit (not shown). Note that the format of the association is not limited, and for example, a single file may contain the 3D model of the subject and information indicating the shooting time. Alternatively, a file name may include the shooting time, and a file containing the 3D model of the subject may be output to the association unit 4. Here, the subject refers to an object for which a 3D model is to be generated, including a person and items handled by the person. Note that the shooting time in this embodiment is also called the shooting date and time and is the time when the image was captured by the image capture unit 1. It may include the year, month, day, hour, minute, and second.

[0033] In addition, in this embodiment, since it is assumed that multiple shooting subjects will be photographed simultaneously, the generated 3D model will include multiple subjects. However, since there is no information indicating which of the components of the generated 3D model corresponds to which subject, in order to properly manage the 3D models of the subjects for each shooting subject, a process is performed to separate the generated 3D models by subject. In other words, the generated 3D models are separated or classified by subject. In this embodiment, the process of separating the generated three-dimensional shapes by subject is performed by the object separation unit 5.

[0034] The object separation unit 5 performs a process of separating the 3D models generated by the three-dimensional shape estimation unit 3 so that they correspond to each of the multiple objects being photographed. In this embodiment, the 3D models generated by the three-dimensional shape estimation unit 3 are point clouds, and the object separation unit 5 converts the point clouds into mesh models to generate 3D models corresponding to each of the objects. A point cloud is a collection of points and does not contain information indicating which 3D model each point corresponds to. On the other hand, a mesh model contains information indicating the combination of coordinates forming the polygons that are its components. If connected polygons are considered to be a single 3D model, the coordinates contained in the polygons can be used to identify which 3D model each corresponds to. In other words, by converting the point cloud into a mesh model, multiple 3D models comprising multiple objects can be generated. Note that it is unknown which photographic projects each of the multiple 3D models corresponds to, so this identification is performed by the project identification unit 6. Note that the marching cubes method is used to convert the point cloud into a mesh model. The generated multiple 3D models are each assigned an identifier and output to the project identification unit 6.

[0035] The process of separating 3D models for each subject is not limited to the above. The point cloud may be separated into multiple sets without being converted into a mesh model by the object separation unit 5. In this case, the multiple separated sets are treated as a single 3D model. In the separation process, for each point constituting the point cloud, adjacent points or points existing within a predetermined range from the point being processed are treated as points constituting the point cloud of the same subject. In other words, the 3D model generated by the three-dimensional shape estimation unit 3 is separated into 3D models for each subject by grouping components with close three-dimensional positions. The object separation unit 5 may be included in the three-dimensional shape estimation unit 3 rather than in the association unit 4. The object separation unit 5 is not an essential component, and a configuration without the object separation unit 5 will be described in Example 3.

[0036] The case identification unit 6 identifies which photography case the 3D models of each of the multiple subjects correspond to, and generates information indicating the combination. The specific process of generating information indicating the combination will be described later. The information indicating the combination is then output to the input / output unit 7, and the administrator confirms the combination of the photography case and the 3D model via the input / output unit 7, and generates a file for each photography case. The file for each photography case includes a 3D model of the subject corresponding to each photography case. The generated file is called a photography case file, and will be specifically explained in Figure 10. The case identification unit 6 generates multiple photography case files and outputs them to the recording unit 8.

[0037] The input / output unit 7 is responsible for managing information about photography projects and generating information (hereinafter referred to as "project identification information") used in the process of identifying which photography projects a 3D model corresponds to. A user requesting a photography project inputs information such as the shooting time, the number of people to be photographed, and details of the tools to be brought via the user terminal 10. The input / output unit 7 acquires this information and determines multiple photography projects that can be photographed simultaneously, taking into account the number of people who can be photographed simultaneously in the photography area, and manages the photography projects. For example, if the number of people who can be photographed simultaneously in the photography area is 10, the photography projects are managed so that 10 or fewer subjects are photographed at the target photography time. For example, photography projects can be managed on a first-come, first-served basis so that 10 or fewer people are photographed. Then, after the multiple photography projects to be photographed at the target photography time are determined, project identification information is generated according to the number of photography projects. In this embodiment, the photography area is divided into photography projects, and information indicating a combination of photography projects and area information is generated as project identification information. This enables the project identification unit 6 to generate information indicating a combination of a 3D model and photography project. For example, if there are three photographic subjects to be photographed simultaneously, the photographic area is divided into three, and a table is created showing the combination of the position information of each area and the photographic subjects. Note that this is not limited to the above, and for example, information showing the combination of the photographic subjects and the position tracking device may be generated. This example will be specifically explained in Example 2. The input / output unit 7 outputs information used in the process of identifying which photographic subject the generated 3D model corresponds to to the subject identification unit 6. Note that although an example of automatically dividing the photographic area has been described above, this is not limiting. The photographic area may also be set by an administrator who manages the photography.

[0038] Furthermore, the input / output unit 7 performs a process of associating the generated 3D model with the photography subject through an operation by the administrator during or after photography. At this time, the input / output unit 7 acquires information indicating the combination of the 3D model and the photography subject from the subject identification unit 6 and displays it in a manner that is recognizable to the administrator, thereby assisting the administrator in the process of associating the 3D model with the photography subject.

[0039] The recording unit 8 records the photography case file generated by the case identification unit 6. The photography case file is a file that compiles 3D models corresponding to each photography case. When a 3D model of a specific photography case is requested by the management server 9, the recording unit 8 outputs the data included in the target photography case file to the management server 9.

[0040] The management server 9 requests the data included in the photography case file of the target photography case from the recording unit 8 based on the information indicating the photography case acquired from the user terminal 10. Then, the management server 9 generates a virtual viewpoint image using the acquired data and outputs it to the user terminal 10. Note that information on the position and attitude of the virtual camera used to generate the virtual viewpoint image is acquired from the user terminal 10.

[0041] The user terminal 10 is, for example, a smartphone or tablet terminal. It outputs information indicating a specific photography case to the management server 9, acquires a virtual viewpoint image generated using a 3D model of a subject corresponding to the specific photography case, and displays it on the display unit shown in the accompanying drawings. For example, when photography is completed, the administrator provides the user with a card bearing a QR code associated with an identifier of the photography case. The user reads the QR code using a camera installed in the user terminal 10 to output the identifier of the photography case to the management server 9. The management server 9 uses the identifier of the photography case to acquire data of the target photography case file from the recording unit 8 and generates a virtual viewpoint image. The user terminal 10 acquires and displays the virtual viewpoint image. The acquired virtual viewpoint image may be displayed superimposed on another captured image. This can be displayed as a so-called AR image. In this case, the captured image acquired by the camera installed in the user terminal and the virtual viewpoint image are superimposed and combined. When superimposing, the position of the 3D model of the subject included in the virtual viewpoint image may be determined based on the position of the QR code included in the captured image. Furthermore, a plane such as a table or chair included in the captured image may be recognized, and the 3D model of the subject may be positioned on that plane. Furthermore, it is not necessary to use a card with a QR code printed on it, but a marker associated with an identifier of the photography subject may be printed on the card, and the identifier of the photography subject may be output to the management server 9 using image recognition. Furthermore, the medium is not limited to a card, and may be an acrylic stand, a figure, or the like.

[0042] (Hardware configuration) FIG. 2 is a diagram illustrating an example of the hardware configuration of an information processing device.

[0043] The hardware configuration of the information processing device 200 will be described with reference to Fig. 2. The information processing device 200 includes a CPU 211, a ROM 212, a RAM 213, an auxiliary storage device 214, a display unit 215, an operation unit 216, a communication I / F 217, and a bus 218.

[0044] The CPU 211 controls the entire information processing device 200 using computer programs and data stored in the ROM 212 and RAM 213, thereby realizing each function of the information processing device 200 shown in FIG. 1 . Note that the information processing device 200 may have one or more dedicated hardware components different from the CPU 211, and at least a portion of the processing by the CPU 211 may be executed by the dedicated hardware components. Examples of the dedicated hardware components include an ASIC (application-specific integrated circuit), an FPGA (field-programmable gate array), and a DSP (digital signal processor). The ROM 212 stores programs that do not require modification. The RAM 213 temporarily stores programs and data supplied from the auxiliary storage device 214, as well as data supplied from the outside via the communication I / F 217. The auxiliary storage device 214 is configured, for example, by a hard disk drive or the like, and stores various data such as image data and audio data.

[0045] The display unit 215 is configured with, for example, a liquid crystal display, an LED, or the like, and displays a GUI (Graphical User Interface) or the like for the user to operate the information processing device 200. The operation unit 216 is configured with, for example, a keyboard, a mouse, a joystick, a touch panel, or the like, and receives operations by the user to input various instructions to the CPU 211. The CPU 211 operates as a display control unit that controls the display unit 215 and an operation control unit that controls the operation unit 216. The communication I / F 217 is used for communication with an external device of the information processing device 200. For example, when the information processing device 200 is connected to an external device via a wired connection, a communication cable is connected to the communication I / F 217. When the information processing device 200 has a function of wirelessly communicating with an external device, the communication I / F 217 is equipped with an antenna. The bus 218 connects each unit of the information processing device 200 to transmit information.

[0046] In this embodiment, the display unit 215 and the operation unit 216 are assumed to exist inside the information processing device 200, but at least one of the display unit 215 and the operation unit 216 may exist outside the information processing device 200 as a separate device.

[0047] (Generation process of case identification information) The case identification information is generated by the input / output unit 7. After a plurality of photographing cases to be photographed at the target photographing time are determined, the photographing area is divided into each photographing case, thereby generating information indicating a combination of photographing cases and area information.

[0048] FIG. 3 is a diagram illustrating an example of case identification information according to the first embodiment. The case identification information includes a case identifier, which is an identifier for a photographed case, and area information indicating the coordinates of the four corners of a partial area generated by dividing the photographed area. Since no restrictions are placed on the height direction of the photographed area, the coordinates of the partial area are expressed using the X-axis and Y-axis coordinates in a virtual space represented by three axes: X, Y, and Z. The name of the photographed case may be included instead of the case identifier. This case identification information is assumed to be created in advance before photographing. Using this case identification information, it is possible to identify which photographed case the generated 3D model corresponds to. While FIG. 3 illustrates case identification information for four photographed cases in which the photographed area is equally divided into four, this is not limiting. For example, if the number of photographers for a photographed case with case identifier A is four and the number of photographers for each of the photographed cases with case identifiers B to D is one, the partial area corresponding to half of the photographed area may be used as the area information for case identifier A. In this case, the remaining half of the partial area is divided into three equal parts, and these parts are set as area information corresponding to the photographed cases with case identifiers B to D, respectively. If there are many subjects to be photographed for one photography subject, multiple partial areas may correspond to one photography subject. It is sufficient to be able to identify which photography subject corresponds to which partial area. Furthermore, the photography subject is not limited to being automatically set by the input / output unit 7, but may be arbitrarily set by an administrator who manages photography.

[0049] (Identifying the 3D model and the corresponding photography project) FIG. 4 is a diagram illustrating a process for identifying a photographing subject corresponding to a 3D model of a subject according to the first embodiment. Imaging devices 401a, 401b, 401c, and 401d are installed so as to surround a photographing area 402. In this embodiment, four photographing subjects are simultaneously photographed within the photographing area 402, and 3D models of the subjects corresponding to the four photographing subjects are simultaneously created. At this time, it is assumed that the photographing subjects corresponding to each 3D model are identified using position information of the 3D model corresponding to each subject, and the photographing area 402 is divided into four partial areas for each photographing subject. In other words, one photographing subject corresponds to one partial area, and the subject performs within the specified partial area. Partial area 403a includes subject 404a. Partial area 403b includes subjects 404b and 404c. Partial area 403c includes subject 404d. Partial area 403d includes subject 404e.

[0050] 5 is a flowchart illustrating the process of the case identifying unit 6 according to the embodiment 1. It is assumed that this process is executed for each frame.

[0051] In S501, the case identification unit 6 acquires multiple 3D models corresponding to multiple subjects from the subject separation unit 5. At this time, position information of the multiple 3D models is acquired. The position information of the 3D models is, for example, the position of the center of gravity of the 3D models. In this embodiment, it is assumed that the case identification information has been acquired in advance from the input / output unit 7, but if it has not been acquired, it is acquired in this step.

[0052] In S502, the case identification unit 6 selects a 3D model that is not linked to the case identifier of the photographing case. Note that the process may be performed in the order of the identifiers of the 3D models assigned by the subject separation unit 5, thereby selecting a 3D model that is not linked to the case identifier.

[0053] In S503, the case identification unit 6 identifies the partial area to which the selected 3D model belongs among the multiple partial areas. Specifically, the case identification unit 6 identifies the partial area that includes the center of gravity of the selected 3D model. In other words, the case identification unit 6 identifies the corresponding partial area using the position information of the 3D model.

[0054] In S504, the case identification unit 6 associates the case identifier corresponding to the identified partial area with the 3D model. Since the case identification information includes the case identifier corresponding to the partial area, this processing is performed based on the case identification information. This makes it possible to clarify which photographing case each 3D model corresponds to.

[0055] In S505, the case identification unit 6 outputs information indicating the combination of the 3D model and the photographing case to the recording unit 8 based on the information linked in S504.

[0056] In S506, the case identification unit 6 determines whether or not the process of linking the photographing case identifier to all 3D models has been executed. If the process of linking the photographing case identifier to all 3D models has been executed, this process ends, and if there is a 3D model for which the process of linking the photographing case identifier has not been executed, the process proceeds to S502.

[0057] By the above process, it is possible to identify the corresponding photography subject for all generated 3D models.

[0058] FIG. 6 is a diagram showing information indicating a combination of a 3D model and a photography subject generated by the subject identification unit 6 according to the first embodiment. Based on this information, the subject identification unit 6 can manage an appropriate 3D model for each photography subject. Note that the table of 3D model identifiers and subject identifiers shown in FIG. 6 is an example and does not necessarily have to be in a tabular format. It is sufficient to know the correspondence between the 3D model and the photography subject. For example, when managing a 3D model for each subject in the recording unit 8, the metadata of each 3D model may include the subject identifier or the name of the corresponding photography subject. Of course, the metadata of the 3D model may include both the subject identifier and the name of the photography subject. The existence of information indicating this correspondence makes it possible to generate a photography subject file including an appropriate 3D model for each photography subject.

[0059] If a subject performs near the boundary of a partial area, there is a risk that the 3D model of the subject may be erroneously associated with a different filming project. Therefore, the administrator may check whether there are any errors in the information indicating the combination of the 3D model and the filming project generated by the project identification unit 6. The administrator can check whether there are any errors in the information indicating the combination of the 3D model and the filming project via the input / output unit 7. When filming one filming project, it is conceivable to film the same performance multiple times, and use the filming that produces the desired performance as the deliverable. Because the subject may adjust their performance position during these multiple filmings and perform away from the boundary of the partial area, the administrator's process of checking and correcting the information indicating the combination of the 3D model and the filming project is not a required configuration.

[0060] FIG. 7 is a diagram illustrating a confirmation screen for information indicating a combination of a 3D model and a photography subject, which is displayed on the input / output unit 7 according to the first embodiment. The administrator uses this screen to check whether there are any errors in the combination of the 3D model and the photography subject, and corrects any errors. A screen area 701 displays images indicating the photography area 402, partial areas 403a to 403d, and 3D models 404a to 404e of the subjects. The screen also includes an area 704 indicating the selected photography subject, buttons 705 and 706 for selecting the photography subject, and a button 707 for confirming the combination when confirmation and correction are complete. 3D models associated with the selected photography subject are displayed in an identifiable manner. For example, when photography subject B is selected, information indicating that photography subject B has been selected is displayed in the area 704, and 3D models 404b and 404c associated with photography subject B are highlighted. On the other hand, 3D models not associated with photography subject B are not highlighted. The highlighting is not limited to shading; color information may be changed, or the size of the 3D model may be increased. If the combination information is incorrect, the administrator corrects the combination information by moving the cursor 704 and clicking the 3D model. Clicking a 3D model that is not highlighted causes it to become highlighted. Conversely, clicking a highlighted 3D model causes it to become unhighlighted. The administrator can correct the combination information by highlighting the appropriate 3D model for the selected photography project. If the input / output unit 7 is a device equipped with a touch panel, the administrator may use a touch operation instead of a click operation. After confirming and correcting the information on the combination of the 3D model and the photography project, the administrator clicks button 707 to end the confirmation and correction process. The screen area 702 displays information indicating the combination of the photography project and the 3D model. It is expected that the information displayed here will be updated sequentially according to adjustments made by the administrator. When confirmation and adjustment begins, information indicating the combination of the 3D model and the photography project generated by the project identification unit 6 is used as initial information. The screen area 703 displays project management information generated by the input / output unit 7.The information displayed in the screen area 702 and the screen area 703 may be displayed in a tabular format. The confirmation screen may be generated by the input / output unit 7 or the case identification unit 6.

[0061] FIG. 8 is a diagram illustrating a photography case file containing data for each photography case generated by the case identification unit 6 according to the first embodiment. This file is output to the recording unit 8, and therefore can be referred to as a file recorded in the recording unit 8. This photography case file is generated for each photography case and contains data associated with the photography case. Specifically, the photography case file contains the name of the photography case, the case identifier for the photography case, the name of the subject of the 3D model associated with the photography case, the 3D model of the subject, and audio data of the subject. Note that the photography case file need not necessarily contain other information as long as it contains the case identifier for the photography case and the 3D model of the subject. Note that if multiple subjects correspond to one photography case, data is recorded distinguishing between the subjects. In the example shown in FIG. 8, data for subject A and subject B are recorded. Specifically, the name of subject A, the 3D model of subject A, audio data for subject A, the name of subject B, the 3D model of subject B, and audio data for subject B are recorded. Furthermore, if a 3D model of the background to be used for the photography case is predetermined, the 3D model of the background may also be included. The name of the subject may be included in the metadata of the 3D model of the subject and the metadata of the audio data of the subject. In this embodiment, 3D models for a predetermined time period are generated, and therefore, the 3D models for that time period are collectively included in a file. Note that, since 3D models are generated for each shooting time, multiple 3D models may be managed by arranging them in the order of shooting time. FIG. 8 shows, as an example, a file storing 3D models of subject A arranged in the order of shooting time A to shooting time N. Furthermore, instead of the shooting time, the 3D models may be recorded in association with a time code expressed as "00 (hour): 00 (minute): 00 (second): 000 (frame)", which is the time when shooting by the imaging device begins.

[0062] By creating and recording a photography case file containing appropriate data for each photography case as described above, the management server 9 can acquire the necessary data for each photography case and generate a virtual viewpoint image.

[0063] The above embodiment makes it easy to simultaneously photograph multiple photography projects and appropriately manage 3D models for each photography project. Conventionally, an administrator had to check and associate each 3D model with each photography project. The above embodiment makes it easy to associate them.

[0064] If the data to be used in combination with the 3D model of the subject is predetermined for each shooting project, that data or information indicating that data may be stored in the metadata of the 3D model of the subject, or the 3D model of the subject and that data may be associated and managed. Possible data to be used in combination with the 3D model of the subject include a 3D model of the background, background material images, and 3D models of tools. This allows deliverables to be managed together for each shooting project.

[0065] (How to use the shooting case file) For example, the generated photography project file is used to generate AR images. The photography project identifier contained in the photography project file is converted into a QR code and printed on a medium such as a card or acrylic stand. By reading the card with the QR code using a user device, the user can display an AR image generated using a 3D model of the subject corresponding to the photography project.

[0066] FIG. 9 is a sequence diagram illustrating a process flow for displaying an AR image of a 3D model of a subject corresponding to a photography case according to the first embodiment.

[0067] In S901, the user terminal 10 photographs a QR code. Note that the QR code does not have to be a QR code, and may be, for example, a specific marker or barcode. The QR code indicates the case identifier of the photographed case, and the user terminal 10 acquires the case identifier of the photographed case by photographing it.

[0068] In S902, the user terminal 10 transmits the case identifier of the photographing case that has been read to the management server 9. Note that the information read by photographing the QR code is not limited to the case identifier of the photographing case, and may be, for example, the address at which the photographing case file corresponding to the photographing case is recorded in the recording unit 8, or the name of the photographing case.

[0069] In S903, the management server 9 requests the recording unit 8 for data included in the photography case file corresponding to the case identifier of the acquired photography case.

[0070] In S904, the recording unit 8 identifies the photography case file corresponding to the request.

[0071] In S905, the recording unit 8 outputs the data included in the identified photography case file to the management server 9.

[0072] In S906, the management server 9 completes acquisition of the data included in the photography case file and notifies the user terminal 10 that the data has been acquired.

[0073] In S907, the user terminal 10 measures the position and orientation of the user terminal 10. These are measured using a gyro sensor, a magnetic sensor, LiDAR, image recognition using a QR code included in a captured image, or the like, that are installed in the user terminal 10. Note that S907 to S912 are looped until the user inputs an end instruction. When the user inputs an end instruction, the process proceeds to S913, where the loop processing ends.

[0074] In S908, the user terminal 10 outputs information indicating the measured position and posture to the management server 9.

[0075] In S909, the management server 9 generates a virtual viewpoint image using data included in the photography case file acquired from the recording unit 8 and information indicating the position and attitude of the user terminal acquired from the user terminal 10. The position and attitude of the virtual camera in the virtual space is determined using the information indicating the position and attitude of the user terminal, and a virtual viewpoint image is generated using the position and attitude of the virtual camera and a 3D model of the subject included in the photography case file. Note that the virtual viewpoint image may be generated using only the data included in the photography case file, or may be generated by combining data separately recorded in the management server 9 and data included in the photography case file. If the data included in the photography case file is only a 3D model of the subject, a virtual viewpoint image including only the 3D model of the subject is generated.

[0076] In S910, the management server 9 outputs the generated virtual viewpoint image to the user terminal 10.

[0077] In S911, the user terminal 10 generates a composite image by combining a captured image acquired from a camera mounted on the user terminal 10 with a virtual viewpoint image acquired from the management server 9. Specifically, the composite image is generated by superimposing and combining the captured image and the virtual viewpoint image. When superimposing the virtual viewpoint image, the position of the QR code included in the captured image may be used as a reference, and the superimposition may be performed so that a 3D model of the subject is displayed on top of the QR code.

[0078] In S912, the user terminal 10 displays the generated composite image.

[0079] In S913, the user terminal 10 acquires an instruction to end the AR display through a user operation.

[0080] In S914, the user terminal 10 outputs an end instruction to the management server 9. This process ends the loop process.

[0081] By the above process, the user can view the AR image generated using the 3D model corresponding to the specific photography case. Note that the user is not limited to the client who requests the photography case, but may be a photographer who takes a photo of the card with the QR code, different from the client.

[0082] FIG. 10 is a diagram illustrating an example of an AR image displayed on the user terminal 10 according to the first embodiment.

[0083] 10(a) is a diagram showing a card with a QR code being photographed by a camera mounted on the user terminal 10. The photographed image includes a card 1001 and a QR code 1002.

[0084] FIG. 10(b) shows an AR image displayed by reading a QR code. In this embodiment, a composite image generated by superimposing a captured image and a virtual viewpoint image is referred to as an AR image. The AR image is generated by superimposing a captured image and a virtual viewpoint image so that a 3D model 1003 of the subject is displayed on the QR code included in the captured image. The user changes the position and orientation of the virtual camera by changing the position and orientation of the user terminal 10. If the subject performs a 30-second performance during the shoot, a 3D model for 30 seconds is generated. Therefore, a 3D model moving for 30 seconds can also be displayed in the AR image. After the 30 seconds of movement are completed, the 3D model performing the performance again from the beginning is displayed, and the display continues in a loop until the user issues a command to end the performance. The user can pause the movement of the 3D model by touching the screen. Alternatively, a 3D model from 5 seconds before the shooting time corresponding to the 3D model displayed at the time of operation can be displayed by swiping to the left of the screen. By swiping to the right of the screen, a 3D model 5 seconds after the shooting time corresponding to the 3D model displayed at the time of the operation may be displayed.

[0085] In the above embodiment, a case where a photography project file is utilized for AR has been described. However, the present invention is not limited to this, and a 3D model may be delivered to the user terminal 10.

[0086] <Example 2> In Example 1, the corresponding photography project was identified based on which partial area multiple 3D models corresponding to multiple subjects belong to. However, because the subjects must perform within the partial area, it may be difficult to perform within the partial area when filming a subject that moves significantly, such as a skateboard. Therefore, in this example, a tracker for acquiring position information is distributed to each subject, and the subjects are photographed while wearing the tracker. By linking the tracker identifier with the photography project in advance, it is possible to identify which photography project the generated 3D model corresponds to.

[0087] In this embodiment, a device capable of acquiring and tracking location information is referred to as a tracker. Specifically, a GPS tracker is utilized. However, this is not limiting, and location information may be acquired and tracked by capturing images of RFID tags, barcodes, etc. using multiple imaging devices and utilizing image recognition. Alternatively, a UWB sensor may be utilized.

[0088] 11 is a diagram illustrating an example of an overall system according to Example 2. In addition to the overall system illustrated in FIG.

[0089] The tracking information acquisition unit 11 acquires the location information of each tracker distributed to each subject. When a GPS tracker is used as the tracker, the location information of each tracker is acquired from an external server (not shown). The tracking information acquisition unit 11 is not an essential component, and the input / output unit 7 may acquire the location information of each tracker. Furthermore, for example, the multiple image capture units 1 may each have an infrared sensor or the like, so that the multiple image capture units 1 acquire the location information of each tracker.

[0090] FIG. 12 is a diagram illustrating an example of case identification information according to Example 2. The case identification information according to Example 2 is configured as a table including case identifiers, which are identifiers of photographed cases, and tracker identifiers, which are identifiers of trackers. In other words, the case identification information is information indicating which photographed case each tracker corresponds to. This case identification information is generated by the input / output unit 7 before photographing. In the example shown in FIG. 12, the case identifier A of the photographed case corresponds to the tracker identifier 1. The case identifier B corresponds to the tracker identifier 2 and the tracker identifier 3. The case identifier C corresponds to the tracker identifier 4. The case identifier D corresponds to the tracker identifier 5.

[0091] FIG. 13 is a diagram illustrating an imaging system according to a second embodiment. In this embodiment, a tracker is distributed to each subject, and the subjects perform while wearing the tracker. In the example shown in FIG. 13, subject 404a performs while wearing tracker 1301a. Subject 404b performs while wearing tracker 1301b. Subject 404c performs while wearing tracker 1301c. Subject 404d performs while wearing tracker 1301d. Subject 404e performs while wearing tracker 1301e.

[0092] In skateboarding, depending on the trick, the skater and the board may become physically separated. In this way, depending on the subject of the shoot, the person and the equipment may become physically separated. Therefore, when performing with such equipment, a tracker is attached to the equipment. By doing this, even when 3D models are generated when the person and equipment are physically separated, it is possible to identify what each 3D model corresponds to and appropriately link them to the shooting project.

[0093] 14 is a flowchart illustrating the processing of the case identifying unit 6 according to the embodiment 2. Note that the description of the same processing steps as those in the flow illustrated in FIG. 5 will be omitted.

[0094] In S1401, the case identification unit 6 acquires a plurality of 3D models corresponding to a plurality of subjects from the subject separation unit 5 and the position information of each tracker from the tracking information acquisition unit 11. If the case identification information has not been acquired, the case identification information is acquired from the input / output unit 7 in this step.

[0095] In S1402, the case identification unit 6 identifies the tracker from among the multiple trackers that is closest to the selected 3D model. At this time, the identification is performed using the position information of the 3D model acquired in S1401 and the position information of each tracker. Specifically, the case identification unit 6 identifies the tracker that is closest to the center of gravity of the 3D model.

[0096] In S1403, the case identification unit 6 associates the case identifier of the photography case corresponding to the identified tracker with the 3D model.

[0097] When photographing a tool with a tracker attached, if the person and tool are physically separated, a 3D model of the person and a 3D model of the tool are generated separately, and different trackers are identified for each 3D model. On the other hand, when the person and tool are physically in contact and a single 3D model including the person and tool is generated, two trackers are identified for the single 3D model. In photography projects where multiple subjects are photographed, there is a demand for generating virtual viewpoint images that include only 3D models of specific subjects. Therefore, when two trackers are identified for a single 3D model, it is necessary to determine which tracker corresponds to which subject. Therefore, when attaching trackers to tools during photography, the input / output unit 7 adds information indicating the type of subject corresponding to each tracker identifier to the photography project information shown in FIG. 12. For example, information indicating the type of subject can be person or tool. When multiple trackers are associated with a single 3D model, trackers whose subject type corresponding to the tracker identifier is set to person are prioritized for association with the 3D model. This allows for the generation of a virtual viewpoint image that includes only a 3D model of a specific subject among multiple subjects who is performing using props, regardless of whether the person and the prop are in contact with each other or not.

[0098] With the above process, the subject is not limited to a specific area in which to perform their performance, making it possible to accommodate a wider variety of photography projects.

[0099] Example 3 In the first embodiment, the object separation unit 5 separates the 3D model generated by the three-dimensional shape estimation unit 3 into multiple 3D models corresponding to multiple objects. If the data provided to the user terminal 10 is a virtual viewpoint image generated by the management server 9, the above process is not essential. In this embodiment, an example will be described in which the object separation unit 5 does not exist.

[0100] In this embodiment, an example will be described in which the 3D model generated by the three-dimensional shape estimation unit 3 is a point cloud. If the 3D model is a point cloud, without the object separation unit 5, it cannot be separated into point clouds corresponding to the objects, and therefore the case identifier of the 3D model cannot be identified. Therefore, the case identification unit 6 cannot generate information indicating the combination of the 3D model and the photographed case. Therefore, unlike in the first embodiment, the case identification unit 6 in this embodiment does not generate a photographed case file, but instead generates a file including multiple 3D models and case identification information and outputs it to the recording unit 8. Since this file is generated for each photographed time, it will hereinafter be referred to as a photographed time file.

[0101] FIG. 15 is a diagram showing a shooting time file including data for each shooting time according to the third embodiment. This file is generated by the case identification unit 6. The shooting time file includes the shooting time, a foreground 3D model including multiple subjects, case identification information, and a background 3D model. The background 3D model is not essential. It may also include audio data.

[0102] 16 is a sequence diagram illustrating a process flow for displaying an AR image of a 3D model of a subject corresponding to a photography case according to Example 3. Note that a description of the same processes as those in FIG. 9 will be omitted.

[0103] In S1601, the user terminal 10 captures a QR code. The QR code indicates the case identifier of the photography case and the photography time when the photography case was photographed, and these are acquired by the photography. Note that if a 30-second photography session is performed on the photography case, the QR code indicates the photography time for 30 seconds.

[0104] In S1602, the user terminal 10 outputs the read case identifier and the photographing time to the management server 9.

[0105] In S1603, the management server 9 requests the recording unit 8 for the data corresponding to the acquired shooting time.

[0106] In S1604, the recording unit 8 identifies the corresponding shooting time file based on the acquired request.

[0107] In S1605, the recording unit 8 outputs the data of the identified shooting time file.

[0108] In S1606, the management server 9 uses the case identifier acquired in S1602 and the case identification information included in the shooting time file to perform a process of making transparent 3D models that do not correspond to the photographed case among the data included in the shooting time file at the photographed time to be processed. Specifically, among the foreground 3D models (3D models including multiple subjects) included in the shooting time file, 3D models that are not included in the partial area corresponding to the case identifier of the photographed case acquired in S1602 are identified. Then, the color of the identified 3D models is set to colorless and transparent. This process makes it possible to generate a virtual viewpoint image that includes only 3D models that correspond to the case identifier of the photographed case whose QR code the user photographed and read.

[0109] In S1607, the management server 9 generates a virtual viewpoint image using the data in the shooting time file at the shooting time of the processing target and the information indicating the position and orientation of the user terminal. In S1606, since the 3D models corresponding to the non-target shooting subjects are made transparent, the generated virtual viewpoint image does not include the 3D models corresponding to the non-target shooting subjects.

[0110] By using the above process, even if multiple shooting cases are taken simultaneously to generate 3D models containing multiple subjects, a virtual viewpoint image containing only the 3D model of the subject corresponding to a specific shooting case is generated. Note that if the shooting time file contains a background model, the background model will also be included in the virtual viewpoint image.

[0111] Although the above describes a case where the 3D model is a point cloud, this is not limiting. If the 3D model generated by the three-dimensional shape estimation unit 3 is a mesh model, multiple mesh models corresponding to multiple subjects can be separated depending on whether the polygons are connected. Therefore, the three-dimensional shape estimation unit 3 may assign a 3D model case identifier to each mesh model, allowing the case identification unit 6 to generate information indicating the combination of the 3D model and the photographed case. In this case, a photographed case file or a photographed time file may be generated.

[0112] In the above example, the case where the shooting time file contains the case identification information is described, but the present invention is not limited to this. For example, the case identification unit 6 may identify which partial area each component of the 3D model is included in, and add information indicating the combination of each component and the photographed case corresponding to the identified partial area to the metadata of the 3D model.

[0113] <Other Examples> In the above embodiment, multiple photography projects are photographed simultaneously, and recording the voices of multiple subjects simultaneously can result in inappropriate audio recording due to the voices of other subjects being mixed in. Therefore, it is assumed that audio is collected for each photography project after the photography for generating a 3D model of the subject is completed. In this case, audio is collected while viewing a virtual viewpoint image generated using a 3D model associated with each photography project. The audio data acquired by the audio collection is managed in association with the 3D model associated with each photography project. For example, a folder may be created for each photography project, and the 3D model and audio data corresponding to the photography project may be stored in that folder for management.

[0114] Although the above describes an example in which the image of the subject and the sound of the subject are captured separately, the present invention is not limited to this. For example, each subject may be provided with an earphone and a directional microphone, and the image of the subject and the sound of the subject may be captured simultaneously.

[0115] The present disclosure can also be realized by providing a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0116] The disclosure of the present embodiment includes the following configurations, methods, systems, and programs.

[0117] (Configuration 1) an acquisition means for acquiring a plurality of 3D models generated based on a plurality of captured images obtained by simultaneously capturing a plurality of subjects corresponding to each of a plurality of photography projects; an association means for associating one of the plurality of photographing cases with one of the plurality of 3D models; an output means for outputting information indicating the combination of the photography subject and the 3D model associated by the association means; An information processing system comprising:

[0118] (Configuration 2) The information processing system according to configuration 1, wherein the association means associates one of the plurality of photographing subjects with one of the plurality of 3D models based on position information of one of the plurality of 3D models.

[0119] (Configuration 3) The plurality of photography subjects correspond to different areas, The information processing system according to configuration 2, characterized in that a photography subject corresponding to an area among the plurality of photography subjects that includes one of the plurality of 3D models is associated with one of the plurality of 3D models.

[0120] (Configuration 4) The plurality of photographic subjects are each associated with a different tracking device; The information processing system according to configuration 2, characterized in that a photography subject among the plurality of photography subjects, which corresponds to the tracking device located closest to the position of one of the plurality of 3D models, is associated with one of the plurality of 3D models.

[0121] (Configuration 5) The information processing system according to any one of configurations 1 to 4, further comprising a generation unit that generates a plurality of files including 3D models of a subject corresponding to the photography project based on information indicating a combination of the photography project and a 3D model.

[0122] (Configuration 6) An identification means for identifying a file corresponding to a photography subject identified by a first user operation from among the plurality of files; a generating means for generating a virtual viewpoint image using a 3D model included in the identified file; 6. The information processing system according to configuration 5, comprising:

[0123] (Configuration 7) 7. The information processing system according to configuration 6, further comprising a display control means for controlling the display of a composite image generated based on a captured image acquired by a second user operation and the virtual viewpoint image.

[0124] (Configuration 8) The information processing system according to configuration 7, further comprising a modification unit that modifies information indicating a combination of the photography subject and the 3D model based on a third user operation.

[0125] (Configuration 9) the first user operation and the second user operation are performed by the same user; 9. The information processing system according to claim 8, wherein the first user operation and the third user operation are performed by different users.

[0126] (method) an acquisition step of acquiring a plurality of 3D models generated based on a plurality of captured images obtained by simultaneously capturing a plurality of subjects corresponding to each of a plurality of photography projects; a correspondence process for correlating one of the plurality of photographing cases with one of the plurality of 3D models; an output step of outputting information indicating the combination of the photographing subject and the 3D model associated by the association step; An information processing method comprising:

[0127] (program) A program for causing a computer to function as each means of the information processing system according to any one of configurations 1 to 9. [Explanation of symbols]

[0128] 4 Mapping section 5 Subject separation section 6 Project identification department 7 Input / output section 8 Recording section

Claims

1. an acquisition means for acquiring a plurality of 3D models generated based on a plurality of captured images obtained by simultaneously capturing a plurality of subjects corresponding to each of a plurality of photography projects; an association means for associating one of the plurality of photographing subjects with one of the plurality of 3D models; an output means for outputting information indicating the combination of the photographing subject and the 3D model associated by the association means; An information processing system comprising:

2. The information processing system according to claim 1, wherein the associating means associates one of the plurality of photography subjects with one of the plurality of 3D models based on position information of one of the plurality of 3D models.

3. The plurality of photography subjects correspond to different areas, The information processing system according to claim 2, wherein a photography subject corresponding to an area of ​​the plurality of photography subjects that includes one of the plurality of 3D models is associated with one of the plurality of 3D models.

4. The plurality of photographic subjects are each associated with a different tracking device; The information processing system according to claim 2, characterized in that a photography subject corresponding to the tracking device that is located closest to the position of one of the plurality of 3D models among the plurality of photography subjects is associated with one of the plurality of 3D models.

5. The information processing system according to claim 1, further comprising a generation means for generating a plurality of files including 3D models of a subject corresponding to the photography project based on information indicating a combination of the photography project and a 3D model.

6. An identification means for identifying a file corresponding to a photography subject identified by a first user operation from among the plurality of files; a generating means for generating a virtual viewpoint image using a 3D model included in the identified file; 6. The information processing system according to claim 5, further comprising:

7. 7. The information processing system according to claim 6, further comprising a display control means for controlling the display of a composite image generated based on a captured image acquired by a second user operation and the virtual viewpoint image.

8. The information processing system according to claim 7, further comprising a modification unit that modifies information indicating a combination of the photography subject and the 3D model based on a third user operation.

9. the first user operation and the second user operation are performed by the same user; 9. The information processing system according to claim 8, wherein the first user operation and the third user operation are performed by different users.

10. an acquisition step of acquiring a plurality of 3D models generated based on a plurality of captured images obtained by simultaneously capturing a plurality of subjects corresponding to each of a plurality of photography projects; a correspondence process of associating one of the plurality of photographing subjects with one of the plurality of 3D models; an output step of outputting information indicating the combination of the photographing subject and the 3D model associated by the association step; An information processing method comprising:

11. A program for causing a computer to function as each of the means of the information processing system according to any one of claims 1 to 9.

Images