Information processing device, information processing method, and program

Abstract

The aim is to provide users with virtual viewpoint images that offer higher added value depending on the number of figures included in the captured image. [Solution] The information processing device analyzes the captured image obtained by photographing a three-dimensional object to acquire information about the three-dimensional object. Then, based on the number of three-dimensional objects included in the captured image, it determines and generates the content of a virtual viewpoint image.

Description

Technical Field

[0001] The present disclosure relates to a technique for generating a virtual viewpoint image from a 3D figure.

Background Art

[0002] There is a technique of superimposing and displaying a virtual viewpoint image of a scene related to a 3D model figure (hereinafter referred to as "figure") on an image obtained by photographing the figure with a mobile terminal such as a smartphone or a tablet. This technique, which enables a person who does not exist in the real world to be displayed as if they were in the real world, is called AR (Augmented Reality), and is widely used in fields such as entertainment, education, and training. Patent Document 1 discloses a technique of extracting a two-dimensional marker from an image obtained by photographing a figure representing a decisive moment in sports, and generating a virtual viewpoint image when viewing the figure at a decisive moment from an arbitrary viewpoint. With the technique of Patent Document 1, the user can deepen their knowledge about the decisive moment represented by the figure by means of virtual viewpoint images representing how the figure looks from various viewpoints.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The technology described in Patent Document 1 generates a virtual viewpoint image of a figure as seen from the viewpoint of the camera by photographing the figure with a camera. Therefore, the virtual viewpoint image produced by this technology was relatively monotonous and depended on the user's viewpoint. This disclosure has been made in view of this matter and aims to provide the user with a virtual viewpoint image that has higher added value depending on the number of figures included in the photographed image. [Means for solving the problem]

[0005] The information processing device according to this disclosure is characterized by comprising: a first acquisition means for acquiring a captured image obtained by photographing a three-dimensional object and camera parameters indicating the position and orientation of the photographing device used for the photograph; a second acquisition means for acquiring virtual viewpoint images that are generated based on the shape data of the object and the camera parameters, and whose content differs based on the number of three-dimensional objects included in the captured image; and a display control means for controlling the display of the virtual viewpoint images. [Effects of the Invention]

[0006] According to this disclosure, it will be possible to provide users with diverse and attractive virtual viewpoint images, thereby increasing the value of utilizing 3D model figures. [Brief explanation of the drawing]

[0007] [Figure 1] A block diagram showing an example configuration of an information processing system 10 that generates virtual viewpoint images. [Figure 2] A diagram showing an example of a sensor system installation. [Figure 3] Examples of tables that a database stores and manages. [Figure 4] A diagram showing an example of the functional configuration of an image generation device according to Embodiment 1. [Figure 5] A diagram showing an example of the hardware configuration of an image generation device. [Figure 6]A flowchart showing the flow of the virtual viewpoint image generation process according to Embodiment 1. [Figure 7] (a) and (b) are diagrams showing examples of captured images including figures. [Figure 8] (a) and (b) are diagrams showing examples of tables for storing figure information. [Figure 9] (a) and (b) are figures showing examples of augmented reality images. [Figure 10] A diagram showing an example of the functional configuration of an image generation device according to Embodiment 2. [Figure 11] A flowchart showing the flow of the virtual viewpoint image generation process according to Embodiment 2. [Figure 12] A diagram showing an example of a photograph that includes a figure. [Figure 13] A diagram showing an example of a table for storing figure information. [Figure 14] A diagram showing an example of an augmented reality image. [Figure 15] A diagram showing an example of the functional configuration of an image generation device according to Embodiment 3. [Figure 16] A flowchart showing the flow of the virtual viewpoint image generation process according to Embodiment 3. [Figure 17] A diagram showing an example of a photograph of a figure. [Figure 18] A diagram showing an example of a table for storing figure information. [Figure 19] A diagram showing an example of an augmented reality image. [Modes for carrying out the invention]

[0008] Embodiments of the present invention will be described below with reference to the drawings. Note that the following embodiments are not intended to limit the scope of the claims of this disclosure, and not all combinations of features described in these embodiments are necessarily essential to the solutions of this disclosure. The same reference numerals are used for identical components, and their descriptions are omitted.

[0009] [Embodiment 1] This embodiment will describe a mode of generating virtual viewpoint images with different contents according to the number of 3D model figures reflected in a photographed image in AR technology. In this specification, the 3D model figure will be simply referred to as a "figure". Also, a virtual viewpoint image is an image representing the view from a non-existent virtual camera (virtual camera), and it may be a video or a still image.

[0010] <Configuration of the system> FIG. 1 is a block diagram showing a configuration example of an information processing system 10 that generates virtual viewpoint images according to this embodiment. The information processing system 10 includes a mobile terminal 11, an image generation device 12, a database 13, and a modeling device 14.

[0011] The mobile terminal 11 is an information processing device such as a smartphone or a tablet terminal having a camera function. The mobile terminal 11 superimposes and displays in real time the virtual viewpoint video generated by the image generation device 12 on the video of the figure being photographed.

[0012] The image generation device 12 is, for example, a server device. It receives a photographed image from the mobile terminal 11, acquires a 3D model corresponding to the figure reflected in the photographed image, and generates and provides a virtual viewpoint image. Here, the 3D model refers to, for example, shape data representing the three-dimensional shape of an object such as a person to be photographed as a set of points (point cloud) including its color information. The 3D model is generated, for example, based on a plurality of images (multi-viewpoint images) obtained by photographing an object from various angles with a plurality of photographing devices. The image generation device 12 generates various virtual viewpoint images according to the number of figures in the photographed image using the acquired 3D model, and transmits them to the mobile terminal 11 that is the transmission source of the photographed image.

[0013] Database 13 stores and manages data that serves as material for figures and virtual viewpoint images. Specifically, it stores and manages data such as multi-view images captured by the sensor system described later, 3D models of objects shown in those multi-view images, and time codes representing the time of day represented by those 3D models, for example, in a table like the one described later. The time code is specified in the format "hour:minute:second.frame number". In this case, the frame number corresponds to the frame rate used during shooting; for example, if shot at 60fps, it will take a value from "0" to "59".

[0014] The molding device 14 creates a figure 15, a three-dimensional object made of a material such as resin or plastic, using a 3D model stored and managed by the database 13, for example, by a known method such as 3D printing. The created figure 15 has a code that encodes information about the 3D model it is based on (object ID, time code, table ID) attached to the base or surface of the figure. In this embodiment, a two-dimensional marker is used as an example of the code. A two-dimensional marker refers to a code composed of patterns, dots, symbols, etc., placed on a plane. Note that the code is not limited to a two-dimensional marker; a one-dimensional marker such as a barcode may also be used. Furthermore, the information about the 3D model may include time information of a highlight scene in a specific event linked to the table ID of the table that stores the target 3D model. An example of a highlight scene is a walk-off home run in a baseball game, in which case the time information would be a time code indicating the start and end times that specify a few seconds before and after the walk-off home run.

[0015] <Generating and saving 3D models> Figure 2 shows an example of a sensor system setup for obtaining multi-view images during a baseball game. Sensor systems 50a-50m, each having at least one camera, are installed to surround the target area 51, which includes the baseball field and its surroundings. Multi-view images are obtained by having each sensor system 50a-50m capture the target area 51 in time-synchronized manner.

[0016] Once multi-view images are obtained as described above, the foreground region corresponding to the players, ball, etc., is extracted from each of the multiple captured images constituting the multi-view image by detecting the difference between it and, for example, an image taken beforehand without the players or ball (background image), thereby obtaining a foreground image. Then, based on the multiple foreground images obtained, point cloud data showing the three-dimensional shape of the players, ball, etc., is generated using a shape estimation method such as Visual Hull. By assigning the pixel values ​​of the captured images derived based on the position and orientation of each camera to each point constituting the generated point cloud, a point cloud with color information is obtained. The color of each point is determined by identifying the position on the camera coordinate system using camera parameters that indicate the position and orientation of the camera, based on the coordinates of the point in 3D, and adopting the color of that camera coordinate position. In this case, if the object is visible from multiple cameras, the color of any one camera may be used, or the colors of multiple cameras may be blended.

[0017] In this embodiment, a point cloud format with color information is used as the data format for the 3D model, but it is not limited to this. For example, the format representing the three-dimensional shape may be a voxel format or a mesh format, and it may not even have color information. In the case of a 3D model without color information, color can be added when rendering the image corresponding to the virtual viewpoint, for example, by using an image taken from a viewpoint close to the virtual viewpoint. Furthermore, although this embodiment describes a method of generating a 3D model using Visual Hull with multi-view images, the 3D model may also be generated by other methods, such as 3D scanning or computer graphics.

[0018] The number of sensor systems to be installed is not limited. Furthermore, the sensor systems do not need to be installed around the entire circumference of the area to be photographed; depending on location constraints, they may be installed only around a portion of the area. Also, the cameras in each of the multiple sensor systems may include cameras with different functions, such as telephoto and wide-angle cameras. In addition to cameras, each of the multiple sensor systems may also have a microphone (not shown). If each of the multiple sensor systems has a microphone, each microphone will synchronize to capture sound. Based on the captured sound, an audio signal can be generated that will be played back along with the display of the virtual viewpoint image. For the sake of simplicity, the description of audio will be omitted hereafter, but it will be assumed that images and audio are processed together.

[0019] In this embodiment, the database 13 stores the 3D model data generated as described above in a table structure, for example, as shown in Figure 3, for each captured image file. In the table shown in Figure 3, an object ID (001, 002, 003, ...) is assigned to each group of 3D models relating to the same object. The 3D model data (Data A100, Data B100, ...) corresponding to each time indicated by the time code is stored in association with the object ID. In addition, each table is assigned a table ID (tbl123) to uniquely identify it, and the database 13 will maintain a table for each captured image file. When 3D model data is stored using a table like the one shown in Figure 3, the 3D model for each object at any moment in a specific event can be read and retrieved by specifying the table ID, time code, and object ID. Note that the table in Figure 3 is just an example, and for example, the table item values ​​could include the centroid coordinates of each 3D model or the object name (for example, player name).

[0020] <Example of image generation device configuration> An example of the configuration of the image generation device 12 according to this embodiment will be explained with reference to the figures. Figure 4 is a diagram showing the software configuration (functional configuration) of the image generation device 12, and Figure 5 is a diagram showing the hardware configuration of the image generation device 12. The following explanation will be based on these figures.

[0021] ≪Example of software configuration for an image generation device≫ As shown in Figure 4, the image generation device 12 according to this embodiment includes a data receiving unit 101, an image analysis unit 102, a content determination unit 103, a virtual viewpoint setting unit 104, a 3D model acquisition unit 105, an image generation unit 106, and a data output unit 107.

[0022] The data receiving unit 101 receives data of the captured image of the figure from the mobile terminal 11. The data receiving unit 101 also receives data from the mobile terminal 11 such as sensor values ​​indicating changes in acceleration and attitude obtained from an acceleration sensor and gyroscope (not shown) of the mobile terminal 11, and the focal length of the built-in camera. The received data, such as the captured image, is input to the image analysis unit 102.

[0023] The image analysis unit 102 analyzes the input captured image and extracts and decodes the code (in this embodiment, a 2D marker) associated with each figure in the captured image to obtain information about the figures shown in the image. Here, the information about the figures includes an object ID that uniquely identifies the person or other entity represented by each figure, a time code representing the moment in time represented by each figure, and a table ID that uniquely identifies the table in which the 3D model of each figure is stored. In the following description, this information about the figures will be referred to as "figure information." The extraction of the 2D markers is performed, for example, by performing grayscale conversion, noise reduction, contrast adjustment, etc., on the captured image, and then detecting the area corresponding to the 2D marker in the captured image by pattern matching. The 2D markers thus obtained are represented by a pattern of black and white cells, and by reading this pattern, the figure information embedded in the 2D marker is obtained. The obtained figure information is input to the content determination unit 103. Furthermore, the image analysis unit 102 can determine the position and orientation (direction) of the mobile terminal 11 based on the code or image of the three-dimensional object in the captured image. In this embodiment, the image analysis unit 102 calculates the relative position and orientation of the built-in camera to the figure based on the positions of the four corners of the two-dimensional markers extracted from the captured image, the characteristic points of the figure being shown, etc.

[0024] The content determination unit 103 determines the display size of objects in the virtual viewpoint image according to the number of figures in the captured image and outputs it to the virtual viewpoint setting unit 104. The content determination unit 103 also determines the effect according to the number of figures in the captured image and outputs it to the image generation unit 106. The determination of the display size and effect according to the number of figures will be described later. In addition, the content determination unit 103 outputs the object ID, time code, and table ID to the 3D model acquisition unit 105 based on the figure information extracted from the captured image. Furthermore, the content determination unit 103 outputs camera parameters (shooting parameters) indicating shooting conditions such as the position and orientation of the built-in camera and the focal length, which are identified based on the sensor values ​​and analysis results received from the mobile terminal 11, to the virtual viewpoint setting unit 104.

[0025] The virtual viewpoint setting unit 104 sets virtual viewpoint information, such as the position and orientation of the virtual camera, based on the shooting parameters and display size information input from the content determination unit 103. Here, the virtual camera is a virtual camera that does not exist in reality, placed in a virtual space (CG space) corresponding to the real space where the shooting took place. The image representing how the object looks from this virtual camera becomes the virtual viewpoint image. First, the virtual viewpoint setting unit 104 sets the position and orientation of the mobile terminal 11 indicated by the input shooting parameters as the position and orientation of the virtual camera. Furthermore, while maintaining that orientation, it changes the position of the virtual camera according to the input display size. In this case, if the virtual camera is close to the 3D model of the figure, the object will be displayed larger (enlarged compared to the captured image) in the virtual viewpoint image. Conversely, if the virtual camera is far from the 3D model of the figure, the object will be displayed smaller (reduced compared to the captured image) in the virtual viewpoint image.

[0026] The 3D model acquisition unit 105 acquires a 3D model from the database 13, which is identified based on the object ID, table ID, and time code input from the content determination unit 103. The acquired 3D model data is input to the image generation unit 106.

[0027] The image generation unit 106 generates a virtual viewpoint image by performing rendering processing based on the 3D model input from the 3D model acquisition unit 105, the virtual viewpoint information input from the virtual viewpoint setting unit 104, and the performance effect information input from the content determination unit 103.

[0028] The data output unit 107 transmits the virtual viewpoint image data generated by the image generation unit 106 to the mobile terminal 11, which is the source of the captured image data received by the data receiving unit 101. However, the destination to which the data output unit 107 outputs the virtual viewpoint image is not limited to the mobile terminal 11.

[0029] (Hardware configuration of the image generation device) Next, the hardware configuration of the image generation device 12, which is an information processing device, will be described. Figure 5 shows an example of the hardware configuration of the image generation device 12.

[0030] The CPU 201 is an arithmetic processing unit that controls the overall operation of the image generation device 12, and by executing a predetermined program stored in the ROM 203, it realizes each of the functional units shown in Figure 4. The image generation device 12 may have one or more dedicated hardware components separate from the CPU 201, and at least a portion of the processing performed by the CPU 201 may be executed by the dedicated hardware. Examples of dedicated hardware include ASICs (Application-Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), and DSPs (Digital Signal Processors).

[0031] ROM203 holds programs and various data corresponding to each functional unit shown in Figure 4. RAM202 has a work area for temporarily storing programs and data read from ROM203. RAM202 also provides a work area used by CPU201 when executing each process.

[0032] The operation input unit 204 accepts user input via input devices such as a keyboard or mouse, or a touch panel. The display unit 205 is, for example, a liquid crystal display, and performs functions such as displaying the status of the image generation device 12 and displaying the generated virtual viewpoint image.

[0033] The communication interface unit 206 is an interface that controls communication between the database 13 and external devices such as the mobile terminal 11, for example, via a network such as a LAN. For example, it receives 3D models from the database 13 via Ethernet. It also receives captured image data and transmits virtual viewpoint image data to the mobile terminal 11 via short-range wireless communication such as Ethernet or Bluetooth®. In addition, it may transmit and receive various types of data via image output ports such as HDMI® or SDI.

[0034] <Operation Flow of Image Generation Device> Next, the flow of the virtual viewpoint image generation process in the image forming apparatus 12 according to this embodiment will be explained with reference to the flowchart in Figure 6. In the following explanation, the symbol "S" represents a step.

[0035] In S601, the data receiving unit 101 receives captured images of the figures from the mobile terminal 11. Figures 7(a) and 7(b) show examples of images obtained when a user photographs baseball player figures placed on a desk using the camera function of the mobile terminal 11. Figure 7(a) is an example of a captured image showing one figure f11 of a certain batter, and Figure 7(b) is an example of a captured image showing three figures f11, f12, and f13 of the same batter in the same pose. Two-dimensional markers m11 to m13, which store figure information, are attached to the base of each figure.

[0036] In S602, the image analysis unit 102 analyzes the captured image received in S601 and obtains figure information from each 2D marker in the captured image. Figure 8(a) is an example of a table that stores figure information obtained from the captured image in Figure 7(a) above, and Figure 8(b) is an example of a table that stores figure information obtained from the captured image in Figure 7(b) above. The table in Figure 8(a) stores the object ID, time code, and table ID information of the 3D model corresponding to one figure f11. The table in Figure 8(b) stores the object ID, time code, and table ID information of the 3D models corresponding to three figures f11 to f13.

[0037] In S603, the content determination unit 103 counts the number of figures in the input captured image based on the figure information acquired in S602. In the case of the captured image in Figure 7(a), the count value will be 1, and in the case of the captured image in Figure 7(b), the count value will be 3.

[0038] In S604, the content determination unit 103 determines the display size and visual effects of the 3D models (=objects) in the virtual viewpoint image generated in S607 (described later), based on the number of figures acquired in S603.

[0039] ≪Determining the display size≫ For example, the display size is determined using a threshold: "Large" if there are 3 or more figures, "Medium" if there are 2 figures, and "Small" if there is 1 figure. In the case of the image in Figure 7(a), the display size is determined to be "Small" because there is 1 figure. In the case of the image in Figure 7(b), the display size is determined to be "Large" because there are 3 figures. Note that the representation of the display size is not limited to large, medium, and small. For example, it could be represented by a number from 1 to 10, where the display size increases as the number increases.

[0040] ≪Decision on the effects≫ Special effects are visual effects applied to attract the viewer's attention, and are added by compositing virtual representations such as flames, sparks, smoke, lightning, and luminescence onto the foreground (the area corresponding to an object) or its surroundings in a virtual viewpoint image. For example, a threshold is used to determine whether special effects are "present" if the number of figures is 3 or more, and "absent" if it is less than 3. In the case of the captured image in Figure 7(a), the number of figures is 1, so special effects are determined to be "absent". In the case of the captured image in Figure 7(b), the number of figures is 3, so special effects are determined to be "present". Note that the determination of special effects is not limited to a binary choice of "present" or "absent". For example, the content or scale of the special effects may be determined so that they become more elaborate or larger as the number of figures increases.

[0041] In S605, the 3D model acquisition unit 105 acquires the corresponding 3D model based on the figure information acquired in S602. Specifically, it acquires the 3D model identified by the table ID, object ID, and time code indicated by the figure information from the database 13. In the case of the captured image in Figure 7(a), based on the single figure information indicated by the table in Figure 8(a), the 3D model identified by table ID=“tbl20200101”, time code=“18:30:02.001”, and object ID=“3” is acquired from the database 13. In the case of the captured image in Figure 7(b), based on the three figure information indicated by the table in Figure 8(b), the 3D model identified by table ID=“tbl20200101”, time code=“18:30:02.001”, and object ID=“3” is acquired from the database 13. In the case of the table in Figure 8(b), all three object IDs are the same, and all three time codes are also the same. In this case, since the 3D model to be acquired is the same, it is sufficient to acquire the 3D model only once.

[0042] In S606, the virtual viewpoint setting unit 104 sets virtual viewpoint information based on the shooting parameters of the mobile terminal 11 acquired in S602 and the display size determined in S606. Specifically, it places a virtual camera in the virtual three-dimensional space at the same position and orientation as indicated by the shooting parameters, and further adjusts the position of the virtual camera according to the determined display size.

[0043] In S607, the image generation unit 106 places the 3D model acquired in S605 into the virtual space and generates a virtual viewpoint image by rendering based on the virtual viewpoint information set in S606 and the visual effects determined in S604. At this time, when the generated virtual viewpoint image is displayed superimposed on the captured image on the mobile terminal 11, the 3D model is positioned such that, for example, the foreground of the virtual viewpoint image does not overlap with the figure in the captured image. Alternatively, the 3D model may be deliberately positioned to overlap with the figure.

[0044] In S608, the data output unit 107 transmits the virtual viewpoint image generated in S607 to the mobile terminal 11. Upon receiving the virtual viewpoint image, the mobile terminal 11 displays the image superimposed on the image captured by the built-in camera, thereby realizing an augmented reality image. Figure 9(a) shows the augmented reality image corresponding to the image captured in Figure 7(a), with a display size of "small" and no special effects. As mentioned above, in the case of Figure 7(a), where there is only one figure in the captured image, the foreground a11 (=virtual viewpoint image) in the augmented reality image is displayed at approximately the same size as the figure in the captured image, and without any special effects. Figure 9(b) shows the augmented reality image corresponding to the image captured in Figure 7(b), with a display size of "large" and special effects. As mentioned above, in the case of Figure 7(b), where there are three figures in the captured image, the foreground a12 (= virtual viewpoint image) in the augmented reality image is enlarged compared to the figures in the captured image and is displayed in a manner that includes special effects (in this case, sparks).

[0045] In S609, it is determined whether there are any unprocessed frames based on the timecode of the figure information acquired in S602. If there are any unprocessed frames, the process returns to S605 and the same process is repeated. The above is the flow of the virtual viewpoint image generation process in the image forming apparatus 12 according to this embodiment.

[0046] <Variation> In the embodiment described above, the display size of the objects and the visual effects when generating the virtual viewpoint image were set according to the number of figures in the captured image, but other parameters related to the virtual viewpoint image may also be set. Other parameters include, for example, the trajectory of the virtual viewpoint movement (virtual camera path), the resolution and frame rate of the virtual viewpoint image, and the playback time of the virtual viewpoint image. When setting a virtual camera path according to the number of figures, for example, as the number of figures increases, the movement distance and zoom magnification of the virtual camera may be changed (for a small number of figures, half a circle around the 3D model with a fixed magnification; for a large number of figures, a full circle around the model with gradual zoom in / out, etc.).

[0047] In the above embodiment, the position of the virtual camera was changed according to the determined display size, but this is not limited to that. For example, the quality of the 3D model itself that is the target of the rendering process may be varied according to the determined display size. For example, three types of point cloud data, "large," "medium," and "small," may be prepared and acquired according to the determined display size.

[0048] Furthermore, in the above-described embodiment, the image generation device 12, which is a server device, generated the virtual viewpoint image and provided it to the mobile terminal 11, which is a user terminal, but the invention is not limited to this. For example, both figure photography and virtual viewpoint image generation may be performed on a user terminal that also has the function of an image generation device. As described above, this embodiment makes it possible to generate virtual viewpoint images with different content depending on the number of figures in the captured image. This makes it possible to provide users with diverse and attractive augmented reality images.

[0049] [Embodiment 2] Next, Embodiment 2 will describe a method for generating a virtual viewpoint image that reproduces a continuous motion (same motion) when multiple figures in the captured image represent different moments in a sequence of movements (same motion) by the same object. Note that the system configuration and other aspects common to Embodiment 1 will be omitted, and the following explanation will focus on the differences.

[0050] <Example of software configuration for an image generation device> Figure 10 shows an example of the configuration of the image generation device 12 according to this embodiment. As shown in Figure 10, the image generation device 12 of this embodiment includes a data receiving unit 101, an image analysis unit 102, a content determination unit 103, a virtual viewpoint setting unit 104, a 3D model acquisition unit 105, an image generation unit 106, a data output unit 107, and a motion determination unit 108. The differences from Embodiment 1 will be described below.

[0051] First, the figure information in this embodiment includes a motion ID. The motion ID will now be explained. In this embodiment, motion refers to a series of actions related to the same object (in this embodiment, a baseball player), such as a batter's bat swing or a pitcher's throw in baseball. The motion ID is identification information for uniquely identifying the motion. This motion ID is added to the figure information by the user when creating the figure in the modeling apparatus 14, for example. The figure information including the motion ID is then encoded to generate a code such as a 2D marker, which is then attached to the figure's base or the like. The motion ID may also be automatically generated and added to the figure information based on the shape of the 3D model corresponding to the created figure.

[0052] The motion determination unit 108, when multiple figures are present in the captured image, uses the motion IDs stored in the figure information corresponding to the multiple figures to determine whether the 3D models corresponding to each figure are related to the same motion. The method for determining whether or not they are the same motion is not limited to using motion IDs; for example, it may be determined whether or not they are the same motion by a method such as pattern matching based on the shape of the 3D model corresponding to each figure.

[0053] If the motion determination unit 108 determines that the motions are the same, the content determination unit 103 identifies the time code indicating the earliest time and the time code indicating the latest time of the motion based on the information attached to the motion ID. Then, it determines the generation period of the virtual viewpoint image by setting the identified time code indicating the earliest time as the "start time code" and the time code indicating the latest time as the "end time code".

[0054] When the above-mentioned generation period is determined, the image generation unit 106 generates a virtual viewpoint image that reproduces the series of actions indicated by the motion ID, using a 3D model associated with the timecode between the time indicated by the start timecode and the time indicated by the end timecode for the said generation period.

[0055] The above summarizes the main differences between the image generation device 12 according to this embodiment and Embodiment 1 of each functional unit.

[0056] <Operation Flow of Image Generation Device> Next, the flow of the virtual viewpoint image generation process in the image forming apparatus 12 according to this embodiment will be explained, focusing on the differences from Embodiment 1, with reference to the flowchart in Figure 11. In the following explanation, the symbol "S" represents a step.

[0057] In step S1101, the data receiving unit 101 receives the captured image of the figure from the mobile terminal 11. Figure 12 shows an example of an image obtained when a user photographs a baseball player figure placed on a desk using the camera function of the mobile terminal 11. In the example in Figure 12, an example of a captured image shows three figures f21, f22, and f23 capturing the moment of the same batter's bat swing. Two-dimensional markers m21 to m23, which store figure information, are attached to the base of each figure.

[0058] In S1102, the image analysis unit 102 analyzes the captured image received in S1101 and obtains figure information from the 2D markers in the captured image. Figure 13 is an example of a table showing figure information obtained from the captured image in Figure 12. The table in Figure 13 stores the object ID, time code, table ID, and motion ID for each of the 3D models corresponding to the three figures f21 to f23. The motion ID (mtn_10010) also stores the time codes (18:30:02.001-18:30:04.020) of the start and end times of the motion as supplementary information.

[0059] In S1103, the content determination unit 103 counts the number of figures in the input captured image based on the figure information acquired in S1102. In the case of the captured image in Figure 12, the count value will be 3.

[0060] In S1104, the next process to be executed is determined by whether the count value obtained in S1103 is multiple. If the count value is multiple (2 or more), S1105 is executed next; if it is not multiple (less than 2), S1106 is executed next.

[0061] In S1105, the motion determination unit 108 determines whether the 3D models identified in each figure information are related to the same motion, based on the motion ID contained in each of the multiple figure information acquired in S1102. As shown in the table in Figure 13, in the case of the captured image in Figure 12, the motion IDs stored in each of the three figure information are all the same, so it is determined that they are the same motion.

[0062] In S1106, the content determination unit 103 determines the display size and effect of the objects in the virtual viewpoint image generated in S1109 (described later) based on the number of figures acquired in S1103. If it is determined in S1105 that the motions are the same, as in the captured image in Figure 12, the generation period of the virtual viewpoint image is determined based on the attached information of the motion ID included in the figure information.

[0063] In S1107, the 3D model acquisition unit 105 acquires the corresponding 3D model based on the figure information acquired in S1102. If it was determined in S1105 that the motions are the same, the 3D models corresponding to each time from the start timecode to the end timecode of the generation period determined in S1106 will be acquired sequentially. In this case, instead of acquiring 3D models for all frames from the start timecode to the end timecode, it is also possible to acquire 3D models only for frames that are thinned out at equal intervals, such as once every two frames. In this way, if there are multiple figure information sets determined to be the same motion, 3D models will be acquired to fill in the gaps between the timecodes indicated by each of the multiple figure information sets. If it was determined in S1105 that the motions are not the same, the process described in Embodiment 1 will be executed.

[0064] In S1108, similar to S606 mentioned above, the virtual viewpoint setting unit 104 sets virtual viewpoint information based on the shooting parameters of the mobile terminal 11 acquired in S1102 and the display size determined in S1106.

[0065] In S1109, similar to S607 described above, the image generation unit 106 places the 3D model acquired in S1107 into the virtual space and generates a virtual viewpoint image by rendering based on the virtual viewpoint information set in S1108 and the visual effects determined in S1106. At this time, when the generated virtual viewpoint image is displayed superimposed on the captured image on the mobile terminal 11, the 3D model is positioned such that, for example, the foreground of the virtual viewpoint image does not overlap with the figures in the captured image. Alternatively, the 3D model may be deliberately positioned to overlap with one of the figures.

[0066] In S1110, similar to S608 mentioned above, the data output unit 107 transmits the virtual viewpoint image generated in S1109 to the mobile terminal 11. Upon receiving the virtual viewpoint image, the mobile terminal 11 displays the image superimposed on the image captured by the built-in camera, thereby realizing an augmented reality image.

[0067] In S1111, similar to S609 mentioned above, it is determined whether there are any unprocessed frames based on the timecode of the figure information obtained in S1102. If there are unprocessed frames, the process returns to S1107 and is repeated. If it was determined in S1105 that the motion is the same, the process will be repeated up to the frame with the end timecode of the generation period determined in S1106.

[0068] The above describes the flow of the virtual viewpoint image generation process in the image forming apparatus 12 according to this embodiment. Figure 14 shows an augmented reality image corresponding to the captured image in Figure 12. In the augmented reality image shown in Figure 14, which is realized by this embodiment, the bat swing of the player with object ID=3, represented by three figures f21~f23, is displayed as a video in the foreground a21 (=virtual viewpoint image). In the example of Figure 14, the object is enlarged and an effect is added to emphasize the trajectory of the bat swing. In the example of Figure 14, the position and orientation of the virtual viewpoint are fixed during the determined generation period, but they may be changed according to the number of figures (for example, gradually approaching, moving from a viewpoint near the ground to an overhead viewpoint, etc.).

[0069] As described above, according to this embodiment, when multiple figures represent different moments in a series of actions, it is possible to generate a virtual viewpoint image that interpolates the time intervals between each moment. This allows the user to enjoy an augmented reality image that reproduces the series of actions represented by the figures.

[0070] [Embodiment 3] Next, Embodiment 3 will describe a method for generating a virtual viewpoint image that reproduces a specific scene when multiple figures in a captured image represent a specific scene composed of multiple objects. Note that the system configuration and other aspects common to Embodiment 1 will be omitted, and the following explanation will focus on the differences.

[0071] <Example of software configuration for an image generation device> Figure 15 shows an example of the configuration of the image generation device 12 according to this embodiment. As shown in Figure 15, the image generation device 12 of this embodiment includes a data receiving unit 101, an image analysis unit 102, a content determination unit 103, a virtual viewpoint setting unit 104, a 3D model acquisition unit 105, an image generation unit 106, a data output unit 107, and a scene determination unit 109. The differences from Embodiment 1 will be explained below.

[0072] First, the figure information in this embodiment includes a scene ID. The scene ID will now be explained. In this embodiment, the scene ID is identification information used to uniquely identify a highlight scene during a game, such as a showdown between the ace pitcher and the cleanup hitter in baseball. Similar to the motion ID in Embodiment 2, this scene ID is added to the figure information by the user when creating the figure in the modeling apparatus 14, for example. The figure information including the scene ID is then encoded to generate a code such as a 2D marker, which is then attached to the figure's base or the like. Note that, similar to the motion ID, the scene ID may also be automatically generated and added to the figure information based on the shape of the created 3D model of the figure.

[0073] The scene determination unit 109, when multiple figures are present in the captured image, uses the scene ID stored in the figure information corresponding to the multiple figures to determine whether the 3D models corresponding to each figure belong to the same scene. The method for determining whether or not they belong to the same scene is not limited to using the scene ID. For example, it may be determined whether or not they belong to the same scene by a method such as pattern matching based on the shape of the 3D model corresponding to each figure. Alternatively, for example, if the difference between the timecode of one 3D model and the timecode of the other 3D model is below a threshold and they are temporally close, it may be determined that they belong to the same scene.

[0074] If the scene determination unit 109 determines that the scenes are the same, the content determination unit 103 identifies the earliest time code and the latest time code for that scene based on the information attached to the scene ID. The earliest time code identified is then designated as the "start time code," and the latest time code as the "end time code," and the generation period for the virtual viewpoint image is determined.

[0075] When the scene determination unit 109 determines that the scenes are the same, the virtual viewpoint setting unit 104 sets the position of the virtual camera so that the 3D model of the object ID stored in each figure information fits within the field of view and generates virtual viewpoint information.

[0076] When the above-mentioned generation period is determined, the image generation unit 106 generates a virtual viewpoint image that reproduces the specific scene indicated by the scene ID, using a 3D model associated with the timecode between the time indicated by the start timecode and the time indicated by the end timecode for the said generation period.

[0077] The above outlines the main differences between the image generation device 12 according to this embodiment and embodiments 1 and 2 of the respective functional units.

[0078] <Operation Flow of Image Generation Device> Next, the flow of the virtual viewpoint image generation process in the image forming apparatus 12 according to this embodiment will be explained, focusing on the differences from embodiments 1 and 2, with reference to the flowchart in Figure 16. In the following explanation, the symbol "S" represents a step.

[0079] In step S1601, the data receiving unit 101 receives the captured image of the figures from the mobile terminal 11. Figure 17 shows an example of an image obtained when a user photographs baseball player figures placed on a desk using the camera function of the mobile terminal 11. In the example in Figure 17, an example of a captured image shows the batter figure f31 and the pitcher figure f32. Two-dimensional markers m31 and m32, which store figure information, are attached to the base of each figure.

[0080] In S1602, the image analysis unit 102 analyzes the captured image received in S1601 and obtains figure information from the 2D markers in the captured image. Figure 18 is an example of a table showing figure information obtained from the captured image in Figure 17. The table in Figure 18 stores the object ID, time code, table ID, and scene ID for the 3D models corresponding to the two figures f31 and f32, respectively. The scene ID (scn_20010) also stores the time codes (18:30:02.001-18:30:07.001) of the start and end times of the motion as supplementary information.

[0081] In S1603, the content determination unit 103 counts the number of figures in the input captured image based on the figure information acquired in S1602. In the case of the captured image in Figure 17, the count value will be figure count = 2.

[0082] In S1604, the next process to be executed is determined by whether the count value obtained in S1603 is multiple. If the count value is multiple (2 or more), S1605 is executed next; if it is not multiple (less than 2), S1606 is executed next.

[0083] In S1605, the scene determination unit 109 determines whether the 3D model identified in each figure information pertains to a specific scene, based on the scene ID contained in each of the multiple figure information obtained in S1602. As shown in the table in Figure 18, in the case of the captured image in Figure 17, the scene IDs stored in each of the two figure information are the same, so it is determined that the 3D model identified in each figure information pertains to a specific scene.

[0084] In S1606, the content determination unit 103 determines the display size and visual effects of the objects in the virtual viewpoint image generated in S1609 (described later) based on the number of figures acquired in S1603. If it is determined in S1605 that the image relates to a specific scene, as shown in the captured image in Figure 17, the generation period of the virtual viewpoint image is determined based on the accompanying information of the scene ID included in the figure information.

[0085] In S1607, the 3D model acquisition unit 105 acquires the corresponding 3D model based on the figure information acquired in S1602. If it was determined in S1605 that the figure information pertains to a specific scene, the 3D models corresponding to each time from the start timecode to the end timecode of the generation period determined in S1606 are acquired in sequence. In this case, instead of acquiring 3D models for all frames from the start timecode to the end timecode, it is also possible to acquire 3D models only for frames that are thinned out at equal intervals, such as once every two frames. In this way, if there is multiple figure information that is determined to pertain to a specific scene, 3D models are acquired to fill in the gaps between the timecodes indicated by each of the multiple figure information. If it was determined in S1605 that the figure information does not pertain to a specific scene, the process described in Embodiment 1 is executed.

[0086] In S1608, similar to S606 mentioned above, the virtual viewpoint setting unit 104 sets virtual viewpoint information based on the shooting parameters of the mobile terminal 11 acquired in S1602 and the display size determined in S1606.

[0087] In S1609, similar to S607 mentioned above, the image generation unit 106 places the 3D model acquired in S1607 into the virtual space and generates a virtual viewpoint image by rendering based on the virtual viewpoint information set in S1608 and the visual effects determined in S1606. In this case, if the specific scene is, for example, a pitcher vs. batter showdown, it is desirable to place the respective 3D models according to the relative positions of the pitcher and batter in an actual game.

[0088] In S1610, similar to S608 mentioned above, the data output unit 107 transmits the virtual viewpoint image generated in S1609 to the mobile terminal 11. Upon receiving the virtual viewpoint image, the mobile terminal 11 displays the image superimposed on the image captured by the built-in camera, thereby realizing an augmented reality image.

[0089] In S1611, similar to S609 mentioned above, it is determined whether there are any unprocessed frames based on the timecode of the figure information acquired in S1602. If there are unprocessed frames, the process returns to S1607 and is repeated. If it was determined in S1605 that the process relates to a specific scene, the process will be repeated up to the frame with the end timecode of the generation period determined in S1606.

[0090] The above describes the flow of the virtual viewpoint image generation process in the image forming apparatus 12 according to this embodiment. Figure 19 shows an augmented reality image corresponding to the captured image in Figure 17. In the augmented reality image shown in Figure 19, which is realized by this embodiment, the pitcher throwing the ball and the batter swinging the bat are represented by two figures f31 and f32 in the foreground a31 and a32 (= virtual viewpoint image) and displayed as a video. In the example of Figure 19, in order to create a more realistic virtual viewpoint image, the position of the virtual camera is set so that the 3D model of the batter fits within the field of view when viewed from behind the 3D model of the pitcher. In the example of Figure 19, as in Embodiment 2, the position and orientation of the virtual viewpoint are fixed during the determined generation period, but they may be changed according to the number of figures (for example, gradually approaching, moving from a viewpoint near the ground to an overhead viewpoint, etc.).

[0091] <Variation> In this embodiment, a virtual viewpoint image of the entire scene was generated based on the start and end time codes indicated in the accompanying information of the scene ID. However, a virtual viewpoint image of only a portion of the scene may also be generated.

[0092] Furthermore, in this embodiment, when it is determined from the scene ID that a 3D model relates to a specific scene, a virtual viewpoint image is generated that reproduces the specific scene using the 3D model. However, this is not limited to this. For example, a group ID that identifies the team may be assigned to each 3D model corresponding to all players belonging to the same team. Then, a group ID may be extracted from each of the multiple figure information obtained by photographing multiple figures that appear in the captured image, and if it is determined that they belong to the same team, a pre-set team-specific visual effect may be added.

[0093] As described above, according to this embodiment, when multiple figures represent different moments in a specific scene, a virtual viewpoint image is generated that interpolates the time intervals between each moment. This allows the user to enjoy an augmented reality image that recreates the specific scene represented by the figures.

[0094] (Other embodiments) This disclosure can also be implemented by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions.

[0095] Furthermore, this disclosure includes the following configurations and methods.

[0096] [Configuration 1] A first acquisition means for acquiring a captured image obtained by photographing a three-dimensional object, and camera parameters indicating the position and orientation of the photographing device used for the photograph, A second acquisition means for acquiring virtual viewpoint images whose content differs based on the number of three-dimensional objects included in the captured image, wherein the virtual viewpoint image is generated based on the shape data of the object and the camera parameters, and the content of the virtual viewpoint image differs based on the number of three-dimensional objects included in the captured image. Display control means for controlling the display of the virtual viewpoint image An information processing device characterized by having the following features.

[0097] [Configuration 2] Analysis means for analyzing the acquired captured images to obtain information about the three-dimensional object and camera parameters indicating the position and orientation of the imaging device used for the imaging, A determination means for determining the content of a virtual viewpoint image based on the number of three-dimensional objects included in the acquired captured image, A third acquisition means for acquiring shape data of the object based on the information about the three-dimensional object that has been acquired, The information processing device according to configuration 1, further comprising

[0098] [Configuration 3] The information processing apparatus according to configuration 2, characterized in that the second acquisition means acquires a virtual viewpoint image of the content determined by the determination means by generating it using the acquired shape data and virtual viewpoint information based on the camera parameters.

[0099] [Structure 4] The information processing apparatus according to configuration 3, characterized in that the determination means determines the size of the object in the virtual viewpoint image based on the number of three-dimensional objects, as content of the virtual viewpoint image.

[0100] [Composition 5] The information processing device according to configuration 4, characterized in that the size of the object when the number of three-dimensional objects is a first number is larger than the size of the object when the number of three-dimensional objects is a second number, which is smaller than the first number.

[0101] [Composition 6] The virtual viewpoint information includes information on the position and orientation of the virtual camera. The position of the virtual camera is the position of the imaging device indicated by the acquired camera parameters, modified according to the determined size of the object. The information processing apparatus according to configuration 4 or 5, characterized by the above.

[0102] [Composition 7] The information processing apparatus according to configuration 2 or 3, characterized in that the determination means determines the content of the virtual viewpoint image, based on the number of three-dimensional objects, the effects of the virtual viewpoint image.

[0103] [Structure 8] The aforementioned effect is a virtual representation of the foreground in the virtual viewpoint image, The content of the performance effect when the number of three-dimensional objects is a first number is more flashy or larger in scale than the performance effect when the number of three-dimensional objects is a second number, which is smaller than the first number. The information processing apparatus according to configuration 7, characterized by the features described above.

[0104] [Composition 9] The aforementioned effect is a virtual representation of the foreground in the virtual viewpoint image, The aforementioned determination means is When the number of three-dimensional objects is less than two, it is decided not to perform the aforementioned effect. When the number of the three-dimensional objects is two or more, it is decided to perform the aforementioned effect. The information processing apparatus according to configuration 7 or 8, characterized by the above.

[0105] [Configuration 10] The information processing apparatus according to any one of configurations 2 to 9, characterized in that the analysis means obtains information about the three-dimensional object by extracting a code encoded with information about the three-dimensional object from the captured image and decoding the code.

[0106] [Composition 11] The shape data represents the three-dimensional shape of the object at a certain time, The information relating to the three-dimensional object includes an object ID that identifies the object and a time code indicating the time, The third acquisition means acquires the shape data based on the object ID and the time code. An information processing apparatus according to any one of configurations 2 to 9, characterized by the above.

[0107] [Composition 12] The information processing apparatus according to any one of configurations 2 to 11, characterized in that the second acquisition means generates a virtual viewpoint image that reproduces the same motion when the number of three-dimensional objects included in the acquired captured image is multiple and the multiple three-dimensional objects relate to the same motion of the same object.

[0108] [Composition 13] The information processing device according to configuration 12, further comprising motion determination means for determining whether, when the number of three-dimensional objects included in the acquired captured image is multiple, the multiple three-dimensional objects relate to the same motion relating to the same object.

[0109] [Composition 14] The information relating to the three-dimensional object includes identification information for uniquely identifying the motion, The information processing device according to configuration 13, characterized in that the motion determination means determines whether the plurality of three-dimensional objects are related to the same motion relating to the same object, based on the identification information contained in the acquired information about the three-dimensional objects.

[0110] [Composition 15] The information processing apparatus according to any one of configurations 2 to 11, characterized in that the second acquisition means generates a virtual viewpoint image that reproduces the same scene when the number of three-dimensional objects included in the acquired captured image is multiple and the multiple three-dimensional objects relate to the same scene.

[0111] [Composition 16] The information processing device according to configuration 15, further comprising a scene determination means for determining whether multiple three-dimensional objects included in the acquired captured image belong to the same scene.

[0112] [Composition 17] The information relating to the three-dimensional object includes identification information for uniquely identifying the scene, The information processing device according to configuration 16, characterized in that the scene determination means determines whether the plurality of three-dimensional objects belong to the same scene based on the identification information contained in the acquired information about the three-dimensional objects.

[0113] [Composition 18] The information processing device according to configuration 10, characterized in that the analysis means acquires the camera parameters based on a code that encodes information about the three-dimensional object contained in the captured image or an image of the three-dimensional object.

[0114] [Method 1] A first acquisition step involves acquiring a captured image obtained by photographing a three-dimensional model of an object, and camera parameters indicating the position and orientation of the photographing device used for the photograph. A second acquisition step involves acquiring virtual viewpoint images, the virtual viewpoint images being generated based on the shape data of the object and the camera parameters, wherein the content of the virtual viewpoint images differs based on the number of three-dimensional objects included in the captured image. A display control step that controls the display of the virtual viewpoint image, An information processing method characterized by having the following features.

[0115] [Composition 19] A program for causing a computer to function as an information processing device as described in any one of the configurations 1 to 18.

Claims

1. A first acquisition means for acquiring a captured image obtained by photographing a three-dimensional object, and camera parameters indicating the position and orientation of the photographing device used for the photograph, A second acquisition means for acquiring virtual viewpoint images whose content differs based on the number of three-dimensional objects included in the captured image, wherein the virtual viewpoint image is generated based on the shape data of the object and the camera parameters. Display control means for controlling the display of the virtual viewpoint image An information processing device characterized by having the following features.

2. Analysis means for analyzing the acquired captured images to obtain information about the three-dimensional object and camera parameters indicating the position and orientation of the imaging device used for the imaging, A determination means for determining the content of a virtual viewpoint image based on the number of three-dimensional objects included in the acquired captured image, A third acquisition means for acquiring shape data of the object based on the information about the three-dimensional object that has been acquired, The information processing apparatus according to claim 1, further comprising the features.

3. The information processing apparatus according to claim 2, characterized in that the second acquisition means acquires a virtual viewpoint image of the content determined by the determination means by generating it using the acquired shape data and virtual viewpoint information based on the camera parameters.

4. The information processing apparatus according to claim 3, characterized in that the determination means determines the size of the object in the virtual viewpoint image based on the number of three-dimensional objects, as content of the virtual viewpoint image.

5. The information processing apparatus according to claim 4, characterized in that the size of the object when the number of three-dimensional objects is a first number is larger than the size of the object when the number of three-dimensional objects is a second number which is smaller than the first number.

6. The virtual viewpoint information includes information on the position and orientation of the virtual camera. The position of the virtual camera is the position of the imaging device indicated by the acquired camera parameters, modified according to the determined size of the object. The information processing apparatus according to feature 4.

7. The information processing apparatus according to claim 2, characterized in that the determination means determines the content of the virtual viewpoint image, based on the number of three-dimensional objects, the effects of the virtual viewpoint image.

8. The aforementioned effect is a virtual representation of the foreground in the virtual viewpoint image, The content of the performance effect when the number of three-dimensional objects is a first number is more flashy or larger in scale than the performance effect when the number of three-dimensional objects is a second number, which is smaller than the first number. The information processing apparatus according to feature 7.

9. The aforementioned visual effect is a virtual representation of the foreground in the virtual viewpoint image, The aforementioned determination means is When the number of the three-dimensional objects is less than two, it is decided not to perform the aforementioned effect. When the number of the three-dimensional objects is two or more, it is decided to perform the aforementioned effect. The information processing apparatus according to feature 7.

10. The information processing apparatus according to claim 2, characterized in that the analysis means obtains information about the three-dimensional object by extracting a code encoded with information about the three-dimensional object from the captured image and decoding the code.

11. The shape data represents the three-dimensional shape of the object at a certain time, The information relating to the three-dimensional object includes an object ID that identifies the object and a time code indicating the time, The third acquisition means acquires the shape data based on the object ID and the time code. The information processing apparatus according to feature 2.

12. The information processing apparatus according to claim 2, characterized in that the second acquisition means generates a virtual viewpoint image that reproduces the same motion when the number of three-dimensional objects included in the acquired captured image is multiple and the multiple three-dimensional objects relate to the same motion of the same object.

13. The information processing apparatus according to claim 12, further comprising motion determination means for determining whether, when the number of three-dimensional objects included in the acquired captured image is multiple, the multiple three-dimensional objects relate to the same motion relating to the same object.

14. The information relating to the three-dimensional object includes identification information for uniquely identifying the motion, The information processing apparatus according to claim 13, characterized in that the motion determination means determines whether the plurality of three-dimensional objects are related to the same motion relating to the same object, based on the identification information contained in the acquired information regarding the three-dimensional objects.

15. The information processing apparatus according to claim 2, characterized in that the second acquisition means generates a virtual viewpoint image that reproduces the same scene when the number of three-dimensional objects included in the acquired captured image is multiple and the multiple three-dimensional objects relate to the same scene.

16. The information processing apparatus according to claim 15, further comprising a scene determination means for determining whether the multiple three-dimensional objects included in the acquired captured image belong to the same scene.

17. The information relating to the three-dimensional object includes identification information for uniquely identifying the scene, The information processing apparatus according to claim 16, characterized in that the scene determination means determines whether the plurality of three-dimensional objects belong to the same scene based on the identification information contained in the acquired information about the three-dimensional objects.

18. The information processing apparatus according to claim 10, characterized in that the analysis means acquires the camera parameters based on a code that encodes information about the three-dimensional object contained in the captured image or an image of the three-dimensional object.

19. A first acquisition step involves acquiring a captured image obtained by photographing a three-dimensional model of an object, and camera parameters indicating the position and orientation of the photographing device used for the photograph. A second acquisition step involves acquiring virtual viewpoint images, the virtual viewpoint images being generated based on the shape data of the object and the camera parameters, wherein the content of the virtual viewpoint images differs based on the number of three-dimensional objects included in the captured image. A display control step that controls the display of the virtual viewpoint image, An information processing method characterized by having the following features.

20. A program for causing a computer to execute the information processing method described in claim 19.

Images