Information processing device, figure manufacturing device, information processing method, figure manufacturing method, and program

Abstract

Displaying various images depending on the image captured of the figure. [Solution] The information processing device includes an acquisition means for acquiring an image containing a figure, a receiving means for receiving a virtual viewpoint image stored in association with the state of the figure included in the image from a storage device in which a different virtual viewpoint image is associated with each of a plurality of states of the figure, and a display control means for performing processing to display the received virtual viewpoint image.

Description

Technical Field

[0001] The present disclosure relates to processing based on data obtained by imaging.

Background Art

[0002] Based on three-dimensional shape data of an object obtained by imaging or scanning a real person or the like, a figure such as a person is generated using a 3D printer or the like. Also, there is a technique for calculating the position and orientation of a mobile terminal based on a captured image obtained by imaging a two-dimensional marker with a mobile terminal such as a smartphone or a tablet, and generating a virtual viewpoint image viewed from a virtual viewpoint corresponding to the calculated position and orientation.

[0003] Patent Document 1 discloses a technique for generating a virtual viewpoint image of a target scene viewed from a virtual viewpoint corresponding to the position and orientation of a camera at the time of imaging when imaging a figure generated based on three-dimensional shape data of a player in a target scene.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the method of Patent Document 1, only a virtual viewpoint image of a specific object in a specific scene viewed from a specific viewpoint is generated, so there are cases where a user who has imaged a figure cannot be made to appreciate various images.

Means for Solving the Problems

[0006] The information processing apparatus of the present disclosure is characterized by comprising: acquisition means for acquiring an image image including a figure; receiving means for receiving a virtual viewpoint image stored in association with the state of the figure included in the image image from a storage device in which a different virtual viewpoint image is associated with each of a plurality of states of the figure; and display control means for performing processing for displaying the received virtual viewpoint image. [Effects of the Invention]

[0007] According to this disclosure, various images can be displayed depending on the image captured of the figure. [Brief explanation of the drawing]

[0008] [Figure 1] A diagram showing the system configuration. [Figure 2] A diagram showing an example of the hardware configuration for an image acquisition device and a figure manufacturing device. [Figure 3] A sequence diagram used to generate the flow of processing between systems. [Figure 4] A diagram illustrating the functional configuration of a figure-making device. [Figure 5] A diagram to explain the figure. [Figure 6] A diagram illustrating the data used to manage figure information in relation to virtual viewpoint images. [Figure 7] A flowchart to explain the figure production process. [Figure 8] A diagram illustrating the functional configuration of an image acquisition device. [Figure 9] A flowchart to explain the process of displaying virtual viewpoint images. [Figure 10] A diagram illustrating the display process of virtual viewpoint images. [Figure 11] A diagram showing the system configuration. [Figure 12] A flowchart to explain the process of displaying virtual viewpoint images. [Modes for carrying out the invention]

[0009] The embodiments of this disclosure will be described below with reference to the drawings. Note that the following embodiments are not intended to limit the technology of this disclosure, and not all combinations of features described in the following embodiments are necessarily essential to the solutions of this disclosure. Furthermore, identical components will be denoted by the same reference numerals.

[0010] <Embodiment 1> This embodiment describes a system that allows a user who has purchased a 3D model figure to take an image of the 3D model figure and then display an associated virtual viewpoint image to the user.

[0011] [About volumetric data] Volumetric data consists of data for multiple frames representing 3D models of objects such as multiple players participating in a match held in a stadium, and texture images corresponding to those 3D models. The 3D models are also called three-dimensional shape data. For example, in this embodiment, the 3D models of players who participated in the match and the texture images corresponding to those 3D models for each of the multiple frames (timecodes) corresponding to one match are included in one volumetric data.

[0012] Figure 1(a) shows an example of the arrangement of imaging devices for generating volumetric data consisting of 3D models of players, balls, etc., in a stadium. Multiple imaging devices 10 are installed to surround the target area 11, which is the stadium field and its surrounding area, and image the target area 11 from various angles in time-synchronized manner. The target of volumetric data generation is not limited to any sport such as soccer, baseball, rugby, volleyball, table tennis, or concerts and various events.

[0013] The target area 11 for volumetric data generation is imaged by a plurality of imaging devices 10, and the volumetric data is generated by processing the imaging data. For example, although not shown in FIG. 1(a), in a stadium having the target area 11, for example, 100 imaging devices are installed, and volumetric data is generated using 100 imaging images.

[0014] For example, 100 imaging devices image the target area 11 in time synchronization, and a process of acquiring imaging images of the 100 imaging devices for each frame is performed. Then, a process of extracting a foreground area, which is an area of an object such as a player, from each of the 100 imaging images is performed. As a method of extracting the foreground area from the imaging image, there is a method of extracting the foreground area by comparing a background image generated based on the input imaging image with the input imaging image. For example, among the pixels of the input imaging image, pixels whose difference in RGB values of the corresponding pixels in the background image is different by a determination threshold value or more can be determined as the foreground area. In addition, any method of image comparison may be used as long as it is a method of extracting the foreground area. The information indicating the extracted foreground area may be, for example, a 1Bit image having the same image size as the imaging image and representing the foreground area as white and the background area as black for each pixel. Or, the information indicating the foreground area may be any information that can indicate the foreground area, and is not particularly limited.

[0015] A 3D model corresponding to the foreground region is generated using the extracted foreground region, external parameters (position and orientation information of 100 pre-acquired imaging devices), internal parameters (such as lens focal length, field of view, and aperture value), and distortion parameters. One method for generating the 3D model is to estimate the three-dimensional shape using the Visual Hull method. As a result of the 3D model generation process, point cloud information representing objects such as players in three dimensions is obtained. Furthermore, the obtained point cloud is divided into point cloud information for each object by processing it using a general 3D labeling method based on the presence or absence of adjacent points. Each point is assigned an object ID as a result of the labeling, and point cloud information for each object can be obtained by specifying the object ID. The method of generating the 3D model is not limited to this method, and various methods can be used. Also, various formats can be used for the 3D model; for example, meshes can be used. By repeating this process for each frame, volumetric data for all players in one match can be obtained.

[0016] [About virtual viewpoint images] A virtual viewpoint image is an image generated to represent an image of an object, such as a player, in the target area 11 as shown in Figure 1(a), viewed from a virtual viewpoint different from the viewpoint from the actual imaging device. The virtual viewpoint image may be a moving image or a still image. In this embodiment, the virtual viewpoint image will be described as a moving image (video) composed of multiple frames. By using volumetric data, the 3D space of the target area 11 where 3D models of objects such as players are placed is reproduced, and the virtual viewpoint image is generated by rendering that 3D space to represent an image viewed from a virtual viewpoint. If the virtual viewpoint is replaced with a virtual camera (virtual camera), the virtual viewpoint image corresponds to the captured image obtained by virtually capturing images with the virtual camera. The position of the virtual camera corresponds to the position of the virtual viewpoint, and the orientation of the virtual camera corresponds to the direction of the line of sight. In this way, if volumetric data is available, it is possible to generate virtual viewpoint images that capture the target area 11 at any time and from any viewpoint.

[0017] [System Configuration] FIG. 1(b) is a diagram showing an example of the configuration of the system of the present embodiment. The system of the present embodiment includes a volumetric data recording system 110, a figure production system 120, an image acquisition device 130, and a storage device 140.

[0018] The volumetric data recording system 110 is a system that includes a generation device that generates volumetric data based on captured images of sports or the like performed in, for example, a stadium. The generated volumetric data is uploaded to a storage device 140 such as a database on the cloud and stored in the storage device 140.

[0019] The figure production system 120 is a system that includes at least a figure production device 121 and a 3D printer 122. The figure production device 121 generates data for the 3D printer 122 to model a 3D model figure (also simply referred to as a figure), which is a physical model such as a doll, from the volumetric data. The 3D printer 122 produces a figure by modeling the three-dimensional shape of an object based on the data generated by the figure production device 121. Further, the figure production device 121 generates a virtual viewpoint image associated with the produced figure and stores it in the storage device 140. The figure production device 121 stores the virtual viewpoint image in the storage device 140 in association with the information of the produced figure. By storing the virtual viewpoint image in this way, the user can view the virtual viewpoint image in cooperation with the figure. The association of the virtual viewpoint image may be changed according to the state of the figure.

[0020] The image acquisition device 130 is an information processing device that acquires a virtual viewpoint image associated with a figure by the user imaging the figure produced by the figure production system 120 and performs display processing on the acquired virtual viewpoint image.

[0021] [Hardware Configuration] Figure 2 shows an example of the hardware configuration of the figure manufacturing apparatus 121 and the image acquisition apparatus 130 of this embodiment.

[0022] The figure making apparatus 121 and the image acquisition apparatus 130 of this embodiment have a CPU 211, ROM 212, RAM 213, HDD 214, communication interface 215, operation unit 216, and display unit 217, and are connected by a system bus 210.

[0023] The CPU 211 is the central processing unit and controls the entire device, including loading computer programs or data stored in the ROM 212 into the RAM 213 and executing the computer programs. The ROM 212 is a memory unit that stores configuration data, boot programs, etc. The RAM 213 is a memory unit that has an area for temporarily storing computer programs and data loaded from the ROM 212, data acquired from the outside via the communication interface 215, etc. The RAM 213 has a work area that the CPU 211 uses when executing various processes. The RAM 213 can be allocated as, for example, frame memory, or various other areas can be provided as appropriate.

[0024] The operation unit 216 is composed of, for example, a keyboard or mouse, and receives user input and inputs various instructions based on the user's input to the CPU 211. The display unit 217 is composed of, for example, a liquid crystal display, and displays the processing results from the CPU 211. The CPU 211 operates as a display control unit that controls the screen displayed on the display unit 217, and as an operation control unit that controls the operation unit 216. In this embodiment, the display unit 217 and the operation unit 216 have been described as being located within the device, but at least one of the display unit 217 and the operation unit 216 may be located as a separate device outside the device.

[0025] HDD214 is a large-capacity information storage unit. HDD214 stores the OS (operating system), computer programs for implementing the functional units included in the device, and other similar data. HDD214 may also store image data for processing. The computer programs and data stored in HDD214 are loaded into RAM213 as appropriate, according to the control of CPU211, and become the target of processing by CPU211.

[0026] The communication interface 215 can connect to networks such as LANs and the Internet, as well as other devices such as projection devices and display devices. The CPU 211 can acquire and transmit various information via the communication interface 215. The system bus 210 is a bus that connects the above-mentioned parts.

[0027] [Sequence Diagram] Figure 3 is a sequence diagram illustrating the overall processing flow of the system.

[0028] In S301, multiple imaging devices 10, as shown in Figure 1, begin imaging the target area 11. For example, when a match starts in the target area 11, the multiple imaging devices 10 begin imaging. In S302, the multiple imaging devices 10 transmit the captured images to the volumetric data recording system 110.

[0029] In S311, the volumetric data recording system 110 generates volumetric data based on the captured images transmitted from the multiple imaging devices 10 and stores the generated volumetric data in the storage device 140. In S303, the multiple imaging devices 10 terminate imaging. For example, when the match being played in the target area 11 ends, the multiple imaging devices 10 terminate imaging.

[0030] In S321, the figure-making device 121 selects the scene for which the figure will be made and the target 3D model corresponding to the athlete in that scene from the volumetric data stored in the storage device 140. The method for selecting the target 3D model will be described later.

[0031] In S322, the figure manufacturing apparatus 121 generates figure modeling data by converting the data of the target 3D model selected in S321 into data that can be input to a 3D printer. S322 also includes data processing such as adding reinforcing members to areas that are too thin and may break when making a figure.

[0032] In S323, the figure production device 121 generates multiple virtual viewpoint images, each with a different perspective.

[0033] In S324, the figure production system 120 associates information indicating the state of the figure at the time of imaging with the virtual viewpoint image generated in S323, and stores the virtual viewpoint image in the storage device 140. The method for generating the virtual viewpoint image will be described later.

[0034] In S331, the 3D printer 122 manufactures a figure based on the figure modeling data generated in S322. The figure manufactured in S331 is associated with the virtual viewpoint image generated in S323. In this embodiment, the figure associated with the virtual viewpoint image is sold.

[0035] A user purchases a figure created with S331. The user who purchased the figure then transitions to a flow to view the virtual viewpoint image associated with the purchased figure.

[0036] In S341, the image acquisition device 130 acquires an image obtained when the user photographs the figure manufactured in S331. For example, if the image acquisition device 130 is a terminal such as a smartphone with an imaging unit, then in S341, the image obtained when the imaging unit of the image acquisition device 130 photographs the figure is acquired.

[0037] In S342, the image acquisition device 130 analyzes the captured image acquired in S341 and receives a virtual viewpoint image from the storage device 140 based on the analysis results.

[0038] In S343, the image acquisition device 130 displays the virtual viewpoint image received in S342 to the user. For example, if the image acquisition device 130 is a terminal such as a smartphone with a display unit, the virtual viewpoint image received in S342 is displayed on the display unit of the image acquisition device 130 in S343.

[0039] In this embodiment, when a figure is imaged, a virtual viewpoint image that has been pre-stored and associated with the figure is displayed. Details of the method for analyzing the captured image and the method for acquiring the virtual viewpoint image will be described later.

[0040] [Functional Configuration of the Figure Production System] Figure 4 is a block diagram illustrating an example of the functional configuration of the figure-making device 121 included in the figure-making system.

[0041] The figure production apparatus 121 includes a model selection unit 401, a modeling data generation unit 402, a virtual viewpoint image generation unit 404, a virtual viewpoint image management unit 405, and an external access unit 406. As will be described later, the figure production apparatus 121 may also include a figure image registration unit 403.

[0042] The model selection unit 401 selects a 3D model (referred to as the target 3D model) that shows the three-dimensional shape of the object in the scene for which the figure will be created, based on the specifications of the figure creator.

[0043] The figure maker specifies the scene (referred to as the target scene) and object (referred to as the target object) to be used as the subject of the figure's creation, following the procedure outlined below. Note that the figure maker is not limited to a single person; all individuals involved in the creation of a figure are collectively referred to as the figure maker. Users who own the figure to be created can also be considered figure makers.

[0044] First, the figure maker specifies the target volumetric data, which is volumetric data containing the 3D model of the athlete for whom the figure will be made. For example, the figure maker specifies the target event (match) by inputting volumetric data information, such as the volumetric data ID, target team (matchup), and match date, into the storage device 140. The volumetric data ID is an identifier used to identify the volumetric data. The user may also specify the volumetric data by inputting any of the information contained in the volumetric data information. The model selection unit 401 accesses the storage device 140 via the external access unit 406 and obtains the target volumetric data, which is the volumetric data specified by the user. As mentioned above, the volumetric data contains a 3D model for each frame (timecode) of one match.

[0045] Next, the figure maker specifies a time range (scene) by specifying a start timecode and an end timecode, and sets the position and orientation of the virtual camera. The model selection unit 401 generates a virtual viewpoint image corresponding to the specified time range and virtual camera from the target volumetric data, and displays the generated virtual viewpoint image on the display unit of the figure production device 121. From the virtual viewpoint image, the figure maker selects a target frame and then selects a target object, such as a player, to be made into a figure from the target frame. The model selection unit 401 selects a 3D model showing the three-dimensional shape of the target object selected from the target frame as the target 3D model for making the figure. At this time, the scene specified by the figure maker becomes the target scene. The model selection unit 401 assigns a scene ID to the target scene to uniquely identify the scene. The information indicating the target scene specified by the figure maker is called scene information. Scene information includes the scene ID, start timecode, and end timecode. In this way, when a target scene is specified from the target volumetric data, a target frame is specified from the target scene, and a target object is specified from the target frame, the model selection unit 401 selects a target 3D model that shows the target object.

[0046] When the model selection unit 401 selects a target 3D model, it outputs volumetric data information to identify the target volumetric data, scene information to identify the target scene, and information to identify the target frame to the modeling data generation unit 402. Furthermore, it outputs information to identify the target 3D model to the modeling data generation unit 402. This information is used to retrieve the target 3D model from the storage device 140.

[0047] The modeling data generation unit 402 generates figure modeling data corresponding to the target 3D model.

[0048] First, the modeling data generation unit 402 obtains the target 3D model and a texture for coloring the three-dimensional shape represented by the target 3D model from the storage device 140 via the external access unit 406, based on the information output from the model selection unit 401. Using the target 3D model and the texture, the modeling data generation unit 402 generates figure modeling data, such as a mesh model, that can be fabricated by the 3D printer 122.

[0049] Furthermore, the modeling data generation unit 402 generates information to identify the state in the captured image when the figure, which is manufactured based on the target 3D model, is captured. In this embodiment, it is explained that information to identify the orientation of the figure is generated. The information to identify the orientation is assumed to be a code image such as a two-dimensional code. The modeling data generation unit 402 generates modeling data for the base so that the generated two-dimensional code is applied to the base of the figure. The modeling data generation unit 402 generates figure modeling data that combines a mesh model of a 3D model representing the three-dimensional shape of an object such as a player, and a base to which the two-dimensional code is applied as a texture. The modeling data generation unit 402 outputs the figure modeling data to the 3D printer 122.

[0050] The 3D printer 122 receives figure modeling data from the modeling data generation unit 402 and uses the received figure modeling data to create a physical figure by three-dimensionally modeling the target object.

[0051] Figure 5 is a diagram illustrating the manufactured figure. Figure 5(a) is a front view of figure 500. Figure 5(b) is a rear view of figure 500. Figure 5(c) is a bottom view of figure 500.

[0052] The modeling data generation unit 402 generates a two-dimensional code that encodes the orientation information of the figure in order to identify the orientation of the figure in the captured image when the user takes a picture of the figure. For example, the modeling data generation unit 402 generates a two-dimensional code that encodes the orientation information for each of the six directions (front, right, left, back, top, and bottom). Then, when generating the modeling data for the figure, the modeling data generation unit 402 applies a two-dimensional code for the front as a texture to the front side of the base and a two-dimensional code for the back side as a texture to the back side. In this embodiment, the top code is applied to the top surface of the base and the bottom code is applied to the bottom surface of the base. As a result, the generated figure 500 consists of at least a figure body 501 and a base 502, as shown in Figure 5, and two-dimensional codes 510 to 515 representing the respective orientations are applied to each of the six directions of the base 502. The figure has two-dimensional codes: 510 on the front, 511 on the right, 512 on the left, 513 on the back, 514 on the top, and 515 on the bottom. By analyzing the two-dimensional codes contained in the captured image obtained by photographing the figure 500, it is possible to determine the orientation of the figure in the captured image at the time of photography.

[0053] The information encoded in the two-dimensional codes 510-515 includes not only information about the orientation of the figure, but also information for accessing the storage device 140 and information for identifying the imaged figure. For example, the information encoded in the two-dimensional code includes a figure ID for identifying the figure. Furthermore, it may also include a scene ID for identifying the target scene used in the production of the figure, and a volumetric ID for identifying the target volumetric data.

[0054] In this embodiment, the information that identifies the orientation of the figure in the captured image was described as a two-dimensional code. However, figure modeling data may also be generated by assigning a code to the figure 500 that is difficult for the human eye to see.

[0055] The virtual viewpoint image generation unit 404 generates a virtual viewpoint image associated with the orientation of the figure 500. In this embodiment, as an example, the virtual viewpoint image generation unit 404 generates a virtual viewpoint image based on the instructions of the figure maker. The figure maker specifies the camera work of the virtual camera using the operation unit. The virtual viewpoint image generation unit 404 generates a virtual viewpoint image that is an image captured by the virtual camera specified by the figure maker. The virtual viewpoint image generation unit 404 generates a different virtual viewpoint image for each orientation of the figure 500.

[0056] The virtual viewpoint image management unit 405 uploads multiple virtual viewpoint images generated by the virtual viewpoint image generation unit 404 to the storage device 140 and processes the virtual viewpoint images to be pre-stored in the storage device 140. The virtual viewpoint image management unit 405 also stores the virtual viewpoint images in the storage device 140 in association with a figure ID for identifying the figure 500. Furthermore, in this embodiment, each generated virtual viewpoint image is stored in association with information indicating the orientation of the figure. For example, the two-dimensional code on the figure's base encodes information indicating the orientation of the figure as the state of the figure. Therefore, the virtual viewpoint image management unit 405 processes the virtual viewpoint images generated to be associated with a certain orientation of the figure to be stored in the storage device 140 in association with the figure ID that identifies that figure and the information indicating the orientation of that figure. The information including the figure ID, the information indicating the orientation of the figure, and the information indicating the virtual viewpoint image is called figure information.

[0057] Figure 6 shows an example of data for managing figure information, which is information associated with a virtual viewpoint image generated by the virtual viewpoint image generation unit 404 and stored in the storage device 140. As shown in Figure 6, the figure information may also be associated with volumetric information that identifies the volumetric data used to manufacture the figure, and scene information that identifies the target scene. By associating volumetric information and scene information with the figure information in this way, the searchability when searching for a target figure or virtual viewpoint image can be improved. For this reason, volumetric data information, scene information, and figure information are acquired from the modeling data generation unit 402 and uploaded to the storage device 140 via the external access unit 406.

[0058] There are various possible methods for managing the virtual viewpoint images generated by the virtual viewpoint image generation unit 404 in the storage device 140.

[0059] For example, the virtual viewpoint image generation unit 404 generates six different virtual viewpoint images A to F to be associated with a certain figure. The virtual viewpoint image management unit 405 stores the virtual viewpoint images A to F in the storage device 140, associating them with the orientation of the figure in six directions. In this case, for example, as shown in the figure information in Figure 6(a), the figure ID for identifying the figure is associated with the virtual viewpoint images A to F. Furthermore, the virtual viewpoint images A to F are stored by associating each of the six orientations A to F of the figure with one virtual viewpoint image.

[0060] Alternatively, the virtual viewpoint image generation unit 404 generates two different virtual viewpoint images, A and B, to be associated with a certain figure. The virtual viewpoint image management unit 405 stores virtual viewpoint images A and B in the storage device 140, associating them with the six orientations of the figure. In this case, as shown in Figure 6(b), among the six orientations A to F, virtual viewpoint image A is associated with orientations A, B, and C, and virtual viewpoint image B is associated with orientations D, E, and F, and the virtual viewpoint images are stored in the storage device 140. Thus, the number of orientations of the figure and the number of virtual viewpoint images do not have to match.

[0061] Each functional unit shown in Figure 4 is realized by the CPU of the figure-making apparatus 121 executing a predetermined program, but is not limited to this. Other hardware such as a GPU (Graphics Processing Unit) or FPGA (Field Programmable Gate Array) may also be used to speed up calculations. Each functional unit may be realized through the cooperation of software and hardware such as a dedicated IC, or some or all of the functions may be realized by hardware alone.

[0062] [Regarding virtual viewpoint images associated with figures] As an example of a virtual viewpoint image associated with a figure, four methods can be considered for generating the virtual viewpoint image. The target scene is assumed to be a sports scene, and the object from which the figure is created is player Z.

[0063] (1) Generate a virtual viewpoint image that reflects the direction in which the image was captured. Method (1) involves generating virtual viewpoint images of player Z, the target object in the target scene used to create the figure, from each orientation corresponding to the two-dimensional code. For example, a virtual viewpoint image A is generated, captured by a virtual camera with camera work that primarily captures player Z in the target scene from the front. Virtual viewpoint image A is stored in association with the orientation indicated by the two-dimensional code 510 for the front. Similarly, a virtual viewpoint image B is generated, captured by a virtual camera with camera work that captures player Z in the target scene from the right side, and virtual viewpoint image B is stored in association with the orientation indicated by the two-dimensional code 512 for the right side. In this way, virtual viewpoint images are generated that appear as if the target object in the target scene were captured with camera work centered on the orientation of the virtual camera corresponding to the orientation of the figure, and the generated virtual viewpoint images are stored in association with the orientation of the figure. According to Method (1), since the virtual viewpoint image is linked to the orientation captured by the user, it has the effect of making it easier for the user to imagine the virtual viewpoint image that can be viewed.

[0064] (2) Generate a virtual viewpoint image of the target object in a scene other than the target scene. Method (2) is a method that generates a virtual viewpoint image of player Z, the target object, in a scene different from the target scene, captured by a virtual camera. For example, a virtual viewpoint image B is generated by capturing player Z in a different scene with a virtual camera, and the virtual viewpoint image B of the different scene is stored in association with directions other than the front, such as the right, left, and back, as indicated by the two-dimensional codes 511 to 515. Alternatively, a virtual viewpoint image A is generated by a virtual camera that captures player Z in the target scene mainly from the front, and the virtual viewpoint image A is stored in association with the direction indicated by the two-dimensional code 510 for the front. Since users usually photograph figures from the front, the virtual viewpoint image A of the target scene from which the figure originated may be stored in association with the direction indicated by the two-dimensional code 510 indicating the front. For example, in the case of baseball, the target scene could be the scene of player Z hitting a home run, and the other scene could be the scene of player Z's previous at-bat. Or, the target scene could be the scene of player Z celebrating, and the other scene could be the home run scene that preceded the scene of player Z celebrating. By associating and storing virtual viewpoint images from different scenes in this way, users can enjoy virtual viewpoint images of related scenes, such as the scene that triggered the target scene, by photographing the figure from an angle other than the front. Furthermore, if a player achieves a "multi-hit game" by hitting three or more home runs in a single day, virtual viewpoint images of other hits and home runs may be generated as separate scenes. By associating and storing virtual viewpoint images of other scenes related to the target object in relation to the object's orientation, users can enjoy viewing many scenes in which the player, the target object, performs well.

[0065] (3) Generate a virtual viewpoint image of another player in the target scene. Method (3) is a method for generating a virtual viewpoint image by capturing a player other than the target object player in the target scene with a virtual camera. For example, a virtual viewpoint image B is generated by capturing a player Y, who is different from the target object player in the target scene, with a virtual camera, and virtual viewpoint image B is stored in association with directions other than the front, such as the right, left, and back, as indicated by the two-dimensional codes 511 to 515. Alternatively, for the same reasons as in method (2), a virtual viewpoint image A may be generated by capturing a virtual camera with camera work that mainly captures player Z in the target scene from the front, and virtual viewpoint image A may be stored in association with the direction indicated by the two-dimensional code 510 for the front. For example, in the case of baseball, if the target scene is a scene of player Z hitting a home run, a virtual viewpoint image B is generated as if captured by a virtual camera so that the facial expressions of player Y, such as the pitcher or catcher who gave up the home run, can be seen. In the case of soccer, if the target scene is a goal scored by player Z, a virtual viewpoint image B is generated, which is a virtual camera image of the opposing player Y defending, or the celebratory expression of another player Y on the same team as player Z. In this way, the user can view the target scene from a different angle.

[0066] (4) Generate a virtual viewpoint image of a different player in a different scene. Method (4) is a method for generating a virtual viewpoint image by capturing a different player from the target object player in a different scene using a virtual camera. A different scene is, for example, a scene before or after the target scene. For example, a virtual viewpoint image B is generated by capturing a different player Y from the target object in the different scene using a virtual camera, and virtual viewpoint image B is stored in association with directions other than the front, such as the right, left, or back, as indicated by the two-dimensional codes 511-515. Alternatively, for the same reasons as in method (2), a virtual viewpoint image A may be generated by capturing player Z in the target scene with a virtual camera using camera work that mainly captures the player from the front, and virtual viewpoint image A may be stored in association with the direction indicated by the two-dimensional code 510 for the front. For example, in the case of baseball, if the target scene is a scene where a timely hit is made, a different scene could be a scene of player Y, the batter before that timely hit, getting a hit or a walk. In the case of soccer, if the target scene is a scene where player Z scores a goal, a different scene could be a scene of another player Y starting to move, which leads to the goal. In this way, the user can view related images.

[0067] The position and orientation of the virtual camera used to generate the virtual viewpoint image B in the above-described methods (2) to (4) do not need to be related to the orientation of the associated figure 500. The virtual viewpoint image B in the above-described methods (2) to (4) shall include multiple virtual viewpoint images different from virtual viewpoint image A.

[0068] Furthermore, the virtual viewpoint image associated with the orientation of Figure 500 is not limited to the virtual viewpoint image of the event (match) used to generate Figure 500. A virtual viewpoint image associated with the orientation of Figure 500 may be generated by specifying volumetric data different from the target volumetric data selected when generating Figure 500. Also, the image associated with the orientation of Figure 500 is not limited to a virtual viewpoint image. For example, the image associated with the orientation of Figure 500 may be an actual video or still image captured by any of the multiple imaging devices 10, or it may be a video or still image captured separately for the purpose of associating it with the figure. In addition, the data associated with Figure 500 is not limited to images; data such as the performance of the athlete who served as the model for Figure 500 may also be associated. Alternatively, these images or data may be combined and associated with the orientation of the figure.

[0069] [flowchart] Figure 7 is a flowchart illustrating the process of manufacturing a figure and associating a virtual viewpoint image with the figure in this embodiment. The series of processes shown in Figure 7 are realized by loading a program stored in the ROM or HDD of the figure manufacturing device 121 into RAM and executing it on the CPU. Alternatively, some or all of the functions of the steps in Figure 7 may be realized by hardware such as an ASIC or electronic circuit. The symbol "S" in the description of each process means that it is a step in the flowchart, and the same applies to subsequent flowcharts.

[0070] In S701, the model selection unit 401 generates a virtual viewpoint image corresponding to the target scene in the event (match) specified by the figure maker, and displays the generated virtual viewpoint image on the display unit of the figure making device 121. When the figure maker selects a target frame and target object from the virtual viewpoint image, the model selection unit 401 selects a target 3D model corresponding to the target object in the target scene (target frame). The model selection unit 401 transmits information for identifying the target 3D model to the modeling data generation unit 402.

[0071] In S702, the modeling data generation unit 402 acquires the target 3D model from the storage device 140 and generates figure modeling data based on the target 3D model.

[0072] In S703, the modeling data generation unit 402 outputs figure modeling data to the 3D printer 122, and the 3D printer 122 models the figure 500 based on the figure modeling data.

[0073] In S704, the virtual viewpoint image generation unit 404 determines whether the generation of the virtual viewpoint image associated with the figure generated in S703 has finished. For example, if the figure creator inputs an instruction to finish generating the virtual viewpoint image, the virtual viewpoint image generation unit 404 determines that the generation of the virtual viewpoint image has finished. If the virtual viewpoint image generation unit 404 determines that the generation of the virtual viewpoint image has finished (S704 is YES), it proceeds to S708; otherwise (S704 is NO), it proceeds to S705.

[0074] In step S705, the virtual viewpoint image generation unit 404 selects the orientation of the figure. Alternatively, in step S705, a two-dimensional code indicating the orientation of the figure may be selected.

[0075] In S706, the virtual viewpoint image generation unit 404 generates a virtual viewpoint image associated with the orientation of the figure selected in S705. The figure creator generates a virtual viewpoint image associated with the orientation of the figure selected in S706 using one of the methods (1) to (4) described above. For example, the figure creator selects a scene from the target volumetric data used to generate the figure and sets the position and orientation of the virtual camera. The virtual viewpoint image generation unit 404 generates a virtual viewpoint image that appears as if the scene reproduced by the volumetric data used to generate the figure was captured by the set virtual camera.

[0076] In S707, the virtual viewpoint image management unit 405 associates the virtual viewpoint image generated in S706 with the orientation of the figure selected in S705.

[0077] As steps S705 to S707 are repeatedly executed, a virtual viewpoint image is generated that is associated with information indicating the orientation of the figure, as shown in Figure 6.

[0078] In S708, the virtual viewpoint image management unit 405 aggregates scene information, figure information, generated virtual viewpoint images, and figure orientation information associated with the virtual viewpoint images, and uploads them to the storage device 140 via the external access unit 406. As a result, the virtual viewpoint images are stored in the storage device 140 in association with the figure orientations.

[0079] [Display processing of virtual viewpoint images] This document explains how a figure buyer can photograph the figure and view a virtual viewpoint image associated with the figure's orientation at the time of the photograph.

[0080] Figure 8 shows an example of the functional configuration of the image acquisition device 130. The image acquisition device 130 includes an image acquisition unit 801, an image recognition unit 802, a determination unit 804, a specification unit 803, a data access unit 805, and a display control unit 806.

[0081] The image acquisition unit 801 acquires the captured image obtained by imaging the figure 500. For example, suppose the image acquisition device 130 is a user terminal having an imaging unit and a display unit. In this case, the imaging unit of the image acquisition device 130 images the figure 500 in response to the user's operation of the image acquisition device 130, and the image acquisition unit 801 acquires the resulting captured image.

[0082] The image recognition unit 802 acquires the captured image acquired by the image acquisition unit 801, analyzes the acquired image, and extracts the two-dimensional code contained in the image. The image recognition unit 802 outputs the position and size information of the extracted two-dimensional code to the determination unit 804. If no two-dimensional code is extracted, the image recognition unit 802 determines that recognition has failed and presents an error to the user.

[0083] The determination unit 804 determines which of the two-dimensional codes contained in the captured image will be decoded, based on the size and position of the two-dimensional codes contained in the captured image. As mentioned above, if two-dimensional codes 510 to 515 are assigned to the six directions of the figure 500, the largest and least distorted two-dimensional code among the two-dimensional codes contained in the captured image is identified as the two-dimensional code to be decoded. The two-dimensional code to be decoded is used as a two-dimensional code to identify the orientation of the captured figure 500 at the time of imaging. In this way, in this embodiment, the orientation of the figure 500 can be identified from the captured image obtained by imaging the figure 500.

[0084] The identification unit 803 decodes the two-dimensional code to be decoded and outputs the resulting information to the data access unit 805. The information obtained by decoding the two-dimensional code includes access information to the storage device 140, information for identifying the captured figure 500, and information on the orientation of the captured figure 500. Therefore, the identification unit 803 can identify the orientation of the figure 500 in the captured image as the state of the captured figure 500 in the captured image. As mentioned above, the information for identifying the captured figure 500 is the figure ID. Furthermore, if the volumetric ID and scene ID are encoded in the two-dimensional code, the volumetric ID and scene ID are also obtained as information for identifying the captured figure 500. The information obtained by decoding the two-dimensional code may also be a URL for accessing a virtual viewpoint image associated with the orientation of the figure 500 in the captured image. In this case, the URL will be the access information to the storage device 140, the information for identifying the captured figure 500, and the information on the orientation of the captured figure.

[0085] The data access unit 805 accesses the storage device 140 based on the access information. It then inputs the figure ID and figure orientation information to the storage device 140 to identify the captured figure 500, and receives the virtual viewpoint image stored in the storage device 140 in association with the input figure ID and figure orientation.

[0086] The display control unit 806 performs the process of displaying the received virtual viewpoint image. If the image acquisition device 130 is a user's terminal, the display control unit 806 displays the virtual viewpoint image received by the data access unit 805 on the display unit of the image acquisition device 130.

[0087] Each functional unit shown in Figure 8 is realized by the CPU of the image acquisition device 130 executing a predetermined program, but is not limited to this. Other hardware such as a GPU (Graphics Processing Unit) or FPGA (Field Programmable Gate Array) may also be used to speed up calculations. Each functional unit may be realized through the cooperation of software and hardware such as a dedicated IC, or some or all of the functions may be realized by hardware alone.

[0088] Figure 9 is a flowchart illustrating the display process of a virtual viewpoint image stored in association with the orientation of a figure. The series of processes shown in Figure 9 are realized by loading a program stored in the ROM or HDD of the image acquisition device 130 into RAM and executing it on the CPU. Alternatively, some or all of the functions of the steps in Figure 9 may be realized by hardware such as an ASIC or electronic circuit.

[0089] In S901, the image acquisition unit 801 acquires the image obtained when the user takes a picture of the figure 500. The image is output to the image recognition unit 802.

[0090] Figure 10 is a diagram illustrating the display process of a virtual viewpoint image. As shown in Figure 10(a), when the figure 500 is imaged from the rear, S901 acquires an image 1001 that includes the figure 500 in the rear-facing position.

[0091] In S902, the image recognition unit 802 analyzes the captured image acquired in S901 and extracts the two-dimensional codes contained in the captured image. For example, if the captured image 1001 shown in Figure 10(a) is acquired, the two-dimensional codes 512 to 514 contained in the captured image 1001 are extracted.

[0092] In S903, the determination unit 804 determines the two-dimensional code to be decoded from among the extracted two-dimensional codes. For example, in the case of Figure 10(a), among the two-dimensional codes 512 to 514 contained in the captured image 1001, the two-dimensional code 513, which is the largest and has the least distortion, is determined to be decoded.

[0093] In S904, the identification unit 803 decodes the determined two-dimensional code and obtains access information to the storage device 140 and a figure ID that identifies the figure 500. Furthermore, the identification unit 803 identifies the orientation of the figure 500 in the captured image from the information obtained by decoding the determined two-dimensional code and obtains orientation information. The identification unit 803 outputs the acquired information to the data access unit 805.

[0094] For example, by decoding the two-dimensional code 513 shown in Figure 10(a), information for accessing the storage device 140 and a figure ID for identifying the figure 500 are obtained. In addition, the identification unit 803 determines that the orientation of the figure 500 in the captured image is the back by decoding the two-dimensional code 513.

[0095] In S905, the data access unit 805 accesses the storage device 140 and receives a virtual viewpoint image stored in association with the orientation of the figure 500 included in the captured image.

[0096] In S906, the display control unit 806 displays the virtual viewpoint image received in S905 on the display unit of the image acquisition device 130, which is the user's terminal.

[0097] For example, let's assume that the storage device 140 stores a virtual viewpoint image as shown in Figure 6(a). Also, let's assume that the figure ID obtained in S904 from the two-dimensional code 513 contained in the captured image 1001 is "222", and the orientation of figure 500 in the captured image is orientation D showing the back. In this case, the virtual viewpoint image D associated with figure ID "222" and orientation D showing the back will be received from the storage device 140. As a result, as shown in Figure 10(b), the virtual viewpoint image 1010, which is the virtual viewpoint image D, is displayed on the user terminal's display. For example, let's assume that figure 500 is a figure generated from a 3D model of player Z in a scene celebrating a home run. In this case, if the user captures figure 500 from the back, the user terminal can display a virtual viewpoint image D that is similar to the home run scene that was the premise for the scene where player Z is celebrating.

[0098] In this embodiment, the virtual viewpoint image based on the analysis results of the captured image is described as being displayed on the user terminal. Alternatively, the user terminal may display options for which virtual viewpoint image to display. For example, when figure 500 is captured, a list of displayable virtual viewpoint images A to F may be displayed on the user terminal in list format. The list of virtual viewpoint images may be configured so that the virtual viewpoint image D associated with the orientation D of figure 500 in the captured image is displayed at the top of the list.

[0099] Furthermore, when figure 500 is positioned within the field of view of the imaging unit, thumbnails of virtual viewpoint images A to F associated with figure 500 may be superimposed. In this case, the thumbnails may also display which orientation of figure 500 they are associated with, so that the user can see which virtual viewpoint image will be displayed depending on the orientation in which the image is captured.

[0100] Figure 11 shows another example of the system configuration of this embodiment. The image acquisition device 130 has been described as, for example, a user terminal having an imaging unit and a display unit. Alternatively, the image acquisition device 130 may be an information processing device such as a server that transmits virtual viewpoint images to a user terminal 1100, such as a smartphone having an imaging unit and a display unit. For example, when a user images figure 500 with terminal 1100, terminal 1100 transmits the resulting image to the image acquisition device 130. As a result, in S901, the image acquisition unit 801 may acquire an image containing figure 500. Also, in S906, the display control unit 806 may transmit the virtual viewpoint image data received in S905 to the user terminal 1100, thereby displaying the virtual viewpoint image associated with the orientation of the imaged figure 500 on the display unit of terminal 1100.

[0101] [Example 1] In the above explanation, the virtual viewpoint image was described as being associated with the orientation of the figure in the captured image as the state of the figure in the captured image. Alternatively, instead of the orientation of the figure, or in addition to the orientation of the figure, the virtual viewpoint image may be associated with the position (distance) of the figure relative to the imaging device at the time of imaging, and the virtual viewpoint image may be stored in the storage device 140.

[0102] In that case, for example, in the methods (1) to (4) described as virtual viewpoint image generation methods, two virtual viewpoint images may be generated that are associated with one orientation of the figure. For example, the virtual viewpoint image generation unit 404 of the figure production device 121 generates two types of virtual viewpoint images corresponding to the size of the figure 500 included in the captured image obtained when the user photographs the figure 500. When the virtual viewpoint image generation unit 404 generates virtual viewpoint image A using methods (1) to (4), it may generate two types: virtual viewpoint image A1 associated with a position close to the imaging device, which is the zoom-in center, and virtual viewpoint image A2 associated with a position far from the imaging device, which is the zoom-out center. By associating and storing the position (distance) of the figure 500 at the time of imaging with the virtual viewpoint image in this way, the size of the figure 500 in the captured image obtained when the user photographs it can be linked to the size of the player in the virtual viewpoint image. This makes it easier for the user to predict the virtual viewpoint image that will be displayed.

[0103] When a virtual viewpoint image is generated in association with the position (distance) of figure 500 relative to the imaging device, the virtual viewpoint image management unit 405 shall store the virtual viewpoint image in association with the position of figure 500. Alternatively, the virtual viewpoint image management unit 405 may store the virtual viewpoint image in association with the orientation and position of figure 500.

[0104] Figure 6(c) shows an example of a method for associating and storing two types of virtual viewpoint images, one for a close-up position and the other for a wide-angle position, for the same orientation. As shown in Figure 6(c), virtual viewpoint image A1 is stored in association with "close-up," which indicates that it was taken from a frontal orientation A and a close-up position. Virtual viewpoint image A2 is stored in association with "wide-angle," which indicates that it was taken from a frontal orientation A and a wide-angle position. Therefore, if the user takes a close-up image of figure 500 from the front, virtual viewpoint image A1 can be displayed. Other virtual viewpoint images can be generated in a similar manner, and the virtual viewpoint images may be managed by further associating them not only with the orientation of figure 500 but also with information about the position of figure 500 at the time of imaging.

[0105] In this case, the image acquisition device 130, the identification unit 803 determines the position (distance) of the figure 500 relative to the imaging device by analyzing the captured image obtained when the user photographs the figure 500. For example, the identification unit 803 determines the position (distance) of the figure 500 relative to the imaging device using the size of the two-dimensional code to be decoded included in the captured image. For example, it is possible to determine whether the figure 500 was photographed up close or from a distance by checking whether the number of pixels (ratio) of the area of ​​the two-dimensional code to be decoded relative to the size of the entire captured image is above a threshold. For example, if the threshold is set to 0.5%, the identification unit 803 determines that the figure 500 was photographed up close if the ratio of the area of ​​the two-dimensional code relative to the entire captured image is 0.5% or more, and determines that it was photographed from a distance if it is smaller. Information on whether the position of the figure 500 was photographed up close or from a distance is output to the data access unit 805.

[0106] The data access unit 805 of the image acquisition device 130 further inputs information into the storage device 140 regarding whether the figure 500 is in a wide or close position during imaging, and receives a virtual viewpoint image of the object to be displayed. Alternatively, the data access unit 805 may input information about the size of the two-dimensional code to be decoded included in the captured image into the storage device 140, and receive a virtual viewpoint image from the storage device 140 corresponding to the size of the two-dimensional code. The size of the two-dimensional code is, for example, the ratio of the size of the area of ​​the two-dimensional code (number of pixels) to the overall size of the captured image, as described above.

[0107] [Differentiation 2] In the above description, the virtual viewpoint image was stored in association with at least one of the orientation or position of the figure as the state of the figure in the captured image. Alternatively, instead of the orientation or position of the figure, or in addition to at least one of the orientation or position of the figure, the virtual viewpoint image may be associated with the user's posture of the figure at the time of imaging, and the virtual viewpoint image may be stored in the storage device 140.

[0108] For example, the virtual viewpoint image generation unit 404 of the figure production device 121 generates a virtual viewpoint image A of a scene where the player is standing and a virtual viewpoint image B of a scene where the player is sliding. The virtual viewpoint image management unit 405 may store virtual viewpoint image A in the storage device 140 in association with information that the figure 500 is in a standing position, and virtual viewpoint image B in the storage device 140 in association with information that the figure 500 is in a lying position.

[0109] In this case, the image acquisition device 130's identification unit 803 identifies the pose of the figure 500 when the user takes an image of it by analyzing the captured image obtained when the user takes an image of the figure 500. For example, if the captured image contains multiple two-dimensional codes, the identification unit 803 identifies the pose of the figure 500 based on the positional relationship of the multiple two-dimensional codes and their positions on the captured image. If the captured image contains one two-dimensional code, the identification unit 803 determines whether the orientation of the two-dimensional code is up, down, left, or right, and identifies the pose of the figure 500 based on that determination result. The information on the pose of the figure 500 is output to the data access unit 805. The data access unit 805 of the image acquisition device 130 further inputs the information on the pose of the figure 500 at the time of imaging into the storage device 140 and receives the virtual viewpoint image to be displayed. As a result, the user is shown a virtual viewpoint image associated with the pose of the figure 500 when the user took an image of it.

[0110] [Difference 3] The above description explains a method for identifying the orientation, position, and posture of figure 500 from an image obtained by a user capturing a picture of figure 500, based on the two-dimensional code assigned to figure 500. This method is just one example, and the method for identifying the orientation, position, posture, and other states of figure 500 in the captured image is not limited to this method.

[0111] For example, the figure image registration unit 403 of the figure manufacturing device 121 pre-registers images of the figure 500. The figure maker uses an imaging device (not shown) to image the figure 500, which has been fabricated by the 3D printer 122, from multiple directions and positions, or to image the figure 500 in various poses. The resulting images are then input to the figure manufacturing device 121. The figure image registration unit 403 registers the input images in the storage device 140. In this case, the identification unit 803 of the image acquisition device 130 may identify the orientation, position, or posture of the figure 500 by comparing the images obtained by the user imaging the figure 500 with the images registered by the figure image registration unit 403.

[0112] Alternatively, the identification unit 803 may recognize various parts of the figure 500, such as hands and feet, included in the captured image by image recognition, and determine the orientation, position, or posture of the figure 500 based on the recognized parts. Alternatively, the orientation, position, or posture of the figure 500 may be determined from the shape and color of the figure 500.

[0113] [Differentiation Example 4] The above explanation describes how imaging figure 500 displays a virtual viewpoint image of the target scene associated with figure 500, i.e., multiple virtual viewpoint images corresponding to the target scene. Alternatively, a countdown may be displayed to indicate when the virtual viewpoint image generated using the 3D model used to create the figure will be displayed among the multiple virtual viewpoint images corresponding to the target scene.

[0114] Figure 12 is a diagram illustrating the display process of a virtual viewpoint image according to Modification 4. As described above, a virtual viewpoint image associated with a figure may be generated using the target object in the target scene used to create the figure. In this case, the time from the frame corresponding to the virtual viewpoint image displayed on the image acquisition device 130 to the target frame corresponding to the 3D model used to create the figure is calculated, and a countdown is displayed along with the virtual viewpoint image. By displaying the countdown in this way, the user can easily grasp the target frame in which the figure was created.

[0115] For example, suppose the target scene is the scene in which player Z hits a home run. Figure 12(a) shows that the figure 500 included in the captured image 1200 is a figure generated from a 3D model of player Z celebrating in the frame after the home run (target frame) in the target scene. Figure 12(b) shows a video that includes a virtual viewpoint image 1201 corresponding to the frame in which player Z hits the home run that is the premise for player Z's celebration, up to a virtual viewpoint image 1203 corresponding to the target frame in which player Z is celebrating, which was used for figure 500. In this way, a video composed of multiple virtual viewpoint images, including from the frame in which the event that was the premise for figure 500 occurred up to the target frame used for figure 500, may be stored in association with a certain state of figure 500. At this time, the virtual viewpoint images may be stored in a way that allows the target scene to be identified from among the frames that make up the video. When the image acquisition device 130 displays multiple virtual viewpoint images, the display control unit 806 calculates the time until the virtual viewpoint image 1203 of the target frame used to create figure 500 is displayed. The display control unit 806 may also display a countdown 1212 indicating the time until the virtual viewpoint image 1203 corresponding to the target frame is displayed, as shown in the virtual viewpoint image 1202 corresponding to the frame prior to the target frame.

[0116] Furthermore, the method of display is not limited to a countdown, as long as the user can easily identify the frame in which the figure was created. For example, the target object may be highlighted in the virtual viewpoint image corresponding to the frame in which the figure was created. Examples of highlighting include changing the color, overlaying CG on the target object, and various other display methods. In addition to highlighting, the display time of the frame in which the figure was created may also be extended. For example, if multiple virtual viewpoint images corresponding to the target scene are displayed at 60fps each, multiple virtual viewpoint images corresponding to 2 seconds before and after the frame in which the figure was created may be displayed at 5fps. This allows the period before and after the frame in which the figure was created to be displayed in slow motion, making it easy for the user to identify the frame in which the figure was created. In addition to slow motion, only the display time of the frame in which the figure was created may also be extended.

[0117] As described above, according to this embodiment, different images are associated with and stored for each of the multiple states (orientation, position, or posture) of the figure. Therefore, when the figure is photographed, the user can see the image associated with the figure's state (orientation, position, or posture) in the captured image. As a result, by photographing the figure in various orientations, postures, and positions, the user can enjoy viewing different images depending on the figure's orientation, posture, and position at the time of photography. Furthermore, in this embodiment, a virtual viewpoint image is generated in advance, and the generated virtual viewpoint image is associated with and stored in advance. Therefore, in this embodiment, the processing load until the virtual viewpoint image is displayed can be reduced compared to the case where a virtual viewpoint image is generated and displayed each time.

[0118] <Other Embodiments> This embodiment can also be realized by supplying a storage medium containing the code of a computer program that realizes the above-described functions to a system, and having the system read and execute the computer program code. In this case, the computer program code read from the storage medium itself realizes the functions of the embodiment described above, and the storage medium containing the computer program code constitutes the technology of this disclosure. Furthermore, this also includes cases in which an OS or other system running on a computer uses the above-described hardware resources to perform some or all of the actual processing based on the instructions in the program code, and the above-described functions are realized through that processing.

[0119] Furthermore, it may also be implemented in the following form: that is, computer program code read from a storage medium is written to the memory of a function expansion card inserted into a computer or a function expansion unit connected to a computer. Then, based on the instructions in that computer program code, the CPU or other components of the function expansion card or function expansion unit perform some or all of the actual processing to realize the above-mentioned functions.

[0120] When this embodiment is applied to the storage medium described above, the storage medium will store the code of a computer program corresponding to the process described earlier.

[0121] This disclosure can also be implemented by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions. Furthermore, the program may be recorded on a recording medium readable by a computer and provided.

[0122] The above-described embodiments include the following configurations.

[0123] (Composition 1) A means for acquiring an image containing a figure, A receiving means for receiving a virtual viewpoint image stored in association with the state of the figure included in the captured image from a storage device in which different virtual viewpoint images are associated with each of the multiple states of the figure, A display control means that performs processing for displaying the received virtual viewpoint image, An information processing device characterized by having the following features. (Configuration 2) The aforementioned multiple states are at least one of the multiple orientations, multiple postures, or multiple positions of the figure. The information processing device according to configuration 1, characterized by the above. (Composition 3) The system further includes a means for identifying the state of the figure included in the captured image, The receiving means receives a virtual viewpoint image recorded in association with the state of the identified figure. An information processing device according to configuration 1 or 2, characterized by the above. (Composition 4) The aforementioned figure is provided with first information for identifying the state of the figure, The aforementioned identification means identifies the state of the figure from the first information contained in the captured image. The information processing apparatus according to configuration 3, characterized by the features described herein. (Composition 5) The aforementioned multiple states are multiple orientations of the figure, Each figure is provided with a code image at a position corresponding to its orientation, in which information indicating that orientation is encoded. The aforementioned identification means determines the orientation of the figure by decoding the code image included in the captured image. An information processing apparatus according to configuration 3 or 4, characterized by the above. (Composition 6) The figure is provided with second information for accessing the storage device. An information processing device according to any one of configurations 1 to 5, characterized by the above. (Composition 7) The aforementioned storage device stores, for each of the multiple figures, a virtual viewpoint image associated with the multiple states. The aforementioned figure is provided with a third piece of information for identifying the figure, The receiving means receives from the storage device a virtual viewpoint image stored in association with the figure indicated by the third information and the state of the identified figure. An information processing device according to any one of configurations 3 to 5, characterized by the above. (Composition 8) The aforementioned figure is a figure generated based on the three-dimensional shape data of the target object in the target scene, which is among the three-dimensional shape data for each frame generated based on imaging of the target area by multiple imaging devices. Each of the aforementioned multiple states includes a virtual viewpoint image associated with it, which is generated based on the three-dimensional shape data for each frame and shows a predetermined object in a predetermined scene viewed from the virtual viewpoint. An information processing device according to any one of configurations 1 to 7, characterized by the above. (Composition 9) Each of the aforementioned states is associated with a virtual viewpoint image that includes a virtual viewpoint image of the target object in the target scene as seen from the virtual viewpoint. The information processing apparatus according to configuration 8, characterized by the above. (Composition 10) Each of the aforementioned states includes a virtual viewpoint image associated with the virtual viewpoint image of the target object in a scene different from the target scene. The information processing apparatus according to configuration 8 or 9, characterized by the above. (Composition 11) Each of the aforementioned states includes a virtual viewpoint image associated with a virtual viewpoint image of an object different from the target object in the target scene, viewed from that virtual viewpoint. An information processing apparatus according to any one of the configurations 8 to 10, characterized by the above. (Composition 12) Each of the aforementioned states includes a virtual viewpoint image associated with a different object from the target object in a different scene from the target scene, viewed from the virtual viewpoint. An information processing apparatus according to any one of the configurations 8 to 11, characterized by the above. (Composition 13) It further comprises an imaging unit and a display unit, The acquisition means acquires the captured image containing the figure obtained as a result of the imaging unit imaging the figure, The display control means displays the received virtual viewpoint image on the display unit. An information processing device according to any one of configurations 1 to 12, characterized by the above. (Composition 14) It is possible to communicate with a terminal having an imaging unit and a display unit. The acquisition means receives the captured image containing the figure obtained as a result of the imaging unit imaging the figure from the terminal, The display control means transmits the data of the received virtual viewpoint image to the terminal so that the received virtual viewpoint image is displayed on the display unit. An information processing device according to any one of configurations 1 to 12, characterized by the above. (Composition 15) A first generation means for generating data for sculpting figures, A second generation means for generating multiple different virtual viewpoint images, Processing means for storing the plurality of different virtual viewpoint images in a storage device, associated with each of the plurality of states of the figure, so that when the figure is imaged, a virtual viewpoint image corresponding to the state of the figure is displayed. A figure-making apparatus characterized by having the following features. (Composition 16) The first generation means generates the modeling data such that information for identifying the state of the figure when the figure is imaged is assigned to the figure. The figure-making apparatus according to configuration 15, characterized by the features described above. (Composition 17) Acquisition step to obtain an image containing a figure, A receiving step of receiving a virtual viewpoint image stored in association with the state of the figure included in the captured image from a storage device in which different virtual viewpoint images are associated with each of the multiple states of the figure, A display step which involves processing to display the received virtual viewpoint image, An information processing method characterized by having the following features. (Composition 18) The first generation step generates data for sculpting the figure, A second generation step involves generating multiple different virtual viewpoint images, A processing step of storing the multiple different virtual viewpoint images in a storage device, associated with each of the multiple states of the figure, so that when the figure is photographed, a virtual viewpoint image corresponding to the state of the figure is displayed. A method for manufacturing a figure, characterized by having the following features. (Composition 19) A program for causing a computer to execute each of the means of the information processing device described in any one of configurations 1 to 14. (Composition 20) A program for causing a computer to perform each means of the figure-making apparatus described in either item 15 or 16 of the configuration. [Explanation of Symbols]

[0124] 130 Image acquisition device 801 Image acquisition unit 805 Data Access Section 806 Display Control Unit

Claims

1. A means for acquiring an image containing a figure, A receiving means for receiving a virtual viewpoint image stored in association with the state of the figure included in the captured image from a storage device in which different virtual viewpoint images are associated with each of the multiple states of the figure, A display control means that performs processing for displaying the received virtual viewpoint image, An information processing device characterized by having the following features.

2. The aforementioned multiple states are at least one of the multiple orientations, multiple postures, or multiple positions of the figure. The information processing apparatus according to feature 1.

3. The system further includes a means for identifying the state of the figure included in the captured image, The receiving means receives a virtual viewpoint image recorded in association with the state of the identified figure. The information processing apparatus according to feature 1.

4. The aforementioned figure is provided with first information for identifying the state of the figure, The aforementioned identification means identifies the state of the figure from the first information contained in the captured image. The information processing apparatus according to claim 3.

5. The aforementioned multiple states are multiple orientations of the figure, Each figure is provided with a code image at a position corresponding to its orientation, in which information indicating that orientation is encoded. The aforementioned identification means determines the orientation of the figure by decoding the code image included in the captured image. The information processing apparatus according to claim 3.

6. The figure is provided with second information for accessing the storage device. The information processing apparatus according to feature 1.

7. The aforementioned storage device stores, for each of the multiple figures, a virtual viewpoint image associated with the multiple states. The aforementioned figure is provided with a third piece of information for identifying the figure, The receiving means receives from the storage device a virtual viewpoint image stored in association with the figure identified by the third information and the state of the figure. The information processing apparatus according to claim 3.

8. The aforementioned figure is a figure generated based on the three-dimensional shape data of the target object in the target scene, which is among the three-dimensional shape data for each frame generated based on imaging of the target area by multiple imaging devices. Each of the aforementioned multiple states includes a virtual viewpoint image associated with it, which is generated based on the three-dimensional shape data for each frame and shows a predetermined object in a predetermined scene viewed from the virtual viewpoint. The information processing apparatus according to feature 1.

9. Each of the aforementioned states is associated with a virtual viewpoint image that includes a virtual viewpoint image of the target object in the target scene as seen from the virtual viewpoint. The information processing apparatus according to feature 8.

10. Each of the aforementioned states includes a virtual viewpoint image associated with the virtual viewpoint image of the target object in a scene different from the target scene. The information processing apparatus according to feature 8.

11. Each of the aforementioned states includes a virtual viewpoint image associated with a virtual viewpoint image of an object different from the target object in the target scene, viewed from that virtual viewpoint. The information processing apparatus according to feature 8.

12. Each of the aforementioned states includes a virtual viewpoint image associated with a different object from the target object in a different scene from the target scene, viewed from the virtual viewpoint. The information processing apparatus according to feature 8.

13. It further comprises an imaging unit and a display unit, The acquisition means acquires the captured image containing the figure obtained as a result of the imaging unit imaging the figure, The display control means displays the received virtual viewpoint image on the display unit. The information processing apparatus according to feature 1.

14. It is possible to communicate with a terminal having an imaging unit and a display unit. The acquisition means receives the captured image containing the figure obtained as a result of the imaging unit imaging the figure from the terminal, The display control means transmits the data of the received virtual viewpoint image to the terminal so that the received virtual viewpoint image is displayed on the display unit. The information processing apparatus according to feature 1.

15. A first generation means for generating data for sculpting figures, A second generation means for generating multiple different virtual viewpoint images, Processing means for storing the plurality of different virtual viewpoint images in a storage device, associated with each of the plurality of states of the figure, so that when the figure is imaged, a virtual viewpoint image corresponding to the state of the figure is displayed. A figure-making apparatus characterized by having the following features.

16. The first generation means generates the modeling data such that information for identifying the state of the figure when the figure is imaged is assigned to the figure. The figure manufacturing apparatus according to feature 15.

17. Acquisition step to obtain an image containing a figure, A receiving step of receiving a virtual viewpoint image stored in association with the state of the figure included in the captured image from a storage device in which different virtual viewpoint images are associated with each of the multiple states of the figure, A display step which involves processing to display the received virtual viewpoint image, An information processing method characterized by having the following features.

18. A first generation step to generate data for sculpting a figure, A second generation step involves generating multiple different virtual viewpoint images, A processing step of storing the multiple different virtual viewpoint images in a storage device, associated with each of the multiple states of the figure, so that when the figure is photographed, a virtual viewpoint image corresponding to the state of the figure is displayed. A method for manufacturing a figure, characterized by having the following features.

19. A program for causing a computer to execute each of the means of the information processing apparatus described in any one of claims 1 to 14.

20. A program for causing a computer to perform each means of the figure-making apparatus described in any one of claims 15 or 16.

Images