Information processing device, information processing system, information processing method, and program

Abstract

This makes it easier for users to select an image of an object to be composited into the captured image from a wider variety of options. [Solution] Identification information is detected from the medium to identify 3D model data representing 3D models of objects corresponding to each of multiple time codes. User input is accepted to specify the time code to be displayed. An image of the object based on the 3D model of the object corresponding to the time code according to the identification information and user input is displayed on the display unit. A virtual image, which is composited with the captured image and includes the image of the object displayed on the display unit, is recorded according to the user input.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing system, an information processing method, and a program, and particularly relates to AR technology.

Background Art

[0002] Techniques for synthesizing an image of a virtual object into a captured image are known. Such an image of a virtual object is also known as an AR frame. For example, a user can select an AR frame on a terminal and synthesize the selected AR frame with a desired captured image.

[0003] Techniques for controlling the position and shape of a virtual object are also known. For example, Patent Document 1 discloses arranging a virtual object at a position where a landmark exists in a captured image. Patent Document 1 also discloses performing a projective transformation on a virtual object that is a 3D image based on the shape of an image of the landmark on the captured image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] According to the technique described in Patent Document 1, the position and shape of a virtual object are automatically determined. On the other hand, a user may desire to select an object to be synthesized into a captured image according to their own wishes.

[0006] The technique according to the present disclosure can facilitate a user to select an image of an object to be synthesized into a captured image from a greater variety of variations.

Means for Solving the Problems

[0007] An information processing apparatus according to one embodiment of this disclosure has the following configuration. That is, A detection means for detecting identification information from a medium to identify 3D model data representing 3D models of objects corresponding to each of multiple time codes, A means for receiving user input to specify the timecode to be displayed, Display control means for displaying an image of the object based on a 3D model of the object corresponding to the identification information and the time code according to the user operation on the display unit, A recording means for recording a virtual image that is composited with an captured image, including an image of the object displayed on the display unit, in accordance with user operation, It is equipped with. [Effects of the Invention]

[0008] This makes it easier for users to select an image of an object to be composited into the captured image from a wider variety of options. [Brief explanation of the drawing]

[0009] [Figure 1] A diagram showing an example configuration of an information processing system according to one embodiment. [Figure 2] A diagram showing an example of the hardware configuration of an information processing device according to one embodiment. [Figure 3] A diagram showing an example of the functional configuration of an information processing device according to one embodiment. [Figure 4] A diagram illustrating an example of a user interface. [Figure 5] A diagram showing a flowchart of an information processing method according to one embodiment. [Figure 6] A diagram illustrating an example of a user interface. [Figure 7] A diagram showing a flowchart of an information processing method according to one embodiment. [Modes for carrying out the invention]

[0010] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the scope of the claims. While the embodiments describe multiple features, not all of these features are necessarily essential, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0011] (System Configuration) An information processing system according to one embodiment will be described with reference to Figure 1. Figure 1 shows an example of the configuration of the information processing system according to one embodiment. This information processing system has a terminal 10 and a server 20.

[0012] Terminal 10 is an information processing device operated by the user to display an image of an object. Server 20 is an information processing device that generates an image of an object. In this embodiment, Server 20 stores 3D model data of the object, which is used to generate the image of the object.

[0013] A 3D model can represent the three-dimensional shape of an object. Furthermore, a 3D model can represent the color at each point in the object's three-dimensional shape. In this way, a 3D model can represent the appearance of an object.

[0014] In this embodiment, the 3D model data includes a 3D model of an object corresponding to each of several time codes. Such 3D model data can represent the three-dimensional shape of an object that changes over time. A time code is information indicating the time associated with the 3D model of an object.

[0015] The types of objects and 3D model data are not particularly limited. In one embodiment, the 3D model data is 3D model data of a subject generated using volumetric capture technology. Such 3D model data of a subject can be generated using captured images of the subject from multiple viewpoints. In this case, the object can represent a real subject.

[0016] FIG. 1 shows a configuration example of an imaging system 30 that generates 3D model data of a subject. The imaging system 30 includes a generation device 31 and a plurality of imaging devices 32. The plurality of imaging devices 32 are a plurality of cameras installed so as to image the subject from different directions. The plurality of imaging devices 32 can perform imaging a plurality of times synchronously during an imaging period. Thus, the plurality of imaging devices 32 can generate a group of captured images of the subject from different viewpoints at each of a plurality of times. During the imaging period, the position and shape of the subject can change.

[0017] The generation device 31 generates 3D model data of the subject using the captured images obtained by the plurality of imaging devices 32. The method for generating the 3D model data is not particularly limited. The generation device 31 can generate 3D model data of the subject based on, for example, the visual volume intersection method or the PhotoHull method. As a specific example, the generation device 31 can extract the subject region from each of the captured image groups of the subject from different viewpoints obtained by synchronous imaging at a certain time. In order to extract the subject region, for example, the background difference method can be used. Then, the generation device 31 can estimate the three-dimensional shape of the subject based on the extraction result of the subject region and the camera parameters of each of the plurality of imaging devices 32. Further, the generation device 31 can generate a texture to be applied to the 3D model of the subject based on the captured image group of the subject. Thus, the generation device 31 can generate a 3D model of the subject at a certain time. By performing such processing using the captured image groups at each of a plurality of times, the generation device 31 can generate 3D models of the subject at each of the plurality of times. In this case, the time code corresponds to the imaging time.

[0018] On the other hand, the object may be a virtual object. For example, the object may represent an anime character or a virtual idol. The 3D model data of such an object can be generated using a 3DCG creation device.

[0019] The terminal 10 is connected to the server 20 via the network 40. As shown in FIG. 1, a plurality of terminals 10 can be connected to the server 20. The type of the network 40 is not particularly limited. The network 40 can be, for example, the Internet or an intranet. Also, the network 40 can be a wireless network or a wired network.

[0020] Next, an example of the hardware configuration of a terminal 10, which is an information processing device according to one embodiment, and a server 20, which is an information processing device according to one embodiment, will be described with reference to Figure 2. The terminal 10 and the server 20 can be implemented using a computer. Examples of computers include general-purpose desktop computers, laptop computers, tablet PCs, or smartphones. Note that Figures 2 and 3 are merely examples of information processing devices. For example, the information processing device according to one embodiment may be composed of multiple information processing devices connected via a network.

[0021] As shown in Figure 2, the terminal 10 includes a processor 11, memory 12, storage medium 13, input interface 14, output interface 15, communication unit 16, display unit 17, imaging unit 18, and bus 19. The processor 11 is, for example, a CPU, which controls the operation of the entire computer. The memory 12 is, for example, RAM, which temporarily stores programs and data. The computer-readable storage medium 13 is, for example, a hard disk or CD-ROM, which stores programs and data long-term. In this embodiment, the programs that realize the functions of each part shown in Figure 3, which are stored in the storage medium 13, are read into the memory 12. Then, the processor 11 operates according to the programs in the memory 12, thereby realizing the functions of each part.

[0022] The input interface 14 is an interface for acquiring information. For example, the input interface 14 may be connected to an input device that accepts user input, such as a keyboard, mouse, or joystick. The output interface 15 is an interface for outputting information. For example, the output interface 15 may be connected to an output device such as an external display. The communication unit 16 is an interface for connecting to a network. The display unit 17 is a screen that can display information. The display unit 17 can display a graphical user interface (GUI) for the user to operate the system. The display unit 17 is, for example, a liquid crystal display or a touch panel. The imaging unit 18 generates an image by taking an image. The imaging unit 18 is, for example, a camera. The bus 19 connects the above-mentioned units and enables data exchange.

[0023] Server 20 includes a processor 21, memory 22, storage medium 23, input interface 24, output interface 25, communication unit 26, and bus 27. The functions of each of these components are the same as those of terminal 10.

[0024] Next, an example of the functional configuration of terminal 10 will be described with reference to Figure 3. Terminal 10 has a detection unit 110, a reception unit 120, a transmission unit 130, a receiving unit 140, a display control unit 160, a recording unit 170, and a synthesis unit 180.

[0025] The detection unit 110 acquires identification information to identify the 3D model data. In this embodiment, the identification information is attached to a medium. The medium has the function of transmitting the identification information, and its type is not particularly limited. For example, the medium may be a flat object such as paper, or a three-dimensional object. Examples of mediums include printed materials such as business cards, letters, advertisements, or booklets. Another example of a medium is a character figurine. Examples of character figurine include resin plates with the character printed on them (e.g., acrylic stands), character figurines, or stuffed animals.

[0026] Identification information can be information that uniquely identifies 3D model data. For example, identification information may be a Uniform Resource Identifier (URI). Alternatively, identification information may be the file name of a specific 3D model data stored by server 20. Furthermore, identification information may indicate the location of 3D model data stored on a device other than server 20. As will be described later, the generation unit 220 of server 20 can acquire 3D model data corresponding to the identification information. On the other hand, identification information may also be an identifier such as the ID of the 3D model data. In this case, server 20 can acquire 3D model data corresponding to the identification information by referring to a database. Such a database can manage information indicating the location of 3D model data in association with the identification information.

[0027] In one embodiment, the detection unit 110 detects identification information based on an image of the medium. The image of the medium may be, for example, an image of the medium obtained using the imaging unit 18. In such an embodiment, the identification information may be printed on the medium as a code. Specific examples of the code include barcodes and QR codes (registered trademarks) that encode the identification information.

[0028] On the other hand, the detection unit 110 may detect identification information by image recognition processing on the image of the medium. Image recognition processing may be, for example, image identification processing that recognizes the type of subject. For example, if a picture of a character is printed on the medium, the detection unit 110 can identify the character by image recognition processing. Furthermore, the detection unit 110 can determine, by image recognition processing, a specific variation of the character's picture that is printed on the medium. The detection unit 110 can perform such image recognition processing using, for example, a trained neural network. In such an example, the identification information may be an ID representing the type of subject. In this case, the server 20 can refer to the database and obtain 3D model data corresponding to the ID.

[0029] Furthermore, the identification information may be the feature quantities of the subject (e.g., feature vectors). The detection unit 110 can detect such identification information by performing a feature extraction process on the image of the medium. Moreover, the identification information may be the image of the medium itself. The server 20 can recognize the type of subject by performing image recognition processing or identification processing using this identification information. The server 20 can also obtain 3D model data corresponding to the recognized subject by referring to a database. Therefore, such identification information can also be used as information to identify the 3D model data. Thus, detecting the identification result from the medium includes capturing an image.

[0030] In another embodiment, the detection unit 110 detects identification information based on information transmitted by the medium. For example, the medium may have an information transmission circuit such as RFID. In this case, the detection unit 110 can obtain the identification information transmitted from the information transmission circuit of the medium via the communication unit 16.

[0031] The reception unit 120 receives user input to specify the time code to be displayed. As described above, 3D model data can represent the three-dimensional shape of an object that changes over time. In this embodiment, based on user input, the three-dimensional shape of the object corresponding to a specific time code is displayed on the display unit 17. In this way, in this embodiment, the user can select a desired object from among the objects that change over time.

[0032] Furthermore, the reception unit 120 can accept user operations specifying at least one of the following: the object's position, the object's orientation, and the object's size. For example, the reception unit 120 may accept user operations specifying the object's orientation. The user operation specifying the object's orientation may also be a user operation specifying the viewpoint relative to the object's 3D model. The reception unit 120 may also accept user operations specifying the object's position. The reception unit 120 may also accept user operations specifying the object's size. The user can change the display manner of the object on the image through these user operations. The process for accepting user operations will be described later with reference to Figures 4(A) and (B).

[0033] The types of user operations and the methods for acquiring user operations are not particularly limited. For example, the reception unit 120 can receive various user operations via the input interface 14. In this embodiment, the display unit 17 is a touch panel, and the reception unit 120 can receive user operations on the contact-sensing display.

[0034] The transmitting unit 130 transmits the identification information and time code received by the receiving unit 120 to the server 20 via the communication unit 16. The transmitting unit 130 may also transmit to the server 20 information used when rendering other objects. Such information may include information indicating the display mode of an object according to user operations, such as information specifying the viewpoint for the object's 3D model.

[0035] The receiving unit 140 receives an image of an object transmitted from the server 20 via the communication unit 16. As will be described later, the image of the object is generated by the server 20 according to the identification information and time code transmitted to the server 20. In other words, the image of the object received by the receiving unit 140 is an image of the object based on the 3D model data of the object corresponding to the identification information and the time code according to the user operation.

[0036] The display control unit 160 causes the object image to be displayed on the display unit 17. For example, the display control unit 160 can display the object image received by the receiving unit 140 on the display unit 17.

[0037] Furthermore, the display control unit 160 can display a user interface on the display unit 17 for specifying at least one of the following: time code, object position, object orientation, and object size. For example, the display control unit 160 can display a user interface on the display unit 17 for specifying a time code. Figure 4(A) shows an example of such a user interface. The screen 400 displayed on the display unit 17 includes an object 410, a time bar 430, and a button 440.

[0038] Object 410 is an image of an object generated by the server 20 and received by the receiving unit 140. Object 410 is generated according to user operations received by the reception unit 120. For example, a user can specify one time code from among several time codes by operating the time bar 430. The user may also specify a time code by touching or sliding on the time bar 430. When the user changes the time code to be displayed by operating the time bar 430, the transmission unit 130 sends identification information and the changed time code to the server 20. The receiving unit 140 also acquires an image of the object corresponding to the changed time code. The display control unit 160 then updates the object 410 displayed on the screen 400 with the image of the object acquired by the receiving unit 140. In this way, the user can determine the desired time code while viewing the object corresponding to the specified time code.

[0039] Furthermore, the user can perform other operations on the user interface displayed on the display unit 17 to change the display mode of the object. For example, as described above, the reception unit 120 can receive user operations to specify the orientation of the object. The reception unit 120 can also receive user operations to specify the viewpoint of the object. For example, the user can perform a flick or rotate operation on the object 410 to rotate the object. Rotating the object is equivalent to changing the viewpoint of the 3D model used when rendering the object. In this case, as described above, the transmission unit 130 can send information to the server 20 specifying the viewpoint of the 3D model of the object according to the user operation. The server 20 can then generate an image of the object corresponding to the specified time code from the specified starting point. In this case as well, the display control unit 160 can update the object 410 displayed on the screen 400 with the image of the object generated and transmitted by the server 20. In this way, the display control unit 160 can display an image of the object from the specified viewpoint on the display unit 17.

[0040] Furthermore, the user can perform operations to specify the position of an object. For example, the user can perform a drag operation (or a long-press followed by a drag operation) from object 410 to translate the object. In this case, the display control unit 160 can display the image of the object generated and transmitted by the server 20 at a position on the screen 400 according to the user operation. The transmission unit 130 may also transmit information specifying the position of the object to the server 20. In this case, the server 20 can render the 3D model so that the object is displayed at the specified position.

[0041] Furthermore, the user can perform operations to specify the size of the object. For example, the user can perform pinch-out or pinch operations on object 410 to enlarge or reduce the object. In this case, the display control unit 160 can perform enlargement or reduction processing on the image of the object generated and transmitted by the server 20 according to the user operation. The display control unit 160 can then display the enlarged or reduced image of the object on screen 400. The transmission unit 130 may also transmit information to the server 20 specifying the size of the object. For example, the transmission unit 130 may transmit information specifying the viewpoint so that the distance from the object 3D model to the viewpoint is closer or further. In this case, the server 20 can render the 3D model so that the object is displayed at the specified size.

[0042] When the user determines that the image of the desired object has been displayed on screen 400 as a result of their user operation, they can press button 440. At this time, the recording unit 170 can record the image of the object, as shown below.

[0043] The recording unit 170 records a virtual image, which is composited with the captured image, including an image of an object displayed on the display unit 17, according to user operation. The data format of the virtual image is not particularly limited. The recording unit 170 can record the virtual image in memory within the terminal 10, such as a storage medium 13. Alternatively, the recording unit 170 may record the virtual image in a device other than the terminal 10, such as a server 20.

[0044] In one embodiment, the recording unit 170 records the image of the object generated by the server 20 and received by the receiving unit 140 as a virtual image. On the other hand, as described above, the display control unit 160 can perform processing to change the display characteristics of the object, such as its position or size. In this case, the recording unit 170 can generate and record a virtual image that includes the image of the object whose display characteristics have been changed according to user operation.

[0045] The synthesis unit 180 generates a composite image of the virtual image recorded by the recording unit 170 and the captured image. The synthesis unit 180 can generate a composite image of a virtual image selected by the user from among multiple virtual images recorded by the recording unit 170 and the captured image. For this purpose, the display control unit 160 can display one or more virtual images in a list on the display unit 17. For example, the display control unit 160 can display a thumbnail of each virtual image recorded by the recording unit 170 on the display unit 17. The reception unit 120 can also receive user operations to select a virtual image from the list of virtual images.

[0046] Furthermore, in this embodiment, the captured image is an image captured in real time by the imaging unit 18. The synthesis unit 180 may synthesize each of the captured images sequentially obtained by the imaging unit 18 with a common virtual image. In this case, the synthesis unit 180 can sequentially generate a composite image of the captured image and the virtual image from the imaging unit 18 in real time. Alternatively, the captured image may be an image stored in the terminal 10.

[0047] The image synthesis unit 180 can further record the synthesized image. The image synthesis unit 180 can record the synthesized image in the memory of the terminal 10, for example, in the storage medium 13. Alternatively, the image synthesis unit 180 may record the synthesized image in a device other than the terminal 10, such as the server 20. The image synthesis unit 180 may also record the synthesized image according to user instructions. For example, the image synthesis unit 180 may record only the synthesized image selected by the user from among multiple synthesized images. Furthermore, when the image synthesis unit 180 sequentially generates a synthesized image of an captured image and a virtual image, the image synthesis unit 180 can record the latest synthesized image at a timing specified by the user. For example, the reception unit 120 can receive a user operation to record a synthesized image. The user operation to record a synthesized image may be pressing the button 460 shown in Figure 4(B). Button 460 corresponds to the shutter button.

[0048] In one embodiment, the virtual image has a foreground region, which is the area of ​​the object's image, and a transparent region. When such a virtual image is combined with an captured image, the object's image is superimposed on the captured image in the foreground region, and the captured image is maintained in the transparent region. That is, in the combined image, the area corresponding to the foreground region shows the object's image, and the area corresponding to the transparent region shows the captured image. Figure 4(B) shows an example of a screen 450 displaying the combined image thus generated. Screen 450 shows an object 410 and an captured image 420.

[0049] Here, the recording unit 170 can record information indicating the image region of an object in the virtual image. For example, the virtual image data recorded by the recording unit 170 may include the pixel values ​​of each pixel in the foreground region indicating the color information of the object's image, and the pixel values ​​of each pixel in the transparent region indicating the color information corresponding to the transparent color. In another embodiment, the recording unit 170 may record metadata indicating the image region of an object in the virtual image, associated with the virtual image.

[0050] In one embodiment, the virtual image has the same aspect ratio as the image captured by the imaging unit 18, in order to facilitate the synthesis of the virtual image and the captured image. On the other hand, the virtual image may be represented by image data that includes shape information indicating the two-dimensional shape of the object, position information indicating the two-dimensional position of the object, and color information of the object at each pixel.

[0051] Next, an example of the functional configuration of server 20 will be described with reference to Figure 3. Server 20 has a receiving unit 210, a generating unit 220, and a transmitting unit 230.

[0052] The receiving unit 210 receives identification information and timecode transmitted from the terminal 10 as described above via the communication unit 26. The receiving unit 210 can also receive information specifying the viewpoint for an object, or other information used when rendering an object, as described above.

[0053] The generation unit 220 generates an image of an object based on the 3D model data of the object corresponding to the identification information transmitted from the terminal. As described above, the generation unit 220 can acquire 3D model data according to the identification information. The generation unit 220 can also identify the 3D model of the object to be displayed according to the specified time code. The generation unit 220 can then render an image (virtual viewpoint image) of the 3D model of the object to be displayed from a viewpoint specified using the terminal 10, or from a predetermined viewpoint. The rendering method is not particularly limited, and for example, ray tracing can be used.

[0054] The format of the object image generated by the generation unit 220 is not particularly limited. For example, the generation unit 220 may generate image data having a foreground region, which is the area of ​​the object's image, and a transparent region. Here, the foreground region may correspond to the area in which the 3D model of the object is captured. The transparent region may correspond to the area in which the 3D model of the object is not captured. The generation unit 220 may also generate image data including shape information indicating the two-dimensional shape of the rendered object and color information of the object at each pixel. This image data may also include position information indicating the position of the object on the screen.

[0055] Furthermore, the image of the object generated by the generation unit 220 may have a background. For example, the generation unit 220 may generate an image of the object by rendering a 3D model of the object corresponding to a specified time code and then compositing a background onto the rendering result. The background may be common to multiple time codes. Such background data may be included in the 3D model data. In this case, the generation unit 220 may generate image data having a foreground region which is the area of ​​the object image and the background region, and a transparent region.

[0056] The transmitting unit 230 transmits the image of the object generated by the generation unit 220 to the terminal 10 via the communication unit 26.

[0057] Next, the information processing method according to one embodiment will be described with reference to the flowchart in Figure 5, which shows the operation of terminal 10 and server 20 according to one embodiment. A virtual image is recorded according to user operation through the operation described below.

[0058] In S510, the detection unit 110 acquires identification information as described above. In S520, the reception unit 120 transmits the identification information and time code to the server 20. At this time, the reception unit 120 may also transmit information specifying the viewpoint of the 3D model to the server 20. Note that in the first S520, the reception unit 120 may transmit information specifying the default time code and / or default viewpoint to the server 20. The default time code may be, for example, the time code corresponding to the start time. The default viewpoint may be set to face the 3D model from a position in front of the 3D model, at a predetermined distance from the 3D model.

[0059] In S530, the generation unit 220 generates an image of the 3D model of the object as described above, according to the identification information and time code transmitted from the terminal 10 and received by the receiving unit 210. In S540, the transmission unit 230 transmits the image of the object generated by the generation unit in S530 to the terminal 10.

[0060] In S550, the display control unit 160 displays the image of the object transmitted from the server 20 and received by the receiving unit 140 on the display unit 17. In S560, the display control unit 160 determines whether the user operation has ended. For example, if button 440 is pressed, the display control unit 160 can determine that the user operation has ended. If it is determined that the user operation has ended, the process proceeds to S580. Otherwise, the process proceeds to S570.

[0061] In S570, the reception unit 120 accepts user operations as described above. For example, the reception unit 120 can accept user operations such as specifying a time code or specifying a viewpoint. After that, processing returns to S520. In S520, the time code, viewpoint, or other information used when rendering the object specified in S570 is sent to the terminal 10.

[0062] In S580, the recording unit 170 records a virtual image that is composited with the captured image, as described above.

[0063] Next, the information processing method according to one embodiment will be described with reference to the flowchart in Figure 7, which shows the operation of terminal 10 according to one embodiment. A composite image of a virtual image and an captured image is generated by the following operations.

[0064] In S710, the display control unit 160 displays a list of virtual images recorded by the recording unit 170 on the display unit 17, as described above. In S720, the synthesis unit 180 selects a virtual image to be synthesized with the captured image from the virtual images recorded by the recording unit 170. The synthesis unit 180 can select a virtual image according to the user operation received by the reception unit 120, as described above.

[0065] In S730, the synthesis unit 180 acquires the captured image obtained by the imaging unit 18. The synthesis unit 180 can acquire the captured image obtained by the imaging unit 18 in real time. In S740, the synthesis unit 180 generates a composite image by combining the virtual image selected in S720 and the captured image acquired in S730 as described above. In S750, the display control unit 160 displays the composite image generated in S740 on the display unit 17. For example, the display control unit 160 can display the screen 450 shown in Figure 4(B) on the display unit 17.

[0066] In S760, the synthesis unit 180 determines whether a user operation to record a composite image has been performed, as described above. If it is determined that a user operation to record a composite image has been performed, the process proceeds to S770. Otherwise, the process returns to S730, and a composite image of another captured image and a virtual image is generated. In S770, the synthesis unit 180 records the composite image generated in S740.

[0067] Thus, according to this embodiment, the identification information detected from the medium is associated with 3D model data of objects corresponding to each of multiple time codes. Furthermore, the user of terminal 10 can select the desired object while viewing the image of the object corresponding to the specified time code on the display unit. This makes it easier for the user to select an object image from a wider variety of options.

[0068] Furthermore, as described above, terminal 10 displays an object corresponding to the identification information detected from the medium and records a virtual image containing this object. With this configuration, the owner of the medium can generate a composite image containing the object associated with the medium, thereby increasing the value of the medium. In addition, terminal 10 can automatically display an image of the object corresponding to the medium based on the detection result of the identification information. Therefore, according to this embodiment, the user's operational burden for generating a composite image can be reduced.

[0069] (modified version) In the embodiment described above, the user could specify the position of the object to be displayed. Alternatively, the display control unit 160 may display the object at the position of the medium in the captured image. For example, the detection unit 110 can detect the medium (or a code attached to the medium) from the captured image through image recognition processing. The synthesis unit 180 can generate a composite image of the captured image and the object image such that the object image is superimposed at the position of the medium detected in the captured image. The display control unit 160 can then display the composite image generated by the synthesis unit 180 on the display unit 17. With such a configuration, objects related to the medium can be displayed in the vicinity of the medium. For example, an object image of the person described on a business card can be displayed in the vicinity of a business card on which a code is printed. Also, an object image representing a character can be displayed superimposed on an acrylic stand on which a character is printed.

[0070] The orientation or size of an object displayed on the display unit 17, or the method for specifying the viewpoint for the 3D model of an object, is not limited to the methods described above. For example, a user operation to specify a viewpoint may include an operation to change the physical position and orientation of the terminal 10. For example, the viewpoint for the 3D model of an object may be a viewpoint that follows the position and orientation of the terminal 10.

[0071] In one embodiment, the viewpoint of an object is indicated by the physical position and orientation of the information processing device relative to the medium. For example, when displaying an image of a 3D model of an object from above, the user can move the terminal 10 above the medium and control the orientation of the terminal 10 so that the optical axis of the imaging unit 18 faces the medium. According to such an embodiment, the user can set the viewpoint of an object through intuitive operation. The position and orientation of the terminal 10 relative to the medium can be determined, for example, based on the image of the medium or code in the captured image obtained by the imaging unit 18.

[0072] In the embodiment described above, an image of one object was displayed on the display unit 17. However, images of multiple objects may be displayed on the display unit 17. In such an embodiment, the identification information can identify the 3D model data of each of the multiple objects. For example, multiple codes may be printed on the medium, and each of the multiple codes may indicate the location of the object's 3D model data. Alternatively, a database referenced by the server 20 may indicate the location of the 3D model data of each of the multiple objects corresponding to the identification information.

[0073] In this case, the reception unit 120 can independently accept user input to specify the time code to be displayed for each of the multiple objects. The generation unit 220 can generate images for each of the multiple objects based on the 3D model data of the object corresponding to the time code according to the user input. The display control unit 160 can then display the images of each of the multiple objects generated by the generation unit 220 on the display unit 17. The recording unit 170 can record a virtual image containing the images of the multiple objects displayed on the display unit 17.

[0074] Figure 6 shows an example of a screen 400 providing a user interface as displayed in this embodiment. Objects 611 to 613 are displayed on screen 400. In addition, time bars 631 to 633, which specify the time codes of objects 611 to 613, are displayed on screen 400 in association with objects 611 to 613. In this example, each of the time bars 631 to 633 is displayed directly below objects 611 to 613. The user can independently specify the time codes of the objects 611 to 613 to be displayed by manipulating each of the time bars 631 to 633.

[0075] Furthermore, the reception unit 120 may accept user operations to change the display mode of multiple objects independently. For example, the reception unit 120 may accept user operations that specify at least one of the following: the position of an object, the orientation of an object, and the size of an object. Specifically, a user can perform a flick or rotation operation on object 612 in order to rotate only object 612. Alternatively, a user can perform a drag operation (or a long-press followed by a drag operation) from object 613 in order to translate only object 613.

[0076] In this case, the recording unit 170 can generate and record a virtual image containing images of multiple objects, each according to an independently specified time code and / or independently modified display mode.

[0077] In the embodiment described above, the compositing unit 180 combined a virtual image, which is a still image, with an captured image, which is a still image. However, at least one of the virtual image and the captured image may be a moving image. For example, the compositing unit 180 may combine the same virtual image with each frame of a moving image captured by the imaging unit 18. The user may specify the shooting period of the moving image to be combined. For example, the user can specify the start point of the shooting period by pressing button 460, and specify the end point of the shooting period by pressing button 460 again. In this case, the compositing unit 180 can generate multiple combined images by combining a virtual image with each captured image obtained by the imaging unit 18 repeatedly taking images within the shooting period. The compositing unit 180 can then record a moving image composed of multiple combined images.

[0078] Furthermore, the synthesis unit 180 may generate a composite video of the video recorded by the recording unit 170 and the captured image. In this case, the reception unit 120 can receive a user operation to specify a timecode range. The generation unit 220 can then generate an image of the object corresponding to each timecode based on the 3D model data of the object corresponding to each timecode included in the timecode range specified by the user operation. That is, the generation unit 220 can generate a video of an object based on the 3D model of the object corresponding to the timecode range specified by the user operation. The recording unit 170 can also record a video composed of multiple virtual images corresponding to each of the multiple timecodes. Here, each of the multiple virtual images contains an image of the object corresponding to each timecode. That is, the recording unit 170 can record a video of an object based on the 3D model of the object corresponding to the timecode range specified by the user operation. The display control unit 160 may sequentially display the images of the object corresponding to each timecode generated by the generation unit 220 on the display unit 17. The user can specify a timecode range while viewing the image of the object displayed on the display unit 17.

[0079] The combining unit 180 can generate a composite video of the video recorded by the recording unit 170 and the captured video. For example, the combining unit 180 may combine each frame of the video captured by the imaging unit 18 with a virtual image corresponding to each of the multiple time codes, frame by frame. The display control unit 160 can also display the composite video generated by the combining unit 180 on the display unit 17.

[0080] Furthermore, when the generation unit 220 generates a video of an object, the viewpoint may change with each timecode. In this case, the reception unit 120 can accept user input specifying the movement of the viewpoint relative to the object. This information indicating the viewpoint that changes over time is called the camera path. In one embodiment, the user can specify such a camera path by changing the physical position and orientation of the terminal 10. For example, the viewpoint relative to the object in each timecode may correspond to the position and orientation of the terminal 10 at each time. The generation unit 220 can then generate a video of the object from a viewpoint that moves according to the user input. The recording unit 170 can also record such a video. According to this embodiment, the user can specify the camera path through intuitive operation.

[0081] The method of specifying time codes is not limited to the examples described above. For example, a video of an object may be played on screen 400. That is, the display control unit 160 may sequentially display the images of objects corresponding to each time code, generated by the generation unit 220, on the display unit 17. When the image of the desired object is displayed, the user can stop the playback of the video. This operation is equivalent to the user specifying a desired time code.

[0082] The method for generating the composite image by the compositing unit 180 is not particularly limited. In the above embodiment, the compositing unit 180 superimposed a virtual image having a transparent region onto the captured image. On the other hand, a depth value may be set for each pixel of the virtual image. The generation unit 220 can set such depth values ​​based on the distance between the viewpoint and the 3D model. Alternatively, a depth value may be set for each pixel of the captured image. The depth value of each pixel of the captured image may be, for example, a constant predetermined value, or it may be determined according to the distance to the subject. In this case, the compositing unit 180 may combine the captured image and the virtual image based on the depth value of each pixel. In this case, for each pixel, an image with a smaller depth value can be superimposed onto the other image. The generation unit 220 may also combine the captured image and the virtual image using techniques such as alpha blending.

[0083] Furthermore, the synthesis unit 180 may separate the captured image into foreground and background using machine learning techniques or background subtraction methods. The synthesis unit 180 may then superimpose the captured image and the object image such that a virtual image is superimposed on the background of the captured image, and the foreground of the captured image is superimposed on the virtual image.

[0084] The display control unit 160 may display a composite image of the image captured by the imaging unit 18 and the image of the object received by the receiving unit 140 on a user interface for changing the display mode of an object. For example, a composite image of the object 410 and the image captured 420 may be displayed on a user interface for receiving user operations, such as screen 400. Such a composite image can be generated by the synthesis unit 180. In this case, the display control unit 160 can update the displayed composite image in real time based on real-time imaging by the imaging unit 18.

[0085] In the embodiment described above, the detection unit 110 detected identification information from the medium. However, in another embodiment, 3D model data of an object corresponding to each of a plurality of time codes is selected according to user operation. For example, the user may be able to select a desired 3D model data from among a plurality of 3D model data on the terminal 10. In such a configuration as well, the user can easily select an image of a desired object from among many variations corresponding to each of a plurality of time codes.

[0086] In the embodiment described above, the server 20 generated an image of an object. However, the terminal 10 may have at least some of the functions of the server 20. That is, the terminal 10 may store 3D model data or be able to access 3D model data. The server 20 may also receive identification information and transmit 3D model data corresponding to the identification information to the terminal 10. In this case, the terminal 10 can generate an image of an object in the same way as the generation unit 220. Conversely, the server 20 may have at least some of the functions of the terminal 10. Furthermore, the information processing device according to one embodiment may be realized by a combination of the terminal 10 and the server 20.

[0087] In the embodiment described above, the composite unit 180 of the terminal 10 generated the composite image. However, it is not essential that the terminal 10 generates the composite image. For example, the server 20 or other information processing device may generate the composite image using a virtual image recorded by the recording unit 170 of the terminal 10.

[0088] (Other examples) The contents of this disclosure can also be realized by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by a process in which one or more processors in the computer of that system or device read and execute the program. Furthermore, it can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0089] The disclosures herein include the following information processing devices, information processing systems, information processing methods, and programs. (Item 1) A detection means for detecting identification information from a medium to identify 3D model data representing 3D models of objects corresponding to each of multiple time codes, A means for receiving user input to specify the timecode to be displayed, Display control means for displaying an image of the object based on a 3D model of the object corresponding to the identification information and the time code according to the user operation on the display unit, A recording means for recording a virtual image that is composited with an captured image, including an image of the object displayed on the display unit, in accordance with user operation, An information processing device characterized by comprising: (Item 2) The information processing apparatus according to item 1, wherein the receiving means further receives a user operation specifying at least one of the position of the object, the orientation of the object, and the size of the object. (Item 3) The information processing apparatus according to item 2, characterized in that the display control means causes the display unit to display a user interface for specifying at least one of the time code, the position of the object, the orientation of the object, and the size of the object. (Item 4) The receiving means further receives user input specifying the viewpoint for the object, The information processing device according to any one of items 1 to 3, characterized in that the display control means causes the display unit to display an image of the object from the viewpoint. (Item 5) The information processing apparatus according to item 4, characterized in that the viewpoint with respect to the object is indicated by the physical position and orientation of the information processing apparatus with respect to the medium. (Item 6) The information processing apparatus according to any one of items 1 to 5, wherein the detection means detects the identification information based on the image of the medium in the captured image. (Item 7) The information processing device according to item 6, characterized in that the identification information is printed as a code on the medium. (Item 8) The information processing device according to any one of items 6 to 7, characterized in that the medium is a printed document. (Item 9) The information processing device according to any one of items 6 to 7, characterized in that the medium is a character sculpture. (Item 10) The aforementioned identification information identifies the 3D model data of each of the multiple objects, The receiving means independently receives a user operation for each of the multiple objects to specify the time code to be displayed. The information processing device according to any one of items 1 to 9, characterized in that the display control means causes the display unit to display images of each of the plurality of objects, and each image of the plurality of objects is based on a 3D model of the object corresponding to a time code according to the user operation. (Item 11) The information processing apparatus according to item 10, wherein the receiving means further receives a user operation in which, for each of the plurality of objects, the user specifies at least one of the position of the object, the orientation of the object, and the size of the object. (Item 12) The aforementioned identification information identifies the 3D model data of each of the multiple objects, The information processing apparatus according to any one of items 1 to 11, characterized in that the receiving means receives a user operation in which, for each of the plurality of objects, the user specifies at least one of the position of the object, the orientation of the object, and the size of the object. (Item 13) The aforementioned receiving means receives user operations to specify a timecode range, The information processing device according to any one of items 1 to 12, characterized in that the recording means records a moving image of the object based on a 3D model of the object corresponding to a time code range in accordance with the user operation. (Item 14) The receiving means further receives user input specifying the movement of the viewpoint relative to the object, The information processing apparatus according to item 13, characterized in that the recording means records a moving image of the object from a viewpoint that moves according to the user operation. (Item 15) The information processing apparatus according to any one of items 1 to 14, characterized in that the recording means records information indicating an image region of the object in the virtual image. (Item 16) The information processing apparatus according to any one of items 1 to 15, further comprising a synthesis means for generating a composite image of the virtual image recorded by the recording means and the captured image. (Item 17) The information processing apparatus according to any one of items 13 to 14, further comprising a synthesis means for generating a composite video of the video recorded by the recording means and an image captured. (Item 18) An information processing system, A detection means for detecting identification information from a medium to identify 3D model data representing 3D models of objects corresponding to each of multiple time codes, A means for receiving user input to specify the timecode to be displayed, A generation means for generating an image of the object based on the identification information and the 3D model of the object corresponding to the time code according to the user operation, A display control means for displaying an image of the aforementioned object on a display unit, A recording means for recording a virtual image that is composited with an captured image, including an image of the object displayed on the display unit, in accordance with user operation, An information processing system characterized by comprising the following features. (Item 19) An information processing method performed by an information processing device, A process of detecting identification information from a medium to identify 3D model data representing 3D models of objects corresponding to each of multiple time codes, A process for receiving user input to specify the timecode to be displayed, A step of displaying an image of the object based on a 3D model of the object corresponding to the identification information and the time code according to the user operation on the display unit, A step of recording a virtual image to be composited with the captured image, which includes an image of the object displayed on the display unit, in accordance with user operation; An information processing method characterized by including (Item 20) A program that causes a computer to function as an information processing device as described in one of items 1 through 17.

[0090] This disclosure is not limited to the embodiments described above, and various modifications and variations are possible without departing from its spirit and scope. Accordingly, the claims are attached to make the scope of this disclosure public. [Explanation of Symbols]

[0091] 10: Terminal, 20: Server, 110: Detection unit, 120: Reception unit, 130: Transmission unit, 140: Receiving unit, 160: Display control unit, 170: Recording unit, 180: Synthesis unit, 210: Receiving unit, 220: Generation unit, 230: Transmission unit

Claims

1. A detection means for detecting identification information from a medium to identify 3D model data representing 3D models of objects corresponding to each of multiple time codes, A means for receiving user input to specify the timecode to be displayed, Display control means for displaying an image of the object based on a 3D model of the object corresponding to the identification information and the time code according to the user operation on the display unit, A recording means for recording a virtual image that is composited with an captured image, including an image of the object displayed on the display unit, in accordance with user operation, An information processing device characterized by comprising:

2. The information processing apparatus according to claim 1, wherein the receiving means further receives a user operation specifying at least one of the position of the object, the orientation of the object, and the size of the object.

3. The information processing apparatus according to claim 2, characterized in that the display control means causes the display unit to display a user interface for specifying at least one of the time code, the position of the object, the orientation of the object, and the size of the object.

4. The receiving means further receives user input specifying the viewpoint for the object, The information processing apparatus according to claim 1, characterized in that the display control means causes the display unit to display an image of the object from the viewpoint.

5. The information processing apparatus according to claim 4, characterized in that the viewpoint with respect to the object is indicated by the physical position and orientation of the information processing apparatus with respect to the medium.

6. The information processing apparatus according to claim 1, wherein the detection means detects the identification information based on the image of the medium in the captured image.

7. The information processing apparatus according to claim 6, characterized in that the identification information is printed as a code on the medium.

8. The information processing apparatus according to claim 6, characterized in that the medium is a printed document.

9. The information processing apparatus according to claim 6, characterized in that the medium is a character sculpture.

10. The aforementioned identification information identifies the 3D model data of each of the multiple objects, The receiving means independently receives a user operation for each of the multiple objects to specify the time code to be displayed. The information processing apparatus according to claim 1, characterized in that the display control means causes the display unit to display images of each of the plurality of objects, and each image of the plurality of objects is based on a 3D model of the object corresponding to a time code according to the user operation.

11. The information processing apparatus according to claim 10, wherein the receiving means further receives a user operation in which, for each of the plurality of objects, the user specifies at least one of the position of the object, the orientation of the object, and the size of the object.

12. The aforementioned identification information identifies the 3D model data of each of the multiple objects, The information processing apparatus according to claim 1, characterized in that the receiving means receives a user operation for each of the plurality of objects, independently specifying at least one of the position of the object, the orientation of the object, and the size of the object.

13. The aforementioned receiving means receives user operations to specify a timecode range, The information processing apparatus according to claim 1, characterized in that the recording means records a moving image of the object based on a 3D model of the object corresponding to a time code range according to the user operation.

14. The receiving means further receives user input specifying the movement of the viewpoint relative to the object, The information processing apparatus according to claim 13, characterized in that the recording means records a moving image of the object from a viewpoint that moves according to the user operation.

15. The information processing apparatus according to claim 1, wherein the recording means records information indicating an image region of the object in the virtual image.

16. The information processing apparatus according to claim 1, further comprising a synthesis means for generating a composite image of the virtual image recorded by the recording means and the captured image.

17. The information processing apparatus according to claim 13, further comprising a synthesis means for generating a composite video of the video recorded by the recording means and an image captured.

18. An information processing system, A detection means for detecting identification information from a medium to identify 3D model data representing 3D models of objects corresponding to each of multiple time codes, A means for receiving user input to specify the timecode to be displayed, A generation means for generating an image of the object based on the identification information and the 3D model of the object corresponding to the time code according to the user operation, A display control means for displaying an image of the aforementioned object on a display unit, A recording means for recording a virtual image that is composited with an captured image, including an image of the object displayed on the display unit, in accordance with user operation, An information processing system characterized by comprising the following features.

19. An information processing method performed by an information processing device, A process of detecting identification information from a medium to identify 3D model data representing 3D models of objects corresponding to each of multiple time codes, A process for receiving user input to specify the timecode to be displayed, A step of displaying an image of the object based on a 3D model of the object corresponding to the identification information and the time code according to the user operation on the display unit, A step of recording a virtual image to be composited with the captured image, which includes an image of the object displayed on the display unit, in accordance with user operation; An information processing method characterized by including

20. A program for causing a computer to function as an information processing device according to any one of claims 1 to 17.

Images