Program and information processing device

Abstract

To provide a program that improves the amusing property of presentation of a video, and an information processing device.SOLUTION: A program causes a user terminal 12 as a computer to function as: acquisition means 52 which acquires a plurality of kinds of figures based upon a video photographed by a camera when the plurality of figures are photographed by the camera; determination means 56 which determines effect data based upon a combination of the plurality of kinds of figures acquired by the acquisition means 52 and an arrangement relation of the plurality of figures in the video; and display control means 58 which adds the effect data determined by the determination means 56 to the video to display the video.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a program and an information processing device. [Background technology]

[0002] Augmented reality (AR) is a technology that adds additional information such as digital content to real-world information and outputs it through a smartphone, etc. For example, there is a technology that detects AR markers based on video captured by a camera (hereinafter referred to as "photographed video"), and adds a virtual video of a virtual object generated based on the AR markers to the photographed video as a special effect.

[0003] Regarding this technology, for example, Patent Document 1 listed below discloses that a composite AR marker constructed by combining multiple blocks, which are marker construction elements, is detected based on a captured image of a combination of multiple blocks, and the display of a virtual image is changed in accordance with the composite AR marker.

[0004] Furthermore, Patent Document 2 below discloses that an image obtained by photographing a specific mark attached to a base for placing a figure is recognized, and if there are multiple specific marks recognized in the image, the virtual image for the performance is changed depending on the combination of the marks. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 6262127 [Patent Document 2] Patent No. 5551205 Summary of the Invention [Problem to be solved by the invention]

[0006] The technology disclosed in the above-mentioned Patent Document 1 generates a virtual image based on a composite AR marker detected when a combination of multiple blocks is photographed. This composite AR marker is defined by the arrangement order of multiple combined blocks. Therefore, with this technology, when multiple uncombined blocks are photographed, the composite AR marker cannot be defined and a virtual image cannot be generated.

[0007] Furthermore, in the technology disclosed in Patent Document 2, when multiple unconnected objects are photographed, the virtual image can be changed based on the combination of the objects, but the relative positions of the multiple objects, the photographing position, the photographing posture, etc. are not taken into consideration, and therefore the same effect is produced for the photographed image regardless of these conditions.

[0008] The present invention has been made in view of the above-mentioned problems, and aims to provide a program and an information processing device that can enhance the creativity of video presentation. [Means for solving the problem]

[0009] A program according to a first aspect of the present invention is a program for causing a computer to function as an acquisition means for acquiring identification information of multiple objects based on the image captured by an imaging means when the multiple objects are captured by the imaging means, a determination means for determining additional information based on a combination of the identification information of the multiple objects acquired by the acquisition means and the relative positions of the multiple objects within the image, and a display control means for adding the additional information determined by the determination means to the image and displaying the image.

[0010] In the program according to the second aspect of the present invention, the determination means determines the position or size of the additional information to be displayed on the image based on at least the layout relationship, and the display control means displays the image with the additional information added at the position or size determined by the determination means.

[0011] In a program according to a third aspect of the present invention, when the layout relationship changes, the determination means determines a plurality of pieces of additional information corresponding to the changing layout relationship, and the display control means switches the additional information to be added to the image to one of the plurality of pieces of additional information determined by the determination means in accordance with the change in the layout relationship.

[0012] In a fourth aspect of the present invention, the program is configured such that, when an arrangement sequence indicating a series of sequences including the arrangement relationships that change in a changing order is a predetermined arrangement sequence, the determination means determines additional information corresponding to the predetermined arrangement sequence, and the display control means adds the additional information corresponding to the predetermined arrangement sequence determined by the determination means to the image and displays the image.

[0013] A program according to a fifth aspect of the present invention causes the computer to further function as a storage means for storing a plurality of pieces of additional information and, when an image to which the additional information has been added is displayed by the display control means, storing information indicating a plurality of objects included in the image as displayed objects, and a setting means for setting the plurality of pieces of additional information stored in the storage means to change or add to the plurality of pieces of additional information when the displayed objects satisfy predetermined conditions.

[0014] In a program according to a sixth aspect of the present invention, when the three-dimensional shooting position and shooting attitude of the shooting means relative to the three-dimensional object change, the determination means determines multiple pieces of additional information based on the shooting position and shooting attitude, and the display control means switches the additional information to be added to the image to one of the multiple pieces of additional information determined by the determination means in accordance with the change in the shooting position and shooting attitude.

[0015] In the program according to a seventh aspect of the present invention, when camerawork showing a series of sequences including the shooting positions and shooting postures that change in a changing order is a predetermined camerawork, the determination means determines additional information corresponding to the predetermined camerawork, and the display control means adds the additional information corresponding to the predetermined camerawork determined by the determination means to the image and displays the image.

[0016] In the program according to an eighth aspect of the present invention, the shape of the object can be changed by moving it, attaching or detaching parts, or rearranging parts, and when the shape of the object changes, the determination means determines multiple pieces of additional information corresponding to the changing shape, and the display control means switches the additional information to be added to the image to one of the multiple pieces of additional information determined by the determination means in accordance with the change in shape.

[0017] In the program according to a ninth aspect of the present invention, the determining means determines the additional information based on a current position when the object is photographed by the photographing means.

[0018] A program according to a tenth aspect of the present invention causes the computer to further function as a judgment means for judging whether or not to add the additional information to the image based on the current position when the object is photographed by the photographing means and the position stored in correspondence with the identification information of the object, and the display control means, when the judgment means makes a positive judgment, adds the additional information to the image and displays the image, and when the judgment means makes a negative judgment, displays the image without adding the additional information to the image.

[0019] In the program according to an eleventh aspect of the present invention, the determining means determines the additional information based on a current period or weather.

[0020] In a program according to a twelfth aspect of the present invention, the additional information includes a background image to be superimposed on the background of the object, and when the background image is superimposed on the image, the display control means switches the image of the object photographed by the imaging means to an image of a virtual object, makes a predetermined change, and then switches the image of the virtual object back to the image of the object photographed by the imaging means.

[0021] A program according to a twelfth aspect of the present invention causes a computer to further function as game control means for controlling a predetermined game based on the additional information added to the video by the display control means.

[0022] An information processing device according to a thirteenth aspect of the present invention comprises: an acquisition means for acquiring identification information of a plurality of objects based on the image captured by the imaging means when the plurality of objects are captured by the imaging means; a determination means for determining additional information based on a combination of the identification information of the plurality of objects acquired by the acquisition means and the relative positions of the plurality of objects within the image; and a display control means for adding the additional information determined by the determination means to the image and displaying the image. [Effects of the Invention]

[0023] According to the present invention, it is possible to enhance the ingenuity of the presentation of video. [Brief explanation of the drawings]

[0024] [Figure 1] 1 is a block diagram illustrating an example of the overall configuration of an augmented reality system according to a first embodiment. [Figure 2] 2 is a block diagram showing an example of a hardware configuration of a server device shown in FIG. 1. FIG. [Figure 3] 2 is a block diagram showing an example of the hardware configuration of a smartphone as the user terminal shown in FIG. 1. FIG. [Figure 4] FIG. 2 is a block diagram illustrating an example of a functional configuration of a user terminal. [Figure 5]10 is a diagram showing an example of the structure of effect information stored in a storage unit; FIG. [Figure 6A] FIG. 10 is a diagram conceptually explaining the layout relationship. [Figure 6B] FIG. 10 is a diagram conceptually explaining the layout relationship. [Figure 6C] FIG. 10 is a diagram conceptually explaining the layout relationship. [Figure 6D] FIG. 10 is a diagram conceptually explaining the layout relationship. [Figure 7A] FIG. 10 is a diagram illustrating an example of the configuration of correspondence relationship information. [Figure 7B] FIG. 10 is a diagram illustrating an example of the configuration of correspondence relationship information. [Figure 7C] FIG. 10 is a diagram illustrating an example of the configuration of correspondence relationship information. [Figure 7D] FIG. 10 is a diagram illustrating an example of the configuration of correspondence relationship information. [Figure 7E] FIG. 10 is a diagram illustrating an example of the configuration of correspondence relationship information. [Figure 8] 10 is a table for explaining the types of special effect modes. [Figure 9] 5 is a flowchart showing an example of the flow of processing performed by each functional component shown in FIG. 4 in the augmented reality system according to the first embodiment. [Figure 10] 10 is a flowchart showing an example of the process flow of step SP18 in FIG. 9. [Figure 11A] 10A and 10B are diagrams showing an example of a captured image displayed on a display of a user terminal under display control by a display control means. [Figure 11B] 10A and 10B are diagrams showing an example of a captured image displayed on a display of a user terminal under display control by a display control means. [Figure 11C] 10A and 10B are diagrams showing an example of a captured image displayed on a display of a user terminal under display control by a display control means. [Figure 11D] 10A and 10B are diagrams showing an example of a captured image displayed on a display of a user terminal under display control by a display control means. [Figure 11E] 10A and 10B are diagrams showing an example of a captured image displayed on a display of a user terminal under display control by a display control means. [Figure 12A] 1A and 1B are diagrams conceptually explaining a photographing position and a photographing posture. [Figure 12B] 1A and 1B are diagrams conceptually explaining a photographing position and a photographing posture. [Figure 12C] 1A and 1B are diagrams conceptually explaining a photographing position and a photographing posture. [Figure 12D] 1A and 1B are diagrams conceptually explaining a photographing position and a photographing posture. [Figure 13A] FIG. 11 is a diagram illustrating an example of the configuration of correspondence relationship information according to the second embodiment. [Figure 13B] FIG. 11 is a diagram illustrating an example of the configuration of correspondence relationship information according to the second embodiment. [Figure 13C] FIG. 11 is a diagram illustrating an example of the configuration of correspondence relationship information according to the second embodiment. [Figure 13D] FIG. 11 is a diagram illustrating an example of the configuration of correspondence relationship information according to the second embodiment. [Figure 14] 10 is a flowchart showing an example of the process flow of step SP18 in the second embodiment. [Figure 15] FIG. 11 is a diagram showing an example of a captured image displayed on a display of a user terminal by display control of a display control unit in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0025] [First embodiment] A first embodiment of the present invention will be described in detail below with reference to Figures 1 to 11. To facilitate understanding of the description, the same components and steps in each drawing are denoted by the same reference numerals as much as possible, and duplicated descriptions will be omitted.

[0026] <Overall structure> FIG. 1 is a block diagram showing an example of the overall configuration of an augmented reality system 1 according to the first embodiment.

[0027] 1, the augmented reality system 1 includes a server device 10 and one or more user terminals 12. The server device 10 and the user terminals 12 are configured to be able to communicate with each other via a communication network NT such as an intranet, the Internet, or a telephone line.

[0028] The server device 10 is an information processing device that provides the execution results obtained by executing a program 14 of an augmented reality application (hereinafter referred to as an "augmented reality app") for experiencing augmented reality (AR), or the program 14 itself, to each user terminal 12 via a communication network NT.

[0029] Each user terminal 12 is an information processing device owned by a user. Examples of the user terminal 12 include a smartphone, a mobile phone, a tablet, a personal computer, etc. In this embodiment, the user terminal 12 will be described as a smartphone.

[0030] After the program 14 is installed in the user terminal 12, the augmented reality application in this embodiment can be activated based on a predetermined operation by the user on the user terminal 12. When multiple objects are photographed using the user terminal 12, the augmented reality application adds additional information such as video and audio to the photographed images, and displays an image that combines the real world with the additional information on the screen of the user terminal 12.

[0031] Examples of objects photographed by the user terminal 12 include objects having a predetermined shape, such as figurines, toys, plastic models, and stuffed toys. In this embodiment, the object is described as a three-dimensional object, particularly a figurine. A figurine is a three-dimensional object that is a three-dimensional representation of a character, animal, person, or the like from a predetermined animation or the like.

[0032] In this embodiment, the download and installation of the program 14 and the launch of the augmented reality application may be enabled, for example, by reading predetermined download information into the user terminal 12. The download information may be, for example, a QR code (registered trademark) or a URL attached to an instruction manual or the like inside the figure package, and may be obtained by purchasing the figure.

[0033] Note that the execution result of the program 14 in the server device 10 may be provided to each user terminal 12 via the communication network NT, without installing the program 14 of the augmented reality application in the user terminal 12. In this case, for example, the augmented reality application is launched as a web application used on a web browser via the communication network NT based on a predetermined operation by the user on the user terminal 12. Examples of the predetermined operation here include clicking a link on a predetermined website on the user terminal 12, or reading a predetermined QR code (registered trademark) displayed at a predetermined store, event venue, etc. with the user terminal 12.

[0034] <Hardware configuration> FIG. 2 is a block diagram showing an example of a hardware configuration of the server device 10 shown in FIG.

[0035] 2, the server device 10 includes a control device 20, a communication device 26, and a storage device 28. The control device 20 is mainly configured to include a CPU (Central Processing Unit) 22 and a memory 24.

[0036] In the control device 20, the CPU 22 executes a predetermined program stored in the memory 24, the storage device 28, or the like.

[0037] The communication device 26 is configured with a communication interface for communicating with an external device, etc. This communication device 26 transmits and receives various information to and from the user terminal 12, for example.

[0038] The storage device 28 is configured with a hard disk, etc. The storage device 28 stores various programs and information necessary for executing the processes in the control device 20, including the program 14, and information on the results of the processes.

[0039] The server device 10 can be realized using an information processing device such as a dedicated or general-purpose server computer. The server device 10 may be configured by a single information processing device or by multiple information processing devices distributed over a communication network NT. Fig. 2 shows only a portion of the main hardware configuration of the server device 10, and the server device 10 may also have other configurations that are generally included in a server.

[0040] FIG. 3 is a block diagram showing an example of the hardware configuration of a smartphone serving as the user terminal 12 shown in FIG.

[0041] As shown in FIG. 3, the user terminal 12 includes a main control unit 30, a touch panel 32, a camera 34, a mobile communication unit 36, a wireless LAN (Local Area Network) communication unit 38, a memory unit 40, a speaker 42, an acceleration / direction sensor 44, and a GPS (Global Positioning System) receiving unit 46.

[0042] The main control unit 30 is configured to include a CPU, memory, etc. Connected to the main control unit 30 are a touch panel 32, a camera 34, a mobile communication unit 36, a wireless LAN communication unit 38, a storage unit 40, a speaker 42, an acceleration / direction sensor 44, and a GPS receiving unit 46. The main control unit 30 has a function to control these connected devices.

[0043] The touch panel 32 has the functions of both a display device and an input device, and is composed of a display 32A that performs the display function and a touch sensor 32B that performs the input function. The display 32A is composed of a general display device such as a liquid crystal display or an organic EL (Electro Luminescence) display. The display 32A displays, for example, an augmented reality application execution screen generated by execution of the program 14.

[0044] The touch sensor 32B is composed of an element for detecting a touch operation on the screen displayed by the display 32A. The touch sensor 32B can use any known method for detecting a touch operation, such as a capacitance type, a resistive film type (pressure-sensitive type), or an electromagnetic induction type. The touch sensor 32B accepts a user's operation input by detecting the movement of a user's finger, stylus, or other operating element that contacts the screen. When the touch sensor 32B detects the movement of the user's finger, stylus, or other operating element, it detects coordinates indicating the position of contact on the screen and outputs the coordinates to the main control unit 30. The coordinates indicating the position are expressed, for example, as coordinate values ​​on an xy plane along the screen displayed by the display 32A.

[0045] The camera 34 is a photographing means for taking still images and / or moving images, and has a function of storing the photographed results in the storage unit 40.

[0046] The mobile communication unit 36 ​​has a function of connecting to a mobile communication network via an antenna 36A and communicating with other communication devices connected to the mobile communication network.

[0047] The wireless LAN communication unit 38 has a function of connecting to the communication network NT via an antenna 38A and communicating with other devices such as the server device 10 connected to the communication network NT.

[0048] The storage unit 40 stores various programs including the program 14 and various pieces of information.

[0049] The speaker 42 has a function of outputting sounds and the like while an augmented reality application is running.

[0050] The acceleration and direction sensor 44 has a function of acquiring information for calculating the direction and inclination of the user terminal 12, and includes various sensors such as an electronic magnetic compass that detects geomagnetism, a gyrocompass, and an acceleration sensor.

[0051] The GPS receiver 46 has a function of receiving GPS signals for identifying the position of the user terminal 12 from GPS satellites via an antenna 46A.

[0052] Note that Figure 3 only shows some of the main hardware components of the user terminal 12, and the user terminal 12 may also have other components that are generally included in smartphones, such as a microphone for inputting voice, a real-time clock, and short-range wireless communication.

[0053] <Functional configuration> FIG. 4 is a block diagram showing an example of the functional configuration of the user terminal 12. As shown in FIG.

[0054] 4, the user terminal 12 includes, as functional components, a storage unit 50, an acquisition unit 52, an analysis unit 54, a determination unit 56, a display control unit 58, a determination unit 60, a setting unit 62, and a game control unit 64. These functional components are realized by executing a predetermined program stored in the storage unit 40 under the control of the main control unit 30 and operating the various components of the user terminal 12 shown in FIG. 3. Note that all or part of these functional components may be provided in the server device 10.

[0055] The storage means 50 stores figure information 50A, effect information 50B, correspondence relationship information 50C, already-displayed figure information 50D, and collection condition information 50E.

[0056] The figure information 50A is information relating to the figure as an object photographed by the camera 34. Video including a figure photographed by the camera 34 is hereinafter referred to as "photographed video." The figure information 50A includes feature point data of the figure. The feature point data of the figure is acquired in advance from three-dimensional model data of the figure, multiple photographs, etc. Multiple pieces of feature point data of the figure are stored in association with the type of figure. The type of figure is identification information indicating what type of figure the figure is, and is indicated by, for example, the product name, character name, genre name, figure ID, product ID, etc.

[0057] Furthermore, the feature point data of a figure may be stored in association with the shape, orientation, etc. of the figure for each type of figure. The shape of a figure is information regarding the three-dimensional shape or the two-dimensional shape of a three-dimensional figure when viewed from above. The shape of a figure can be changed by moving the figure, attaching or detaching detachable parts of the figure, or rearranging such parts. Furthermore, the orientation of a figure is information indicating the orientation of the placed figure in three-dimensional space. The orientation of a figure can also be changed by changing the position of the figure or in the same way as the shape of the figure.

[0058] Effect information 50B is information generated as additional information to be added to the captured video. Effect information 50B includes effect data configured as various digital content such as video, including still images and video, text, music, audio, and sound effects. When the effect data is video effect data, the effect data includes background video to be superimposed on the background of the figure, video of a virtual object to be superimposed on the video of the figure itself, and the like. Hereinafter, effect data output by displaying video, text, etc. will also be referred to as "display effect." Furthermore, effect data output by sound, such as music, audio, and sound effects, will also be referred to as "sound effect."

[0059] Fig. 5 is a diagram showing an example of the configuration of effect information 50B. As shown in Fig. 5, effect information 50B is a data table that stores path names of effect data in association with effect IDs, which are identification information of the effect data. The path names of the effect data indicate the storage locations of the effect data in the storage unit 40, etc. Note that the effect IDs and the path names of the effect data are not limited to a one-to-one correspondence, and one effect data may be associated with multiple effect IDs, or one effect ID may be associated with multiple effect data.

[0060] The correspondence information 50C is information indicating the correspondence between the situation information and the effect data. The situation information is, for example, information about the situation of the figure included in the photographed video, and is predetermined information that can be acquired by analyzing the photographed video. In addition to the type and shape of the figure described above, the situation information includes the combination of multiple figures, the positional relationship of multiple figures, or the positional sequence of multiple figures.

[0061] The combination of multiple figures is information indicating the combination of types of multiple figures photographed by the camera 34, i.e., the combination of types of figures included in the photographed video. Hereinafter, the combination of multiple figures will also be simply referred to as a "combination."

[0062] The positional relationship of multiple figures is information indicating the positional relationship of multiple figures within a captured image (captured range), and includes not only the position of each figure itself, but also the distance between each figure, the orientation of each figure, and the combination of the orientations of multiple figures within the captured range. The coordinates for calculating the position of each figure and the distance between each figure can be, for example, the center coordinates of each figure, but are not limited to this and may also be the coordinates of any position on each figure. Hereinafter, the positional relationship of multiple figures will also be simply referred to as the "positional relationship." Furthermore, the positional sequence of multiple figures is information indicating a series of sequences including the positional relationship that changes in the order in which it changes when the positional relationship changes. Hereinafter, the positional sequence of multiple figures will also be simply referred to as the "positional sequence."

[0063] Here, the layout relationship will be conceptually explained with reference to Figures 6A to 6D. Figures 6A to 6D are diagrams conceptually explaining the layout relationship. Note that although Figures 6A to 6D illustrate the figures F01 to F03 included in the photographed video 100 in a two-dimensional plane, the figures F01 to F03 actually photographed are three-dimensional solid objects.

[0064] As shown in Figures 6A to 6D, the figures F01 to F03 are arranged, for example, in an unconnected state. An unconnected state means that the figures are not directly physically connected to each other and are independent. This allows the figures F01 to F03 to be arranged in various orientations with a predetermined distance between them. Note that the unconnected state is not necessarily limited to the state shown in the figures, and may include, for example, a state in which the figures are indirectly connected to each other via a predetermined connecting element, a state in which there is overlapping between the figures, etc.

[0065] 6A, the figures F01 to F03 are all positioned side by side in the photographed video 100. The distance between the figures F01 to F03 is, for example, 5 cm or less. Each figure is facing forward.

[0066] 6B, the figure F01 and the figures F02 and F03 are positioned side by side in the photographed video 100. The distance between the figure F02 and the figure F03 is, for example, 5 cm or less, while the distance between the figure F01 and the figures F02 and F03 is, for example, 5 to 10 cm. Each figure is facing forward.

[0067] 6C, the figure F01 and each of the figures F02 and F03 are positioned side by side in the photographed video 100. The distance between the figure F02 and the figure F03 is, for example, 5 cm or less, while the distance between the figure F01 and each of the figures F02 and F03 is, for example, 5 to 10 cm. The figure F01 and each of the figures F02 and F03 face each other.

[0068] 6D, the figures F01 to F03 are positioned in a triangular arrangement within the photographed image 100. The distance between each of the figures F01 to F03 is, for example, 5 to 10 cm. The figures F01 to F03 have their backs to each other.

[0069] As described above, a predetermined arrangement relationship is formed by the positions of multiple figures in the captured video 100, the distance between each figure, the orientation of each figure, etc. Furthermore, when the arrangement relationship changes, for example, when a user changes the arrangement of each figure, an arrangement sequence is formed that shows a series of sequences including the changing arrangement relationships in the changing order. For example, when the arrangement relationship changes from that shown in FIG. 6B to that shown in FIG. 6C within a predetermined time period, an arrangement sequence is formed that includes the arrangement relationship shown in FIG. 6B and the arrangement relationship shown in FIG. 6C in that changing order.

[0070] 7A to 7E are diagrams showing examples of the configuration of the correspondence information 50C. As shown in Fig. 7A to 7E, the correspondence information 50C is a data table that stores effect IDs in association with situation information (type, shape, combination, placement relationship, and placement sequence of figures).

[0071] Correspondence information 50C shown in Fig. 7A stores effect IDs in association with the type of figure. Correspondence information 50C shown in Fig. 7B stores effect IDs in association with the shape of the figure. Correspondence information 50C shown in Fig. 7B shows an example where the type of figure is, for example, "human character A."

[0072] Furthermore, the correspondence relationship information 50C shown in FIG. 7C stores effect IDs in association with combinations of multiple figures. Furthermore, the correspondence relationship information 50C shown in FIG. 7D stores effect IDs in association with the positional relationship of multiple figures. Furthermore, the correspondence relationship information 50C in FIG. 7E stores effect IDs in association with the positional sequence of multiple figures. The correspondence relationship information 50C in FIGS. 7D and 7E shows an example where the combination is a set of three types of figures, for example, "F01, F02, F03."

[0073] Each piece of correspondence information 50C as shown in Figures 7A to 7E is linked to one another by predetermined situation information, etc. For example, the storage means 50 stores a plurality of pieces of correspondence information 50C between figure shapes and effect IDs as shown in Figure 7B, linked to the type of figure. The storage means 50 also stores a plurality of pieces of correspondence information 50C between placement relationships and effect IDs as shown in Figure 7D and a plurality of pieces of correspondence information 50C between placement sequences and effect IDs as shown in Figure 7E, linked to combinations.

[0074] Returning to Fig. 4, the displayed figure information 50D is information indicating a displayed figure (displayed object). A displayed figure is a figure included in the photographed video 100 that has been displayed on the display 32A with effect data added. For example, when the photographed video 100 with effect data added by the display control means 58 is displayed on the display 32A, the storage means 50 stores the type or number of multiple figures included in the photographed video 100 as displayed figure information 50D.

[0075] The collection condition information 50E is information that indicates collection conditions related to the collection of multiple photographed figures. The collection condition information 50E stores various conditions related to the types and numbers of figures, such as "five types of figures," "three animal figures," "one or more figures of type △△," and "ten figures."

[0076] When multiple figures are photographed by the camera 34 while an augmented reality application is running on the user terminal 12, the acquisition means 52 sequentially acquires the photographed images 100. The acquisition means 52 sequentially outputs the acquired photographed images 100 to the analysis means 54 and the display control means 58.

[0077] The analysis means 54 sequentially analyzes the photographed video 100 acquired by the acquisition means 52, and acquires the types of multiple figures included within the shooting range captured as the photographed video 100. The analysis means 54 functions as an acquisition means that acquires the types of multiple figures as identification information of the multiple figures based on the photographed video 100.

[0078] For example, the analysis means 54 uses known video analysis technology to analyze the captured video 100 and detect feature points of the figure. The analysis means 54 acquires the type of figure corresponding to the detected feature points based on the detected feature points and the figure information 50A stored in the storage means 50. More specifically, the analysis means 54 extracts feature point data whose degree of similarity to the detected feature points is equal to or greater than a predetermined value from the feature point data stored in association with the type of figure. Then, the analysis means 54 acquires the type of figure corresponding to the feature point data.

[0079] Furthermore, the analysis means 54 identifies a combination of figures included in the photographed video 100 based on the types of figures obtained. For example, if the types of figures obtained are "human character A," "bear," and "dog," the analysis means 54 identifies a set of these three figures, "human character A," "bear," and "dog," as a combination. Furthermore, if the types of figures obtained have a common attribute, such as being a character from a predetermined animation, the attribute itself may be identified as a combination.

[0080] Furthermore, when the analysis means 54 acquires the type of figure, it may further detect feature points related to the shape and orientation of the figure in detail. In this case, the analysis means 54 acquires information indicating the shape and orientation of the figure corresponding to the detected feature points based on the feature points related to the detected shape and orientation and the figure information 50A stored in the storage means 50. For example, the analysis means 54 extracts feature point data whose degree of approximation to the detected feature points is equal to or greater than a predetermined value from feature point data stored in association with the shape and orientation of each figure type. Then, the analysis means 54 acquires information indicating the shape and orientation of the figure corresponding to the feature point data.

[0081] The analysis means 54 also analyzes the filmed video 100 using known video analysis techniques to acquire the positional relationship of multiple figures in the filmed video 100. That is, the analysis means 54 acquires the position of each figure, the distance between each figure, the orientation of each figure, etc. For example, the analysis means 54 acquires the position of each figure, the distance between each figure, or the orientation of each figure based on the feature points of each figure detected from the filmed video 100. More specifically, the analysis means 54 determines the outer shape of each figure from the detected feature points of each figure, identifies position coordinates included in the outer shape, and acquires the position of each figure and the distance between each figure by calculating the difference between the identified position coordinates. The analysis means 54 also extracts feature point data whose similarity to feature points stored in association with the orientation of the figure is equal to or greater than a predetermined value. Then, the analysis means 54 acquires the orientation of the figure corresponding to the feature point data.

[0082] Furthermore, when the layout relationship changes within a predetermined period, for example, the analysis means 54 acquires a layout sequence based on the layout relationship. For example, when the orientation of each of the figures F01 to F03 changes from facing forward to facing each other, from the layout relationship of Fig. 6B to the layout relationship of Fig. 6C, the analysis means 54 acquires information including the layout relationship of Fig. 6B and the layout relationship of Fig. 6C in the order of change as the layout sequence. The analysis means 54 outputs the video analysis information acquired as a result of the analysis to the determination means 56.

[0083] The determination means 56 determines effect data as additional information to be added to the photographed video 100 based on the video analysis information output from the analysis means 54. First, the determination means 56 refers to the correspondence information 50C to identify the effect ID corresponding to the output video analysis information. Next, the determination means 56 refers to the effect information 50B to identify the path name of the effect data corresponding to the identified effect ID. As a result, the determination means 56 determines the effect data stored in the storage destination indicated by the path name as the effect data to be added to the photographed video 100. Then, the determination means 56 acquires the effect data stored in the storage destination indicated by the path name and outputs the acquired effect data to the display control means 58.

[0084] Specifically, when the type of figure is output from the analysis means 54, the determination means 56 refers to the correspondence information 50C in Fig. 7A and identifies an effect ID corresponding to the type of figure. Also, when the shape of a figure is output from the analysis means 54, the determination means 56 refers to the correspondence information 50C in Fig. 7B and identifies an effect ID corresponding to the shape of the figure. Also, when both the type and shape of a figure are output from the analysis means 54, the determination means 56 refers to the correspondence information 50C in Fig. 7B linked to the type of figure and identifies an effect ID corresponding to the type and shape of the figure.

[0085] Furthermore, when a combination is output from the analysis means 54, the determination means 56 refers to the correspondence information 50C in FIG. 7C and identifies an effect ID corresponding to the combination. Furthermore, when a layout relationship is output from the analysis means 54, the determination means 56 refers to the correspondence information 50C in FIG. 7D and identifies an effect ID corresponding to the layout relationship. Furthermore, when both a combination and a layout relationship are output from the analysis means 54, the determination means 56 refers to the correspondence information 50C in FIG. 7D linked to the combination and identifies an effect ID corresponding to the combination and layout relationship. That is, the determination means 56 identifies an effect ID based on the combination and the layout relationship.

[0086] Furthermore, when the video analysis information changes, the determination means 56 determines multiple effect data corresponding to the changing video analysis information. For example, when the shape of a figure changes, the determination means 56 refers to the correspondence relationship information 50C in Fig. 7B in the same manner as above to identify multiple effect IDs corresponding to the changing shape. Furthermore, when the positional relationship changes, the determination means 56 refers to the correspondence relationship information 50C in Fig. 7D in the same manner as above to identify multiple effect IDs corresponding to the changing positional relationship.

[0087] Furthermore, when a placement sequence is output from the analysis means 54, the determination means 56 determines whether the placement sequence is a predetermined placement sequence stored in the storage means 50. For example, the determination means 56 references the correspondence information 50C in FIG. 6E linked to the combination output from the analysis means 54. If the output placement sequence corresponds to a placement sequence stored in the correspondence information 50C, the determination means 56 makes a positive determination. Note that the case where the output placement sequence corresponds to a placement sequence stored in the correspondence information 50C is not limited to a case where the placement sequences match each other, but may also include, for example, a case where the placement sequences are within a predetermined range of similarity with each other. If the determination means 56 makes a positive determination, it identifies an effect ID corresponding to the predetermined placement sequence.

[0088] Furthermore, the determination means 56 determines the position and size of the effect data to be added to the photographed video 100 based on the positional relationship output from the analysis means 54. In other words, the determination means 56 determines the position at which the display effect is to be superimposed on the photographed video 100 and the size of the display effect to be superimposed.

[0089] For example, when the figures F01 to F03 are arranged side by side as shown in Fig. 6A, the determination means 56 determines the position and size of the display effect in the photographed video 100 so that the display effect is positioned and sized to overlap as the background of all of the figures F01 to F03. Also, when the figure F01 and each of the figures F02 and F03 are arranged facing each other as shown in Fig. 6C, the determination means 56 determines the position and size of the display effect in the photographed video 100 so that the display effect is positioned and sized to fit between the figure F01 and each of the figures F02 and F03.

[0090] Furthermore, the determination means 56 may determine the position and size of the display effect in the photographed video 100 based on other video analysis information, such as the combination of multiple figures, in addition to or instead of the positional relationship. For example, if the combination is a combination of predetermined characters, the determination means 56 determines a position and size appropriate for that combination. Examples of a position and size appropriate for the combination include a position and size that fits between the figures in the case of a combination of characters who are enemies, and a position and size that overlaps the background of the figures in the case of a combination of characters who are allies. Furthermore, examples of a position and size that can recreate a scene appearing in that animation in the case of a combination of characters who are allies include a position and size that can recreate a scene appearing in that animation. Furthermore, the determination means 56 may determine either the position or the size of the display effect in the photographed video 100, rather than both the position and the size.

[0091] Furthermore, the determination means 56 may determine the manner in which various effect data, not limited to display effects, is added to the captured video 100 based on video analysis information such as layout relationships and combinations. For example, the determination means 56 may determine the volume and sound direction of a sound effect based on layout relationships and combinations. The specific determination method is the same as for display effects.

[0092] The method of determining effect data by the determination means 56 can be changed, for example, depending on whether the mode setting of the augmented reality application is set to normal effect mode or special effect mode by a designer or an operator. For example, depending on whether the normal effect mode or special effect mode is set, the determination means 56 changes the correspondence information 50C that is referenced when identifying the effect ID. Note that the normal effect mode or special effect mode may be set before the augmented reality application is launched, or may be switched during launch.

[0093] For example, when the normal presentation mode is set, the determination means 56 identifies the effect ID by referring to the correspondence information 50C shown in Figure 7A or 7B. On the other hand, when the special presentation mode is set, the determination means 56 identifies the effect ID by referring to the correspondence information 50C shown in Figures 7C to 7E.

[0094] Fig. 8 is a table for explaining the types of special effect modes. As shown in Fig. 8, special effect mode "A" is a "combination mode" in which effect data corresponding to a combination is added to the photographed video 100. Special effect mode "B" is a "figure placement mode" in which effect data corresponding to a placement relationship or placement sequence is added to the photographed video 100. Special effect modes "C" to "E" will be explained in the second embodiment.

[0095] When the special effect mode is set to the ON state, at least one of the multiple special effect modes shown in Fig. 8 may be selected and set. The determination means 56 may refer to different correspondence information 50C depending on at least one selected special effect mode from the multiple special effect modes. In other words, the method of determining effect data may be changed depending on the type of mode selected as the special effect mode.

[0096] For example, if the special effect mode "A" is selected, the determination means 56 refers to the correspondence information 50C shown in Fig. 7C to identify the effect ID. On the other hand, if the special effect mode "B" is selected, the determination means 56 refers to the correspondence information 50C shown in Fig. 7D or 7E to identify the effect ID. Furthermore, if both the special effect modes "A" and "B" are selected, the determination means 56 refers to the correspondence information 50C shown in Fig. 7D or 7E that is linked to the combination output from the analysis means 54 to identify the effect ID.

[0097] Returning to FIG. 4 , the display control means 58 controls the display of the photographed video 100 acquired by the acquisition means 52 on the display 32A. At this time, the display control means 58 adds the effect data output by the determination means 56 to the photographed video 100 and displays the photographed video 100. For example, if the effect data is a display effect, the display control means 58 displays the photographed video 100 with the display effect superimposed at the position and size determined by the determination means 56. Furthermore, for example, if the effect data is a sound effect, the display control means 58 displays the photographed video 100 while outputting the sound effect at the volume and direction determined by the determination means 56.

[0098] Furthermore, when the shape or positional relationship of the figures changes, the display control means 58 switches the effect data to be added to the photographed video 100 to one of the plurality of effect data determined by the determination means 56 in accordance with the change. Furthermore, when a predetermined placement sequence is formed, the display control means 58 displays the photographed video 100 with the effect data corresponding to the placement sequence added.

[0099] The determination means 60 determines whether the displayed figures stored as displayed figure information 50D satisfy predetermined conditions. For example, the determination means 60 determines whether the type or number of displayed figures satisfy at least one of the collection conditions stored as collection condition information 50E. The determination means 60 may make this determination for all of the collection conditions, or may make this determination for at least one of the collection conditions selected by the designer or operator. The determination means 60 outputs the determination result to the setting means 62.

[0100] The setting means 62 performs settings to change or add effect data stored in the storage means 50 when a displayed figure stored as displayed figure information 50D satisfies a predetermined condition. For example, when the type or number of displayed figures satisfies the collection conditions stored in the collection condition information 50E, i.e., when a positive determination is made by the determination means 60, the setting means 62 changes or adds at least one piece of effect data stored in the storage means 50. Changing effect data means changing the effect data by editing or modifying the effect data itself. Adding effect data means storing new effect data in the storage unit 40 separately from the effect data in question, and storing the storage destination as effect information 50B.

[0101] The game control means 64 controls a predetermined game based on the effect data added to the photographed video 100 by the display control means 58. The game control means 64 links the effect data to a predetermined video game or the like by, for example, reading and setting information indicated by the effect data into the video game. For example, when a display effect of an item related to the video game, such as a weapon or equipment, is displayed on the photographed video 100 of a figure of a character in the video game, the game control means 64 sets the item so that the character in the video game possesses the item.

[0102] <Processing flow> 9 is a flowchart showing an example of the flow of processing performed by each functional component shown in FIG. 4 in the augmented reality system 1 according to the first embodiment. The content and order of the processing shown in FIG. 9 can be changed as appropriate. The processing in FIG. 9 is started, for example, when the user starts an augmented reality application on the user terminal 12.

[0103] (Step SP10) When the user points the camera 34 of the user terminal 12 at a plurality of figures, the acquisition means 52 sequentially acquires the captured images 100 captured by the camera 34. Then, the process proceeds to step SP12.

[0104] (Step SP12) The analysis means 54 sequentially analyzes the photographed video 100 acquired in the processing of step SP10 to acquire the types of multiple figures. The analysis means 54 may also acquire the shapes of multiple figures by analyzing the photographed video 100. Then, the processing proceeds to the processing of step SP14.

[0105] (Step SP14) The determination means 56 determines whether the special effect mode is ON. If the determination is negative, the process proceeds to step SP16. On the other hand, if the determination is positive, the process proceeds to step SP18.

[0106] (Step SP16) The determination means 56 acquires effect data corresponding to the normal rendering mode. That is, the determination means 56 acquires effect data corresponding to the type and shape of the figure. For example, the determination means 56 refers to the correspondence information 50C in FIG. 7A or 7B regarding the type and shape of the figure acquired in the processing of step SP12, and identifies the effect ID corresponding to the type and shape of the figure. Next, the determination means 56 refers to the effect information 50B in FIG. 5, identifies the effect data path corresponding to the identified effect ID, and acquires the effect data stored in the storage destination indicated by the effect data path. Then, the processing proceeds to the processing of step SP20.

[0107] (Step SP18) The determination means 56 acquires effect data corresponding to the special effect mode. For example, if both "A" and "B" of the special effect modes are selected, the determination means 56 acquires effect data corresponding to the combination and arrangement relationship. Then, the process proceeds to step SP20.

[0108] (Step SP20) The display control means 58 sequentially displays the photographed video 100 acquired in the processing of step SP12 on the display 32A. At this time, when effect data is acquired in the processing of step SP16 or step SP18, the display control means 58 adds the acquired effect data to the photographed video 100 and sequentially displays the photographed video 100. Then, the processing proceeds to the processing of step SP22.

[0109] (Step SP22) The analysis means 54 determines whether there is any change in the video analysis information acquired as the analysis result. The analysis means 54 determines, for example, whether the positional relationship has not changed within a predetermined time. If the determination is negative, the process returns to the process of step SP18, and the processes of steps SP18 and SP20 are repeatedly executed. As a result, in the process of step SP20, the display control means 58 switches the effect data added to the shot video 100 to one of the multiple effect data acquired in the process of step SP18 in accordance with the change in the positional relationship. On the other hand, if the determination is positive, the process proceeds to the process of SP24.

[0110] (Step SP24) The storage means 50 stores the figures included in the photographed video 100 displayed on the display 32A in the process of step SP20, i.e., the type or number of the figures whose types were acquired in the process of step SP12, as displayed figure information 50D. Then, the process proceeds to the process of step SP26.

[0111] (Step SP26) The determination means 60 determines whether the type or number of figures stored as the displayed figure information 50D satisfies at least one collection condition stored in the collection condition information 50E. If the determination is affirmative, the process proceeds to step SP28. If the determination is negative, the process proceeds to step SP30.

[0112] (Step SP28) The setting means 62 changes or adds effect data stored in the storage means 50. Then, the process proceeds to step SP30.

[0113] (Step SP30) The game control means 64 determines whether the game-linked setting is in the ON state. If the determination is affirmative, the process proceeds to step SP32. If the determination is negative, the series of processes shown in FIG. 9 ends.

[0114] (Step SP32) The game control means 64 links the effect data added to the photographed video 100, that is, the effect data acquired in the processing of step SP16 or step SP18, with the game. Then, the series of processing shown in FIG.

[0115] FIG. 10 is a flowchart showing an example of the process flow of step SP18 in FIG.

[0116] (Step SP40) The analysis means 54 identifies a combination of a plurality of figures based on the types of the plurality of figures acquired in the process of step SP12, and then the process proceeds to the process of step SP42.

[0117] (Step SP42) The analysis means 54 acquires the positional relationship of the multiple figures by sequentially analyzing the photographed video 100 acquired in the processing of step SP10. Furthermore, the analysis means 54 acquires the positional sequence when the positional relationship changes. Then, the processing proceeds to the processing of step SP44.

[0118] (Step SP44) When a placement sequence is acquired in the processing of step SP42, the determination means 56 refers to the correspondence information 50C in FIG. 7E that is linked to the combination identified in the processing of step SP40. Next, the determination means 56 determines whether the acquired placement sequence is a predetermined placement sequence stored in the correspondence information 50C. If the determination is affirmative, the processing proceeds to processing of step SP46. On the other hand, if the determination is negative, the processing proceeds to processing of step SP48. Also, if a placement sequence is not acquired in the processing of step SP42, the processing proceeds to processing of step SP48.

[0119] (Step SP46) The determination means 56 identifies the effect ID corresponding to the predetermined arrangement sequence for which a positive decision was made in the processing of step SP44. Next, the determination means 56 refers to the effect information 50B in FIG. 5 to identify the effect data path corresponding to the identified effect ID. Next, the determination means 56 acquires the effect data stored in the storage destination indicated by the effect data path. Then, the series of processing steps shown in FIG. 10 ends.

[0120] (Step SP48) The determination means 56 refers to the correspondence relationship information 50C in FIG. 7D linked to the combination identified in the processing of step SP40. Next, the determination means 56 identifies the effect ID corresponding to the placement relationship acquired in the processing of step SP42. Next, the determination means 56 refers to the effect information 50B in FIG. 5 and identifies the effect data path corresponding to the identified effect ID. Next, the determination means 56 acquires the effect data stored in the storage destination indicated by the effect data path. Then, the series of processing shown in FIG. 10 ends.

[0121] <Screen display example> 11A to 11E are diagrams showing an example of a captured image 100 displayed on the display 32A of the user terminal 12 under display control of the display control means 58. In the examples of Fig. 11A to 11E, a display effect is displayed as effect data superimposed on the captured image 100.

[0122] The photographed video 100 shown in Fig. 11A shows an example in which display effects are superimposed when the normal presentation mode is set. When the normal presentation mode is set, display effects AR01, AR02, and AR03 corresponding to the types and shapes of multiple figures F01, F02, and F03 are obtained in the processing of step SP16. In this case, as shown in Fig. 11A, the display effects AR01, AR02, and AR03 are superimposed and displayed on the photographed video 100.

[0123] In contrast, the photographed video 100 shown in Figures 11B to 11E shows an example in which a display effect is superimposed when a special effect mode is set. When "A" or "B" is selected as the special effect mode, a display effect AR102 corresponding to the combination and arrangement of multiple figures F01, F02, and F03 is acquired in the processing of step SP18. In this case, as shown in Figure 11B, the display effect AR102 is superimposed on the photographed video 100. The display effect AR102 is, for example, a rainbow image as a background image, and is displayed at a position and size that bridges the gap between the figures F01 and F03 at both ends based on the arrangement.

[0124] Furthermore, for example, if the layout relationship changes from Fig. 6B to Fig. 6C, the processing of step SP18 acquires a display effect AR104 corresponding to the layout relationship of Fig. 6B and a display effect AR106 corresponding to the layout relationship of Fig. 6C. In this case, as shown in Fig. 11C and Fig. 11D, the display effect in the shot video 100 is switched from the display effect AR104 to the display effect AR106 in accordance with the change in the layout relationship.

[0125] Furthermore, for example, when a predetermined arrangement sequence is formed by changing the arrangement relationship from Fig. 6B to Fig. 6C, AR106 may be acquired as a display effect corresponding to the arrangement sequence in the processing of step SP18. In this case, in response to the formation of the arrangement sequence, the display effect AR106 may be superimposed and displayed on the captured video 100, as shown in Fig. 11D.

[0126] Furthermore, for example, after the captured video 100 with the display effect AR102 superimposed thereon is displayed on the display 32A as shown in FIG. 11B, the video of the figures F01, F02, and F03 may be switched to the display effects AR108, AR110, and AR112 of the virtual objects, respectively, at a predetermined timing as shown in FIG. 11E. These display effects AR108, AR110, and AR112 may be, for example, moving images that are displayed while undergoing a predetermined change. Then, after such a predetermined change has occurred, the display effects AR108, AR110, and AR112 may be switched back to the video of the figures F01, F02, and F03 at a predetermined timing as shown in FIG. 11B. The switching of these display effects may not be limited to predetermined timing, but may also be performed in response to a user operation or the like.

[0127] <Action and effect> As described above, in the first embodiment, the user terminal 12 as a computer functions as an acquisition means 52 that, when multiple figures are photographed by the camera 34, acquires the types of multiple figures based on the photographed video 100 captured by the camera 34, a determination means 56 that determines effect data based on the combination of the types of multiple objects acquired by the acquisition means 52 and the positional relationship of the multiple figures in the photographed video 100, and a display control means 58 that adds the effect data determined by the determination means 56 to the photographed video 100 and displays the photographed video 100. Furthermore, the user terminal 12 according to the first embodiment includes: an acquisition means 52 that, when multiple figures are photographed by the camera 34, acquires the types of multiple figures based on the photographed video 100 captured by the camera 34, a determination means 56 that determines effect data based on the combination of the types of multiple figures acquired by the acquisition means 52 and the positional relationship of the multiple figures in the photographed video 100, and a display control means 58 that adds the effect data determined by the determination means 56 to the photographed video 100 and displays the photographed video 100.

[0128] According to this configuration, it is possible to display the photographed video 100 to which effect data corresponding to the combination and arrangement relationship has been added. As a result, by arranging a plurality of figures in a predetermined combination and arrangement relationship, the user can add to the photographed video 100 unique effect data that can only be obtained with that combination and arrangement relationship. Furthermore, by changing the combination and arrangement relationship, different effect data can be added to the photographed video 100. In this way, a variety of effect data according to the situation can be added to the photographed video 100, thereby enhancing the creativeness of the presentation of the photographed video 100.

[0129] In addition, in this embodiment, the determination means 56 determines the position or size of the display effect to be displayed on the captured image 100 based on at least the layout relationship, and the display control means 58 displays the captured image 100 with the display effect added at the position or size determined by the determination means 56.

[0130] According to this configuration, the position or size of the display effect can be set to a position or size that is suitable for the layout relationship, or can be changed to a variety of positions or sizes depending on the layout relationship.

[0131] In addition, in this embodiment, when the layout relationship changes, the determination means 56 determines multiple effect data corresponding to the changing layout relationship, and the display control means 58 switches the effect data to be added to the captured video 100 to one of the multiple effect data determined by the determination means 56 in accordance with the change in the layout relationship.

[0132] With this configuration, the user can change the positional relationship of the figures, thereby switching the effect data added to the photographed video 100 to effect data that corresponds to the positional relationship. Therefore, instead of effect data that simply follows the change in the positional relationship, a variety of effect data can be added to the photographed video 100, such as effect data that is specific to a specific positional relationship, and the presentation of the photographed video 100 can be made more creative.

[0133] In addition, in this embodiment, when an arrangement sequence indicating a series of sequences including arrangement relationships that change in a changing order is a predetermined arrangement sequence, the determination means 56 determines effect data corresponding to the predetermined arrangement sequence, and the display control means 58 adds the effect data corresponding to the predetermined arrangement sequence determined by the determination means 56 to the photographed video 100 and displays the photographed video 100.

[0134] According to this configuration, the user can change the positional relationship of the multiple figures over time to form a predetermined positional sequence, thereby adding, as effect data corresponding to the predetermined positional sequence, for example, a reenactment scene from a predetermined anime to the photographed video 100. Therefore, unique effect data can be added to the photographed video 100 in response to the formation of the predetermined positional sequence, thereby further enhancing the creativeness of the presentation of the photographed video 100.

[0135] In addition, in this embodiment, the user terminal 12 as a computer further functions as a storage means 50 that stores multiple pieces of effect data and, when a photographed video 100 with effect data added thereto is displayed by a display control means 58, stores information indicating multiple figures included in the photographed video 100 as displayed figures, and a setting means 62 that performs settings to change or add multiple pieces of effect data stored in the storage means 50 when the displayed figures satisfy predetermined collection conditions.

[0136] With this configuration, the effect data options can be changed or added depending on the type or number of displayed figures, etc. Therefore, a wider variety of effect data can be added to the photographed video 100. Furthermore, a user who wants to change or add the effect data options will try to collect figures so that the type or number of displayed figures satisfies a predetermined collection condition, which increases the user's desire to collect and also leads to an increase in the purchase of figures.

[0137] In addition, in this embodiment, the shape of the figure can be changed by making it movable, attaching and detaching parts, or rearranging parts, and when the shape of the figure changes, the determination means 56 determines multiple effect data corresponding to the changing shape, and the display control means 58 switches the effect data to be added to the captured video 100 to one of the multiple effect data determined by the determination means 56 in accordance with the change in shape.

[0138] According to this configuration, it is possible to switch to one of the plurality of effect data in response to a change in the shape of the figure, and thus the user can add a variety of effect data to the photographed video 100 by changing the shape of the figure.

[0139] In addition, in this embodiment, the effect data includes a background image to be superimposed on the background of the figure, and when the background image is superimposed on the photographed image 100, the display control means 58 switches the image of the figure photographed by the camera 34 to an image of a virtual object, makes a predetermined change, and then switches the image of the virtual object back to the image of the figure photographed by the camera 34.

[0140] According to this configuration, when a background image is superimposed on the background of the figure, the photographed figure is switched to an image of a virtual object and a predetermined change is made, so that the figure, which is actually stationary, appears to be moving within the photographed image 100.

[0141] In this embodiment, the user terminal 12 as a computer further functions as a game control means 64 that controls a predetermined game based on the effect data added to the captured video 100 by the display control means 58.

[0142] This configuration allows various effect data to be linked to a specific video game depending on the situation. Furthermore, users who want to link effect data to a video game will first try to collect figurines in order to add the effect data to the filmed video 100, which will increase users' desire to collect and also encourage them to purchase figurines.

[0143] [Second embodiment] Next, a second embodiment of the present invention will be described with reference to Figures 12 to 15. Hereinafter, the same configurations or functions as those in the first embodiment will be denoted by the same reference numerals, and descriptions thereof will be omitted as appropriate, and differences from the first embodiment will be described in detail.

[0144] <Functional configuration> The augmented reality system according to the second embodiment, like the augmented reality system 1 according to the first embodiment, includes a storage means 50, an acquisition means 52, an analysis means 54, a determination means 56, a display control means 58, a judgment means 60, a setting means 62, and a game control means 64.

[0145] In the second embodiment, the method by which the determination means 56 determines effect data, specifically the correspondence relationship information 50C referenced by the determination means 56, differs from that in the first embodiment. In the second embodiment, the situation information in the correspondence relationship information 50C includes, in addition to or instead of the situation information according to the first embodiment (type of figure, shape of figure, combination, positional relationship, and positional sequence), information on the real world situation such as location, period, and weather, and information on the shooting situation such as the shooting position and posture, and camera work.

[0146] The position as situation information is predetermined position information. The position may be a position in three-dimensional space determined by a three-dimensional position vector, or may be a position determined by a two-dimensional position vector in three-dimensional space, i.e., a position on a two-dimensional plane. The position is indicated, for example, by coordinates including latitude, longitude, and altitude, or by a place name indicating a predetermined location. Note that the altitude may be constant or may change depending on the latitude or longitude.

[0147] The period and weather as situation information is information indicating a predetermined period or weather. The period is indicated by a name indicating a predetermined period, such as the name of a season, such as spring, summer, autumn, or winter, or a predetermined date and time. The weather is indicated by a name indicating a predetermined weather, such as sunny or rainy, or by precipitation amount, temperature, etc.

[0148] The shooting position and shooting attitude as situation information are information that indicates the three-dimensional shooting position and shooting attitude of the camera 34 relative to the three-dimensional figure. The shooting position is the relative position of the camera 34 with respect to the figure. The shooting attitude is the direction (azimuth) and inclination (elevation angle) of the camera 34 with respect to the figure. Furthermore, the camera work as situation information is the camera work of the camera 34 with respect to the figure, and is information that indicates a series of sequences including the shooting position and shooting attitude that change in the order in which they change when the camera's shooting position and shooting attitude change.

[0149] Here, the photographing position and the photographing posture will be conceptually explained with reference to Fig. 12A to Fig. 12D. Fig. 12A to Fig. 12D are diagrams conceptually explaining the photographing position and the photographing posture. Note that Fig. 12A to Fig. 12D illustrate one figure F01, but the present invention is not limited to photographing one figure, and multiple figures may also be photographed. When multiple figures are photographed, the determination means 56 may determine effect data based on the photographing position and the photographing posture for each figure, or may recognize the multiple figures as a single object and determine effect data based on the photographing position and the photographing posture for that object.

[0150] In the photographing position and posture shown in FIG. 12A, the camera 34 of the user terminal 12 is positioned, for example, within 5 cm of the figure F01, and is directed toward the left side of the figure F01.

[0151] In the photographing position and posture shown in FIG. 12B, the camera 34 of the user terminal 12 is positioned, for example, within 5 cm of the figure F01, and is directed toward the back of the figure F01.

[0152] In the photographing position and posture shown in FIG. 12C, the camera 34 of the user terminal 12 is positioned, for example, within 5 cm of the figure F01, and is directed toward the right side of the figure F01.

[0153] In the shooting position and shooting posture shown in Figure 12D, the camera 34 of the user terminal 12 is positioned, for example, 5 to 10 cm away from the figure F01, and the camera 34 is pointed toward the head of the figure F01 so as to overlook the figure F01.

[0154] As described above, a predetermined shooting position and shooting posture are configured by the position, orientation, posture, etc. of the camera 34 relative to the figure F01. Furthermore, when the shooting position and shooting posture change, for example, by the user moving the camera 34, a camerawork is configured that shows a series of sequences including the changing shooting positions and shooting postures in the changing order. For example, when the shooting position and shooting posture change from Figure 12A to Figure 12B to Figure 12C within a predetermined time period, a camerawork is configured that includes the shooting positions and shooting postures shown in Figure 12A, Figure 12B, and Figure 12C in the changing order.

[0155] 13A to 13D are diagrams showing an example of the configuration of correspondence information 50C according to the second embodiment. As shown in FIGS. 13A to 13D, the correspondence information 50C stores effect IDs in association with situation information (location, period, weather, shooting position and posture, camera work). The correspondence information 50C in FIGS. 13A to 13D all show an example in which the type of figure is, for example, "human character A."

[0156] The correspondence relationship information 50C shown in Figures 6A to 6E and Figures 13A to 13D are linked to one another by predetermined situation information or the like. For example, the storage means 50 stores a plurality of pieces of correspondence relationship information 50C shown in Figures 13A to 13D, linked to the type of figure. Note that the storage means 50 may store a plurality of pieces of correspondence relationship information 50C shown in Figures 12A to 12D, linked to other situation information, not limited to the type of figure.

[0157] Furthermore, in the second embodiment, the analysis means 54 analyzes the captured video 100 using a known analysis technique and acquires the shooting position and shooting posture in addition to or instead of the above-mentioned video analysis information. The analysis means 54 estimates and acquires the shooting position and shooting posture using a known technique such as VSLAM (Visual Simultaneous Localization and Mapping). VSLAM uses information acquired by a predetermined device such as the camera 34 and acceleration / orientation sensor 44 of the user terminal 12 to extract feature points of a figure captured by the camera 34 and track the feature points of the figure in real time to estimate the shooting position and shooting posture.

[0158] Furthermore, when the photographing position and the photographing posture change within a predetermined period, for example, the analysis means 54 acquires camerawork based on the photographing position and the photographing posture. For example, when the photographing position and the photographing posture for the figure F01 change sequentially from Figure 12A to Figure 12B to Figure 12C, the analysis means 54 acquires, as camerawork, information including the photographing positions and the photographing postures shown in Figures 12A, 12B, and 12C in the order of change.

[0159] Furthermore, the determination means 56 determines effect data based on the current location, the actual period and weather, the shooting position and posture, camera work, and the like, in addition to or instead of the video analysis information described in the first embodiment.

[0160] For example, the determination means 56 acquires the current location and determines effect data based on the current location. In this case, the determination means 56 refers to the correspondence information 50C in FIG. 13A and identifies a predetermined location that coincides with the current location or is within an approximate range. Then, the determination means 56 identifies an effect ID corresponding to the predetermined location. The current location acquired by the determination means 56 is information indicating the current location when the figure is photographed by the camera 34, i.e., the location of the user terminal 12 that is photographing the photographed video 100. This current location is acquired, for example, based on a position measurement technique based on a GPS signal received by the GPS receiver 46 of the user terminal 12, the IP address of the user terminal 12, etc.

[0161] The determination means 56 also acquires the actual period and weather and determines effect data based on the actual period and weather. In this case, the determination means 56 references the correspondence information 50C in FIG. 13B to identify a predetermined period and weather that matches or is within an approximate range to the actual period and weather. The determination means 56 then identifies an effect ID corresponding to the predetermined period and weather. The period and weather acquired by the determination means 56 is information indicating the period or weather in the real world at the time the photographed video 100 was captured. This actual period or weather is acquired, for example, from the real-time clock of the user terminal 12 or a website connected to the user terminal 12 via the communication network NT.

[0162] Furthermore, the determination means 56 determines effect data based on the shooting position and shooting posture output from the analysis means 54. In this case, the determination means 56 refers to the correspondence relationship information 50C shown in Fig. 13C and identifies the effect ID corresponding to the shooting position and shooting posture.

[0163] Furthermore, when the shooting position and shooting posture change, the determination means 56 determines a plurality of effect data corresponding to the changing shooting position and shooting posture. For example, when the shooting position and shooting posture change, the determination means 56 refers to the correspondence information 50C in Fig. 13C in the same manner as above to identify a plurality of effect IDs corresponding to the changing shooting position and shooting posture.

[0164] Furthermore, when camerawork is output from the analysis means 54, the determination means 56 determines whether the camerawork is a predetermined camerawork stored in the storage means 50. The determination means 56, for example, references the correspondence information 50C in FIG. 13D. If the output camerawork corresponds to the camerawork stored in the correspondence information 50C, the determination means 56 makes a positive determination. Note that the case where the output camerawork corresponds to the camerawork stored in the correspondence information 50C is not limited to a case where the cameraworks match, but may also include, for example, a case where the cameraworks are within a predetermined range of similarity with each other. If the determination means 56 makes a positive determination, it identifies an effect ID corresponding to the predetermined camerawork.

[0165] Furthermore, the determination means 56 may specify an effect ID based on other situation information (type, shape, combination, current location, period, weather, etc.) in addition to the photographing position and photographing posture. In this case, the determination means 56 refers to the correspondence information 50C linked to other situation information (type, shape, combination, current location, period, weather, etc.) and specifies the effect ID corresponding to the photographing position and photographing posture.

[0166] In addition, the determination means 56 determines the output manner (position, size, volume, direction of sound, etc.) of the effect data to be added to the captured video 100, similar to the first embodiment, based on at least the shooting position and shooting posture output from the analysis means 54.

[0167] Furthermore, in the second embodiment, when a special effect mode "C" to "E" is selected, the determination means 56 may identify the effect ID by referring to the correspondence information 50C corresponding to the selected mode. As shown in FIG. 8, the special effect mode "C" is a "shooting position mode" in which effect data corresponding to the shooting position, shooting posture, and camera work is added to the photographed video 100. The special effect mode "D" is a "current position mode" in which effect data corresponding to the current position is added to the photographed video 100. The special effect mode "E" is a "period / weather mode" in which effect data corresponding to a period or weather is added to the photographed video 100.

[0168] Furthermore, in the second embodiment, the determination means 60 determines whether or not to add effect data to the captured video 100 based on the current position and the position stored in association with the type of figure, etc. For example, the determination means 60 refers to the correspondence information 50C in FIG. 13A linked to the type of figure. Then, the determination means 60 determines whether or not the current position acquired by the determination means 56 corresponds to a predetermined position stored in the correspondence information 50C. Note that the case where the current position corresponds to a predetermined position is not limited to the case where the respective positions coincide with each other, but may also include the case where the respective positions are within a predetermined approximate range on a two-dimensional plane, for example, where a position is included within a predetermined range including the current position.

[0169] Furthermore, in the second embodiment, when the determination means 60 determines that the current position acquired by the determination means 56 corresponds to the predetermined position stored in the correspondence information 50C, the display control means 58 adds effect data to the photographed video 100 and displays the photographed video 100. On the other hand, when the determination means 60 determines that the current position acquired by the determination means 56 does not correspond to the predetermined position stored in the correspondence information 50C, the display control means 58 displays the photographed video 100 without adding effect data to the photographed video 100.

[0170] Furthermore, when the shooting position and shooting posture change, the display control means 58 switches the effect data to be added to the shot video 100 to one of the plurality of effect data determined by the determination means 56 in accordance with the change. Furthermore, when a predetermined camerawork is formed, the display control means 58 adds effect data corresponding to the camerawork to the shot video 100 and displays the shot video 100.

[0171] <Processing flow> The processing flow performed by each functional component in the augmented reality system according to the second embodiment is the same as the flowchart shown in Fig. 9. In the second embodiment, the process of acquiring effect data corresponding to the special effect mode in step SP18 of Fig. 9 differs from that in the first embodiment. In the first embodiment, the processing flow when modes "A" and "B" are selected among the special effect modes shown in Fig. 8 is described, but in the second embodiment, the processing flow when modes "C", "D", and "E" are selected among the special effect modes shown in Fig. 8 is described.

[0172] In the second embodiment, the processing flow of step SP18 is shown in the flowchart shown in Fig. 14 instead of the flowchart shown in Fig. 10. Fig. 14 is a flowchart showing an example of the processing flow of step SP18 in the second embodiment.

[0173] (Step SP60) The determination means 56 acquires the current position where the photographed video 100 is being acquired in the process of step SP10, and then the process proceeds to the process of step SP62.

[0174] (Step SP62) The determination means 60 refers to the correspondence information 50C in Figure 13A linked to the type of figure acquired in the processing of step SP12. Next, the determination means 60 determines whether or not to add effect data to the photographed video 100 based on the current position acquired in the processing of step SP60 and the position stored in the referenced correspondence information 50C in Figure 13A. If this determination is negative, the processing proceeds to the processing of step SP64. If this determination is positive, the processing proceeds to the processing of step SP66.

[0175] (Step SP64) The display control means 58 displays the photographed video 100 on the display 32A without adding effect data to the photographed video 100. Then, the series of processes shown in FIG.

[0176] (Step SP66) The determining means 56 acquires the period and weather, and then the process proceeds to step SP68.

[0177] (Step SP68) The analysis means 54 sequentially analyzes the photographed video 100 acquired in the processing of step SP10, and acquires the photographing position and photographing posture. Furthermore, the analysis means 54 acquires the camerawork when the photographing position and photographing posture change. Then, the processing proceeds to the processing of step SP70.

[0178] (Step SP70) When camerawork is acquired in the processing of step SP68, the determination means 56 refers to the correspondence information 50C of FIG. 13D linked to the acquired situation information (the type of figure, current location, period, and weather acquired in the processing of steps SP12, SP60, and SP66, respectively). Next, the determination means 56 determines whether the acquired camerawork is the predetermined camerawork stored in the correspondence information 50C. If the determination is affirmative, the processing proceeds to processing of step SP72. If the determination is negative, the processing proceeds to processing of step SP74. Also, if camerawork is not acquired in the processing of step SP68, the processing proceeds to processing of step SP74.

[0179] (Step SP72) The determination means 56 acquires effect data corresponding to the camerawork, etc. (camerawork, type of figure, current position, and period / weather). Specifically, the determination means 56 identifies an effect ID corresponding to the camerawork for which a positive determination was made in the processing of step SP70 in the correspondence relationship information 50C of FIG. 13D referenced in the processing of step SP70. Next, the determination means 56 references the effect information 50B of FIG. 5 to identify an effect data path corresponding to the identified effect ID. Next, the determination means 56 acquires the effect data stored in the storage destination indicated by the effect data path. Then, the series of processing shown in FIG. 14 ends.

[0180] (Step SP74) The determination means 56 acquires effect data corresponding to the shooting position and shooting posture, etc. (shooting position and shooting posture, type of figure, current location, and period / weather). Specifically, the determination means 56 refers to the correspondence information 50C in FIG. 13C linked to the acquired situation information (type of figure, current location, period / weather acquired in the processing of steps SP12, SP60, and SP66, respectively). Next, the determination means 56 identifies the effect ID corresponding to the shooting position and shooting posture acquired in the processing of step SP68. Next, the determination means 56 refers to the effect information 50B in FIG. 5 and identifies the effect data path corresponding to the identified effect ID. Next, the determination means 56 acquires the effect data stored in the storage destination indicated by the effect data path. Then, the series of processes shown in FIG. 14 ends.

[0181] <Screen display example> FIG. 15 is a diagram showing an example of a captured video 100 displayed on the display 32A of the user terminal 12 under the display control of the display control means 58 in the second embodiment.

[0182] For example, when a predetermined camerawork is configured in which the shooting position and shooting posture change sequentially as shown in Figures 12A, 12B, and 12C, moving from the left side of the figure F01 around to the back and then to the right side, a display effect AR120 corresponding to the camerawork is acquired in the processing of step SP18. In this case, as shown in Figure 15, in response to the configuration of the camerawork, the shot video 100 with the display effect AR120 superimposed thereon is displayed on the display 32A.

[0183] Although not shown, the switching of display effects in response to changes in the shooting position and shooting posture is performed in a manner similar to the switching of display effects in response to changes in the positional relationship described in the first embodiment with reference to Figures 11C and 11D. That is, switching of display effects does not simply follow changes in the shooting position and shooting posture, but also includes switching to a completely different display effect in response to changes in the shooting position and shooting posture. Also, as in the first embodiment, after the photographed video 100 is displayed with a display effect based on the shooting position and shooting posture superimposed, the photographed video of the figure F01 may be switched to the display effect of a virtual object, and after a predetermined change, the display effect of the virtual object may be switched back to the photographed video of the figure F01.

[0184] <Action and effect> As described above, in the second embodiment, when the three-dimensional shooting position and shooting posture of the camera 34 relative to the three-dimensional figure changes, the determination means 56 determines multiple pieces of effect data based on the shooting position and shooting posture, and the display control means 58 switches the effect data to be added to the video to one of the multiple pieces of effect data determined by the determination means 56 in accordance with the change in the shooting position and shooting posture.

[0185] With this configuration, the user can change the shooting position and shooting posture of the figure, thereby switching the effect data added to the photographed video 100 to effect data that corresponds to the shooting position and shooting posture. Therefore, instead of effect data that simply follows changes in the shooting position and shooting posture, a variety of effect data can be added to the photographed video 100, such as effect data that is specific to a specific shooting position and shooting posture, thereby further enhancing the creativeness of the presentation of the photographed video 100.

[0186] In addition, in the second embodiment, when camerawork showing a series of sequences including changing shooting positions and shooting postures in a changing order is a predetermined camerawork, the determination means 56 determines effect data corresponding to the predetermined camerawork, and the display control means 58 adds the effect data corresponding to the predetermined camerawork determined by the determination means 56 to the shot video 100 and displays the shot video 100.

[0187] According to this configuration, by changing the shooting position and shooting posture so as to compose a predetermined camerawork, the user can add, as effect data corresponding to the predetermined camerawork, for example, a reenactment scene of a predetermined anime to the shot video 100. Therefore, unique effect data can be added to the shot video 100 in response to the shooting position and shooting posture that have been changed over time constituting the predetermined camerawork, thereby further enhancing the ingenuity of the presentation of the shot video 100.

[0188] In the second embodiment, the determining means 56 determines the effect data based on the current position when the camera 34 photographs the figure.

[0189] According to this configuration, different effect data can be added to the photographed video 100 depending on the current location. Therefore, by executing the augmented reality app at a predetermined location, the user can view the photographed video 100 to which unique effect data that is only added at the predetermined location has been added.

[0190] In addition, in the second embodiment, the user terminal 12 as a computer further functions as a judgment means 60 that judges whether or not to add effect data to the photographed video 100 based on the current position when the figure is photographed by the camera 34 and the position stored in correspondence with the type of figure, and if the judgment means 60 makes a positive judgment, the display control means 58 adds the effect data to the photographed video 100 and displays the photographed video 100, and if the judgment means 60 makes a negative judgment, displays the photographed video 100 without adding the effect data to the photographed video 100.

[0191] With this configuration, for example, if the current position does not correspond to the stored position, it is possible to prevent the effect data from being added, thereby limiting the locations where the effect data can be added to the photographed image 100 of the figure and displayed to predetermined locations.

[0192] In the second embodiment, the determining means 56 determines the effect data based on the actual period or weather.

[0193] According to this configuration, different effect data can be added to the photographed video 100 depending on the actual period or weather.

[0194] <Modification> The present invention is not limited to the above-described embodiments. In other words, variations on the above-described embodiments, which are appropriately modified by a person skilled in the art, are also included within the scope of the present invention as long as they incorporate the features of the present invention. Furthermore, the elements of the above-described embodiments and the modifications described below can be combined to the extent technically possible, and such combinations are also included within the scope of the present invention as long as they incorporate the features of the present invention.

[0195] For example, in the first embodiment, an example of determining effect data based on combinations and positional relationships was described, and in the second embodiment, an example of determining effect data based on the shooting position and shooting posture was described. However, the information used to determine effect data is not limited to this. For example, effect data may be determined based on a predetermined combination of various information acquired as video analysis information of the captured video 100, such as determining effect data based on the shooting position and shooting posture in addition to the combinations and positional relationships. Furthermore, effect data may be determined based on real-world information such as the current location, period, and weather in addition to video analysis information such as combinations and positional relationships. In this case, as in the above embodiments, effect data corresponding to desired real-world information is identified and acquired by referencing information storing the correspondence between the desired real-world information and effect data.

[0196] Furthermore, in the first embodiment, an example was described in which effect data was determined based on the combination and positional relationship when multiple figures were shot, but when a single figure was shot, for example, effect data may be determined based on the positional relationship and positional sequence of the single figure within the shot video 100. In this case, the specific determination method is the same as in the case of multiple figures described in the above embodiment. That is, effect data may be changed depending on the position and orientation of the figure included in the shot video 100 within the shot video 100, or specific effect data may be added to the shot video 100 when the figure moves to a predetermined position and orientation within a predetermined time.

[0197] Furthermore, regardless of whether the normal presentation mode or special presentation mode is set, the presentation method may be changed automatically based on predetermined information, etc. For example, regardless of the setting, the method for determining effect data may be automatically changed according to the analysis results of the photographed video 100. In this case, if the analysis results indicate that the photographed video 100 contains a single figure, the determination means 56 determines effect data corresponding to the type and shape of the figure. On the other hand, if the analysis results indicate that the photographed video 100 contains multiple figures, the determination means 56 determines effect data corresponding to their combination and positional relationship. Furthermore, if the analysis results indicate that the positional relationship, shooting position, and shooting attitude will not change, the determination means 56 determines effect data corresponding to the positional relationship, shooting position, and shooting attitude. On the other hand, if the analysis results indicate that the positional relationship, shooting position, and shooting attitude will change, the determination means 56 determines effect data corresponding to the positional sequence and camerawork resulting from the change.

[0198] Furthermore, the method of analyzing the captured video 100 by the analysis means 54 is not limited to the above method. The analysis means 54 may acquire video analysis information by, for example, performing video analysis of the captured video 100 using a predetermined trained model. The predetermined trained model is constructed based on a predetermined machine learning algorithm using, for example, previously collected figures' videos and three-dimensional model data as training data, and is stored in the storage means 50. For example, the analysis means 54 inputs the captured video into a predetermined inference program using the trained model, and acquires video analysis information output as an inference result by the inference program.

[0199] Furthermore, the display screen including the photographed video 100 displayed on the display 32A of the user terminal 12 by the display control of the display control means 58 is not limited to the example described in the above embodiment. For example, when displaying the photographed video 100, the display control means 58 may output suggestive information suggesting a predetermined layout relationship, layout sequence, shooting position and shooting posture, and camerawork. The suggestive information may be, for example, a predetermined display or audio. In this case, the user can configure a predetermined layout relationship, layout sequence, shooting position and shooting posture, and camerawork by moving the figures and camera 34 according to the output suggestive information. Furthermore, the display control means 58 may output or switch effect data in response to, for example, a user's operation on the photographed video 100 or display effect displayed on the display 32A.

[0200] Furthermore, the correspondence relationship information 50C as shown in Figures 7A to 7E and Figures 13A to 13D may be linked to each other in a predetermined relationship using various information including predetermined situation information, without being limited to the linking relationship described above.

[0201] The display effect may also be a display effect (image) based on a three-dimensional model such as a polygon. In this case, in order to superimpose the display effect correctly in the orientation and depth direction of the photographed object such as a figure, video analysis information or the like that associates the image of the figure or the like with three-dimensional model data with the placement position of the display effect may be used to identify areas where the display effect is hidden by the photographed object such as a figure, and a partial erasure process may be performed, such as partially masking the display effect so that this area is not displayed. Furthermore, the display effect may be animated by changing shape, movement, or deformation, or the above process may be repeated to ensure the display effect is correctly superimposed even when the photographing position or the orientation of the photographed object such as a figure is changed. [Explanation of symbols]

[0202] 10: Server device (computer, information processing device), 12: User terminal (computer), 34: Camera (photographing means), 50: Storage means, 52: Acquisition means, 56: Determination means, 58: Display control means, 60: Determination means, 62: Setting means, 64: Game control means

Claims

1. Computer, an acquisition means for acquiring, when a plurality of objects are photographed by the photographing means, identification information of the plurality of objects based on the images photographed by the photographing means; a determination means for determining additional information based on a combination of the identification information of the plurality of objects acquired by the acquisition means and a positional relationship of the plurality of objects in the video, and, when the positional relationship changes, acquiring a positional sequence showing a series of sequences including the positional relationship that changes over time in a predetermined order within a predetermined time period, and, when the acquired positional sequence is a predetermined positional sequence, determining additional information corresponding to the predetermined positional sequence; a display control means for adding the additional information determined by the determination means to the image and displaying the image; A program to function as a

2. The display control means displays suggestion information suggesting the predetermined arrangement sequence. The program according to claim 1.

3. the determining means determines a position or a size of the additional information to be displayed on the video based on at least the layout relationship; the display control means displays the image to which the additional information has been added at the position or size determined by the determination means. The program according to claim 1.

4. the determining means determines, when the layout relationship changes, a plurality of pieces of additional information corresponding to the changing layout relationship; The display control means controls additional information to be added to the video in accordance with the change in the positional relationship, Switching to one of the plurality of pieces of additional information determined by the determining means; The program according to any one of claims 1 to 3.

5. The computer a storage means for storing a plurality of pieces of additional information and, when a video to which the additional information has been added is displayed by the display control means, for storing information indicating a plurality of objects included in the video as displayed objects; a setting means for changing or adding the plurality of pieces of additional information stored in the storage means when the displayed object satisfies a predetermined condition; Further function as The program according to any one of claims 1 to 4.

6. the determining means determines a plurality of pieces of additional information based on the photographing position and the photographing attitude of the photographing means when the three-dimensional photographing position and the photographing attitude of the photographing means change with respect to the three-dimensional object; the display control means switches the additional information to be added to the video to one of the plurality of pieces of additional information determined by the determination means in accordance with changes in the shooting position and the shooting posture. The program according to any one of claims 1 to 5.

7. the determining means determines additional information corresponding to a predetermined camerawork when the camerawork showing a series of sequences including the photographing positions and the photographing postures that change in a changing order is a predetermined camerawork; the display control means adds additional information corresponding to the predetermined camerawork determined by the determination means to the video and displays the video; The program according to claim 6.

8. The object can change its shape by being movable, by attaching or detaching parts, or by rearranging parts; when the shape of the object changes, the determining means determines a plurality of pieces of additional information corresponding to the changing shape; the display control means switches the additional information to be added to the video image to one of the plurality of pieces of additional information determined by the determination means in response to the change in the shape. The program according to any one of claims 1 to 7.

9. the determining means determines the additional information based on a current position when the object is photographed by the photographing means. The program according to any one of claims 1 to 8.

10. the determining means determines the additional information based on a current period or weather. The program according to any one of claims 1 to 9.

11. the additional information includes a background image to be superimposed on a background of the object, when the background image is superimposed on the image, the display control means switches the image of the object photographed by the photographing means to an image of a virtual object, and after making a predetermined change, switches the image of the virtual object back to the image of the object photographed by the photographing means. The program according to any one of claims 1 to 10.

12. The computer: a game control means for controlling a predetermined game based on the additional information added to the image by the display control means; The program according to any one of claims 1 to 11, further functioning as:

13. an acquisition means for acquiring, when a plurality of objects are photographed by the photographing means, identification information of the plurality of objects based on the images photographed by the photographing means; a determination means for determining additional information based on a combination of the identification information of the plurality of objects acquired by the acquisition means and a positional relationship of the plurality of objects in the video, and, when the positional relationship changes, acquiring a positional sequence indicating a series of sequences including the positional relationship that changes over time in a predetermined order within a predetermined time period, and, when the acquired positional sequence is a predetermined positional sequence, determining additional information corresponding to the predetermined positional sequence; a display control means for adding the additional information determined by the determination means to the image and displaying the image; An information processing device comprising:

14. A computer, an acquisition means for acquiring, when a plurality of objects are photographed by the photographing means, identification information of the plurality of objects based on the images photographed by the photographing means; a determination means for determining additional information based on a combination of the identification information of the plurality of objects acquired by the acquisition means and a positional relationship of the plurality of objects in the video; a display control means for adding the additional information determined by the determination means to the image and displaying the image; a determination means for determining whether or not to add the additional information to the image based on a current position when the object is photographed by the photographing means and a position stored in association with identification information of the object; It functions as When the determination means makes a positive determination, the display control means adds the additional information to the image and displays the image, and when the determination means makes a negative determination, the display control means displays the image without adding the additional information to the image. program.

15. An acquisition means for acquiring identification information of a plurality of objects based on images captured by said image capturing means when said plurality of objects are captured by said image capturing means; a determination means for determining additional information based on a combination of the identification information of the plurality of objects acquired by the acquisition means and a positional relationship of the plurality of objects in the video; a display control means for adding the additional information determined by the determination means to the image and displaying the image; a determination means for determining whether or not to add the additional information to the image based on a current position when the object is photographed by the photographing means and a position stored in association with identification information of the object, When the determination means makes a positive determination, the display control means adds the additional information to the image and displays the image, and when the determination means makes a negative determination, the display control means displays the image without adding the additional information to the image. Information processing device.

Images