Content display system, information terminal, information processing apparatus, and program

Abstract

To provide a technique for easily modifying the content of additional information to be displayed.SOLUTION: A content display system includes: a real-world object that represents superimposing virtual content; and an information terminal which acquires a first image including the real-world object and causes a display unit to display a second image formed by superimposing virtual content corresponding to user information on a user of a terminal associated with the real-world object on the first image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a content display system, an information terminal, an information processing device, and a program. [Background technology]

[0002] 2. Description of the Related Art A technology called Augmented Reality (AR) is known that blends the real world with the virtual world by adding additional virtual information to a real landscape.

[0003] For example, Patent Document 1 discloses a technique for determining a superimposition area for additional information (eg, advertising information) on a video, and superimposing the additional information on the determined superimposition area. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2012-10117 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in many cases, advertising as a sales promotion measure in stores is limited to simply listing information about products and services on craft tools made of paper or polypropylene (so-called POP (point of purchase advertising)). Therefore, it is difficult to make changes to such information once it has been printed and processed.

[0006] In view of the above, an object of the present disclosure is to provide a technique that enables easy change of the display content of additional information. [Means for solving the problem]

[0007] A content display system according to one embodiment of the present disclosure includes a real-world object indicating that virtual content is to be overlaid, and an information terminal that acquires a first image including the real-world object and displays on a display unit a second image in which virtual content corresponding to the real-world object and corresponding to user information related to a user of the terminal is overlaid on the first image. Effect of the Invention

[0008] According to one aspect of the present disclosure, virtual content corresponding to the real-world object and corresponding to user information is superimposed on an image including the real-world object, thereby making it possible to easily change the display content of additional information according to the user information. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram illustrating a schematic configuration example of an advertisement delivery system according to an embodiment of the present disclosure. [Diagram 2] FIG. 1 is a block diagram illustrating an example of a hardware configuration of a user device according to an embodiment of the present disclosure. [Diagram 3] FIG. 1 is a block diagram illustrating an example of a functional configuration of a user device according to an embodiment of the present disclosure. [Figure 4] FIG. 1 is a block diagram illustrating an example of a hardware configuration of an advertisement distribution device according to an embodiment of the present disclosure. [Diagram 5] FIG. 1 is a block diagram illustrating an example of a functional configuration of an advertisement distribution device according to an embodiment of the present disclosure. [Figure 6] FIG. 1 is an external perspective view showing an example of a fixture according to an embodiment of the present disclosure. [Figure 7A] FIG. 1 is a diagram for explaining an example of access by a user device to an advertisement distribution device according to an embodiment of the present disclosure. [Figure 7B] FIG. 1 is a diagram illustrating an example of sending login information by a user device according to an embodiment of the present disclosure. [Figure 8A] FIG. 1 is a diagram for explaining an example of display of display data on a user device according to an embodiment of the present disclosure. [Figure 8B]FIG. 1 is a diagram for explaining an example of display of display data on a user device according to an embodiment of the present disclosure. [Figure 9] FIG. 1 is a diagram showing a data table that is an example of association information between AR content and user information according to an embodiment of the present disclosure. [Figure 10A] 1 is a flowchart illustrating an example of the operation of a user device according to an embodiment of the present disclosure. [Figure 10B] 1 is a flowchart illustrating an example of the operation of a user device according to an embodiment of the present disclosure. [Figure 10C] 1 is a flowchart illustrating an example of the operation of a user device according to an embodiment of the present disclosure. [Figure 10D] 1 is a flowchart illustrating an example of the operation of a user device according to an embodiment of the present disclosure. [Figure 11A] 1 is a flowchart illustrating an example of an operation of an advertisement distribution device according to an embodiment of the present disclosure. [Figure 11B] 1 is a flowchart illustrating an example of an operation of an advertisement distribution device according to an embodiment of the present disclosure. [Figure 11C] 1 is a flowchart illustrating an example of an operation of an advertisement distribution device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. However, the scope of the invention is not limited to the examples shown in the drawings. In the following description, the same reference numerals are used to designate components having the same functions and configurations, and the description thereof will be omitted.

[0011] (Embodiment) <Summary> According to an embodiment of the present disclosure, a content display system (e.g., advertisement distribution system 1 described later) includes a real-world object (e.g., fixture 30 described later or advertisement surface 31 that is part of fixture 30) indicating that virtual content (e.g., AR content) is to be superimposed, and an information terminal (e.g., user device 10 described later). The information terminal acquires a first image including the real-world object, for example, via an imaging device (e.g., imaging device 15 described later) included in the information terminal. The information terminal displays, on a display unit, a second image in which virtual content corresponding to the real-world object and corresponding to user information (e.g., age, gender, etc.) related to the user of the information terminal is superimposed on the first image.

[0012] The content display system further includes an information processing device (for example, an advertisement distribution device 20 described later). The information processing device generates a second image in response to recognition of a real-world object in a first image transmitted from the information terminal to the information processing device, and transmits the second image to the information terminal.

[0013] In this way, the content display system displays virtual content corresponding to the real-world object and corresponding to the user information, superimposed on an image including the real-world object, thereby making it possible to easily change the display content of the additional information according to the user information.

[0014] <Advertisement distribution system configuration> FIG. 1 is a block diagram illustrating a schematic configuration example of an advertisement delivery system according to an embodiment of the present disclosure.

[0015] As shown in Fig. 1, the advertisement delivery system 1 includes a user device 10, an advertisement delivery device 20, a fixture 30, an associated service providing device 40, and a network 50. The user device 10, the advertisement delivery device 20, and the associated service providing device 40 are communicatively connected to each other via the network 50. The network 50 may be a local area network (LAN), the Internet, a virtual private network (VPN), a mobile communication network, or a combination of these. The advertisement delivery system 1 is an example of a content display system according to the present disclosure.

[0016] The user device 10 has a configuration capable of connecting to the advertisement distribution device 20 via, for example, a web browser or a dedicated application installed on the user device 10. For example, the following processing of the user device 10 may be performed via the web browser or the dedicated application.

[0017] The user device 10 acquires image data (also simply referred to as an image) generated by a camera (for example, an imaging device 15 described below) included in the user device 10, and transmits the image data to the advertisement distribution device 20 via the network 50. Here, such an image may include a part or all of the fixtures 30. The part or all of the fixtures 30 are examples of real-world objects that indicate the superimposition of virtual content (hereinafter also referred to as AR content, AR content data, or AR objects), as described below.

[0018] The user device 10 receives an image (hereinafter also referred to as display data) output (transmitted) by the advertisement distribution device 20 via the network 50. Here, such an image may be an image in which AR content is superimposed by the advertisement distribution device 20 on the image transmitted by the user device 10, or may be an image transmitted by the user device 10 to the advertisement distribution device 20. The user device 10 displays the received image.

[0019] 1 shows one user device 10, there may be multiple user devices 10. The user device 10 may be called an information terminal, a user terminal, a terminal, a client, a client device, or the like.

[0020] The advertisement distribution device 20 is installed in a cloud environment to host an advertisement distribution web service. The advertisement distribution device 20 is accessed by the user device 10 when the user device 10 specifies a uniform resource locator (URL) that hosts the advertisement distribution web service via a web browser or a dedicated application. As a result, the advertisement distribution web service is provided to the user device 10.

[0021] The advertisement distribution device 20 receives login information (login ID and password) of the user of the user device 10 transmitted by the user device 10 via the network 50. The login information is information for using a service (an integrated service) that is integrated with the advertisement distribution web service provided by the advertisement distribution device 20. If authentication is successful in the authentication process by the integrated service providing device 40 using the login information (information necessary for authentication process), the advertisement distribution device 20 receives (acquires) user information on the user corresponding to the login information from the integrated service providing device 40 via the network 50.

[0022] The advertisement distribution device 20 receives the image transmitted by the user device 10 via the network 50. As described above, such an image may include a part or all of the fixture 30.

[0023] When the advertisement distribution device 20 recognizes a target for superimposing AR content (also called an AR target or a recognition target) in an image, that is, a real-world object indicating that virtual content is to be superimposed, the advertisement distribution device 20 performs the following in response to this recognition: the advertisement distribution device 20 generates display data (image) in which the AR content associated with the AR target in the image is superimposed, for example, at the AR target position in the image, at a position within the AR target, or at a predetermined position a predetermined distance away from the AR target. At this time, if the advertisement distribution device 20 has acquired user information, it generates display data in which the AR content corresponding to the user information is superimposed.

[0024] On the other hand, if the advertisement distribution device 20 does not recognize the AR target in the image, it determines to use the received image as display data.

[0025] The advertisement distribution device 20 outputs (transmits) the generated display data to the user device 10 via the network 50.

[0026] Although one advertisement distribution device 20 is shown in FIG. 1, the functions of the advertisement distribution device 20 may be distributed among a plurality of devices, for example, for load balancing.

[0027] In this embodiment, the fixture 30 is a product shelf for sales promotion on which a product (e.g., cosmetics) is placed. On the fixture 30, advertising information (illustrations, photos, characters, etc.) of the product, a two-dimensional code (e.g., QR code (registered trademark)) including information for accessing the advertisement distribution device 20 (the above-mentioned URL), and the like are written. The two-dimensional code may or may not include information (called transition information) for transitioning to a screen that prompts the user to input login information for using a linked service immediately after the user device 10 accesses the advertisement distribution device 20.

[0028] The linked service providing device 40 provides the linked service. The linked service provided by the linked service providing device 40 is, for example, a service including a BtoC (Business to Customer) electronic commerce (EC) service operated by a manufacturer / distributor who manufactures and sells various products including the above-mentioned product.

[0029] The linked service providing device 40 receives the login information of the user of the user device 10 transmitted by the advertisement distribution device 20 via the network 50. The linked service providing device 40 executes an authentication process using the received login information, and if the authentication is successful, transmits user information about the user corresponding to the login information to the advertisement distribution device 20 via the network 50.

[0030] <User device configuration> 2 is a block diagram showing an example of a hardware configuration of a user device according to an embodiment of the present disclosure. The user device 10 may be realized by any computer or information processing device, such as a tablet, a smartphone, a notebook PC (Personal Computer), or a wearable device such as smart glasses or a head-mounted display.

[0031] 2, the user device 10 may include a storage device 11, a processing device 12, a user interface (UI) device 13, a communication device 14, an imaging device (photographing device) 15, and a bus 16. The storage device 11, the processing device 12, the UI device 13, the communication device 14, and the imaging device 15 may be connected to each other via the bus 16.

[0032] The programs or instructions for implementing the functions and processes of the user device 10 described above and below may be downloaded from some external device (e.g., a server) via a network, etc. Also, such programs or instructions may be provided from a removable storage medium such as a CD-ROM (Compact Disc Read Only Memory) or a flash memory.

[0033] Storage device 11 is realized by a random access memory (RAM), a flash memory, a hard disk drive, etc., and stores installed programs or instructions as well as files, data, etc. used in executing the programs or instructions. Storage device 11 may include a non-transitory storage medium.

[0034] The processing device 12 may be realized, for example, by a general-purpose processor or controller (circuit), or may be realized by a dedicated processor or controller (circuit). When realized by a general-purpose processor, the processing device 12 may be realized by one or more central processing units (CPUs), graphics processing units (GPUs), processing circuitry, etc., which may be composed of one or more processor cores. In this case, the processing device 12 executes the functions and processes of the user device 10 described above and below according to a program or instruction stored in the storage device 11, data such as parameters used to execute the program or instruction, etc. Such a program may be a program that causes the computer or processing device 12 to function as the user device 10 according to the present disclosure when executed by the computer or processing device 12. In addition, such a program may be a program that causes the computer or processing device 12 to execute a method related to advertisement delivery according to the present disclosure when executed by the computer or processing device 12.

[0035] The UI device 13 may be composed of input devices such as a keyboard, a mouse, a camera, a microphone, etc., output devices such as a display, a speaker, a headset, a printer, etc., and input / output devices such as a haptic device, a touch panel, etc., and realizes an interface between a user and the user device 10. For example, a user operates the user device 10 by operating a graphical user interface (GUI) displayed on a display or a touch panel using a keyboard, a mouse, etc.

[0036] The communication device 14 may be realized by various communication circuits that execute communication processes with external devices, the Internet, a LAN, a VPN, or other communication networks.

[0037] The imaging device 15 may be realized by including a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The imaging device 15 captures an image of a scene around the user device 10 to generate image data. The imaging device 15 outputs the generated image data to a communication control unit 101 and an image acquisition unit 102 described later, and stores the image data in the storage device 11 (an image storage unit 111 described later). The imaging device 15 generates image data including the above-mentioned two-dimensional code, a part or all of the fixture 30, etc.

[0038] The above-described hardware configuration of the user device 10 is merely an example, and the user device 10 according to the present disclosure may be realized by any other appropriate hardware configuration.

[0039] FIG. 3 is a block diagram illustrating an example of a functional configuration of a user device according to an embodiment of the present disclosure.

[0040] 3, the user device 10 includes a communication control unit 101, an image acquisition unit 102, a display data acquisition unit 103, a display control unit 104, an image storage unit 111, and a display data storage unit 112. The communication control unit 101, the image acquisition unit 102, the display data acquisition unit 103, and the display control unit 104 may be realized by the processing device 12 as hardware, as software (e.g., an application executed on a web browser or a dedicated application), or as a combination of hardware and software. The image storage unit 111 and the display data storage unit 112 may be realized by the storage device 11.

[0041] The communication control unit 101 acquires as an input an image including a two-dimensional code output by the imaging device 15, and connects to the advertisement distribution device 20 by accessing the URL included in the two-dimensional code via the communication device 14 (if the user permits access). Note that this process of the communication control unit 101 does not necessarily need to be realized via a web browser or a dedicated application, and may be realized via a general-purpose function of the user device 10.

[0042] When the two-dimensional code includes transition information, when the user inputs login information for using the linked service on the screen displayed immediately after the user device 10 accesses the URL, the communication control unit 101 accepts the input login information and transmits the accepted login information to the advertising distribution device 20 via the communication device 14.

[0043] The communication control unit 101 controls communication with the advertisement distribution device 20 by the image acquisition unit 102 and the display data acquisition unit 103 .

[0044] When the user inputs to the user device 10 via the UI device 13 to end image capture (photography) by the imaging device 15 in order to end advertisement display, the communication control unit 101 transmits image capture end information to the advertisement distribution device 20.

[0045] When a connection with the advertisement distribution device 20 is established by the communication control unit 101, the image acquisition unit 102 acquires, as an input, an image output by the imaging device 15. For example, the image acquisition unit 102 acquires a first image including a real-world object indicating that virtual content is to be superimposed. The image acquisition unit 102 transmits the acquired image to the advertisement distribution device 20 via the communication device 14. Note that the image acquisition unit 102 may transmit, together with the image, coordinate information when the image is displayed on the display device of the user device 10 to the advertisement distribution device 20.

[0046] The display data acquisition unit 103 acquires (receives) display data in which the AR content is or is not superimposed on the transmitted image from the advertisement distribution device 20 via the communication device 14. The display data acquisition unit 103 outputs the acquired display data to the display control unit 104 and stores it in the display data storage unit 112.

[0047] The display control unit 104 receives the display data output by the display data acquisition unit 103 as an input, generates a screen including the received display data, and displays the screen on a display device (display unit) included in the UI device 13. For example, the display control unit 104 displays on the display unit a second image in which virtual content corresponding to a real-world object and corresponding to user information related to the user of the user device 10 is superimposed on a first image.

[0048] The image storage unit 111 stores image data generated by the imaging device 15 .

[0049] The display data storage unit 112 stores the display data acquired by the display data acquisition unit 103 .

[0050] <Configuration of the ad distribution device> 4 is a block diagram showing an example of a hardware configuration of an advertisement distribution device according to an embodiment of the present disclosure. The advertisement distribution device 20 may be realized by a calculation device (calculator, computer) such as a server or an information processing device.

[0051] 4, the advertisement distribution device 20 may include a storage device 21, a processing device 22, a UI device 23, a communication device 24, and a bus 25. The storage device 21, the processing device 22, the UI device 23, and the communication device 24 may be connected to each other via the bus 25.

[0052] The programs or instructions for implementing the functions and processes of the advertisement distribution device 20 described above and below may be downloaded from some external device (e.g., a server) via a network, etc. Also, such programs or instructions may be provided from a removable storage medium such as a CD-ROM or a flash memory.

[0053] Storage device 21 may be implemented by RAM, flash memory, a hard disk drive, etc., and stores installed programs or instructions as well as files, data, etc. used in executing the programs or instructions. Storage device 11 may include a non-transitory storage medium.

[0054] The processing device 22 may be realized by, for example, a general-purpose processor or controller (circuit), or may be realized by a dedicated processor or controller (circuit). When realized by a general-purpose processor, the processing device 22 may be realized by one or more CPUs, GPUs, processing circuits, etc., which may be composed of one or more processor cores. In this case, the processing device 22 executes the functions and processes of the advertisement distribution device 20 described above and below according to a program or instruction stored in the storage device 21, data such as parameters used to execute the program or instruction, etc. Such a program may be a program that causes the computer or processing device 22 to function as the advertisement distribution device 20 according to the present disclosure when executed by the computer or processing device 22. In addition, such a program may be a program that causes the computer or processing device 22 to execute a method related to advertisement distribution according to the present disclosure when executed by the computer or processing device 22.

[0055] The UI device 23 may be composed of input devices such as a keyboard, a mouse, a camera, a microphone, etc., output devices such as a display, a speaker, a headset, a printer, etc., and input / output devices such as a haptic device, a touch panel, etc., and realizes an interface between a user and the advertisement distribution device 20. For example, a user operates the advertisement distribution device 20 by operating a GUI displayed on a display or a touch panel using a keyboard, a mouse, etc.

[0056] The communication device 24 may be realized by various communication circuits that execute communication processes with external devices, the Internet, a LAN, a VPN, or other communication networks.

[0057] The above-described hardware configuration of the advertisement distribution device 20 is merely an example, and the advertisement distribution device 20 according to the present disclosure may be realized by another appropriate hardware configuration.

[0058] FIG. 5 is a block diagram illustrating an example of a functional configuration of an advertisement distribution device according to an embodiment of the present disclosure.

[0059] 5, the advertisement distribution device 20 includes a communication control unit 201, a user information acquisition unit 202, an image acquisition unit 203, a display data generation unit 204, a display data output unit 205, a content data storage unit 211, a user information storage unit 212, an image storage unit 213, and a display data storage unit 214. The communication control unit 201, the user information acquisition unit 202, the image acquisition unit 203, the display data generation unit 204, and the display data output unit 205 may be realized as hardware, software, or a combination of hardware and software by the processing device 12. The content data storage unit 211, the image storage unit 213, and the display data storage unit 214 may be realized by the storage device 21.

[0060] The communication control unit 201 establishes a connection with the user device 10 that has accessed the URL (connects to the user device 10) via the communication device 24. The communication control unit 201 may also receive login information transmitted by the user device 10 (communication control unit 101) via the communication device 24. In this case, the communication control unit 201 outputs the login information to the user information acquisition unit 202.

[0061] The communication control unit 201 controls communication with the integrated service providing apparatus 40 by the user information acquisition unit 202, and communication with the user device 10 by the image acquisition unit 203 and the display data output unit 205.

[0062] The communication control unit 201 receives, via the communication unit 24, the shooting end information transmitted by the user device 10 (communication control unit 101).

[0063] The user information acquisition unit 202 receives as an input the login information output by the communication control unit 201. The user information acquisition unit 202 transmits the received login information to the linked service providing device 40 via the communication device 24. If the authentication is successful in the authentication process by the linked service providing device 40 using the login information, the user information acquisition unit 202 receives user information about the user corresponding to the login information from the linked service providing device 40 via the communication device 24, thereby acquiring the user information. The user information acquisition unit 202 stores the acquired user information in the user information storage unit 212, and outputs the login information to the display data generation unit 204 to indicate that the authentication using the login information has been successful (i.e., that the user information has been acquired). Note that if the authentication is unsuccessful in the authentication process by the linked service providing device 40 using the login information, the user information acquisition unit 202 receives a message from the linked service providing device 40 via the communication device 24 indicating that the authentication using the login information has failed, and outputs the message to the display data generation unit 204.

[0064] The image acquisition unit 203 acquires (receives) an image transmitted by the user device 10 (image acquisition unit 102) via the communication device 24. For example, the image acquisition unit 203 acquires a first image including a real-world object indicating that virtual content is to be superimposed from an information terminal. The image acquisition unit 203 outputs the acquired image to the display data generation unit 204 and stores it in the image storage unit 213.

[0065] The display data generating unit 204 receives the image output by the image acquiring unit 203 as an input.

[0066] When the display data generating unit 204 recognizes (or identifies or detects) an AR target in an image, the display data generating unit 204 reads out, from the content data storage unit 211, AR content data associated with the recognized AR target.

[0067] The display data generating unit 204 calculates (local) features of the image, and compares the calculated features with features of an AR target stored in advance, thereby recognizing the AR target within the image.

[0068] For example, the AR target may be a flat surface such as an advertising surface on which an advertising photo or the like is attached, a curved surface (such as the side of a cone or cylinder), or a floor surface. The AR target may exist on a flat surface, a curved surface, or a floor surface as an advertising photo or the like. The AR target may also be the sky.

[0069] The display data generation unit 204 refers to the correspondence information described later, and superimposes the read AR content on the received image data at a specified or predetermined position, such as the recognition position where the AR target is recognized, near the recognition position, or a position a predetermined distance away from the recognition position, and generates the image data on which the AR content is superimposed as display data.

[0070] When the display data generating unit 204 receives the login information output by the user information acquiring unit 202 as an input, the display data generating unit 204 generates display data in which AR content corresponding to the user information is superimposed on an image when generating display data. For example, in response to recognizing a real-world object in the first image, the display data generating unit 204 generates a second image in which virtual content corresponding to the real-world object and corresponding to the user information related to the user of the user device 10 is superimposed on the first image.

[0071] When the display data generation unit 204 does not receive information regarding login information, and when it receives as input information output by the user information acquisition unit 202 indicating that authentication using the login information has failed, it generates display data in which general AR content not based on user information is superimposed on an image, rather than AR content based on the user information.

[0072] On the other hand, if the display data generating unit 204 does not recognize the AR target in the image, it determines to use the received image data as display data.

[0073] The display data generating unit 204 outputs the display data (image data on which the AR content is superimposed or the accepted image data) to the display data output unit 205, and also causes the display data storage unit 214 to store the same.

[0074] The display data output unit 205 receives the display data output by the display data generation unit 204 as an input, and transmits (outputs) the display data to the user device 10 via the communication device 24. For example, the display data output unit 205 transmits a second image generated by the display data generation unit 204 to the user device 10.

[0075] The content data storage unit 211 stores AR content data. The AR content data may be any content data that can be displayed via a display device, such as a still image, a moving image, an animation effect, a three-dimensional (3D) object, a 3D animation, or text.

[0076] The content data storage unit 211 also stores association information (for example, in a table format) that associates the AR target, the AR content, and the user information (for example, see FIG. 9). Thus, when the display data generation unit 204 recognizes an AR target in an image, the display data generation unit 204 can read out, from the content data storage unit 211, the AR content that is associated with the recognized AR target and conforms to the user information by referring to the association information.

[0077] The user information storage unit 212 stores the user information acquired by the user information acquisition unit 202. The user information acquired from the integrated service providing device 40 by the user information acquisition unit 202 may include the user's profile such as the user's age, sex, place of residence, place of origin, hobbies, and preferences, and the user's purchase data such as products purchased by the user in the past and purchase amounts. However, the user information is not limited to these and may be any information related to the user.

[0078] The image storage unit 213 stores the image data acquired by the image acquisition unit 203 .

[0079] The display data storage unit 214 stores the display data generated by the display data generation unit 204 and image data accepted by the display data generation unit 204 and used as the display data.

[0080] <Configuration of Collaboration Service Providing Device> The linked service providing device 40 may have the same or similar hardware configuration as the advertisement distribution device 20, and therefore may be realized by a computing device (calculator, computer) such as a server or an information processing device.

[0081] The linked service providing device 40 receives login information transmitted by the advertisement distribution device 20 (user information acquisition unit 202) via the network 50. The linked service providing device 40 executes an authentication process to determine whether or not a user corresponding to the received login information is a legitimate user, using the received login information and a user ID and password stored in the linked service providing device 40 or another device. If the authentication is successful, the linked service providing device 40 transmits, via the network 50, a message indicating that the authentication was successful and user information related to the user to the advertisement distribution device 20. If the authentication is unsuccessful, the linked service providing device 40 transmits, via the network 50, a message indicating that the authentication was unsuccessful to the advertisement distribution device 20.

[0082] <Furniture composition> 6 is an external perspective view showing an example of a fixture according to an embodiment of the present disclosure. As shown in FIG. 6, the fixture 30 includes an advertising surface 31, an advertising area 32, an advertising distribution device access information area 33, and a product display area 34.

[0083] The advertising surface 31 is a surface on which various advertising information (illustrations, photos, text, etc.) of the product is written. In this embodiment, the advertising surface 31 is a flat surface. However, the advertising surface 31 may be a curved surface.

[0084] Advertisement information is written in the advertisement area 32. There may be a plurality of advertisement areas 32.

[0085] In the advertisement distribution device access information area 33, a two-dimensional code including a URL that hosts an advertisement distribution web service, which is information for accessing the advertisement distribution device 20, is described. The two-dimensional code may optionally include transition information.

[0086] In the product display area 34, products (lipstick in FIG. 6) are displayed.

[0087] In this embodiment, the advertisement surface 31 is the AR target. However, some or all of one or more advertisement areas 32 may be the AR target, or the entire fixture 30 may be the AR target. The fixture 30, the advertisement surface 31, and the advertisement areas 32 are examples of real-world objects.

[0088] <Access to ad distribution device by user device> FIG. 7A is a diagram illustrating an example of access by a user device to an advertisement distribution device according to an embodiment of the present disclosure.

[0089] As shown in FIG. 7A, for example, when a user turns on the camera function of the user device 10 and takes a picture of the advertisement distribution device access information area 33 of the fixture 30, the user device 10 can access the URL included in the two-dimensional code and connect to the advertisement distribution device 20.

[0090] <User device sends login information> FIG. 7B is a diagram illustrating an example of transmission of login information by a user device according to an embodiment of the present disclosure.

[0091] 7B, immediately after the user device 10 accesses the advertisement distribution device 20, a screen for inputting and transmitting login information (user ID and password) for using the integrated service provided by the integrated service providing device 40 is displayed on the display unit of the user device 10. When the user inputs the login information and presses a send button, the user device 10 transmits the login information to the advertisement distribution device 20.

[0092] <Displaying data for display on user device> 8A and 8B are diagrams for explaining an example of display of display data on a user device according to an embodiment of the present disclosure. Fig. 9 is a diagram showing a data table as an example of association information between AR content and user information according to an embodiment of the present disclosure. Although not shown, the data table 900 shown in Fig. 9A is associated with the advertisement surface 31 as an AR target.

[0093] As shown in FIG. 9A, the data table 900 has a “content ID” field 901 , a “user information” field 902 , and a “content description” field 903 .

[0094] In the “content ID” field 901, information for identifying the AR content to be superimposed on the image transmitted from the user device 10 is stored.

[0095] In the "user information" field 902, user information associated with the content (content ID) is stored.

[0096] The "content description" field 903 stores a description of the corresponding content.

[0097] The advertisement distribution device 20 can refer to a data table such as the data table 900 to generate display data on which the AR content is superimposed, and cause the user device 10 to display the display data.

[0098] As a premise for explanation with reference to FIGS. 8A and 8B, it is assumed that the user device 10 is connected to the advertisement delivery device 20, and that the user information relating to the user of the user device 10, which indicates that the user is female and 35 years old, is transmitted from the linked service providing device 40 to the advertisement delivery device 20. Images generated by the imaging device 15 of the user device 10 are sequentially transmitted from the user device 10 to the advertisement delivery device 20, and images (display data) with or without AR content superimposed on the images generated by the imaging device 15 are sequentially transmitted from the advertisement delivery device 20 to the user device 10 and displayed on the user device 10. At this time, since the advertisement surface 31 is an AR target, the advertisement delivery device 20 recognizes the advertisement surface 31 in the image as the AR target, and superimposes the AR content by reading out from the content data storage unit 211 a video for a woman in her 30s having a content ID "c3" corresponding to the user information of a woman in her 30s. 8A shows a state in which the user device 10 displays display data (an image generated by the imaging device 15) before the advertising surface 31, which is an AR target, is recognized by the advertising distribution device 20. FIG. 8B shows a state in which the user device 10 displays display data (an image generated by the imaging device 15 with AR content 81 (a video for women in their 30s) with a content ID of "c3" superimposed thereon) after the advertising distribution device 20 recognizes the advertising surface 31, which is an AR target. As a result, the user device 10 is associated with the fixture 30 (the advertising surface 31 of the fixture 30) and displays AR content corresponding to the user information superimposed on the advertising surface 31.

[0099] Depending on the fixture 30, a two-dimensional code not including transition information may be attached. In this case, the advertisement distribution device 20 does not receive the login information and does not acquire the user information, but reads out from the content data storage unit 211 a general video (AR content) having a content ID "cN" corresponding to the user information "-" in the data table 900, thereby superimposing the AR content.

[0100] Furthermore, multiple AR contents (AR targets) may exist for one piece of fixture 30. In this case, another field associated with the AR content (for example, a field in which information identifying the AR target is stored when multiple AR targets exist) may exist in one data table 900. Alternatively, multiple data tables 900 each corresponding to one AR content may exist.

[0101] <Content based on user information> Next, an example of providing different recommendations (AR content) depending on user information will be described.

[0102] As a first example, when the user information is age and gender as described above, the advertisement distribution device 20 (display data generation unit 204) may, for example, generate display data on which AR content recommending skin care products is superimposed for women in their 20s, generate display data on which AR content recommending shaving cream is superimposed for men in their 30s, and generate display data on which AR content recommending basic cosmetic cream is superimposed for women in their 40s.

[0103] As a second example, when the user information is past purchase amount (e.g., total purchase amount in a specified period), the advertising distribution device 20 (display data generation unit 204) may generate display data on which an image of a coupon for a 15% discount on the next purchase is superimposed for people who have purchased 10,000 yen or more, and may generate display data on which an image of a coupon for a 10% discount on the next purchase is superimposed for people who have purchased less than 10,000 yen.

[0104] As a third example, when the user information is nationality, the advertisement distribution device 20 (display data generation unit 204) may, for example, generate display data on which content in English is superimposed for users of American nationality, and generate display data on which content in Chinese is superimposed for users of Chinese nationality.

[0105] However, the recommendations (AR contents) according to the user information are not limited to the above examples. Also, the first to third examples and other examples not described may be combined.

[0106] By having the above-mentioned configuration, the user device 10 acquires a first image including a real-world object indicating that virtual content is to be overlaid, and displays on the display unit a second image in which virtual content corresponding to the real-world object and corresponding to user information about the user of the user device 10 is overlaid on the first image.

[0107] By having the above-mentioned configuration, the advertisement distribution device 20 receives a first image from the user device 10, and in response to recognizing a real-world object in the first image, generates a second image and transmits the second image to the user device 10.

[0108] As described above, the advertisement delivery system 1 can easily change the display content of the additional information by superimposing virtual content according to user information on an image including a real-world object. In addition, since the virtual content according to the user is superimposed, it is possible to increase the appeal to the user.

[0109] <Operation of the ad distribution system> Next, an example of the operation of the advertisement delivery system 1 will be described with reference to FIGS. 10A to 11C.

[0110] 10A is a flowchart illustrating an example of an operation of a user device according to an embodiment of the present disclosure. The example of the operation illustrated in FIG. 10A relates to access to the advertisement distribution device 20 by the user device 10.

[0111] In step S101, the user device 10 (communication control unit 101) connects to the advertisement distribution device 20 by accessing a URL included in a two-dimensional code in an image of the fixture 30 generated by the imaging device 15. If transition information is included in the two-dimensional code, in step S101, the user device 10 (communication control unit 101) transmits the transition information to the advertisement distribution device 20. Then, the flow ends.

[0112] Fig. 10B is a flowchart showing an example of an operation of a user device according to an embodiment of the present disclosure. The example of the operation shown in Fig. 10B is related to the transmission of an image by the user device 10 to the advertisement distribution device 20. In the example of the operation shown in Fig. 10B, it is assumed that the user device 10 is connected to the advertisement distribution device 20.

[0113] In step S121, the user device 10 (image acquisition unit 102) acquires an image of the fixture 30 or the like from the imaging device 15.

[0114] In step S122, the user device 10 (image acquisition unit 102) transmits the captured image of the fixture 30 or the like to the advertisement distribution device 20. Then, the flow ends.

[0115] The process shown in FIG. 10B is repeatedly executed until the user inputs an instruction to end image capture to the user device 10 (see FIG. 10D).

[0116] 10C is a flowchart illustrating an example of an operation of a user device according to an embodiment of the present disclosure. The example of the operation illustrated in FIG. 10C relates to reception and display of display data from the advertisement distribution device 20 by the user device 10. In the example of the operation illustrated in FIG. 10C, it is assumed that the user device 10 is connected to the advertisement distribution device 20.

[0117] In step S141, the user device 10 (the display data acquisition unit 103) receives display data from the advertisement distribution device 20. The display data is an image in which AR content is superimposed on an image transmitted from the user device 10 to the advertisement distribution device 20, or an image transmitted from the user device 10 to the advertisement distribution device 20.

[0118] In step S142, the user device 10 (display control unit 104) generates a screen including the display data, and displays the screen including the display data on a display device (display unit) included in the user device 10. Then, the flow ends.

[0119] The process shown in FIG. 10C is repeatedly executed until the user inputs an instruction to end image capture to the user device 10 (see FIG. 10D).

[0120] 10D is a flowchart showing an example of an operation of a user device according to an embodiment of the present disclosure. The example of the operation shown in FIG. 10D relates to the end of the method related to advertisement distribution according to the present disclosure. In the example of the operation shown in FIG. 10D, it is assumed that the user device 10 is connected to the advertisement distribution device 20.

[0121] In step S161, the user device 10 (communication control unit 101) transmits shooting end information to the advertisement distribution device 20 in response to a user's input to end shooting to the user device 10. Then, the flow ends. This ends the method related to advertisement distribution according to the present disclosure.

[0122] 11A is a flowchart illustrating an example of an operation of the advertisement distribution device according to an embodiment of the present disclosure. A part of the example of the operation illustrated in FIG. 11A corresponds to the example of the operation illustrated in FIG. 10A and relates to a connection between the advertisement distribution device 20 and the user device 10.

[0123] In step S201, the advertisement distribution device 20 (the communication control unit 201) connects to the user device 10 in response to the user device 10 accessing a URL.

[0124] In step S202, the advertisement distribution device 20 (the communication control unit 201) determines whether or not login information has been received from the user device 10.

[0125] If the login information has not been received (step S202; NO), the flow ends.

[0126] On the other hand, if the login information has been received (step S202; YES), the advertisement distribution device 20 (user information acquisition unit 202) connects to the integrated service providing device 40 in step S203.

[0127] In step S204, the advertisement distribution device 20 (user information acquisition unit 202) transmits the login information to the linked service providing device 40, and in response to successful authentication using the login information, receives (acquires) user information about the user corresponding to the login information from the linked service providing device 40. Then, the flow ends.

[0128] 11B and 11C are flowcharts showing an operation example of the advertisement delivery device according to an embodiment of the present disclosure. The operation examples shown in Fig. 11B and 11C correspond to the operation examples shown in Fig. 10B to Fig. 10D, and relate to the reception of an image from the user device 10 by the advertisement delivery device 20, the transmission of display data to the user device 10, the end of the method for advertisement delivery according to the present disclosure, and the like. In the operation example shown in Fig. 11B, it is assumed that the advertisement delivery device 20 is connected to the user device 10.

[0129] In step S251, the advertisement distribution apparatus 20 (the communication control unit 201) determines whether or not image-taking by the user device 10 has ended by determining whether or not image-taking end information has been received.

[0130] If the shooting has not ended (step S251; NO), the advertisement distribution device 20 (image acquisition unit 203) acquires the image transmitted by the user device 10 in step S252.

[0131] In step S253, the advertisement distribution device 20 (the display data generating unit 204) determines whether or not a recognizable AR target is present in the image.

[0132] If there is no recognizable AR target in the image, that is, if the AR target is not recognized (step S253; NO), in step S254, the advertisement distribution device 20 (display data generation unit 204) determines whether or not AR content exists in the previous display data.

[0133] If the AR content does not exist (step S254; NO), in step S255, the advertisement distribution device 20 (display data generation unit 204) determines to use the image as display data without superimposing the AR content. Then, the flow proceeds to step S257. Note that step S255 is also executed when the image acquired in step S252 is the image initially transmitted by the user device 10.

[0134] On the other hand, if the AR content exists (step S254; YES), in step S256, the advertisement distribution device 20 (display data generation unit 204) decides to clear the AR content and use the image as display data. Then, the flow proceeds to step S257.

[0135] In step S253, if there is a recognizable AR target in the image, i.e., if the AR target is recognized (step S253; YES), in step S258, the advertising distribution device 20 (display data generation unit 204) determines whether or not there is login information (i.e., whether or not authentication using the login information was successful).

[0136] If the login information is present (step S258; YES), in step S259, the advertisement distribution device 20 (display data generation unit 204) refers to the association information and generates display data in which the AR content corresponding to the user information is superimposed on the image. Then, the flow proceeds to step S257.

[0137] On the other hand, if there is no login information (step S258; NO), in step S260, the advertisement distribution device 20 (display data generation unit 204) refers to the association information and generates display data in which AR content (for example, a general video) not associated with specific user information is superimposed on an image. Then, the flow proceeds to step S257.

[0138] In step S257, the advertisement-distribution device 20 (display data output unit 205) outputs (transmits) the display data to the advertisement-distribution device 20. Then, the flow returns to step S251.

[0139] If the shooting is completed (step S251; YES), in step S261, the advertisement distribution device 20 (image acquisition unit 203, etc.) clears all processing (for example, deletes the image from the image storage unit 213, etc.), and the flow ends.

[0140] <Modification> Modifications of the above embodiment will now be described.

[0141] [Variation 1] In the above embodiment, an example has been described in which the advertisement surface 31 is an AR target and AR content is superimposed on the advertisement surface 31, but the present disclosure is not limited to this example. For example, the AR target may be the entire fixture 30, and the display data generating unit 204 may generate display data in which AR content is superimposed on the advertisement surface 31, which is a part of the AR target (furniture 30). Similarly, when the advertisement surface 31 is an AR target, the display data generating unit 204 may generate display data in which AR content is superimposed on a no-information area (an area in which no photographs, characters, etc. exist) on the advertisement surface 31, which is a part of the AR target (advertising surface 31), for example.

[0142] [Variation 2] In the above embodiment, an example of recognizing an AR target in an image by so-called markerless AR has been described, but the present disclosure is not limited to this example. For example, the AR target may be recognized by the display data generating unit 204 using an AR marker, which is a pattern image that serves as a marker for specifying a position where the AR content is to be displayed.

[0143] [Variation 3] In the above embodiment, an example has been described in which the linked service is a service including a BtoC EC service operated by a manufacturer / distributor, but the present disclosure is not limited to this example. For example, the linked service may be any service, such as a service including a BtoC EC service operated by a distributor such as a supermarket or a convenience store, a service operated by a transportation company such as a railway company, or a social networking service (SNS). For example, when the linked service is an SNS, the advertisement distribution device 20 may transmit (post) an image on which the generated AR content is superimposed to the SNS using login information of the user of the user device 10 (if the user allows posting).

[0144] [Variation 4] In the above embodiment, an example in which user information is acquired from the linked service providing device 40 has been described, but the present disclosure is not limited to this example. For example, when the user device 10 connects to the advertisement distribution device 20, the advertisement distribution device 20 may obtain a face image of the user of the user device 10, the user may photograph his / her face using the imaging device 15, and the user device 10 may transmit the generated face image to the advertisement distribution device 20. Then, the advertisement distribution device 20 (for example, the display data generating unit 204) may estimate user information regarding the user using the received face image and a previously trained estimation model for estimating user information such as attributes such as age and gender from the face image, and may generate display data on which AR content is superimposed as described above based on the estimated user information.

[0145] Furthermore, when the user information is a face image, the advertisement delivery device 20 (the display data generation unit 204) may generate, as display data, a superimposed image that applies makeup suited to the user to the face image and recommends products for applying the makeup. The advertisement delivery device 20 (for example, the display data generation unit 204) may generate such a superimposed image using a generation model that has been trained in advance.

[0146] The user information may also include information about the user device 10, such as the model and specifications of the user device 10. For example, when the user information is the model of the user device 10, the advertisement distribution device 20 (the display data generating unit 204) may generate, as display data, a superimposed image in accordance with the model of the user device 10, such that if the model is an older model, the latest model is recommended, or if the model is the latest model, a currently popular application is recommended.

[0147] [Variation 5] In the above embodiment, an example in which AR content according to user information is superimposed has been described, but the present disclosure is not limited to this example.

[0148] For example, AR content according to where the user device 10 is located (i.e., information on the location of the user device 10) may be superimposed and displayed. This makes it possible to provide different superimposed AR content (e.g., area-specific coupons, store-specific coupons, etc.) depending on the store in which the fixture 30 is installed.

[0149] In addition, for example, AR content according to the season or the date and time may be displayed in a superimposed manner, thereby making it possible to display different AR content (for example, seasonal selections) in a superimposed manner depending on the time of year.

[0150] Also, for example, AR content according to the weather may be superimposed and displayed. This allows different AR content to be superimposed depending on whether the weather is sunny or rainy (for example, AR content related to products that can be enjoyed at home when it is raining).

[0151] [Variation 6] In the above embodiment, an example has been described in which the user device 10 transmits the user's login information to the advertisement distribution device 20, and the advertisement distribution device 20 transmits the received login information to the linked service providing device 40, but the present disclosure is not limited to this example. For example, the user device 10 may transmit the user's login information to the linked service providing device 40, and when authentication is successful in an authentication process by the linked service providing device 40 using the login information, the linked service providing device 40 may transmit the user information to the advertisement distribution device 20.

[0152] [Variation 7] In the above embodiment, an example has been described in which the advertisement distribution device 20 generates display data in which virtual content is superimposed on an image including a real-world object, but the present disclosure is not limited to this example. For example, the user device 10 may have a function (the same or similar functional unit as described above) provided in the advertisement distribution device 20 in the form of an application. That is, the user device 10 may generate and display display data in which virtual content corresponding to the real-world object and corresponding to user information related to the user of the user device 10 is superimposed on an image including a real-world object.

[0153] [Other variations] The names of the above-mentioned devices (user device 10, advertisement distribution device 20, etc.) and the functional units constituting the devices (communication control unit 101, image acquisition unit 102, etc.) are merely examples and may be changed as appropriate. Furthermore, two or more of the functional units constituting the devices may be combined and integrated, or one functional unit may be divided into functional units. Furthermore, the transmission function and / or reception function possessed by each functional unit may be summarized as a notification unit, a notified unit, a communication unit, a transmission / reception unit, a transmission unit, a reception unit, etc.

[0154] The order of steps in the flowcharts shown in the figures and described above is merely an example and may be rearranged as appropriate, and some steps may be performed in parallel.

[0155] According to one aspect of the present disclosure, virtual content according to user information is superimposed on an image including a real-world object, thereby making it possible to easily change the display content of additional information.

[0156] Although the embodiments have been described above with reference to the drawings, the present disclosure is not limited to such examples. It is clear that a person skilled in the art can come up with various modified or amended examples within the scope of the claims. It is understood that such modified or amended examples also belong to the technical scope of the present disclosure. In addition, each component or each item in the embodiment (including modified or amended examples) may be arbitrarily combined within the scope of the present disclosure. [Explanation of symbols]

[0157] 1. Advertisement distribution system 10. User Devices 101 Communication control unit 102 Image acquisition unit 103 Display data acquisition unit 104 Display control unit 111 Image storage unit 112 Display data storage unit 20 Advertisement distribution device 201 Communication control section 202 User information acquisition unit 203 Image Acquisition Unit 204 Display data generation unit 205 Display data output section 211 Content data storage unit 212 User information storage unit 213 Image storage unit 214 Display data storage unit 30 Furniture 31 Advertising space 32 Advertising Area 33 Ad distribution device access information area 34 Product display area 40 Collaborative service providing device

Claims

1. a real-world object indicating overlay of virtual content; an information terminal that acquires a first image including the real-world object, and displays on a display unit a second image in which virtual content corresponding to the real-world object and corresponding to user information related to a user of the information terminal is superimposed on the first image; A content display system comprising:

2. The content display system further includes an information processing device, The information terminal transmits the first image to the information processing device, the information processing device receives the first image from the information terminal, and in response to recognizing the real-world object in the first image, generates the second image and transmits the second image to the information terminal. The content display system according to claim 1 .

3. The content display system further includes a second information processing device, the information processing device receives the user information from the second information processing device in response to a successful authentication in an authentication process related to the user executed in the second information processing device; The content display system according to claim 2 .

4. the information processing device receives information necessary for the authentication process from the information terminal, and transmits the received information necessary for the authentication process to the second information processing device; The content display system according to claim 3 .

5. the information terminal transmits information necessary for the authentication process to the second information processing device; The content display system according to claim 3 .

6. The information terminal generates the second image in response to recognizing the real-world object in the first image. The content display system according to claim 1 .

7. An information terminal, an image capture unit that captures a first image including a real-world object indicative of an overlay of virtual content; a display control unit that causes a display unit to display a second image in which virtual content corresponding to the real-world object and corresponding to user information related to a user of the information terminal is superimposed on the first image; An information terminal comprising:

8. an image acquisition unit that acquires, from an information terminal, a first image including a real-world object indicating that virtual content is to be overlaid; a display data generating unit that generates a second image in response to the recognition of the real-world object in the first image, in which virtual content corresponding to the real-world object and corresponding to user information related to a user of the information terminal is superimposed on the first image; a display data output unit that transmits the second image to the information terminal; An information processing device comprising:

9. On the information terminal, acquiring a first image including a real-world object indicative of overlaying virtual content; displaying, on a display unit, a second image in which virtual content corresponding to the real-world object and corresponding to user information related to a user of the information terminal is superimposed on the first image; A program for executing a process including the above.

10. In the information processing device, acquiring a first image from an information terminal, the first image including a real-world object indicating that virtual content is to be overlaid; generating a second image in response to recognizing the real-world object in the first image, the second image having virtual content associated with the real-world object and corresponding to user information about a user of the information terminal superimposed on the first image; transmitting the second image to the information terminal; A program for executing a process including the above.

Images