Information processing device, information processing method, information processing program, and information processing system

Abstract

To provide an information processing device, an information processing method, an information processing program, and an information processing system capable of optimally overlaying and displaying augmented reality content on a captured image.SOLUTION: The information processing device is provided with an image identification unit, a superimposed image determination unit, and a display control unit. The image identification unit identifies a content image included in contents based on detected information. The superimposed image determination unit determines a superimposed image to be overlaid on a captured image, constituting at least a portion of a content image, based on the size of a target area that satisfies predetermined conditions within the captured image captured by the imaging unit. The display control unit displays the superimposed image on the captured image.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, an information processing program, and an information processing system. [Background technology]

[0002] BACKGROUND ART Technologies relating to Augmented Reality (AR) that superimpose additional information on the real world and present it to a user have been developed (see, for example, Patent Document 1).

[0003] For example, smartphone applications that add augmented reality (AR) content to captured images of cityscapes are now in practical use. Examples include services that superimpose information about buildings and stores (hereinafter referred to as "buildings, etc.") that appear in the captured image onto the buildings, services that superimpose translations of text that appears in the captured image onto the text, and services that superimpose virtual creatures, etc. onto the captured image.

[0004] In recent years, there have been services that allow you to superimpose manga characters onto a photograph. With this service, you can take a photo at a location (sacred site) that served as a model for the setting of a manga, allowing you to photograph the manga characters together with the reader, and the background will be the actual location that appeared in the manga. This allows you to take a photo that looks like a scene (frame) from a manga. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-155654 Summary of the Invention [Problem to be solved by the invention]

[0006] However, there was a problem in that the augmented reality content (superimposed image) could not be appropriately superimposed and displayed depending on the captured image, for example, in an image in which the augmented reality content is superimposed on the captured image, and people who are unrelated to the user, specifically tourists visiting the location that served as the model for the setting of the manga, were displayed.

[0007] An object of the present invention is to provide an information processing device, an information processing method, an information processing program, and an information processing system that are capable of appropriately overlaying and displaying augmented reality content in accordance with a captured image. [Means for solving the problem]

[0008] The information processing device according to the present invention comprises: The image specifying unit, the superimposed image determining unit, and the display control unit are provided, the image specification unit specifies a content image included in the content based on the detected detection information; the superimposed image determination unit determines a superimposed image to be superimposed on at least a part of the content image, based on a size of a target area that satisfies a predetermined condition within a captured image captured by an imaging unit; The display control unit causes the superimposed image to be superimposed on the captured image and displayed. [Effects of the Invention]

[0009] According to the present invention, augmented reality content can be suitably superimposed and displayed in accordance with a captured image. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating a configuration of an information processing system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing the hardware configuration of a server according to the present embodiment. [Figure 3] FIG. 2 is a block diagram showing the hardware configuration of a user terminal according to the present embodiment. [Figure 4]FIG. 2 is a block diagram illustrating an example of a functional configuration of a server according to the present embodiment. [Figure 5] 10A and 10B are diagrams illustrating an example in which a superimposed image is superimposed on a captured image and displayed. [Figure 6] 10A and 10B are diagrams illustrating an example in which a superimposed image is superimposed on a captured image and displayed. [Figure 7] 10 is a flowchart illustrating an example of a control process performed by a server. [Figure 8] 10A and 10B are diagrams illustrating an example in which a superimposed image is superimposed on a captured image and displayed. [Figure 9] 10A and 10B are diagrams illustrating an example in which a superimposed image is superimposed on a captured image and displayed. [Figure 10] 10A and 10B are diagrams illustrating an example in which a superimposed image is superimposed on a captured image and displayed. [Figure 11] 10A and 10B are diagrams illustrating an example in which a superimposed image is superimposed on a captured image and displayed. [Figure 12] FIG. 10 is a block diagram showing a modified example of the functional configuration of the server in the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described with reference to the accompanying drawings, in which the same reference numerals denote the same or similar components.

[0012] (Configuration of Information Processing System 1) FIG. 1 shows the configuration of an information processing system 1 in this embodiment. The information processing system 1 shown in FIG. 1 includes a server 10 (information processing server) and multiple user terminals 20 (user terminal 20a, user terminal 20b, user terminal 20c, and user terminal 20d). The server 10 and the user terminals 20 are connected to each other so as to be able to communicate with each other via a communication line 5 such as the Internet, an intranet, a wireless LAN, or mobile communication. The server 10 functions as the "information processing device" of the present invention. The user terminal 20 may also function as the "information processing device" of the present invention.

[0013] In this embodiment, the users who use the user terminal 20a, the user terminal 20b, the user terminal 20c, and the user terminal 20d are referred to as user U1, user U2, user U3, and user U4, respectively. When there is no need to distinguish between user U1, user U2, user U3, and user U4, user U1, user U2, user U3, and user U4 will each be referred to as user U.

[0014] Furthermore, when there is no need to distinguish between the user terminal 20a, the user terminal 20b, the user terminal 20c, and the user terminal 20d, the user terminal 20a, the user terminal 20b, the user terminal 20c, and the user terminal 20d will each be referred to as the user terminal 20.

[0015] The number of user terminals 20 included in the information processing system 1 is not limited to four, but may be at least one user terminal 20.

[0016] The user terminal 20 is an information processing terminal such as a smartphone or a tablet terminal. Any terminal may be used as the user terminal 20, such as a mobile phone, a personal computer (PC), a notebook PC, a personal digital assistant (PDA), or a home game console. The user terminal 20 may notify the server 10 of its own location detected using a global positioning system (GPS) or the like.

[0017] The information processing system 1 provides an augmented reality (AR) service to a user U, which superimposes additional information on the real world and presents it to the user. For example, the user U can add augmented reality content to a captured image of a city taken by the user terminal 20 by executing an application that is installed in the user terminal 20 and provides the augmented reality service in cooperation with the server 10 and the user terminal 20. The augmented reality service includes, for example, a service that displays information about buildings and stores (hereinafter referred to as "buildings, etc.") that appear in the captured image superimposed on the buildings, etc., a service that displays a translation of text that appears in the captured image superimposed on the text, a service that displays virtual creatures, etc., superimposed on the captured image, and a service that displays cartoon characters superimposed on the captured image.

[0018] In this embodiment, a case will be described in which a service is provided to a user U as an augmented reality service that displays manga characters superimposed on a captured image. In this service, for example, the user U can take a photo at a location (sacred place) that served as the model for the setting of the manga, so that the manga characters and the reader of the manga (user U) can be photographed together, and the background will be the actual location that appeared in the manga. This allows the user U to take a photo that looks like a scene (frame) from the manga.

[0019] The server 10 is a device that performs some of the functions of the user terminal 20 when providing an augmented reality service, such as managing various types of information related to the user U and executing some of the processing for the augmented reality service. The server 10 may be configured using one or more information processing devices, or may be configured using a virtual server (such as a cloud server).

[0020] The information processing system 1 solves the following problem that occurs when overlaying and displaying manga characters on a captured image: In other words, there is a problem that the augmented reality content (superimposed image) may not be appropriately overlaid and displayed according to the captured image, such as when a person unrelated to the user, specifically a tourist visiting a place that served as a model for the setting of the manga, is displayed in the image overlaid with the augmented reality content on the captured image.

[0021] Therefore, in this embodiment, the information processing system 1 is capable of appropriately superimposing and displaying augmented reality content in accordance with the captured image.

[0022] (Server 10 hardware configuration) 2 is a block diagram showing the hardware configuration of the server 10 in this embodiment. As shown in FIG. 2, the server 10 includes a control unit 11, a storage unit 12, a communication unit 13, an operation input unit 14, and a monitor 15.

[0023] The control unit 11 is, for example, a CPU (Central Processing Unit), a microprocessor, a DSP (Digital Signal Processor), or the like, and controls the overall operation of the server 10.

[0024] A part of the storage unit 12 is configured with, for example, a RAM (Random Access Memory) or a DRAM (Dynamic Random Access Memory), and is used as a work area when the control unit 11 executes processes based on various programs.

[0025] Furthermore, a part of the storage unit 12 is, for example, a non-volatile memory such as a ROM (Read Only Memory) or an HDD (Hard Disk Drive), and stores various data and programs used in the processing of the control unit 11. The storage unit 12 can hold a database including one or more tables for recording various information, processing results, and the like.

[0026] The programs stored in the memory unit 12 include, for example, an OS (Operating System) for realizing the basic functions of the server 10, drivers for controlling various hardware, programs for realizing various functions, etc., and include programs that function as the ``information processing program'' of the present invention.

[0027] The communication unit 13 is, for example, a NIC (Network Interface Controller) and has a function of connecting to the communication line 5. Instead of or together with the NIC, the communication unit 13 may have a function of connecting to a wireless LAN (Local Area Network), a function of connecting to a wireless WAN (Wide Area Network), a function of enabling short-range wireless communication such as Bluetooth (registered trademark), infrared communication, etc. The server 10 is connected to the user terminal 20 etc. via the communication line 5 and can transmit and receive various data to and from the user terminal 20 etc.

[0028] The operation input unit 14 is composed of a keyboard, a mouse, etc., and accepts input of various operations by a user who uses the server 10. The monitor 15 is, for example, a liquid crystal display device, etc., and displays various images.

[0029] The control unit 11, storage unit 12, communication unit 13, operation input unit 14, and monitor 15 are electrically connected to one another via a system bus 16. Therefore, the control unit 11 can access the storage unit 12, display images on the monitor 15, grasp the operation status of the operation input unit 14 by the user, and access various communication networks and user terminals 20 via the communication unit 13.

[0030] (Hardware configuration of user terminal 20) Fig. 3 is a block diagram showing the hardware configuration of user terminal 20 in this embodiment. As shown in Fig. 3, user terminal 20 is configured to include control unit 21, storage unit 22, communication unit 23, display unit 24, speaker 25, microphone 26, camera 27, and operation buttons 28. Camera 27 functions as the "imaging unit" of the present invention.

[0031] The control unit 21 is, for example, a CPU (Central Processing Unit), a microprocessor, a DSP (Digital Signal Processor), or the like, and controls the overall operation of the user terminal 20.

[0032] A part of the storage unit 22 is configured, for example, with RAM (Random Access Memory) or DRAM (Dynamic Random Access Memory), and is used as a work area when the control unit 21 executes processes based on various programs.

[0033] Furthermore, a part of the storage unit 22 is, for example, a non-volatile memory such as a ROM (Read Only Memory) or an HDD (Hard Disk Drive), and stores various data and programs used in the processing of the control unit 21. The storage unit 22 can hold a database including one or more tables for recording various information, processing results, and the like.

[0034] The programs stored in the storage unit 22 include, for example, an OS (Operating System) for realizing the basic functions of the user terminal 20, drivers for controlling various hardware, programs for realizing various functions, and the like.

[0035] The communication unit 23 is, for example, a network interface controller (NIC), and has a function of connecting to the communication line 5. Instead of or together with the NIC, the communication unit 23 may have a function of connecting to a wireless local area network (LAN), a function of connecting to a wireless wide area network (WAN), a function of enabling short-range wireless communication such as Bluetooth (registered trademark), infrared communication, etc. The user terminal 20 is connected to the server 10, other user terminals 20, etc. via the communication line 5, and can transmit and receive various data to and from the server 10, other user terminals 20, etc.

[0036] The display unit 24 is a touch panel display or the like, and is capable of displaying images and the like and accepting operations by the user U.

[0037] The speaker 25 outputs various sounds under the control of the control unit 21. The microphone 26 inputs various sounds under the control of the control unit 21. The camera 27 captures an image of a subject under the control of the control unit 21.

[0038] The operation button 28 is provided on, for example, the side surface of the user terminal 20, and includes a power button for starting or stopping the user terminal 20, a button for adjusting the volume of the sound output from the speaker 25, and the like.

[0039] The control unit 21, memory unit 22, communication unit 23, display unit 24, speaker 25, microphone 26, camera 27, and operation buttons 28 are electrically connected to one another via a system bus 29. Therefore, the control unit 21 can access the memory unit 22, display images on the display unit 24, grasp the operation state of the touch panel display (display unit 24) and operation buttons 28 by the user U, input sound to the microphone 26, output sound from the speaker 25, control the camera 27, and access various communication networks, the server 10, other user terminals 20, etc. via the communication unit 23.

[0040] Fig. 4 is a block diagram showing an example of the functional configuration of the control unit 11 included in the server 10 in this embodiment. As shown in Fig. 4, the control unit 11 includes, as its functional configuration, a captured image acquisition unit 11a, a detected information acquisition unit 11b, an image specification unit 11c, a superimposed image determination unit 11d, and a display control unit 11e. Note that the control unit 11 of the server 10 generally has various functions in addition to those described above, but here, only the functions that are characteristic of the augmented reality service (a service that displays cartoon characters superimposed on a captured image) provided by the information processing system 1 in this embodiment will be described, and other known functions and the like will not be illustrated or described.

[0041] The captured image acquiring unit 11a acquires captured images captured by the camera 27 (imaging unit) of the user terminal 20 from the user terminal 20. The captured images acquired by the captured image acquiring unit 11a include, for example, a location (sacred place) that served as a model for the setting of a scene in a manga.

[0042] The detection information acquisition unit 11b acquires detection information detected in the user terminal 20 from the user terminal 20. The detection information is, for example, location information of the user terminal 20 detected using the GPS or the like of the user terminal 20, a QR code (registered trademark) read by the camera 27 of the user terminal 20, for example, a QR code installed at a location that served as a model for the setting of a manga, and a captured image captured by the camera 27 of the user terminal 20.

[0043] The image specifying unit 11c specifies a content image (e.g., a frame image) included in the content (e.g., a manga formed of a series of still images) based on the detection information (location information, QR code (registered trademark), captured image) detected by the detection information acquiring unit 11b. For example, the image specifying unit 11c uses the camera 27 of the user terminal 20 to capture location information detected by GPS or a QR code installed at a location that served as a model for the setting of the manga, and searches the Internet for the location specified by reading the QR code or a symbolic object (e.g., a temple, park, station, or inn) in the captured image, or accesses an Internet site corresponding to the QR code, thereby specifying the content image included in the content.

[0044] The superimposed image determination unit 11d identifies a target area in the captured image that satisfies a predetermined condition (for example, the presence of a predetermined object (tourist)) by performing, for example, a known image recognition process on the captured image acquired by the captured image acquisition unit 11a. Then, based on the size of the identified target area, the superimposed image determination unit 11d determines a superimposed image (at least a part of an augmented reality content, for example, a frame image) that constitutes at least a part of the content image identified by the image identification unit 11c and is to be superimposed on the captured image acquired by the captured image acquisition unit 11a. In this embodiment, the superimposed image determination unit 11d increases the size of the superimposed image as the size of the identified target area increases. The superimposed image determination unit 11d determines the superimposed image so that it includes at least an area in the content image where a specific object (for example, a character) is located.

[0045] The display control unit 11e superimposes the superimposed image (for example, a part of a frame image) determined by the superimposed image determination unit 11d on the captured image acquired by the captured image acquisition unit 11a, and displays the superimposed image on, for example, the display unit 24 of the user terminal 20. In this embodiment, the display control unit 11e superimposes and displays the superimposed image on the target area of ​​the captured image identified by the superimposed image determination unit 11d so that the positional relationship between the captured image and the content image matches when viewed by the user U.

[0046] Fig. 5 shows an example of displaying a superimposed image on a captured image. Fig. 5A shows a captured image 30 acquired by the captured image acquisition unit 11a. The captured image 30 shows a train station that served as the model for the setting of a scene in the manga, along with several tourists U11, U12, U13, U14, U15, and U16 visiting the station. As shown in Fig. 5A, the train station that served as the model for the setting of the manga scene is crowded with tourists.

[0047] 5B shows a content image 32 (frame image) included in a content (manga) identified by the image identification unit 11c based on the detection information (captured image 30) detected by the detection information acquisition unit 11b. A specific object 33 (e.g., a character) is arranged in the content image 32.

[0048] 5C shows an image in which a superimposed image 35 (a part of a frame image) determined by the superimposed image determination unit 11d is superimposed on a captured image 30 acquired by the captured image acquisition unit 11a. Here, the superimposed image determination unit 11d identifies a target area within the captured image 30 that satisfies a predetermined condition (the presence of tourists) by, for example, performing a known image recognition process on the captured image 30 acquired by the captured image acquisition unit 11a. Then, based on the size of the identified target area, the superimposed image determination unit 11d determines a superimposed image 35 that constitutes at least a part of the content image 32 determined by the image determination unit 11c and is to be superimposed on the captured image 30 acquired by the captured image acquisition unit 11a.

[0049] 5A, when many tourists U11, U12, U13, U14, U15, and U16 are captured in the captured image 30, the superimposed image 35 is determined so that the tourists U11, U12, U13, U14, U15, and U16 are hidden in half of the content image 32 in FIG. 5B. The display control unit 11e then superimposes the superimposed image 35 determined by the superimposed image determination unit 11d on the captured image 30 acquired by the captured image acquisition unit 11a, and displays the superimposed image 35 on, for example, the display unit 24 of the user terminal 20. This prevents, for example, people unrelated to the user U, specifically, tourists U11, U12, U13, U14, U15, and U16 who are visiting a location that served as a model for the setting of a manga, from being displayed in the image in which the superimposed image 35 (augmented reality content) is superimposed on the captured image 30, and allows the augmented reality content (superimposed image 35) to be suitably superimposed and displayed in accordance with the captured image 30.

[0050] Fig. 6 also shows an example of superimposing an overlay image on a captured image. Fig. 6A shows a captured image 40 acquired by the captured image acquisition unit 11a. The captured image 40 shows a train station that served as the model for the setting of a scene in a manga, and a tourist U11 visiting the station. As shown in Fig. 6A, the train station that served as the model for the setting of the scene in the manga is not crowded with tourists (empty).

[0051] 6B shows a content image 42 (frame image) included in the content (manga) identified by the image identification unit 11c based on the detection information (captured image 40) detected by the detection information acquisition unit 11b. A specific object 43 (e.g., a character) is arranged in the content image 42.

[0052] 6C shows an image in which a superimposed image 45 (a part of a frame image) determined by the superimposed image determination unit 11d is superimposed on a captured image 30 acquired by the captured image acquisition unit 11a. Here, the superimposed image determination unit 11d identifies a target area in the captured image 40 that satisfies a predetermined condition (the presence of tourists) by, for example, performing a known image recognition process on the captured image 40 acquired by the captured image acquisition unit 11a. Then, based on the size of the identified target area, the superimposed image determination unit 11d determines a superimposed image 45 that constitutes at least a part of the content image 42 determined by the image determination unit 11c and is to be superimposed on the captured image 40 acquired by the captured image acquisition unit 11a.

[0053] When only one tourist U11 is shown in the captured image 40 as shown in Fig. 6A, the superimposed image 45 is determined so that the tourist U11 is hidden in the peripheral area of ​​the content image 42 in Fig. 6B where a specific object 43 (character) is located. Then, the display control unit 11e superimposes the superimposed image 45 determined by the superimposed image determination unit 11d on the captured image 40 acquired by the captured image acquisition unit 11a, and displays the superimposed image 45 on, for example, the display unit 24 of the user terminal 20. This prevents, for example, a person unrelated to the user U, specifically, the tourist U11 visiting a place that served as a model for the setting of a manga, from being displayed in the image displayed by superimposing the superimposed image 45 (augmented reality content) on the captured image 40, and allows the augmented reality content (superimposed image 45) to be suitably superimposed and displayed in accordance with the captured image 40.

[0054] 5C and 6C, the superimposed images 35, 45 are determined so as to include at least the area in the content images 32, 42 where the specific objects 33, 43 (characters) are located, thereby satisfying the expectations of the user U who expects the specific objects 33, 43 to be superimposed and displayed in the captured images 30, 40. Therefore, the area included in the superimposed images 35, 45 other than the area in the content images 32, 42 where the specific objects 33, 43 (characters) are located is changed according to the size of the target area in the captured images 40, 50 that satisfies the predetermined condition (the presence of tourists).

[0055] FIG. 7 is a flowchart showing an example of control processing performed by the control unit 11 of the server 10 in providing an augmented reality service (a service in which cartoon characters are superimposed on a captured image) in this embodiment.

[0056] First, the captured image acquiring unit 11a acquires, from the user terminal 20, a captured image captured by the camera 27 (imaging unit) of the user terminal 20 (step S100).

[0057] Next, the detection information acquisition unit 11b acquires the detection information detected in the user terminal 20 from the user terminal 20 (step S110).

[0058] Next, the image identification unit 11c identifies a content image (e.g., a frame image) included in the content (e.g., a manga) based on the detection information (location information, QR code (registered trademark), captured image) detected by the detection information acquisition unit 11b in step S110 (step 120).

[0059] Next, the superimposed image determination unit 11d identifies a target area within the captured image that satisfies a predetermined condition (for example, the presence of a predetermined object (tourist)) by performing, for example, a known image recognition process on the captured image acquired by the captured image acquisition unit 11a in step S100 (step S130).

[0060] Next, the superimposed image determination unit 11d determines a superimposed image (augmented reality content, for example, at least a part of a frame image) that constitutes at least a part of the content image identified by the image identification unit 11c in step S120 and is to be superimposed on the captured image acquired by the captured image acquisition unit 11a, based on the size of the target area identified in step S130.

[0061] Finally, the display control unit 11e superimposes the superimposed image (e.g., a part of the frame image) determined by the superimposed image determination unit 11d in step S140 on the target region identified by the superimposed image determination unit 11d in step S130 in the captured image acquired by the captured image acquisition unit 11a in step S100, and displays the superimposed image on, for example, the display unit 24 of the user terminal 20. When the process of step S150 is completed, the server 10 ends the control process shown in FIG.

[0062] As described above in detail, in this embodiment, the server 10 (information processing device) includes an image specification unit 11c, a superimposed image determination unit 11d, and a display control unit 11e. The image specification unit 11c specifies a content image included in the content based on detection information detected in the user terminal 20. The superimposed image determination unit 11d determines a superimposed image to form at least a part of the content image and to be superimposed on the captured image based on the size of a target area that satisfies a predetermined condition in a captured image captured by the camera 27 (imaging unit) of the user terminal 20. The display control unit 11e superimposes the superimposed image on the captured image and displays it on, for example, the display unit 24 of the user terminal 20.

[0063] According to this embodiment configured in this manner, it is possible to prevent people who are unrelated to the user U, such as tourists visiting the location that served as the model for the setting of the manga, from being displayed in the image in which the superimposed image (augmented reality content) is superimposed on the captured image 40, and the augmented reality content (superimposed image) can be appropriately superimposed and displayed in accordance with the captured image 40.

[0064] In the above embodiment, the display control unit 11e may superimpose and display a superimposed image (a part of a frame image) on at least a part of the captured image other than a region (display region) that satisfies the second condition (e.g., a user who wants to appear with a specific object (character) placed in the content image exists). This prevents the superimposed image from being displayed superimposed on the region of the captured image that satisfies the second condition. Whether or not a user who wants to appear with a specific object (character) placed in the content image exists can be determined, for example, by judging the similarity with the face or overall image of the user that is pre-stored (registered) in the storage unit 22 of the user terminal 20. Alternatively, the region that satisfies the second condition may be determined by the user U touching the user who wants to appear with a specific object on the captured screen or setting the region where the user exists before being captured by the camera 27 of the user terminal 20. Alternatively, the region that satisfies the second condition may be determined by the user U being present in a predetermined position (e.g., the center) of the captured image captured by the camera 27 of the user terminal 20 or the region around the user who is performing a predetermined pose. Alternatively, after a target area that satisfies a predetermined condition is identified, a display area may be set from within the target area.

[0065] Fig. 8 shows an example of displaying a superimposed image on a captured image. Fig. 8A shows a captured image 50 acquired by the captured image acquisition unit 11a. The captured image 50 shows a train station that served as the model for the setting of a scene from a manga, along with several tourists U11, U12, and U13 visiting the station. Here, tourist U11 is a user that user U (the photographer) wants to photograph together with a specific object (character) placed in the content image.

[0066] 8B shows a content image 52 (frame image) included in a content (manga) identified by the image identification unit 11c based on the detection information (captured image 50) detected by the detection information acquisition unit 11b. A specific object 53 (e.g., a character) is arranged in the content image 52.

[0067] 8C shows an image in which a superimposed image 55 (a part of a frame image) determined by the superimposed image determination unit 11d is superimposed on a captured image 50 acquired by the captured image acquisition unit 11a. Here, the display control unit 11e superimposes and displays the superimposed image 55 on at least a part of the captured image 50 other than the area (display area) that satisfies the second condition (tourist U11 who wants to be photographed with specific object 53 located in content image 52). This fulfills the wish of tourist U11 who wants to be photographed with specific object 53 located in content image 52. Furthermore, the superimposed image 55 is determined so that tourists U12 and U13 are hidden in a part of the content image 52 in FIG. 8B, and the superimposed image 55 is displayed superimposed on the surrounding area of ​​the captured image 50 where tourists U12 and U13 are present. This prevents people who are unrelated to the user U, such as tourists U12 and U13 who are visiting the location that served as the model for the setting of the manga, from appearing in the image in which the superimposed image 55 (augmented reality content) is superimposed on the captured image 50, and allows the augmented reality content (superimposed image 55) to be displayed in an appropriate manner in accordance with the captured image 50.

[0068] Furthermore, in the above embodiment, when the display control unit 11e superimposes a superimposed image (part of a frame) on the captured image, if a first region in the superimposed image in which a specific object (character) is located overlaps with a second region that satisfies a second condition (for example, there is a user who wants to be photographed with the specific object (character) located in the content image), the display control unit 11e may change the superimposed position of the superimposed image in the captured image so that the first region and the second region do not overlap. This allows the superimposed image to be displayed so that the first region in which the specific object is located does not overlap with the second region that satisfies the second condition, i.e., so that the first region does not obscure the second region.

[0069] Fig. 9 shows an example of displaying a superimposed image on a captured image. Fig. 9A shows a captured image 60 acquired by the captured image acquisition unit 11a. The captured image 60 shows a train station that served as the model for the setting of a scene from a manga, and a tourist U11 visiting the station. Here, the tourist U11 is a user that the user U (photographer) wants to photograph together with a specific object (character) placed in the content image.

[0070] 9B shows a content image 62 (frame image) included in a content (manga) identified by the image identification unit 11c based on the detection information (captured image 60) detected by the detection information acquisition unit 11b. A specific object 63 (e.g., a character) is arranged in the content image 62.

[0071] 9C shows an image in which a superimposed image 65 (a part of a frame image) determined by the superimposed image determination unit 11d is superimposed on a captured image 60 acquired by the captured image acquisition unit 11a. Here, when the display control unit 11e superimposes the superimposed image 65 on the captured image 60, a first region in which the specific object 63 is located overlaps with a second region that satisfies the second condition (the tourist U11 who wants to be photographed with the specific object 63 located in the content image 62 is present). Therefore, the display control unit 11e shifts the superimposed position of the superimposed image 65 in the captured image 60 to the right so that the first region and the second region do not overlap. This allows the superimposed image 65 to be displayed superimposed so that the first region in which the specific object 63 is located does not overlap with the second region that satisfies the second condition, i.e., so that the second region (the region where the tourist U11 is present) is not hidden by the first region. Ultimately, it is possible to meet the wishes of tourist U11 who wants to be photographed with a specific object 63 placed in the content image 62, and the augmented reality content (superimposed image 65) can be appropriately superimposed and displayed according to the captured image 60.

[0072] Furthermore, in the above embodiment, the display control unit 11e may display the superimposed image (part of the frame image) as a region (display region) satisfying the second condition in the captured image by superimposing it on at least a part of the captured image other than the peripheral region (display region) within a predetermined range from the region in which the specific object is located in the superimposed image (part of the frame image). This makes it possible to prevent the superimposed image from being displayed superimposed on the peripheral region (region satisfying the second condition) within a predetermined range from the region in which the specific object is located in the captured image.

[0073] Fig. 10 shows an example of displaying a superimposed image on a captured image. Fig. 10A shows a captured image 70 acquired by the captured image acquisition unit 11a. The captured image 70 shows a train station that is the model for the setting of a scene from a manga, along with several tourists U11 and U12 visiting the station. Here, the tourists U11 and U12 are users that the user U (photographer) wants to photograph together with a specific object (character) placed in the content image.

[0074] 10B shows a content image 72 (frame image) included in a content (manga) identified by the image identification unit 11c based on the detection information (captured image 70) detected by the detection information acquisition unit 11b. A specific object 73 (e.g., a character) is arranged in the content image 72.

[0075] FIG. 10C shows an image in which a superimposed image 75 (a part of a frame image) determined by the superimposed image determination unit 11d is superimposed on a captured image 70 acquired by the captured image acquisition unit 11a. Here, the display control unit 11e sets a predetermined peripheral area (display area; in the illustrated example, the area between tourists U11 and U12) from the area in which the specific object 73 is located in the superimposed image 75 as an area that satisfies the second condition in the captured image 70. The superimposed image 75 is then displayed superimposed on the area in which the specific object 73 (character) is located. This satisfies the desire of tourists U11 and U12 to be photographed with the specific object 73 located in the content image 72. That is, the augmented reality content (superimposed image 75) can be appropriately superimposed and displayed in accordance with the captured image 70.

[0076] Furthermore, in the above embodiment, when the ratio of the size of the target area to the size of the captured image is equal to or greater than a predetermined ratio (e.g., 80%), the display control unit 11e may restrict the process of superimposing and displaying a superimposed image (e.g., a part of a frame image) on the captured image, compared to when the ratio is less than the predetermined ratio. This allows the process of superimposing and displaying a superimposed image on the captured image to be restricted when the ratio of the size of the target area to the size of the captured image is equal to or greater than the predetermined ratio. This is because when the ratio of the size of the target area to the size of the captured image is equal to or greater than the predetermined ratio, a large portion of the captured image is superimposed on the superimposed image, which reduces the significance for the user U of taking a photo at, for example, a location (sacred site) that served as a model for the setting of a manga, compared to when the ratio is less than the predetermined ratio. Note that when the ratio of the size of the target area to the size of the captured image is equal to or greater than the predetermined ratio, the display control unit 11e may display, on the captured image acquired by the captured image acquisition unit 11a, a message indicating that the ratio of the size of the target area to the size of the captured image is equal to or greater than the predetermined ratio. Furthermore, the display control unit 11e may cancel the restriction on the process of superimposing and displaying a superimposed image on a captured image when the size of the target area has decreased by a predetermined percentage (e.g., 20%) or when the number of tourists appearing in the captured image has decreased by a predetermined number (e.g., 2 people). Furthermore, even if the ratio of the size of the target area to the size of the captured image is less than a predetermined percentage (e.g., 80%) but is equal to or greater than a second predetermined percentage (e.g., 50%), the display control unit 11e may restrict the process of superimposing and displaying a superimposed image on a captured image (i.e., a tourist taking a photo with a specific object) even if there is a tourist who wants to be photographed with a specific object (character) placed in the content image in the captured image acquired by the captured image acquisition unit 11a.

[0077] Furthermore, in the above embodiment, the display control unit 11e may adjust color parameters (e.g., color tone) of at least one of the captured image and the superimposed image so as to reduce the difference between the color parameters (e.g., color tone) of the captured image and the color parameters (e.g., color tone) of the superimposed image, and then superimpose the superimposed image (part of the frame image) on the captured image. This can reduce the sense of incongruity felt by the user due to the difference in color between the captured image and the superimposed image when the superimposed image is superimposed on the captured image. In particular, when the content is a cartoon, content images are available in both black and white and color. Therefore, even if overlaying and displaying a superimposed image (augmented reality content) on the target area of ​​the captured image can prevent, for example, a person unrelated to the user U from being displayed, the sense of incongruity felt by the user is likely to occur when the captured image is color and the superimposed image is black and white. This is particularly likely to occur when the ratio of the size of the target area is equal to or greater than a predetermined ratio. Therefore, by matching the colors of the captured image and the superimposed image (for example, by changing the color of the captured image to black and white, or by changing the color of the superimposed image to color), the sense of discomfort felt by the user due to the difference in color between the captured image and the superimposed image can be reduced.

[0078] Furthermore, in the above embodiment, the superimposed image determination unit 11d may determine a superimposed image (part of a frame image) based on the size of a target area that satisfies a predetermined condition (for example, the presence of a predetermined object (tourist)) and is an area selected by the user U in the captured image. This allows the superimposed image to be suitably superimposed and displayed on the captured image based on the size of the area (target area) selected by the user in the captured image. For example, if the user U checks the captured image and performs an operation to select an area containing a person unrelated to the user U, specifically a tourist visiting a location that served as a model for the setting of a manga, the superimposed image determination unit 11d may determine a superimposed image based on the size of the target area that is the area selected by the user U.

[0079] Fig. 11 shows an example of displaying a superimposed image on a captured image. Fig. 11A shows a captured image 80 acquired by the captured image acquisition unit 11a. The captured image 80 shows a train station that served as the model for the setting of a scene from a manga, along with several tourists U11, U12, U13, U14, and U15 who are visiting the station. Here, tourists U11 and U12 are users that user U (the photographer) wants to photograph together with a specific object (character) placed in the content image.

[0080] 11B shows a content image 82 (frame image) included in a content (manga) identified by the image identification unit 11c based on the detection information (captured image 80) detected by the detection information acquisition unit 11b. A specific object 83 (e.g., a character) is arranged in the content image 82.

[0081] 11C shows an image in which superimposed images 85, 86, and 87 (parts of frame images) determined by the superimposed image determination unit 11d are superimposed on a captured image 80 acquired by the captured image acquisition unit 11a. Here, the display control unit 11e sets a peripheral area (display area, in the illustrated example, the area between tourists U11 and U12) within a predetermined range from the area in which the specific object 83 is located in the superimposed image 86 as an area that satisfies the second condition in the captured image 80. The superimposed image 86 is then superimposed and displayed on the area in which the specific object 83 (character) is located. The display control unit 11e also superimposes and displays the superimposed images 85 and 87 on a target area selected by the user U in the captured image 80 (an area containing people unrelated to the user U, specifically tourists U13, U14, and U15 who are visiting the location that served as the model for the setting of the manga). This fulfills the wishes of tourists U11 and U12 who want to be photographed with specific object 83 located in content image 82. Furthermore, superimposed images 85 and 87 are determined so that tourists U13, U14, and U15 are hidden in a portion of content image 82 in FIG. 11B , and superimposed image 85 is displayed superimposed on the area of ​​captured image 80 where tourists U13 and U14 exist, while superimposed image 87 is displayed superimposed on the area where tourist U15 exists. This prevents, for example, people unrelated to user U, such as tourists U13, U14, and U15 who are visiting a location that served as a model for the setting of a manga, from appearing in the image in which superimposed images 85, 86, and 87 (augmented reality content) are superimposed on captured image 80, and allows the augmented reality content (superimposed images 85, 86, and 87) to be appropriately superimposed and displayed in accordance with the captured image 80.

[0082] In the above embodiment, the control unit 11 of the server 10 may further include a reservation time slot receiving unit 11f, as shown in FIG. 12. The reservation time slot receiving unit 11f receives a reservation time slot (e.g., from 2:00 PM to 4:00 PM) from the user U via the user U's operation on the display unit 24 of the user terminal 20 as a time slot during which a superimposed image (a part of a frame image) can be displayed superimposed on a captured image. The display control unit 11e then displays the superimposed image (a part of a frame image) superimposed on a captured image only if the current time is included in the reservation time slot. This makes it possible to limit the time slot during which a superimposed image can be displayed superimposed on a captured image to the reservation time slot received from the user U. As a result, it is possible to prevent users U receiving augmented reality (AR) services from the information processing system 1 from gathering at a location (sacred site) that served as a model for the setting of a manga at the same time to take photos at the same location and overcrowding. When the reservation time slot receiving unit 11f receives a reservation time slot, the display control unit 11e may display the expected congestion status for the reservation time slot on the display unit 24 of the user terminal 20. Furthermore, the display control unit 11e may display the (estimated) size of a target area that satisfies a predetermined condition (for example, the presence of a predetermined object (tourist)) within the captured image according to the expected congestion status.

[0083] It should be noted that the above embodiments are merely examples of specific embodiments for carrying out the present invention, and the technical scope of the present invention should not be construed as being limited by these embodiments. In other words, the present invention can be carried out in various forms without departing from the gist or main features thereof.

[0084] <Summary> [General-purpose assignment] One of the objects of the present invention is to display augmented reality content in a suitable manner overlaid on a captured image.

[0085] Issues corresponding to [Appendix 1] One object of the present invention is to appropriately overlay and display augmented reality content in accordance with a captured image. [Appendix 1] The information processing device according to this embodiment includes an image identification unit that identifies a content image included in the content based on the detected detection information, a superimposed image determination unit that determines a superimposed image to be superimposed on the captured image, constituting at least a part of the content image, based on the size of a target area that satisfies predetermined conditions within the captured image captured by the imaging unit, and a display control unit that superimposes and displays the superimposed image on the captured image. According to the information processing device described above, it is possible to suitably superimpose and display augmented reality content (superimposed image) on a captured image based on the size of a target area that satisfies a predetermined condition within the captured image.

[0086] Issues corresponding to [Appendix 2] One object of the present invention is to prevent an overlay image from being displayed superimposed on an area of ​​a captured image that satisfies a specific condition. [Supplementary Note 2] The information processing device according to Supplementary Note 1, wherein the display control unit displays the superimposed image in a superimposed manner on at least a part of the captured image other than the area that satisfies the second condition. This makes it possible to prevent the superimposed image from being displayed superimposed on an area of ​​the captured image that satisfies the second condition.

[0087] Issues corresponding to [Appendix 3] One object of the present invention is to display a superimposed image in a captured image so that an area in which a specific object is located does not overlap an area that satisfies a specific condition. [Appendix 3] An information processing device as described in Appendix 2, wherein the display control unit superimposes and displays a superimposed image on the captured image so that the positional relationship between the captured image and the content image matches as seen by the user, and when superimposing the superimposed image on the captured image, if a first area in which a specific object is located in the superimposed image overlaps with a second area that satisfies a second condition, the display control unit changes the superimposed position of the superimposed image so that the first area and the second area do not overlap. This allows the superimposed image to be displayed in a superimposed manner so that the first area in which the specific object is located and the second area that satisfies the second condition do not overlap in the captured image.

[0088] Issues corresponding to [Appendix 4] One object of the present invention is to prevent a superimposed image from being displayed superimposed on a peripheral area within a predetermined range from an area where a specific object is located in a captured image. [Appendix 4] An information processing device as described in Appendix 2, wherein the display control unit superimposes and displays the superimposed image on the captured image so that the positional relationship between the captured image and the content image matches as seen by the user, and the second condition is that the superimposed image is a peripheral area within a predetermined range from the area in the captured image where the specific object is located in the superimposed image. This makes it possible to prevent the superimposed image from being displayed superimposed on a peripheral area (area that satisfies the second condition) within a predetermined range from the area in which the specific object is located in the captured image.

[0089] Issues corresponding to [Appendix 5] One object of the present invention is to appropriately restrict the process of superimposing a superimposed image on a captured image and displaying it. [Appendix 5] An information processing device described in any one of Appendices 1 to 4, wherein the display control unit restricts the process of superimposing and displaying an overlay image on the captured image when the ratio of the size of the target area to the size of the captured image is equal to or greater than a predetermined ratio, compared to when the ratio is less than the predetermined ratio. This makes it possible to restrict the process of superimposing and displaying the superimposed image on the captured image when the ratio of the size of the target region to the size of the captured image is equal to or greater than a predetermined ratio.

[0090] Issues corresponding to [Appendix 6] One object of the present invention is to display a superimposed image on a captured image so as to include at least the area in the content image where a specific object is located. [Supplementary Note 6] The information processing device according to any one of Supplementary Notes 1 to 5, wherein the superimposed image determination unit determines the superimposed image so as to include at least an area in the content image where the specific object is located. This allows the superimposed image to be displayed superimposed on the captured image so as to include at least the area in the content image where the specific object is located.

[0091] Issues corresponding to [Appendix 7] One of the objects of the present invention is to suppress the sense of discomfort felt by the user when a superimposed image is superimposed on a captured image. [Appendix 7] An information processing device described in any one of Appendices 1 to 6, wherein the display control unit adjusts color parameters of at least one of the captured image and the superimposed image so as to reduce the difference between color parameters related to the color of the captured image and color parameters related to the color of the superimposed image, and superimposes the superimposed image on the captured image. This makes it possible to suppress the sense of discomfort felt by the user due to the difference in color between the captured image and the superimposed image when the superimposed image is superimposed on the captured image.

[0092] Issues corresponding to [Appendix 8] One object of the present invention is to display a superimposed image in an appropriate manner on a captured image based on an area selected by a user within the captured image. [Appendix 8] An information processing device described in any one of Appendices 1 to 7, wherein the superimposed image determination unit determines the superimposed image based on the size of a target area that satisfies the predetermined condition of being an area selected by a user within the captured image. This makes it possible to suitably superimpose the superimposed image on the captured image and display it based on the size of the area (target area) selected by the user within the captured image.

[0093] Issues corresponding to [Appendix 9] One object of the present invention is to appropriately limit the time period during which a superimposed image can be displayed superimposed on a captured image. [Appendix 9] An information processing device described in any one of Appendices 1 to 8, comprising a reservation time slot receiving unit that receives from a user a reservation time slot as a time slot during which a superimposed image can be superimposed and displayed on a captured image, and a display control unit that superimposes and displays the superimposed image on a captured image only if the current time is included in the reservation time slot. This makes it possible to limit the time period during which the superimposed image can be displayed superimposed on the captured image to the reserved time period accepted from the user.

[0094] Issues corresponding to [Appendix 10] One object of the present invention is to appropriately overlay and display augmented reality content in accordance with a captured image. [Appendix 10] In the information processing method of this embodiment, the processor identifies a content image included in the content based on the detected detection information, and the processor constructs at least a portion of the content image based on the size of a target area that satisfies predetermined conditions within the captured image captured by the imaging unit, and determines a superimposed image to be superimposed on the captured image, and the processor superimposes the superimposed image on the captured image and displays it. According to the above information processing method, it is possible to suitably superimpose and display augmented reality content (superimposed image) on a captured image based on the size of a target area that satisfies a predetermined condition within the captured image.

[0095] Issues corresponding to [Appendix 11] One object of the present invention is to appropriately overlay and display augmented reality content in accordance with a captured image. [Appendix 11] The information processing program according to this embodiment causes a processor to execute the following process: identify a content image included in the content based on the detected detection information; construct at least a portion of the content image based on the size of a target area that satisfies predetermined conditions within the captured image captured by the imaging unit; determine a superimposed image to be superimposed on the captured image; and display the superimposed image on the captured image. According to the information processing program, it is possible to suitably superimpose and display augmented reality content (superimposed image) on a captured image based on the size of a target area that satisfies a predetermined condition within the captured image.

[0096] Issues corresponding to [Appendix 12] One object of the present invention is to appropriately overlay and display augmented reality content in accordance with a captured image. [Appendix 12] The information processing system according to this embodiment includes a server and a terminal device. The server includes an image identification unit that identifies a content image included in the content based on detection information detected by the user's terminal device, a superimposed image determination unit that determines a superimposed image to be superimposed on the captured image, constituting at least a part of the content image, based on the size of a target area that satisfies predetermined conditions within a captured image captured by an imaging unit included in the terminal device, and a display control unit that superimposes the superimposed image on the captured image and displays it. According to the information processing system described above, it is possible to suitably superimpose and display augmented reality content (superimposed image) on a captured image based on the size of a target area that satisfies a predetermined condition within the captured image. [Explanation of symbols]

[0097] 1: Information processing system, 10: Server, 11: Control unit, 11a: Captured image acquisition unit, 11b: Detection information acquisition unit, 11c: Image identification unit, 11d: Superimposed image determination unit, 11e: Display control unit, 11f: Reservation time slot reception unit, 12: Memory unit, 13: Communication unit, 14: Operation input unit, 15: Monitor, 16: System bus, 20, 20a, 20b, 20c: User terminal, 21: Control unit, 22: Memory unit, 23: Communication unit, 24: Display unit, 25: Peaker, 26: Microphone, 27: Camera, 28: Operation button, 29: System bus, 30, 40, 50, 60, 70, 80: Captured image, 32, 42, 52, 62, 72, 82: Content image, 33, 43, 53, 63, 73, 83: Specific object, 35, 45, 55, 65, 75, 85, 86, 87: Superimposed image, U, U1, U2, U3, U4: User, U11, U12, U13, U14, U15, U16: Tourist

Claims

1. An information processing device, The image specifying unit, the superimposed image determining unit, and the display control unit are provided, the image specification unit specifies a content image included in the content based on the detected detection information; the superimposed image determination unit determines a superimposed image to be superimposed on at least a part of the content image, based on a size of a target area that satisfies a predetermined condition within a captured image captured by an imaging unit; the display control unit causes the superimposed image to be superimposed on the captured image and displayed; Information processing device.

2. the display control unit displays the superimposed image in a superimposed manner on at least a part of the captured image other than an area that satisfies a second condition; The information processing device according to claim 1 .

3. the display control unit displays the superimposed image on the captured image so that a positional relationship between the captured image and the content image matches that seen from a user; when the display control unit superimposes the superimposed image on the captured image, if a first area in which a specific object is located in the superimposed image overlaps with a second area that satisfies the second condition, the display control unit changes a superimposed position of the superimposed image so that the first area and the second area do not overlap. The information processing device according to claim 2 .

4. the display control unit displays the superimposed image on the captured image so that a positional relationship between the captured image and the content image matches that seen from a user; the second condition is that the specific object is located in a peripheral area of ​​the captured image within a predetermined range from an area in which the specific object is located in the superimposed image; The information processing device according to claim 2 .

5. the display control unit restricts the process of superimposing the superimposed image on the captured image when the ratio of the size of the target region to the size of the captured image is equal to or greater than a predetermined ratio, compared to when the ratio is less than the predetermined ratio. The information processing device according to claim 1 .

6. the superimposed image determination unit determines the superimposed image so as to include at least an area in the content image where a specific object is located. The information processing device according to claim 1 .

7. the display control unit adjusts color parameters of at least one of the captured image and the superimposed image so that a difference between a color parameter related to a color of the captured image and a color parameter related to a color of the superimposed image is reduced, and superimposes the superimposed image on the captured image. The information processing device according to claim 1 .

8. the superimposed image determination unit determines the superimposed image based on a size of the target area that satisfies the predetermined condition of being an area selected by a user within the captured image. The information processing device according to claim 1 .

9. Equipped with a reservation time slot reception desk, the reservation time slot receiving unit receives from a user a reservation time slot as a time slot during which the superimposition image can be superimposed on the captured image and displayed; the display control unit causes the superimposition image to be superimposed on the captured image only when the current time is included in the reserved time slot; The information processing device according to claim 1 .

10. The processor identifies a content image included in the content based on the detected detection information; a processor determines a superimposed image to be superimposed on the captured image, the superimposed image constituting at least a part of the content image, based on the size of a target area that satisfies a predetermined condition within the captured image captured by the imaging unit; a processor causes the superimposed image to be superimposed on the captured image and displayed; Information processing methods.

11. The processor Identifying a content image included in the content based on the detected detection information; determining a superimposed image to be superimposed on the captured image, which constitutes at least a part of the content image, based on the size of a target area that satisfies a predetermined condition within the captured image captured by the imaging unit; displaying the superimposed image superimposed on the captured image; An information processing program that executes processing.

12. An information processing system, A server and a terminal device are provided, The server The image specifying unit, the superimposed image determining unit, and the display control unit are provided, the image specification unit specifies a content image included in the content based on detection information detected by the terminal device of the user; the superimposed image determination unit determines a superimposed image that constitutes at least a part of the content image and is to be superimposed on the captured image based on a size of a target area that satisfies a predetermined condition within a captured image captured by an imaging unit included in the terminal device; the display control unit causes the superimposed image to be superimposed on the captured image and displayed; Information processing system.

Images