System and computer program

Abstract

To provide a system and a computer program which can provide an interesting AR synthesis technique which has high visibility.SOLUTION: A system executes following steps: a display step of displaying a predetermined image on a display device; a photographing step of photographing the predetermined image displayed on the display device, by a predetermined photographing device; a recognition step of executing image recognition processing with respect to the photographed predetermined image; a setting step of setting visual recognition information of a user who operates a predetermined terminal device; an addition step of executing addition processing, by using a result of the image recognition processing, wherein the addition processing is processing for newly adding information which assists the predetermined image to generate an auxiliary image, on the basis of the visual recognition information and the predetermined image; and a superposition step of superimposing the auxiliary image on the photographed predetermined image, and causing the terminal device to display it.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present invention relates to a system and a computer program. [Background technology]

[0002] 2. Description of the Related Art Conventionally, television systems using AR (Augmented Reality) technology are known (see, for example, Patent Documents 1 to 3).

[0003] With this system, a user can take a picture of an image (television image) displayed on a television using a camera mounted on the mobile terminal they use, and then have an AR image related to that television image displayed on the mobile terminal (AR synthesis).

[0004] Furthermore, by using a gyro sensor built into the mobile terminal, this system can also estimate the position and orientation of the television even when the television image is not being captured by the camera.

[0005] This means that by detecting and registering the position and orientation of the television in advance using a mobile device, it is possible to estimate the position and orientation of the television even if the position or orientation of the mobile device changes thereafter. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent No. 5259519 [Patent Document 2] Patent No. 5265468 [Patent Document 3] Patent No. 6010373 DISCLOSURE OF THEINVENTION [Problem to be solved by the invention]

[0007] However, in the conventional technology, only an AR image related to content synchronized with a television broadcast is displayed on a mobile terminal.

[0008] In addition, when the user is away from the television and the television image is not captured by the mobile terminal, the conventional technology does not perform AR synthesis, so the user may miss the provided content.

[0009] An object of the present invention is to provide a system and a computer program capable of providing an AR synthesis technique that is highly visible and interesting. [Means for solving the problem]

[0010] The present invention relates to A system for causing a computer to execute the following steps: In the display step, a predetermined image is displayed on a display device; In the photographing step, the predetermined image displayed on the display device is photographed by a predetermined photographing device; In the recognition step, an image recognition process is performed on the captured predetermined image, In the setting step, visual recognition information of a user who operates a predetermined terminal device is set, In the addition step, an addition process is performed using a result of the image recognition process, the addition process is a process of generating an auxiliary image by newly adding information that supplements the predetermined image based on the visual recognition information and the predetermined image, In the superimposing step, the auxiliary image is superimposed on the predetermined image and displayed on the terminal device. It is a system. Effect of the Invention

[0011] According to the present invention, it is possible to provide a system and a computer program capable of providing an AR synthesis technique that is highly visible and interesting. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing a system configuration in the present embodiment. [Diagram 2] FIG. 2 is a diagram showing the configuration of a game device included in the system in this embodiment. [Diagram 3] FIG. 1A is a diagram showing the configuration of an AR terminal device included in a system, and FIG. 1B is a perspective view showing the appearance of the AR terminal device, in this embodiment. [Figure 4] 4 is a diagram showing an example of an image displayed on the AR terminal device in the present embodiment. FIG. [Diagram 5] 4 is a diagram showing an example of an image displayed on the AR terminal device in the present embodiment. FIG. [Figure 6] 4 is a diagram showing an example of an image displayed on the AR terminal device in the present embodiment. FIG. [Figure 7] 4 is a diagram showing an example of an image displayed on the AR terminal device in the present embodiment. FIG. [Figure 8] 4 is a diagram showing an example of an image displayed on the AR terminal device in the present embodiment. FIG. [Figure 9] 4 is a diagram showing an example of an image displayed on the AR terminal device in the present embodiment. FIG. [Figure 10] 11 is a flowchart showing AR synthesis processing in the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] [Embodiment] A system 1 according to an embodiment of the present invention will be described with reference to Fig. 1 to Fig. 10. Note that the system 1 and the processing procedure described below are merely examples, and the embodiment of the present invention is not limited thereto. The design of the system 1 and the processing procedure can be appropriately changed without departing from the gist of the present invention.

[0014] <System Description> The system 1 of this embodiment is composed of a game device 2 (part of the "computer"; an example of a distribution device), a monitor 3 (part of the "computer"; an example of a display device), an AR terminal device 4 (part of the "computer"; an example of a terminal device), and a network 5 connecting these devices, as shown in Figures 1 to 3.

[0015] The system 1 executes AR synthesis processing based on the functions of the game device 2, the monitor 3, and the AR terminal device 4.

[0016] The game device 2 in this embodiment is a stationary game device to which a monitor 3 for displaying game images and a controller 250 for accepting user operations are externally connected, as shown in Fig. 2. In this embodiment, the game device 2 is, for example, wirelessly connected to the controller 250 and wired connected to the monitor 3.

[0017] As shown in Figures 3(a) and (b), the AR terminal device 4 is a head-mounted display, which is a type of wearable terminal, that is equipped with an optically transmissive transparent display 430 (e.g., a display including a transmissive liquid crystal display, a transmissive organic EL display, a transmissive inorganic EL display, etc.) that allows the user to see behind the display, a speaker 440, a camera 450 (an example of an imaging device), and setting buttons, etc. 460.

[0018] As shown in FIG. 4, the user can view the real space RS directly (that is, without going through a computer image captured by the camera 450) via the see-through display 430.

[0019] In addition, the AR terminal device 4 can generate an auxiliary image (hereinafter referred to as an "AR image AI") to which new information that supplements the game image GI has been added by using the camera 450 to capture a game image GI (an example of a specified image) displayed on a monitor 3 installed in the real space RS, as shown in Figure 5.

[0020] This allows the AR terminal device 4 to display (AR composite) on the transparent display 430, in addition to the real space RS (including the game image GI) that can be viewed through the transparent display 430, an AR image AI that supplements the game image GI.

[0021] When generating the AR image AI, the AR terminal device 4 executes image recognition processing on the game image GI captured by the camera 430. Specifically, the AR terminal device 4 analyzes the captured game image GI to grasp the progress of the game.

[0022] 5, when it is determined as a result of the image recognition process that an enemy character is performing a special move in the game image GI, the AR terminal device 4 executes additional processing to generate an AR image AI that is composed of an AR image AI showing a wave to suggest (supplementarily explain) the power of the special move and a text message such as "BOOOM!", in accordance with the game image GI. In this case, the AR terminal device 4 determines the progress of the game based on, for example, the pose of the character displayed in the game image GI, VFX (visual effects), and the like, and generates the AR image AI.

[0023] Also, as shown in FIG. 6, if the image recognition process determines based on the game image GI that the playable character's stamina is low (for example, the playable character's stamina gauge HG is at 10 or less), the AR terminal device 4 executes additional processing to generate an AR image AI (texture TX) showing blood splattered all over the screen of the transparent display 430.

[0024] In addition, when it is determined that the game is being executed, the AR terminal device 4 can generate an AR image in which fireworks are exploding. Note that the AR terminal device 4 determines that the game is being executed, for example, when an operation standby screen (for example, a pause screen) is not displayed.

[0025] In addition, when a game image GI is captured while the game is paused (for example, when there is no screen transition for more than five minutes, when the operation standby screen is displayed, or during loading), the AR terminal device 4 can also generate information (for example, an advertisement for beer) to recommend food or drink to the user.

[0026] Furthermore, in the system 1 of this embodiment, the AR terminal device 4 generates an AR image AI to supplement the game image GI using not only the captured game image GI but also the user's visual information acquired in the setting process described below.

[0027] This "visual recognition information" is information on the user's visual acuity (hereinafter referred to as "visual acuity information") and is information related to whether or not the user can visually recognize the game image GI. In other words, the visual acuity information is information including an index itself related to the user's visual acuity, and is information indicating whether or not the user can recognize images such as characters and letters in the game image GI with the user's visual acuity in a state in which the user has the game image GI in his / her field of view (for example, central vision at 1 to 2 degrees from the line of sight, or effective field of view at 4 to 20 degrees from the line of sight; the same applies below). This visual acuity information can be registered in advance in the AR terminal device 4. In addition to this, for example, a test may be performed in advance in the AR terminal device 4 to determine whether or not the user can read letters and the like displayed on the monitor 3, and the visual acuity value may be input based on a user operation. In addition, whether or not the user can recognize an image may be determined according to the distance from the AR terminal device 4 (user) to the monitor 3 and the visual acuity information.

[0028] When the AR terminal device 4 determines based on the user's visual acuity information that the size of an image of a character, text (e.g., a character name) displayed in the game image GI is too small for the user to recognize the image, it generates an AR image AI (hereinafter referred to as an "enlarged image EI") to supplement the image that is too small for the user to recognize, as shown in Figure 7.

[0029] From another perspective, the "visual information" is information that indicates whether or not the user can visually recognize the game image GI based on information related to the user's vision (hereinafter referred to as "visual information"). Specifically, the visual information is information that is used when the AR terminal device 4 determines whether or not the user has the game image GI in their field of vision (whether or not the monitor 3 is in the user's field of vision).

[0030] Whether or not the monitor 3 is within the user's field of view is determined by detecting the user's line of sight with an in-camera (not shown) provided on the user's side while the user is wearing the AR terminal device 4. Specifically, the AR terminal device 4 determines (estimates) whether or not the monitor 3 is within the user's field of view based on whether or not the game image GI is within the central vision or effective field of view with respect to the user's line of sight detected by the in-camera. This allows the AR terminal device 4 to automatically determine whether or not the monitor 3 is within the user's field of view without relying on user input. In FIG. 8, the user is looking to the left side of the paper in FIG. 8, and it is determined that the game image GI is not within the user's field of view.

[0031] When the AR terminal device 4 uses the user's visual information to determine that the monitor 3 is within the field of view of the camera 450, but determines that the monitor 3 is not within the user's field of view (when out-of-field-of-view information described below is generated), it generates an AR image (a text image TI saying "It's dangerous! Protect yourself!") to draw the user's attention, as shown in Figure 8.

[0032] In addition, when the AR terminal device 4 uses the user's visual information to determine that the monitor 3 is not within the field of view of the camera 450 (when outside field of view information described below is generated), it generates an assist AR image (a wipe image WI and a text image TI saying "Check!") to assist the game image GI, as shown in Figure 9.

[0033] The AR synthesis process executed by the system 1 will be described in detail below.

[0034] The system 1 can execute the following processes related to the AR synthesis process by using the respective functions of the game device 2, the monitor 3, and the AR terminal device 4.

[0035] <Game progress management processing> The game device 2 executes a game application to control the progress of the video game. Information related to the progress of the game is transmitted to the monitor 3.

[0036] <Display processing> The monitor 3 displays a game image GI on the screen based on the information relating to the progress of the game transmitted from the game device 2.

[0037] <Monitor recognition processing> The AR terminal device 4 can use the camera 450 to cause the AR terminal device 4 to recognize the monitor 3 .

[0038] One possible method for making the AR terminal device 4 recognize the monitor 3 is, for example, to match a frame object 4F displayed on the transparent display 430 with a frame 3F arranged around the monitor 3, as shown in Figure 4 (note that the frame object 4F and the frame 3F do not need to match perfectly, and the frame 3F may be included within the frame object 4F).

[0039] Other methods that can be adopted include a method of using a camera 450 to detect a light-emitting element (not shown) arranged, for example, on the top (or around) of the monitor 3, and a method of setting the position and orientation of the monitor 3 relative to the acquired room information (e.g., a room map) based on a user operation on a setting button or the like 460 of the AR terminal device 4.

[0040] When the monitor 3 is recognized by the AR terminal device 4, the AR terminal device 4 can thereafter detect the location and direction of the AR terminal device 4 using information from at least one of the GPS function, gyro sensor, acceleration sensor, and magnetic sensor of the AR terminal device 4, and can recognize the location and direction of the monitor 3 based on the information. The position and orientation of the monitor 3 (the orientation of the screen on which the game image GI is displayed) are stored in the storage unit 41 of the AR terminal device 4.

[0041] By executing this monitor recognition process, the AR terminal device 4 can execute image recognition process, visual field inside / outside determination process, angle of view inside / outside determination process, and the like, which will be described later.

[0042] <Shooting process> The AR terminal device 4 can use the camera 450 to capture a game image GI displayed on the monitor 3. The captured game image GI is stored in the storage unit 41 of the AR terminal device 4. Note that the subject of the capture includes not only the game image GI but also the entire real space RS, such as the monitor 3, the television stand, and the bookshelf.

[0043] <Image recognition processing> The AR terminal device 4 can analyze the captured game image GI (image recognition processing). For example, in a fighting game, the AR terminal device 4 analyzes the user's fighting style, such as whether or not the user is weak at defense, and in an action game, the AR terminal device 4 analyzes the user's skill, such as whether or not the game has been over too many times, and in a baseball game, the AR terminal device 4 performs a tendency analysis of the user's pitching. For example, the AR terminal device 4 can determine whether or not the user is weak at defense based on the degree to which a vitality gauge is decreasing, and can determine whether or not the game has been over too many times based on the number of times the word "GAME OVER" is displayed on the monitor 3 within a predetermined period. The AR terminal device 4 can also determine the tendency analysis of play based on game images of the game (images such as play images and UIs) and the like.

[0044] <Additional Processing> By executing the above-mentioned image recognition process, the AR terminal device 4 can analyze the captured game image GI and generate (or select; the same applies below) an AR image AI based on the analysis results, without linking with the game device 2 or the like (without assuming that information synchronized with the game progress on the game device 2 is obtained in real time from a device other than the AR terminal device 4).

[0045] For example, the AR terminal device 4 can generate the above-mentioned AR image AI of a wave (FIG. 5) and AR image AI of a blood splatter texture TX (FIG. 6), and in the case where the type of video game is a baseball game, can analyze the distance of a home run and generate an AR image according to the distance (for example, a fireworks image of a size according to the distance). For example, the AR terminal device 4 analyzes the trajectory of the ball and the characters indicating the distance displayed on the game side (for example, "distance 125 m") regarding the distance of the home run.

[0046] <Visual acuity information setting process> The AR terminal device 4 can set the user's visual acuity information on, for example, a menu screen displayed on the transmissive display 430. For example, the user can register the user's visual acuity information in the AR terminal device 4 by inputting information such as the user's right eye visual acuity being "0.7", the user's left eye visual acuity being "0.5", and "has farsightedness" based on the user's operation. The user's visual acuity information is stored in the storage unit 41 of the AR terminal device 4. This visual acuity information setting process is one of the setting processes.

[0047] Furthermore, the AR terminal device 4 can display images of various sizes (for example, Landolt rings, letters, or numbers used in visual acuity tests) on the transmissive display 430 (or may be displayed on the monitor 3) and allow the user to confirm whether or not they can be seen. Then, the user's visual acuity information can be registered in the AR terminal device 4 by the user's input as to whether or not the user can see each image.

[0048] <Processing for determining whether it is inside or outside the field of view and inside or outside the angle of view> The AR terminal device 4 executes a visual field inside / outside determination process to determine whether or not the monitor 3 is within the visual field of the user.

[0049] If it is determined that the monitor 3 is within the user's field of view, neither out-of-field information nor out-of-angle-of-view information is generated (neither the flag for out-of-field information nor the flag for out-of-angle-of-view information is set).

[0050] On the other hand, if it is determined that the monitor 3 is not within the user's field of view, the AR terminal device 4 executes a process of determining whether the monitor 3 is within or outside the field of view, and generates visual information according to the situation.

[0051] The monitor 3 is determined to be out of the user's field of vision in the following cases, for example. (1) When it is determined that the monitor 3 is photographed by the camera 450 of the AR terminal device 4 (the monitor 3 is present within the angle of view of the camera 450), but it is determined that the monitor 3 is not within the user's field of view (when out-of-view information is generated, i.e., when a flag related to the out-of-view information is set) (2) When it is determined that the monitor 3 is not photographed by the camera 450 of the AR terminal device 4 (the monitor 3 is outside the angle of view of the camera 450), and therefore it is determined that neither the user nor the AR terminal device 4 can recognize the monitor 3 (when outside-angle-of-view information is generated, i.e., when a flag related to the outside-angle-of-view information is set)

[0052] In the case of (1) above, it is determined that the camera 450 of the AR terminal device 4 is able to recognize the game image GI displayed on the monitor 3, and therefore the AR terminal device 4 generates an AR image AI based on the captured game image GI based on the image recognition processing, as shown in Figure 8.

[0053] In this case, in addition to the image recognition process, the AR terminal device 4 executes an in-angle of view process based on the out-of-field information, and determines information that the user should particularly pay attention to (for example, a text image TI saying "It's dangerous! Defend yourself!") from the game progress status in order to give the user an opportunity to focus on the monitor 3 (to bring the game image GI into the user's field of view), as shown in Fig. 8. The AR image AI (text image TI) related to the information that the user should pay attention to is displayed near the user's line of sight on the transparent display 430, as shown in Fig. 8, for example.

[0054] In the case of (2) above, the AR terminal device 4 transmits outside-angle information to the game device 2, and the game device 2 can transmit information for displaying an assist AR image (e.g., whether the operated character has won, whether it is a heads-up attack or a tails-up attack, etc.) to the AR terminal device 4 via the Internet line.

[0055] As a result, even in the case (2) above where it is determined that the monitor 3 is not within the field of view of the camera 450, the AR terminal device 4 can display an assist AR image (in the example of Figure 9, a wipe image WI and, as a further auxiliary image, a text image TI reading "Check! →") that corresponds to at least a part of the image displayed on the monitor 3 (for example, a situation in which an enemy character unleashes a special move, etc.) on the transparent display 430.

[0056] By executing the inside / outside of the angle of view determination process, even if the user looks away or moves to a position where the monitor 3 is not visible, the user can check the progress of the game through the AR terminal device 4. This makes it possible to prevent the user from missing the progress of the game.

[0057] <Information expansion processing> When the AR terminal device 4 executes the visual acuity information setting process and it is determined that the user cannot visually recognize characters and the like displayed on the monitor 3, that is, when it is determined that the size of characters and the like is too small for the user (or, more specifically, when it is determined that the characters and the like are difficult to see with the user's visual acuity), the AR terminal device 4 generates an enlarged image EI by enlarging the information, as shown in Fig. 7. Note that the enlarged image EI in this case may be displayed anywhere on the transparent display 430, but it is preferable that it is disposed at a position that does not affect the visibility of the game image GI displayed on the monitor 3 (where the game image GI is not hidden or the area of ​​the hidden image is small), or the enlarged image E1 is displayed transparently.

[0058] Whether or not the user can visually recognize characters and the like displayed on the monitor 3 is determined as follows. For example, a set of information on the size of characters and whether or not the characters can be visually recognized is prepared for each level of visual acuity (for example, if the visual acuity of both eyes is 1.0, the image can be visually recognized if it is at least this size), and the AR terminal device 4 detects characters and the like displayed on the monitor 3, compares it with the visual acuity information of the user registered in advance, and determines (estimates) whether or not the user can visually recognize the image. In this way, the AR terminal device 4 can automatically determine whether or not the user can visually recognize characters and the like displayed on the monitor 3 without relying on the user's input.

[0059] <Superimposition processing> The AR terminal device 4 can superimpose the AR image AI on the real space RS (including the game image GI) visually recognized through the transparent display 430 and display it on the transparent display 430. This allows the user to grasp, for example, information supplementing information related to the game image GI from the AR image AI.

[0060] <Hardware configuration of System 1> As shown in FIG. 1, the system 1 includes a game device 2 on which a game is executed, a monitor 3 on which a game image GI is displayed, and an AR terminal device 4 equipped with a camera 450 capable of photographing the monitor 3.

[0061] A user account is assigned to each game device 2 in association with the game device 2. This user account is managed as a game user ID (user identification information).

[0062] Furthermore, a user account is assigned to the AR terminal device 4 in association with the AR terminal device 4. This user account is managed as a terminal user ID (user identification information).

[0063] The game user ID and the terminal user ID are stored in the game device 2 and the AR terminal device 4 in association with each other based on a user's operation.

[0064] When the AR terminal device 4 communicates with the game device 2 via the communication network 5, a terminal user ID is transmitted from the AR terminal device 4. The transmitted terminal user ID is subjected to predetermined authentication in the game device 2. This enables communication between the game device 2 and the AR terminal device 4.

[0065] It should be noted that the present invention is not limited to the above, and the user may also register an existing terminal user ID (or an existing game user ID) as a game user ID (or a terminal user ID).

[0066] <Hardware configuration of game device 2> The game device 2 is a stationary game device, and has various functions for providing the monitor 3 with information regarding the progress of a game application.

[0067] The game device 2 can provide the monitor 3 with, for example, various types of image information for outputting game images GI and game sounds.

[0068] In addition, when the game device 2 receives outside-angle-of-view information from the AR terminal device 4, it can provide the AR terminal device 4 with information related to an assist AR image (e.g., the wipe image WI and text image TI in FIG. 9) for assisting the game image GI.

[0069] The game device 2 is composed of a game console equipped with a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). The game device 2 is connected to a monitor 3 and a controller 250.

[0070] As shown in FIG. 2, the game device 2 includes a control unit 20, a storage unit 21, a network interface 22, a graphics processing unit 23, an audio processing unit 24, and an operation unit 25.

[0071] The storage unit 21 , the network interface 22 , the graphics processing unit 23 , the audio processing unit 24 , and the operation unit 25 are electrically connected to the control unit 20 via a bus 29 .

[0072] The control unit 20 controls the operation of the game device 2. The control unit 20 transmits and receives various information and data, and processes various information and data. The control unit 20 includes a CPU, a semiconductor memory, and the like (not shown). The functions of the control unit 20 will be described later.

[0073] The storage unit 21 stores various types of information and data and is configured with a hard disk drive (HDD), a solid state drive (SSD), a RAM, a ROM, and the like.

[0074] The storage unit 21 stores various data downloaded from a predetermined server (for example, a server provided by an application store), various programs such as game application programs, etc. The storage unit 21 also stores a game user ID, a terminal user ID, etc.

[0075] The network interface 22 is communicatively connected to a communication network 5 in order to transmit and receive various data between the game device 2 and the AR terminal device 4.

[0076] The graphics processing unit 23 generates various images in a moving image format in accordance with various image information output from the control unit 20. The images (game images GI) generated by the graphics processing unit 23 are output (displayed) on the monitor 3.

[0077] The audio processing unit 24 generates various sounds in accordance with various types of audio information output from the control unit 20. Sounds (e.g., background music, character voices) related to the progress of the game generated by the audio processing unit 24 are output as various sounds from a speaker built into the monitor 3.

[0078] An operation signal from the user is input to the operation unit 25. The operation signal from the user is input to the operation unit 25 via a controller 250, which is an input position detection device.

[0079] <Description of Functions of the Control Unit of Game Device 2> The control unit 20 of the game device 2 executes a game application program to function as each of the following means and execute each of the following steps.

[0080] <Explanation of game progress management steps> The control unit 20 executes the game progress management step by functioning as a game progress management means. The following describes the process executed by the control unit 20 as the game progress management means.

[0081] The control unit 20 controls the progress of the game executed on the game device 2 and manages the progress status. For example, the control unit 20 controls the actions of the player character based on an operation signal from the controller 250, and also controls the non-player characters controlled by the control unit 20 to progress the game.

[0082] <Description of Communication Step> The control unit 20 executes the communication step by functioning as a communication means. Hereinafter, the processing executed by the control unit 20 as the communication means will be described.

[0083] The control unit 20 transmits the progress status of the game executed on the game device 2 (for example, score, operation character, physical strength of non-player characters, BGM, vibration pattern) to the monitor 3 and the controller 250.

[0084] In addition, the control unit 20 receives the operation signal from the controller 250 and the off-angle information from the AR terminal device 4.

[0085] When the control unit 20 receives the off-angle information, it transmits the information related to the assist AR image (for example, wipe image WI, text image TI) to the AR terminal device 4.

[0086] <Hardware Configuration of AR Terminal Device 4> The AR terminal device 4 is, for example, a head-mounted display.

[0087] As shown in FIG. 3, the AR terminal device 4 includes a transmissive display 430 having an optical system such as a half mirror or a prism, a speaker 440, a camera 450 (not shown, including an in-camera attached to the user side), setting buttons etc. 460, a frame 470 supporting the transmissive display 430, and a band 480 for fixing the AR terminal device 4 to the user's head.

[0088] The AR terminal device 4 includes a CPU, a ROM, and a RAM.

[0089] As shown in FIG. 3, the AR terminal device 4 includes a control unit 40, a storage unit 41, a network interface 42, a graphic processing unit 43, an audio processing unit 44, an imaging unit 45, and an operation unit 46.

[0090] The storage unit 41 , the network interface 42 , the graphics processing unit 43 , the audio processing unit 44 , the imaging unit 45 , and the operation unit 46 are electrically connected to the control unit 40 via a bus 49 .

[0091] The control unit 40 controls the operation of the AR terminal device 4. The control unit 40 transmits and receives various information and data, and processes various information and data. The control unit 40 includes a CPU, a semiconductor memory, etc. The functions of the control unit 40 will be described later.

[0092] The storage unit 41 stores various types of information and data and is configured with an HDD, an SSD, a RAM, a ROM, and the like.

[0093] The storage unit 41 stores various data downloaded from a predetermined server (for example, a server of a manufacturer of the AR terminal device 4), various programs such as an AR application program, etc. The storage unit 41 also stores a game user ID, a terminal user ID, user's eyesight information, user's visual acuity information, etc.

[0094] The network interface 42 is communicatively connected to the communication network 5 in order to transmit and receive various data between the AR terminal device 4 and the game device 2.

[0095] The graphic processing unit 43 can render various images in a video format. The AR image AI and the like rendered by the graphic processing unit 43 are output on the transmissive display 430. As described above, the transmissive display 430 includes an optical system such as a half mirror or a prism, and the user can directly view the real space RS (without going through a computer image based on the real space RS captured by the camera 450).

[0096] Furthermore, the graphics processing unit 43 executes monitor recognition processing, image recognition processing, and the like on the game image GI captured by the camera 450.

[0097] The audio processing unit 44 outputs (plays) various voices to the speaker 440 according to various voice information output from the control unit 40. Note that the audio processing unit 44 can also output voices related to the progress of the game (for example, BGM, character voices) to the speaker 440 as various voices.

[0098] The imaging unit 45 activates the camera 450 and captures the real space RS including the game image GI displayed on the monitor 3.

[0099] An operation signal from the user is input to the operation unit 46 via the setting button 460 etc.

[0100] <Description of the functions of the control unit 40 of the AR terminal device 4> By executing the AR application program, the control unit 40 functions as the following respective means and executes the following respective steps.

[0101] <Description of the recognition step> The control unit 40 executes the recognition step by functioning as the recognition means. Hereinafter, the processing executed by the control unit 40 as the recognition means will be described.

[0102] The control unit 40 executes monitor recognition processing. For example, the control unit 40 executes processing for matching the frame 3F of the monitor 3 displayed on the transmissive display 430 and the frame object 4F displayed on the transmissive display 430 (in this case, the control unit 40 can also execute processing for matching the computer image based on the frame 3F of the monitor 3 captured by the camera 450 and the frame object 4F). When the frame 3F and the frame object 4F match, the control unit 40 recognizes the monitor 3 (registers the position and orientation of the monitor 3).

[0103] Furthermore, the control unit 40 executes image recognition processing on the game image GI captured in a capturing step described below. Specifically, the control unit 40 captures the game image GI displayed on the monitor 3 by the camera 450, and analyzes the game image GI. As described above, the control unit 40 changes the method of analyzing the game image GI depending on the type of video game (e.g., action game, sports game). In this way, the AR terminal device 4 grasps the progress of the game and the like from information such as the image displayed in the game image GI.

[0104] In addition, when outside-of-field-of-view information is generated but outside-angle-of-view information is not generated, the control unit 40 executes within-angle-of-view processing and determines information that should prompt the user's attention from the game image GI (game progress).

[0105] <Explanation of shooting steps> The control unit 40 executes the photographing step by functioning as a photographing unit. The process executed by the control unit 40 as a photographing unit will be described below.

[0106] The control unit 40 activates the camera 450 to capture an image of the real space RS including the game image GI. The captured game image GI is stored in the storage unit 41 and is used in the above-mentioned recognition step.

[0107] <Setup Steps> The control unit 40 executes the setting step by functioning as a setting unit. The process executed by the control unit 40 as the setting unit will be described below.

[0108] The control unit 40 sets the visual acuity information of the user (executes visual acuity information setting processing) on, for example, a menu screen, etc. The control unit 40 also causes the storage unit 41 to store the visual acuity information of the user (part of the visual recognition information).

[0109] Furthermore, the control unit 40 generates visual information (part of visual recognition information / out-of-view information, out-of-view information) such as determination information as to whether the monitor 3 is within the user's field of view, and determination information as to whether the monitor 3 is within the angle of view of the camera 450, based on the registered position and orientation of the monitor 3 and the position and orientation of the AR terminal device 4. Furthermore, the control unit 40 stores the user's visual information in the storage unit 41.

[0110] <Description of additional steps> The control unit 40 executes the adding step by functioning as an adding unit. The process executed by the control unit 40 as the adding unit will be described below.

[0111] The control unit 40 executes an addition process. In this addition process, the control unit 40 generates an AR image AI that supplements the game image GI captured by the camera 450, based on the user's visual information and the result of the image recognition process (the analysis result of the game image GI).

[0112] Specifically, for example, in a fighting game, when the control unit 40 determines that a character is about to unleash a special move (detects an image of a character unleashing a special move), it generates an AR image AI that expresses a wave suggesting the power of the special move, and an AR image AI that is composed of a text message such as "BOOOM!", as shown in Figure 5.

[0113] In addition, based on the generation of the out-of-field information in the setting step, the control unit 40 generates a text image TI as an AR image AI to encourage the user to pay attention to the game image GI, as shown in Figure 8.

[0114] Furthermore, the control unit 40 transmits the outside angle of view information generated in the setting step to the game device 2, thereby acquiring information related to the assist AR image (a wipe image WI and a text image TI in the example of FIG. 9) from the game device 2. Then, the control unit 40 generates the wipe image WI and the text image TI based on the information related to the assist AR image. Note that the addition process when the angle of view information has been generated is hereinafter referred to as a "wipe addition process."

[0115] Furthermore, when the control unit 40 determines that the user cannot visually recognize information because the size of characters, etc., in the game image GI is determined to be too small for the user's eyesight, the control unit 40 executes information enlargement processing. This makes it easier for the user to perceive information by displaying, for example, an enlarged AR image (in the example of FIG. 7, an enlarged image EI obtained by enlarging an image, etc., that is too small).

[0116] <Explanation of the convolution step> The control unit 40 executes the superimposing step by functioning as a superimposing unit. The process executed by the control unit 40 as the superimposing unit will be described below.

[0117] The control unit 40 superimposes the generated AR image AI on the real space RS (including the game image GI) that can be viewed through the transparent display 430, on the transparent display 430. This allows the user to view, in addition to the game image GI, the AR image AI that supplements the game image GI (for example, in the example of FIG. 5, a wave image that suggests the power of the special move used by the character).

[0118] <Communication step description> The control unit 40 executes the communication step by functioning as a communication unit. The process executed by the control unit 40 as a communication unit will be described below.

[0119] The control unit 40 transmits the terminal user ID, the outside angle of view information, and the like to the game device 2.

[0120] Further, the control unit 40 receives information regarding the wipe image WI and the text image TI, which are assist AR images, from the game device 2.

[0121] <Explanation of AR synthesis process> Hereinafter, with reference to FIG. 10, the AR synthesis process of the present invention performed in the system 1 will be described.

[0122] First, based on the user's operation, the AR terminal device 4 executes a monitor recognition process (step S101). As described above, when the user aligns the frame object 4F with the frame 3F of the monitor 3, the AR terminal device 4 registers the position and orientation of the monitor 3.

[0123] Next, based on the user's operation, the AR terminal device 4 executes a user's visual acuity information setting process (step S102). As described above, based on the user's operation, by inputting the user's visual acuity information (right eye visual acuity "0.7", left eye visual acuity "0.5") into the AR terminal device 4, the user's visual acuity information is registered in the AR terminal device 4. Note that the monitor recognition process (step S101) and the visual acuity information setting process (step S102) may be executed in any order, not limited to the flow of FIG. 10.

[0124] Next, the AR terminal device 4 captures the real space RS (which may include the game image GI displayed on the monitor 3) by the camera 450 (step S103).

[0125] Next, the AR terminal device 4 executes a field-of-view inside / outside determination process, detects the user's line-of-sight direction with an in-camera (not shown), and determines whether the monitor 3 is within the user's field of view (step S104).

[0126] When it is determined that the monitor 3 is within the user's field of view (S104: YES), the AR terminal device 4 executes image recognition processing on the captured game image GI (step S105). Specifically, when the video game is an action game, the AR terminal device 4 analyzes information from the image displayed on the monitor 3, such as the current round number in a battle in which the first player to win two rounds out of three rounds wins, whether an image of a character unleashing a special move is displayed, and whether the operated character or the enemy character is in an advantageous position.

[0127] Next, the AR terminal device 4 determines whether or not the user can visually recognize all the information in the game image GI (images of characters, text, UI, etc.) based on the user's visual acuity information (step S106).

[0128] If it is determined that the user is able to visually recognize all the information in the game image GI (S106: YES), the process proceeds to step S108, which will be described later.

[0129] On the other hand, when the AR terminal device 4 determines that the user cannot visually recognize all the information of the game image GI (S106: NO), it executes information enlargement processing (step S107). For example, the AR terminal device 4 generates information for generating an enlarged image EI related to the information (images of characters, text, UI, etc.) that the AR terminal device 4 determines that the user cannot visually recognize.

[0130] Next, the AR terminal device 4 executes additional processing, and generates an AR image AI (or an enlarged image EI, if applicable) to be displayed on the transparent display 430 based on the results of the executed processing (processing related to determining whether the user can see all the information of the game image GI, processing to determine whether it is inside or outside the field of view, and image recognition processing) (step S108).

[0131] Next, the AR terminal device 4 executes a superimposition process, and displays the AR image AI by superimposing it on the real space RS and the game image GI displayed via the transmissive display 430 (step S109).

[0132] Returning to step S104, if the AR terminal device 4 determines that the monitor 3 is not within the user's field of view (S104: NO), it executes an inside / outside field of view determination process and determines whether the monitor 3 is within the field of view of the camera 450 (step S110).

[0133] When the AR terminal device 4 determines that the monitor 3 is outside the angle of view of the camera 450 (S110: NO), it generates outside-angle-of-view information, transmits it to the game device 2, and executes wipe addition processing (step S111). Specifically, the AR terminal device 4 obtains information such as a wipe image WI, which is an assist AR image, and a text image TI from the game device 2 via the Internet line.

[0134] Next, the AR terminal device 4 displays a wipe image WI and a text image TI as shown in FIG. 9, superimposed on the real space RS displayed via the transmissive display 430 (step S109).

[0135] On the other hand, if the AR terminal device 4 determines that the monitor 3 is within the angle of view of the camera 450 of the AR terminal device 4 (S110: YES), it executes image recognition processing (step S112) and generates out-of-field information to execute within-angle of view processing (step S113).

[0136] Next, the AR terminal device 4 executes additional processing to generate an AR image AI (AR image AI related to information that the user should particularly pay attention to) to be displayed together with the real space RS and the game image GI (step S108). For example, as shown in Fig. 8, the AR terminal device 4 generates, as the AR image AI, a text image TI saying "Oh no! Defend yourself!" to inform the user who is looking away from the game image GI that the enemy character is about to activate a technique, in addition to a motion image of the enemy character about to activate a technique.

[0137] Next, the AR terminal device 4 executes a superimposition process, and displays the AR image AI (the AR image AI and the text image TI as shown in FIG. 8) together with the image of the real space RS (including the game image GI) on the transmissive display 430 (step S109). The AR synthesis process of the present invention is carried out according to the above procedure.

[0138] In summary, the present invention provides: A system 1 for causing a computer to execute the following steps: In the display step, a game image GI (an example of a predetermined image) generated by the game device 2 is displayed on the monitor 3 (an example of a display device), In the photographing step, the game image GI displayed on the monitor 3 is photographed by a camera 450 (an example of a photographing device) provided in the AR terminal device 4 (an example of a terminal device), In the recognition step, an image recognition process is performed on the captured game image GI. In the setting step, visual recognition information (for example, visual acuity information, information outside the visual field, information outside the angle of view) of the user who operates the AR terminal device 4 is set, In the addition step, an addition process is performed using the result of the image recognition process; The addition process is a process of generating an auxiliary image (AR image AI) by adding new information that supplements the game image GI based on the visual information and the game image GI. In the superimposing step, the AR image AI is superimposed on the game image GI and displayed on the transmissive display 430 of the AR terminal device 4.

[0139] <Effects of the Invention> A technology has been known in the past in which a television image is captured by a terminal device, and an AR image related to the television image is displayed on the terminal device. However, in the conventional technology, only an AR image related to content synchronized with a television broadcast is displayed on a mobile terminal.

[0140] Even if the terminal device can estimate the position of the television as in the conventional technology, if the terminal device does not capture the television image, AR synthesis is not performed, so the user may miss the provided content.

[0141] In contrast, the present invention can display an auxiliary image (AR image) that supplements the game image on an AR terminal device by using the user's visual information (eyesight information, visual information) along with the results of image recognition processing on the game image displayed on the monitor.

[0142] This allows the user to expand the real world and provide a new type of entertainment that has never been seen before. For example, if it is determined that the user has poor eyesight and cannot fully see the information related to the game image, the text etc. will be enlarged and displayed as an AR image to complement the information.

[0143] In addition, if it is determined that the monitor is not within the user's field of view, an AR image can be used to prompt the user to pay attention to the monitor. Also, if it is determined that the monitor is outside the camera's field of view, information related to the content displayed on the monitor (for example, information that the user should pay attention to) is displayed as an AR image, thereby preventing the user from missing the content.

[0144] Furthermore, according to the present invention, by using an AR terminal device with multiple users (e.g., family members, friends), the users can synchronize content and enjoy the content together as a family while doing housework, raising children, chatting, etc. Therefore, for example, a family member who is out hanging out the laundry and not in the living room can enjoy a TV program that other family members are watching in the living room at the same time.

[0145] In addition, from the perspective of content providers (e.g., game developers), linking with AR will enable them to provide more interesting games that will prevent content from being missed.

[0146] As described above, according to the present invention, it is possible to provide a system and a computer program capable of providing an AR synthesis technique that is highly visible and interesting.

[0147] [Other embodiments] In the above embodiment, the distribution device is a game device, but the present invention is not limited to this. For example, the distribution device may be a server installed in a television station, an online video distribution company, or the like.

[0148] In this case, the AR terminal device can capture and analyze a television program broadcast on a television screen or an online video stream displayed on a tablet, and display an AR image that matches the displayed image.

[0149] Furthermore, when an advertisement is displayed on the monitor, the advertisement may be image-recognized, and an AR image of the product (for example, an image showing how to use the product) may be displayed on the AR terminal device.

[0150] In addition, if the content being distributed from the distribution device is a live broadcast of a professional baseball game and a monitor is located outside the user's field of view, the image displayed on the monitor may include information about the game situation on the AR terminal device, such as whether it is the first batting or second batting (which team is batting), whether or not a commercial is being displayed, and the scores of both teams.

[0151] In this case, when an event that the user should pay attention to is displayed on the monitor, such as when one of the teams has a chance or is in a pinch, the AR terminal device can also display a text message indicating that the user should pay attention (for example, the words "Attention!") as an AR image.

[0152] In addition, conversely, if it is determined that the user is watching the television and the camera of the AR terminal device detects the presence of a person or living thing other than the user, an image or text message (e.g., "!" or "Attention") informing the user of their presence may be displayed as an AR image.

[0153] In the above embodiment, the game device is a stationary game device in which the game device main body, which is a distribution device, the monitor, and the controller are configured as separate devices, but the present invention is not limited to this. For example, the game device may be a portable game device in which the game device main body, the monitor, and the controller are integrated.

[0154] In addition, in the above embodiment, an example is described in which the AR terminal device includes a transparent display and a camera, but the present invention is not limited to this. For example, the AR terminal device may be configured by fitting a smartphone, tablet, or the like that includes a transparent display and a camera into a frame for a head mounted display.

[0155] Furthermore, various types of monitors are possible, such as television receivers, smartphones, and tablets.

[0156] In addition, in the above embodiment, an example is described in which the AR terminal device is an optically transparent head-mounted display, but the present invention is not limited to this. The AR terminal device may be a video-transparent head-mounted display, a fully immersive head-mounted display, or the like. In the case of a video-transparent head-mounted display, the real space captured by the camera is displayed on the display as a computer image, and an AR image related to the image displayed on the monitor is superimposed on the display.

[0157] In addition, in the above embodiment, an example is described in which the terminal device (AR terminal device) and the photographing device (camera) are integrated into one device, but the present invention is not limited to this. For example, the photographing device and the terminal device may be configured as separate devices, and each device may be used by a different user.

[0158] In this case, for example, based on an image taken by user A (for example, a video of Mt. Fuji), user B who is located far away from user A can visually recognize an AR image (for example, a monster clinging to Mt. Fuji) based on the image taken by user A. In this case, the AR terminal device used by user B may be a see-through head-mounted display, or may be a fully immersive head-mounted display.

[0159] In the above embodiment, an example is described in which an AR image is superimposed on an image of real space (an image of furniture actually placed in addition to a game image), but the present invention is not limited to this. For example, only a game image and an AR image may be displayed on the display of the AR terminal device.

[0160] Furthermore, without being limited to the above embodiment, information for display as an AR image (for example, a two-dimensional code for AR display of blood splatters that is displayed when the health gauge falls to 1 / 3 or less) may be pre-embedded in the game image, and the AR terminal device may generate the AR image by photographing the game image with the embedded information using the camera of the AR terminal device.

[0161] In addition, when an image of food is displayed on the monitor, a UI (user interface) related to an online supermarket or food delivery service where that food is sold may be displayed on the liquid crystal screen of the AR terminal device.

[0162] In addition, the image displayed on the monitor is not limited to a moving image, but may be a still image. In this case, the user can use the camera of the AR terminal device to take a memorable photo displayed on the monitor, and superimpose the AR image on the memorable photo. In this case, the AR terminal device can recognize the face of a person (e.g., a grandfather) in the memorable photo, and generate an AR image (e.g., a photo of the grandparents' wedding) based on the processing.

[0163] In addition, the image captured by the AR terminal device is not limited to the image displayed on the monitor. For example, if it is determined that the user cannot visually recognize the image captured by the camera, the AR terminal device may display an auxiliary image by the AR image. This configuration is particularly effective for users with poor eyesight.

[0164] The AR terminal device can also display the positions, directions, and distances of surrounding objects and buildings using AR images. The AR terminal device can also provide AR images in combination with audio information, tactile information, and the like.

[0165] The AR terminal device can also navigate routes when reaching a destination, display destinations using AR images, and provide real-time guidance by identifying and warning of dangerous situations.

[0166] The AR terminal device can display information such as books, product labels, restaurant menus, museum exhibits, and historical places in enlarged AR images, and can also display translations.

[0167] The present invention can also be applied when a user is working using a PC (personal computer) while wearing special glasses (an example of a terminal device or an image capture device). In this case, the user can change the size of the characters displayed on the PC monitor, and display auxiliary images such as the meaning of words (a dictionary function) and the translation of sentences (a translation function) on the glasses.

[0168] Also, guidance may be displayed on the glasses when the user does not know how to operate the PC.

[0169] Furthermore, when driving a car (an example of a distribution device), information captured by an onboard camera (an example of a photographing device) (for example, a car in the passing lane turning on its turn signal to move into the driving lane) may be displayed as an AR image (for example, a text message saying "Watch out for the car on your right (or left)!") on special glass (an example of a terminal device, such as a windshield).

[0170] In addition, in the above embodiment, an example is described in which the AR terminal device executes image recognition processing and additional processing, but these processes may be executed by devices other than the AR terminal device, such as a server, a distribution device, or a broadcasting device connected to the AR terminal device via a network line.

[0171] The distribution device and the monitor may be gaming machines such as pachinko and pachinko. In this case, the camera of the AR terminal device may photograph the liquid crystal screen (monitor) of the gaming machine (one example of the distribution device) to superimpose an AR image on the AR terminal device. For example, if the gaming machine is a pachinko slot machine, the AR terminal device may display an AR image of the timing of pressing the buttons when the pressing role is hit, or an AR image of the correct pressing order when the pressing order role is hit, or the displayed color may express the chance level or display a setting suggestion.

[0172] In the above embodiment, an example is described in which an AR image that supplements a game image is displayed on an AR terminal device, but the present invention is not limited to this. For example, an auxiliary sound that supplements a game image (for example, a name of a technique, a voice message such as "Danger, get out of the way!") may be output from a speaker of a monitor.

[0173] In addition, in the above embodiment, an example is described in which the AR terminal device executes image recognition processing on a captured game image and generates an AR image based on the image recognition processing, but the present invention is not limited to this. For example, the AR terminal device may execute voice recognition processing on a sound in the real space acquired by a microphone, and generate an AR image (or an auxiliary voice that supports and explains the acquired sound in the real space) based on the sound recognition processing.

[0174] In addition, by linking home appliances such as refrigerators and lighting with terminal devices using the IoT (Internet of Things), when content (such as a baseball game) is being displayed on the terminal device, the refrigerator may output a voice such as, "The beer is cold. Things are starting to get exciting, would you like another glass?"

[0175] Also, advertisements may be displayed on the AR terminal device. This allows advertisements to be added to television, games, and video content using AR images. For example, advertisements can be shown to the user when watching sports at a ballpark or stadium, or even when the user is in the city. This allows appropriate products and services to be provided to the user as auxiliary images.

[0176] In addition, in the above embodiment, the setting of the visual acuity information is described as an example based on the user's operation, but the present invention is not limited to this. For example, the visual acuity information may be registered in the AR terminal device by reading the two-dimensional code in which the visual acuity information of the user is recorded with the camera of the AR terminal device, or by reading the two-dimensional code in which the visual acuity information of the user is recorded with the AR terminal device by NFC (near field communication).

[0177] The method for measuring visual acuity may be the Snellen chart, the Randle ring chart, the E chart, the Jagger chart, etc. Furthermore, visual acuity may be determined based on factors such as the brightness of the screen, the brightness of the room, and the presence or absence of color vision deficiency.

[0178] Furthermore, the present invention is not limited to AR display of images displayed on a monitor. For example, when it is determined that a display object that should be visually communicated to a user, such as a clock, a bulletin board, or a guide display, is in a state in which the user cannot (or has difficulty in) viewing the display object, the information may be supplemented by AR display. This can further assist users with poor eyesight.

[0179] In the above embodiment, an example is described in which the visual acuity information setting process is executed prior to the information enlargement process, but the present invention is not limited to this. For example, when a user inputs to a terminal device that the displayed characters are difficult to see, the visual acuity information setting process may be executed on the terminal device, or the information enlargement process may be executed based on the user's enlargement operation without executing the visual acuity information setting process.

[0180] In addition, in the above embodiment, an example is described in which it is assumed that the terminal device and the distribution device are not linked (AR images relating to content synchronized with the content distributed by the distribution device are not displayed on the mobile terminal), but it may also be assumed that the terminal device and the distribution device are linked. [Explanation of symbols]

[0181] 1 System 2. Distribution Device 3 Monitor (display device) 4 AR terminal device (terminal device) 450 Camera (photography device) 5. Communication Network

Claims

1. A system for causing a computer to execute the following steps: In the display step, a predetermined image is displayed on a display device; In the photographing step, the predetermined image displayed on the display device is photographed by a predetermined photographing device; In the recognition step, an image recognition process is performed on the captured predetermined image, In the setting step, visual recognition information of a user who operates a predetermined terminal device is set, In the addition step, an addition process is performed using a result of the image recognition process, the addition process is a process of generating an auxiliary image by newly adding information that supplements the predetermined image based on the visual recognition information and the predetermined image, In the superimposing step, the auxiliary image is superimposed on the predetermined image and displayed on the terminal device. system.

2. the predetermined image is an image related to a video game, In the image recognition process, an analysis based on a progress of the video game is performed based on the predetermined image, In the addition step, the addition process is performed based on the analysis. The system of claim 1 .

3. The visual recognition information is information regarding the visual acuity of the user (hereinafter referred to as “visual acuity information”) and is information indicating whether the user can visually recognize the predetermined image, In the adding step, when it is determined that the user cannot visually recognize the predetermined image based on the visual acuity information, an image related to at least a part of information of the captured predetermined image is enlarged to generate the auxiliary image. The system of claim 1 .

4. The visual recognition information is information indicating whether or not the predetermined image can be visually recognized based on predetermined information regarding the user's vision (hereinafter referred to as "visual information"); In the adding step, when it is determined that the user cannot visually recognize the predetermined image based on the visual information, an image related to the captured predetermined image is generated as the auxiliary image. The system of claim 1 .

5. In the adding step, when it is determined that the user cannot visually recognize the predetermined image based on the visual information, an image related to information related to the captured predetermined image is generated as the auxiliary image. The system of claim 4.

6. In the adding step, when it is determined that the user cannot visually recognize the predetermined image based on the visual information, information related to the predetermined image is received via an Internet line, and the auxiliary image is generated based on the received information. The system of claim 5.

7. A computer program causing a computer to execute the following steps: In the display step, a predetermined image is displayed on a display device; In the photographing step, the predetermined image displayed on the display device is photographed by a predetermined photographing device; In the recognition step, an image recognition process is performed on the captured predetermined image, In the setting step, visual recognition information of a user who operates a predetermined terminal device is set, In the addition step, an addition process is performed using a result of the image recognition process, the addition process is a process of generating an auxiliary image by newly adding information that supplements the predetermined image based on the visual recognition information and the predetermined image, In the superimposing step, the auxiliary image is superimposed on the predetermined image and displayed on the terminal device. Computer program.

Images