Systems and computer programs
The system enhances AR synthesis by capturing and recognizing images, generating auxiliary content, and superimposing it on the terminal device, ensuring continuous engagement with AR content.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- CAPCOM CO LTD
- Filing Date
- 2023-10-08
- Publication Date
- 2026-05-12
Smart Images
Figure 0007856907000001 
Figure 0007856907000002 
Figure 0007856907000003
Abstract
Description
Technical Field
[0001] The present invention relates to a system and a computer program.
Background Art
[0002] Conventionally, a television system using AR (Augmented Reality) technology has been known (see, for example, Patent Documents 1 to 3).
[0003] According to this system, by using a camera mounted on a mobile terminal used by a user to photograph an image (television image) displayed on a television, an AR image related to the television image can be displayed (AR synthesis) on the mobile terminal.
[0004] Also, according to this system, by using a gyro sensor built in the mobile terminal in combination, even when not photographing a television image with the camera, the position and orientation of the television can be estimated.
[0005] This means that by previously detecting and registering the position and orientation of the television with the mobile terminal, thereafter, even when the position and orientation of the mobile terminal change, the position and orientation of the television can be estimated.
Prior Art Documents
Patent Documents
[0006]
Patent Document 1
Patent Document 2
Patent Document 3
Disclosure of the Invention
Problems to be Solved by the Invention
[0007] However, with the aforementioned conventional technology, AR images related to content synchronized with television broadcasts are merely displayed on the mobile device.
[0008] Furthermore, if the user moves away from the television and the television image is not captured by the mobile device, AR synthesis will not be performed with conventional technology. As a result, the user may miss out on the content being offered.
[0009] The objective of the present invention is to provide a system and computer program that can provide highly visible and interesting AR synthesis technology. [Means for solving the problem]
[0010] The present invention A system that causes a computer to perform the following steps: In the display step, a predetermined image is displayed on the display device. In the shooting step, the predetermined image displayed on the display device is captured using a predetermined shooting device. In the recognition step, image recognition processing is performed on the predetermined image that has been captured. In the setup step, you set the user's visual information for operating a designated terminal device. In the additional step, the additional processing is performed using the results of the image recognition processing. The aforementioned additional processing is a process that generates an auxiliary image by adding new information to support the predetermined image based on the visual information and the predetermined image. In the superposition step, the auxiliary image is superimposed on the predetermined image and displayed on the terminal device. It is a system. [Effects of the Invention]
[0011] According to the present invention, it is possible to provide a system and computer program that can provide highly visible and interesting AR synthesis technology. [Brief explanation of the drawing]
[0012] [Figure 1] This is a diagram showing the configuration of the system in the present embodiment. [Figure 2] This is a diagram showing the configuration of the game device included in the system in the present embodiment. [Figure 3] This is a diagram showing the configuration of the AR terminal device (a) included in the system and (b) an external perspective view of the AR terminal device in the present embodiment. [Figure 4] This is a diagram showing an example of an image displayed on the AR terminal device in the present embodiment. [Figure 5] This is a diagram showing an example of an image displayed on the AR terminal device in the present embodiment. [Figure 6] This is a diagram showing an example of an image displayed on the AR terminal device in the present embodiment. [Figure 7] This is a diagram showing an example of an image displayed on the AR terminal device in the present embodiment. [Figure 8] This is a diagram showing an example of an image displayed on the AR terminal device in the present embodiment. [Figure 9] This is a diagram showing an example of an image displayed on the AR terminal device in the present embodiment. [Figure 10] This is a flowchart showing the AR synthesis process in the present embodiment. [Embodiment for Carrying Out the Invention]
[0013] [Embodiment] The system 1 according to the embodiment of the present invention will be described with reference to FIGS. 1 to 10. Note that the system 1 and the processing procedures described below are examples, and the embodiments of the present invention are not limited thereto. The system 1 and the processing procedures can be appropriately modified in design without changing the gist of the present invention.
[0014] [Description of the System] System 1 of this embodiment consists of a game device 2 (part of the "computer," an example of a distribution device), a monitor 3 (part of the "computer," an example of a display device), an AR terminal device 4 (part of the "computer," an example of a terminal device), and a network 5 that connects these devices, as shown in Figures 1 to 3.
[0015] System 1 performs AR synthesis processing based on the functions of the game device 2, monitor 3, and AR terminal device 4.
[0016] In this embodiment, the game device 2, as shown in Figure 2, is a stationary game device to which a monitor 3 for displaying game images and a controller 250 for receiving user input are externally connected. In this embodiment, for example, the game device 2 is wirelessly connected to the controller 250 and wired to the monitor 3.
[0017] As shown in Figures 3(a) and (b), the AR terminal device 4 is a head-mounted display, a type of wearable terminal, equipped with an optically transmissive display 430 (for example, a display including a transmissive liquid crystal display, a transmissive organic EL display, a transmissive inorganic EL display, etc.) that allows the user to see behind the display, a speaker 440, a camera 450 (an example of a shooting device), and setting buttons, etc. 460.
[0018] As shown in Figure 4, the user can directly view the real-world RS through the transparent display 430 (i.e., without going through computer images captured by the camera 450).
[0019] Furthermore, the AR terminal device 4 can use its camera 450 to capture a game image GI (an example of a predetermined image) displayed on the monitor 3 installed in the real space RS, as shown in Figure 5, thereby generating an auxiliary image (hereinafter referred to as "AR image AI") with newly added information to support the game image GI.
[0020] As a result, the AR terminal device 4 can display (AR composite) on the transparent display 430 not only the real-world RS (including game image GI) that can be viewed through the transparent display 430, but also AR image AI that assists the game image GI.
[0021] In generating AR image AI, the AR terminal device 4 performs image recognition processing on the game image GI captured by the camera 430. Specifically, the AR terminal device 4 understands the game's progress by analyzing the captured game image GI.
[0022] For example, as shown in Figure 5, if the image recognition process determines that an enemy character is performing a special move in the game image GI, the AR terminal device 4 performs additional processing to generate an AR image AI consisting of a wave indicating the power of the special move (to provide supplementary explanation) and a text message such as "BOOOM!", in accordance with the game image GI. In this case, the AR terminal device 4 determines the progress of the game based on, for example, the character's pose and VFX (visual effects) displayed in the game image GI, and generates the AR image AI.
[0023] Furthermore, as shown in Figure 6, if the image recognition process determines that the player character's health is low based on the game image GI (for example, the player character's health gauge HG is 10 or less remaining), the AR terminal device 4 performs additional processing to generate an AR image AI (texture TX) that shows blood splattered across the entire screen of the transparent display 430.
[0024] In addition, if the AR terminal device 4 determines that the game is running, it can also generate an AR image of fireworks being launched. The AR terminal device 4 determines that the game is running, for example, if the operation waiting screen (for example, the pause screen) is not displayed.
[0025] Furthermore, if a game image GI is captured while the game is interrupted (for example, when there has been no screen transition for more than 5 minutes, when the operation waiting screen is displayed, or when loading is in progress), the AR terminal device 4 can also generate information (such as a beer advertisement) to recommend food or drink (for example, beer) to the user.
[0026] Furthermore, in System 1 of this embodiment, the AR terminal device 4 generates AR image AI to support the game image GI, using not only the captured game image GI but also the user's visual information acquired in the setting process described later.
[0027] This "visibility information" refers to information about the user's visual acuity (hereinafter referred to as "visual acuity information"), specifically information relating to whether or not the user can visually perceive the game image GI. In other words, visual acuity information is information that includes an index of the user's visual acuity itself, indicating whether or not the user can recognize images such as characters and text within the game image GI, given the user's visual acuity while the game image GI is within the user's field of vision (for example, central vision of 1 to 2 degrees from the line of sight, or effective field of vision of 4 to 20 degrees from the line of sight; the same applies hereinafter). This visual acuity information can be registered in advance in the AR terminal device 4. In addition to this, for example, the AR terminal device 4 may perform a test in advance to determine whether or not the user can read text displayed on the monitor 3, or the visual acuity value may be input based on user operation. Furthermore, it may be determined whether or not the user can recognize the image based on the distance from the AR terminal device 4 (user) to the monitor 3 and the visual acuity information.
[0028] Based on the user's visual acuity information, if the AR terminal device 4 determines that the size of images such as characters and text (e.g., character names) displayed in the game image GI is too small for the user to recognize, it generates an AR image AI (hereinafter referred to as "enlarged image EI") to assist the user in recognizing images that are too small to recognize, as shown in Figure 7.
[0029] From another perspective, "visibility information" refers to information indicating whether or not the user is able to see the game image GI, based on information related to the user's vision (hereinafter referred to as "visual information"). Specifically, visual information is the information used by the AR terminal device 4 to determine whether or not the user has the game image GI in their field of view (whether or not the monitor 3 is in the user's field of view).
[0030] Whether or not monitor 3 is within the user's field of view is determined by detecting the user's gaze direction using an in-camera (not shown) provided on the user's side while the user is wearing the AR terminal device 4. Specifically, the AR terminal device 4 determines (predicts) whether or not monitor 3 is within the user's field of view by checking whether the game image GI is within the central vision or effective field of view relative to the user's gaze direction detected by the in-camera. As a result, the AR terminal device 4 can automatically determine whether or not monitor 3 is within the user's field of view without relying on user input. In Figure 8, the user is looking to the left of the page in Figure 8, and it is determined that the game image GI is not within the user's field of view.
[0031] If the AR terminal device 4 determines, using the user's visual information, that the monitor 3 is within the camera 450's field of view, but that the monitor 3 is not within the user's field of view (i.e., out-of-field information is generated as described later), it generates an AR image (text image TI with the message "Oh no! Defend yourself!") to draw the user's attention, as shown in Figure 8.
[0032] Furthermore, if the AR terminal device 4 determines, using the user's visual information, that the monitor 3 is not within the field of view of the camera 450 (i.e., if out-of-field information, as described later, is generated), it generates an assist AR image (wipe image WI, text image TI with the word "Check!") to support the game image GI, as shown in Figure 9.
[0033] The AR synthesis process performed by System 1 will be described in detail below.
[0034] System 1 can perform the following processes related to AR synthesis processing through the respective functions of the game device 2, monitor 3, and AR terminal device 4.
[0035] <Game progress management processing> Game device 2 runs a game application and controls the progress of the video game. Information regarding the game progress is transmitted to monitor 3.
[0036] <Display Processing> Monitor 3 displays game images (GI) on its screen based on information about the game's progress transmitted from game device 2.
[0037] <Monitor Recognition Process> The AR terminal device 4 can recognize the monitor 3 by using the camera 450.
[0038] One way to make the AR terminal device 4 recognize the monitor 3 is, for example, to match the frame object 4F displayed on the transparent display 430 with the frame 3F arranged around the monitor 3, as shown in Figure 4 (note that the frame object 4F and frame 3F do not need to be a perfect match, and the frame 3F may be enclosed within the frame object 4F).
[0039] In addition, other methods can be employed, such as detecting light-emitting elements (not shown) located, for example, on the top (or around) of the monitor 3 with the camera 450, or setting the position and orientation of the monitor 3 in relation to acquired room information (e.g., a room map) based on user operations on setting buttons 460 of the AR terminal device 4.
[0040] Once monitor 3 is recognized by the AR terminal device 4, the AR terminal device 4 uses information from at least one of its GPS function, gyroscope, accelerometer, and magnetic sensor to detect its location and orientation, and based on that, can recognize the location and orientation of monitor 3. The location and orientation of monitor 3 (the orientation of the screen on which the game image GI is displayed) are stored in the storage unit 41 of the AR terminal device 4.
[0041] Once this monitor recognition process is executed, the AR terminal device 4 can perform image recognition processing, field of view determination processing, and field of view determination processing, which will be described later.
[0042] <Image Processing> The AR terminal device 4 can use the camera 450 to capture game images (GI) displayed on the monitor 3. The captured game images (GI) are stored in the memory unit 41 of the AR terminal device 4. The objects to be captured include not only the game images (GI) but also the entire real-world space (RS), such as the monitor 3, TV stand, and bookshelf.
[0043] <Image Recognition Processing> The AR terminal device 4 can analyze captured game images (GI) (image recognition processing). For example, in a fighting game, the AR terminal device 4 can analyze the user's fighting style, such as whether or not they are bad at defense; in an action game, it can analyze the user's skill level, such as whether or not they are getting game over too often; and in a baseball game, it can analyze the user's pitching tendencies. For example, the AR terminal device 4 can determine whether or not the user is bad at defense by the degree to which the health gauge is decreasing, and it can determine whether or not they are getting game over too often by the number of times the word "GAMEOVER" is displayed on the monitor 3 within a predetermined period. The AR terminal device 4 can also analyze play tendencies using game images (play images, UI images, etc.).
[0044] <Additional processing> Once the aforementioned image recognition process is executed, the AR terminal device 4 can analyze the captured game image GI and generate (or select; the same applies hereinafter) an AR image AI based on the analysis results, without needing to be linked with the game device 2 or other devices (without requiring real-time acquisition of information synchronized with the game progress on the game device 2 from other devices other than the AR terminal device 4).
[0045] For example, in addition to the aforementioned AR image AI for waves (Figure 5) and AR image AI for blood splatter texture TX (Figure 6), if the video game is a baseball game, the AR terminal device 4 can analyze the distance of a home run and generate an AR image corresponding to the distance (for example, a fireworks image of a size corresponding to the distance). For example, regarding the distance of a home run, the AR terminal device 4 analyzes the trajectory of the ball and the text indicating the distance displayed on the game side (for example, "Distance 125m").
[0046] <Visual Acuity Information Setting Process> The AR terminal device 4 can set the user's visual acuity information, for example, on a menu screen displayed on the transparent display 430. For example, the user can register their visual acuity information with the AR terminal device 4 by inputting information such as "0.7" for the user's right eye, "0.5" for the user's left eye, and "presbyopia" based on the user's actions. The user's visual acuity information is stored in the memory unit 41 of the AR terminal device 4. Note that this visual acuity information setting process is one of the setting processes.
[0047] Furthermore, the AR terminal device 4 can display images of varying sizes (for example, Landolt rings used in vision tests, letters, or numbers) on the transparent display 430 (or on the monitor 3), and allow the user to confirm whether they can see them. The user's vision information can also be registered in the AR terminal device 4 based on user input indicating whether they can see each image.
[0048] <In-field of view determination process and field of view determination process> The AR terminal device 4 performs a field of view determination process to determine whether or not the monitor 3 is within the user's field of view.
[0049] If it is determined that Monitor 3 is within the user's field of view, neither out-of-field information nor out-of-view information will be generated (neither the flag for out-of-field information nor the flag for out-of-view information will be set).
[0050] On the other hand, if it is determined that monitor 3 is not within the user's field of view, the AR terminal device 4 performs a field of view determination process and generates visual information appropriate to the situation.
[0051] The following are examples of situations in which it is determined that Monitor 3 is not within the user's field of view: (1) If it is determined that the monitor 3 is being photographed by the camera 450 of the AR terminal device 4 (i.e., the monitor 3 is within the field of view of the camera 450), but it is determined that the monitor 3 is not within the user's field of view (i.e., out-of-field information is generated, i.e., a flag related to out-of-field information is set) (2) If it is determined that the monitor 3 is not being photographed by the camera 450 of the AR terminal device 4 (i.e., the monitor 3 is outside the field of view of the camera 450), and therefore neither the user nor the AR terminal device 4 is able to recognize the monitor 3 (if out-of-field information is generated, i.e., if a flag related to out-of-field information is set)
[0052] In the case of (1) described above, the camera 450 of the AR terminal device 4 is determined to be able to recognize the game image GI displayed on the monitor 3, and therefore, as shown in Figure 8, the AR terminal device 4 generates an AR image AI based on the captured game image GI based on the image recognition process.
[0053] In this case, in addition to image recognition processing, the AR terminal device 4 performs in-field processing based on out-of-field information, and as shown in Figure 8, determines from the game's progress information information that it particularly wants the user to pay attention to in order to give them a reason to focus on the monitor 3 (to bring the game image GI into the user's field of view). The AR image AI (text image TI) related to the information that it wants the user to pay attention to is displayed near the user's line of sight on the transparent display 430, for example, as shown in Figure 8.
[0054] In the case of (2) above, the AR terminal device 4 transmits out-of-field information to the game device 2, which in turn transmits information for displaying an assist AR image (for example, whether the controlled character won or not, whether it was a heads attack or a tails attack, etc.) to the AR terminal device 4 via the internet connection.
[0055] As a result, even in the case of (2) above, where it is determined that the monitor 3 is not within the field of view of the camera 450, the AR terminal device 4 can display an assist AR image (in the example of Figure 9, a wipe image WI and a text image TI with "Check! →" written on it as a further auxiliary image) relating to at least a part of the image displayed on the monitor 3 (for example, the situation when an enemy character unleashes a special move) on the transparent display 430.
[0056] By performing a field-of-view detection process, users can check the game's progress via the AR terminal device 4 even if they are looking away or have moved to a position where monitor 3 is not visible. This prevents them from missing out on the game's progress.
[0057] <Information expansion processing> When the AR terminal device 4 performs the vision information setting process, if it is determined that the user cannot see the characters or other elements displayed on the monitor 3, that is, if the size of the characters or other elements is determined to be too small for the user (or more precisely, if it is determined to be difficult for the user to see with their eyesight), the AR terminal device 4 generates an enlarged image EI, as shown in Figure 7. In this case, the enlarged image EI may be displayed at any location on the transparent display 430, but it is preferable that it be placed in a position that does not affect the visibility of the game image GI displayed on the monitor 3 (i.e., the game image GI is not hidden, or only partially hidden), or that the enlarged image E1 is displayed transparently.
[0058] Whether or not the user can see the text displayed on monitor 3 is determined as follows. For example, information on the size of the text and whether or not the text is visible is provided as a set for each visual acuity (for example, if the visual acuity of both eyes is 1.0, the user can see the image if the text is of a certain size or larger). The AR terminal device 4 detects the text displayed on monitor 3 and compares it with the user's visual acuity information that has been registered in advance to determine (predict) whether or not the user can see the image. As a result, the AR terminal device 4 can automatically determine whether or not the user can see the text displayed on monitor 3 without relying on user input.
[0059] <Superimposition> The AR terminal device 4 can superimpose an AR image AI onto the real-world space RS (including game image GI) viewed through the transparent display 430 and display it on the transparent display 430. This allows the user to obtain information from the AR image AI to supplement the information related to the game image GI, for example.
[0060] <System 1 Hardware Configuration> As shown in Figure 1, System 1 comprises a game device 2 on which the game is executed, a monitor 3 on which game images GI are displayed, and an AR terminal device 4 equipped with a camera 450 capable of taking pictures of the monitor 3.
[0061] Game device 2 is assigned a user account associated with it. This user account is managed as a game user ID (user identification information).
[0062] Furthermore, a user account is assigned to the AR terminal device 4. This user account is managed as the terminal user ID (user identification information).
[0063] The game user ID and terminal user ID are associated with each other and stored in the game device 2 and AR terminal device 4 based on the user's actions.
[0064] When the AR terminal device 4 communicates with the game device 2 via the communication network 5, the terminal user ID is transmitted from the AR terminal device 4. The transmitted terminal user ID is then authenticated by the game device 2. This enables communication between the game device 2 and the AR terminal device 4.
[0065] Furthermore, although not limited to the foregoing, users can also register an existing terminal user ID (or an existing game user ID) as their game user ID (or terminal user ID).
[0066] <Hardware configuration of game device 2> Game device 2 is a stationary game device equipped with various functions for providing information regarding the progress of the game application to monitor 3.
[0067] The game device 2 can, for example, provide the monitor 3 with various image information such as game image GI and game sound output.
[0068] Furthermore, when the game device 2 receives out-of-field information from the AR terminal device 4, it can provide the AR terminal device 4 with information related to assist AR images (for example, the wipe image WI and text image TI in Figure 9) to support the game image GI.
[0069] Game device 2 consists of a game console body equipped with a CPU (Central Processing Unit), ROM (Read Only Memory), and RAM (Random Access Memory). Game device 2 is connected to monitor 3 and controller 250.
[0070] As shown in Figure 2, the game device 2 comprises a control unit 20, a storage unit 21, a network interface 22, a graphics processing unit 23, an audio processing unit 24, and an operation unit 25.
[0071] The memory unit 21, the network interface 22, the graphics processing unit 23, the audio processing unit 24, and the operation unit 25 are electrically connected to the control unit 20 via the bus 29.
[0072] The control unit 20 controls the operation of the game device 2. The control unit 20 transmits and receives various information and data, and processes various information and data. The control unit 20 is equipped with a CPU and semiconductor memory, etc. (not shown). The functions of the control unit 20 will be described later.
[0073] The storage unit 21 stores various types of information and data. The storage unit 21 consists of an HDD (Hard Disk Drive), an SSD (Solid State Drive), RAM, ROM, etc.
[0074] The memory unit 21 stores various data downloaded from a designated server (for example, a server provided by an application store), various programs such as game application programs, etc. The memory unit 21 also stores game user IDs, terminal user IDs, etc.
[0075] The network interface 22 is connected to the communication network 5 in a way that allows it to communicate with the game device 2 and the AR terminal device 4 in order to send and receive various types of data between them.
[0076] The graphics processing unit 23 generates various images in video format according to the various image information output from the control unit 20. The images generated by the graphics processing unit 23 (game image GI) are output (displayed) on monitor 3.
[0077] The audio processing unit 24 generates various sounds according to the various audio information output from the control unit 20. The sounds related to the game's progress (e.g., background music, character voices, etc.) generated by the audio processing unit 24 are output as various sounds through the speakers built into the monitor 3.
[0078] Operation signals from the user are input to the operation unit 25. Operation signals from the user are input to the operation unit 25 via the controller 250, which is an input position detection device.
[0079] <Description of the functions of the control unit of game device 2> The control unit 20 of the game device 2 functions as the following means and performs the following steps by executing a game application program.
[0080] <Explanation of Game Progress Management Steps> The control unit 20 executes the game progress management step by functioning as a game progress management means. The following describes the processes performed by the control unit 20 as a game progress management means.
[0081] The control unit 20 controls the progress of the game being played on the game device 2 and manages its progress. For example, the control unit 20 controls the movement of the player character based on the operation signals from the controller 250, and also controls the non-player character controlled by the control unit 20 to advance the game.
[0082] <Description of Communication Step> The control unit 20 executes the communication step by functioning as a communication means. Hereinafter, the processes executed by the control unit 20 as a communication means will be described.
[0083] The control unit 20 transmits the progress status of the game executed on the game device 2 (for example, score, operation character, physical strength of non-player characters, BGM, vibration pattern) to the monitor 3 and the controller 250.
[0084] In addition, the control unit 20 receives an operation signal from the controller 250 and off-angle information from the AR terminal device 4.
[0085] When the control unit 20 receives off-angle information, it transmits information related to assist AR images (for example, wipe image WI, text image TI) to the AR terminal device 4.
[0086] <Hardware Configuration of AR Terminal Device 4> The AR terminal device 4 is, for example, a head-mounted display.
[0087] As shown in FIG. 3, the AR terminal device 4 includes a transmissive display 430 having an optical system such as a half mirror or a prism, a speaker 440, a camera 450 (not shown, including an in-camera attached to the user side), setting buttons 460, etc., a frame 470 supporting the transmissive display 430, and a band 480 for fixing the AR terminal device 4 to the user's head.
[0088] The AR terminal device 4 includes a CPU, a ROM, and a RAM.
[0089] As shown in FIG. 3, the AR terminal device 4 includes a control unit 40, a storage unit 41, a network interface 42, a graphic processing unit 43, an audio processing unit 44, an imaging unit 45, and an operation unit 46.
[0090] The memory unit 41, the network interface 42, the graphics processing unit 43, the audio processing unit 44, the imaging unit 45, and the operation unit 46 are electrically connected to the control unit 40 via the bus 49.
[0091] The control unit 40 controls the operation of the AR terminal device 4. The control unit 40 transmits and receives various types of information and data, and processes various types of information and data. The control unit 40 is equipped with a CPU and semiconductor memory, etc. The functions of the control unit 40 will be described later.
[0092] The storage unit 41 stores various types of information and data. The storage unit 41 is composed of an HDD, SSD, RAM, ROM, etc.
[0093] The memory unit 41 stores various data downloaded from a designated server (for example, the server of the manufacturer that produces the AR terminal device 4), various programs such as AR application programs, etc. The memory unit 41 also stores the game user ID, terminal user ID, user's eyesight information, user's visual information, etc.
[0094] The network interface 42 is connected to the communication network 5 in a way that allows it to communicate with the AR terminal device 4 and the game device 2 in order to send and receive various types of data between them.
[0095] The graphics processing unit 43 can render various images in video format. The AR image AI rendered by the graphics processing unit 43 is output to the transparent display 430. As mentioned above, this transparent display 430 is equipped with an optical system such as a half-mirror or prism, allowing the user to directly view the real-world RS (without going through a computer image based on the real-world RS captured by the camera 450).
[0096] Furthermore, the graphics processing unit 43 performs monitor recognition processing, image recognition processing, and other operations on the game image GI captured by the camera 450.
[0097] The audio processing unit 44 outputs (plays) various voices to the speaker 440 in accordance with various voice information output from the control unit 40. Note that the audio processing unit 44 can also output voices related to the progress of the game (e.g., BGM, character voices) to the speaker 440 as various voices.
[0098] The imaging unit 45 activates the camera 450 and captures the real space RS including the game image GI displayed on the monitor 3.
[0099] An operation signal from the user is input to the operation unit 46 via a setting button or the like 460.
[0100] <Explanation of the functions of the control unit 40 of the AR terminal device 4> By executing the AR application program, the control unit 40 functions as the following respective means and executes the following respective steps.
[0101] <Explanation of the recognition step> The control unit 40 executes the recognition step by functioning as a recognition means. Hereinafter, the processing executed by the control unit 40 as the recognition means will be described.
[0102] The control unit 40 executes monitor recognition processing. For example, the control unit 40 executes processing for matching the frame 3F of the monitor 3 displayed on the transmissive display 430 with the frame object 4F displayed on the transmissive display 430 (in this case, the control unit 40 can also execute processing for matching the computer image based on the frame 3F of the monitor 3 captured by the camera 450 with the frame object 4F). When the frame 3F and the frame object 4F match, the control unit 40 recognizes the monitor 3 (registers the position and orientation of the monitor 3).
[0103] Furthermore, the control unit 40 performs image recognition processing on the game image GI captured in the shooting step described later. Specifically, the control unit 40 captures the game image GI displayed on the monitor 3 by the camera 450 and analyzes the game image GI. As mentioned above, the control unit 40 changes the method of analyzing the game image GI depending on the type of video game (for example, action game, sports game). As a result, the AR terminal device 4 grasps the progress of the game from the information such as images displayed in the game image GI.
[0104] Furthermore, if out-of-field information is generated but out-of-field information is not, the control unit 40 performs in-field processing and determines information that will draw the user's attention from the game image GI (game progress).
[0105] <Explanation of shooting steps> The control unit 40 executes the shooting step by functioning as an imaging means. The following describes the processes performed by the control unit 40 as an imaging means.
[0106] The control unit 40 activates the camera 450 to capture the real-world space RS, including the game image GI. The captured game image GI is stored in the memory unit 41 and used in the recognition step described above.
[0107] <Explanation of setup steps> The control unit 40 executes the setting step by functioning as a setting means. The following describes the processes performed by the control unit 40 as a setting means.
[0108] The control unit 40 sets the user's visual acuity information (performs a visual acuity information setting process), for example, on a menu screen. The control unit 40 also stores the user's visual acuity information (part of the visual perception information) in the storage unit 41.
[0109] Furthermore, the control unit 40 generates visual information (part of the visual information / out-of-field information, out-of-field information) such as determination information on whether the monitor 3 is within the user's field of view and determination information on whether the monitor 3 is within the camera 450's field of view, based on the registered position and orientation of the monitor 3 and the position and orientation of the AR terminal device 4. The control unit 40 also stores the user's visual information in the storage unit 41.
[0110] <Explanation of steps> The control unit 40 performs the addition step by functioning as an addition means. The following describes the processing performed by the control unit 40 as an addition means.
[0111] The control unit 40 performs additional processing. In this additional processing, the control unit 40 generates AR image AI that assists the game image GI captured by the camera 450, based on the user's visual information and the results of the image recognition processing (analysis results of the game image GI).
[0112] Specifically, if the control unit 40 determines, for example, that a character is about to unleash a special move in a fighting game (i.e., it detects an image of a character unleashing a special move), it generates an AR image AI consisting of a wave representing the power of the special move, and a text message such as "BOOOM!", as shown in Figure 5.
[0113] Furthermore, based on the fact that out-of-field information was generated in the setting step, the control unit 40 generates a text image TI as an AR image AI, as shown in Figure 8, to prompt the user to focus on the game image GI.
[0114] Furthermore, the control unit 40 transmits the out-of-field information generated in the setting step to the game device 2, thereby acquiring information about the assist AR image (wipe image WI and text image TI in the example in Figure 9) from the game device 2. The control unit 40 then generates the wipe image WI and text image TI based on the information about the assist AR image. The additional processing performed when field-of-field information has been generated will be referred to below as the "wipe addition processing".
[0115] Furthermore, if the control unit 40 determines that the user is unable to see the information because the size of the text or other elements in the game image GI is too small for the user's eyesight, it will perform information enlargement processing. As a result, for example, an enlarged AR image (in the example in Figure 7, an enlarged image EI created by enlarging an image that is too small) is displayed, making it easier for the user to perceive the information.
[0116] <Explanation of overlapping steps> The control unit 40 performs the superposition step by functioning as a superposition means. The following describes the processes performed by the control unit 40 as a superposition means.
[0117] The control unit 40 superimposes the generated AR image AI onto the real-world space RS (including the game image GI) which can be viewed through the transparent display 430. This allows the user to view not only the game image GI but also the AR image AI that complements it (for example, in the example in Figure 5, an image of waves suggesting the power of a special move unleashed by a character).
[0118] <Explanation of communication steps> The control unit 40 executes the communication step by functioning as a communication means. The following describes the processes performed by the control unit 40 as a communication means.
[0119] The control unit 40 transmits the terminal user ID, out-of-view information, etc., to the game device 2.
[0120] Further, the control unit 40 receives information regarding the wipe image WI and the text image TI, which are assist AR images, from the game device 2.
[0121] <Explanation of AR synthesis processing> Hereinafter, the AR synthesis processing of the present invention performed in the system 1 will be described with reference to FIG. 10.
[0122] First, based on the user's operation, the AR terminal device 4 executes monitor recognition processing (step S101). As described above, when the user aligns the frame object 4F with the frame 3F of the monitor 3, the AR terminal device 4 registers the position and orientation of the monitor 3.
[0123] Next, based on the user's operation, the AR terminal device 4 executes user visual acuity information setting processing (step S102). As described above, based on the user's operation, by inputting the user's visual acuity information (right eye visual acuity "0.7", left eye visual acuity "0.5") into the AR terminal device 4, the user's visual acuity information is registered in the AR terminal device 4. Note that the monitor recognition processing (step S101) and the visual acuity information setting processing (step S102) may be executed in any order, not limited to the flow of FIG. 10.
[0124] Next, the AR terminal device 4 captures the real space RS (which may include the game image GI displayed on the monitor 3) with the camera 450 (step S103).
[0125] Next, the AR terminal device 4 executes a field-of-view inside / outside determination process, detects the user's line-of-sight direction with an in-camera (not shown), and determines whether the monitor 3 is within the user's field of view (step S104).
[0126] If it is determined that monitor 3 is within the user's field of view (S104: YES), the AR terminal device 4 performs image recognition processing on the captured game image GI (step S105). Specifically, if the video game is an action game, the AR terminal device 4 analyzes the image displayed on monitor 3 to determine information such as the current round number in a battle where the first to win two out of three rounds wins, whether or not an image of a character performing a special move is displayed, and whether the player character or the enemy character is in an advantageous position.
[0127] Next, the AR terminal device 4 determines, based on the user's visual acuity information, whether or not the user can see all the information in the game image GI (images such as characters, text, and UI) (step S106).
[0128] If it is determined that the user can see all the information in the game image GI (S106: YES), the process proceeds to process S108, which will be described later.
[0129] On the other hand, if the AR terminal device 4 determines that the user is unable to see all the information in the game image GI (S106: NO), it performs information augmentation processing (step S107). For example, the AR terminal device 4 generates information for generating an enlarged image EI for the information (images such as characters, text, and UI) that the user has determined to be unable to see.
[0130] Next, the AR terminal device 4 performs additional processing and generates an AR image AI (or an enlarged image EI, if applicable) to be displayed on the transparent display 430 based on the results of the executed processing (processing to determine whether the user can see all the information in the game image GI, in / out of field of view determination processing, and image recognition processing) (step S108).
[0131] Next, the AR terminal device 4 performs superposition processing and displays the AR image AI by superimposing it onto the real space RS and game image GI displayed via the transparent display 430 (step S109).
[0132] Returning to step S104, if the AR terminal device 4 determines that the monitor 3 is not within the user's field of view (S104: NO), it performs a field of view determination process to determine whether or not the monitor 3 is within the field of view of the camera 450 (step S110).
[0133] If the AR terminal device 4 determines that the monitor 3 is outside the field of view of the camera 450 (S110: NO), it generates out-of-field information and sends it to the game device 2 to perform the wipe addition process (step S111). Specifically, the AR terminal device 4 obtains information such as the wipe image WI and text image TI, which are assist AR images, from the game device 2 via the internet connection.
[0134] Next, the AR terminal device 4 displays a wipe image WI and a text image TI, as shown in Figure 9, superimposed on the real space RS displayed via the transparent display 430 (step S109).
[0135] On the other hand, if the AR terminal device 4 determines that the monitor 3 is within the field of view of the camera 450 of the AR terminal device 4 (S110: YES), it performs image recognition processing (step S112) and also generates out-of-field information and performs in-field processing (step S113).
[0136] Next, the AR terminal device 4 performs additional processing and generates AR image AI (AR image AI relating to information that the user should pay particular attention to) for display along with the real space RS and game image GI (step S108). For example, as shown in Figure 8, the AR terminal device 4 generates, as AR image AI, a motion image of an enemy character about to use a technique, as well as a text image TI that says "Danger! Defend!" to inform users who have taken their eyes off the game image GI that the enemy character is about to use a technique.
[0137] Next, the AR terminal device 4 performs superposition processing and displays the AR image AI (AR image AI and text image TI as shown in Figure 8) on the transparent display 430 along with the image RS of the real space (including the game image GI) (step S109). The AR synthesis process of the present invention is performed according to the above procedure.
[0138] In summary, the present invention is: System 1 causes a computer to perform the following steps: In the display step, the game image GI (an example of a predetermined image) generated by the game device 2 is displayed on the monitor 3 (an example of a display device). In the shooting step, the camera 450 (an example of a shooting device) provided by the AR terminal device 4 (an example of a terminal device) captures the game image GI displayed on the monitor 3. In the recognition step, image recognition processing is performed on the captured game image GI. In the setup step, the user's visual information (e.g., visual acuity information, out-of-field information, out-of-view information) for operating the AR terminal device 4 is set. In the additional step, the results of the image recognition process are used to perform additional processing. The additional processing is a process that generates an auxiliary image (AR image AI) by adding new information that supports the game image GI based on visual information and game image GI. In the superposition step, the AR image AI is superimposed on the game image GI and displayed on the transparent display 430 of the AR terminal device 4.
[0139] <Effects of the invention> A technology has been known for some time in which an AR image related to a television image is displayed on a terminal device by capturing the television image with the terminal device. However, with the aforementioned conventional technology, only AR images related to content synchronized with the television broadcast are displayed on the mobile device.
[0140] Furthermore, even if the terminal device can estimate the location of the television, as with conventional technology, AR synthesis will not be performed if the terminal device has not captured a television image in the first place. As a result, users may miss out on the content being offered.
[0141] In contrast, the present invention allows for the display of auxiliary images (AR images) that complement the game images on an AR terminal device, using the results of image recognition processing on the game images displayed on the monitor, along with the user's visual information (visual acuity information, visual information).
[0142] This allows us to provide users with an expanded real world and offer them a new kind of entertainment unlike anything before. For example, if a user has poor eyesight and it is determined that they cannot fully see the information in a game image, the text and other elements will be enlarged and displayed as an AR image to compensate for the missing information.
[0143] Furthermore, if it is determined that the monitor is not within the user's field of view, an AR image can be used to prompt the user to focus on the monitor. Also, if it is determined that the monitor is outside the camera's field of view, information related to the content displayed on the monitor (for example, information that the user should pay attention to) will be displayed as an AR image, preventing the user from missing the content.
[0144] Furthermore, according to the present invention, multiple users (for example, family members and friends) can use an AR terminal device to synchronize content among them, allowing the whole family to enjoy the content while doing housework, childcare, or chatting. For example, a family member who is not in the living room, such as hanging up laundry, can enjoy the same TV program that other family members are watching in the living room at the same time.
[0145] Furthermore, from the perspective of content providers (for example, game developers), integrating with AR allows them to offer more engaging games that prevent players from missing out on important content.
[0146] As described above, the present invention provides a system and computer program that can provide highly visible and interesting AR synthesis technology.
[0147] [Other embodiments] In the above embodiment, the distribution device is described as a game device, but the present invention is not limited to this. For example, the distribution device may be a server installed at a television station, an internet video distribution company, etc.
[0148] In this case, the AR terminal device can capture and analyze television programs broadcast on a television screen, online streaming videos displayed on a tablet, etc., and display AR images that match the displayed images.
[0149] Furthermore, if an advertisement is displayed on the monitor, the advertisement may be recognized as an image, and an AR image of the product (for example, an image showing how to use the product) may be displayed on the AR terminal device.
[0150] Furthermore, if the content being distributed from the distribution device is a live broadcast of a professional baseball game, and the monitor is located outside the user's field of view, the AR terminal device may display information about the game situation, such as whether it is the top of the inning or the bottom of the inning (which team is batting), whether or not commercials are being displayed, and the scores of both teams, as images displayed on the monitor.
[0151] In this case, if an event that the user should pay attention to, such as a scoring opportunity or a critical situation for either team, is displayed on the monitor, the AR terminal device can also display a text message (for example, "Pay Attention!") as an AR image to indicate that the user should pay attention to it.
[0152] In addition, if the system determines that the user is viewing the television and the AR terminal's camera detects the presence of another person or living creature, an image or text message (for example, "!" or "Attention") may be displayed as an AR image to inform the user of its presence.
[0153] Furthermore, while the above embodiment describes a stationary game device in which the game device consists of a game device body (which is a distribution device), a monitor, and a controller, all of which are separate devices, the present invention is not limited to this. For example, the game device may be a portable game device in which the game device body, monitor, and controller are integrated into a single unit.
[0154] Furthermore, although the above embodiments describe an example in which the AR terminal device includes a transparent display and a camera, the present invention is not limited thereto. The AR terminal device may be configured, for example, by fitting a smartphone or tablet equipped with a transparent display and a camera into a frame for a head-mounted display.
[0155] Furthermore, various types of monitors can be considered, including television receivers, smartphones, and tablets.
[0156] Furthermore, although the above embodiment describes an example in which the AR terminal device is an optically transparent head-mounted display, the present invention is not limited thereto. The AR terminal device may also be a video transparent head-mounted display, a fully immersive head-mounted display, or the like. In the case of a video transparent head-mounted display, the real space captured by the camera is displayed on the display as a computer image, and an AR image relating to the image displayed on the monitor is superimposed on the display.
[0157] Furthermore, although the above embodiment describes an example in which the terminal device (AR terminal device) and the imaging device (camera) are integrated into a single device, the present invention is not limited to this. For example, the imaging device and the terminal device may be composed of separate devices, and the users of each device may be different.
[0158] In this case, for example, based on an image taken by user A (for example, a video of Mount Fuji), an AR image based on the image taken by user A (for example, a monster clinging to Mount Fuji) can be made visible to user B, who is located far away from user A. In this case, the AR terminal device used by user B may be a transparent head-mounted display or a fully immersive head-mounted display.
[0159] Furthermore, while the above embodiment describes an example in which an AR image is superimposed on an image of real space (including game images and images of furniture and other objects actually placed in the real world), the present invention is not limited to this. For example, the display of the AR terminal device may show only game images and AR images.
[0160] Furthermore, the embodiment is not limited to the above, and information for displaying as an AR image (for example, a two-dimensional code for displaying blood splatters in AR when the health gauge falls below 1 / 3) may be pre-integrated into the game image, and the AR terminal device may generate an AR image when the game image with the incorporated information is captured by the camera of the AR terminal device.
[0161] Furthermore, if an image of food is displayed on the monitor, the user interface (UI) for the online supermarket or food delivery service where that food is sold may be displayed on the LCD screen of the AR terminal device.
[0162] Furthermore, the images displayed on the monitor are not limited to videos; they may also be still images. In this case, the user can, for example, use the camera of the AR terminal device to take a picture of a memorable photo displayed on the monitor, thereby superimposing an AR image onto the memorable photo. In this case, the AR terminal device can, for example, recognize the faces of people in the memorable photo (e.g., a grandfather) and generate an AR image (e.g., a photo of grandparents' wedding) based on that recognition.
[0163] Furthermore, the images captured by the AR terminal device are not limited to those displayed on the monitor. For example, if it is determined that the user cannot see the image captured by the camera, the AR terminal device may display an auxiliary image using AR technology. This configuration is particularly effective for users with poor eyesight.
[0164] Furthermore, AR terminals can display the location, direction, and distance of surrounding objects and buildings using AR images. They can also provide AR images in combination with audio information, haptic information, and other data.
[0165] The AR terminal device can provide route guidance to reach a destination, display destinations using AR images, and provide real-time guidance, including detection and warning of dangerous situations.
[0166] AR terminals can, for example, enlarge and display information such as books, product labels, restaurant menus, museum exhibits, and historical sites using AR images, and also display translations.
[0167] Furthermore, the present invention can also be applied when a user is working on a PC (personal computer) while wearing special glasses (an example of a terminal device or imaging device). In this case, the glasses worn by the user can change the size of the characters displayed on the PC monitor, or display auxiliary images such as word meanings (function as a dictionary) or sentence translations (function as a translator).
[0168] Additionally, guidance could be displayed on the glasses when the user is unsure how to operate the PC.
[0169] Furthermore, while driving a vehicle (an example of a distribution device), information captured by an in-vehicle camera (an example of a recording device) (for example, a car in the passing lane turning on its turn signal to move into the driving lane) may be displayed as an AR image (for example, a text message such as "Watch out for the car on the right (or left)!") on special glass (an example of a terminal device, for example, the windshield).
[0170] Furthermore, although the above embodiment describes an example in which the AR terminal device performs image recognition processing and additional processing, these processes may also be performed by other devices other than the AR terminal device, such as a server, distribution device, or broadcasting device connected to the AR terminal device via a network line.
[0171] Furthermore, the distribution device and monitor may be a gaming machine such as a pachislot or pachinko machine. In this case, by taking a picture of the LCD screen (monitor) of the gaming machine (an example of a distribution device) with the camera of the AR terminal device, an AR image can be superimposed and displayed on the AR terminal device. For example, if the gaming machine is a pachislot machine, the AR terminal device may display an AR image of the timing for pressing the buttons when a winning combination is achieved, or an AR image of the correct button press sequence when a winning combination is achieved. The displayed color may also represent the degree of chance, or hints about the machine's settings may be displayed.
[0172] The above embodiment describes an example in which an AR image assisting the game image is displayed on an AR terminal device, but the present invention is not limited thereto. For example, auxiliary audio assisting the game image (e.g., the name of a move, or voice messages such as "Watch out, dodge!") may be output from the monitor's speaker.
[0173] Furthermore, while the above embodiment describes an example in which an AR terminal device performs image recognition processing on a captured game image and generates an AR image based on the results, the present invention is not limited to this. For example, an AR terminal device may perform speech recognition processing on audio from the real world acquired by a microphone and generate an AR image (or auxiliary audio that assists and explains the acquired audio from the real world) based on the results.
[0174] Furthermore, by linking home appliances such as refrigerators and lights with terminal devices via IoT (Internet of Things), when content (for example, a baseball game) is being displayed on the terminal device, the refrigerator may output voice messages such as, "The beer is cold. The atmosphere is getting exciting, would you like another one?"
[0175] Furthermore, advertisements may be displayed on the AR terminal device. This allows advertisements to be added to television, games, and video content using AR images. For example, advertisements can be shown to users while they are watching sports at a stadium or other venue, or even when they are out in the city. This allows for the provision of appropriate products and services to users as supplementary images.
[0176] Furthermore, although the above embodiment describes an example in which the setting of visual acuity information is based on user operation, the present invention is not limited thereto. For example, the visual acuity information may be registered in the AR terminal device by reading a two-dimensional code on which the user's visual acuity information is recorded with the camera of the AR terminal device, or by reading the information using NFC (Near Field Communication) between the recording medium on which the user's visual acuity information is recorded and the AR terminal device.
[0177] Visual acuity may be measured using methods such as the Snellen chart, the Landol ring chart, the E-chart, or the Jaguar chart. Furthermore, factors such as screen brightness, room brightness, and the presence or absence of color vision deficiency may also be included as criteria for determining visual acuity.
[0178] Furthermore, the present invention is not limited to AR display applied to images displayed on a monitor. For example, if it is determined that a display object that should communicate information to the user visually, such as a clock, a bulletin board, or a sign, is not visible (or difficult to see), the information may be supplemented by AR display. This can further assist users with poor eyesight.
[0179] In the above embodiment, an example is described in which a visual acuity information setting process is performed prior to the information magnification process, but the present invention is not limited thereto. For example, if a user inputs to the terminal device that the displayed characters are difficult to see, the terminal device may perform a visual acuity information setting process, or the information magnification process may be performed based on the user's magnification operation without the visual acuity information setting process being performed first.
[0180] Furthermore, although the above embodiment describes an example where it is assumed that the terminal device and the distribution device are not linked (i.e., the AR image related to the content synchronized with the content distributed by the distribution device is not displayed on the mobile terminal), it may also be assumed that the terminal device and the distribution device are linked. [Explanation of Symbols]
[0181] 1 System 2 Distribution device 3. Monitor (display device) 4. AR terminal device (terminal device) 450 Camera (imaging device) 5. Communication Network
Claims
1. A system that causes a computer to perform the following steps: In the display step, a predetermined image is displayed on the display device. In the shooting step, the predetermined image displayed on the display device is captured using a predetermined shooting device. In the recognition step, image recognition processing is performed on the predetermined image that has been captured. In the setup step, you set the user's visual information for operating a designated terminal device. In the additional step, the additional processing is performed using the results of the image recognition processing. The aforementioned additional processing is a process that generates an auxiliary image by adding new information to support the predetermined image based on the visual information and the predetermined image. In the superposition step, the auxiliary image is superimposed on the predetermined image and displayed on the terminal device. In the certification step, it is determined whether the predetermined image is within the user's field of view and whether the predetermined image is within the field of view of the imaging device. In the superposition step, if it is determined that the predetermined image is not within the user's field of view, and / or if it is determined that the predetermined image is not within the field of view of the shooting device, the content of the auxiliary image is changed. system.
2. The aforementioned predetermined image is an image related to a video game, In the image recognition process, an analysis is performed based on the predetermined image, according to the progress of the video game. In the above additional step, the additional processing is performed based on the above determination. The system according to claim 1.
3. The aforementioned visual information is information relating to the user's visual acuity (hereinafter referred to as "visual acuity information"), which indicates whether or not the user can visually perceive the predetermined image. In the aforementioned additional step, if it is determined that the user cannot see the predetermined image based on the visual acuity information, the auxiliary image is generated by enlarging an image relating to at least a portion of the information of the captured predetermined image. The system according to claim 1.
4. The aforementioned visual information is information indicating whether or not the predetermined image can be viewed based on predetermined information relating to the user's vision (hereinafter referred to as "visual information"). In the aforementioned additional step, if it is determined based on the visual information that the user cannot see the predetermined image, an image related to the captured predetermined image is generated as the auxiliary image. The system according to claim 1.
5. In the aforementioned additional step, if it is determined that the user cannot see the predetermined image based on the visual information, an image relating to the information of the captured predetermined image is generated as the auxiliary image. The system according to claim 4.
6. In the aforementioned additional step, if it is determined that the user cannot see the predetermined image based on the visual information, information relating to the predetermined image is received via the internet, and the auxiliary image is generated based on the received information. The system according to claim 5.
7. A computer program that causes a computer to perform the following steps: In the display step, a predetermined image is displayed on the display device. In the shooting step, the predetermined image displayed on the display device is captured using a predetermined shooting device. In the recognition step, image recognition processing is performed on the predetermined image that has been captured. In the setup step, you set the user's visual information for operating a designated terminal device. In the additional step, the additional processing is performed using the results of the image recognition processing. The aforementioned additional processing is a process that generates an auxiliary image by adding new information to support the predetermined image based on the visual information and the predetermined image. In the superposition step, the auxiliary image is superimposed on the predetermined image and displayed on the terminal device. In the certification step, it is determined whether the predetermined image is within the user's field of view and whether the predetermined image is within the field of view of the imaging device. In the superposition step, if it is determined that the predetermined image is not within the user's field of view, and / or if it is determined that the predetermined image is not within the field of view of the shooting device, the content of the auxiliary image is changed. Computer program.