Information processing device, program, system, and information processing method
By inserting markers into AR device displays using visual masking effects, the system addresses the challenge of positioning in low-light conditions, ensuring accurate content placement and maintaining user immersion.
Patent Information
- Application Number
- JP2022186056
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-11-21
- Publication Date
- 2024-12-04
- Estimated Expiration
- 2042-11-21
AI Technical Summary
Augmented Reality (AR) devices struggle to accurately determine their position in low-light environments, such as movie theaters or event venues, leading to potential disorientation and reduced immersive experience for users.
The system employs a mechanism to insert markers into the video displayed on AR devices for a short duration using visual masking effects, allowing the device to recognize feature points without being perceivable by the user, thereby enabling accurate content positioning in low-light conditions.
This approach enhances the immersive experience by ensuring accurate content placement on the AR device display without requiring the user to recognize the markers, thus maintaining the AR experience in dark environments.
Smart Images

Figure 0007594573000001 
Figure 0007594573000002 
Figure 0007594573000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an information processing device, a program, a system, a display device, and an information processing method. [Background technology]
[0002] Patent Document 1 describes a matching service that uses gaze data of a wearer wearing an eyeglass-type device such as AR (Augmented Reality) glasses. Patent Document 2 describes an AR display control device that can reduce the sense of discomfort when sharing a space. Patent Document 3 describes an image processing device that creates a highly stealthy latent image. Patent Document 4 describes an apparatus for adapting default encoding of a digital video signal during a scene change period. [Prior art document] [Patent documents] [Patent Document 1] JP 2022-085730 A [Patent Document 2] JP 2021-131741 A [Patent Document 3] JP 2021-004957 A [Patent Document 4] International Publication No. 2008 / 019525 Summary of the Invention [Means for solving the problem]
[0003] According to an embodiment of the present invention, there is provided an information processing device, which may include a marker insertion unit that inserts a marker for determining a display position of content related to an image displayed on a display unit of a display device while an imaging unit mounted on the display device is imaging the image displayed in a display target area, into the image for a period shorter than a period recognizable by a user of the display device.
[0004] The marker insertion unit may insert the marker into the video by executing a video processing process on the video.
[0005] In any of the information processing devices, the marker insertion section may insert the marker into the image while the image is being displayed in the display target area.
[0006] In any of the information processing devices, the marker insertion unit may insert the marker into the video by causing a marker projection device to project the marker.
[0007] Any of the information processing devices may further include an information receiving unit configured to receive user operation information indicating an operation of the user of the display device, and the marker inserting unit may insert the marker into the video based on the user operation information of the user of the display device.
[0008] Any of the information processing devices may further include an information receiving unit that receives an image captured by the imaging unit mounted on the display device, and the marker insertion unit may insert the marker into the video based on the captured image.
[0009] Any of the information processing devices may further include an information storage unit that stores attention area information indicating an attention area within the video where attention is focused, and the marker insertion unit may determine an insertion position of the marker so that the marker is inserted into an area outside the attention area indicated by the attention area information.
[0010] Any of the information processing devices may further include an information receiving unit that receives gaze information indicating the gaze of the user of the display device, and an attention area identification unit that identifies an attention area within the image where attention is focused based on the gaze of the user indicated by the gaze information, and the marker insertion unit may determine an insertion position of the marker so that the marker is inserted outside the attention area identified by the attention area identification unit.
[0011] Any of the information processing devices may further include an information storage unit that stores color-related information related to colors in the image, and the marker insertion unit may determine an insertion position of the marker based on the color-related information so that when the marker is inserted, the color difference between the color of the marker and the color in the image is greater.
[0012] Any of the information processing devices may further include an information storage unit that stores color-related information related to colors in the image, and the marker insertion unit may determine the color of the marker based on the color-related information so that when the marker is inserted at an insertion position, the color difference between the color of the marker and the color in the image is greater.
[0013] Any of the information processing devices may further include an information receiving unit that receives user-related information related to the user of the display device, and the marker insertion unit may determine an insertion period for inserting the marker into the image based on the user-related information.
[0014] Any of the information processing devices further includes an information receiving unit that receives recognition feasibility information indicating whether the display device is able to recognize the marker, and when the recognition feasibility information indicates that the display device is unable to recognize the marker, the marker insertion unit may change at least one of the insertion position of the marker, the color of the marker, and the size of the marker so that the display device can recognize the marker.
[0015] Any of the information processing devices may further include a content delivery unit that delivers the content to the display device.
[0016] According to one embodiment of the present invention, there is provided a program for causing a computer to function as the information processing device.
[0017] According to an embodiment of the present invention, there is provided a system, the system may include the information processing apparatus, and the display device.
[0018] According to an embodiment of the present invention, there is provided a display device. The display device may include a display unit. The display device may include an imaging unit. The display device may include a marker recognition unit that recognizes a marker inserted in a video displayed in a display target area from an image captured by the imaging unit for a period shorter than a period recognizable by a user of the display device. The display device may include a display position determination unit that determines a display position of a content displayed on the display unit based on the marker recognized by the marker recognition unit.
[0019] The display device may further include an information storage unit that stores angle of view information indicating an angle of view of the display target area included in the captured image, and the display position determination unit may determine the display position of the content further based on the angle of view information.
[0020] In any of the display devices, the marker recognition unit may recognize a plurality of markers inserted in the video from the captured image, and the display position determination unit may determine the display position of the content, which is 3D content, based on the plurality of markers recognized by the marker recognition unit.
[0021] According to one embodiment of the present invention, there is provided a program for causing a computer to function as the display device.
[0022] According to an embodiment of the present invention, there is provided an information processing method executed by a computer, which may include a marker insertion step of inserting, into an image displayed in a display target area, a marker for determining a display position of content related to the image displayed on a display unit of the display device, while an imaging unit mounted on the display device is imaging the image, for a period shorter than a period recognizable by a user of the display device.
[0023] According to one embodiment of the present invention, there is provided an information processing method executed by a computer. The information processing method may include a marker recognition step of recognizing a marker inserted in a video displayed in a display target area from an image captured by an imaging unit mounted on the computer for a period shorter than a period recognizable by a user of the computer. The information processing method may include a display position determination step of determining a display position of a content displayed on a display unit mounted on the computer based on the marker recognized in the marker recognition step.
[0024] The above summary of the invention does not list all of the necessary features of the present invention. Also, subcombinations of these features may also be inventions. [Brief description of the drawings]
[0025] [Figure 1] An example of a system 10 is shown diagrammatically. [Diagram 2] 1 illustrates an example of the configuration of a display device 100. [Diagram 3] FIG. 2 is an explanatory diagram for explaining an example in which the display device 100 recognizes a marker. [Figure 4] FIG. 2 is an explanatory diagram for explaining an example in which the display device 100 displays content. [Diagram 5] FIG. 11 is an explanatory diagram for explaining another example in which the display device 100 displays content. [Figure 6] FIG. 11 is an explanatory diagram for explaining another example in which the display device 100 recognizes a marker. [Figure 7] FIG. 11 is an explanatory diagram for explaining another example in which the display device 100 displays content. [Figure 8] FIG. 11 is an explanatory diagram for explaining another example in which the display device 100 displays content. [Figure 9] 1 shows an example of a captured image 80 captured by the imaging section 120 of the display device 100. [Figure 10]FIG. 11 is an explanatory diagram for explaining another example in which the display device 100 recognizes a marker. [Figure 11] 2 illustrates an example of a functional configuration of an information processing device 200. [Figure 12] 2 illustrates an example of a functional configuration of the display device 100. [Figure 13] 1 shows an example of a process flow by the system 10. [Figure 14] An example of a hardware configuration of a computer 1200 functioning as the information processing device 200 or the display device 100 is shown in schematic form. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0026] When a wearer of AR glasses watches an image projected on a screen through the AR glasses, the AR glasses need to perform self-location estimation in order to display content at an appropriate position on the display unit of the AR glasses. However, the AR glasses may not be able to grasp the feature points required to perform self-location estimation in a dark place. The system 10 according to the present embodiment employs a mechanism for inserting a marker used as a feature point for estimating the self-location of the AR glasses into the image projected on the screen for a short period of time that cannot be recognized by humans.
[0027] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.
[0028] 1 illustrates an example of a system 10. The system 10 includes a display device 100 and an information processing device 200. The system 10 may further include a communication device 300. The system 10 may further include a video display device 400. The system 10 may further include a communication terminal 150.
[0029] The system 10 is used in a facility that provides a service using AR technology that virtually expands a real space by adding virtual visual information to an image displayed in a display target area 50. The system 10 is used, for example, in a movie theater. The system 10 is used, for example, in a digital art event venue such as projection mapping. The system 10 is used, for example, in a live venue for a virtual live performance. The system 10 is used, for example, in a theme park such as an amusement park. The system 10 is used, for example, in a game center. The system 10 may be used at home. FIG. 1 shows an example of the case where the system 10 is used in a movie theater.
[0030] The display target area 50 is, for example, a screen area of a screen device. The display target area 50 is, for example, a display area of a display device. The display target area 50 is, for example, a wall area, ceiling area, floor area, etc. of a room or a building. The display target area 50 may be any other area in which an image can be displayed. FIG. 1 shows an example in which the display target area 50 is a screen area of a screen device.
[0031] The display device 100 may be any device having a wireless communication function and a display function for displaying content on a display unit. The display device 100 is, for example, a glasses-type device. The glasses-type device is, for example, AR glasses. The display device 100 may be a smartphone, a tablet terminal, or the like. FIG. 1 shows an example in which the display device 100 is AR glasses.
[0032] The content may be in any data format. For example, the content is 2D CG (Computer Graphics) data. For example, the content is 3D CG data. For example, the content is a still image. Also, for example, the content is a video. For example, the content is text. The content may include a combination of these.
[0033] The display device 100 may be a device capable of giving the user 102 the sensation that the content is placed in real space by displaying the content on a transparent or semi-transparent display unit. That is, the display device 100 may be a so-called transmissive (see-through) device. The display device 100 may be a video pass-through device. In this case, the display device 100 may display a captured image capturing a range corresponding to the field of view of the user 102 on a non-transmissive display, and may superimpose and display the content.
[0034] The display device 100 has a function of estimating its own position, for example. The display device 100 estimates its own position based on, for example, a captured image captured by an imaging unit mounted on the display device 100. The display device 100 estimates its own position by, for example, extracting feature points for estimating its own position from the captured image. The display device 100 estimates its own position based on, for example, the shape and size of the extracted feature points. The display device 100 may display content on the display unit based on a result of estimating its own position.
[0035] The information processing device 200 executes information processing related to the video. For example, the information processing device 200 executes an insertion process of inserting information into the video. For example, while an imaging unit mounted on the display device 100 is capturing an image displayed in the display target area 50, the information processing device 200 inserts a marker 60 for determining a display position of content related to the video displayed on the display unit of the display device 100 into the video for a period shorter than a period recognizable by a user 102 of the display device 100. Note that the user 102 of the display device 100 may be a person wearing the display device 100.
[0036] The information processing device 200 uses, for example, the visual masking effect to insert the marker 60 into the image for a period shorter than the period that the user 102 of the display device 100 can recognize. The visual masking effect is an effect in which, when two visual stimuli, a target stimulus and a mask stimulus, are displayed close to each other in time and space, the perception of the target stimulus is hindered compared to when only the target stimulus is displayed. Note that a case in which the mask stimulus is displayed before the target stimulus in time is called forward masking, and a case in which the mask stimulus is displayed after the target stimulus in time is called backward masking. For example, the information processing device 200 uses the visual masking effect to insert the marker 60 into the image for 70 to 80 ms or less.
[0037] The frame rate of the video is, for example, 60 fps (frames per second). In this case, the information processing device 200 may insert the marker 60 into 4 to 5 or fewer consecutive frames by utilizing a visual masking effect. The frame rate of the video may be any other value.
[0038] The marker 60 is, for example, a QR (Quick Response) code. The marker 60 is, for example, a barcode. The marker 60 may be a graphic of any shape, such as a rectangle or a circle, as long as the display device 100 can recognize it as a marker.
[0039] The marker 60 includes, for example, content-related information related to the content, content identification information for identifying the content, and data format information indicating the data format of the content.
[0040] The content related to the video may be anything as long as it is related to the video. The content related to the video may be, for example, content used in the direction of the video, content used as subtitles for the video, content used to explain the scenes in the video, content used to advertise products that appear in the video, etc.
[0041] The information processing device 200 provides the video to the communication device 300. The information processing device 200 provides the video to the communication device 300 by communicating with the communication device 300 via the network 20, for example.
[0042] The network 20 may include a core network provided by a telecommunications carrier. The core network may be compliant with, for example, a 5G (5th Generation) communication system. The core network may be compliant with a mobile communication system after the 6G (6th Generation) communication system. The core network may be compliant with a 3G (3rd Generation) communication system or an LTE (Long Term Evolution) communication system. The network 20 may include the Internet. The network 20 may be compatible with a Wi-Fi (registered trademark) communication method. The network 20 may be accessible via a Wi-Fi access point. The network 20 may be accessible using other general wireless communication. The network 20 may be accessible via a wired connection.
[0043] The communication device 300 may be any device capable of communicating with the information processing device 200. The communication device 300 is, for example, a PC such as a laptop PC (Personal Computer) or a desktop PC. The communication device 300 may also be a smartphone or a tablet terminal. Fig. 1 shows an example in which the communication device 300 is a laptop PC.
[0044] The communication device 300 transmits, for example, a video provided by the information processing device 200 to the video display device 400. The communication device 300 transmits the video to the video display device 400, for example, via a wired connection. The communication device 300 may establish a wired connection with the video display device 400 using an HDMI (registered trademark) (High-Definition Multimedia Interface) cable or the like. The communication device 300 may transmit the video to the video display device 400 via a wireless connection. The communication device 300 may establish a wireless connection with the video display device 400 using a short-range wireless communication method such as Wi-Fi, Bluetooth (registered trademark), and Zigbee (registered trademark).
[0045] The image display device 400 may be any device capable of displaying an image received from the communication device 300. The image display device 400 is, for example, a projector. The image display device 400 may also be a display device. Fig. 1 shows an example in which the image display device 400 is a projector.
[0046] The information processing device 200 distributes, for example, video-related content to the communication terminal 150. The information processing device 200 distributes the content to the communication terminal 150 by communicating with the communication terminal 150 via the network 20, for example.
[0047] The communication terminal 150 may be any device capable of communicating with the information processing device 200. The communication terminal 150 is, for example, a smartphone or a tablet terminal. The communication terminal 150 may be a PC. The communication terminal 150 may be a wearable terminal. The communication terminal 150 may be AR glasses. FIG. 1 illustrates an example in which the communication terminal 150 is a smartphone.
[0048] The communication terminal 150 transmits content related to the video distributed by the information processing device 200 to the display device 100. The communication terminal 150 transmits the content to the display device 100, for example, via a wireless connection. The communication terminal 150 may establish a wireless connection with the display device 100 using a short-range wireless communication method such as Wi-Fi, Bluetooth, or Zigbee. The communication terminal 150 may transmit the content to the display device 100 via a wired connection. The communication terminal 150 and the display device 100 may be integrated.
[0049] For example, the display device 100 recognizes a marker 60 inserted in a captured image of a video displayed in the display target area 50 captured by an imaging unit mounted on the display device 100, for a period shorter than the period during which the video can be recognized by the user 102 of the display device 100. Next, the display device 100 determines the display position of the content received from the communication terminal 150 based on the recognized marker 60. Thereafter, the display device 100 displays the content on the display unit of the display device 100 according to the determined display position.
[0050] The information processing device 200 may function as a Mobile Edge Computing or Multi-access Edge Computing (MEC). That is, the information processing device 200 may be an MEC server.
[0051] In order to display content at an appropriate position on the display unit of a display device using AR technology, it is necessary to have the display device recognize feature points for determining the display position of the content. However, in dark places such as movie theaters and event venues, the display device may not be able to recognize the feature points and may not be able to display the content at an appropriate position on the display unit. On the other hand, if the wearer of the display device recognizes the feature points, the wearer of the display device may lose interest. For the above reasons, it is desirable to display feature points for determining the display position of the content in a dark place in a manner that the display device can recognize and the wearer of the display device cannot recognize.
[0052] According to the system 10 of this embodiment, the information processing device 200 uses the visual masking effect to insert the marker 60, which is a feature point for determining the display position of the content, into the image for such a short period of time that the user 102 of the display device 100 cannot recognize it. Since the display target area 50 in which the image is displayed is a bright area in a dark place, the information processing device 200 can make the display device 100 recognize the marker 60 inserted into the image displayed in the display target area 50. Thus, the information processing device 200 can display the content at an appropriate position on the display unit of the display device in a dark place. Furthermore, since the period during which the marker 60 is inserted into the image is shorter than the period during which the user 102 of the display device 100 can recognize the marker 60, the information processing device 200 does not make the user 102 of the display device 100 recognize the marker 60 inserted into the image displayed in the display target area 50. As described above, the system 10 of this embodiment can improve the sense of immersion of a user of a display device watching video through the display device in a dark place by having the display device recognize markers and display content at an appropriate position on the display unit of the display device without the user of the display device being aware of markers.
[0053] 2 shows a schematic example of a configuration of the display device 100. The display device 100 includes a frame 110 having a rim 112 and temples 118. The display device 100 may include a battery.
[0054] The display device 100 includes, for example, an imaging unit 120. The imaging range of the imaging unit 120 may be a range corresponding to the field of view of a user 102 wearing the display device 100. The arrangement of the imaging unit 120 in Fig. 2 is an example, and the imaging unit 120 may be arranged in another position.
[0055] The display device 100 includes, for example, a sensor 122. The sensor 122 may be a sensor for detecting the movement of the user 102 wearing the display device 100. The sensor 122 is, for example, an acceleration sensor or a gyro sensor. Fig. 2 illustrates an example in which the sensor 122 is disposed on the bridge 114, but the arrangement of the sensor 122 in Fig. 2 is only an example, and the sensor 122 may be disposed in another position.
[0056] The display device 100 includes, for example, a gaze detection unit 124. The gaze detection unit 124 may detect the gaze of the user 102 who is a wearer of the display device 100. The gaze detection unit 124 detects the gaze of the user 102, for example, by capturing an image of the eyes of the user 102 and monitoring them. The arrangement of the gaze detection unit 124 in FIG. 2 is one example, and the gaze detection unit 124 may be arranged in another position.
[0057] The display device 100 may accept various inputs. The display device 100 may accept various inputs from the user 102 via an operation unit arranged at any position on the display device 100. The display device 100 accepts various inputs from the user 102 via an input unit such as a button arranged at any position on the frame 110, for example. The display device 100 may accept various inputs from the user 102 via a communication terminal 150, for example.
[0058] The display device 100 may have a display function onto the lens 116. The lens 116 may be an example of a display unit of the display device 100. The lens 116 may be a transparent or semi-transparent display, and the display device 100 may cause the lens 116 to perform various displays. That is, the display device 100 may be a transparent or semi-transparent display type device. Furthermore, the display device 100 may have a function of projecting a display onto the lens 116. That is, the display device 100 may be a projection type device.
[0059] The display device 100 may have a function of performing communication via a network 20. The display device 100 may access the network 20 by at least one of mobile communication, Wi-Fi communication, and other general wireless communication. The display device 100 may communicate with the information processing device 200 via the network 20. The information processing device 200 may deliver content related to video to the display device 100 via the network 20.
[0060] Fig. 3 is an explanatory diagram for explaining an example in which the display device 100 recognizes a marker. In Figs. 3 to 8 and 10, the x-axis direction in the diagram is the horizontal direction, and the y-axis direction in the diagram is the vertical direction.
[0061] For example, the display device 100 recognizes a marker 60, which is one marker inserted into the video by the information processing device 200, from a captured image of the video displayed in the display target area 50 captured by the imaging unit 120. The following explanations of Figs. 4 and 5 each describe an example in which, when the display device 100 recognizes the marker 60 shown in Fig. 3, the display device 100 determines the display position of the content to be displayed on the lens 116 based on the marker 60.
[0062] 4 is an explanatory diagram for explaining an example in which the display device 100 displays content. The display device 100 recognizes, for example, the marker 60 shown in FIG. 3, and determines the display position of one piece of content to be displayed on the lens 116 based on the recognized marker 60. The display device 100 may determine the display positions of a plurality of pieces of content to be displayed on the lens 116 based on the recognized marker 60. Here, an example in which the display device 100 determines the display position of one piece of content 70 to be displayed on the lens 116 based on the recognized marker 60 will be mainly described.
[0063] For example, the display device 100 determines the display position of the content 70 based on positional relationship information indicating the positional relationship between the reference position and the display position of the content 70, with the center position of the recognized marker 60 as the reference position. The display device 100 receives the positional relationship information from, for example, the information processing device 200. The display device 100 may receive the positional relationship information from the communication terminal 150. The display device 100 may read content-related information including the positional relationship information from the marker 60.
[0064] In the example shown in FIG. 4, the display device 100 is rotated in the x-axis direction from the reference position. c , y-axis direction c The remote position is determined to be the display position of the content 70. The display device 100 may cause the lens 116 to display the content 70 such that the content 70 is displayed at the determined display position.
[0065] Fig. 5 is an explanatory diagram for explaining another example in which the display device 100 displays the content. In the example shown in Fig. 5, the display device 100 determines that the center position of the marker 60 shown in Fig. 3 is the display position of the content 70. The display device 100 may cause the lens 116 to display the content 70 so that the content 70 is displayed at the determined display position.
[0066] 6 is an explanatory diagram for explaining another example in which the display device 100 recognizes a marker. For example, the display device 100 recognizes three markers, marker 62, marker 64, and marker 66, inserted into the video by the information processing device 200 from a captured image captured by the imaging unit 120 of a video displayed in the display target area 50. Each of the markers 62, 64, and 66 may have the same characteristics as the marker 60. In the following explanations of FIGS. 7 and 8, an example will be explained in which the display device 100 determines the display position of the content displayed on the lens 116 based on the markers 62, 64, and 66 when the display device 100 recognizes the markers 62, 64, and 66 shown in FIG. 6.
[0067] Fig. 7 is an explanatory diagram for explaining another example in which the display device 100 displays the content. Here, an example in which the display device 100 recognizes the markers 62, 64, and 66 shown in Fig. 6, and determines the display position of the content 70, which is one 3D content displayed on the lens 116, based on the recognized markers 62, 64, and 66, will be mainly described.
[0068] The display device 100 determines the display position of the content 70 based on, for example, correspondence information indicating that the recognized markers 62, 64, and 66 are markers for determining the display position of the content 70, and the positional relationship between the markers 62, 64, and 66. The display device 100 receives the correspondence information from, for example, the information processing device 200. The display device 100 may also receive the correspondence information from the communication terminal 150. The display device 100 may read content-related information including the correspondence information from each of the markers 62, 64, and 66.
[0069] The correspondence information includes, for example, content identification information and marker identification information for identifying a marker.
[0070] The display device 100 may determine a positional relationship between the markers 62, 64, and 66 based on the respective central positions of the markers 62, 64, and 66. The display device 100 may cause the lens 116 to display the content 70 such that the content 70 is displayed at the determined display position.
[0071] Fig. 8 is an explanatory diagram for explaining another example in which the display device 100 displays the content. Here, an example will be mainly explained in which the display device 100 recognizes the markers 62, 64, and 66 shown in Fig. 6, and determines the display positions of the three contents, that is, the content 72, the content 74, and the content 76, displayed on the lens 116, based on the recognized markers 62, 64, and 66, respectively.
[0072] For example, the display device 100 determines the display positions of the contents 72, 74, and 76 based on a plurality of pieces of correspondence information indicating that the recognized markers 62, 64, and 66 are markers for determining the display positions of the contents 72, 74, and 76. In the example shown in Fig. 8, the display device 100 determines that the center position of the marker 62 is the display position of the content 72, the center position of the marker 64 is the display position of the content 74, and the center position of the marker 66 is the display position of the content 76. The display device 100 may cause the lens 116 to display the contents 72, 74, and 76 so that the contents 72, 74, and 76 are displayed at the determined display positions.
[0073] Fig. 9 shows an example of a captured image 80 captured by the imaging section 120 of the display device 100. Fig. 9 shows an example of a captured image captured by the imaging section 120 when the user 102 and the user 502 of the display device 100 are playing a snowball fight video game displayed in the display target area 50. The captured image 80 includes a video of the snowball fight video game displayed in the display target area 50 and the user 502. The display device 100 may transmit the captured image 80 to the information processing device 200.
[0074] The information processing device 200 inserts the marker 60 into the video based on, for example, the captured image 80 received from the display device 100. For example, the information processing device 200 detects the motion of the user 502 by performing image analysis on the captured image 80. The information processing device 200 may insert the marker 60 into the video based on the detected motion of the user 502. For example, the information processing device 200 inserts the marker 60 into the video when the motion of the user 502 is a specific motion related to the video. When the video displayed in the display target area 50 is a video of a snowball fight video game, the information processing device 200 inserts the marker 60 into the video when, for example, the motion of the user 502 is a pitching motion.
[0075] The information processing device 200 may detect an overlapping area where the display target region 50 and the user 502 overlap by performing image analysis on the captured image 80. In this case, the information processing device 200 may insert the marker 60 into the video so that the marker 60 is displayed in an area outside the overlapping area.
[0076] 10 is an explanatory diagram for explaining another example of how the display device 100 recognizes a marker. The display device 100 may recognize a marker 60 that has been inserted into the outer area of the overlapping area by the information processing device 200.
[0077] 11 illustrates an example of a functional configuration of the information processing device 200. The information processing device 200 includes an information storage unit 202, an information receiving unit 204, a request transmitting unit 206, a marker inserting unit 208, an attention area identifying unit 210, a video providing unit 212, and a content distributing unit 214. Note that it is not essential that the information processing device 200 includes all of these components.
[0078] The information storage unit 202 stores various information. The information storage unit 202 stores, for example, video-related information related to a video into which the marker 60 is to be inserted. The video-related information includes, for example, a video into which the marker 60 is to be inserted. The video-related information includes, for example, attention area information indicating an attention area in the video where attention is focused. The attention area is, for example, an area occupied by an attention target such as a character or a person appearing in a video. The attention area may be a central area of the video. The video-related information includes, for example, color-related information related to a color in the video. The color-related information includes, for example, hue information indicating a hue in the video. The color-related information includes, for example, brightness information indicating a brightness in the video. The color-related information may include saturation information indicating a saturation in the video. The video-related information includes, for example, main target information indicating a main target of the video. The main target of the video is, for example, a child. The main target of the video is, for example, an adult. The main target of the video is, for example, a man. The main target of the video is, for example, a woman. The video related information may include specific action information indicating a specific action related to the video.
[0079] The information storage section 202 stores content related to the video into which the marker 60 is to be inserted. The information storage section 202 may store the content in association with video-related information.
[0080] The information storage unit 202 may store content-related information of the content. The information storage unit 202 may store the content-related information in association with the video-related information.
[0081] The information receiving unit 204 receives various information from an external device. For example, the information receiving unit 204 receives various information from an external device via the network 20. The information receiving unit 204 may store the received various information in the information storage unit 202.
[0082] The information receiving unit 204 receives various information from, for example, the display device 100. The information receiving unit 204 receives, for example, user-related information related to a user of the display device 100. The user-related information includes, for example, generation information indicating the generation of the user. The user-related information includes, for example, age information indicating the age of the user. The user-related information may include gender information indicating the gender of the user. The information receiving unit 204 may receive gaze information indicating the gaze of the user.
[0083] The information receiving unit 204 receives various types of information, for example, from the communication terminal 150. The information receiving unit 204 may receive information from the communication terminal 150 similar to the information received from the display device 100.
[0084] The information receiving unit 204 receives various information while, for example, a video is displayed in the display target area 50. The information receiving unit 204 receives, for example, user action information indicating an action of a user of the display device 100. The action of the user of the display device 100 is detected, for example, by a sensor 122 provided in the display device 100. The action of the user of the display device 100 is detected, for example, by a sensor such as an acceleration sensor or a gyro sensor provided in the communication terminal 150. The action of the user of the display device 100 may be detected by a camera installed in a space including the display target area 50. In this case, the information receiving unit 204 may receive a captured image including the user of the display device 100 captured by the camera.
[0085] The information receiving unit 204 receives, for example, a captured image captured by the imaging unit 120 mounted on the display device 100 of a video displayed in the display target area 50. The information receiving unit 204 may receive recognition feasibility information indicating whether the display device 100 has recognized the marker 60.
[0086] The request transmitting unit 206 requests the external device to transmit various information. The request transmitting unit 206 requests the external device to transmit various information, for example, via the network 20. The information receiving unit 204 may receive the various information transmitted by the external device in response to the request by the request transmitting unit 206.
[0087] The request sending unit 206 sends a request to, for example, the display device 100. The request sending unit 206 sends a request to, for example, the communication terminal 150. The request sending unit 206 may send a request to the communication device 300.
[0088] The marker inserting unit 208 inserts the marker 60 into the video into which the marker 60 is to be inserted. The marker inserting unit 208 inserts, for example, a plurality of markers 60 into the video into which the marker 60 is to be inserted.
[0089] The marker insertion unit 208 inserts a marker 60 for determining a display position of content related to an image displayed on the display unit of the display device 100 while the imaging unit 120 mounted on the display device 100 is capturing the image displayed in the display target area 50, into the image for a period shorter than a period recognizable by a user of the display device 100. The marker insertion unit 208 inserts the marker 60 into the image, for example, by executing an image processing process on the image. The marker insertion unit 208 may insert the marker 60 into the image by causing a marker projection device to project the marker 60. The marker projection device is, for example, a projector.
[0090] The marker insertion unit 208, for example, determines the insertion position of the marker 60. The marker insertion unit 208 determines the color of the marker 60. The marker insertion unit 208 determines the size of the marker 60. The marker insertion unit 208 determines the timing of inserting the marker 60 into the video. The marker insertion unit 208 may determine the insertion period for inserting the marker 60 into the video.
[0091] The marker insertion unit 208 inserts the marker 60 into the video based on, for example, various information stored in the information storage unit 202. The marker insertion unit 208 inserts the marker 60 into the video based on, for example, content-related information stored in the information storage unit 202 in association with the video-related information.
[0092] The marker insertion unit 208 inserts the marker 60 into the video based on, for example, the correspondence information included in the content-related information. The marker insertion unit 208 inserts, for example, the marker 60 associated with the content by the correspondence information into the video. The marker insertion unit 208 may insert the marker 60 into the video so that the marker 60 includes the content-related information.
[0093] The marker insertion unit 208 inserts the marker 60 into the video based on, for example, the video-related information. The marker insertion unit 208 inserts the marker 60 into the video based on, for example, area-of-attention information included in the video-related information. The marker insertion unit 208 determines the insertion position of the marker 60 based on, for example, the area-of-attention information. The marker insertion unit 208 determines the insertion position of the marker 60 such that, for example, the marker 60 is inserted into an area outside the area of attention indicated by the area-of-attention information.
[0094] A user of the display device 100 tends to focus on the attention area when viewing an image displayed in the display target area 50. Therefore, the information processing device 200 can further reduce the possibility that the marker 60 will be recognized by the user of the display device 100 by inserting the marker 60 into the image so that the marker 60 is inserted into an area outside the attention area.
[0095] The marker insertion unit 208 inserts the marker 60 into the video based on, for example, color-related information included in the video-related information. The marker insertion unit 208 determines an insertion position of the marker 60 based on, for example, the color-related information. The marker insertion unit 208 determines an insertion position of the marker 60 such that, when the marker 60 is inserted, a color difference between the color of the marker 60 and a color in the video becomes larger. The marker insertion unit 208 may determine a color of the marker 60 based on, for example, the color-related information. The marker insertion unit 208 may determine a color of the marker 60 such that, when the marker 60 is inserted at the insertion position, a color difference between the color of the marker 60 and a color in the video becomes larger.
[0096] The greater the color difference between the color of the marker 60 and the color in the video, the easier it is for the display device 100 to recognize the marker 60 inserted in the video. Therefore, the information processing device 200 can make the display device 100 more reliably recognize the marker 60 by inserting the marker 60 into the video so that the color difference between the color of the marker 60 and the color in the video becomes greater.
[0097] The marker insertion unit 208 inserts the marker 60 into the video based on the user-related information of the user of the display device 100. The marker insertion unit 208 inserts the marker 60 into the video based on, for example, a plurality of pieces of user-related information of respective users of a plurality of display devices 100.
[0098] The marker insertion unit 208 determines an insertion period for inserting the marker 60 into the video based on, for example, the user-related information. The marker insertion unit 208 determines an insertion period for inserting the marker 60 into the video so that the marker 60 is inserted into the video for a period shorter than the period during which the user of the display device 100 of the age indicated by the age information included in the user-related information can recognize the marker 60. The marker insertion unit 208 determines an insertion period for inserting the marker 60 into the video so that the marker 60 is inserted into the video for a period shorter than the period during which the user of the display device 100 of the age indicated by the age information included in the user-related information can recognize the marker 60. The marker insertion unit 208 may determine an insertion period for inserting the marker 60 into the video so that the marker 60 is inserted into the video for a period shorter than the period during which the user of the display device 100 of the gender indicated by the gender information included in the user-related information can recognize the marker 60.
[0099] The marker insertion unit 208 may insert the marker 60 into the video based on main target information of the video into which the marker 60 is to be inserted. The marker insertion unit 208 determines an insertion period for inserting the marker 60 into the video, for example, based on the main target information of the video. The marker insertion unit 208 determines an insertion period for inserting the marker 60 into the video, for example, so that the marker 60 is inserted into the video for a period shorter than a period during which the main target of the video indicated by the main target information is recognizable.
[0100] The insertion period of the marker 60 during which the user of the display device 100 becomes able to recognize the marker 60 tends to differ depending on the age and gender of the user of the display device 100. Therefore, the marker insertion unit 208 can further reduce the possibility that the marker 60 will be recognized by the user of the display device 100 by inserting the marker 60 into the video in consideration of the age and gender of the user of the display device 100.
[0101] The marker insertion unit 208 inserts the marker 60 into the video, for example, while the video is being displayed in the display target area 50. That is, the marker insertion unit 208 inserts the marker 60 in real time while the video is being played back. The marker insertion unit 208 may also insert the marker 60 into the video in advance before the video is displayed in the display target area 50.
[0102] The marker insertion unit 208 inserts the marker 60 into the video based on, for example, various information received by the information receiving unit 204 while the video is displayed in the display target area 50. The marker insertion unit 208 inserts the marker 60 into the video based on, for example, user operation information of the user of the display device 100. The marker insertion unit 208 determines the timing of inserting the marker 60 into the video based on, for example, the user operation information. The marker insertion unit 208 inserts the marker 60 into the video when, for example, the operation of the user of the display device 100 indicated by the user operation information is a specific operation related to the video indicated by the specific operation information. In this case, the marker insertion unit 208 may insert the marker 60, which is associated with the content corresponding to the specific operation by the correspondence information, into the video. The marker insertion unit 208 may insert the marker 60 into the video when the user of the display device 100 moves a moving distance longer than a predetermined moving distance threshold.
[0103] The marker insertion unit 208 inserts the marker 60 into the video based on, for example, a captured image captured by the imaging unit 120 mounted on the display device 100. The marker insertion unit 208 inserts the marker 60 into the video by, for example, performing image analysis on the captured image.
[0104] The marker insertion unit 208 inserts the marker 60 into the video based on, for example, the video displayed in the display target area 50 and the captured image including other users who are not the user of the display device 100. The marker insertion unit 208 determines the timing of inserting the marker 60 into the video based on, for example, the captured image. For example, the marker insertion unit 208 detects the action of the other user from the captured image, and inserts the marker 60 into the video when the detected action of the other user is a specific action related to the video indicated by the specific action information. In this case, the marker insertion unit 208 may insert the marker 60 associated with the content corresponding to the specific action by the correspondence information into the video. The marker insertion unit 208 may insert the marker 60 into the video when the other user moves a moving distance longer than a predetermined moving distance threshold.
[0105] The marker insertion unit 208 determines, for example, based on the captured image, the insertion position of the marker 60. For example, the marker insertion unit 208 detects an overlapping area where the display target area 50 and another user overlap from the captured image, and determines the insertion position of the marker 60 so that the marker 60 is displayed in an area outside the overlapping area.
[0106] The marker insertion unit 208 may insert the marker 60 into the video based on a captured image including the video displayed in the display target area 50 and obstacles other than other users. The marker insertion unit 208 determines the insertion position of the marker 60 based on the captured image, for example. The marker insertion unit 208 detects an overlapping area where the display target area 50 and the obstacle overlap from the captured image, for example, and determines the insertion position of the marker 60 so that the marker 60 is displayed in an area outside the overlapping area.
[0107] The marker insertion unit 208 may insert the marker 60 into the video based on the recognition feasibility information. For example, when the recognition feasibility information indicates that the display device 100 cannot recognize the marker 60, the marker insertion unit 208 changes at least one of the insertion position of the marker 60, the color of the marker 60, and the size of the marker 60 so that the display device 100 can recognize the marker 60. When the information receiving unit 204 does not receive the recognition feasibility information from the display device 100 within a predetermined period from the time when the marker 60 is displayed in the display target area 50, the marker insertion unit 208 may change at least one of the insertion position of the marker 60, the color of the marker 60, and the size of the marker 60 so that the display device 100 can recognize the marker 60.
[0108] For example, when changing the insertion position of the marker 60, the marker insertion unit 208 changes the insertion position of the marker 60 so that the color difference between the color of the marker 60 at the changed insertion position of the marker 60 and the color in the video becomes larger. When changing the color of the marker 60, for example, the marker insertion unit 208 changes the color of the marker 60 so that the color difference between the color of the marker 60 after the change and the color in the video becomes larger. When changing the size of the marker 60, for example, the marker insertion unit 208 changes the size of the marker 60 so that the size of the marker 60 after the change is larger than the size of the marker 60 before the change.
[0109] The attention area identifying unit 210 identifies the attention area. For example, the attention area identifying unit 210 identifies the attention area based on the gaze of the user indicated by the gaze information of the user of the display device 100 stored in the information storage unit 202. For example, the attention area identifying unit 210 identifies the attention area based on the gaze of each of the multiple users indicated by the gaze information of each of the users of the multiple display devices 100. For example, the attention area identifying unit 210 identifies the attention area by identifying an area in the video to which the gaze of the user is more directed. The attention area identifying unit 210 may identify the attention area by receiving an input from the creator or director of the video by an input device included in the information processing device 200.
[0110] The attention area identifying unit 210 identifies the attention area, for example, while the video is being displayed in the display target area 50. That is, the attention area identifying unit 210 identifies the attention area in real time while the video is being played back. The attention area identifying unit 210 may identify the attention area in advance before the video is displayed in the display target area 50.
[0111] The attention area identifying unit 210 may store the identified attention area as attention area information in the information storage unit 202. The marker insertion unit 208 may insert a marker 60 into the video based on the attention area identified by the attention area identifying unit 210.
[0112] The video providing unit 212 provides the video into which the marker 60 has been inserted by the marker inserting unit 208. The video providing unit 212 provides the video to the communication device 300, for example.
[0113] The video providing unit 212 provides the video by, for example, transmitting the video via the network 20. The video providing unit 212 transmits the video by, for example, a streaming method. The video providing unit 212 may transmit the video by a download method. The video providing unit 212 may provide the video by recording the video on a recording medium such as a USB (Universal Serial Bus).
[0114] The content distribution unit 214 distributes the content stored in the information storage unit 202. The content distribution unit 214 distributes the content via the network 20, for example.
[0115] The content delivery unit 214 delivers the content to, for example, the display device 100. The content delivery unit 214 delivers the content to, for example, the communication terminal 150. The content delivery unit 214 may deliver the content to the communication device 300.
[0116] The content distribution unit 214 distributes the content based on, for example, the content-related information that is associated with the video-related information and stored in the information storage unit 202. The content distribution unit 214 distributes the content that is associated with the marker 60 by the correspondence information included in the content-related information.
[0117] The content distribution unit 214 distributes content, for example, while a video is being displayed in the display target area 50. The content distribution unit 214 distributes content in real time while a video is being played back. The content distribution unit 214 distributes content related to a video, for example, while the video providing unit 212 is providing the video. The content distribution unit 214 may distribute content in advance before the video is displayed in the display target area 50.
[0118] 12 shows an example of a schematic functional configuration of the display device 100. The display device 100 includes an information storage unit 104, an information receiving unit 106, an angle of view determination unit 107, a marker recognition unit 108, a display position determination unit 109, an imaging unit 120, a sensor 122, a gaze detection unit 124, a display unit 126, a request receiving unit 128, and an information transmitting unit 130. It is not essential that the display device 100 includes all of these components.
[0119] The information storage unit 104 stores various types of information. The information storage unit 104 stores, for example, user-related information of a user of the display device 100. The information storage unit 104 stores, for example, captured images captured by the imaging unit 120. The information storage unit 104 stores, for example, user operation information indicating an operation of the user of the display device 100 detected by the sensor 122. The information storage unit 104 may store gaze information indicating the gaze of the user of the display device 100 detected by the gaze detection unit 124.
[0120] The information receiving unit 106 receives various types of information. For example, the information receiving unit 106 receives the various types of information via the network 20. The information receiving unit 106 may store the received various types of information in the information storage unit 104.
[0121] The information receiving unit 106 receives various information from, for example, the information processing device 200. The information receiving unit 106 receives various information from, for example, the communication terminal 150. The information receiving unit 106 may receive various information from, for example, the communication device 300.
[0122] The information receiving unit 106 receives, for example, content related to the video displayed in the display target area 50. The information receiving unit 106 may also receive content-related information related to the content.
[0123] The angle of view determination unit 107 determines the angle of view of the display target area 50 included in the captured image captured by the imaging unit 120. The angle of view determination unit 107 determines the angle of view of the display target area 50 based on, for example, a captured image of the display target area 50 captured by the imaging unit 120 in a state in which the space including the display target area 50 is bright before the image is displayed in the display target area 50. When it is possible to specify in advance a viewing position from which the user of the display device 100 views the image displayed in the display target area 50, such as in a movie theater or a live concert venue, the angle of view determination unit 107 may determine the angle of view of the display target area 50 based on the viewing position of the user of the display device 100. The angle of view determination unit 107 may store the determined angle of view of the display target area 50 as angle of view information in the information storage unit 202.
[0124] The marker recognition unit 108 recognizes the marker 60 inserted in the video displayed in the display target area 50. For example, the marker recognition unit 108 recognizes the marker 60 inserted in the video from a captured image captured by the imaging unit 120 of the video displayed in the display target area 50 for a period shorter than a period recognizable by the user of the display device 100. The marker recognition unit 108 may recognize a plurality of markers 60 inserted in the video from the captured image. The marker recognition unit 108 may read content-related information from the recognized marker 60.
[0125] The marker recognition unit 108 recognizes, for example, the shape of the marker 60. The marker recognition unit 108 may also recognize the size of the marker 60.
[0126] The display position determination unit 109 determines the display position of the content stored in the information storage unit 104 to be displayed on the display unit 126. The display position determination unit 109 determines the display position of the content based on, for example, the marker 60 recognized by the marker recognition unit 108. The display position determination unit 109 determines the display position of the content, which is 3D content, based on, for example, a plurality of markers 60 recognized by the marker recognition unit 108. The display position determination unit 109 may determine the display positions of a plurality of contents based on a plurality of markers 60 recognized by the marker recognition unit 108.
[0127] The display position determination unit 109 determines the display position of the content based on, for example, the shape and size of the marker 60. For example, the display position determination unit 109 estimates the position of the display device 100 based on the shape and size of the marker 60. The display position determination unit 109 may determine the display position of the content so as to more closely match the estimated position of the display device 100. In this case, the marker 60 may be a feature point for estimating the position of the display device 100.
[0128] The display position determination unit 109 determines the display position of the content based on, for example, the positional relationship of the multiple markers 60. For example, the display position determination unit 109 estimates the position of the display device 100 based on the positional relationship of the multiple markers 60. The display position determination unit 109 may determine the display position of the content so as to more closely match the estimated position of the display device 100. In this case, the multiple markers 60 may be feature points for estimating the position of the display device 100. Note that the display position determination unit 109 may determine the positional relationship of the multiple markers 60 by identifying the center positions of the multiple markers 60.
[0129] The display position determination unit 109 determines the display position of the content further based on the content-related information. The display position determination unit 109 determines the display position of the content based on positional relationship information included in the content-related information. The display position determination unit 109 determines the display position of the content based on the positional relationship between the reference position, which is the center position of the marker 60 recognized by the marker recognition unit 108, and the display position of the content, which is indicated by the positional relationship information, for example.
[0130] The display position determination unit 109 determines the display position of the content based on, for example, the correspondence information included in the content-related information. For example, when the correspondence information indicates that one marker 60 is a marker for determining the display position of one piece of content, the display position determination unit 109 determines that the center position of the marker 60 is the display position of the content. For example, when the correspondence information indicates that multiple markers 60 recognized by the marker recognition unit 108 are markers for determining the display position of one piece of content that is 3D content, the display position determination unit 109 determines the display position of the content that is 3D content based on the positional relationship of the multiple markers 60. In this case, the multiple markers 60 are, for example, three or more markers 60.
[0131] The display position determination unit 109 determines the display position of the content, for example, further based on the angle of view information stored in the information storage unit 104. For example, the display position determination unit 109 estimates the position of the display device 100 based on the angle of view of the display target area 50 included in the captured image captured by the imaging unit 120, which is indicated by the angle of view information. The display position determination unit 109 may determine the display position of the content so as to more closely match the estimated position of the display device 100. In this case, the angle of view of the display target area 50 may be a feature point for estimating the position of the display device 100.
[0132] The display unit 126 displays the content stored in the information storage unit 104. The display unit 126 displays the content at a display position determined by the display position determination unit 109, for example.
[0133] The display unit 126 displays the content based on, for example, the content-related information stored in the information storage unit 104. The display unit 126 displays the content associated with the marker 60 recognized by the marker recognition unit 108 based on, for example, the correspondence information included in the content-related information.
[0134] The request receiving unit 128 receives requests to transmit various pieces of information stored in the information storage unit 104. The request receiving unit 128 receives the requests via the network 20, for example.
[0135] The request receiving unit 128 receives a request from, for example, the information processing device 200. The request receiving unit 128 may receive a request from the communication terminal 150.
[0136] The information transmitting unit 130 transmits the various pieces of information stored in the information storage unit 104. The information transmitting unit 130 transmits the various pieces of information via the network 20, for example.
[0137] For example, the information transmitting unit 130 transmits the various information in response to a request received by the request receiving unit 128. The information transmitting unit 130 may transmit the various information periodically.
[0138] The information transmitting unit 130 transmits, for example, various types of information to the information processing device 200. The information transmitting unit 130 may transmit the various types of information to the communication terminal 150.
[0139] 13 shows an example of a processing flow by the system 10. Here, a state in which no video is displayed in the display target area 50 is taken as a starting state.
[0140] In step (sometimes abbreviated to S) 102, the marker insertion unit 208 inserts the marker 60 into the video into which the marker 60 is to be inserted. In S104, the video providing unit 212 transmits to the communication device 300 the video into which the marker insertion unit 208 inserted the marker 60 in S102.
[0141] In S106, communications device 300 transmits the video received from information processing device 200 in S104 to video display device 400. In S108, video display device 400 starts displaying in display target region 50 the video received from communications device 300 in S106.
[0142] In S110, the content delivery unit 214 delivers content to the display device 100. In S112, the imaging unit 120 captures an image displayed in the display target area 50. In S114, the marker recognition unit 108 recognizes, from the captured image captured by the imaging unit 120 in S112, a marker 60 inserted in the image for a period shorter than a period recognizable by a user of the display device 100. In S116, the display position determination unit 109 determines a display position of the content to be displayed on the display unit 126 based on the marker 60 recognized by the marker recognition unit 108 in S114. In S118, the display unit 126 displays the content at the display position determined by the display position determination unit 109 in S116.
[0143] S110 to S118 may be repeatedly executed while the image display device 400 is displaying the image in the display target region 50. In S120, the image display device 400 ends the display of the image in the display target region 50, and the processing by the system 10 ends.
[0144] 14 is a schematic diagram showing an example of a hardware configuration of a computer 1200 functioning as the information processing device 200 or the display device 100. A program installed in the computer 1200 can cause the computer 1200 to function as one or more "parts" of the device according to the above embodiment, or cause the computer 1200 to execute operations or one or more "parts" associated with the device according to the above embodiment, and / or cause the computer 1200 to execute a process or steps of the process according to the above embodiment. Such a program can be executed by the CPU 1212 to cause the computer 1200 to execute specific operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.
[0145] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are connected to each other by a host controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive 1226, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive 1226 may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid state drive, or the like. The computer 1200 also includes legacy input / output units such as a ROM 1230 and a keyboard 1242, which are connected to the input / output controller 1220 via an input / output chip 1240.
[0146] The CPU 1212 operates according to a program stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphic controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into itself, and causes the image data to be displayed on the display device 1218.
[0147] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive 1226 reads programs or data from a DVD-ROM 1227 or the like and provides the programs or data to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.
[0148] The ROM 1230 stores therein a boot program or the like that is executed by the computer 1200 upon activation, and / or a program that depends on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, and the like.
[0149] The programs are provided by a computer-readable storage medium such as a DVD-ROM 1227 or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, the RAM 1214, or the ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by implementing operations or processing of information according to the use of the computer 1200.
[0150] For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 may execute a communication program loaded in the RAM 1214 and instruct the communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer area provided in the RAM 1214, the storage device 1224, the DVD-ROM 1227, or a recording medium such as an IC card, and transmits the read transmission data to a network, or writes reception data received from the network to a reception buffer area or the like provided on the recording medium.
[0151] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, the DVD drive 1226 (DVD-ROM 1227), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.
[0152] Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and undergo information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and write back the results to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.
[0153] The above-described programs or software modules may be stored in a computer-readable storage medium on the computer 1200 or in the vicinity of the computer 1200. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.
[0154] The blocks in the flowcharts and block diagrams in the present embodiment may represent stages of a process in which an operation is performed or "parts" of an apparatus responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as, for example, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like, including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.
[0155] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.
[0156] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages such as the “C” programming language or similar programming languages.
[0157] Computer readable instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, either locally or over a local area network (LAN), a wide area network (WAN), such as the Internet, etc., to cause the processor of the general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, to execute the computer readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.
[0158] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the description of the claims that such modifications and improvements can also be included in the technical scope of the present invention.
[0159] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and that the process may be performed in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, it does not mean that the process must be performed in this order. [Explanation of symbols]
[0160] 10 system, 20 network, 50 display target area, 60 marker, 62 marker, 64 marker, 66 marker, 70 content, 72 content, 74 content, 76 content, 80 captured image, 100 display device, 102 user, 104 information storage unit, 106 information receiving unit, 107 field of view determination unit, 108 marker recognition unit, 109 display position determination unit, 110 frame, 112 rim, 114 bridge, 116 lens, 118 temple, 120 imaging unit, 122 sensor, 124 gaze detection unit, 126 display unit, 128 request receiving unit, 130 information transmitting unit, 150 communication terminal, 200 information processing device, 202 information storage unit, 204 information receiving unit, 206 request transmitting unit, 208 Marker insertion unit, 210, attention area identification unit, 212, video providing unit, 214, content distribution unit, 300, communication device, 400, video display device, 502, user, 1200, computer, 1210, host controller, 1212, CPU, 1214, RAM, 1216, graphic controller, 1218, display device, 1220, input / output controller, 1222, communication interface, 1224, storage device, 1230, ROM, 1240, input / output chip, 1242, keyboard
Claims
1. a marker insertion unit that inserts a marker for determining a display position of content related to an image displayed on a display unit of a display device while an imaging unit mounted on the display device is imaging the image displayed in a display target area, into the image for a period shorter than a period recognizable by a user of the display device while the image is being displayed in the display target area by having a marker projection device project the marker; an information storage unit for storing attention area information indicating an attention area in the video that attracts attention; Equipped with The marker insertion unit determines an insertion position of the marker such that the marker is inserted in an area outside the area of interest indicated by the area of interest information. Information processing device.
2. a marker insertion unit that inserts a marker for determining a display position of content related to an image displayed on a display unit of a display device while an imaging unit mounted on the display device is imaging the image displayed in a display target area, into the image for a period shorter than a period recognizable by a user of the display device while the image is being displayed in the display target area by having a marker projection device project the marker; an information receiving unit that receives line-of-sight information indicating a line of sight of the user of the display device; an attention area specifying unit that specifies an attention area in the video where attention is focused based on the line of sight of the user indicated by the line of sight information; Equipped with the marker insertion unit determines an insertion position of the marker such that the marker is inserted in an area outside the area of interest specified by the area of interest specification unit. Information processing device.
3. a marker insertion unit that inserts a marker for determining a display position of content related to an image displayed on a display unit of a display device while an imaging unit mounted on the display device is imaging the image displayed in a display target area, into the image for a period shorter than a period recognizable by a user of the display device while the image is being displayed in the display target area by having a marker projection device project the marker; an information receiving unit that receives user operation information indicating an operation of the user of the display device; Equipped with The marker insertion unit inserts the marker into the video based on the user operation information of the user of the display device. Information processing device.
4. a marker insertion unit that inserts a marker for determining a display position of content related to an image displayed on a display unit of a display device while an imaging unit mounted on the display device is imaging the image displayed in a display target area, into the image for a period shorter than a period recognizable by a user of the display device while the image is being displayed in the display target area by having a marker projection device project the marker; an information receiving unit for receiving user-related information related to the user of the display device; Equipped with The marker insertion unit determines an insertion period for inserting the marker into the video based on the user-related information. Information processing device.
5. The information storage unit further stores color-related information related to colors in the image; The marker insertion unit determines an insertion position of the marker based on the color-related information such that a color difference between a color of the marker and the color in the image becomes larger when the marker is inserted. The information processing device according to claim 1 .
6. The information storage unit further stores color-related information related to colors in the image; the marker insertion unit determines a color of the marker based on the color-related information such that a color difference between the color of the marker and the color in the video becomes larger when the marker is inserted at an insertion position. The information processing device according to claim 1 .
7. a content delivery section that delivers the content to the display device; The information processing device according to claim 1 , further comprising:
8. A program for causing a computer to function as the information processing device according to any one of claims 1 to 7.
9. An information processing device according to any one of claims 1 to 7; The display device; A system comprising:
10. 1. A computer-implemented information processing method, comprising: a marker insertion step of inserting a marker for determining a display position of content related to an image displayed on a display unit of a display device while an imaging unit mounted on the display device is imaging the image displayed on the display target area, into the image for a period shorter than a period recognizable by a user of the display device while the image is being displayed on the display target area by projecting the marker from a marker projection device; Equipped with The marker insertion step includes a determination step of determining an insertion position of the marker so that the marker is inserted in an area outside an attention area in the image where attention is focused, the attention area being indicated by attention area information stored in the computer. Information processing methods.
11. 1. A computer-implemented information processing method, comprising: a marker insertion step of inserting a marker for determining a display position of content related to an image displayed on a display unit of a display device while an imaging unit mounted on the display device is imaging the image displayed in a display target area, into the image for a period shorter than a period recognizable by a user of the display device while the image is being displayed in the display target area by projecting the marker using a marker projection device; receiving gaze information indicating a gaze of the user of the display device; an attention area identifying step of identifying an attention area in the video where attention is focused based on the gaze of the user indicated by the gaze information; Equipped with The marker insertion step includes a determination step of determining an insertion position of the marker so that the marker is inserted in an area outside the area of interest identified in the area of interest identification step. Information processing methods.
12. 1. A computer-implemented information processing method, comprising: a marker insertion step of inserting a marker for determining a display position of content related to an image displayed on a display unit of a display device while an imaging unit mounted on the display device is imaging the image displayed in a display target area, into the image for a period shorter than a period recognizable by a user of the display device while the image is being displayed in the display target area by projecting the marker using a marker projection device; receiving user action information indicative of an action of the user of the display device; Equipped with The marker insertion step includes inserting the marker into the image based on the user operation information of the user of the display device. Information processing methods.
13. 1. A computer-implemented information processing method, comprising: a marker insertion step of inserting a marker for determining a display position of content related to an image displayed on a display unit of a display device while an imaging unit mounted on the display device is imaging the image displayed in a display target area, into the image for a period shorter than a period recognizable by a user of the display device while the image is being displayed in the display target area by projecting the marker using a marker projection device; receiving user-related information associated with the user of the display device; Equipped with The marker insertion step includes a determination step of determining an insertion period for inserting the marker into the image based on the user-related information. Information processing methods.
Citation Information
Patent Citations
Information input device
JP1998222287A
Superimposing apparatus, screening apparatus, screening method and program
JP2009076985A
Display device, information service system, information service method, and program
JP2012163853A
Projection system, projection device, projection method, and control program
JP2012233962A
Augmented reality device, augmented reality method, and augmented reality program
JP2015038696A
