Information processing device, information processing method, and program

Abstract

An information processing device according to one embodiment of the present invention is equipped with: a gaze point database; an acquisition unit; a prediction unit; and an annotation generation unit. The gaze point database holds time information and position information in which a gaze point appears in video content. The acquisition unit acquires viewpoint information, acquires the time information and the position information from the gaze point database, and acquires a video content playback time. The prediction unit predicts the appearance of the gaze point in the rendering range of the video content, on the basis of orientation information, the time information, the position information, and the playback time. The annotation generation unit generates an annotation if the appearance of the gaze point is predicted, and superimposes the annotation on the video content in association with the gaze point.

Description

Information processing device, information processing method, and program

[0001] One aspect of the present invention relates to an information processing device, an information processing method, and a program used to construct a virtual space known as XR (Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), etc.).

[0002] Head-mounted displays (HMDs), also known as VR goggles, have become commonplace. Also, 360-degree cameras, or spherical cameras, that can capture images from any viewing direction, are now available for purchase by individuals. As a result, a large amount of spherical video content, known as virtual reality content, has been produced in recent years. Users can wear an HMD and freely move their viewpoint to enjoy the content from any angle. This type of technology is not only used to create virtual worlds, such as scenes from quest games, but is also increasingly being used in business situations, such as real estate previews.

[0003] However, the viewpoint of the content displayed on the HMD follows the direction in front of the user's head. This means that the user's viewpoint moves freely, making it difficult for the user to enjoy the content as intended by the creator, and users may miss highlights or lose track of the content they want to see.

[0004] Trepkowski, Christina, et al. "Multisensory proximity and transition cues for improving target awareness in narrow field of view augmented reality displays." IEEE transactions on visualization and computer graphics 28.2 (2021): 1342-1362.

[0005] In content such as spherical video, a specific gaze point is often preset for the user. However, with existing technology, it has been difficult to reliably direct the user's gaze to the gaze point. While there are known technologies that provide separate notifications such as audio, there is a demand for technology that can draw attention in a more seamless manner.

[0006] The present invention has been made in light of the above-mentioned circumstances, and has an object to provide a technique that can prevent a user from missing a point of interest in video content.

[0007] An information processing device according to one aspect of the present invention includes a gaze point database, an acquisition unit, a prediction unit, and an annotation generation unit. The gaze point database holds time information and position information for the appearance of gaze points in video content. The acquisition unit acquires viewpoint information, time information and position information from the gaze point database, and acquires the playback time of the video content. The prediction unit predicts the appearance of gaze points in the drawing range of the video content based on the posture information, time information, position information, and playback time. The annotation generation unit generates an annotation when the appearance of a gaze point is predicted, and superimposes the annotation on the video content in association with the gaze point.

[0008] According to one aspect of the present invention, it is possible to provide a technique that can prevent a user from missing a point of interest in video content.

[0009] FIG. 1 is a diagram illustrating an example of a system to which an information processing device according to an embodiment is applied. FIG. 2 is a functional block diagram illustrating an example of the head-mounted display 3 illustrated in FIG. 1. FIG. 3 is a flowchart illustrating an example of a processing procedure of the information processing device 1 according to an embodiment. FIG. 4 is a flowchart illustrating an example of a processing procedure of the annotation generation unit 70c. FIG. 5 is a flowchart illustrating an example of a processing procedure of the time information representation unit 70d. FIG. 6 is a diagram illustrating an example of an annotation including time information. FIG. 7 is a diagram illustrating another example of an annotation including time information. FIG. 8 is a diagram illustrating another example of an annotation including time information. FIG. 9 is a diagram illustrating another example of an annotation including time information. FIG. 10 is a flowchart illustrating an example of a processing procedure of the distance information representation unit 70e. FIG. 11 is a diagram illustrating an example of an annotation including distance information. FIG. 12 is a diagram illustrating another example of an annotation including distance information. FIG. 13 is a diagram illustrating another example of an annotation including distance information. FIG. 14 is a diagram illustrating another example of an annotation including distance information. FIG. 15 is a diagram illustrating an example of video content including a gaze point. Fig. 16 is a diagram showing that the user's gaze point is guided by annotations. Fig. 17 is a diagram showing an example of video content that does not include annotations. Fig. 18 is a diagram for explaining that the user's gaze cannot be guided. Fig. 19 is a diagram for explaining a case where the gaze point is outside the field of view. Fig. 20 is a diagram for explaining a case where the gaze point is outside the field of view. Fig. 21 is a diagram for explaining a case where the gaze point is inside the field of view.

[0010] 1 is a diagram showing an example of a system to which an information processing device according to an embodiment is applied. The system includes an information processing device 1 connected to a network 4 and a head-mounted display 3 that can access the information processing device 1 via the network 4.

[0011] The head-mounted display 3 includes a posture sensor for detecting a three-dimensional posture. The posture sensor is, for example, an angular velocity sensor (gyro sensor), and acquires posture information (gyro information) of the head-mounted display 3. The posture information is, for example, a vector indicating the forward direction of the head-mounted display 3, i.e., the user's line of sight. The head-mounted display 3 may also include a camera for detecting viewpoint information indicating the user's viewpoint direction and blinks. The viewpoint information and blinks can be detected, for example, by capturing an image of the user's eyes with a camera and analyzing the image data. The posture information and viewpoint information can be treated as independent information.

[0012] 1, an information processing device 1 as a computer includes a processor 70 and a memory 60. In addition, the information processing device 1 includes a storage 50 and a communication unit 80 that functions as a communication interface with a network 4.

[0013] The communication unit 80 waits for access from the head mounted display 3 and transmits the requested video content to the accessing head mounted display 3. In the embodiment, a spherical video is taken as an example of the video content. Of course, the video content is not limited to a spherical video.

[0014] Storage 50 stores video content 52 and a gaze point database 53 in addition to a program 51 for causing a computer to function as information processing device 1 according to the embodiment. Gaze point database 53 holds appearance time information and appearance position information for each gaze point preset in video content 52. Appearance time information for each gaze point is registered as time-series information in time information table 53a. Appearance position information for each gaze point is registered as position information relative to the entire celestial sphere in position information table 53b.

[0015] The program 51 is loaded into the memory 60 by the OS (Operating System) of the information processing device 1 and executed by the processor 70. The program 51 causes the processor 70 to function as an acquisition unit 70a, a prediction unit 70b, an annotation generation unit 70c, a time information representation unit 70d, and a distance information representation unit 70e.

[0016] The acquisition unit 70a acquires posture information and user viewpoint information from the head-mounted display 3. The acquisition unit 70a also measures and acquires the playback time of the video content 52 that is streamed, and acquires appearance time information and appearance position information of gaze points that appear over time from the gaze point database 53.

[0017] The prediction unit 70b calculates, based on the acquired posture information, the rendering range of the video content in the field of view of the user wearing the head-mounted display 3. Then, the prediction unit 70b predicts the position and time (appearance time) at which the gaze point will appear in this rendering range, based on the acquired time information, position information, and playback time.

[0018] The annotation generation unit 70c generates an annotation when the prediction unit 70b predicts that a gaze point will appear, for example, t seconds into the future. The generated annotation is displayed at the position where the predicted gaze point will appear or in the vicinity thereof. That is, the annotation is superimposed on the video content in association with the gaze point and displayed so as to be captured by the user's line of sight.

[0019] The annotation is displayed in different ways depending on "how far in the future the gaze point will appear" and "how far the gaze point is from the user's line of sight." That is, the time information representation unit 70d changes the shape of the annotation depending on the time it takes for the gaze point to appear. Furthermore, the distance information representation unit 70e changes the display color of the annotation based on the viewpoint information acquired from the acquisition unit 70a and the appearance position of the gaze point.

[0020] Fig. 2 is a functional block diagram showing an example of the head-mounted display 3 shown in Fig. 1. The head-mounted display 3 is, for example, in the form of goggles and is detachably attached to the user's head. The HMD may be a goggle-type or a glasses-type HMD, or may be a combination of goggles or glasses and a mobile terminal such as a smartphone.

[0021] The head mounted display 3 includes a control unit 5 having a hardware processor such as a CPU. A program storage unit 6, a data storage unit 7, a communication interface (communication I / F) 8, and an input / output interface (input / output I / F) 9 are connected to the control unit 5 via a bus.

[0022] The program storage unit 6 is a non-volatile memory such as an SSD (Solid State Disk), and stores a program for controlling the head mounted display 3. The data storage unit 7 is, for example, a combination of an SSD and a RAM, and stores data for controlling the head mounted display 3. The communication I / F 8 communicates with the information processing device 1 via the network 4.

[0023] An attitude sensor 91 that outputs attitude information, a camera 92 that outputs viewpoint information, a display 93, and an input unit 94 are connected to the control unit 5 via the input / output I / F 9. The display 93 is, for example, a panel display that gently curves to fit the shape of the goggles, and displays video content in the user's field of view. The input unit 94 includes operation buttons, switches, etc., and is used by the user to operate the head-mounted display 3.

[0024] 3 is a flowchart showing an example of a processing procedure of the information processing device 1 according to the embodiment. In FIG. 3, when viewing of content is started by a user operation (step S1), the information processing device 1 acquires the playback time of the viewing content (spherical video) and acquires the time-series gaze point times from the time information table 53a (step S2). Then, it determines whether or not a gaze point exists t seconds after the current playback time (step S3). If a gaze point exists (Yes), an annotation generation process is performed (step S4). If no gaze point exists between the present and t seconds into the future (No in step S3), no annotation is presented.

[0025] 4 is a flowchart showing an example of a processing procedure of the annotation generation unit 70c. In FIG. 4, when viewing of content is started by a user operation (step S11), the annotation generation unit 70c measures the user's head direction and the viewpoint position relative to the celestial sphere from gyro information of the head mounted display 3, and acquires viewpoint information (step S12). In addition, the annotation generation unit 70c acquires the screen drawing range displayed on the display 93 from viewing angle information of the display 93 of the head mounted display 3.

[0026] Next, the annotation generation unit 70c acquires the position information of the gaze point from the position information table 53b and determines whether the gaze point exists within the screen drawing range (step S13). If the gaze point exists within the drawing range (Yes), the annotation generation unit 70c generates an annotation with time information (step S14) and superimposes it near the gaze point (step S15). Here, the annotation generation unit 70c changes the display color of the annotation based on the user's viewpoint information and the appearance position of the gaze point. For example, the annotation is displayed in the color with the longest wavelength (close to red).

[0027] On the other hand, if the gaze point does not exist within the field of view (step S13), the annotation generation unit 70c measures a vector along the celestial sphere from the viewpoint position with respect to the celestial sphere toward the gaze point (step S17), and according to the result, the annotation generation unit 70c provides an expression of time information to the annotation (step S18) and also an expression of distance information (step S19). The expression of time information and the expression of distance information will be described in detail immediately below.

[0028] The annotation generation unit 70c then superimposes the annotations representing time information and distance information onto the vector of the viewpoint and gaze point within the field of view (step S20). The superimposed position is preferably within the vector. However, the closer to the center the annotation is, the higher the probability that it will obstruct the field of view, but the higher the probability that the user will notice it. Conversely, the farther from the center the annotation is, the lower the probability that the annotation will be noticed and the lower the probability that it will obstruct the field of view. In other words, it is known that there is a trade-off between these two (Reference [1]).

[0029] When an annotation for one gaze point is superimposed, the processing procedure proceeds to step S16, and the processing from step S13 is repeated until annotations for all gaze points are superimposed (No in step S16).

[0030] <Regarding Representation of Time Information> Fig. 5 is a flowchart showing an example of the processing procedure of the time information representation unit 70d. In Fig. 5, when a user starts viewing content (step S31), the time information representation unit 70d acquires the playback time of the viewed content and the chronological gaze point time from the time information table 53a (step S32). The time information representation unit 70d then compares the appearance time stored in the gaze point with the current playback time (step S33). If the difference between the two is within t seconds (step S34), the time information representation unit 70d calculates the ratio of the obtained number to t (step S35) and generates an annotation containing information about the obtained ratio (step S36).

[0031] 6 to 9 are diagrams showing examples of annotations that include time information. Figures 6(a) and 6(b) are both rectangular (vertical bar) annotations, with Figure 6(a) showing the case where the ratio calculated in step S35 is 0.4, and Figure 6(b) showing the case where the ratio is 0.8. In other words, the proportion of colored portions (hatching) increases as the time until the gaze point appears shortens. This means that Figure 6(b) shows a shorter time until the gaze point appears than Figure 6(a).

[0032] 7(a) and 7(b) show circular annotations. Fig. 7(a) shows the case where the ratio is 0.4, and Fig. 7(b) shows the case where the ratio is 0.8. In this way, the proportion of colored areas (hatching) may be reduced in inverse proportion to the shortening of the time until the fixation point appears.

[0033] Fig. 8(a) and Fig. 8(b) are annotations that mimic an hourglass. Fig. 8(a) shows the case where the ratio is 0.4, and Fig. 8(b) shows the case where the ratio is 0.8. The sand is mimicked as time passes.

[0034] 9(a) and 9(b) show the passage of time by changing the size of the circular annotation. Fig. 9(a) shows the case where the ratio is 0.4, and Fig. 9(b) shows the case where the ratio is 0.8.

[0035] As described above, the ratio between the appearance time of the fixation point and the current playback time can be expressed by the proportion of the colored part of the annotation, the size of the annotation itself, etc. In addition, any expression that changes seamlessly according to the proportion of time, such as a progress bar, can be used.

[0036] 5, the annotation disappears when the appearance time stored in the time information table 53a is reached. However, the annotation may continue to be displayed. In such a case, a process for maintaining the ratio value at 1 for s seconds may be added.

[0037] <Regarding Representation of Distance Information> Fig. 10 is a flowchart showing an example of the processing procedure of the distance information representation unit 70e. In Fig. 10, when the user starts viewing content (step S41), the distance information representation unit 70e measures the user's viewpoint from gyro information of the HMD and acquires viewpoint information (step S42). Here, the viewpoint is represented by a vector (λ ｘ , λ ｙ , λ ｚ ) is given by

[0038] Next, the distance information representation unit 70e generates (in memory) an arc that is tangent to the gaze point from the intersection point with the sphere when a vector is extended from the center of the celestial sphere (step S43), and determines the intersection point of the arc with a figure that indicates the drawing range when this arc is drawn on the celestial sphere.Furthermore, the distance information representation unit 70e determines the angle of the arc that is drawn from the intersection point to the gaze point (step S44).

[0039] Generally, the field of view of a head-mounted display is divided by a screen centered on the viewpoint. Therefore, if the viewpoint is unique, the drawing range becomes a unique figure drawn on the entire celestial sphere. Therefore, in this embodiment, the wavelength of the color of the annotation is determined to be proportional to the angle calculated in step S44 (step S45).

[0040] Generally, humans perceive colors with short wavelengths as being close and colors with long wavelengths as being far away (Reference [2]). Therefore, in this embodiment, colors are determined so that the wavelength increases as the angle increases and the wavelength decreases as the angle decreases. There are no particular limitations on the specific colors that will be obtained as long as they are proportional within the range of visible light.

[0041] It is also known that saturation is proportional to the distance (reference [2]). Therefore, it is possible to adjust the saturation within the same color wavelength. Specifically, the larger the angle, the lower the saturation, and the smaller the angle, the higher the saturation to represent the distance.

[0042] 11 to 14 are diagrams showing examples of annotations including distance information. Both Fig. 11(a) and Fig. 11(b) show bar-shaped annotations, indicating that it takes the same amount of time for the gaze point to appear. Here, Fig. 11(a) shows the case where the angle calculated in step S44 is large, while Fig. 11(b) shows the case where the angle is smaller than that in Fig. 11(b). That is, the color wavelength is changed according to the distance to the gaze point (the two are distinguished by hatching). In other words, the user is looking at a position closer to the gaze point in Fig. 11(b) than in Fig. 11(a). Similarly, Fig. 12(a) and Fig. 12(b) show circular annotations, with the angle smaller in Fig. 11(b) than in Fig. 11(a). In this way, by changing the color wavelength according to the angle, the user's gaze can be guided to the gaze point.

[0043] 12 and 13 show how color saturation can be changed depending on the angle. (a) in Fig. 12 and Fig. 13 shows the case where the angle is large, and (b) shows the case where the angle is small. In this way, color saturation can be changed depending on the angle (different saturations are indicated by hatching).

[0044] FIG. 15 is a diagram showing an example of video content including a gaze point. The area within the rectangular frame corresponds to the field of view of the display 93 of the head-mounted display 3, and reference numeral 100 indicates the gaze point. When the user notices an annotation displayed on the left side of their field of view, they make a guess such as, "Should I look left? It looks a little far away, so maybe it's near the door?" and turn left (the field of view moves to the left). Then, as shown in FIG. 16, a person appears from the door, giving the impression, "Oh, someone's coming out!" In this way, the user's gaze can be guided as intended by the content creator.

[0045] Incidentally, if annotations are not displayed, it will not be possible to guide the user's gaze as shown in Figures 17 and 18. Therefore, even if a person comes out of the door (Figure 18), the user may get the impression that "I was just casually looking at the shelves, but nothing happened. This is a boring video..."

[0046] 19 and 20 are diagrams for explaining a case where the gaze point is outside the field of view. In Fig. 19 and Fig. 20, the small circle inside the celestial sphere indicates the field of view of the head-mounted display 3, and the center indicates the user's viewpoint. The line from the user's viewpoint to the gaze point 100 indicates an auxiliary line for drawing a vector (the drawing of the auxiliary line is optional). Annotation 200 is drawn at the intersection of this vector and the field of view.

[0047] 21 is a diagram for explaining a case where the gaze point is inside the field of view. In FIG. 21, an annotation 200 is drawn inside the field of view of the head mounted display 3.

[0048] As described above, in the embodiment, an annotation that notifies the appearance of a gaze point is presented in an image (viewpoint image) in the viewpoint direction of a user wearing the head-mounted display 3. The annotation is displayed, for example, near the position where the gaze point will appear. In other words, if a gaze point including the future is outside the user's field of view, the annotation is displayed in the direction of the gaze point. The annotation has a different color depending on the distance to the gaze point. Also, the annotation is displayed in a different shape depending on the time until the gaze point appears. Furthermore, for annotations outside the user's field of view, the color change is used to represent pseudo-seamless distance information.

[0049] In this way, by notifying the user of the appearance of a gaze point by an annotation, it is possible to prevent the user from missing the gaze point in video content. Furthermore, the change in color can make the user aware of the distance between the gaze point and the user's viewpoint, and the change in shape can make the user aware of the time until the gaze point appears. In other words, by providing the annotation presented when viewing spherical video with time series information and distance information, it is possible to present both pieces of information to the user simultaneously.

[0050] Therefore, according to the embodiment, it is possible to solve problems that users may have when watching spherical video, such as "I don't know what's going to happen where" or "I don't know where to point my gaze." Furthermore, content providers can encourage consumers to have a consumer experience that follows the flow intended by the creator.

[0051] The annotations indicate the distance to the gaze point by color, and the time until the gaze point appears by shape. For annotations outside the field of view, the color changes to represent distance information in a pseudo-seamless manner.

[0052] In existing technologies, when a gaze point comes into view, a notification is given by voice or the like. In contrast, according to the embodiment, it is possible to seamlessly express the approach of the gaze point by changing the visual user interface.

[0053] Furthermore, according to the embodiment, for 360° content in which a gaze point is specified in advance, the user can avoid missing the gaze point and can track a specific person or object. This improves the user experience and allows the content to be experienced as intended by the creator. Furthermore, by adding two pieces of information, time and distance, to the annotation, the user's viewpoint can be moved efficiently.

[0054] Furthermore, according to the embodiment, a user can intuitively grasp the position of a gaze point, the time when the gaze point occurs, and the distance to the gaze point. Therefore, a user can efficiently view spherical video by directing the viewpoint in the direction in advance or minimizing head movement for searching. Furthermore, by referring to a different gaze point DB for each user or situation, it is possible to meet individual needs, such as wanting to follow a specific person.

[0055] Furthermore, by presenting not only the current gaze point but also future gaze points, the user can proactively move their viewpoint to the gaze point. Furthermore, by expressing distance in a pseudo-planar manner using color, a human sensory characteristic, rather than representing distance using relative distance on a pseudo-plane like radar, the distance to the gaze point beyond the annotation can be intuitively grasped. Therefore, according to the embodiment, a technology can be provided that can prevent users from missing gaze points in video content.

[0056] It should be noted that the present invention is not limited to the above-described embodiment. In the embodiment, an example in which video content is displayed on the display 93 of the head-mounted display 3 has been described. Alternatively, it goes without saying that the annotation generation technology described above can be applied to a configuration in which a spherical video is displayed on a display device such as a normal display or a personal computer monitor. For example, in recent years, a wide variety of spherical videos have been made available on video sites, and users can directly input viewpoint information by clicking up, down, left, and right arrow icons. The viewpoint information acquired from this type of user interface may be provided to the acquisition unit 70a, and the appearance of a gaze point set in the video content may be predicted.

[0057] Furthermore, in the implementation stage, the components of this invention can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0058] <References> [1] Naohiro Kishishita. "Investigation of the influence of peripheral vision in a wide-field see-through HMD for outdoor AR." Proceedings of the Virtual Reality Society of Japan annual conference. Vol. 18. [Virtual Reality Society of Japan], 2013. [2] Tadashi Oyama. "Measurement of the advancement and recession phenomenon of colored surfaces." Journal of the Illuminating Engineering Institute of Japan 42.12 (1958): 526-532.

[0059] 1...Information processing device 3...Head-mounted display 4...Network 5...Control unit 6...Program memory unit 7...Data memory unit 8...Communication interface 9...Input / output interface 50...Storage 51...Program 52...Video content 53...Gaze point database 53a...Time information table 53b...Position information table 60...Memory 70...Processor 70a...Acquisition unit 70b...Prediction unit 70c...Annotation generation unit 70d...Time information representation unit 70e...Distance information representation unit 80...Communication unit 91...Orientation sensor 92...Camera 93...Display 94...Input unit 100...Gaze point 200...Annotation.

Claims

1. An information processing device comprising: a gaze point database that holds time information and position information of the appearance of gaze points in video content; an acquisition unit that acquires viewpoint information, acquires the time information and the position information from the gaze point database, and acquires the playback time of the video content; a prediction unit that predicts the appearance of the gaze point in the drawing range of the video content based on the viewpoint information, the time information, the position information, and the playback time; and an annotation generation unit that generates an annotation when the appearance of the gaze point is predicted, and superimposes the annotation on the video content in association with the gaze point.

2. An information processing device as described in claim 1, further comprising: a display unit for displaying the video content; and a communication unit for communicating with a head-mounted display having a gaze sensor for acquiring user viewpoint information, wherein the acquisition unit acquires the user viewpoint information from the head-mounted display; a time information representation unit for changing the shape of the annotation according to the time it takes for the gaze point to appear; and a distance information representation unit for changing the display color of the annotation based on the acquired viewpoint information and the appearance position of the gaze point.

3. An information processing method executed by the processor of an information processing device having a processor and a storage, comprising: a step of the processor acquiring viewpoint information; a step of the processor acquiring time information and position information from a gaze point database that holds time information and position information of the appearance of gaze points in video content; a step of the processor acquiring the playback time of the video content; a step of the processor predicting the appearance of the gaze point in the drawing range of the video content based on the viewpoint information, the time information, the position information and the playback time; and a step of the processor generating an annotation when the appearance of the gaze point is predicted, and superimposing the annotation on the video content in association with the gaze point.

4. A program that causes a computer to function as the information processing device according to claim 1 or 2.

Images