Information processing device, system for information processing device, method for controlling information processing device, and program

Abstract

An information processing apparatus includes: a display control unit configured to display a video on a display section; a line-of-sight acquisition unit configured to acquire line-of-sight information of a user; a recording unit configured to record the line-of-sight information when the display of the video is stopped; and a playback control unit configured to stop the video when the display of the video is stopped, and to restore the video based on the line-of-sight information recorded in the recording unit when the display of the video is restored.

Description

Technical Field

[0001] This invention relates to an information processing device for detecting gaze. Background Technology

[0002] In recent years, virtual reality (VR) technology, which uses head-mounted displays (HMDs) to allow users to experience spaces different from real space, has become known. HMDs allow users to view VR videos with freely changeable viewing orientations, and during viewing, a portion of the video is cropped based on the viewing orientation and displayed on the HMD's display. Many HMDs are equipped with a function that allows users to confirm the scene in real-time through live framing (pass-through) while wearing the HMD. One known technology involves automatically pausing the video if the user uses this pass-through function, and automatically resuming playback from the point of interruption after the pass-through function ends, adjusting the viewpoint information based on the HMD's movement and the viewpoint from the HMD's position. PTL 1 can be cited as prior art related to this type of VR video.

[0003] Reference List

[0004] Patent documents

[0005] PTL 1: Japanese Patent Application Publication No. 2021-144599 Summary of the Invention

[0006] One aspect of the present invention is an information processing apparatus. The information processing apparatus includes: a display control unit configured to display video on a display unit; a gaze acquisition unit configured to acquire gaze information of a user; a recording unit configured to record the gaze information when the display of the video stops; and a playback control unit configured to stop the video when the display of the video stops, and to restore the video based on the gaze information recorded in the recording unit when the display of the video resumes.

[0007] According to the present invention, an information processing apparatus can be provided that can prevent video viewing from being missed due to changes in viewing angle during the period from when the video display stops to when the video display resumes. Attached Figure Description

[0008] [ Figure 1A ] Figure 1A This is an example of the appearance of a smartphone 100, which is an information processing device of a type according to the first embodiment of the present invention.

[0009] [ Figure 1B ] Figure 1B This is an example of the appearance of another side of the smartphone 100 in the first embodiment of the present invention.

[0010] [ Figure 1C ] Figure 1C This is an example of the configuration of the smartphone 100 in the first embodiment of the present invention.

[0011] [ Figure 1D ] Figure 1D This is an external view of the VR glasses (headset adapter) 200 that can be equipped with a smartphone 100 according to the first embodiment of the present invention.

[0012] [ Figure 2A ] Figure 2A This is an external view of the handheld controller of the display device in the first embodiment of the present invention.

[0013] [ Figure 2B ] Figure 2B This is an external view and wearing example of the ring controller of the display device in the first embodiment of the present invention.

[0014] [ Figure 3 ] Figure 3 This is a flowchart illustrating the video viewing mode processing of the display device in the first embodiment of the present invention.

[0015] [ Figure 4A ] Figure 4A This is an example of a VR video being played in the video viewing mode of the display device in the first embodiment of the present invention.

[0016] [ Figure 4B ] Figure 4B This is an example of a screen after a pause operation has been performed in the video viewing mode of the display device in the first embodiment of the present invention.

[0017] [ Figure 4C ] Figure 4C This is an example of a screen after playback operation has been performed in the video viewing mode of the display device in the first embodiment of the present invention.

[0018] [ Figure 4D ] Figure 4D This is an example of the screen after switching to perspective display in the video viewing mode of the display device in the first embodiment of the present invention.

[0019] [ Figure 4E ] Figure 4E In the video viewing mode of the display device in the first embodiment of the present invention, from Figure 4D An example of a viewpoint viewed from directly behind (180° opposite direction).

[0020] [ Figure 4F] Figure 4F This is an example of the screen after the perspective view ends in the video viewing mode of the display device in the first embodiment of the present invention.

[0021] [ Figure 4G ] Figure 4G In the video viewing mode of the display device in the first embodiment of the present invention, from Figure 4E Example of a view rotated 90° to the left.

[0022] [ Figure 4H ] Figure 4H This illustrates the video viewing mode of the display device in the first embodiment of the present invention. Figure 4G Further change the viewing direction and orient it towards... Figures 4A to 4C Examples of images with the same orientation.

[0023] [ Figure 4I ] Figure 4I This is an example of a screen showing the restoration of a video state in the video viewing mode of the display device in the first embodiment of the present invention.

[0024] [ Figure 4J ] Figure 4J This is an example of a screen display provided in the video viewing mode of the display device in the first embodiment of the present invention.

[0025] [ Figure 5A ] Figure 5A This is an example of a screen when a predetermined object 501 is stored in the video viewing mode of the display device in the first embodiment of the present invention.

[0026] [ Figure 5B ] Figure 5B This is an example of a screen showing an arrow 502 indicating the location of a stored coordinate or area in the video viewing mode of the display device in the first embodiment of the present invention.

[0027] [ Figure 5C ] Figure 5C This is an example of a screen showing the mark 503 on the viewing direction indicator 402 in the video viewing mode of the display device in the first embodiment of the present invention.

[0028] [ Figure 5D ] Figure 5D This is an example of adding a direction indicator 504 to the screen in the video viewing mode of the display device in the first embodiment of the present invention.

[0029] [ Figure 5E ] Figure 5E This is an example of changing the display in the video viewing mode of the display device in the first embodiment of the present invention to show the relative position of the screen. Detailed Implementation

[0030] In the aforementioned prior art, when the HMD's viewpoint position changes during perspective viewing, the area being viewed before the perspective begins may differ from the area of ​​the automatically playing video at the end of the perspective, thus potentially causing the user to miss the area they want to view. Therefore, this invention aims to provide an information processing device that can prevent the user from missing the VR video before and after the perspective view while maintaining immersion. An example of the described perspective function is that when VR video viewing is interrupted, other functions are used, and then VR video viewing is resumed, the viewing direction is likely to change. However, this is not limited to this situation; even when using other functions besides VR video, if the viewing direction changes before and after the VR video viewing is interrupted, such a problem may occur.

[0031] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0032] The embodiments described below are examples of means for implementing the present invention, and can be modified or changed as needed depending on the configuration of the apparatus to which the present invention is applied and various conditions. These embodiments can be combined as needed. In the accompanying drawings, the same reference numerals denote the same or similar components, and repeated descriptions will be omitted.

[0033] (First Embodiment)

[0034] Figure 1A An example of the appearance of a smartphone 100, which is a type of information processing device, is shown.

[0035] Here, we will describe a smartphone as an example of an information processing device, and VR glasses (headset adapters) that allow viewing VR videos (VR content) when the smartphone is mounted. The information processing device can be a standalone HMD (standalone HMD), or the HMD can be connected to the information processing device to share processing. The smartphone and VR glasses can also have an integrated housing. The smartphone can be used without being mounted on the VR glasses (headset adapter). The information processing device can be a tablet or a PC.

[0036] Display 101 is a display unit that displays images and various information. Display 101 is integrated with touch panel 102a, which will be described later, and is capable of detecting touch operations on the display surface of display 101. Smartphone 100 is capable of displaying VR video (VR content) in a VR manner on display 101. As shown, operation unit 102 includes touch panel 102a, power button 102b, volume buttons 102c and 102d, and home button 102e. Operation unit 102 is an input device for accepting user operations and includes character input devices such as a keyboard, pointing devices such as a mouse and touch panel, buttons, dials, joysticks, touch sensors, and touchpads. Touch panel 102a is arranged in a planar shape and laid on display 101, and is an input device configured to output coordinate information corresponding to the touch position. Operation unit 102b is a power button that accepts operations to switch the power of smartphone 100 between an on and off state. Volume buttons 102c and 102d are operation units for increasing or decreasing the volume of sound output from the audio output unit 103. The home button 102e is an operation unit for displaying the home screen on the display 101. The audio output unit 103 includes an audio output terminal 103a and a speaker 103b. The audio output terminal 103a is a headphone jack, which is a terminal (headphone plug or microphone plug) for inserting headphones, external speakers, etc., to output sound. The speaker 103b is a built-in speaker that emits sound. The audio output unit 103 may include only one of these two components. The headphone jack of the audio output terminal 103a can be of any size, and can be any of two-pole, three-pole, or four-pole types, and may also use a lightning bolt connector. The gaze sensor unit 104 is a sensor for detecting the user's gaze on the smartphone 100.

[0037] Figure 1B An example view showing the appearance of another side of the smartphone 100 is provided. The image capturing unit 105 is a camera capable of capturing images. The image capturing unit 105 includes, for example, an optical system and an image capturing element. The optical system controls an optical lens unit, aperture, zoom, and focus, and the image capturing element converts light (image) introduced through the optical lens unit into an electrical image signal. Complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) devices are commonly used as image capturing elements. Under the control of the control unit 107, the image capturing unit 105 converts the light from the subject focused by the lens included in the image capturing unit 105 into an electrical signal, performs noise reduction processing, etc., and outputs digital data as image data.

[0038] Figure 1CAn example configuration of smartphone 100 is shown. Smartphone 100 can be configured using a display device such as a smartphone. Control unit 107, working memory 108, non-volatile memory 109, image processing unit 110, display 101, operation unit 102, storage medium I / F 111, external I / F 112, and communication I / F 113 are connected to internal bus 106. Audio output unit 103, gaze sensor unit 104, attitude detection unit 114, and self-position estimation unit 115 are also connected to internal bus 106. Components connected to internal bus 106 are designed to exchange data with each other via internal bus 106.

[0039] The control unit 107 is the CPU that controls the entire smartphone 100 and consists of at least one processor or circuit.

[0040] The working memory 108 is, for example, composed of RAM (such as volatile memory using semiconductor elements). The control unit 107 controls the various units of the smartphone 100 by using the working memory 108 as working memory, according to a program stored in, for example, non-volatile memory 109.

[0041] The non-volatile memory 109 is an electrically erasable and recordable non-volatile memory that stores image data, audio data, other data, and various programs operated by the control unit 107. The non-volatile memory 109 is, for example, composed of flash memory, ROM, etc.

[0042] The image processing unit 110, under the control of the control unit 107, performs various image processing operations on images stored in the non-volatile memory 109 and storage medium 116, image signals acquired via external I / F 112, and images acquired via communication I / F 113. The image processing performed by the image processing unit 110 includes analog-to-digital conversion, digital-to-analog conversion, and image data encoding, compression, decoding, scaling (reduction), noise reduction, and color conversion. The image processing unit 110 also performs various types of image processing on VR videos of omnidirectional images or wide-area images with a wide range of data, such as panoramic unfolding, mapping, and transformation. The image processing unit 110 can be composed of dedicated circuit modules for specific image processing. Depending on the type of image processing, the control unit 107 can perform image processing according to a program without using the image processing unit 110.

[0043] Display 101, under the control of control unit 107, displays images, GUI screens constituting a graphical user interface (GUI), etc. Control unit 107 generates display control signals according to a program and controls the units of smartphone 100 to generate image signals for display on display 101 and outputs the image signals to display 101. Display 101 displays images based on the output image signals. Components of smartphone 100 itself may include interfaces for outputting image signals for display on display 101, and display 101 may be composed of an external monitor (such as a television).

[0044] Storage medium I / F 111 can load storage medium 116 such as memory card, CD, DVD, and, based on the control unit 107, read data from the loaded storage medium 116 and write data to the storage medium 116.

[0045] The external I / F 112 is an interface for connecting to external devices via wired or wireless means to input and output image and audio signals.

[0046] The Communication I / F 113 is an interface for communicating with external devices, the Internet, etc., to transmit and receive various data such as files and commands. The Communication I / F 113 can also communicate with wirelessly connected controllers 300, 310, etc. (described later). The Communication I / F 113 communicates via wired methods such as USB cables or wireless methods such as Bluetooth or Wi-Fi.

[0047] The audio output unit 103 outputs audio from video and music data, operation tones, ringtones, and various notification tones. The audio output unit 103 includes an audio output terminal 103a for connecting headphones and a speaker 103b, and can also output sound via wireless communication, etc.

[0048] The attitude detection unit 114 detects the attitude of the smartphone 100 relative to the direction of gravity and the tilt of the attitude relative to the roll axis, pitch axis, and yaw axis. Based on the attitude detected by the attitude detection unit 114, it can be determined whether the smartphone 100 is held horizontally, vertically, facing upwards, facing downwards, or in a tilted posture. At least one of an accelerometer, gyroscope, magnetometer, orientation sensor, and altitude sensor can be used as the attitude detection unit 114, and multiple sensors can be used in combination.

[0049] The self-position estimation unit 115 estimates the self-position of the smartphone 100 or VR glasses 200 (described later) in space and the surrounding environment.

[0050] "Self-position" refers to the location of the smartphone 100 or VR glasses 200 (described later) in space. For example, self-position is represented by three parameters expressing position in a coordinate system that sets the predetermined location in a predetermined range of space as the origin and defines three mutually orthogonal axes as the X-axis, Y-axis, and Z-axis, respectively. Self-position can also be represented by three additional parameters expressing attitude (or orientation).

[0051] The line-of-sight sensor unit 104 is composed of a CCD sensor and the like and performs photoelectric conversion on the infrared image.

[0052] The image capturing unit 105 is a camera capable of acquiring images. The pose detection unit 102f (described later), the self-position estimation unit 115, and others can perform various detection processes using the acquired images. The control unit 107 can output the information acquired by the image capturing unit to the pose detection unit 102f, and can also output external images to the display 101.

[0053] The operation unit 102 includes a touch panel 102a. The control unit 107 is capable of detecting the following operations or states of the touch panel 102a.

[0054] • When a finger or stylus that has not previously touched the touch panel 102a makes a new contact with the touch panel 102a, that is, when a touch begins (hereinafter referred to as a touch-down).

[0055] • Continued contact of a finger or stylus with the touch panel 102a (hereinafter referred to as touch-on)

[0056] • A finger or stylus moves while touching the touch panel 102a (hereinafter referred to as touch-move).

[0057] • When the finger or stylus that was touching the touch panel 102a leaves the touch panel 102a, that is, the touch ends (hereinafter referred to as touch-up).

[0058] • No object is in contact with the touch panel 102a (hereinafter referred to as touch-off)

[0059] When a touch press is detected, a touch hold is also detected simultaneously. After a touch is pressed, a touch hold is typically detected until a touch release is detected. Touch hold is also detected when touch movement is also detected. Even when a touch hold is detected, touch movement is not detected unless the touch location changes. Touch release is detected when all fingers or pens that are touching the object are released.

[0060] These operations and states, along with notifications of the position coordinates of a finger or stylus touching the touch panel 102a, are provided to the control unit 107 via the internal bus 106. The control unit 107 then determines, based on the provided information, what type of operation (touch operation) was performed on the touch panel 102a. Regarding touch movement, the direction of movement of the finger or stylus on the touch panel 102a can also be determined based on changes in position coordinates, specifically the vertical and horizontal components on the touch panel 102a. When a touch movement of a predetermined distance or more is detected, a swipe operation is determined. A flick is an operation where a finger quickly moves a certain distance while touching the touch panel 102a and then releases it. In other words, a flick is a quick swipe across the touch panel 102a like a light tap. When a touch movement of a predetermined distance or more at a predetermined speed is detected, followed by a touch release, a flick can be determined (a swipe operation followed by a flick operation can be determined). Furthermore, simultaneously touching multiple points (e.g., two points) and bringing the touch positions closer together is called pinch-in, while moving the touch positions and further separating them is called pinch-out. Pinch-out and pinch-in are collectively referred to as pinch operations (or simply pinch). Touch panel 102a can be used with any of the following types of touch panels: resistive film, capacitive, surface acoustic wave, infrared, electromagnetic induction, image recognition, and optical sensor. There are two methods for detecting touch: one is to detect a touch when there is contact with the touch panel, and the other is to detect a touch when a finger or pen approaches the touch panel; however, either of these methods can be used.

[0061] The operation unit 102 includes a posture detection unit 102f. The control unit 107 can acquire posture images of the user's hand, etc., through the image capturing unit 105, and detect a predetermined posture in the posture detection unit 102f. In other words, instructions can be obtained through posture.

[0062] The operation unit 102 includes a gaze input detection unit 102g. The control unit 107 can acquire the user's gaze information through the gaze sensor unit 104, and interpret the gaze information as a predetermined input operation in the gaze input unit 102g. For example, based on the output signal from the gaze sensor unit 104, the control unit 107 detects the user's gaze (the direction the user is looking) using methods such as corneal reflex and scleral reflex, and interprets the detected gaze information as a predetermined input operation in the gaze input unit 102g. In other words, instructions can be obtained through gaze.

[0063] <Headband Configuration>

[0064] Figure 1D This is an external view of a VR glasses (headset adapter) 200 that can accommodate a smartphone 100. The smartphone 100 can also be used as a head-mounted display by mounting it onto the VR glasses 200. The insertion port 201 is for inserting the smartphone 100. The smartphone 100 can be inserted entirely into the VR glasses 200 with the display surface of the display 101 facing the headband 202 (i.e., the user) used to secure the VR glasses 200 to the user's head. In this way, the user does not need to hold the smartphone 100; they can view the display 101 of the smartphone 100 simply by wearing the VR glasses 200 on their head. In this case, the posture of the smartphone 100 changes when the user moves their head or entire body. The posture detection unit 114 detects this posture change of the smartphone 100, and the control unit 107 performs VR display processing based on the detected posture change. In other words, instructions for changing the viewing direction of the VR video can be obtained based on the posture change. In this case, the posture detection unit 114 detects the posture of the smartphone 100 in the same way it detects the posture of the user's head (the direction the user's gaze is directed). The lens 203 of the image capturing device allows the image capturing unit 105 of the display device 100 to still acquire images of the real space and the user's posture even when mounted on the VR glasses 200.

[0065] The smartphone 100 can detect the user's gaze, facial expressions, etc., and can use this information as the operating method in the smartphone 100.

[0066] <Controller Configuration>

[0067] Figure 3 A is an external view of the controller 300 that can communicate with the smartphone 100. When the user holds the holding part 301 and operates the components on the operating surface 302, the handheld controller 300 notifies the smartphone 100 of the operation event.

[0068] Figure 3 The ring controller 310 shown in B includes a ring portion 311 for wearing the ring controller 310 on a user's finger 313 and a ring operation unit 312. The ring operation unit 312 may be a button or a component capable of detecting finger contact, such as a touch panel, a rotary dial, and an optical trackpad.

[0069] The controller communicates wirelessly with the smartphone 100 via Bluetooth through the communication I / F 113.

[0070] (Flowchart in this embodiment)

[0071] The following text will describe a user wearing and using VR glasses 200 equipped with a smartphone 100. (Refer to...) Figure 3 The control flow of this embodiment is described. Figure 3 The control flow shown is processed within the control unit 107. The flowchart is processed when the control unit 107 reads a predetermined program from the memory, deploys the program to the working memory, and runs the program.

[0072] Figure 3 This is an example of video viewing mode processing in smartphone 100.

[0073] When the smartphone 100 detects that it has been mounted on the VR glasses 200 or when the user performs a video playback operation, the smartphone 100 executes video viewing mode processing.

[0074] In S301, the control unit 107 reads an image file from the storage medium 116. Here, a VR video file will be used as an example for description.

[0075] In S302, the control unit 107 begins to detect changes in the posture of the smartphone 100 via the posture detection unit 114, and detects the user's gaze input via the gaze input detection unit 102g. In other words, the control unit 107 acquires the user's gaze.

[0076] In S303, the control unit 107 plays the VR video file it reads.

[0077] Figures 4A to 4J This is an example of a smartphone 100's screen in video viewing mode.

[0078] Figure 4A This is an example of a VR video being played. Assume the scenario: VR video 401 is being displayed and played. For clarity, dashed lines are drawn around the VR video, while solid lines are drawn around the perspective image.

[0079] A viewing direction indicator 402 is overlaid on the VR video 401 to indicate the viewing direction relative to the 360° image. The viewing direction indicator 402 includes a viewing direction 402a. A gaze pointer 403 is overlaid on the VR video 401 to acquire the user's gaze information via the gaze sensor unit 104 and to indicate the detected gaze position detected by the gaze detection unit 102c. In other words, the control unit 107 performs gaze display control by displaying the gaze pointer 403 and making the gaze pointer follow the user's gaze. For example, a ray 405 indicating the direction of the grip controller 240 is also overlaid.

[0080] In S304, the control unit 107 determines whether an operation to pause the VR video playback has been performed on the operation unit 102. For example, this is an operation on the operation surface 242 of the handheld controller 240. If it is determined that an operation has been performed, the process proceeds to S305; however, if it is determined that no operation has been performed, the process proceeds to S306.

[0081] In S305, control unit 107 stops (pauses) the playback of VR video.

[0082] Figure 4B This is an example of the screen after a pause operation has been performed. Assume the scenario: The VR video is paused due to user interaction, and the GUI panel 404 overlays the video onto the VR video 401. The GUI panel 404 overlays a background panel 404a, a VR video title 404b, toggle buttons 404c and 404d for switching between video files, a video timeline 404e, and a play icon 404f for playing the video. In other words, since it is currently paused, a play icon is displayed to resume the VR video 401. 404c to 404f represent GUI components that accept user interaction. When the gaze pointer 403 or ray 405 moves nearby and combines with other selection operations, the function indicated by the GUI component located near the position indicated by the gaze pointer 403 or ray 405 can be activated.

[0083] In S306, the control unit 107 determines whether the operation of playing VR video has been performed on the operation unit 102. In other words, in S306, the control unit 107 performs playback control. When it is determined that an operation has been performed, the process proceeds to S307; however, when it is determined that no operation has been performed, the process proceeds to S308.

[0084] In S307, control unit 107 plays (restores) VR video.

[0085] Figure 4C This is an example of the screen after playback has been performed. Figure 4B In the paused screen, assuming the following scenario: the user selects play icon 404f to play a video, and the video resumes. A GUI panel 404 displaying the video is overlaid on the VR video 401. A pause icon 404g for pausing video playback is overlaid on the GUI panel 404. When the video is played by selecting play icon 404f, play icon 404f switches to pause icon 404g. The GUI panel 404 hides after a period of time while the video is playing. The GUI panel 404 can also be hidden by explicit user input on the operation unit 102.

[0086] In S308, the control unit 107 determines whether an attitude change has been detected by the attitude detection unit 114. If an attitude change is detected, the process proceeds to S309; ​​however, if no attitude change is detected, the process proceeds to S310.

[0087] In S309, the viewing direction is changed based on the detected attitude change. The screen is refreshed so that the direction of the viewing direction 402a in the viewing direction indicator 402 is associated with the changed viewing direction.

[0088] In S310, the control unit 107 determines whether the operation to start perspective has been performed on the operation unit 102. If it is determined that the operation has been performed, the process proceeds to S311; however, if it is determined that the operation has not been performed, the process proceeds to S316.

[0089] In S311, the control unit 107 determines whether a VR video is being played. If it is determined that a VR video is being played, the process proceeds to S312; however, if it is determined that a VR video is not being played, the process proceeds to S315.

[0090] In step S312, the control unit 107 stores the gaze position detected by the gaze detection unit 102c in the working memory 108. In this embodiment, the gaze position is used; however, gaze coordinates or gaze region can also be used, as long as information related to the gaze is stored in the working memory 108. The control unit 107 can calculate the information to be stored as needed after acquiring the gaze.

[0091] In S313, the control unit 107 displays a marker (described later) in the VR video at the stored viewing position. This marker indicates the position the user was viewing before the perspective began, and also indicates the position where the play icon will be displayed when the user ends the perspective function and resumes the video.

[0092] In S314, the control unit 107 stops (pauses) the playback of the VR video. The processing in S314 can be performed before S312.

[0093] In S315, control unit 107 ends the VR video display and begins the perspective display. In other words, the video is stopped. Figure 4D This is an example of the screen after switching to perspective view. Assume the scenario: the display starts from... Figure 4C The scene switches to perspective display, and as shown by the viewing direction indicator 402, the viewing direction remains unchanged while the mode switches to perspective mode. A perspective icon 406 indicating that it is in perspective mode is overlaid on the perspective image 407. The viewing direction indicator 402 is also overlaid. Here, Figure 4C VR videos and Figure 4D The viewing direction 402a of the perspective image remains unchanged.

[0094] Figure 4E It is from Figure 4D This is an example of a view showing the user looking directly behind them (180° opposite). Hypothetical scenario: The user is using the perspective function to look directly behind them. An example scenario where this perspective function is used is to check for safety, etc. For example, there might be a situation where the video stops upon collision with an obstacle. The content displayed in perspective image 407 will change, and the viewing direction 402a will also be reversed by 180°.

[0095] In S316, the control unit 107 determines whether the operation to end the perspective has been performed on the operation unit 102. If it is determined that the operation has been performed, the process proceeds to S317; however, if it is determined that the operation has not been performed, the process proceeds to S318.

[0096] In S317, control unit 107 ends the perspective display and begins the VR video display. When the VR video display begins, the process proceeds to S318. When the VR video display begins, the process of S313 can be inserted before proceeding to S318. In this case, it is skipped after S312.

[0097] The process in S313 proceeds to S314.

[0098] Figure 4F This is an example of the scene after perspective has been completed. Assume the scenario: displaying from... Figure 4E The scene switches to the video viewing screen, and as shown by the viewing direction indicator 402, the viewing direction remains unchanged and the mode changes from... Figure 4E The scene switches to the video viewing screen. Figure 4C Compared to the previous scene, the viewing direction is 180° opposite, and according to the viewing direction indicator 402, the viewing direction also appears to be facing the opposite direction. In other words, the viewing direction of the VR video changes in response to the user's operation of changing the viewing direction. A GUI panel 404 of the video is overlaid on the VR video 401. In this state, the video remains paused, and a play icon 404f for playing the video is overlaid on the GUI panel 404.

[0099] Figure 4G The viewing direction is from Figure 4E Example of the scene after rotating 90° to the left. Hypothetical scenario: The user moves the screen while attempting to restore the video from the viewing direction set before using the perspective function. Assume the GUI panel 404 is fixed in the displayed viewing direction. Therefore, when the viewing direction is as shown... Figure 4G When the display position changes as shown, it will follow the change in the VR video 401 in the background.

[0100] Figure 4H The viewing direction is shown from Figure 4G Further changes and towards Figures 4A to 4C The same orientation state. Assume a scenario: When a user attempts to restore a video from a viewing direction set before using the perspective function, they are facing the same direction as the previous viewing direction. Also assume a scenario: The user's current viewpoint is displayed via gaze pointer 403 and is about to reach marker 408, which serves as the playback icon when restoring the video. In other words, assume a scenario: The user is attempting to restore VR video 401 based on the positional relationship between gaze pointer 403 and marker 408. Here, before the perspective display, marker 408, which indicates the stored gaze position and is shown in S313, is overlaid.

[0101] In S318, the control unit 107 determines in the gaze detection unit 102c whether the detected gaze position (i.e., gaze pointer 403) is near the mark 408. When it is determined that the gaze pointer 403 is near the mark 408, the process proceeds to S319; however, when it is determined that the gaze pointer 403 is not near the mark 408, the process proceeds to S320.

[0102] In S319, the control unit 107 plays (restores) the VR video. In other words, in S319, the control unit 107 performs playback control. Figure 4I The status of the restored video is shown. When the user faces the same direction as their previous viewing direction, marker 408 is displayed at the position the user was viewing before the perspective began, and marker 408 is used as the playback icon when the video is restored. Hypothetical scenario: The line of sight pointer 403, representing the user's viewpoint, reaches marker 408 and the VR video 401 is restored. Marker 408 is hidden when the video is restored.

[0103] In S320, the control unit 107 determines whether any other operation has been performed on the operation unit 102. Examples of other operations include reading other video files recorded on the storage medium 116 by operating switch buttons 404c and 404d, and changing the playback position on the video timeline by operating the timeline 404e. When it is determined that an operation has been performed, the process proceeds to S321; however, when it is determined that no operation has been performed, the process proceeds to S322.

[0104] In S321, the control unit 107 performs other processing based on the operation in S320.

[0105] In S322, the control unit 107 determines whether the operation to end the video viewing mode has been performed on the operation unit 102. If it is determined that the operation has been performed, the process ends; however, if it is determined that the operation has not been performed, the process returns to S304.

[0106] In each of the hypothetical scenarios described above, the detected gaze position matches the gaze pointer; however, the range of the detected gaze position detected by the gaze sensor unit 104 and the size of the gaze pointer 403 displayed based on the detected range can be switched as needed. For example, a single coordinate of the smallest unit detectable by the gaze sensor unit 104 can be used as the detected gaze position, or multiple coordinates can be used. A plane or region calculated based on multiple coordinates can be stored as the detected gaze position. The size of the gaze pointer 403 can match or differ from the range of the detected gaze position. For example, when using a coordinate of the smallest detectable unit as the detected gaze position, the accuracy increases if the gaze pointer 403 is displayed in a way that adapts to the range. However, since the gaze pointer 403 is small and there is a risk of reduced visibility, the gaze pointer 403 can be displayed as the smallest unit larger than the range of the gaze pointer.

[0107] The range of the detected gaze position stored in S312 and the size of the area used to determine whether to restore the video in S318 based on this range (i.e., the size of the marker 408 used as a playback icon when restoring the video) can be switched as needed. For example, the detected gaze position stored in S312 can be a single coordinate that is the smallest unit detectable by the gaze sensor unit 104, or it can be multiple coordinates. A plane calculated based on multiple coordinates can be stored as the detected gaze position. The size of the area used to determine whether to restore the video in S318 can match the range of the detected gaze position stored in S312, or it can be different from the range of the detected gaze position. For example, when using a single coordinate that is the smallest unit detectable by the detected gaze position stored in S312, the accuracy will increase if the size of the area used to determine whether to restore the video is set in a way that adapts to the range. However, since it may be difficult to adjust the gaze direction, the marker 408 used as a playback icon when restoring the video can be set to be larger than the range of the detected gaze position or the gaze pointer 403.

[0108] The size and shape of the marker 408 can also be similarly changed as needed. For example, when only the coordinates of the smallest unit detectable by the gaze sensor unit 104 are stored in S312, the marker can indicate a precise point. When the detected gaze position is stored as a plane in S312, the size of the marker 408 can also be changed according to the area. The marker 408 can be displayed over a wider area than the point (plane) stored in S312, and video can be played when the gaze pointer 403 is displayed near or within the marker. Figure 5A The storage pre-defined object 501 shown; or as shown in the figure. Figure 5BThe image shows arrow 502 indicating the stored coordinates or area location. The stored coordinates or area location do not need to be displayed as an overlay on the VR video.

[0109] Regarding the positional relationship between the gaze pointer 403 and the mark 408, VR video 401 can be restored when the gaze pointer 403 and the mark 408 are in contact with each other; VR video 401 can also be restored when the gaze pointer 403 and the mark 408 overlap in a predetermined part.

[0110] The video can be restored based on the fact that the gaze pointer 403 is continuously displayed near the mark 408 (mark 408 is being gazed at) for a predetermined time. In other words, the video can be restored based on the fact that a predetermined time has passed while the gaze pointer 403 is near the mark 408. Alternatively, the video can be restored based on the fact that a predetermined time has passed while the gaze pointer 403 is not near the mark 408 but is in contact with, completely aligned with, partially aligned with, or within the area of ​​the mark 408.

[0111] If the gaze pointer 403 moves out of the range of the marker 408 due to fixation eye movement or other reasons before the predetermined time count for gaze determination ends, a buffer area can be set. Figure 4J This is an example of setting up a buffer area. (Compared to...) Figure 4H Similarly, when a user attempts to restore the video from a viewing direction prior to using the perspective function, the user faces the same direction as before, and the user's viewpoint is indicated by the gaze pointer 403. Assume a scenario where a buffer area 409 is set around marker 408 and the gaze pointer 403 quickly reaches this buffer area 409. This buffer area may be displayed in a visually recognizable form, or it may be unnecessary to display it due to the clutter of displaying multiple elements. An example of using a buffer area is as follows. The count used for gaze determination begins at the point when the gaze pointer 403 enters marker 408. During the period from the start to the end of the count, even if the gaze pointer 403 moves out of the range of marker 408, the count used for gaze determination continues as long as the gaze pointer 403 remains within the buffer area 409.

[0112] When the casing including the smartphone 100 is in a significantly unstable state, the processing of S312 and S318 does not require the aforementioned operations. An example of a state in which the casing including the smartphone 100 is significantly unstable includes a state in which the user is currently wearing VR glasses.

[0113] The detected gaze positions stored in S312 can be multiple positions, and correspondingly, multiple markers 408 can be displayed.

[0114] In S312, the detected gaze position is stored; however, information such as the smartphone 100's posture and its own position can also be stored. Then, if, in S318, when determining whether to resume video, the smartphone 100's posture, its own position, and other information match the stored information, the video can be resumed. In other words, when the current display range of the VR video 401 matches a stored past display range within a predetermined range or a larger range, the video can be resumed. The detected gaze position can also be stored, and the detected gaze position can be used in combination to determine whether to resume video.

[0115] If a perspective-initiating operation is performed while VR video 401 is playing in S311, the detected gaze position is stored in S312. Conversely, the processing in S312 can be performed even when VR video 401 is paused by pausing the video via the pause icon 404g.

[0116] exist Figure 4F In some cases, the user may not remember the previously detected gaze position due to changes in the display range. In such situations, a directional notification can be provided indicating the direction towards the detected gaze position. In other words, the control unit 107 can perform notification control.

[0117] As a method of notification in this direction, it can be as follows: Figure 5C As shown, mark 503 is displayed on the viewing direction indicator 402, or as... Figure 5D The diagram shows an indicator 504 added to the screen to indicate direction. In other words, notifications can be provided within a display unit, such as a monitor. Figure 5E As shown, the indicator can be changed to indicate relative position, just like the indicator 505 which indicates direction.

[0118] The smartphone 100 can be used without being mounted on the VR glasses 200. In this case, the user holds the smartphone 100 to watch VR videos. In this case, unlike when the smartphone 100 is mounted on the VR glasses 200, the user can see objects outside the display of the smartphone 100. In this case, the aforementioned directional notification can be provided using sound, vibration, light, etc. In other words, notification can be provided in a manner other than the display of a monitor. For example, when providing notification via sound, specific instructions can be provided from the audio output unit 103, or a notification tone can be provided from the audio output unit 103.

[0119] This invention can be implemented either before or after the perspective function, or before or after functions other than VR video.

[0120] In addition to the methods described above, the GUI panel 404 can also be designed to display when a touch press operation is detected on the touch panel 102a.

[0121] (Second Embodiment)

[0122] <Ordinary Perspective>

[0123] This invention is not limited to VR videos; images with a normal viewpoint can also be used. When an image has a normal viewpoint but is large enough that the entire image cannot be visually recognized at once, some parts may be missed. In such cases, this invention can be used to reduce these missed views.

[0124] In this example, the invention can be implemented in a digital camera equipped with an electronic viewfinder. In the digital camera's playback mode, when switching to image capture mode is performed on the operation unit 102 while a video recorded on the storage medium 116 is displayed and played on the display 101, the video playback pauses, and a live view is displayed on the display 101. This corresponds to the perspective display described above. Simultaneously, the detected gaze position, which has been continuously observed during video playback, can be maintained, and when switching back to playback mode, the video can be automatically resumed by gazing at the detected gaze position of the stored paused video.

[0125] (Other embodiments)

[0126] This invention is achieved by performing the following process: The process involves providing software (programs) that implement the functions of the above embodiments to a system or device via a network or various storage media, and the computer (or control unit, microprocessor unit (MPU), etc.) of the system or device reading and executing the program code. In this case, the program and the storage medium storing the program constitute this invention.

[0127] The various controls described above, which are performed by a control unit, can be executed by a single piece of hardware, or the control of the entire device can be executed by multiple pieces of hardware sharing processing.

[0128] The present invention has been described in detail according to its preferred embodiments; however, the present invention is not limited to these specific embodiments, and the present invention also includes various modes without departing from the spirit of the invention. Furthermore, the above embodiments are only used to illustrate embodiments of the present invention, and embodiments can be combined as needed.

[0129] This invention can be implemented by providing a program for implementing one or more functions of the above embodiments to a system or device via a network or storage medium, and causing one or more processors in the computer of the system or device to read and run the program. Alternatively, this invention can be implemented by a circuit (e.g., an application-specific integrated circuit (ASIC)) that implements one or more functions.

[0130] This invention is not limited to the embodiments described above. Various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the claims are appended to illustrate the scope of the invention.

[0131] This application claims the benefit of Japanese Patent Application No. 2023-036745, filed on March 9, 2023, the entire contents of which are incorporated herein by reference. Claims (as amended under Article 19 of the Treaty) 1. An information processing apparatus, the information processing apparatus comprising: Display control unit, configured to display video on display unit; A gaze acquisition unit is configured to acquire gaze information, which represents the position or range of the user's viewing within the display range of the video displayed on the display unit. A recording unit, configured to record gaze information acquired by the gaze acquisition unit; and A playback control unit is configured to perform control to stop the video when its display stops while the video is playing, and to restore the video when its display resumes, based on the line-of-sight information indicating the user's viewing position or range within the display area of ​​the video on the display unit and the line-of-sight information recorded by the recording unit before the video's display resumes. 2. The information processing apparatus according to claim 1, further comprising: An instruction acquisition unit is configured to, in response to the user's operation, acquire an instruction for changing the position of the display range of the video, wherein... The display control unit is configured to perform control to display a portion of the video as a display area on the display unit, and to change the position of the display area of ​​the video according to the instruction. 3. The information processing apparatus according to claim 2, wherein... The instruction acquisition unit is configured to acquire the detection result of the position or orientation of the display unit as the instruction, and The display control unit is configured to change the position of the display range according to a change in the position or the orientation of the display unit. 4. The information processing apparatus according to claim 2, wherein... The instruction acquisition unit is configured to acquire the detection result of at least one of touch operation on the display unit, operation through gaze, and operation through gesture as the instruction. 5. The information processing apparatus according to claim 3, wherein The display control unit is configured to, when the display of the video resumes and the position or orientation of the display unit changes compared to when the display of the video stopped, change the position of the display range of the video from the position when the video was stopped to the position of the display range when the video was resumed, based on the change in the position or orientation of the display unit. 6. The information processing apparatus according to claim 1, wherein The display control unit is configured to display items indicating the position or range of gaze information previously recorded by the recording unit based on the video display recovery, thereby overlaying the items onto the video. 7. The information processing apparatus according to claim 6, further comprising: A gaze display control unit is configured to perform control to overlay a pointer following the user's current gaze onto the video, wherein... The playback control unit is configured to recover the video based on the positional relationship between the item and the pointer. 8. The information processing apparatus according to claim 7, wherein The playback control unit is configured to perform control to play the video when the pointer and the information indicating the position or range overlap in a predetermined portion. 9. The information processing apparatus according to claim 7, wherein The playback control unit is configured to perform control to play the video when a predetermined time has elapsed since the pointer and the information indicating the position or range overlap in a predetermined portion. 10. The information processing apparatus according to claim 1, further comprising: A notification control unit is configured to, when the display of the video resumes, perform control based on the gaze information and the current display range to provide a notification of direction from the position of the current display range toward the position of the display range corresponding to the gaze information. 11. The information processing apparatus according to claim 10, wherein The notification control unit is configured to perform control to provide a notification of the relative position of the line of sight information with respect to the current display range. 12. The information processing apparatus according to claim 10, wherein The notification control unit is configured to perform control to provide notifications via at least one of sound, vibration, and light, in addition to the display unit. 13. The information processing apparatus according to claim 10, wherein The notification control unit is configured to execute control to provide the notification by overlaying it onto the video within the display unit. 14. The information processing apparatus according to claim 1, wherein The recording unit is configured to further record the display range of the video displayed on the display unit, and The playback control unit is configured to perform control, based on line-of-sight information indicating the user's current viewing position or range and line-of-sight information recorded by the recording unit before the video is displayed and restored, to restore the video when the display range of the video, as recorded by the recording unit before the video is displayed and restored, is in a state where the display range of the video recorded by the recording unit before the video is displayed and restored. 15. The information processing apparatus according to claim 1, wherein The playback control unit is configured to perform control when the video display resumes, to restore the video based on gaze information indicating the user's current viewing position or range and gaze information when the video stopped, the gaze information being recorded by the recording unit. 16. The information processing apparatus according to claim 1, wherein The playback control unit is configured to perform control when the video display is restored, to restore the video based on gaze information indicating the user's current viewing position or range and gaze information when the operation to stop the video display was performed, the gaze information being recorded by the recording unit. 17. A control method for an information processing device, the control method comprising: Display control steps and display video on the display unit; The gaze acquisition step involves acquiring gaze information, which represents the position or range of the user's viewing within the display area of ​​the video displayed on the display unit. Recording steps, recording the gaze information acquired in the gaze acquisition steps; and The playback control step involves executing control to stop the video when its display stops while the video is being played, and to restore the video when its display resumes, based on the line-of-sight information indicating the user's viewing position or range within the display area of ​​the video on the display unit and the line-of-sight information recorded in the recording step prior to the video's display resumption. 18. A program that causes a computer to act as a unit of the information processing apparatus according to claim 1. 19. A system for an information processing apparatus, the system comprising: Display control device, the display control device being configured to display video on a display unit; A gaze acquisition device, the gaze acquisition device being configured to acquire gaze information, the gaze information representing the position or range of a user viewing within the display range of a video displayed on a display unit; A recording device, configured to record gaze information acquired by the gaze acquisition unit; and A playback control device configured to perform control, when the display of the video stops while the video is being played, to stop the video, and when the display of the video resumes, to resume the video based on the line-of-sight information indicating the position or range of the user's view within the display area of ​​the video displayed on the display unit and the line-of-sight information recorded by the recording device before the display of the video resumes.

Claims

1. An information processing apparatus, the information processing apparatus comprising: Display control unit, configured to display video on display unit; A gaze acquisition unit, configured to acquire the user's gaze information; A recording unit configured to record the gaze information when the video display stops; as well as A playback control unit is configured to stop the video when the video display stops, and to restore the video based on the gaze information recorded on the recording unit when the video display resumes.

2. The information processing apparatus according to claim 1, further comprising: An instruction acquisition unit is configured to, in response to the user's operation, acquire an instruction for changing the position of the display range of the video, wherein... The display control unit is configured to perform control to display a portion of the video as a display area on the display unit, and to change the position of the display area of ​​the video according to the instruction.

3. The information processing apparatus according to claim 2, wherein... The instruction acquisition unit is configured to acquire at least one detection result of the position or orientation of the display unit as the instruction, and The display control unit is configured to change the position of the display range based on a change in at least one of the position of the display unit and the orientation of the display unit.

4. The information processing apparatus according to claim 2, wherein... The instruction acquisition unit is configured to acquire the detection result of at least one of touch operation on the display unit, operation through gaze, and operation through gesture as the instruction.

5. The information processing apparatus according to claim 3, wherein The display control unit is configured to, when the display of the video is restored, set the display range to a corresponding position based on the past display range of the video and according to at least one of the orientation of the display unit and the tilt of the display unit.

6. The information processing apparatus according to claim 3, wherein The display control unit is configured to, when the display of the video is restored, set the display range to a corresponding position based on the past display range of the video and according to changes in at least one of the position of the display unit and the orientation of the display unit.

7. The information processing apparatus according to claim 1, wherein... The line-of-sight information is information indicating location or range, and The display control unit is configured to display the information by overlaying the information indicating the location or range onto the video.

8. The information processing apparatus according to claim 7, wherein The playback control unit is configured to restore the video based on the fact that information indicating the location or range is displayed on the display unit.

9. The information processing apparatus according to claim 7, wherein The playback control unit is configured to restore the video based on the information indicating the location or range and the user's current gaze.

10. The information processing apparatus according to claim 9, further comprising: A gaze display control unit is configured to perform control to overlay a pointer following the user's current gaze onto the video, wherein... The playback control unit is configured to recover the video based on the positional relationship between the information indicating the location or range and the pointer.

11. The information processing apparatus according to claim 10, wherein The playback control unit is configured to perform control to play the video when the pointer and the information indicating the position or range overlap in a predetermined portion.

12. The information processing apparatus according to claim 10, wherein The playback control unit is configured to perform control to play the video when a predetermined time has elapsed since the pointer and the information indicating the position or range overlap in a predetermined portion.

13. The information processing apparatus according to claim 1, further comprising: A notification control unit is configured to, when the display of the video resumes, perform control based on the gaze information and the current display range to provide a notification of direction from the current display range toward the display range where the gaze information is displayed.

14. The information processing apparatus according to claim 13, wherein The notification control unit is configured to perform control to provide a notification of the relative position of the line of sight information with respect to the current display range.

15. The information processing apparatus according to claim 13, wherein The notification control unit is configured to perform control to provide notifications via at least one of sound, vibration, and light, in addition to the display unit.

16. The information processing apparatus according to claim 13, wherein The notification control unit is configured to execute control to provide the notification by overlaying it onto the video within the display unit.

17. The information processing apparatus according to claim 1, wherein The playback control unit is configured to restore the video based on the line-of-sight information when the past display range of the video matches the current display range of the video within a predetermined range or a larger range.

18. A control method for an information processing device, the control method comprising: Display control steps and display video on the display unit; The gaze acquisition step involves obtaining the user's gaze information; The recording process includes recording the gaze information when the video display stops. as well as The playback control steps involve stopping the video when its display stops, and resuming the video based on the gaze information when its display resumes.

19. A program that causes a computer to act as a unit of the information processing apparatus according to claim 1.

20. A system for an information processing apparatus, the system comprising: Display control device, the display control device being configured to display video on a display unit; A gaze acquisition device, the gaze acquisition device being configured to acquire a user's gaze information; A recording device configured to record the gaze information when the display of the video stops; as well as A playback control device configured to stop the video when its display stops, and to resume the video based on the gaze information when its display resumes.

Images