Gaze-driven video recording

Abstract

The present disclosure relates to gaze-driven video recording. Systems and methods for gaze-driven video recording are described. Some implementations may include accessing gaze data captured using one or more gaze tracking sensors; applying a temporal filter to the gaze data to obtain a smoothed gaze estimate; determining a region of interest based on the smoothed gaze estimate, wherein the region of interest identifies a subset of a field of view; accessing a video frame; recording a portion of the frame associated with the region of interest as an enhanced video frame, wherein the portion of the frame corresponds to a smaller field of view than the frame; and storing, transmitting, or displaying the enhanced video frame.

Description

[0001] Division Statement

[0002] This application is a divisional application of the Chinese invention patent application with the application date of December 13, 2019, the invention name of “Gaze-driven video recording” and the application number of 201980082400.2. Technical Field

[0003] The present disclosure relates to gaze-driven video recording. Background Art

[0004] A head mounted display is used to provide a virtual reality, augmented reality, and / or mixed reality experience to a user. Video from a virtual reality or mixed reality experience may be recorded for later playback or analysis. Gaze tracking sensors may be used to provide an indication of where the person being filmed is currently looking. Summary of the invention

[0005] This article discloses a specific implementation of gaze-driven video recording.

[0006] In a first aspect, the subject matter described in this specification may be embodied in a system comprising one or more gaze tracking sensors, one or more image sensors, and a processing device, the processing device being configured to: access gaze data captured using the one or more gaze tracking sensors; apply a temporal filter to the gaze data to obtain a smoothed gaze estimate; determine a region of interest based on the smoothed gaze estimate, wherein the region of interest identifies a subset of a field of view of the one or more image sensors; access video frames captured using the one or more image sensors; and apply signal processing to the video frames based on the region of interest to obtain an enhanced video frame.

[0007] In a second aspect, the subject matter described in this specification can be embodied in a method comprising: accessing gaze data captured using one or more gaze tracking sensors; applying a temporal filter to the gaze data to obtain a smoothed gaze estimate; determining a region of interest based on the smoothed gaze estimate, wherein the region of interest identifies a subset of a field of view; accessing a video frame; applying signal processing to the video frame based on the region of interest to obtain an enhanced video frame; and storing, transmitting, or displaying the enhanced video frame.

[0008] In a third aspect, the subject matter described in this specification can be embodied in a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium can include executable instructions that, when executed by a processor, cause operations to be performed, including accessing gaze data captured using one or more gaze tracking sensors; applying a temporal filter to the gaze data to obtain a smoothed gaze estimate; determining a region of interest based on the smoothed gaze estimate, wherein the region of interest identifies a subset of a field of view; accessing a video frame; applying signal processing to the video frame based on the region of interest to obtain an enhanced video frame; and storing, transmitting, or displaying the enhanced video frame.

[0009] In a fourth aspect, the subject matter described in the present specification may be embodied in a system comprising one or more gaze tracking sensors, one or more image sensors, and a processing device, the processing device being configured to: access gaze data captured using the one or more gaze tracking sensors; determine a region of interest based on the gaze data, wherein the region of interest identifies a subset of a field of view of the one or more image sensors; access a video frame captured using the one or more image sensors; and record a portion of the frame associated with the region of interest as an enhanced video frame, wherein the portion of the frame corresponds to a smaller field of view than the frame.

[0010] In a fifth aspect, the subject matter described in the present specification may be embodied in a method comprising: accessing gaze data captured using one or more gaze tracking sensors; determining a region of interest based on the gaze data, wherein the region of interest identifies a subset of a field of view; accessing a video frame; recording a portion of the frame associated with the region of interest as an enhanced video frame, wherein the portion of the frame corresponds to a smaller field of view than the frame; and storing, transmitting, or displaying the enhanced video frame.

[0011] In a sixth aspect, the subject matter described in the specification may be embodied in a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium may include executable instructions that, when executed by a processor, cause operations to be performed, including accessing gaze data captured using one or more gaze tracking sensors; determining a region of interest based on the gaze data, wherein the region of interest identifies a subset of a field of view; accessing a video frame; recording a portion of the frame associated with the region of interest as an enhanced video frame, wherein the portion of the frame corresponds to a smaller field of view than the frame; and storing, transmitting, or displaying the enhanced video frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The following detailed description is best understood in conjunction with the accompanying drawings when reading this disclosure. It should be emphasized that, according to common practice, the various features of the drawings are not to scale. On the contrary, the sizes of the various features are arbitrarily enlarged or reduced for clarity.

[0013] Figure 1A is an illustration of an example of a process for recording video in a mixed reality or virtual reality application with a static region of interest.

[0014] Figure 1B is an illustration of an example of a process for recording video in a mixed reality or virtual reality application with a gaze-driven region of interest.

[0015] Figure 1C is an illustration of an example of a process for recording video in a mixed reality or virtual reality application with smooth gaze-driven regions of interest.

[0016] Figure 2 is a graph of an example of gaze data used to drive video recording in a mixed reality or virtual reality application.

[0017] Figure 3 is a block diagram of an example of a mobile computing device configured to enable gaze-driven video processing.

[0018] Figure 4 is a block diagram of an example of a system configured to enable gaze-driven video processing.

[0019] Figure 5 is a block diagram of an example of a cloud server configured to enable gaze-driven video processing.

[0020] Figure 6 is a flow chart of an example of a process for gaze-driven video processing.

[0021] Figure 7 is a flow chart of an example of a process for annotating an image based on objects in a gaze-driven region of interest.

[0022] Figure 8 is a flow chart of an example of a process for determining a region of interest based on a combination of gaze data and saliency information based on identified objects. DETAILED DESCRIPTION

[0023] A system and method are described herein that provide gaze-driven signal processing to facilitate mixed reality or virtual reality applications, such as mixed reality recording. In mixed reality recording, a portion of what a user sees can be recorded. The portion of what the user sees (e.g., including video frames and / or video frames of the surrounding environment captured using one or more image sensors mounted on a head-mounted display and presented to the user via a head-mounted display) can be referred to as a recording frame. The recording frame can be a fixed area relative to the video input, but then the user may be viewing interesting content that will be missed by recording outside of the area. The system proposed herein identifies an area of ​​interest by tracking the user's gaze and moves the recording frame as the user's gaze moves. One such example can be that the gaze is located at the center of the recording frame, so that the recording frame is included in the original frame. The recording frame can be a variety of shapes, including, for example, an ellipse, a hexagon, or a rectangle, among others. Moving the recording frame with a gaze center point can cause video jitter. Some specific implementations run a smoothing or stabilization algorithm above the gaze center point to avoid jitter in the recorded video and obtain a more pleasant recording experience. The idea includes use cases other than mixed reality recording, such as running feature extraction in an area of ​​interest; and variable bit allocation, which is higher in the area of ​​interest and lower in the surrounding area.

[0024] Some implementations may provide advantages over earlier video encoding systems, such as more efficiently focusing computing resources (e.g., memory, bandwidth, and / or processor cycles) on the portion of the video that the user is looking at. Some implementations may enable recording of mixed reality or virtual reality experiences in a manner that more effectively represents the user's actual experience.

[0025] Figure 1A 1 is an illustration of an example of a process 100 for recording video in a mixed reality or virtual reality application with a static region of interest. The process 100 takes a sequence of video frames (110, 112, 114, and 116) as input and records corresponding portions (120, 122, 124, and 126) of the video frames referred to as recording frames. For example, the video frames (110, 112, 114, and 116) may include video data captured using one or more image sensors (e.g., including images of a physical environment near a user) and / or images of virtual objects of a virtual environment displayed to the user (e.g., using a head-mounted display or a touch screen display of a smart phone or tablet). In this example, the portions (120, 122, 124, and 126) of the video frames correspond to the static region of interest, where the recording frames are each a rectangle of pixels located at the center of the corresponding frame (110, 112, 114, and 116) of the video.

[0026] Portions (120, 122, 124, and 126) are extracted from the video frames (110, 112, 114, and 116), and the portions (120, 122, 124, and 126) are recorded as a still view video 130. For example, the still view video 130 may be an mp4 file, and recording the still view video 130 may include passing the portions (120, 122, 124, and 126) of the video frames as input to an encoder for compression in preparation for storage and / or transmission of the resulting still view video 130. The still view video 130 may not account for changes in gaze or focus of the user during capture and / or presentation of the original video frames (110, 112, 114, and 116), and therefore it may not accurately represent the user's mixed reality or virtual reality experience.

[0027] Recording a fixed area may not be ideal. Something interesting that the user is watching may happen outside the recording frame and therefore not be captured in the recording. Instead, the region of interest can be dynamically identified using gaze data from a gaze tracking sensor, and the recording frame can be moved as the user's gaze moves. In some implementations, the region of interest and the recording frame can be determined so that the gaze is centered in the recording frame so that the recording frame is contained in the original frame. For example, if the gaze rests at the edge of the video frame or beyond, the recording frame can be set to the closest position on the edge of the frame where the recording frame is still full size. The recording frame can take many different shapes, such as an ellipse, a hexagon, or a rectangle.

[0028] Figure 1B1 is an illustration of an example of a process 140 for recording video in a mixed reality or virtual reality application with a gaze-driven region of interest. The process 140 takes a sequence of video frames (110, 112, 114, and 116) as input and records corresponding portions (150, 152, 154, and 156) of the video frames referred to as recording frames. In this example, the portions (150, 152, 154, and 156) of the video frames correspond to regions of interest at different times, wherein the recording frames are each a rectangle of pixels centered about a current estimate of a user's gaze direction during capture and / or rendering of the video frames (110, 112, 114, and 116). For example, gaze data of a user corresponding to frame 110 in time may indicate that the user is gazing in a direction corresponding to the upper left portion of the video frame 110. Based on the gaze data, a portion 150 of the video frame 110 may be selected for recording. For example, gaze data of a user corresponding to frame 112 in time may indicate that the user is gazing in a direction corresponding to the upper right portion of the frame 112. Based on the gaze data, a portion 152 of the video frame 112 may be selected for recording. For example, the gaze data of the user corresponding in time to frame 114 may indicate that the user is gazing in a direction corresponding to the lower left portion of frame 114. Based on the gaze data, a portion 154 of the video frame 114 may be selected for recording. For example, the gaze data of the user corresponding in time to frame 116 may indicate that the user is gazing in a direction corresponding to the lower right portion of frame 116. Based on the gaze data, a portion 156 of the video frame 116 may be selected for recording. Therefore, the region of interest determined based on the gaze data and the corresponding portion or recording frame may jump significantly between adjacent frames in the sequence of video frames (110, 112, 114, and 116), which may cause jitter when recording the user's mixed reality or virtual reality experience.

[0029] Portions (150, 152, 154, and 156) are extracted from the video frames (110, 112, 114, and 116), and the portions (150, 152, 154, and 156) are recorded as a shaken view video 160. For example, the shaken view video 160 may be an mp4 file, and recording the shaken view video 160 may include passing the portions (150, 152, 154, and 156) of the video frames as input to an encoder for compression in preparation for storage, transmission, and / or display of the resulting shaken view video 160. The shaken view video 160 may account for changes in the user's gaze or focus during the capture and / or presentation of the original video frames (110, 112, 114, and 116), but the shaken view video may be uncomfortable to look at due to jitter caused by the user's fine and rapid eye movements around the scene and / or noise in the user's gaze data.

[0030] For example, process 140 may include accessing gaze data captured using one or more gaze tracking sensors and determining a region of interest based on the gaze data. The region of interest may identify a subset of the field of view. For example, process 140 may include accessing a video frame and recording a portion of the frame associated with the region of interest as an enhanced video frame, wherein the portion of the frame corresponds to a smaller field of view than the frame. For example, process 140 may include storing, transmitting, or displaying the enhanced video frame.

[0031] In some implementations, a smoothing or stabilization algorithm can be run to process the sequence of gaze center points from the gaze data to avoid jitter in the recorded video, resulting in a more pleasant recording experience.

[0032] Figure 1C1 is an illustration of an example of a process 170 for recording video in a mixed reality or virtual reality application with smoothed gaze-driven regions of interest. The process 170 takes a sequence of video frames (110, 112, 114, and 116) as input and records corresponding portions (180, 182, 184, and 186) of the video frames referred to as recording frames. In this example, the portions (180, 182, 184, and 186) of the video frames correspond to regions of interest at different times, wherein the recording frames are each a rectangle of pixels centered about a smoothed estimate of a user's gaze direction during capture and / or rendering of the video frames (110, 112, 114, and 116). For example, a smoothed gaze estimate of a user corresponding in time to frame 110 may indicate that the user is gazing in a direction corresponding to the upper left portion of frame 110. Based on the smoothed gaze estimate, a portion 180 of the video frame 110 may be selected for recording. For example, a smoothed gaze estimate of a user corresponding in time to frame 112 may indicate that the user is looking in a direction corresponding to portion 182 slightly above and to the left of the center of frame 112, which may be an interpolation of the gaze position from the gaze data for a time window close to the time of frame 112. Based on the smoothed gaze estimate, portion 182 of video frame 112 may be selected for recording. For example, a smoothed gaze estimate of a user corresponding in time to frame 114 may indicate that the user is looking in a direction corresponding to portion 184 slightly above and to the left of the center of frame 114, which may be an interpolation of the gaze position from the gaze data for a time window close to the time of frame 114. Based on the smoothed gaze estimate, portion 184 of video frame 114 may be selected for recording. For example, a smoothed gaze estimate of a user corresponding in time to frame 116 may indicate that the user is looking in a direction corresponding to the lower right portion of frame 116, which may be an interpolation of the gaze position from the gaze data for a time window close to the time of frame 116. Based on the smoothed gaze estimate, portion 186 of video frame 116 may be selected for recording. Thus, the region of interest based on the smoothed gaze estimation and the corresponding partial sequence or recording frame can smoothly transition between different regions of the video frames (110, 112, 114, and 116) corresponding to the extended attention period of the user's gaze. This smooth evolution of the region of interest can result in a smooth recording of the user's mixed reality or virtual reality experience.

[0033] Portions (180, 182, 184, and 186) are extracted from the video frames (110, 112, 114, and 116), and the portions (180, 182, 184, and 186) are recorded as a smooth view video 190. For example, the smooth view video 190 may be an mp4 file, and recording the smooth view video 190 may include passing the portions (180, 182, 184, and 186) of the video frames as input to an encoder for compression in preparation for storage and / or transmission of the resulting smooth view video 190. The smooth view video 190 may account for changes in the user's gaze or focus during the capture and / or rendering of the original video frames (110, 112, 114, and 116), while suppressing noise in the gaze data and providing a useful representation of the user's mixed reality or virtual reality experience.

[0034] Figure 2 is a graph of an example of gaze data 200 for driving video recording in a mixed reality or virtual reality application. The gaze data is plotted with gaze center versus frame number to show the evolution of the gaze measured during the mixed reality or virtual reality experience. For simplicity, the gaze center is represented as one-dimensional (e.g., a horizontal offset or vertical offset of the user's gaze detected during the capture and / or rendering of the input video), but the gaze center can evolve, and typically does evolve in two dimensions. The solid black dots represent gaze center data points from a gaze tracking sensor with a gaze tracking algorithm. The hollow dots represent interpolated gaze positions that have been determined using temporal filtering to temporally interpolate nearby data points to determine a smoothed gaze estimate. Figure 2 A linear interpolation of gaze data is shown, but other types of interpolation may be used. When the corresponding gaze data for a frame is too far from data for nearby times or frames, the interpolated gaze position may be used as a smoothed gaze estimate for the frame. The curve plotted on the graph shows an example of a sequence of gaze center estimates that may be used to identify evolving regions of interest, which may be used to record a user's mixed reality or virtual reality experience.

[0035] Figure 3 3 is a block diagram of an example of a mobile computing device 300 that is configured to enable gaze-driven signal processing to facilitate mixed reality or virtual reality applications, such as mixed reality recording. The mobile computing device 300 includes a processing device 310, a data storage device 320, one or more gaze tracking sensors 330, one or more image sensors 340, a display 350, a network interface 360, and an interconnect 370 that the processing device 310 can use to access the other components. The mobile computing device 300 can be configured to determine a region of interest of a user based on the user's gaze data, and perform signal processing to enhance a portion of a video corresponding to the region of interest. For example, the mobile computing device 300 can be configured to implement Figure 6For example, the mobile computing device 300 may be configured to implement Figure 7 For example, the mobile computing device 300 may be configured to implement Figure 8 process 800.

[0036] The processing device 310 is operable to execute instructions stored in the data storage device 320. In some specific implementations, the processing device 310 is a processor with a random access memory for temporarily storing instructions read from the data storage device 320 when executing the instructions. The processing device 310 may include a single or multiple processors, each processor having a single or multiple processing cores. Alternatively, the processing device 310 may include another type of device or multiple devices capable of manipulating or processing data. For example, the data storage device 320 may be a non-volatile information storage device, such as a hard disk drive, a solid-state drive, a read-only memory device (ROM), an optical disk, a magnetic disk, or any other suitable type of storage device such as a non-transitory computer-readable memory. The data storage device 320 may include another type of device or multiple devices capable of storing data for retrieval or processing by the processing device 310. The processing device 310 may access and manipulate the data stored in the data storage device 320 via the interconnect 370. For example, the data storage device 320 may store instructions executable by the processing device 310, which, when executed by the processing device 310, causes the processing device 310 to perform operations (e.g., implement Figure 6 In some implementations, the processing device 310 and the data storage device 320 are attached to the display 350 .

[0037] One or more gaze tracking sensors 330 may be configured to detect the gaze of the user's eyes. For example, one or more gaze tracking sensors 330 may include one or more image sensors that can detect light in various light spectra (e.g., infrared or visible) and / or light sources (e.g., infrared light sources irradiated at the user's eyes). For example, one or more gaze tracking sensors 330 may also include hardware and / or software (e.g., gaze tracking software) for processing raw image data using a gaze tracking algorithm to generate gaze data, the gaze data including a sequence of gaze points, each gaze point specifying the direction or focus of the current gaze of the observed user. In some specific implementations, one or more gaze tracking sensors 330 are coupled (e.g., attached) to a display 350 (e.g., a head-mounted display or a touch screen display of a tablet or smart phone). For example, one or more gaze tracking sensors 330 may be mounted on a fastening article (e.g., an eyeglass frame or goggles) of the head-mounted display so that it is pointed at the eyes of a user wearing the head-mounted display. For example, one or more gaze tracking sensors 330 may include a forward-facing camera of a tablet or smart phone that is mounted so that it is pointed at the eyes of a user holding the device and viewing its touch screen display. In some implementations, the user's gaze direction in the coordinates of an attached image sensor or display can be determined based on gaze data from one or more gaze tracking sensors 330 .

[0038] One or more image sensors 340 may be configured to capture images, converting light incident on one or more image sensors 340 into one or more digital images. In some implementations, one or more image sensors 340 are coupled (e.g., attached) to a display 350 (e.g., a head-mounted display). One or more image sensors 340 may detect light of a specific spectrum (e.g., a visible spectrum and / or an infrared spectrum) and transmit information constituting an image as an electrical signal (e.g., an analog or digital signal). For example, one or more image sensors 340 may include an array of sensing elements, each of which may be used to output a signal representing incident light as an image having an array of pixels representing a captured scene (e.g., a picture of a real-world environment in which the image sensor is located). For example, one or more image sensors 340 may include a charge coupled device (CCD) or an effective pixel sensor in a complementary metal oxide semiconductor (CMOS). In some implementations, one or more image sensors 340 include an analog-to-digital converter. For example, one or more image sensors 340 may include an infrared camera and a visible light camera. One or more image sensors 340 may include an image sensor configured to capture images near the mobile computing device 300. In some implementations, the one or more image sensors 340 include an array of image sensors arranged around a device (e.g., head mounted display 410) to provide a collective field of view across a wide angle. For example, the one or more image sensors 340 can be arranged to provide a panoramic view (e.g., a 360 degree panoramic view) of an area surrounding the head mounted display. For example, the one or more image sensors 340 can receive light through corresponding lenses (e.g., fisheye lenses or linear lenses).

[0039] The display 350 includes a screen, a lens, or other type of optical component configured to direct light to the user's eyes so that an image (e.g., a video frame) can be presented to the user. For example, the display 350 may include a touch screen display, where the mobile computing device 300 is a smart phone or a tablet. In some implementations, the display 350 includes a head-mounted display (e.g., smart glasses) that can be held in place on the user's face by a fastener (e.g., a headband or frame). In some implementations, the screen of the display 350 is positioned directly in front of the user's eyes. In some implementations, the display 350 includes an optical component (e.g., a lens and / or a reflector) that is positioned directly in front of the user's eyes and is configured to direct light from the screen or projector of the display 350 to the user's eyes. The optical component can also direct light from the user's surroundings to the user's eyes. For example, the optical component may include a partially reflective polarizing film applied to the inner surface of a transparent visor. The optical component can be used as an optical combiner. For example, the lens of the optical assembly may also allow light from the environment in front of the user to pass through the lens to reach the user's eyes and allow the user to see in front of him or her, while objects of the virtual environment are depicted in the image presented by display 350 overlaid on a view of the physical environment in front of the user.

[0040] The network interface 360 ​​facilitates communication with other devices such as a cloud server. For example, the network interface 360 ​​may facilitate communication via a communication network. For example, the network interface 360 ​​may facilitate communication via a Wi-Fi network, a cellular network, and / or a wired network. For example, the network interface 360 ​​may facilitate communication via a WiMAX network. For example, the network interface 360 ​​may facilitate communication via a fiber optic network.

[0041] For example, interconnect 370 may be a system bus, or a wired or wireless network (eg, a body area network).

[0042] The processing device 310 may be configured to enable gaze-driven signal processing to facilitate mixed reality or virtual reality applications, such as mixed reality recording. For example, the processing device 310 may be configured to access gaze data captured using one or more gaze tracking sensors 330, apply a temporal filter to the gaze data to obtain a smoothed gaze estimate, and determine a region of interest based on the smoothed gaze estimate. For example, the region of interest may identify a subset of the field of view of one or more image sensors 340. For example, the processing device 310 may be configured to access a video frame captured using one or more image sensors 340, and apply signal processing to the video frame based on the region of interest to obtain an enhanced video frame. For example, the processing device 310 may be configured to record a portion of the frame associated with the region of interest as an enhanced video frame. For example, the portion of the frame may correspond to a smaller field of view than the frame. For example, the processing device 310 may be configured to encode the frame using the following method: for a first portion of the frame associated with the region of interest, a first bit per pixel allocation is used, and for one or more corresponding portions of the frame outside the first portion of the frame, one or more bit per pixel allocations that are lower than the first bit per pixel allocation are used. For example, the processing device 310 may be configured to apply computer vision processing to a portion of the frame associated with the region of interest to detect objects appearing in the portion of the frame and determine the classification of the object. For example, the portion of the frame may correspond to a smaller field of view than the frame. For example, the processing device 310 may be configured to generate annotations based on the classification and superimpose the annotations on the video frame to obtain an enhanced video frame. For example, the display 350 may include a head-mounted display attached to one or more gaze tracking sensors 330 and one or more image sensors 340. For example, the head-mounted display may be configured to display an enhanced video frame. In some specific implementations, the temporal filter includes a Kalman filter. In some specific implementations, the temporal filter includes a moving average of the gaze data in a time window. For example, the processing device 310 may be configured to apply the temporal filter to the gaze data by fitting a curve to the gaze data in the time window to obtain a smoothed gaze estimate. For example, the processing device 310 may be configured to determine a region of interest based on a smoothed gaze estimate in the following manner: applying computer vision processing to a video frame to detect and classify an object appearing in the video frame, thereby obtaining a position associated with the object within the video frame; determining a value of a cost function for the region of interest based on the smoothed gaze estimate and the position of the object; and determining the region of interest based on the value of the cost function.

[0043] Figure 44 is a block diagram of an example of a system 400 configured to enable gaze-driven signal processing to facilitate mixed reality or virtual reality applications, such as mixed reality recording. The system 400 includes a head-mounted display 410 and a computing device 450 that communicates via a wireless communication link 425. The head-mounted display 410 includes one or more gaze tracking sensors 412, one or more image sensors 414, a wireless communication interface 416, a display 422, and an interconnect 418 through which components of the head-mounted display can exchange information (e.g., under the control of a microcontroller or other processing device in the head-mounted display 410, Figure 4 4 ), and a fastening article 420 configured to hold the head mounted display 410 in place on the head of a user wearing the head mounted display 410. The computing device 450 includes a processing device 460, a data storage device 470, a wireless communication interface 480, a user interface 490, a network interface 492, and an interconnect 494 through which the processing device 460 can access other components of the computing device 450. The computing device 450 can be located near the user wearing the head mounted display 410, and can perform computing tasks to control the head mounted display 410. For example, the computing device 450 can be a smart phone, a tablet, a desktop computer, or an appliance (e.g., a smart TV or smart speaker) on a table in a room where the user is located, or the computing device 450 can be another wearable device worn on a different body part of the user (e.g., the computing device 450 can be integrated with a vest worn on the chest). The system 400 can be configured to determine a region of interest of the user based on the user's gaze data, and perform signal processing to enhance a portion of the video corresponding to the region of interest. For example, system 400 may be configured to implement Figure 6 For example, the system 400 may be configured to implement Figure 7 For example, the system 400 may be configured to implement Figure 8 process 800.

[0044] One or more gaze tracking sensors 412 may be configured to detect the gaze of the user's eyes. For example, one or more gaze tracking sensors 412 may include one or more image sensors that can detect light in various light spectra (e.g., infrared or visible) and / or light sources (e.g., infrared light sources irradiated at the user's eyes). For example, one or more gaze tracking sensors 412 may also include hardware and / or software (e.g., gaze tracking software) for processing raw image data using a gaze tracking algorithm to generate gaze data, the gaze data including a sequence of gaze points, each gaze point specifying the direction or focus of the current gaze of the observed user. In some specific implementations, one or more gaze tracking sensors 412 are coupled (e.g., attached) to a display 422. For example, one or more gaze tracking sensors 412 may be mounted on a fastening article 420 (e.g., a spectacle frame or goggles) of the head-mounted display 410 so that it is directed to the eyes of the user wearing the head-mounted display 410. In some specific implementations, the gaze direction of the user in the coordinates of the attached image sensor or display may be determined based on the gaze data from the one or more gaze tracking sensors 412.

[0045] The head-mounted display 410 includes one or more image sensors 414. The one or more image sensors 414 may be configured to capture images and convert light incident on the one or more image sensors 414 into one or more digital images. The one or more image sensors 414 are coupled (e.g., attached) to the head-mounted display 410. The one or more image sensors 414 may detect light of a specific spectrum (e.g., a visible spectrum and / or an infrared spectrum) and transmit information constituting the image as an electrical signal (e.g., an analog or digital signal). For example, the one or more image sensors 414 may include a charge coupled device (CCD) or an effective pixel sensor in a complementary metal oxide semiconductor (CMOS). In some specific implementations, the one or more image sensors 414 include an analog-to-digital converter. For example, the one or more image sensors 414 may include an array of sensing elements, each of which may be used to output a signal representing the incident light as an image having an array of pixels representing the captured scene (e.g., a picture of the real-world environment in which the image sensor is located). For example, the one or more image sensors 414 may include an infrared camera and a visible light camera. The one or more image sensors 414 may include an image sensor configured to capture an image near the head-mounted display 410. In some implementations, the one or more image sensors 414 include an array of image sensors arranged around the head mounted display 410 to provide a collective field of view across a wide angle. For example, the one or more image sensors 414 can be arranged to provide a panoramic view (e.g., a 360 degree panoramic view) of the area surrounding the head mounted display 410. For example, the one or more image sensors 414 can receive light through corresponding lenses (e.g., fisheye lenses or linear lenses).

[0046] The head mounted display 410 includes a wireless communication interface 416. The wireless communication interface 416 facilitates communication with other devices including the computing device 450. For example, the wireless communication interface 416 can facilitate communication via a Wi-Fi network, a Bluetooth link, or a ZigBee link. In some implementations, the wireless communication interface 416 can be used to receive data from the computing device 450 and / or invoke instructions for rendering a view of the virtual environment using the display 422. In some implementations, the wireless communication interface 416 can be used to transmit sensor data from one or more gaze tracking sensors 412 and / or images captured using one or more image sensors 414 to the computing device 450.

[0047] The head-mounted display 410 includes a display 422. The display 422 may be configured to present an image, thereby converting a digital image into light projected from the display 422. The display 422 may project light using a pixel array that projects light in the visible spectrum. The display 422 may include a screen, a lens, or other types of optical components configured to direct light to the eyes of a user wearing the head-mounted display 410 so that an image (e.g., a video frame) can be presented to the user. For example, the display 422 may include a screen, such as a liquid crystal display (LCD), a light emitting diode (LED) display (e.g., an OLED display), or other suitable screens. For example, the display 422 may include a projector. In some specific implementations, the screen of the display 422 is positioned directly in front of the user's eyes. In some specific implementations, the display 422 includes an optical component (e.g., a lens and / or a reflector) that is positioned directly in front of the user's eyes and is configured to direct light from the screen or projector of the display 422 to the user's eyes. The optical component may also direct light from the user's surroundings to the user's eyes. For example, the optical assembly may include a partially reflective polarizing film applied to the inner surface of the transparent visor. The optical assembly may function as an optical combiner. For example, the lens of the optical assembly may also allow light from the physical environment in front of the user to pass through the lens to reach the user's eyes and allow the user to see in front of him or her while objects of the virtual environment are depicted in an image presented by the display 422 overlaid on a view of the physical environment in front of the user. In some implementations, the transparency of the optical assembly (e.g., lens) may be adjusted to suit a mixed reality application (e.g., a virtual reality application or an augmented reality application).

[0048] For example, interconnect 418 may be a system bus, or a wired or wireless network (eg, a body area network).

[0049] The head mounted display 410 includes a fastener 420. The fastener 420 may be configured to hold the head mounted display 410 in place on the user's head when the user wears the head mounted display 410. For example, the fastener 420 may include a headband. For example, the fastener 420 may include a frame having arms that rest on the user's ears and hold the lenses or screens of the display 422 in front of the user's eyes.

[0050] The computing device 450 includes a processing device 460. The processing device 460 is operable to execute instructions that have been stored in the data storage device 470. In some specific implementations, the processing device 460 is a processor with a random access memory for temporarily storing instructions read from the data storage device 470 when executing the instructions. The processing device 460 may include a single or multiple processors, each processor having a single or multiple processing cores. Alternatively, the processing device 460 may include another type of device or multiple devices capable of manipulating or processing data. For example, the data storage device 470 may be a non-volatile information storage device, such as a hard drive, a solid-state drive, a read-only memory device (ROM), an optical disk, a magnetic disk, or any other suitable type of storage device such as a non-transitory computer-readable memory. The data storage device 470 may include another type of device or multiple devices capable of storing data for retrieval or processing by the processing device 460. The processing device 460 can access and manipulate the data stored in the data storage device 470 via the interconnect 494. For example, data storage device 470 may store instructions executable by processing device 460 that, when executed by processing device 460, cause processing device 460 to perform operations (e.g., implement Figure 6 600 operations).

[0051] The computing device 450 includes a wireless communication interface 480. The wireless communication interface 480 facilitates communication with other devices including the head mounted display 410. For example, the wireless communication interface 480 may facilitate communication via a Wi-Fi network, a Bluetooth link, or a ZigBee link. The wireless communication interface 480 may be used to establish a wireless communication link 425 with the head mounted display 410. In some implementations, the wireless communication interface 480 may be used to receive sensor data (e.g., motion sensor data and / or images) from the head mounted display 410. In some implementations, the wireless communication interface 480 may be used to transmit data to the head mounted display 410 and / or invoke instructions for rendering a view of the virtual environment using the display 422.

[0052] exist Figure 4 In the example of , computing device 450 includes user interface 490. For example, user interface 490 may include a touch screen display. For example, user interface 490 may include a microphone configured to receive voice commands. For example, user interface 490 may include an image sensor configured to detect gesture commands. For example, user interface 490 may include a keyboard, a mouse, a joystick, and / or another handheld controller. For example, user interface 490 may enable a user (e.g., a user wearing head mounted display 410) to input commands to an augmented reality application running on computing device 450 and / or change the configuration of a process associated with an object of a virtual environment of the augmented reality application.

[0053] Computing device 450 includes network interface 492. Network interface 492 facilitates communication with other devices such as cloud servers. For example, network interface 492 can facilitate communication via a communication network. For example, network interface 492 can facilitate communication via a Wi-Fi network, a cellular network, and / or a wired network. For example, network interface 492 can facilitate communication via a WiMAX network. For example, network interface 492 can facilitate communication via a fiber optic network.

[0054] For example, interconnect 494 may be a system bus, or a wired or wireless network (eg, a body area network).

[0055] The processing device 460 may be configured to enable gaze-driven signal processing to facilitate mixed reality or virtual reality applications, such as mixed reality recording. For example, the processing device 460 may be configured to access gaze data captured using one or more gaze tracking sensors 412, apply a temporal filter to the gaze data to obtain a smoothed gaze estimate, and determine a region of interest based on the smoothed gaze estimate. For example, the region of interest may identify a subset of the field of view of one or more image sensors 414. For example, the processing device 460 may be configured to access a video frame captured using one or more image sensors 414, and apply signal processing to the video frame based on the region of interest to obtain an enhanced video frame. For example, the processing device 460 may be configured to record a portion of the frame associated with the region of interest as an enhanced video frame. For example, the portion of the frame may correspond to a smaller field of view than the frame. For example, the processing device 460 may be configured to encode the frame using the following method: for a first portion of the frame associated with the region of interest, a first bit per pixel allocation is used, and for one or more corresponding portions of the frame outside the first portion of the frame, one or more bit per pixel allocations that are lower than the first bit per pixel allocation are used. For example, the processing device 460 may be configured to apply computer vision processing to a portion of the frame associated with the region of interest to detect an object appearing in the portion of the frame and determine a classification of the object. For example, the portion of the frame may correspond to a smaller field of view than the frame. For example, the processing device 460 may be configured to generate annotations based on the classification and superimpose the annotations on the video frame to obtain an enhanced video frame. For example, the head mounted display 410 may be configured to display the enhanced video frame. In some specific implementations, the temporal filter includes a Kalman filter. In some specific implementations, the temporal filter includes a moving average of the gaze data in a time window. For example, the processing device 460 may be configured to apply the temporal filter to the gaze data by fitting a curve to the gaze data in the time window to obtain a smoothed gaze estimate. For example, the processing device 460 may be configured to determine the region of interest based on the smoothed gaze estimate by: applying computer vision processing to the video frame to detect and classify an object appearing in the video frame, thereby obtaining a position associated with the object in the video frame; determining a value of a cost function for the region of interest based on the smoothed gaze estimate and the position of the object; and determining the region of interest based on the value of the cost function.

[0056] In some specific implementations ( Figure 4410 and the computing device 450). In some implementations, the reality segments may be transmitted via a wireless communication link 425 between the head-mounted display 410 and the computing device 450. Thus, the processing device of the head-mounted display 410 and the processing device 460 may operate as a single processing device distributed between the head-mounted display 410 and the computing device 450 that runs the mixed reality or virtual reality application and implements the processes described herein (e.g., Figure 6 process 600).

[0057] Figure 5 5 is a block diagram of an example of a cloud server 500 configured to enable gaze-driven signal processing to facilitate mixed reality or virtual reality applications, such as mixed reality recording. The cloud server 500 includes a processing device 510, a data storage device 520, a network interface 530, a user interface 540, and an interconnect 570 that the processing device 510 can use to access other components. The cloud server 500 can be configured to determine a user's area of ​​interest based on the user's gaze data, and perform signal processing to enhance a portion of the video corresponding to the area of ​​interest. For example, the cloud server 500 can be configured to implement Figure 6 Process 600.

[0058] The processing device 510 may be used to execute instructions stored in the data storage device 520. In some specific implementations, the processing device 510 is a processor with a random access memory for temporarily storing instructions read from the data storage device 520 when executing the instructions. The processing device 510 may include a single or multiple processors, each processor having a single or multiple processing cores. Alternatively, the processing device 510 may include another type of device or multiple devices capable of manipulating or processing data. For example, the data storage device 520 may be a non-volatile information storage device, such as a hard disk drive, a solid-state drive, a read-only memory device (ROM), an optical disk, a magnetic disk, or any other suitable type of storage device such as a non-transitory computer-readable memory. The data storage device 520 may include another type of device or multiple devices capable of storing data for retrieval or processing by the processing device 510. The processing device 510 may access and manipulate the data stored in the data storage device 520 via the interconnect 570. For example, the data storage device 520 may store instructions executable by the processing device 510, which, when executed by the processing device 510, causes the processing device 510 to perform an operation (e.g., implement a Figure 6 600 operations).

[0059] The network interface 530 facilitates communication with other devices, such as the mobile computing device 300 or the head mounted display 410. For example, the network interface 530 may facilitate communication via a communication network. For example, the network interface 530 may facilitate communication via a Wi-Fi network, a cellular network, and / or a wired network. For example, the network interface 530 may facilitate communication via a WiMAX network. For example, the network interface 530 may facilitate communication via a fiber optic network.

[0060] exist Figure 5 In an example of , cloud server 500 includes user interface 540. For example, user interface 540 may include a touch screen display. For example, user interface 540 may include a keyboard, a mouse, a joystick, and / or another handheld controller. For example, user interface 540 may include a microphone configured to receive voice commands. For example, user interface 540 may include an image sensor configured to detect gesture commands. For example, user interface 540 may enable a user (e.g., a system administrator) to enter commands to cloud server 500 to configure and / or maintain a collection of mixed reality recordings.

[0061] The processing device 510 may be configured to enable gaze-driven signal processing to facilitate mixed reality or virtual reality applications, such as mixed reality recording. For example, the processing device 510 may be configured to access gaze data (e.g., a sequence of gaze points of a user detected when the user views and / or captures a video associated with a mixed reality application). For example, the processing device 510 may be configured to apply a temporal filter to the gaze data to obtain a smoothed gaze estimate, and to determine a region of interest based on the smoothed gaze estimate. For example, the region of interest may identify a subset of the field of view. For example, the processing device 510 may be configured to access a video frame (e.g., a video frame that has been displayed to the user when the gaze data of the user is collected), and to apply signal processing to the video frame based on the region of interest to obtain an enhanced video frame. For example, the processing device 510 may be configured to record a portion of the frame associated with the region of interest as an enhanced video frame. For example, the portion of the frame may correspond to a smaller field of view than the frame. For example, the processing device 510 may be configured to encode the frame using the following method: for a first portion of the frame associated with the region of interest, using a first bit per pixel allocation, and for one or more corresponding portions of the frame outside the first portion of the frame, using one or more bit per pixel allocations lower than the first bit per pixel allocation. For example, the processing device 510 may be configured to apply computer vision processing to a portion of the frame associated with the region of interest to detect an object appearing in the portion of the frame and determine a classification of the object. For example, the portion of the frame may correspond to a smaller field of view than the frame. For example, the processing device 510 may be configured to generate annotations based on the classification, and superimpose the annotations on the video frame to obtain an enhanced video frame. In some specific implementations, the temporal filter includes a Kalman filter. In some specific implementations, the temporal filter includes a moving average of the gaze data in a time window. For example, the processing device 510 may be configured to apply the temporal filter to the gaze data by fitting a curve to the gaze data in the time window to obtain a smoothed gaze estimate. For example, the processing device 510 may be configured to determine a region of interest based on a smoothed gaze estimate in the following manner: applying computer vision processing to a video frame to detect and classify an object appearing in the video frame, thereby obtaining a position associated with the object within the video frame; determining a value of a cost function for the region of interest based on the smoothed gaze estimate and the position of the object; and determining the region of interest based on the value of the cost function.

[0062] Figure 6is a flow chart of an example of a process 600 for gaze-driven video processing. The process 600 includes accessing 610 gaze data captured using one or more gaze tracking sensors; applying 620 a temporal filter to the gaze data to obtain a smoothed gaze estimate; determining 630 a region of interest based on the smoothed gaze estimate, wherein the region of interest identifies a subset of the field of view; accessing 640 a video frame; applying 650 signal processing to the video frame based on the region of interest to obtain an enhanced video frame; and storing, transmitting, or displaying 660 the enhanced video frame. For example, the process 600 may be performed by Figure 3 For example, the process 600 may be implemented by the mobile computing device 300. Figure 4 For example, the process 600 may be implemented by Figure 5 This is achieved by using the cloud server 500.

[0063] Process 600 includes accessing 610 gaze data captured using one or more gaze tracking sensors (e.g., one or more gaze tracking sensors 330 or one or more gaze tracking sensors 412). For example, the one or more gaze tracking sensors may include one or more image sensors that can detect light in various light spectra (e.g., infrared spectra or visible spectra) and / or light sources (e.g., infrared light sources irradiated at the user's eyes). For example, the one or more gaze tracking sensors may also include hardware and / or software (e.g., gaze tracking software) for processing raw image data to generate gaze data. For example, the gaze data may include a sequence of gaze points, each gaze point specifying a direction (e.g., a ray, a rotation matrix, or a quaternion) or a focus (e.g., as a coordinate in a three-dimensional space or a two-dimensional image space) of a gaze of a user being observed at a given time. In some implementations, the gaze points in the sequence are synchronized with frames of an associated video. In some implementations, the gaze points in the sequence are asynchronous with frames of an associated video, and the gaze points at times corresponding to corresponding video frames may be interpolated from nearby gaze points from the sequence (e.g., using a resampling process). In some implementations, one or more gaze tracking sensors are coupled (e.g., attached) to a display device (e.g., a head-mounted display, a tablet computer, or a smart phone). For example, one or more gaze tracking sensors may be mounted on a fastening article (e.g., a spectacle frame or goggles) of the head-mounted display so that it is directed toward the eyes of a user wearing the head-mounted display. In some implementations, the gaze direction of the user in the coordinates of the attached image sensor or display may be determined based on gaze data from one or more gaze tracking sensors. The gaze data 610 may be accessed in various ways. For example, the gaze data 610 may be accessed by reading directly from a sensor or reading from a memory via an interconnect (e.g., interconnect 370). For example, the gaze data 610 may be accessed by receiving the gaze data via a communication link (e.g., wireless communication link 425).

[0064] The process 600 includes applying a temporal filter to the gaze data to obtain a smoothed gaze estimate. For example, the temporal filter may be applied to the gaze data to reduce jitter in the region of interest to be determined 630 based on the gaze data. In some implementations, the temporal filter may be a finite impulse response filter applied to a sliding window (e.g., a window of 1 second or 0.5 second duration) of the gaze point in the gaze data. In some implementations, the duration of the window of the gaze data may be tuned to reduce jitter. For example, applying the temporal filter to the gaze data to obtain a smoothed gaze estimate may include fitting a curve to the gaze data in the time window. Other types of smoothing filters may be applied to the gaze data 620. For example, the temporal filter may include a moving average of the gaze data in the time window. For example, the temporal filter may include a Kalman filter. In some implementations, the Kalman filter may be applied 620 to predict the gaze direction at a future time.

[0065] Process 600 includes determining 630 a region of interest based on a smoothed gaze estimate. The region of interest identifies a subset of a field of view (e.g., a field of view of an array of one or more image sensors or a field of view of a video shown on a display). Once determined 630, the region of interest can be used to focus computing resources (e.g., memory, communication channel bandwidth, and / or processor cycles) on a portion of the video data corresponding to the region of interest, in preference to other portions of the available field of view. For example, determining 630 the region of interest can include converting the smoothed gaze estimate into a coordinate system of one or more image sensors and / or a display. For example, determining 630 the region of interest can include detecting and enforcing boundary conditions at the edge of the available field of view so that the region of interest does not move outside the available field of view and remains a consistent size relative to the available field of view. In some implementations, the region of interest is determined 630 as a region (e.g., a subset of pixels) within a frame of video data.

[0066] Process 600 includes accessing 640 video frames. The video frames may be captured using one or more image sensors (e.g., one or more image sensors 340 or one or more image sensors 414). In some implementations, the video frames may depict a spatial area near a user wearing a head-mounted display (e.g., head-mounted display 410) including one or more image sensors. The video frames may have been captured to generate gaze data while the user is observed using one or more gaze tracking sensors. In some implementations, the one or more image sensors are coupled (e.g., attached) to the one or more gaze tracking sensors (e.g., part of the head-mounted display). In some implementations, the video frames may depict one or more virtual objects of a virtual environment as part of a mixed reality or virtual reality application. For example, the video frame may be a view of a virtual environment rendered based on the user's head orientation data (e.g., from an inertial measurement unit) and presented to the user in a head-mounted display (e.g., VR goggles). The video frames may have been displayed to the user (e.g., using a head-mounted display or a touch screen display) while the user is observed using one or more gaze tracking sensors to generate gaze data. In some implementations, the display is coupled (e.g., attached) to one or more gaze tracking sensors (e.g., as part of a head mounted display or as part of a smartphone or tablet). For example, the video frames can be accessed 640 by reading directly from an image sensor, by reading an image from a memory or another storage device (e.g., via interconnect 370), or by receiving the video frames via a communication link with another device (e.g., wireless communication link 425 or via network interface 530).

[0067] Process 600 includes applying signal processing to 650 video frames based on a region of interest to obtain an enhanced video frame. For example, applying signal processing to 650 video frames based on a region of interest to obtain an enhanced video frame may include recording a portion of the frame associated with the region of interest as an enhanced video frame. The portion of the frame corresponds to a smaller field of view than the frame. For example, applying signal processing to 650 video frames based on a region of interest to obtain an enhanced video frame may include encoding the portion of the frame associated with the region of interest using a higher per-pixel bit allocation than other portions of the frame. For example, the frame may be encoded using the following method: for a first portion of the frame associated with the region of interest, a first per-pixel bit allocation is used, and for one or more corresponding portions of the frame outside the first portion of the frame, one or more per-pixel bit allocations that are lower than the first per-pixel bit allocation are used. For example, applying signal processing to 650 video frames based on a region of interest to obtain an enhanced video frame may include applying computer vision processing to the portion of the frame associated with the region of interest and enhancing or modifying the video frame based on detection and / or classification of objects appearing in the region of interest (e.g., adding metadata to a data structure including annotations in the video frame or overlaying the video frame). For example, Figure 7 The process 700 can be implemented as applying 650 signal processing to video frames based on the region of interest to obtain enhanced video frames. In some implementations, computer vision processing is applied 650 to portions of frames in the sequence of video frames associated with the corresponding region of interest to search the video for times when classified objects appear in the current region of interest (e.g., times when the user may have seen these objects of interest).

[0068] Process 600 includes storing, transmitting, or displaying 660 the enhanced video frame. For example, the enhanced video frame may be stored 660 in a memory or other data storage device (e.g., data storage device 320, data storage device 470, or data storage device 520). For example, the enhanced video frame may be transmitted 660 to another device via a communication interface (e.g., network interface 360, network interface 492, or network interface 530). For example, the enhanced video frame may be displayed 660 using a screen, a projector, or some other type of display (e.g., display 350, display 422, or user interface 540).

[0069] Figure 7 7 is a flow chart of an example of a process 700 for annotating an image based on an object in a gaze-driven region of interest. The process 700 includes applying 710 computer vision processing to a portion of a frame associated with the region of interest to detect objects appearing in the portion of the frame and determine a classification of the objects; generating 720 annotations based on the classifications; and overlaying 730 the annotations on the video frame to obtain an enhanced video frame. For example, the process 700 may be performed by Figure 3 For example, process 700 may be performed by a mobile computing device 300. Figure 4 For example, process 700 may be implemented by Figure 5 This is achieved by using the cloud server 500.

[0070] Process 700 includes applying computer vision processing to a portion of a frame 710 associated with a region of interest to detect objects that appear in the portion of the frame and determine a classification of the object. The portion of the frame corresponds to a smaller field of view than the frame. For example, applying computer vision processing to the portion of the frame 710 associated with the region of interest may include inputting pixel values ​​from the portion into a convolutional neural network that has been trained to detect and classify objects that appear in an image that is equivalent to the size of the portion. The convolutional neural network may output a classification of the detected object. In some specific implementations, one or more transforms (e.g., a Fourier transform or a wavelet transform) may be applied to the portion of the frame, and the resulting transform coefficients may be input to the neural network. For example, the classification may include a vector of class probabilities.

[0071] The method 700 includes generating 720 an annotation based on the classification. For example, the annotation may include text or another symbol associated with a class identified by a computer vision system used to determine the classification. For example, the annotation may be a class label of a class of an object detected in a portion of the frame associated with the region of interest. For example, the annotation may correspond to the class with the largest probability value in the classification, where the classification is a vector of class probabilities.

[0072] The process 700 includes superimposing 730 the annotation on the video frame to obtain an enhanced video frame. For example, the text of the annotation can be superimposed 730 on the video frame in a mode of writing text onto an image by replacing some of the pixel values ​​of the video frame with a color. In some specific implementations, the annotation can be semi-transparent, and the annotation can be superimposed 730 in a mode of forming the annotation by adding an offset to the pixel values ​​of the video frame.

[0073] Figure 8 8 is a flow chart of an example of a process 800 for determining a region of interest based on a combination of gaze data and saliency information based on an identified object. The process 800 includes applying 810 computer vision processing to a video frame to detect and classify an object that appears in the video frame to obtain a location associated with the object within the video frame; determining 820 a value of a cost function for the region of interest based on the smoothed gaze estimate and the location of the object; and determining 830 the region of interest based on the value of the cost function. For example, the process 800 may be performed by Figure 3 For example, process 800 may be implemented by a mobile computing device 300. Figure 4 For example, process 800 may be implemented by Figure 5 This is achieved by using the cloud server 500.

[0074] Process 800 includes applying computer vision processing to 810 a video frame to detect and classify an object that appears in the video frame, thereby obtaining a position associated with the object within the video frame. For example, applying computer vision processing to 810 a video frame may include inputting pixel values ​​from the frame into a convolutional neural network that has been trained to detect and classify objects that appear in an image of a size equivalent to the frame. The convolutional neural network may output a classification of the detected object. In some implementations, one or more transforms (e.g., a Fourier transform or a wavelet transform) may be applied to the frame, and the resulting transform coefficients may be input to the neural network. For example, the classification may include a vector of class probabilities. The convolutional neural network may output an indication of the position of the detected object within the video frame. For example, a block index associated with a pixel block within the frame in which the object has been detected may be output by the convolutional neural network. In some implementations, overlapping blocks at multiple scales and positions are processed by the neural network to detect and / or classify the object. For example, the position may be a pair of pixel offsets associated with a block in which the object has been detected (e.g., at the center of the block).

[0075] Process 800 includes determining 820 a value of a cost function for a region of interest based on a smoothed gaze estimate and a location of an object. The cost function can be used to combine multiple objectives for determining a region of interest. For example, the cost function can include a term based on a distance between a gaze estimate (e.g., a smoothed gaze estimate) and a point (e.g., a center point) within a candidate region of interest. For example, the cost function can include a term based on a distance between a location of an object (e.g., a center of a block in which the object is detected) and a point (e.g., a center point) within a candidate region of interest. For example, the cost function can include a linear combination of a term based on a distance between a gaze estimate and the region of interest and a term based on a distance between a location of the object and the region of interest.

[0076] The process 800 includes determining 830 a region of interest based on a value of the cost function. For example, the region of interest may be determined 830 by selecting a candidate region of interest having a minimum associated value of the cost function.

[0077] The above disclosed implementation or part of the implementation may take the form of a computer program product, which can be accessed from, for example, a computer usable medium or a computer readable medium. A computer usable medium or a computer readable medium can be, for example, any device that can tangibly contain, store, transmit or transfer a program or data structure for use or connection with any processor. The medium can be, for example, an electronic device, a magnetic device, an optical device, an electromagnetic device, or a semiconductor device. Other suitable media may also be used. Such computer usable media or computer readable media may be referred to as non-transitory memory or media, and may include RAM or other volatile memory or storage devices that may change over time.

[0078] The physical environment refers to the physical world that people can sense and / or interact with without the help of electronic systems. A physical environment, such as a physical park, includes physical objects, such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0079] In contrast, a computer generated reality (CGR) environment refers to a fully or partially simulated environment that a person perceives and / or interacts with via an electronic system. In CGR, a subset of a person's physical movements, or representations thereof, are tracked, and in response, one or more features of one or more virtual objects simulated in the CGR environment are adjusted in a manner that complies with at least one law of physics. For example, a CGR system may detect a person's head turning, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), adjustments to features of virtual objects in a CGR environment may be made in response to representations of physical movement (e.g., voice commands).

[0080] A person may sense and / or interact with a CGR object using any of their senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with an audio object that creates a 3D or spatial audio environment that provides a perception of a point audio source in 3D space. As another example, an audio object may enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person may sense and / or interact only with an audio object.

[0081] Examples of CGR include virtual reality and mixed reality.

[0082] A virtual reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input for one or more senses. A VR environment includes a plurality of virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing avatars of people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence within the computer-generated environment and / or through a simulation of a subset of the person's physical movement within the computer-generated environment.

[0083] In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, a mixed reality environment is anything between a fully physical environment at one end and a virtual reality environment at the other end, but not including both ends.

[0084] In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render the MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment or representations thereof). For example, the system can cause movement so that virtual trees appear stationary relative to the physical ground.

[0085] Examples of mixed reality include augmented reality and augmented virtuality.

[0086] An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive virtual objects superimposed on a physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of a physical environment that are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on an opaque display. People use the system to indirectly view the physical environment via an image or video of the physical environment and perceive virtual objects superimposed on the physical environment. As used herein, a video of a physical environment displayed on an opaque display is referred to as a "transparent video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting an AR environment on an opaque display. Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, for example as a hologram or on a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment.

[0087] An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, a representation of a physical environment may be transformed by graphically modifying (e.g., enlarging) portions thereof so that the modified portions may be representative but not true versions of the original captured images. For another example, a representation of a physical environment may be transformed by graphically eliminating portions thereof or blurring portions thereof.

[0088] An augmented virtual (AV) environment refers to a simulated environment in which a virtual or computer-generated environment is combined with one or more sensory inputs from a physical environment. The sensory input may be a representation of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the faces of people are realistically reproduced from images taken of physical people. For another example, a virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, a virtual object may take a shadow that conforms to the positioning of the sun in the physical environment.

[0089] There are many different types of electronic systems that enable people to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on people's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smart phones, tablets, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smart phone). A head-mounted system can incorporate one or more imaging sensors for capturing images or videos of a physical environment and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system can have a transparent or translucent display. A transparent or translucent display can have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0090] Entities implementing the above techniques for recording CGR environments are reminded to comply with established industry or government privacy requirements. For example, entities providing CGR recording features are advised to begin recording in response to a specific instruction from a user and to identify the destination of such recordings to the user. It is also recommended that these entities take steps to remind users that when recordings are transmitted beyond their local devices (e.g., uploaded to social media sites, etc.), they may be processed and / or viewed by Internet users and / or Internet entities. The possibility cannot be ruled out that if a malicious entity obtains a CGR recording, the entity may attempt to reverse engineer where the user is looking during a particular CGR session. However, it should be noted that metadata indicating such information does not need to be saved to implement the above techniques. In addition, specific implementations may further obscure the user's precise gaze, for example by changing the applied time filter from time to time. It is important that personally identifiable information data (if any) should be processed to minimize the risk of unauthorized and / or unintentional access and / or use. The nature of any authorized use should also be clearly indicated to the user.

[0091] As described above, one aspect of the present technology is to collect and use data available from various sources to improve computer-generated reality experiences. The present disclosure contemplates that, in some instances, these collected data may include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, twitter IDs, home addresses, data or records related to the user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying information or personal information.

[0092] The present disclosure recognizes that the use of such personal information data in the present technology can be used to benefit the user. For example, the personal information data can be used to better identify objects of interest to the user within the field of view. In addition, the present disclosure also anticipates other uses of the personal information data that benefit the user. For example, health and fitness data can be used to provide insights into the user's overall health or can be used as positive feedback to individuals using technology to pursue health goals.

[0093] This disclosure envisions that entities responsible for collecting, analyzing, disclosing, transmitting, storing or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be easily accessible to users and should be updated as the collection and / or use of data changes. Personal information from users should be collected for legitimate and reasonable purposes of the entity and not shared or sold outside of these legitimate uses. In addition, such collection / sharing should be performed after receiving informed consent from the user. In addition, such entities should consider taking any necessary steps to defend and secure access to such personal information data and ensure that others who have access to personal information data comply with their privacy policies and processes. In addition, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. In addition, policies and practices should be adjusted to specific types of personal information data collected and / or accessed, and to applicable laws and standards including specific considerations of jurisdiction. For example, in the United States, the collection or access of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA), while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy practices should be maintained in each country for different types of personal data.

[0094] Regardless of the foregoing, the present disclosure also contemplates implementation schemes in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates that hardware elements and / or software elements may be provided to prevent or block access to such personal information data. For example, with respect to an advertising delivery service, the technology of the present invention may be configured to allow a user to choose to "opt in" or "opt out" to participate in the collection of personal information data at any time during or after registration for the service. In addition to providing "opt-in" and "opt-out" options, the present disclosure contemplates providing notifications related to access or use of personal information. For example, a user may be notified that their personal information data will be accessed when downloading an application, and then be reminded again just before the personal information data is accessed by the application.

[0095] Furthermore, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data once it is no longer needed. Additionally, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be facilitated by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at a city level rather than at an address level), controlling how data is stored (e.g., aggregating data across users), and / or other methods, where appropriate.

[0096] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without access to such personal information data. That is, the various embodiments of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data. For example, objects of interest may be identified based on non-personal information data or an absolute minimum amount of personal information, such as content requested by a device associated with a user, other non-personal information suitable for a computer-generated reality system, or publicly available information).

Claims

1. A method comprising: Obtain one or more video frames; processing the one or more video frames according to the first set of image adjustments; detecting a region of interest of the one or more video frames; as well as The one or more video frames are processed according to a second set of image adjustments configured to enhance the detected region of interest.

2. The method of claim 1, wherein the first set of image adjustments comprises passing extracted portions from the one or more video frames to an encoder for compression in preparation for storage and / or transmission of the still view video. The method of claim 2 , wherein the extracted portion has a smaller field of view than the one or more video frames.

4. The method of claim 1, wherein the first set of image adjustments comprises passing gaze-driven portions extracted from the one or more video frames to an encoder for compression in preparation for storage and / or transmission of the jittered view video. 5 . The method of claim 4 , wherein the region of interest is detected based on gaze data captured by one or more gaze tracking sensors, the gaze data corresponding in time to the one or more video frames.

6. The method of claim 5, wherein the second set of image adjustments comprises applying a smoothing or stabilization process to the one or more video frames based on the gaze data to reduce jitter in the detected region of interest.

7. The method of claim 1, wherein the first set of image adjustments comprises encoding a first portion of one or more frames outside of the detected region of interest using a first bit-per-pixel allocation.

8. The method of claim 7, wherein the second set of image adjustments comprises encoding a second portion of one or more frames associated with the detected region of interest using a second bits-per-pixel allocation.

9. The method of claim 8, wherein the second portion has a smaller field of view than the first portion.

10. The method of claim 8, wherein the second bits per pixel allocation is higher than the first bits per pixel allocation to enhance the detected region of interest.

11. A system comprising: one or more gaze tracking sensors; one or more image sensors; as well as A processing device, the processing device being configured to: obtaining a location of an object appearing in a video frame captured by the one or more image sensors; determining a region of interest, the region of interest identifying a portion of the video frame corresponding to a subset of a field of view of the one or more image sensors, wherein the region of interest is based on the position of the object and gaze data captured by the one or more gaze tracking sensors, the gaze data corresponding in time to the video frame; as well as Signal processing is applied to the portion of the video frame based on the region of interest to obtain an enhanced video frame.

12. The system of claim 11, wherein the processing device is configured to apply signal processing to the portion of the video frame based on the region of interest to obtain the enhanced video frame by performing operations comprising: The portion of the frame associated with the region of interest is recorded as the enhanced video frame, wherein the portion of the frame corresponds to a smaller field of view than the frame.

13. The system of claim 11, wherein the processing device is configured to apply signal processing to the video frame based on the region of interest to obtain the enhanced video frame by performing operations comprising: encoding the portion of the frame associated with the region of interest using a first bits-per-pixel allocation; and One or more corresponding portions of the frame other than the portion of the frame associated with the region of interest are encoded using one or more bit-per-pixel allocations lower than the first bit-per-pixel allocation.

14. The system of claim 11, wherein the processing device is configured to apply signal processing to the portion of the video frame based on the region of interest to obtain the enhanced video frame by performing operations comprising: Computer vision processing is applied to the portion of the frame associated with the region of interest to detect a second object appearing in the portion of the frame, and to determine a classification of the second object, wherein the portion of the frame corresponds to a smaller field of view than the frame.

15. The system of claim 14, wherein the processing device is configured to apply signal processing to the portion of the video frame based on the region of interest to obtain the enhanced video frame by performing operations comprising: generating an annotation based on the classification of the second object; and The annotation is superimposed on the portion of the video frame to obtain the enhanced video frame.

16. The system according to claim 11, comprising: A head mounted display is coupled to the one or more gaze tracking sensors and the one or more image sensors, wherein the head mounted display is configured to display the augmented video frame.

17. The system of claim 11, wherein the processing device is configured to: A temporal filter is applied to the gaze data to obtain a smoothed gaze estimate, wherein the region of interest is determined based on the smoothed gaze estimate.

18. The system of claim 17, wherein the temporal filter comprises a Kalman filter.

19. The system of claim 17, wherein the temporal filter comprises a moving average of the gaze data over a temporal window.

20. The system of claim 17, wherein the processing device is configured to apply the temporal filter to the gaze data to obtain the smoothed gaze estimate by performing operations comprising: A curve was fit to the fixation data in a time window.

Images