Energy-efficient context-dependent processing for content

By using trained models to enhance or scale content data in a mixed reality environment, the resolution and processing capability limitations caused by power constraints of electronic devices are solved, and more efficient content capture and transmission are achieved, improving the quality of the mixed reality experience.

CN118974686APending Publication Date: 2024-11-15APPLE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380031839.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-08-16
Filing Date
2023-03-17
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Electronic devices used for presentation in mixed reality environments limit the available resolution and processing power of the sensor due to power constraints, resulting in higher power consumption and delays in capturing content data.

Method used

The context-aware mode of the trained model enhances or scales the content data captured by the sensor, reduces the sensor resolution or compresses the data, and reduces the data size, thereby reducing power consumption and delay.

Benefits of technology

By reducing sensor resolution and data size, power is saved for capture and transmission, improving the processing power of electronic devices and the quality of mixed reality experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118974686A_ABST
    Figure CN118974686A_ABST
Patent Text Reader

Abstract

A method is provided that includes receiving content data captured by a sensor and receiving a context signal representing a context of a user. The received content data is scaled using a trained model, where the context signal is an input to the trained model, and the scaled content data is provided for presentation to a user.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 325,087, filed on March 29, 2022, entitled “Energy Efficient Context-Related Processing for Content,” the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present application generally relates to electronic devices, including, for example, electronic devices for presenting mixed reality experiences. Background Art

[0004] A mixed reality environment may refer to a simulated environment designed to combine sensory input from a physical environment. Electronic devices used to present a mixed reality environment rely on sensors such as cameras and microphones to capture content data from the physical environment. Many of these electronic devices have power constraints that limit the amount of processing that can be performed on the device. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Certain features of the subject technology are set forth in the appended claims. However, for purposes of explanation, several embodiments of the subject technology are set forth in the following figures.

[0006] Figure 1 An example environment is shown in which the subject technology may operate according to one or more implementations.

[0007] Figure 2 is a block diagram illustrating components of an electronic device in accordance with one or more implementations of the subject technology.

[0008] Figure 3 is a block diagram illustrating a process flow in accordance with aspects of the subject technology.

[0009] Figure 4 An example process for processing content data in accordance with aspects of the subject technology is shown.

[0010] Figure 5 An example electronic system is shown with which aspects of the subject technology can be implemented. DETAILED DESCRIPTION

[0011] The specific description shown below is intended to be a description of various configurations of the subject technology and is not intended to represent the only configuration that the subject technology can be practiced. The accompanying drawings are incorporated herein and constitute a part of the detailed description. The detailed description includes specific details intended to provide a thorough understanding of the subject technology. However, the subject technology is not limited to the specific details described herein, and one or more other specific implementations can be used to practice. In one or more specific implementations, structures and components are shown in block diagram form to avoid blurring the concepts of the subject technology.

[0012] A mixed reality environment may refer to a simulated environment designed to combine sensory input from a physical environment. Electronic devices used to present a mixed reality environment rely on sensors such as cameras and microphones to capture content data from the physical environment. The power consumed to capture the content data is proportional to the resolution of the image and audio signals captured using the camera and microphone of the electronic device. In addition, post-processing may be performed on the captured content data to remove artifacts and improve quality. However, many of these electronic devices have power constraints that may limit the available resolution of the camera and / or microphone and the amount of processing that can be performed on the device.

[0013] The subject technology uses a trained model to enhance or scale content data captured by a sensor in a context-aware manner. In addition to the content data, the input to the trained model also includes one or more context signals. The context signal includes information about the user context, which is used by the trained model to enhance or scale the captured content data. Using a trained model in this manner allows the size of the captured content data to be reduced by reducing the resolution of the sensor used to capture the content data or by compressing the content data before sending the captured content data to another electronic device for processing. Reducing the resolution of the sensor reduces the amount of power consumed by the sensor when capturing the content data. The reduced size of the captured content data reduces the amount of power required to send the captured content data to another electronic device, and reduces the latency of sending the captured content data because there is less data to send. Other benefits and advantages will become apparent in the following description that describes the subject technology in more detail.

[0014] Figure 1 An example environment in which the subject technology may operate according to one or more implementations is shown. However, not all of the components depicted may be used in all implementations, and one or more implementations may include additional or different components than those shown in the figures. Changes in the arrangement and type of these components may be made without departing from the spirit or scope of the claims listed herein. Additional components, different components, or fewer components may be provided.

[0015] like Figure 1As shown in , environment 100 includes electronic device 110 and electronic device 120. Electronic device 110 is an electronic device configured to present a mixed reality experience to a user of the device. Electronic device 110 may be a smart phone, a tablet, a head mounted device (HMD), etc. Electronic device 110 includes input sensors configured to capture content data from the user's physical environment for processing to present a mixed reality experience. Input sensors include, but are not limited to, camera 130, microphone 140, depth sensor 150, proximity sensor 160, gyroscope 170, and accelerometer 180. Although Figure 1 Only one instance of each of these input sensors is depicted in , but the electronic device 110 may have multiple instances of one or more of the various input sensors. For example, the electronic device 110 may have multiple cameras configured with different viewpoints and / or with different capabilities (e.g., color, black and white, high resolution, etc.). Similarly, the electronic device 110 may have multiple microphones or microphone arrays configured to capture sound from different directions relative to a user of the device. Although Figure 1 Not depicted in FIG. 1 , but the electronic device 110 also includes a processing component for managing the operation of the electronic device 110 and its various components, a display component for presenting visual content to a user, and an audio output component for presenting audio content to a user.

[0016] The electronic device 120 is a computing device that wirelessly communicates with the electronic device 110. The electronic device 120 is configured to assist the electronic device 110 in processing tasks associated with presenting a mixed reality experience, which are offloaded from the electronic device 110 to the electronic device 120. The processing tasks may be offloaded to save power on the electronic device 110 and / or to utilize the processing power available on the electronic device 120. The wireless communication between the electronic device 110 and the electronic device 120 may utilize any wireless technology, including but not limited to Wi-Fi, Bluetooth, ultra-wideband (UWB), etc. The electronic device 120 is depicted as a laptop computer. However, the electronic device 120 may be implemented using other types of devices, including but not limited to a desktop computer, a tablet device, or any device capable of executing one or more computer programs to provide processing that supports presenting a mixed reality experience on the electronic device 110 or capable of performing other processing tasks offloaded from the electronic device 110.

[0017] Figure 2 is a block diagram illustrating components of an electronic device according to one or more implementations of the subject technology. However, not all of the components depicted may be used in all implementations, and one or more implementations may include additional or different components than those shown in the figures. Changes in the arrangement and type of these components may be made without departing from the spirit or scope of the claims listed herein. Additional components, different components, or fewer components may be provided.

[0018] exist Figure 2 In the depicted example, the electronic device 120 includes a processor 210 and a memory 220. The processor 210 may include appropriate logic components, circuits, and / or code to enable processing of data and / or control of the operation of the electronic device 120. In this regard, the processor 210 may be enabled to provide control signals to various other components of the electronic device 120. The processor 210 may also control the transfer of data between the various parts of the electronic device 120. In addition, the processor 210 may enable implementation of an operating system or otherwise execute code to manage the operation of the electronic device 120.

[0019] Processor 210 or one or more portions thereof may be implemented in software (e.g., instructions, subroutines, and code), hardware (e.g., application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gated logic components, discrete hardware components, or any other suitable devices), and / or a combination of both.

[0020] The memory 220 may include appropriate logic components, circuits, and / or code to enable storage of various types of information, such as received data, generated data, code, and / or configuration information. The memory 220 may include, for example, random access memory (RAM), read-only memory (ROM), flash memory, and / or magnetic storage. Figure 2 As depicted, memory 220 contains experience module 230, model 240, and algorithm 250. The subject technology is not limited to the number and type of components described above, and may use more than Figure 2 More or fewer components than depicted may be implemented.

[0021] According to aspects of the subject technology, experience module 230 includes a computer program having one or more instruction or code sequences and associated data and settings. When executing these instructions or codes, one or more processes are initiated to manage communications with electronic device 110 and support the use of electronic device 110 to present a mixed reality experience. Supporting the presentation of a mixed reality experience may include receiving content data captured using one or more sensors on electronic device 110, enhancing or scaling the content data using a trained model, providing the content data to an algorithm for further processing, and / or providing the content data for presentation to a user. These operations will be described in further detail below.

[0022] According to various aspects of the subject technology, model 240 includes a computer program having one or more instructions or code sequences and associated data and settings. When executing these instructions or codes, one or more processes are initiated to provide a machine learning model that is trained to enhance or scale content data based on context signals. Content can be video data such as image frames or audio data. Enhancing or scaling content data can include increasing the resolution of content data or a portion of content data from a first resolution to a second resolution greater than the first resolution. The increased level of resolution and / or the enhanced or scaled portion of content data can be based on the context signal received as input to model 240. Enhancing content data can also include changing the format of content data, such as color schemes (e.g., 4:4:4, 4:2:2, etc.). Enhancing content data can also include providing missing information in content data. For example, the visual content behind an object in the current view of a scene can be provided based on historical data or context learned from previous images of different viewpoints of the scene.

[0023] The context signal provides information about the context of the user who is participating in the mixed reality experience using the electronic device 110. The context signal may also include specific user contexts, such as areas of interest in the content data, user gaze, user head position, user trajectory, etc. The context signal may also include time information, such as motion vectors indicating the direction of movement of the user or objects in the scene, the marked positions of objects in the scene over time, etc. The context signal may also include information about the type of content consumed in the mixed reality experience. For example, the content type may include participating in a live event, watching an event or movie on a screen, taking a nature walk, etc. The mixed reality experience of taking a nature walk may be improved by providing more resolution to the area of ​​interest that the user may be watching so that the user can see the details of plants or animals, while watching sports events can be enjoyed without emphasizing any part of the captured content data. The context signal may also include an experience goal. For example, a co-presentation experience with other users may focus on audio / video data streamed from other participants in the experience. A collaborative experience may focus on an object or document being processed by a group of users. The subject technology is not limited to the examples of the above-mentioned context signals, and other types of context signals may be used to implement it.

[0024] Model 240 may be trained using a data set including content data (e.g., visual data and / or audio data) processed for different types of context signals. Model 240 may be a single model trained for multiple different types of context signals. Alternatively, model 240 may represent a group of models that are each trained for a corresponding context signal type or group of context signal types. In this alternative arrangement, experience module 230 may select a trained model from the group of models based on a context signal associated with a mixed reality experience running on electronic device 110.

[0025] Algorithm 250 includes a computer program having one or more instructions or code sequences and associated data and settings. When the instructions or codes are executed, one or more processes are initiated to further process the enhanced or scaled content data output by model 240. Algorithm 250 may represent a computer vision algorithm configured to determine scene geometry, track objects, classify objects, provide alternative or peripheral views of scenes, etc. Algorithm 250 may also represent a video and / or audio processing algorithm configured to present the enhanced or scaled content data to the user in a better or different form. Algorithm 250 may represent a single algorithm or a group of algorithms selected by experience module 230 based on the running mixed reality experience. If algorithm 250 has a preferred data format, the identifier of algorithm 250 may be a context signal that model 240 places the enhanced or scaled content data in the preferred data format.

[0026] Figure 3 is a block diagram illustrating a process flow according to various aspects of the subject technology. In the depicted example, image 310 represents content data captured by a sensor (e.g., camera 130) of electronic device 110. In this example, electronic device 110 uses trained model 320 to scale down image 310 by compressing image 310 in a context-aware manner. Trained model 320 receives as input a context signal, wherein the context signal identifies a region of interest (ROI) within image 310. Trained model 320 compresses image 310 to generate compressed image 330. Based on the context signal, trained model 320 may retain most or all of the details within the ROI of image 310 while compressing data of image 310 outside the ROI.

[0027] Compressed image 330 is sent to an electronic device (e.g., electronic device 120) via wireless connection 340. Wireless connection 340 may use any wireless technology, such as Wi-Fi, Bluetooth, UWB, etc. When sent over wireless connection 340, the reduced size of compressed image 330 relative to image 310 saves power and reduces latency.

[0028] Trained model 350 is used to scale up compressed image 330 to generate image 360. Trained model 350 receives a context signal similar to the context signal received by trained model 320, which identifies a ROI within compressed image 330. Based on the context signal, trained model 350 preserves details within the ROI when generating image 360. Image 360 ​​may be the same as image 310, or image 360 ​​may be enhanced or scaled differently than image 310 based on the context signal received by trained model 350.

[0029] exist Figure 3 In the depicted example, trained model 320 is used to compress the size of image 310 before image 310 is sent as compressed image 330. In other examples, the resolution of the image sensor used to capture the image content data can be adjusted to generate an image with a lower resolution. The lower resolution image can be sent to the electronic device and then scaled up by trained model 350 based on the context signal received as input to trained model 350. This example saves additional power due to the lower resolution used by the image sensor.

[0030] Figure 4 An example process for processing content data according to various aspects of the subject technology is shown. For illustrative purposes, the frames of process 400 are described herein as occurring sequentially or linearly. However, multiple frames of process 400 may occur in parallel. In addition, the frames of process 400 need not be performed in the order shown, and / or one or more frames of process 400 need not be performed and / or may be replaced by other operations.

[0031] The example process 400 may be initiated upon receiving content data captured by a sensor (block 410). For example, the electronic device 110 may send content data captured using the camera 130 to the electronic device 120. One or more context signals representing a user context may be received (block 420). The context signal may come from a sensor on the electronic device 110, from the experience module 230, or from any component involved in the mixed reality experience presented on the electronic device 110. The received content data is scaled using a trained model and one or more context signals as input to the trained model (block 430). The scaled content data may be provided for presentation to the user (block 440). For example, the scaled content data may be sent back to the electronic device 110 for presentation, or may first be processed by a processing algorithm such as an image processing algorithm and then provided for display to the user.

[0032] Figure 5 An electronic system 500 is shown that can be utilized to implement one or more implementations of the subject technology. The electronic system 500 may be Figure 1The electronic device 120 shown and / or may be a part thereof. The electronic system 500 may include various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 500 includes a bus 508, one or more processing units 512, a system memory 504 (and / or cache), a ROM 510, a permanent storage device 502, an input device interface 514, an output device interface 506, and one or more network interfaces 516, or a subset and variant thereof.

[0033] The bus 508 generally represents all system buses, peripheral busses, and chipset buses that communicatively connect the many internal devices of the electronic system 500. In one or more implementations, the bus 508 communicatively connects one or more processing units 512 with the ROM 510, the system memory 504, and the permanent storage device 502. The one or more processing units 512 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes disclosed in the present subject matter. In different implementations, the one or more processing units 512 can be a single processor or a multi-core processor.

[0034] ROM 510 stores static data and instructions required by one or more processing units 512 and other modules of electronic system 500. On the other hand, permanent storage device 502 can be a read-write memory device. Permanent storage device 502 can be a non-volatile memory unit that stores instructions and data even when electronic system 500 is turned off. In one or more specific implementations, a mass storage device (such as a magnetic disk or optical disk and its corresponding disk drive) can be used as permanent storage device 502.

[0035] In one or more implementations, a removable storage device (such as a floppy disk, a flash drive, and its corresponding disk drive) may be used as the permanent storage device 502. Like the permanent storage device 502, the system memory 504 may be a read-write memory device. However, unlike the permanent storage device 502, the system memory 504 may be a volatile read-write memory, such as a random access memory. The system memory 504 may store any of the instructions and data that one or more processing units 512 may need at runtime. In one or more implementations, the processes disclosed in the subject matter are stored in the system memory 504, the permanent storage device 502, and / or the ROM 510. The one or more processing units 512 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes of one or more implementations.

[0036] The bus 508 is also connected to an input device interface 514 and an output device interface 506. The input device interface 514 enables a user to transmit information and select commands to the electronic system 500. Input devices that can be used with the input device interface 514 may include, for example, an alphanumeric keyboard and a pointing device (also referred to as a "cursor control device"). The output device interface 506 may, for example, enable the display of images generated by the electronic system 500. Output devices that can be used with the output device interface 506 may include, for example, a printer and a display device, such as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a flexible display, a flat panel display, a solid-state display, a projector, or any other device for outputting information. One or more specific implementations may include a device that acts as both an input device and an output device, such as a touch screen. In these specific implementations, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input.

[0037] Finally, if Figure 5 As shown, bus 508 also couples electronic system 500 to one or more networks and / or to one or more network nodes, such as Figure 1 1. In this manner, electronic system 500 may be part of a computer network, such as a LAN, a wide area network ("WAN"), or an intranet, or may be part of a network of networks, such as the Internet. Any or all components of electronic system 500 may be used with the subject disclosure.

[0038] Implementations within the scope of the present disclosure may be implemented in part or in whole using a tangible computer-readable storage medium (or multiple tangible computer-readable storage media of one or more types) encoded with one or more instructions. The tangible computer-readable storage medium may also be non-transitory in nature.

[0039] Computer-readable storage media can be any storage media that can be read, written, or otherwise accessed by a general or special computing device, including any processing electronics and / or processing circuitry capable of executing instructions. For example, without limitation, computer-readable media can include any volatile semiconductor memory, such as RAM, DRAM, SRAM, T-RAM, Z-RAM, and TTRAM. Computer-readable media can also include any non-volatile semiconductor memory, such as ROM, PROM, EPROM, EEPROM, NVRAM, flash memory, nvSRAM, FeRAM, FeTRAM, MRAM, PRAM, CBRAM, SONOS, RRAM, NRAM, racetrack memory, FJG, and Millipede memory.

[0040] In addition, the computer-readable storage medium may include any non-semiconductor memory, such as optical disk storage, magnetic disk storage, magnetic tape, other magnetic storage devices, or any other medium capable of storing one or more instructions. In one or more specific implementations, the tangible computer-readable storage medium may be directly coupled to the computing device, while in other specific implementations, the tangible computer-readable storage medium may be indirectly coupled to the computing device, for example, via one or more wired connections, one or more wireless connections, or any combination thereof.

[0041] Instructions can be directly executable, or can be used to develop executable instructions. For example, instructions can be implemented as executable or non-executable machine code, or can be implemented as high-level language instructions that can be compiled to produce executable or non-executable machine code. In addition, instructions can also be implemented as data, or can include data. Computer executable instructions can also be organized in any format, including routines, subroutines, programs, data structures, objects, modules, applications, applets, functions, etc. As recognized by those skilled in the art, the details including but not limited to the number, structure, sequence and organization of instructions can be significantly different without changing the underlying logic, function, processing and output.

[0042] Although the above discussion mainly involves microprocessors or multi-core processors that execute software, one or more implementations are performed by one or more integrated circuits such as ASICs or FPGAs. In one or more implementations, such integrated circuits execute instructions stored on the circuits themselves.

[0043] Those skilled in the art will recognize that the various illustrative frames, modules, elements, parts, methods and algorithms described herein can be implemented as electronic hardware, computer software or a combination of the two. In order to illustrate this interchangeability of hardware and software, various illustrative frames, modules, elements, parts, methods and algorithms have been generally described above according to functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. The technician can implement the described functionality in different ways for each specific application. Various parts and frames can be arranged differently (e.g., arranged in different orders, or divided in different ways) without departing from the scope of the subject technology.

[0044] The head-mountable device can be worn by a user to display visual information within the user's field of view. The head-mountable device can be used as a virtual reality (VR) system, an augmented reality (AR) system, and / or a mixed reality (MR) system. The user can observe the output provided by the head-mountable device, such as visual information provided on a display. The display can optionally allow the user to observe the environment outside the head-mountable device. Other outputs provided by the head-mountable device may include speaker output and / or tactile feedback. The user can further interact with the head-mountable device by providing input for processing by one or more components of the head-mountable device. For example, while the device is mounted to the user's head, the user can provide tactile input, voice commands, and other inputs.

[0045] The physical environment refers to the physical world that people can sense and / or interact with without the help of electronic systems. A physical environment, such as a physical park, includes physical objects, such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0046] In contrast, a computer generated reality (CGR) environment refers to a fully or partially simulated environment that a person perceives and / or interacts with via an electronic system. In CGR, a subset of a person's physical movements, or representations thereof, are tracked, and in response, one or more features of one or more virtual objects simulated in the CGR environment are adjusted in a manner that complies with at least one law of physics. For example, a CGR system may detect a person's head turning, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), adjustments to the characteristics of virtual objects in a CGR environment may be made in response to representations of physical movement (e.g., voice commands).

[0047] A person may sense and / or interact with a CGR object using any of their senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with an audio object that creates a 3D or spatial audio environment that provides a perception of a point audio source in 3D space. As another example, an audio object may enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person may sense and / or interact only with an audio object.

[0048] Examples of CGR include virtual reality and mixed reality. A virtual reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input for one or more senses. A VR environment includes a plurality of virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing avatars of people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence within the computer-generated environment and / or through a simulation of a subset of the person's physical movement within the computer-generated environment.

[0049] In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, a mixed reality environment is anything between a fully physical environment at one end and a virtual reality environment at the other end, but not including both ends.

[0050] In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render the MR environment can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment or representations thereof). For example, the system can cause movement so that virtual trees appear stationary relative to the physical ground.

[0051] Examples of mixed reality include augmented reality and augmented virtuality.

[0052] An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive virtual objects superimposed on a physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of a physical environment that are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on an opaque display. People use the system to indirectly view the physical environment via an image or video of the physical environment and perceive virtual objects superimposed on the physical environment. As used herein, a video of a physical environment displayed on an opaque display is referred to as a "transparent video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting an AR environment on an opaque display. Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, for example as a hologram or on a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment.

[0053] An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, a representation of a physical environment may be transformed by graphically modifying (e.g., enlarging) portions thereof so that the modified portions may be representative but not true versions of the original captured images. For another example, a representation of a physical environment may be transformed by graphically eliminating portions thereof or blurring portions thereof.

[0054] An augmented virtual (AV) environment refers to a simulated environment in which a virtual or computer-generated environment is combined with one or more sensory inputs from a physical environment. The sensory input may be a representation of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the faces of people are realistically reproduced from images taken of physical people. For another example, a virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, a virtual object may take a shadow that conforms to the positioning of the sun in the physical environment.

[0055] There are many different types of electronic systems that enable people to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on people's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smart phones, tablets, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smart phone). A head-mounted system can incorporate one or more imaging sensors for capturing images or videos of a physical environment and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system can have a transparent or translucent display. A transparent or translucent display can have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0056] According to the subject disclosure, a method is provided, the method comprising receiving content data captured by a sensor and receiving a context signal representing a user context. The received content data is scaled using a trained model, wherein the context signal is an input to the trained model, and the scaled content data is provided for presentation to the user.

[0057] The received content data may have a first resolution, and the scaled content data may have a second resolution greater than the first resolution. The second resolution may be based on the context signal. The sensor may be a camera, and the content data may include video data. The context signal may identify a region of interest in the content data, and wherein the region of interest is scaled to a higher resolution than a resolution of the content data outside the region of interest. The sensor may be a microphone, and the content data may include audio data.

[0058] The resolution of the sensor may be set based on the context signal. The context signal may include a content type. The method may also include providing the scaled content to a processing algorithm, wherein the content data is scaled based on the processing algorithm. The content data may be received via a wireless connection. The method may also include selecting a trained model from a plurality of models based on the received context signal.

[0059] According to the subject disclosure, a non-transitory computer-readable medium storing instructions is provided, which, when executed by one or more processors, causes the one or more processors to perform operations. The operations include receiving content data captured by a sensor and receiving a context signal representing a user context. The operations also include scaling the received content data using a trained model, wherein the context signal is an input to the trained model, and providing the scaled content data to a processing algorithm for further processing.

[0060] The received content data may have a first resolution, and the scaled content data may have a second resolution greater than the first resolution. The second resolution may be based on the context signal. The context signal may identify a region of interest in the content data, and wherein the region of interest may be scaled to a higher resolution than the resolution of the content data outside the region of interest. The context signal may include a content type. The content data may be received via a wireless connection.

[0061] According to the subject disclosure, a device is provided, the device including a memory storing a plurality of computer programs and one or more processors configured to execute instructions of the plurality of computer programs. When executed, the instructions receive content data captured by a sensor; receive a context signal representing a user context; enhance the received content data using a trained model, wherein the context signal is an input to the trained model; and provide the enhanced content data to a processing algorithm for further processing.

[0062] The received content data may be enhanced by scaling the received content data from a first resolution to a second resolution greater than the first resolution. The context signal may identify a region of interest in the content data, and wherein the region of interest may be scaled to a higher resolution than a resolution of content data outside the region of interest.

[0063] As described herein, aspects of the subject technology may include collecting and transmitting data from applications to other computing devices. The present disclosure contemplates that, in some instances, the collected data may include personal information data that uniquely identifies or can be used to identify a specific person. Such personal information data may include demographic data, location-based data, online identifiers, phone numbers, email addresses, home addresses, images, data or records related to a user's health or fitness level (e.g., vital signs measurements, medication information, exercise information), date of birth, or any other personal information.

[0064] The present disclosure recognizes that the use of such personal information data in the present technology can be used to benefit the user. For example, the personal information data can be used as a context signal. In addition, the present disclosure also anticipates other uses of personal information data that benefit the user. For example, health and fitness data can be used according to the user's preferences to provide insights into their overall health, or can be used as positive feedback to individuals who use technology to pursue health goals.

[0065] The present disclosure envisions that entities responsible for collecting, analyzing, disclosing, transmitting, storing or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities will be expected to implement and consistently apply privacy practices that are generally recognized as meeting or exceeding the industry or government requirements for maintaining user privacy. Such information about the use of personal data should be prominently accessible to users and should be updated as the collection and / or use of data changes. The user's personal information should be collected only for legal use. In addition, such collection / sharing should only occur after receiving user consent or other legal basis specified in applicable law. In addition, such entities should consider taking any necessary steps to protect and safeguard access to such personal information data and ensure that other entities with access to personal information data comply with their privacy policies and procedures. In addition, such entities can subject themselves to third-party assessments to prove that they comply with widely accepted privacy policies and practices. In addition, policies and practices should be adjusted for specific types of personal information data collected and / or accessed, and applied to applicable laws and standards, including specific considerations that are exclusive to jurisdictions that can be used to impose higher standards. For example, in the United States, collection or access to certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while health data in other countries may be subject to other regulations and policies and should be handled accordingly.

[0066] Regardless of the foregoing, the present disclosure also contemplates specific implementations in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates hardware elements and / or software elements to prevent or block access to such personal information data. For example, with respect to video conferencing, the present technology may be configured to allow users to choose to "opt in" or "opt out" to participate in the collection of personal information data at any time during or after registration for the service. In addition to providing "opt-in" and "opt-out" options, the present disclosure also contemplates providing notifications related to access or use of personal information. For example, a user may be notified that their personal information data will be accessed when downloading an application, and then be reminded again just before the personal information data is accessed by the application.

[0067] In addition, the intention of the present disclosure is that personal information data should be managed and processed in a manner that minimizes the risk of unintentional or unauthorized access or use. Once the data is no longer needed, the risk can be minimized by limiting the collection of data and deleting the data. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be promoted when appropriate by removing identifiers, controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than at the address level), controlling how data is stored (e.g., pooling data between users) and / or other methods such as differential privacy.

[0068] Therefore, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may be implemented without access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or part of such personal information data.

[0069] It should be understood that the specific order or the hierarchical structure of the frame in the process disclosed by the present invention are the illustration of the example method.Based on the design preferred requirements, it should be understood that the specific order or the hierarchical structure of the frame in the process can be rearranged or all the illustrated frames are executed.Any frame in these frames can be executed simultaneously.In one or more specific implementations, multitasking and parallel processing may be advantageous.In addition, the division of each system component in the above-mentioned specific implementation should not be understood as requiring this type of division in all specific implementations, and it should be understood that program components and system can be generally integrated together in a single software product or encapsulated in multiple software products.

[0070] As used in this specification and any claims of this patent application, the terms "base station", "receiver", "computer", "server", "processor" and "memory" all refer to electronic devices or other technical devices. These terms exclude people or groups of people. For the purpose of this specification, the term "display" or "displaying" means displaying on an electronic device.

[0071] As used herein, the phrase "at least one of" following a list of items, any of which is separated by the terms "and" or "or", modifies the list as a whole, rather than each member of the list (i.e., each item). The phrase "at least one of" does not require selection of at least one of each item listed; rather, the phrase allows for a meaning that includes at least one of any one item and / or at least one of any combination of items and / or at least one of each item. For example, the phrase "at least one of A, B, and C" or "at least one of A, B, or C" each refers to only A, only B, or only C; any combination of A, B, and C; and / or at least one of each of A, B, and C.

[0072] The predicates "configured to," "operable to," and "programmed to" do not imply any specific tangible or intangible modification of a subject matter but are intended to be used interchangeably. In one or more specific implementations, a processor configured to monitor and control an operation or component may also mean that the processor is programmed to monitor and control the operation or that the processor is operable to monitor and control the operation. Similarly, a processor configured to execute code may be interpreted as a processor programmed to execute code or operable to execute code.

[0073] Phrases such as aspect, this aspect, another aspect, some aspects, one or more aspects, a specific implementation, this specific implementation, another specific implementation, some specific implementations, one or more specific implementations, an embodiment, this embodiment, another embodiment, some embodiments, one or more embodiments, configuration, this configuration, other configurations, some configurations, one or more configurations, the subject technology, disclosure, the present disclosure, other variations thereof, etc. are for convenience and do not mean that the disclosure involving such one or more phrases is essential to the subject technology, nor does it mean that such disclosure applies to all configurations of the subject technology. Disclosures involving such one or more phrases may apply to all configurations or one or more configurations. Disclosures involving such one or more phrases may provide one or more examples. Phrases such as aspect or some aspects may refer to one or more aspects and vice versa, and this applies similarly to the other aforementioned phrases.

[0074] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" or as an "example" is not necessarily to be construed as preferred or advantageous over other implementations. Furthermore, to the extent that the terms "including," "having," and the like are used in the specification or claims, such terms are intended to be inclusive, in a manner similar to how the term "comprising" is interpreted when used as a transitional word in a claim.

[0075] All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later become known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. In addition, nothing disclosed herein is intended to be made available to the public regardless of whether the disclosure is explicitly stated in the claims. No claim element shall be interpreted under the provisions of 35 U.S.C. §112(f) unless the element is explicitly stated using the phrase “means for…” or, in the case of a method claim, the element is stated using the phrase “step for…”

[0076] The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein, but are intended to make the full scope consistent with the language claims, wherein the reference to a singular element is not intended to mean "only one", but to refer to "one or more", unless specifically noted. Unless otherwise specifically stated, the term "some" refers to one or more. Male pronouns (e.g., his) include female and neutral (e.g., her and its), and vice versa. Titles and subtitles (if any) are used only for convenience and do not limit the subject disclosure.

Claims

1. A method comprising: receiving content data captured by a sensor; receiving a context signal representing a user context; scaling the received content data using a trained model, wherein the context signal is an input to the trained model; as well as The scaled content data is provided for presentation to a user. 2 . The method of claim 1 , wherein the received content data has a first resolution and the scaled content data has a second resolution greater than the first resolution. The method of claim 2 , wherein the second resolution is based on the context signal. The method of claim 1 , wherein the sensor is a camera and the content data comprises video data. 5 . The method of claim 4 , wherein the context signal identifies a region of interest in the content data, and wherein the region of interest is scaled to a higher resolution than a resolution of the content data outside the region of interest. The method of claim 1 , wherein the sensor is a microphone and the content data comprises audio data. The method of claim 1 , wherein a resolution of the sensor is set based on the context signal. The method of claim 1 , wherein the context signal comprises a content type.

9. The method according to claim 1, further comprising: feeding the scaled content to a processing algorithm, Wherein the content data is scaled based on the processing algorithm.

10. The method of claim 1, wherein the content data is received via a wireless connection.

11. The method of claim 1, further comprising selecting the trained model from a plurality of models based on the received context signal.

12. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: receiving content data captured by a sensor; receiving a context signal representing a user context; scaling the received content data using a trained model, wherein the context signal is an input to the trained model; as well as The scaled content data is provided to a processing algorithm for further processing.

13. The non-transitory computer-readable medium of claim 12, wherein the received content data has a first resolution and the scaled content data has a second resolution greater than the first resolution. The non-transitory computer-readable medium of claim 13 , wherein the second resolution is based on the context signal.

15. The non-transitory computer-readable medium of claim 14, wherein the context signal identifies a region of interest in the content data, and wherein the region of interest is scaled to a higher resolution than a resolution of the content data outside the region of interest.

16. The non-transitory computer-readable medium of claim 12, wherein the context signal comprises a content type.

17. The non-transitory computer readable medium of claim 12, wherein the content data is received via a wireless connection.

18. A device comprising: a memory storing a plurality of computer programs; and one or more processors configured to execute instructions of the plurality of computer programs to: receiving content data captured by a sensor; receiving a context signal representing a user context; enhancing the received content data using a trained model, wherein the context signal is an input to the trained model; as well as The enhanced content data is provided to a processing algorithm for further processing.

19. The apparatus of claim 18, wherein the received content data is enhanced by scaling the received content data from a first resolution to a second resolution greater than the first resolution.

20. The apparatus of claim 19, wherein the context signal identifies a region of interest in the content data, and wherein the region of interest is scaled to a higher resolution than a resolution of the content data outside the region of interest.