System and method for image re-projection

Abstract

Imaging systems and techniques are described. An imaging system receives image data of an environment according to a first viewing angle. The imaging system detects an object in the image data. The imaging system generates reprojected image data of at least a portion of the environment (e.g., at least representing the object) based on the image data according to a second viewing angle different from the first viewing angle. In some examples, the imaging system generates and outputs an indicator of a state of the object based on the reprojection image data. In some examples, the indicator of the state of the object may indicate a change in the state of the object, such as a change in illumination characteristics of the light source, a change in content displayed on the display screen, a change in movement of the object, etc.

Description

Technical Field

[0001] The present application relates to image processing. More specifically, the present application relates to a system and method for image reprojection representing a detected object from different viewing angles and determining a state of the detected object based on the image reprojection. Background Art

[0002] Many devices include one or more cameras. For example, a smartphone or tablet includes a front camera for capturing selfie images and a rear camera for capturing images of scenes (such as landscapes or other scenes of interest to the device user). A camera can capture images using the camera's image sensor. The representation of a scene in an image captured by a camera depends on the camera's field of view and / or viewport (e.g., the portion of the scene that the camera is able to observe using the image sensor), for example based on the camera's position and orientation relative to the scene. In some examples, if the camera's viewport does not provide a clear view of a particular portion of the scene, the particular portion of the scene may be difficult to see clearly, for example because the field of view is at an acute angle relative to the particular portion of the scene. Various systems can utilize image data from a camera, such as a virtual reality (VR) or augmented reality (AR) system. Summary of the invention

[0003] In some examples, systems and techniques for image processing are described. Imaging systems and techniques are described. An imaging system receives image data of an environment according to a first perspective. The imaging system detects an object in the image data. The imaging system generates reprojected image data of at least a portion of the environment (e.g., at least representing the object) based on the image data according to a second perspective different from the first perspective. The imaging system generates and outputs an indicator of a state of the object based on the reprojected image data. In some examples, the indicator of the state of the object can indicate a change in the state of the object and / or a region of interest (RoI), such as a change in the lighting characteristics of a light source, a change in content displayed on a display screen, a change in the movement of the object, etc.

[0004] In one example, an apparatus for media processing is provided. The apparatus includes a memory and one or more processors (e.g., implemented in a circuit) coupled to the memory. The one or more processors are configured and can: receive image data of an environment, the image data representing the environment from a first perspective; detect an object in the environment in the image data; and generate reprojected image data of at least a portion of the environment based on the image data, the reprojected image data representing at least the object from a second perspective different from the first perspective.

[0005] In another example, a method of image processing is provided. The method includes: receiving image data of an environment, the image data representing the environment from a first perspective; detecting an object in the environment in the image data; and generating reprojected image data of at least a portion of the environment based on the image data, the reprojected image data representing at least the object from a second perspective different from the first perspective.

[0006] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to: receive image data of an environment, the image data representing the environment from a first perspective; detect an object in the environment in the image data; and generate reprojected image data of at least a portion of the environment based on the image data, the reprojected image data representing at least the object from a second perspective different from the first perspective.

[0007] In another example, an apparatus for image processing is provided. The apparatus includes: a device for receiving image data of an environment, the image data representing the environment from a first perspective; detecting an object in the environment in the image data; and generating reprojected image data of at least a portion of the environment based on the image data, the reprojected image data representing at least the object from a second perspective different from the first perspective.

[0008] In some aspects, one or more of the above methods, apparatuses, and computer-readable media further include: receiving input using a user interface, wherein the input indicates the second perspective. In some aspects, one or more of the above methods, apparatuses, and computer-readable media further include: generating gesture information corresponding to the second perspective based on the position of the feature of the object in the environment.

[0009] In some aspects, generating reprojected image data of at least a portion of the environment includes generating the reprojected image data using probabilistic reprojection. In some aspects, generating the reprojected image data of at least a portion of the environment according to the second perspective includes using an output of a trained machine learning model responsive to an input of the image data into the trained machine learning model.

[0010] In some aspects, one or more of the above methods, apparatuses, and computer-readable media further include: receiving previous image data of an environment; and determining that the image data is missing visual information of a portion of the object, wherein the portion of the object is discernible from a second perspective, wherein generating reprojected image data of at least the portion of the environment includes reconstructing the visual information of the portion of the object based on the previous image data. In some aspects, one or more of the above methods, apparatuses, and computer-readable media further include: generating a mask associated with an occlusion on the portion of the object, wherein the image data is missing the visual information of the portion of the object due to the occlusion obscuring the portion of the object in the image data.

[0011] In some aspects, one or more of the above methods, devices, and computer-readable media also include: determining that the image data is missing visual information of a portion of an object, wherein the portion of the object is discernible from a second perspective, and wherein generating reprojected image data of at least a portion of the environment includes reconstructing the visual information of the portion of the object using an output of a trained machine learning model responsive to an input of the image data into the trained machine learning model.

[0012] In some aspects, one or more of the methods, apparatuses, and computer-readable media described above further include: determining an illumination scheme that illuminates an environment, wherein generating reprojected image data of at least a portion of the environment includes reconstructing the illumination scheme according to a second perspective. In some aspects, one or more of the methods, apparatuses, and computer-readable media described above further include: modifying the image data to normalize the illumination in the image data before generating the reprojected image data of at least the portion of the environment based on the image data.

[0013] In some aspects, one or more of the methods, devices, and computer-readable media described above further include: outputting an indicator of the state of the object based on the reprojected image data. In some aspects, one or more of the methods, devices, and computer-readable media described above further include: monitoring the state of the object over a time period based on the reprojected image data, wherein both the image data and the reprojected image data include images corresponding to different times during the time period; and identifying a change in the state of the object during the time period, wherein the indicator of the state of the object indicates the change in the state of the object. In some aspects, the change in the state of the object includes a change in the lighting characteristics of the light source, wherein the object includes the light source. In some aspects, one or more of the methods, devices, and computer-readable media described above further include: parsing one or more characters on the object in the reprojected image data using optical character recognition (OCR), wherein the change in the state of the object is associated with the one or more characters. In some aspects, the change in the state of the object includes a change in the movement of at least a portion of the object.

[0014] In some aspects, image data is received from an image sensor. In some aspects, a first subset of image data is received from a first image sensor, wherein a second subset of image data is received from a second image sensor. In some aspects, detecting the object in the image data includes tracking the object from the first subset of image data to the second subset of image data.

[0015] In some aspects, one or more of the methods, devices, and computer-readable media described above further include: causing a display to display information based on the reprojected image data. In some aspects, one or more of the methods, devices, and computer-readable media described above further include: causing a communication interface to send information based on the reprojected image data to a recipient device.

[0016] In some aspects, the apparatus is part of and / or includes a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a head-mounted display (HMD) device, a wireless communication device, a mobile device (e.g., a mobile phone and / or a mobile handset and / or a so-called "smart phone" or other mobile device), a camera, a personal computer, a laptop computer, a server computer, a vehicle or a computing device or component of a vehicle, another device, or a combination thereof. In some aspects, the apparatus includes a camera or multiple cameras for capturing one or more images. In some aspects, the apparatus also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the apparatus may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyroscopes, one or more accelerometers, any combination thereof, and / or other sensors).

[0017] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0018] The foregoing and other features and aspects will become more fully apparent by reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Illustrative aspects of the present application are described in detail below with reference to the following drawings:

[0020] Figure 1 is a block diagram illustrating an example architecture of an image capture and processing system according to some examples;

[0021] Figure 2 is a block diagram illustrating an example architecture of an imaging system that performs an imaging process according to some examples;

[0022] Figure 3A is a perspective view showing a head mounted display (HMD) used as part of an imaging system according to some examples;

[0023] Figure 3B is a diagram showing a device worn by a user according to some examples Figure 3A A perspective view of a head mounted display (HMD);

[0024] Figure 4A is a perspective view showing a front surface of a mobile handset that includes a front-facing camera and can be used as part of an imaging system according to some examples;

[0025] Figure 4B is a perspective view showing a rear surface of a mobile handset that includes a rear camera and can be used as part of an imaging system according to some examples;

[0026] Figure 5 is a flow chart illustrating an imaging process according to some examples, the imaging process including capturing an image of an object (fire truck), reprojecting the image to generate a reprojected image of the object, and outputting a status indicator indicating a status of the object;

[0027] Figure 6 is a conceptual diagram illustrating a field of view of a virtual camera associated with image reprojection according to some examples;

[0028] Figure 7 is a flow chart illustrating an imaging process according to some examples, the imaging process comprising capturing an image of an object (computer), reprojecting the image to generate a reprojected image of the object, and outputting a status indicator indicating a status of the object;

[0029] Figure 8 is a block diagram illustrating an example of a neural network that may be used for image processing operations according to some examples;

[0030] Fig. 9 is a flow chart illustrating an imaging process according to some examples; and

[0031] Fig.10 is a diagram illustrating an example of a computing system for implementing certain aspects described herein. DETAILED DESCRIPTION

[0032] Certain aspects of the present disclosure are provided below. Some of these aspects can be applied independently, and some of them can be applied in combination, which is obvious to those skilled in the art. In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of various aspects of the application. However, it is obvious that various aspects can be practiced without these specific details. The drawings and description are not intended to be restrictive.

[0033] The following description provides only example aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. Instead, the following description of the example aspects will provide an enabling description for implementing the example aspects for those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0034] A camera is a device that receives light and captures image frames (such as still images or video frames) using an image sensor. The terms "image", "image frame" and "frame" are used interchangeably herein. A camera can be configured with various image capture and image processing settings. Different settings result in images with different appearances. Some camera settings, such as ISO, exposure time, aperture size, f / stop, shutter speed, focal length, and gain, are determined and applied before or during the capture of one or more image frames. For example, settings or parameters may be applied to an image sensor to capture one or more image frames. Other camera settings may configure post-processing of one or more image frames, such as changes to contrast, brightness, saturation, sharpness, level, curves, or colors. For example, settings or parameters may be applied to a processor (e.g., an image signal processor or ISP) for processing one or more image frames captured by an image sensor.

[0035] An extended reality (XR) system or device may provide a user with virtual content and / or a real-world view that may combine a physical environment (scene) and a virtual environment (including virtual content). The XR system facilitates user interaction with such a combined XR environment. The real-world view may include real-world objects (also referred to as physical objects), such as people, vehicles, buildings, tables, chairs, and / or other real-world or physical objects. The XR system or device may facilitate interaction with different types of XR environments (e.g., a user may use an XR system or device to interact with an XR environment). The XR system may include a virtual reality (VR) system that facilitates interaction with a VR environment, an AR system that facilitates interaction with an augmented reality (AR) environment, a mixed reality (MR) system that facilitates interaction with an MR environment, and / or other XR systems. Examples of XR systems or devices include head-mounted displays (HMDs), smart glasses, and the like. In some cases, the XR system may track portions of a user (e.g., a user's hands and / or fingertips) to allow the user to interact with virtual content items.

[0036] The representation of a scene in an image captured by a camera depends on the pose, viewport, and / or field of view of the camera, e.g., based on the position and orientation of the camera relative to the scene. In some examples, a particular portion of the scene may be difficult to see clearly if the pose, viewport, and / or field of view of the camera does not provide a clear view of the particular portion of the scene, e.g., because the pose, viewport, and / or field of view is at an acute angle relative to the particular portion of the scene.

[0037] In some examples, systems and techniques for image processing are described. An imaging system receives image data of an environment according to a first perspective. The imaging system detects an object in the image data. Based on the image data, the imaging system generates reprojected image data of at least a portion of the environment (e.g., at least representing the object) according to a second perspective different from the first perspective. The imaging system generates and outputs an indicator of a state of the object based on the reprojected image data. In some examples, the indicator of the state of the object can indicate a change in the state of the object, such as a change in the lighting characteristics of a light source, a change in content displayed on a display screen, a change in the movement of the object, and the like.

[0038] The imaging systems and techniques described herein provide many technical improvements over existing imaging systems. For example, the imaging systems and techniques described herein provide improved reliability, accuracy, and precision in identifying the state of an object and / or a change in the state of an object based on image reprojection. The imaging systems and techniques described herein can provide improved flexibility for determining the state of an object and / or a change in the state of an object, because the camera does not need to be pointed directly at the surface of the object to determine the state associated with the surface of the object. The imaging systems and techniques described herein can provide improved efficiency for determining the state of multiple objects and / or a change in the state of an object, because a single camera whose field of view includes multiple objects can be used with reprojection to accurately track the state of each object over time without requiring any camera specific to each object.

[0039] Various aspects of the application will be described with respect to the accompanying drawings. Figure 1 1 is a block diagram illustrating the architecture of image capture and processing system 100. Image capture and processing system 100 includes various components for capturing and processing images of one or more scenes (e.g., images of scene 110). Image capture and processing system 100 can capture independent images (or photographs) and / or can capture a video including multiple images (or video frames) of a particular sequence. Lens 105 of system 100 faces scene 110 and receives light from scene 110. Lens 105 bends light toward image sensor 130. Light received by lens 105 passes through an aperture controlled by one or more control mechanisms 120 and is received by image sensor 130. In some examples, scene 110 is a scene in an environment. In some examples, scene 110 is a scene of at least a portion of a user. For example, scene 110 can be a scene of one or both of a user's eyes and / or at least a portion of a user's face.

[0040] The one or more control mechanisms 120 may control exposure, focus, and / or zoom based on information from the image sensor 130 and / or based on information from the image processor 150. The one or more control mechanisms 120 may include a plurality of mechanisms and components; for example, the control mechanisms 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. The one or more control mechanisms 120 may also include additional control mechanisms in addition to those shown, such as control mechanisms that control analog gain, flash, HDR, depth of field, and / or other image capture attributes.

[0041] A focus control mechanism 125B of the control mechanism 120 may obtain a focus setting. In some examples, the focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 125B may adjust the position of the lens 105 relative to the position of the image sensor 130. For example, based on the focus setting, the focus control mechanism 125B may move the lens 105 closer to the image sensor 130 or further away from the image sensor 130 by actuating a motor or a servo mechanism, thereby adjusting the focus. In some cases, additional lenses may be included in the system 100, such as one or more micro lenses above each photodiode of the image sensor 130, each of which bends light received from the lens 105 toward the corresponding photodiode before reaching the photodiode. The focus setting may be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), or some combination thereof. The focus setting may be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus setting may be referred to as an image capture setting and / or an image processing setting.

[0042] Exposure control mechanism 125A of control mechanism 120 may obtain an exposure setting. In some cases, exposure control mechanism 125A stores the exposure setting in a memory register. Based on this exposure setting, exposure control mechanism 125A may control the size of the aperture (e.g., aperture size or f / stop), the duration that the aperture is open (e.g., exposure time or shutter speed), the sensitivity of image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by image sensor 130, or any combination thereof. The exposure setting may be referred to as an image capture setting and / or an image processing setting.

[0043] A zoom control mechanism 125C of the control mechanism 120 may obtain a zoom setting. In some examples, the zoom control mechanism 125C stores the zoom setting in a memory register. Based on the zoom setting, the zoom control mechanism 125C may control the focal length of an assembly of lens elements (lens assembly) including the lens 105 and one or more additional lenses. For example, the zoom control mechanism 125C may control the focal length of the lens assembly by actuating one or more motors or servo mechanisms to move one or more lenses relative to each other. The zoom setting may be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly may include a parfocal zoom lens or a varifocal zoom lens. In some examples, the lens assembly may include a focusing lens (which may be the lens 105 in some cases) that first receives light from the scene 110, wherein the light then passes through an afocal zoom system between the focusing lens (e.g., the lens 105) and the image sensor 130 before the light reaches the image sensor 130. In some cases, the afocal zoom system can include two positive (e.g., converging, convex) lenses with equal or similar focal lengths (e.g., within a threshold difference) with a negative (e.g., diverging, concave) lens between them. In some cases, zoom control mechanism 125C moves one or more lenses in the afocal zoom system, such as a negative lens and one or two positive lenses.

[0044] The image sensor 130 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a specific pixel in the image generated by the image sensor 130. In some cases, different photodiodes may be covered by different color filters, and thus may measure light that matches the color of the color filter covering the photodiode. For example, a Bayer color filter includes a red filter, a blue filter, and a green filter, where each pixel of an image is generated based on red light data from at least one photodiode covered in a red filter, blue light data from at least one photodiode covered in a blue filter, and green light data from at least one photodiode covered in a green filter. Other types of color filters may use yellow, magenta, and / or cyan (also called "emerald") filters instead of or in addition to red, blue, and / or green filters. Some image sensors may lack color filters entirely, and may instead use different photodiodes (vertically stacked in some cases) throughout the pixel array. Different photodiodes throughout the pixel array may have different spectral sensitivity curves, and therefore respond to light of different wavelengths. Monochrome image sensors may also lack color filters and therefore lack color depth.

[0045] In some cases, image sensor 130 may alternatively or additionally include an opaque and / or reflective mask that blocks light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles, which may be used for phase detection autofocus (PDAF). Image sensor 130 may also include an analog gain amplifier to amplify an analog signal output by a photodiode and / or an analog / digital converter (ADC) to convert the analog signal output of the photodiode (and / or amplified by the analog gain amplifier) ​​into a digital signal. In some cases, certain components or functions discussed with respect to one or more of control mechanism 120 may alternatively or additionally be included in image sensor 130. Image sensor 130 may be a charge coupled device (CCD) sensor, an electron multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal oxide semiconductor (CMOS), an N-type metal oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0046] The image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), and / or one or more of any other type of processor 1010 discussed with respect to the computing system 1000. The host processor 152 may be a digital signal processor (DSP) and / or other type of processor. In some embodiments, the image processor 150 is a single integrated circuit or chip (e.g., referred to as a system on a chip or SoC) that includes the host processor 152 and the ISP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth TM, Global Positioning System (GPS), etc.), any combination thereof, and / or other components. The I / O port 156 may include any suitable input / output port or interface according to one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a Serial General Purpose Input / Output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface), an Advanced High-Performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In an illustrative example, the host processor 152 may communicate with the image sensor 130 using an I2C port, and the ISP 154 may communicate with the image sensor 130 using a MIPI port.

[0047] The image processor 150 may perform a number of tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. The image processor 150 may store image frames and / or processed images in random access memory (RAM) 140 and / or 1020, read-only memory (ROM) 145 and / or 1025, cache memory, memory unit, another storage device, or some combination thereof.

[0048] Various input / output (I / O) devices 160 may be connected to the image processor 150. The I / O devices 160 may include a display screen, a keyboard, a keypad, a touch screen, a touchpad, a touch-sensitive surface, a printer, any other output device 1035, any other input device 1045, or some combination thereof. In some cases, subtitles may be input into the image processing device 105B through a physical keyboard or keypad of the I / O device 160 or through a virtual keyboard or keypad of a touch screen of the I / O device 160. The I / O 160 may include one or more ports, sockets, or other connectors that implement a wired connection between the system 100 and one or more peripheral devices, through which the system 100 may receive data from the one or more peripheral devices and / or transmit data to the one or more peripheral devices. The I / O 160 may include one or more wireless transceivers that implement a wireless connection between the system 100 and one or more peripheral devices, through which the system 100 may receive data from the one or more peripheral devices and / or transmit data to the one or more peripheral devices. The peripheral devices may include any of the types of I / O devices 160 discussed previously, and may themselves be considered I / O devices 160 once they are coupled to a port, socket, wireless transceiver, or other wired and / or wireless connector.

[0049] In some cases, the image capture and processing system 100 can be a single device. In some cases, the image capture and processing system 100 can be two or more separate devices, including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some embodiments, the image capture device 105A and the image processing device 105B can be wirelessly coupled together, for example, via one or more wires, cables, or other electrical connectors and / or via one or more wireless transceivers. In some embodiments, the image capture device 105A and the image processing device 105B can be disconnected from each other.

[0050] like Figure 1 As shown, the vertical dashed line will Figure 1 The image capture and processing system 100 is divided into two parts, respectively representing the image capture device 105A and the image processing device 105B. The image capture device 105A includes a lens 105, a control mechanism 120, and an image sensor 130. The image processing device 105B includes an image processor 150 (including an ISP 154 and a host processor 152), a RAM 140, a ROM 145, and an I / O 160. In some cases, some of the components shown in the image capture device 105A (such as the ISP 154 and / or the host processor 152) may be included in the image capture device 105A.

[0051] The image capture and processing system 100 may include an electronic device, such as a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the image capture and processing system 100 may include one or more wireless transceivers for wireless communications (such as cellular network communications, 1002.11 wi-fi communications, wireless local area network (WLAN) communications, or some combination thereof). In some embodiments, the image capture device 105A and the image processing device 105B may be different devices. For example, the image capture device 105A may include a camera device, and the image processing device 105B may include a computing device, such as a mobile handset, a desktop computer, or other computing device.

[0052] Although the image capture and processing system 100 is shown as including certain components, one of ordinary skill will appreciate that the image capture and processing system 100 may include more than Figure 1 . The components of the image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some embodiments, the components of the image capture and processing system 100 may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device that implements the image capture and processing system 100.

[0053] Figure 2 1 is a block diagram showing an example architecture of an imaging system 200 that performs an imaging process. The imaging system 200 may include an image capture and processing system 100, an image capture device 105A, an image processing device 105B, an HMD 310, a mobile handset 410, and an image processing device 105B. Figure 5 The imaging process 500 includes an imaging system, a user device 530, a camera 540, and an execution Figure 7 Imaging process 700 of the imaging system, user equipment 730, camera 740, neural network 800, execution Fig. 9At least one of the imaging system of the imaging process 900, the computing system 1000, or a combination thereof. In some examples, the imaging system 200 may include, for example, one or more laptop computers, phones, tablet computers, mobile handsets, video game consoles, vehicle computers, desktop computers, wearable devices, televisions, media centers, extended reality (XR) systems, virtual reality (VR) systems, augmented reality (AR) systems, mixed reality (MR) systems, head-mounted display (HMD) devices, other types of computing devices discussed herein, or a combination thereof. In some examples, the imaging system 200 and the corresponding imaging process can be used in network-based interactive system applications, such as those used for video conferencing, extended reality (XR), video games, metaverse environments, or a combination thereof.

[0054] The imaging system 200 includes one or more sensors 205. In some examples, the sensor 205 includes one or more image sensors or one or more cameras. The image sensor(s) capture image data, which may include one or more images, one or more videos, portions thereof, or combinations thereof. In some examples, at least one of the sensors 205 may be pointed toward the user (e.g., may be facing the user), and thus may capture (e.g., depict or otherwise represent) sensor data (e.g., image data) of at least a portion of the user. In some examples, at least one of the sensor(s) 205 may be pointed away from the user (e.g., may be facing away from the user) and / or toward the environment in which the user is located, and thus may capture (e.g., depict or otherwise represent) sensor data (e.g., image data) of at least a portion of the environment. In some examples, data captured by at least one of the sensor(s) 205 pointed away from the user and / or toward the user may have a field of view (FoV) that includes, is included by, overlaps with, and / or otherwise corresponds to the FoV of the user's eyes. Figure 2 Within, a graphic representing sensor(s) 205 illustrates sensor(s) 205 as including a camera facing an environment in which a toy fire truck and a pen are on a table. Examples of image sensors and / or cameras for sensor(s) 205 may include image capture and processing system 100, image capture device 105A, image processing device 105B, image sensor 130, image sensor(s) of any of cameras 330A-330D, image sensor(s) of any of cameras 430A-430D, camera 540, camera 740, an image sensor of imaging process 900, an image sensor of input device 1045, or a combination thereof. Figure 2Within, the graphic representing sensor data 210 shows an image of an environment with a toy fire truck and a pen depicted in the graphic representing sensor 205 .

[0055] Sensors 205 of imaging system 200 may include one or more sensors that track information about a user and / or the environment. For example, sensor 205 may track a posture (e.g., position and / or orientation) of a user and / or imaging system 200 within the environment. In some examples, sensor 205 may include an image sensor and / or a camera.

[0056] In some examples, in addition to the image sensor and / or camera, the sensor 205 of the imaging system 200 may include one or more other sensors, such as one or more other image sensors, other cameras, microphones, heart rate monitors, oximeters, biometric sensors, positioning receivers, global navigation satellite system (GNSS) receivers, inertial measurement units (IMUs), accelerometers, gyroscopes, gyroscopes, barometers, thermometers, altimeters, depth sensors, light detection and ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, sound detection and ranging (SODAR) sensors, sound navigation and ranging (SONAR) sensors, time of flight (ToF) sensors, structured light sensors, other sensors discussed herein, or combinations thereof. In some examples, the sensor 205 includes at least one input device 1045 of the computing system 1000. In some examples, the sensor 205 of the imaging system 200 may include one or more sensors that track information about the user and / or the environment. For example, the sensors 205 may track the posture (e.g., position and / or orientation) of the user and / or imaging system 200 within the environment. In some embodiments, one or more of these additional sensors may supplement or refine the sensor readings from the sensors 205. For example, an inertial measurement unit (IMU), accelerometer, gyroscope, or other sensor may be used to identify the posture (e.g., position and / or orientation) and / or movement and / or acceleration of the imaging system 200 and / or the user in the environment, which may be used by the imaging system 200 to reduce motion blur, rotation blur, or a combination thereof.

[0057] The imaging system 200 includes an object detector 215. The object detector 215 is configured to and may detect, identify, and / or track objects within the sensor data 210. For example, the object detector 215 may detect, identify, and / or track a toy fire truck object in the sensor data 210. The object detector 215 may include a feature extractor, a feature detector, a feature identifier, a feature tracker, an object detector, an object identifier, an object tracker, a face detector, a face identifier, a face tracker, a person detector, a person identifier, a person tracker, a classifier, or a combination thereof. The object detector 215 may perform feature extraction, feature detection, feature identification, feature tracking, object detection, object identification, object tracking, face detection, face recognition, face tracking, person detection, person identification, person tracking, classification, or a combination thereof. Figure 2 , a graphic representing object detector 215 illustrates a bounding box around the object (a toy fire truck) as depicted in sensor data 210 .

[0058] In some examples, object detector 215 detects objects in sensor data 210 by inputting sensor data 210 into one or more of one or more trained machine learning (ML) models 250 discussed herein, and receiving output indicating the position and / or orientation of a face from trained ML model 250. Trained ML model 250 may be trained (e.g., by imaging system 200) with training data for use by object detector 215, the training data including training sensor data (e.g., image data), the training sensor data including representations (e.g., depictions) of objects. In some examples, the training data may include already determined positions and / or orientations of objects within the training sensor data. In some examples, the object detector 215 detects the position (e.g., pixel coordinates) of the object within the sensor data 210, the position (e.g., 3D coordinates within the 3D volume of the environment) of the object, the orientation (e.g., pitch, yaw, and / or roll) of the object within the sensor data 210 (e.g., along an axis about which a discernible rotation is made in the sensor data 210), and / or the orientation (e.g., pitch, yaw, and / or roll) of the object within the environment. In some examples, the pose of the object may include a pose of a first portion of the object relative to a second portion of the object. For example, if the object is a person, the pose of the object may indicate whether the user's hand is up or down. If the object is a vehicle, the pose of the object may indicate whether the door is open or closed. In some examples, the object detector 215 detects, identifies, tracks, and / or predicts the pose of the object relative to the sensor 205 (e.g., relative to the sensor 205) within the environment. For example, the pose (e.g., position and / or orientation) of the object in the environment (e.g., relative to the sensor 205) may be based on an established correspondence between the extracted / trained features of the object and the sensor data. For example, the pose of the object may be based on how the distance between two features on the object (e.g., interocular distance) in sensor data 210 compares to a reference distance (e.g., interocular distance) for a particular instance of the object and / or an average instance of the object. The features may be known based on a 3D model of the object that may have been generated (e.g., by imaging system 200 or another system) prior to object detection.Examples of objects detected using object detector 215 may include a person, a face, a vehicle, a computer, a device, a document, a toy, an animal, a plant, a structure, a target, a pattern, a texture (e.g., as in an image applied to a 3D mesh), a texture (e.g., as in the roughness, smoothness, pattern, and / or consistency of a surface), an optical glyph (e.g., a barcode, a Quick Response (QR) code, an Aztec code, a Data Matrix, a MaxiCode, a Codablock F code, a PDF 17 code, and / or a 2D barcode), a region of interest (RoI) on an object, an RoI in an environment, an RoI in sensor data 210, a portion of an object, a portion of an environment, a portion of sensor data 210, or a combination thereof. Changes in the pose of an object tracked by object detector 215 may include changes in position, changes in orientation, changes in the pose of a portion of the object relative to other portions of the object, or a combination thereof. For example, a change in the pose of an object may include the object moving translationally through the environment, the object rotating about one or more axes in the environment, a first portion of the object moving relative to a second portion of the object (e.g., a person waving their arms or turning their head, a vehicle opening its door or rotating its wheels, etc.), or a combination thereof. In some examples, instead of or in addition to detecting / recognizing / tracking an object, object detector 215 may detect, recognize, and / or track a RoI on the object, a RoI in the environment, a RoI in sensor data 210, or a combination thereof.

[0059] In some examples, object detector 215 may generate a three-dimensional (3D) model of at least a portion of an object based on sensor data 210. The 3D model may include a 3D mesh, a texture of the mesh, or a combination thereof. In some examples, sensor data 210 may include depth data (e.g., a point cloud) from at least one depth sensor (e.g., LIDAR, RADAR, SONAR, SODAR, ToF, structured light) of sensor 205, and the 3D model may be generated based on the depth data from at least one depth sensor. In some examples, sensor data 210 may include image data from multiple image sensors of sensor 205, and object detector 215 may generate a 3D model from image data from multiple image sensors, from monocular depth estimation, and / or from a point cloud based on depth data from stereo depth detection. In some examples, sensor data 210 may include video frame data from at least one image sensor of sensor 205 over a period of time, and a 3D model may be generated from video frame data from at least one image sensor based on depth data from depth detection over time.

[0060] In some examples, to detect an object in sensor data 210, object detector 215 is configured to track the object from a first subset of sensor data 210 to a second subset of sensor data 210. For example, the first subset of sensor data 210 may include sensor data captured by a first sensor in sensor(s) 205, and the second subset of sensor data 210 may include sensor data captured by a second sensor in sensor(s) 205. Thus, object detector 215 may track the object from a field of view of the first sensor in sensors 205 to a field of view of the second sensor in sensors 205.

[0061] The imaging system 200 includes a reprojection engine 220. The reprojection engine 220 is configured to and may reproject representations of objects in the sensor data 210 (e.g., as detected via the object detector 215) from different perspectives to generate reprojected sensor data 225. The different perspectives may have different camera poses, with different camera positions within the environment (e.g., 3D coordinates within a 3D volume of the environment) and / or different camera orientations within the environment (e.g., pitch, yaw, and / or roll). For example, an object may be represented (e.g., depicted) in the sensor data 210 from a first perspective. The reprojection engine 220 may generate reprojected sensor data 225 that represents the object from a second perspective different from the first perspective. For example, in an example where the sensor data 210 includes an image depicting the object from a first perspective, the reprojected sensor data 225 may include a reprojected image depicting the object from a second perspective. In some examples, the second perspective points in a direction perpendicular to the surface of the object (e.g., to obtain a clear view of the surface of the object), while the first perspective points in a direction not perpendicular to the surface of the object (e.g., to have a less clear view of the surface of the object). In some examples, the second perspective points in a direction parallel to the surface of the object, while the first perspective points in a direction not parallel to the surface of the object. In some examples, the second perspective points in a direction between the perpendicular perspective discussed above and the parallel perspective discussed above, while the first perspective points in a different direction. In some examples, the reprojection engine 220 performs reprojection of the sensor data 210 using probabilistic reprojection to generate reprojected sensor data 225. In some examples, the reprojection engine 220 performs reprojection of the sensor data 210 using one or more trained ML models 250 to generate reprojected sensor data 225. Figure 2 , a graphic representing the reprojection engine 220 shows a camera pointing at an object (a toy fire truck) from a first camera pose corresponding to a first perspective (represented by a camera icon with a "1"), with an arrow from the camera to a virtual camera pointing at the object from a second camera pose corresponding to a second perspective (represented by a camera icon with a "2"). Figure 22, a graphic representing reprojected sensor data 225 shows the object (a toy fire truck) depicted in sensor data 210 but depicted from a second perspective shown in the graphic representing reprojection engine 220. The second perspective illustrated in the graphic representing reprojection engine 220 and reprojected sensor data 225 is illustrated as being perpendicular (or nearly perpendicular) to the surface of the right side of the object (the toy fire truck). The portion of the object to be viewed by the virtual camera may be planar, polygonal, polyhedral, circular, concave, convex, another shape, or a combination thereof.

[0062] In some examples, the reprojection engine 220 performs reprojection by inputting the sensor data 210 (e.g., and an indication of the second perspective) into one or more of the trained ML models 250 discussed herein and receiving output from the trained ML model 250, the output comprising the reprojected sensor data 225 representing the object from the second perspective. The trained ML model 250 can be trained (e.g., by the imaging system 200) using training data for use by the reprojection engine 220, the training data comprising multiple representations of the object from multiple different perspectives (such as the first perspective and the second perspective discussed above). The different perspectives can have different camera poses, with different camera positions within the environment (e.g., 3D coordinates within the 3D volume of the environment) and / or different camera orientations within the environment (e.g., pitch, yaw, and / or roll).

[0063] In some examples, the reprojection engine 220 determines that the sensor data 210 lacks information (e.g., visual information, image information, or other sensor information) for a portion of an object that is set to be visible in the reprojected sensor data 225 according to the second perspective. The sensor data 210 may lose information due to occlusion, or due to the angle of the first perspective, or due to shadows, or some combination thereof. In some examples, the reprojection engine 220 reconstructs the missing information based on previous sensor data (e.g., previous images) of an environment including a representation of a portion of the object. In some examples, the missing information is missing due to occluding the portion of the object, and the reprojection engine 220 may generate a mask associated with the occlusion on the portion of the object. In some examples, the reprojection engine 220 reconstructs the missing information by feeding the sensor data 210 and / or previous sensor data into a trained ML model 250, which is trained to reconstruct the missing data based on training data, the training data including an image with missing data and a corresponding image where the missing data is located. In some examples, reprojection engine 220 can determine a lighting scheme that illuminates the environment based on sensor data 210, and can reconstruct the lighting scheme based on the second perspective in reprojected sensor data 225. In some examples, reprojection engine 220 can normalize the lighting in sensor data 210 before generating reprojected sensor data 225 based on sensor data 210.

[0064] Imaging system 200 includes state detection engine 230. State detection engine 230 is configured to and may determine the state of an object (detected using object detector 215) based on the representation of the object in reprojected sensor data 225. In some examples, state detection engine 230 may also determine the state of the object based on the representation of the object in sensor data 210. Figure 2 Within, graphics representing the state detection engine 230 include graphics representing re-projected sensor data 225 depicting an object (a toy fire truck) from a second perspective, a first state determination indicating that the object is a fire truck (represented by a fire icon), and a second state determination indicating that the object's warning lights are illuminated (represented by a warning light icon).

[0065] In some examples, the state detection engine 230 can determine the state of the object by determining the type of the object, such as by determining whether the object is a person, a face, a vehicle, a computer, a device, a document, a toy, an animal, a plant, a structure, a target, a pattern, another type of object, a subtype of one of the previously listed object types, or a combination thereof. For example, a subtype of the vehicle type can include a car, a fire truck, a police car, an ambulance, a taxi, an aircraft, an airplane, a helicopter, a vessel, a boat, a submarine, or another type of vehicle. For example, a graphic representing the state detection engine 230 illustrates a first state determination indicating that the type of the object is a fire truck, as represented by a fire icon. In some examples, the state detection engine 230 can include reference data (e.g., a reference image and / or a 3D model) of the object, which the state detection engine 230 can use to compare the object (in the sensor data 210 and / or the reprojected sensor data 225) to determine the state of the object. For example, state detection engine 230 may include reference data for automobiles, and state detection engine 230 may compare an object (in sensor data 210 and / or reprojected sensor data 225) to the reference data to determine that the object is a fire truck.

[0066] In some examples, the state detection engine 230 can determine the state of the object by determining the lighting characteristics of the light source included in the object and discernible in the reprojected sensor data 225. The lighting characteristics may include, for example, lighting level, lighting color, lighting direction, lighting pattern, or a combination thereof. The lighting level may refer to whether the light source is disabled (off) or enabled (on). In some examples, if the light source is enabled (on), the lighting level may also refer to different levels or degrees of lighting, luminosity, brightness, and / or power, including, for example, very dim, dim, medium, bright, and very bright. The lighting color may refer to whether the light source emits white, pink, red, orange, yellow, green, cyan, blue, indigo, purple, brownish yellow, another color, or a combination thereof. The lighting direction may refer to the direction in which the light source emits light and / or the direction in which the light source points. The lighting pattern may refer to a pattern of lighting level, color, and / or direction over time. Examples of lighting patterns may include steady lighting, flashing light, light that fades in and out, light that maintains a single color or set of colors, light that cycles between two or more colors or sets of colors, light that maintains a specific direction, light that changes between two or more directions, or a combination thereof. For example, the graphic representing the state detection engine 230 illustrates a second state determination indicating that the warning lights atop the object (fire truck) are enabled and / or illuminated, as represented by the warning light icon. In another illustrative example, the object may be a device such as a modem, a router, or a computing system 1000 that has one or more indicator lights (e.g., light emitting diode (LED) indicator lights). The state detection engine 230 may determine the state of the object by determining which indicator lights of the object are enabled (on) or disabled (off), what color any enabled indicator lights are emitting, what direction any enabled indicator lights are emitting light, and / or according to what lighting pattern any enabled indicator lights are illuminated.

[0067] In some examples, state detection engine 230 can determine the state of an object by detecting and / or parsing one or more characters (e.g., text, numbers, symbols, logos, pictographs, and / or other character types), which can, for example, be part of a string of characters. In some examples, state detection engine 230 can detect and / or parse characters using optical character recognition (OCR). In some examples, state detection engine 230 can detect and / or parse the character(s) to determine the type of object, such as by detecting and / or parsing text that says "fire" written on the side of a fire truck object to determine that the object is a fire truck, such as Figure 5, and in the first state determination in the graphic representing state detection engine 230, as described above. In some examples, the object may be or may include a display (e.g., output device(s) 240, display(s) 340, display 440, and / or a display of output device 1035) that may display character(s), and state detection engine 230 may detect and / or parse the character(s) displayed on the display of the object.

[0068] In some examples, as part of determining the state of the object, the state detection engine 230 may determine and / or detect the pose of the object in the reprojected sensor data 225 and / or its changes (e.g., or by obtaining such determination from the object detector 215 based on the image data 210). The pose of the object may include the position of the object within the environment (e.g., 3D coordinates within the 3D volume of the environment) and / or the orientation of the object within the environment (e.g., pitch, yaw, and / or roll). In some examples, the pose of the object may also identify the poses of different parts of the object relative to other parts of the object. For example, if the object is a person, the pose of the object may also include a body pose, which may, for example, indicate the corresponding position and / or orientation of the person's torso, head, neck, arms, legs, hands, legs, and / or other body parts. If the object is a vehicle (such as a fire truck), the pose of the object may also include whether the door is open or closed, whether the ladder is extended or retracted, whether the lights or sirens are on, or a combination thereof. In some examples, the state of the object may also include changes in the pose of the object as detected using the state detection engine 230 and / or the object detector 215. For example, the state of an object may indicate whether a fire truck (or another object) is facing a certain direction (e.g., north, south, east, or west), whether the object fire truck (or other object) has an open or closed door (or other component in a certain pose or state), whether the fire truck (or other object) is turning, whether the object has fallen off one or more axes or otherwise rotated about one or more axes, whether the object is already within a particular geo-fenced area, whether the object has moved into or out of a particular geo-fenced area, whether the fire truck (or other object) is moving, whether the fire truck (or other object) is accelerating, whether the fire truck (or other object) is decelerating, whether the fire truck (or other object) is translating, whether a first portion of the object is moving relative to a second portion of the object (e.g., a person waving their arms or turning their head, a vehicle opening its door or rotating its wheels, etc.), or a combination thereof. In some embodiments, the pose of a particular portion of an object may also be tracked and may be used as the basis for state determination. For example, the state of an object may indicate how far a particular window or door of a fire truck is open, whether the driver and / or passengers are sitting in the fire truck, whether the driver has their arms properly positioned at the steering wheel, etc.

[0069] In some examples, the state detection engine 230 can determine the state of the object by detecting changes in the state of the object over time, for example, if the sensor data 210 and / or the reprojected sensor data 225 include multiple video frames of a video. Thus, the state detection engine 230 can determine the state of the object by detecting changes in the type of the object, changes in the appearance of the object, changes in the lighting characteristics of the light source included in the object, changes in (one or more) characters detected and / or parsed from the object, changes in the posture of at least a portion of the object, (one or more) translational movements of at least a portion of the object, (one or more) rotational movements of at least a portion of the object, changes in (one or more) translational movements of at least a portion of the object, changes in (one or more) rotational movements of at least a portion of the object, acceleration of (one or more) translational movements of at least a portion of the object, acceleration of (one or more) rotational movements of at least a portion of the object, changes in acceleration of (one or more) translational movements of at least a portion of the object, changes in acceleration of (one or more) rotational movements of at least a portion of the object, or a combination thereof. In some examples, the change in the appearance of the object can include a change in the shape of the object, movement of a portion of the object, modification of the object, etc. For example, the modification of the object can include applying paint to the object, applying stickers to the object, etc.

[0070] In some examples, the state detection engine 230 can determine the state of the object based on detecting any of the above-mentioned characteristics of the object and querying and / or cross-referencing the detected characteristics of the object using a data structure that can identify the state indicated by the characteristics. Examples of data structures include a lookup table (LUT), a database, a table, a tree, a ledger, a heap, a dictionary, or another type of data structure.

[0071] In an illustrative example, state detection engine 230 may detect that a light indicator on an object (e.g., a Wi-Fi router) is flashing red based on reprojected sensor data 225. State detection engine 230 may query a lookup table or other data structure to determine what a red flashing light indicator means, for example, to determine that a red flashing light indicator means that a connection between the object (e.g., a Wi-Fi router) and a modem has been terminated. Thus, state detection engine 230 may conclude that the state of the object (e.g., a Wi-Fi router) is that a connection between the object (e.g., a Wi-Fi router) and a modem has been terminated.

[0072] In another illustrative example, state detection engine 230 may detect, based on reprojected sensor data 225, that an object (e.g., a computer) includes a display that displays characters corresponding to an error code. State detection engine 230 may query a lookup table or other data structure to determine the meaning of the character(s) in the error code, for example, to determine that the character(s) in the error code means that the computer has experienced a crash. Thus, state detection engine 230 may infer that the state of the object (e.g., a computer) is that the object (e.g., a computer) has crashed.

[0073] State detection engine 230 may generate indicator 235 of the state of the object. Indicator 235 of the state of the object may indicate any type of state of the object described above. In some examples, indicator 235 of the state of the object may include at least a portion of reprojected sensor data 225. In some examples, indicator 235 of the state of the object may identify the object. Imaging system 200 outputs indicator 235, for example by sending indicator 235 to output device 240 of imaging system 200. Figure 2 2, a graphic representation of indicator 235 illustrates a rectangular box including reprojected sensor data 225, a warning symbol, a fire symbol representing the first determination described above, and a warning light symbol representing the second determination described above. In some examples, imaging system 200 may also output reprojected sensor data 225, such as by sending reprojected sensor data 225 to output device 240 of imaging system 200.

[0074] The imaging system 200 includes an output device 240. The output device(s) 240 may include one or more visual output devices, such as display(s) or connector(s) therefor. The output device(s) 240 may include one or more audio output devices, such as speaker(s), headphone(s), and / or connector(s) therefor. The output device(s) 240 may include one or more of the output device 1035 and / or the communication interface 1040 of the computing system 1000. The imaging system 200 causes a display of the output device 240 to display the indicator 235 (and / or the reprojected sensor data 225).

[0075] In some examples, output device 240 includes one or more transceivers. A transceiver may include a wired transmitter, a receiver, a transceiver, or a combination thereof. A transceiver may include a wireless transmitter, a receiver, a transceiver, or a combination thereof. A transceiver may include one or more of output device 1035 and / or communication interface 1040 of computing system 1000. In some examples, imaging system 200 causes (one or more) transceivers to send indicator 235 (and / or reprojected sensor data 225) to a recipient device. The recipient device may include a display, and data sent from the transceiver of output device 240 to the recipient device may cause the display of the recipient device to display indicator 235 (and / or reprojected sensor data 225).

[0076] In some examples, the display of the output device 240 of the imaging system 200 functions as an optical "see-through" display that allows light from the real-world environment (scene) surrounding the imaging system 200 to pass through (e.g., through) the display of the output device 240 to reach one or both eyes of the user. For example, the display(s) of the output device(s) 240 may be at least partially transparent, translucent, light-permissive, light-transmissive, or a combination thereof. In an illustrative example, the display of the output device 240 includes a transparent, translucent, and / or light-transmissive lens and a projector. The display of the output device 240 may include a projector that projects virtual content (e.g., the indicator 235 and / or the re-projected sensor data 225) onto the lens. The lens may be, for example, a lens of a pair of glasses, a lens of goggles, a contact lens, a lens of a head-mounted display (HMD) of a device, or a combination thereof. Light from the real-world environment passes through the lens and reaches one or both eyes of the user. The projector may project virtual content (e.g., indicator 235 and / or reprojected sensor data 225) onto the lens so that the virtual content appears overlaid on the user's view of the environment from the perspective of one or both eyes of the user. In some examples, the projector may project the virtual content onto one or both retinas of one or both eyes of the user, rather than onto the lens, which may be referred to as a virtual retinal display (VRD), a retinal scanning display (RSD), or a retinal projector (RP) display.

[0077] In some examples, the display of the output device 240 of the imaging system 200 is a digital "pass-through" display that allows a user of the imaging system 200 to see a view of the environment by displaying the view of the environment on the display of the output device 240. The view of the environment displayed on the digital pass-through display can be a view of the real-world environment surrounding the imaging system 200, for example based on sensor data (e.g., images, videos, depth images, point clouds, other depth data, or a combination thereof) captured by one or more environment-facing sensors in the sensor(s) 205, in some cases, as modified using an avatar processor 260 (e.g., indicators 235 and / or reprojected sensor data 225). The view of the environment displayed on the digital pass-through display can be a virtual environment (e.g., as in VR), which in some cases can include elements based on the real-world environment (e.g., the boundaries of a room). The view of the environment displayed on the digital pass-through display can be an augmented environment based on the real-world environment (e.g., as in AR). The view of the environment displayed on the digital pass-through display may be a hybrid environment based on the real-world environment (e.g., as in MR). The view of the environment displayed on the digital pass-through display may include virtual content (e.g., indicators 235 and / or reprojected sensor data 225) overlaid on other content that is otherwise incorporated into the view of the environment.

[0078] The trained ML model 250 may include one or more neural networks (NNs) (e.g., neural network 800), one or more convolutional neural networks (CNNs), one or more trained time delay neural networks (TDNNs), one or more deep networks, one or more autoencoders, one or more deep belief nets (DBNs), one or more recurrent neural networks (RNNs), one or more generative adversarial networks (GANs), one or more conditional generative adversarial networks (CGANs), transformer networks, one or more other types of neural networks, one or more trained support vector machines (SVMs), one or more trained random forests (RFs), one or more computer vision systems, one or more deep learning systems, or a combination thereof.

[0079] exist Figure 2 , a graph representing the trained ML model 250 shows a set of circles connected to each other. Each circle can represent a node (e.g., node 816), a neuron, a perceptron, a layer, a portion thereof, or a combination thereof. The circles are arranged in columns. The white circles in the leftmost column represent input layers (e.g., input layer 810). The white circles in the rightmost column represent output layers (e.g., output layer 814). The two columns of shaded circles between the leftmost column of white circles and the rightmost column of white circles each represent a hidden layer (e.g., hidden layers 812A-812N).

[0080] In some examples, the imaging system 200 includes a feedback engine 255. The feedback engine 255 can detect feedback received from a user interface of the imaging system 200. The feedback can include feedback about object detection by the object detector 215, reprojected sensor data 225, object state detection and / or indicator 235, output by the output device 240, or a combination thereof. The feedback engine 255 can detect feedback about one engine of the imaging system 200 received from another engine of the imaging system 200, such as feedback about whether one engine decides to use data from another engine. The feedback received by the feedback engine 255 can be positive feedback or negative feedback. For example, if one engine of the imaging system 200 uses data from another engine of the imaging system 200, or if positive feedback from a user is received through the user interface, the feedback engine 255 can interpret it as positive feedback. If one engine of the imaging system 200 rejects data from another engine of the imaging system 200, or if negative feedback from a user is received through the user interface, the feedback engine 255 can interpret it as negative feedback. Positive feedback can also be based on attributes of sensor data from sensor(s) 205, such as the user smiling, laughing, nodding, uttering a positive statement (e.g., "yes," "confirm," "ok," "next"), or otherwise reacting positively to an output of one of the engines described herein or an indication thereof. Negative feedback can also be based on attributes of sensor data from sensor(s) 205, such as the user frowning, crying, shaking his head (e.g., in a "no" motion), uttering a negative statement (e.g., "no," "negative," "bad," "not this"), or otherwise reacting negatively to an output of one of the engines described herein or an indication thereof.

[0081] In some examples, the feedback engine 255 provides feedback to one or more ML systems of the imaging system 200 as training data to update one or more trained ML models 250 of the imaging system 200. For example, the feedback engine 255 may provide feedback to the ML system and / or the trained ML model 250 as training data to update training for the object detector 215, the reprojection engine 220, the state detection engine 230, or a combination thereof. Positive feedback may be used to strengthen and / or reinforce weights associated with the output of the ML system(s) and / or the trained ML model(s) 250, and / or weaken or remove weights other than the weights associated with the output of the ML system(s) and / or the trained ML model(s) 250. Negative feedback may be used to weaken and / or remove weights associated with the output of the ML system(s) and / or the trained ML model(s) 250, and / or to strengthen and / or reinforce weights other than the weights associated with the output of the ML system(s) and / or the trained ML model(s) 250. In some examples, feedback engine 255 may be used to enable or disable various virtual cameras (eg, virtual camera 610 ) or perspectives based on various rules, logic, or input from a user interface.

[0082] In some examples, certain elements of imaging system 200 (e.g., object detector 215, reprojection engine 220, state detection engine 230, output device 240, trained ML model 250, feedback engine 255, or a combination thereof) include software elements, such as instruction sets corresponding to programs running on a processor (e.g., processor 1010 of computing system 1000, image processor 150, host processor 152, ISP 154, or a combination thereof). In some examples, these elements of imaging system 200 include one or more hardware elements, such as a dedicated processor (e.g., processor 1010 of computing system 1000, image processor 150, host processor 152, ISP 154, or a combination thereof). In some examples, these elements of imaging system 200 may include a combination of one or more software elements and one or more hardware elements.

[0083] Figure 3A340 is a perspective view showing a head mounted display (HMD) 310 used as part of the imaging system 200. The HMD 310 may be, for example, an augmented reality (AR) head mounted device, a virtual reality (VR) head mounted device, a mixed reality (MR) head mounted device, an extended reality (XR) head mounted device, or some combination thereof. The HMD 310 may be an example of the imaging system 200. The HMD 310 includes a first camera 330A and a second camera 330B along the front of the HMD 310. The first camera 330A and the second camera 330B may be examples of the sensor 205 of the imaging system 200. The HMD 310 includes a third camera 330C and a fourth camera 330D facing the eyes of the user when the eyes of the user face the display 340. The third camera 330C and the fourth camera 330D may be examples of the sensor 205 of the imaging system 200. In some examples, the HMD 310 may have only a single camera with a single image sensor. In some examples, the HMD 310 may include one or more additional cameras in addition to the first camera 330A, the second camera 330B, the third camera 330C, and the fourth camera 330D. In some examples, the HMD 310 may include one or more additional sensors in addition to the first camera 330A, the second camera 330B, the third camera 330C, and the fourth camera 330D, which may also include other types of sensors 205 of the imaging system 200. In some examples, the first camera 330A, the second camera 330B, the third camera 330C, and / or the fourth camera 330D may be examples of the image capture and processing system 100, the image capture device 105A, the image processing device 105B, or a combination thereof.

[0084] The HMD 310 may include one or more displays 340 visible to the user 320 wearing the HMD 310 on the head of the user 320. The one or more displays 340 of the HMD 310 may be an example of one or more displays of the output device 240 of the imaging system 200. In some examples, the HMD 310 may include one display 340 and two viewfinders. The two viewfinders may include a left viewfinder for the left eye of the user 320 and a right viewfinder for the right eye of the user 320. The left viewfinder may be oriented so that the left eye of the user 320 sees the left side of the display. The right viewfinder may be oriented so that the right eye of the user 320 sees the right side of the display. In some examples, the HMD 310 may include two displays 340, including a left display that displays content to the left eye of the user 320 and a right display that displays content to the right eye of the user 320. The one or more displays 340 of the HMD 310 may be a digital "pass-through" display or an optical "see-through" display. In some examples, one or more displays 340 of HMD 310 may display sensor data 210 , reprojected sensor data 225 , indicators 235 , or a combination thereof.

[0085] The HMD 310 may include one or more earphones 335, which may be used as speakers and / or headphones to output audio to one or more ears of a user of the HMD 310 and may be an example of an output device 240. Figure 3A and Figure 3B 310, one earphone 335 is shown, but it should be understood that the HMD 310 may include two earphones, one for each ear (left ear and right ear) of the user. In some examples, the HMD 310 may also include one or more microphones (not shown). In some examples, the audio output by the HMD 310 to the user through the one or more earphones 335 may include or be based on audio recorded using the one or more microphones.

[0086] Figure 3B is shown as being worn by the user 320 Figure 3A310 is a perspective view 350 of a head mounted display (HMD) of the user 320. The user 320 wears the HMD 310 on the head of the user 320 above the eyes of the user 320. The HMD 310 can capture images using a first camera 330A and a second camera 330B. In some examples, the HMD 310 displays one or more output images to the eyes of the user 320 using a display 340. In some examples, the output image may include reprojected sensor data 225 and / or indicators 235. The output image may be based on the image captured by the first camera 330A and the second camera 330B (e.g., sensor data 210), for example, overlaid with virtual content (e.g., reprojected sensor data 225 and / or indicators 235). The output image may provide a stereoscopic view of the environment, in some cases, overlaid with virtual content and / or with other modifications. For example, the HMD 310 may display a first display image to the right eye of the user 320, the first display image being based on the image captured by the first camera 330A. The HMD 310 may display a second display image to the left eye of the user 320, the second display image being based on an image captured by the second camera 330B. For example, the HMD 310 may provide overlaid virtual content in a display image overlaid on the image captured by the first camera 330A and the second camera 330B. The third camera 330C and the fourth camera 330D may capture images of the eyes of the user before, during, and / or after viewing the display image displayed by the display 340. In this way, sensor data from the third camera 330C and / or the fourth camera 330D may capture the reaction of the user's eyes (and / or other parts of the user) to the virtual content. The earphone 335 of the HMD 310 is shown in the ear of the user 320. The HMD 310 may output audio to the user 320 through the earphone 335 and / or through another earphone (not shown) of the HMD 310 in the other ear (not shown) of the user 320.

[0087] Figure 4A 4 is a perspective view 400 showing a front surface of a mobile handset 410 that includes a front-facing camera and that can be used as part of the imaging system 200. The mobile handset 410 can be an example of the imaging system 200. The mobile handset 410 can be, for example, a cellular phone, a satellite phone, a portable game console, a music player, a fitness tracking device, a wearable device, a wireless communication device, a laptop computer, a mobile device, any other type of computing device or computing system discussed herein, or a combination thereof.

[0088] The front surface 420 of the mobile handset 410 includes a display 440. The front surface 420 of the mobile handset 410 includes a first camera 430A and a second camera 430B. The first camera 430A and the second camera 430B may be examples of sensors 205 of the imaging system 200. The first camera 430A and the second camera 430B may face the user, including the user's eyes, while content (e.g., reprojected sensor data 225 and / or indicators 235) is displayed on the display 440. The display 440 may be an example of a display of the output device 240 of the imaging system 200.

[0089] The first camera 430A and the second camera 430B are illustrated in a bezel around the display 440 on the front surface 420 of the mobile handset 410. In some examples, the first camera 430A and the second camera 430B may be positioned in a notch or cutout cut out of the display 440 on the front surface 420 of the mobile handset 410. In some examples, the first camera 430A and the second camera 430B may be under-display cameras located between the display 440 and the rest of the mobile handset 410 so that light passes through a portion of the display 440 before reaching the first camera 430A and the second camera 430B. The first camera 430A and the second camera 430B of the perspective view 400 are front-facing cameras. The first camera 430A and the second camera 430B face in a direction perpendicular to the planar surface of the front surface 420 of the mobile handset 410. The first camera 430A and the second camera 430B may be two of one or more cameras of the mobile handset 410. In some examples, the front surface 420 of the mobile handset 410 may have only a single camera.

[0090] In some examples, the display 440 of the mobile handset 410 displays one or more output images to a user using the mobile handset 410. In some examples, the output images may include the reprojected sensor data 225 and / or the indicator 235. The output images may be based on images (e.g., sensor data 210) captured by the first camera 430A, the second camera 430B, the third camera 430C, and / or the fourth camera 430D, for example, overlaid with virtual content (e.g., reprojected sensor data 225 and / or the indicator 235). In some examples, the display 440 of the mobile handset 410 may display the sensor data 210, the reprojected sensor data 225, the indicator 235, or a combination thereof.

[0091] In some examples, the front surface 420 of the mobile handset 410 may include one or more additional cameras in addition to the first camera 430A and the second camera 430B. The one or more additional cameras may also be examples of sensors 205 of the imaging system 200. In some examples, the front surface 420 of the mobile handset 410 may include one or more additional sensors in addition to the first camera 430A and the second camera 430B. The one or more additional sensors may also be examples of sensors 205 of the imaging system 200. In some cases, the front surface 420 of the mobile handset 410 includes more than one display 440. The one or more displays 440 of the front surface 420 of the mobile handset 410 may be examples of displays of the output device 240 of the imaging system 200. For example, the one or more displays 440 may include one or more touch screen displays.

[0092] The mobile handset 410 may include one or more speakers 435A and / or other audio output devices (eg, earphones or headphones or connectors thereof) that may output audio to one or more ears of a user of the mobile handset 410 . Figure 4A 435A, but it should be understood that the mobile handset 410 may include more than one speaker and / or other audio devices. In some examples, the mobile handset 410 may also include one or more microphones (not shown). In some examples, the mobile handset 410 may include one or more microphones along and / or adjacent to the front surface 420 of the mobile handset 410, where these microphones are examples of sensors 205 of the imaging system 200. In some examples, the audio output by the mobile handset 410 to the user through the one or more speakers 435A and / or other audio output devices may include or be based on audio recorded using the one or more microphones.

[0093] Figure 4B 4 is a perspective view 450 showing a rear surface 460 of a mobile handset that includes a rear camera and can be used as part of the imaging system 200. The mobile handset 410 includes a third camera 430C and a fourth camera 430D on the rear surface 460 of the mobile handset 410. The third camera 430C and the fourth camera 430D of the perspective view 450 are rearward facing. The third camera 430C and the fourth camera 430D can be examples of the sensor 205 of the imaging system 200. The third camera 430C and the fourth camera 430D face in a direction perpendicular to the planar surface of the rear surface 460 of the mobile handset 410.

[0094] The third camera 430C and the fourth camera 430D may be two of the one or more cameras of the mobile handset 410. In some examples, the rear surface 460 of the mobile handset 410 may have only a single camera. In some examples, the rear surface 460 of the mobile handset 410 may include one or more additional cameras in addition to the third camera 430C and the fourth camera 430D. The one or more additional cameras may also be examples of the sensor 205 of the imaging system 200. In some examples, the rear surface 460 of the mobile handset 410 may include one or more additional sensors in addition to the third camera 430C and the fourth camera 430D. The one or more additional sensors may also be examples of the sensor 205 of the imaging system 200. In some examples, the first camera 430A, the second camera 430B, the third camera 430C, and / or the fourth camera 430D may be examples of the image capture and processing system 100, the image capture device 105A, the image processing device 105B, or a combination thereof.

[0095] The mobile handset 410 may include one or more speakers 435B and / or other audio output devices (eg, earphones or headphones or connectors thereof) that may output audio to one or more ears of a user of the mobile handset 410 . Figure 4B 435B, but it should be understood that the mobile handset 410 may include more than one speaker and / or other audio devices. In some examples, the mobile handset 410 may also include one or more microphones (not shown). In some examples, the mobile handset 410 may include one or more microphones along and / or adjacent to the back surface 460 of the mobile handset 410, where these microphones are examples of sensors 205 of the imaging system 200. In some examples, the audio output by the mobile handset 410 to the user through the one or more speakers 435B and / or other audio output devices may include or be based on audio recorded using the one or more microphones.

[0096] The mobile handset 410 can use the display 440 on the front surface 420 as a pass-through display. For example, the display 440 can display an output image, such as the reprojected sensor data 225 and / or the indicator 235. The output image can be based on the image captured by the third camera 430C and / or the fourth camera 430D (e.g., sensor data 210), for example, superimposed with virtual content (e.g., (e.g., reprojected sensor data 225 and / or indicator 235)). The first camera 430A and / or the second camera 430B can capture images of the user's eyes (and / or other parts of the user) before, during, and / or after the output image with virtual content is displayed on the display 440. In this way, the sensor data from the first camera 430A and / or the second camera 430B can capture the reaction of the user's eyes (and / or other parts of the user) to the virtual content.

[0097] Figure 5 5 is a flow chart illustrating an imaging process 500 that includes capturing an image 510 of an object 505 (a fire truck), reprojecting the image to generate a reprojected image 515 of the object 505, and outputting a status indicator 520 indicating a status of the object 505. A device having a camera 540 is shown capturing the image 510 of the object 505. The camera 540 is an example of the image capture and processing system 100, the image capture device 105A, the image processing device 105B, the image sensor 130, the sensor 205, the image sensor of any of the cameras 330A-330D, the image sensor of any of the cameras 430A-430D, the camera 740, the image sensor of the imaging process 900, the image sensor of the input device 1045, or a combination thereof.

[0098] An object detector 215 of an imaging system 200 (e.g., a device with camera 540, a user device 530, and / or another computing system 1000 associated with these devices) may detect an object 505 as depicted within an image 510. In some examples, the object detector 215 may generate a 3D model of the object 505 based on the image 510, including a mesh and / or a texture applied to the mesh. The object detector 215 may generate the 3D model of the object 505 based on depth data obtained using the camera 540 and / or other components of the device with camera 540. For example, the device with camera 540 may obtain depth data from a depth sensor of the device capturing the depth data and / or an additional camera other than camera 540 (e.g., the object detector 215 may be used to perform stereo depth detection and / or monocular depth estimation), and / or may capture multiple video frames including the image 510 (e.g., the object detector 215 may be used to perform video-based depth detection).

[0099] The reprojection engine 220 of the imaging system 200 (e.g., a device having a camera 540, a user device 530, and / or another computing system 1000 associated with these devices) can generate a reprojected image 515 of the object 505 based on the image 510 of the object 505. The image 510 of the object 505 is captured from a first perspective corresponding to the pose, field of view, and / or viewport of the camera 540. The reprojection engine 220 can reproject according to the configuration pose (e.g., position and / or orientation) of the virtual camera (e.g., virtual camera 610) so that the viewport and / or field of view of the virtual camera corresponds to the second perspective depicted in the reprojected image 515 of the object 505. Therefore, the reprojected image 515 of the object 505 simulates the image captured by the virtual camera from the perspective and field of view of the virtual camera. In some examples, the pose of the virtual camera can be determined using the object detector 215 and / or the reprojection engine 220. The pose and / or second perspective of the virtual camera can be object-locked, such that the pose of the virtual camera is locked in a specified pose relative to at least a portion of an object, even if the object moves relative to other parts of the environment. The pose and / or second perspective of the virtual camera can be region-locked, such that the virtual camera is locked in a specific pose relative to an area of ​​interest (e.g., of the object, the environment, sensor data 210, or a combination thereof), even if the area of ​​interest moves relative to other areas. The pose and / or second perspective of the virtual camera can be world-locked or environment-locked, such that the virtual camera is locked in a specific pose relative to at least a portion of the environment, scene, and / or world, even if the object moves relative to the environment, scene, and / or world (e.g., even if the object leaves the field of view). The pose and / or second perspective of the virtual camera can be locked to a texture, feature, corner, outline, spot, edge, floor, wall, ceiling, door, or a combination thereof.

[0100] Figure 6 600 is a conceptual diagram illustrating a viewport and / or field of view of a virtual camera associated with image reprojection. For example, virtual camera 610 is formed by Figure 5An example of a virtual camera used by a reprojection engine 220 of an imaging system 200 (e.g., a device with a camera 540, a user device 530, and / or another computing system 1000 associated with these devices) to generate a reprojected image 515 of an object 505 based on an image 510 of the object 505. In some examples, the reprojection engine 220 defines the virtual camera 610 using a pose (e.g., position and / or orientation) of the virtual camera 610 (e.g., relative to the object 505), a resolution (e.g., image resolution) of the virtual camera, a camera projection type (e.g., projection and / or orthogonal) of the virtual camera, a distortion model (e.g., a radial / tangential model, a distortion model, an isometric distortion model, or a combination thereof), a projection matrix of the virtual camera 610, or a combination thereof. The orientation of the virtual camera 610 can indicate a direction in which the virtual camera 610 is facing. For example, the virtual camera 610 points in a direction perpendicular or orthogonal to a surface of a side of the object 505.

[0101] Figure 5 The reprojection engine 220 of the imaging system 200 (e.g., a device with a camera 540, a user device 530, and / or another computing system 1000 associated with these devices) can calculate a static mapping of each pixel of the virtual camera 610 (e.g., each pixel of the reprojected image 515) to a 3D point on the surface of the object 505 (e.g., on a generated 3D model of the object 505 and / or on the surface of the object 505 depicted in the image 510). Given a known pose of the camera 540 (e.g., from tracking the movement of the camera 540 and / or its device based on the image 510), the reprojection engine 220 can project each 3D point from the image 510 to the virtual camera 610 and / or the reprojected image 515. For this projection, the reprojection engine 220 can use the 3D model for occlusion or occlusion mapping to determine whether any part of the object 505 will be occluded in the reprojected image 515, and therefore verify whether certain points are visible or occluded in the reprojected image 515. Using pixel-surface mapping and surface projection, the reprojection engine 220 can aggregate pixel values ​​(e.g., brightness values ​​and / or color values) of pixels to be discerned by the virtual camera 610 as a reprojected image 515. In some examples, the reprojection engine 220 performs reprojection of the image 510 using probabilistic reprojection to generate the reprojected image 515. In some examples, the reprojection engine 220 performs reprojection of the image 510 using one or more trained ML models 250 to generate the reprojected image 515.

[0102] To assist state detection engine 230 in detecting the state of an object, reprojection engine 220 may update reprojected image 515 with any changes in pixel values ​​(e.g., brightness values ​​and / or color values) of pixels of image 510 over time (e.g., if camera 540 is capturing image 510 as video frames in a video). Figure 5 The state detection engine 230 of the imaging system 200 (e.g., a device with camera 540, user device 530, and / or another computing system 1000 associated with these devices) can detect a change in luminosity of a portion of the object 505 and / or detect a change in color of a portion of the object 505. For example, detecting a change in luminosity can help detect whether a light source of the object 505 is disabled, enabled, or otherwise changes the lighting level, or other modifications to the appearance of the object 505. Detecting a change in color can help detect whether a light source of the object 505 changes the lighting color, or other modifications to the appearance of the object 505.

[0103] exist Figure 5 and Figure 6 In the context of Figure 5 The state detection engine 230 of the imaging system 200 (e.g., a device with a camera 540, a user device 530, and / or another computing system 1000 associated with these devices) can use the reprojected image 515 generated by the reprojection engine 220 to detect and / or parse (e.g., using OCR) a string 545 reading "FIRE" on the side surface of the object 505 (fire truck), indicating that the object 505 is a fire truck. The state detection engine 230 can generate a state indicator 520 and send the state indicator 520 to the user device 530 using the output device 240, and cause the user device 530 to output the state indicator 520 to the user 535. In some examples, the state indicator 520 can indicate that the object 505 is a fire truck. In some examples, the state indicator 520 can indicate that the object 505 includes a string 545 reading "FIRE" on its side surface. In some examples, the state indicator 520 can include at least a portion of the reprojected image 515 of the object 505.

[0104] In some examples, the appearance of character 545 and / or the appearance of the illustrated design of which character 545 is a part can be a change in the appearance of object 505 (compared to a previous image of object 505) detected by state detection engine 230 and indicated in state indicator 520. For example, in the illustrative example, a sticker or paint can be applied to object 505 to add character 545 and / or the illustrated design of which character 545 is a part to object 505. State indicator 520 can detect a change in the appearance of the object, particularly the addition of character 545 to object 505, and / or the addition of the illustrated design of which character 545 is a part (e.g., via a sticker and / or paint applied to object 505). State indicator 520 generated using state detection engine 230 can indicate such a change in the appearance of the object to user device 530 and / or user 535.

[0105] In some examples, the device including the camera 540 is different from the user device 530. Thus, even if the user 535 and the user device 530 are far away from the device with the camera 540, the user 535 using the user device can track the state of the object 505 via the state indicator 520. In some examples, the user device 530 includes the camera 540. Examples of user devices 530 and / or devices having the camera 540 include the image capture and processing system 100, the image capture device 105A, the image processing device 105B, the HMD 310, the mobile handset 410, the execution Figure 7 Imaging process 700 imaging system, neural network 800, execution Fig. 9 An imaging system, a computing system 1000, or a combination thereof, for imaging process 900.

[0106] Figure 7700 includes capturing an image 710 of an object 705 (computer), reprojecting the image to generate a reprojected image 715 of the object 705, and outputting a status indicator 720 indicating a status of the object 705. A device having a camera 740 is shown capturing the image 710 of the object 705. The camera 740 is an example of the image capture and processing system 100, the image capture device 105A, the image processing device 105B, the image sensor 130, the sensor 205, the image sensor of any of the cameras 330A-330D, the image sensor of any of the cameras 430A-430D, the camera 540, the image sensor of the imaging process 900, the image sensor of the input device 1045, or a combination thereof. The object detector 215 of the imaging system 200 (e.g., the device having the camera 740, the user device 730, and / or another computing system 1000 associated with these devices) can detect the object 705 (computer) as depicted within the image 710. Object detector 215 may generate a 3D model of object 705 based on image 710 and / or corresponding depth data, including a mesh and / or a texture applied to the mesh.

[0107] The reprojection engine 220 of the imaging system 200 (e.g., a device having a camera 740, a user device 730, and / or another computing system 1000 associated with these devices) can generate a reprojected image 715 of the object 705 based on the image 710 of the object 705, for example using probabilistic reprojection and / or the trained ML model 250 as described herein. The reprojected image 715 of the object 705 depicts the object 705 from the perspective of the virtual camera corresponding to the field of view of the virtual camera. The image 710 of the object 705 is captured from a first perspective corresponding to the field of view of the camera 740. The reprojection engine 220 can reproject according to a configured pose (e.g., position and / or orientation) of a virtual camera having a viewport and / or field of view corresponding to a second perspective depicted in the reprojected image 715 of the object 705. Thus, the reprojected image 715 of the object 705 simulates an image captured by the virtual camera from the perspective and field of view of the virtual camera. The orientation portion of the pose of the virtual camera defines and / or indicates the direction the virtual camera is facing (e.g., relative to the object 505). For example, Figure 7 The virtual camera in points in a direction perpendicular or orthogonal to the surface of the monitor of the object 705 (computer).

[0108] To assist the state detection engine 230 in detecting the state of the object, the reprojection engine 220 may update the reprojected image 715 using any changes in pixel values ​​(e.g., brightness values ​​and / or color values) of pixels of the image 710 over time (e.g., if the camera 740 is capturing the image 710 as a video frame in a video). Figure 7The state detection engine 230 of the imaging system 200 (e.g., a device with a camera 740, a user device 730, and / or another computing system 1000 associated with these devices) can use the reprojected image 715 generated by the reprojection engine 220 to detect a warning 745 displayed on the display of the object 705 (computer). For example, the state detection engine 230 can detect and / or parse (e.g., using OCR) one or more characters of the warning 745 displayed on the display of the object 705 (computer). The state detection engine 230 can detect a change in color displayed on the display of the object 705 (computer) associated with the warning 745, such as a blue or similar color change of a "blue screen" error message. The state detection engine 230 can detect other changes in lighting characteristics (e.g., lighting level, lighting color, lighting direction, and / or lighting pattern) from the display of the object 705 (computer), where the display is considered to be a light source type as described above. In some examples, state detection engine 230 may determine the meaning of aspects of warning 745 by querying a lookup table or other data structure for aspects of warning 745 (e.g., characters of an error code detected and / or parsed using OCR, colors displayed using a display, or a combination thereof).

[0109] The state detection engine 230 may generate a state indicator 720 and send the state indicator 720 to the user device 730 using the output device 240, and cause the user device 730 to output the state indicator 720 to the user 735. In some examples, the state indicator 720 may indicate that the object 705 is a computer. In some examples, the state indicator 720 may indicate that a warning 745 given by the object 705 is detected. In some examples, the state indicator 720 may indicate any characters detected and / or parsed (e.g., using OCR) in the warning 745, such as an error code. In some examples, the state indicator 720 may indicate the color displayed by the display of the object 705 (computer), such as the blue color of a "blue screen" error message. In some examples, the state indicator 720 may indicate the meaning of an aspect of the warning 745 (e.g., the characters of the error code detected and / or parsed using OCR, the color displayed using the display, or a combination thereof), as determined by querying a lookup table or other data structure for aspects of the warning 745. In some examples, status indicator 720 can include at least a portion of reprojected image 715 of object 705. In some examples, status indicator 720 can include at least a portion of warning 745.

[0110] In some examples, the appearance of warning 745 can be a change in the appearance of object 705 (compared to a previous image of object 705) detected by state detection engine 230 and indicated in state indicator 720. For example, in the illustrative example, the display of object 705 (computer) may have previously displayed other content before displaying warning 745. State indicator 720 can detect the change in the appearance of the object, and in particular, the change of the display of object 705 (computer) from displaying other content to displaying warning 745. State indicator 720 generated using state detection engine 230 can indicate such a change in the appearance of the object to user device 730 and / or user 735.

[0111] In some examples, the device including the camera 740 is different from the user device 730. Thus, even if the user 735 and the user device 730 are far away from the device having the camera 740, the user 735 using the user device can track the state of the object 705 via the state indicator 720. In some examples, the user device 730 includes the camera 740. Examples of the user device 730 and / or examples of devices having the camera 740 include the image capture and processing system 100, the image capture device 105A, the image processing device 105B, the HMD 310, the mobile handset 410, the execution Figure 5 Imaging process 500 imaging system, neural network 800, execution Fig. 9 An imaging system, a computing system 1000, or a combination thereof, for imaging process 900.

[0112] Figure 8 800 that can be used for media processing operations. The neural network 800 can include any type of deep network, such as a convolutional neural network (CNN), an autoencoder, a deep belief network (DBN), a recurrent neural network (RNN), a generative adversarial network (GAN), a transformer, and / or other types of neural networks. The neural network 800 can be an example of one of the trained ML models 250. The trained ML model 250 can be an example of the neural network 800. The neural network 800 can be used by the object detector 215, the reprojection engine 220, the state detection engine 230, the trained ML model 250, the feedback engine 255, or a combination thereof.

[0113] The input layer 810 of the neural network 800 includes input data. The input data of the input layer 810 may include data representing pixels of one or more input image frames. In some examples, the input data of the input layer 810 includes data representing pixels of image data, such as an image captured by the image capture and processing system 100, sensor data 210 captured by the sensor 205, reprojected sensor data 225 generated using the reprojection engine 220, an image captured by one of the cameras 330A-330D, an image captured by one of the cameras 430A-430D, image 510, image 710, image data received in operation 905, image data captured using an image sensor of the input device 1045, or a combination thereof.

[0114] The image may include image data from an image sensor, the image data including raw pixel data (including, for example, a single color per pixel based on a Bayer filter) or processed pixel values ​​(e.g., RGB pixels of an RGB image). The neural network 800 includes a plurality of hidden layers 812A, 812B through 812N. The hidden layers 812A, 812B through 812N include "N" hidden layers, where "N" is an integer greater than or equal to 1. The number of hidden layers may include as many layers as required for a given application. The neural network 800 also includes an output layer 814 that provides an output generated by the processing performed by the hidden layers 812A, 812B through 812N.

[0115] In some examples, the output layer 814 may provide output images and / or media data sets, such as a 3D mesh of a 3D model generated using the object detector 215, a texture of a 3D model generated using the object detector 215, a 3D model generated using the object detector 215, reprojected sensor data 225 generated using the reprojection engine 220, an indicator 235 generated using the state detection engine 230, content output using the output device 240, content displayed on the display 340, content displayed on the display 440, a reprojected image 515, a state indicator 520, a reprojected image generated from the perspective of the virtual camera 610, a reprojected image 715, a state indicator 720, reprojected image data generated in operation 915, an indicator of an object state output in operation 920, output data output using the output device 1035, or a combination thereof.

[0116] Neural network 800 is a multi-layer neural network of interconnected filters. Each filter can be trained to learn features that represent input data. Information associated with the filters is shared between different layers, and each layer retains information as it is processed. In some cases, neural network 800 may include a feedforward network, in which case there is no feedback connection in which the output of the network is fed back into itself. In some cases, network 800 may include a recurrent neural network, which may have loops that allow information to be carried across nodes as input is read in.

[0117] In some cases, information can be exchanged between layers by node-to-node interconnection between each layer. In some cases, the network may include a convolutional neural network, which may not link each node in a layer to each other node in the next layer. In a network that exchanges information between layers, the nodes of the input layer 810 can activate the node set in the first hidden layer 812A. For example, as shown in the figure, each input node of the input layer 810 can be connected to each node of the first hidden layer 812A. The nodes of the hidden layer can transform the information by applying an activation function (e.g., signal mapping, truncation function, ReLU, ReLU6, sigmoid, hyperbolic tangent and / or softmax) and / or filter to the information of each input node. Then, the information derived from the transformation can be passed to the nodes of the next hidden layer 812B and the nodes of the next hidden layer 812B can be activated, and the nodes of the next hidden layer 812B can perform their own specified functions. Example functions include convolution functions, reduction, enlargement, data transformation and / or any other suitable functions. Then, the output of the hidden layer 812B can activate the nodes of the next hidden layer, and so on. The output of the last hidden layer 812N may activate one or more nodes of the output layer 814, which provide a processed output image. In some cases, although a node in the neural network 800 (e.g., node 816) is shown as having multiple output lines, the node has a single output, and all lines shown as output from the node represent the same output value.

[0118] In some cases, each node or interconnection between nodes can have a weight that is a set of parameters derived from the training of the neural network 800. For example, an interconnection between nodes can represent a piece of information learned about the interconnected nodes. The interconnections can have tunable digital weights that can be tuned (e.g., based on a training data set), allowing the neural network 800 to adapt to the input and be able to learn as more and more data is processed.

[0119] The neural network 800 is pre-trained to process features of data from the input layer 810 using different hidden layers 812A, 812B to 812N to provide an output through the output layer 814.

[0120] Fig. 9 900 is a flow chart illustrating an imaging process. The imaging process 900 may be performed by an imaging system. In some examples, the imaging system may include, for example, an image capture and processing system 100, an image capture device 105A, an image processing device 105B, an image processor 150, an ISP 154, a host processor 152, an imaging system 200, an object detector 215, a reprojection engine 220, a state detection engine 230, an output device 240, a trained ML model 250, a feedback engine 255, an HMD 310, a mobile handset 410, an execution Figure 5 The imaging process 500 includes an imaging system, a user device 530, a camera 540, and an execution Figure 7 The imaging process 700 may include an imaging system, a user device 730, a camera 740, a neural network 800, a computing system 1000, or a combination thereof.

[0121] At operation 905, an imaging system is configured to and may receive image data of an environment. The image data represents (eg, depicts) the environment from a first perspective.

[0122] In some examples, the imaging system is configured to and can receive image data from an image sensor. In some examples, the imaging system includes an image sensor connector that couples and / or connects the image sensor to the rest of the imaging system (e.g., including a processor and / or memory of the imaging system), and in some examples, the imaging system receives image data from the image sensor by receiving the image data from, through, and / or using the image sensor connector.

[0123] Examples of image sensors include image sensor 130, sensor 205, first camera 330A, second camera 330B, third camera 330C, fourth camera 330D, first camera 430A, second camera 430B, third camera 430C, fourth camera 430D, an image sensor of camera 540, an image sensor of camera 740, an image sensor for capturing images used as input data for input layer 810 of NN 800, input device 1045, another image sensor described herein, another sensor described herein, or a combination thereof.

[0124] Examples of image data include image data captured using image capture and processing system 100, sensor data 210 captured using an image sensor of sensor 205, image data captured using first camera 330A, image data captured using second camera 330B, image data captured using third camera 330C, image data captured using fourth camera 330D, image data captured using first camera 430A, image data captured using second camera 430B, image data captured using third camera 430C, image data captured using fourth camera 430D, image 510, image 710, an image used as input data for input layer 810 of NN800, an image captured using input device 1045, another image described herein, another set of image data described herein, or a combination thereof.

[0125] At operation 910, the imaging system is configured to and can detect objects in the environment in the image data. In some examples, the imaging system can use object detector 215, trained ML model(s) 250, NN 800, or a combination thereof to detect objects in the image data. Examples of objects include objects in scene 110, Figure 2 A fire truck, object 505, and object 705 are shown.

[0126] At operation 915, the imaging system is configured and may generate re-projected image data of at least a portion of the environment based on the image data, the re-projected image data representing at least an object from a second perspective different from the first perspective.

[0127] Examples of reprojected image data include reprojected sensor data 225, indicator 235, reprojected image data displayed using display 340, reprojected image data displayed using display 440, reprojected image 515, a reprojected image simulating the perspective of virtual camera 610, reprojected image 715, a reprojected image generated using NN 800, or a combination thereof.

[0128] In some aspects, the imaging system is configured to and can receive input indicating a second perspective using a user interface. In some examples, the input can indicate an object, and the second perspective can be orthogonal to at least a portion of the surface of the object. In some examples, the input can include a reference image depicting the environment from the second perspective, and the imaging system can use the reference image to determine the second perspective. In some examples, the input can include a request from a user to translate, sweep, rotate, and / or otherwise move (e.g., in a specific direction and / or a specific distance, angle, or amount) a viewport of a virtual camera (e.g., virtual camera 610) relative to the object and / or relative to the environment.

[0129] In some aspects, the imaging system is configured and can generate gesture information corresponding to the second perspective based on the location of features of the object in the environment. In some examples, based on detecting the object in operation 910, the imaging system can automatically generate gesture information for the second perspective so that the second perspective points in a direction orthogonal to at least a portion of the surface of the object. In some examples, the imaging system is configured and can detect an occlusion that blocks the view of the object in the occlusion image data, and the imaging system can automatically generate gesture information for the second perspective so that the occlusion does not block the view of the object in the reprojected image data.

[0130] In some aspects, generating reprojected image data of at least a portion of the environment includes generating the reprojected image data using probabilistic reprojection (e.g., using reprojection engine 220). In some aspects, generating reprojected image data of at least a portion of the environment according to the second perspective includes using an output of a trained machine learning model responsive to input of the image data and / or geometry of the object into the trained machine learning model (e.g., using reprojection engine 220, trained ML model 250, and / or NN 800).

[0131] In some aspects, the imaging system is configured to and can receive previous image data of the environment. The imaging system can determine that the image data lacks visual information of a part of the object. The part of the object is discernible from the second viewing angle. In operation 915, the reprojected image data of at least a part of the environment generated can include visual information of a part of the object reconstructed based on the previous image data. For example, the image 510 of the object 505 is captured from the first viewing angle with a steep angle, which may make the part of the character 545 blurred, unreadable or missing. In this case, the imaging system can retrieve the previous image data of the object 505, and use the previous image data of the object 505 to fill any gaps or missing image data, and / or replace or improve the low-quality image data from the image 510 to generate the reprojected image 515.

[0132] In some aspects, the imaging system is configured to and can determine that the image data lacks visual information of a portion of an object. The portion of the object is discernible from a second perspective. Generating reprojected image data of at least a portion of the environment in operation 915 may include using the output of the interpolated and / or trained machine learning model in response to inputting the geometry of the image data and / or the object into a trained machine learning model (e.g., trained ML model 250 and / or NN 800) to reconstruct visual information of a portion of the object. For example, an image 510 of an object 505 is captured from a first perspective with a steep angle, which may make a portion of a character 545 blurry, unreadable, or missing. In this case, the imaging system can use interpolation and / or a trained ML model to fill in any gaps or missing image data, and / or replace or improve low-quality image data from image 510 to generate a reprojected image 515.

[0133] In some aspects, the imaging system is configured to and can generate a mask associated with an occlusion on a portion of an object, where the image data lacks visual information of the portion of the object because the occlusion obscures the portion of the object in the image data. This can ensure that the imaging system does not consider the occlusion to be part of the object of the imaging system and therefore does not include the occlusion in the reprojected image data. Instead, the imaging system can fill in the missing image data, where the mask (which indicates where the occlusion is located) is indicated based on previous image data, interpolation, and / or a trained ML model, as described above.

[0134] In some aspects, the imaging system is configured and can determine the lighting scheme (e.g., in the image data) that illuminates the environment. Generating reprojected image data of at least a portion of the environment can include reconstructing the lighting scheme according to the second perspective. For example, if the object 505 is illuminated from the right in the image 510, the imaging system can detect this and reconstruct the lighting scheme in the reprojected image 515 to ensure that the object 505 is also depicted as being illuminated from the right in the reprojected image 515. In some examples, reconstructing the lighting scheme can include applying gradient lighting, generating shadows, etc. In some aspects, the imaging system is configured and can modify the image data to normalize the lighting in the image data before generating reprojected image data of at least a portion of the environment based on the image data. For example, if the image data includes lighting effects (e.g., shadows, glare, and / or other lighting effects) that make important parts of the object difficult to discern (e.g., making the characters 545 difficult to read), the lighting in the normalized image data can reduce or remove such lighting effects in the image data, and therefore can also reduce or remove such lighting effects in the reprojected image data.

[0135] In some aspects, the imaging system is configured to and can output the reprojected image data. In some aspects, the imaging system is configured to and can generate and / or output an indicator of the state of the object based on the reprojected image data. Examples of indicators include indicator 235, an indicator displayed on display 340, an indicator displayed on display 440, a state indicator 520, a state indicator 720, a state indicator generated using NN 800, or a combination thereof.

[0136] In some aspects, the imaging system is configured to and can monitor the state of the object over a period of time based on the reprojected image data. Both the image data and the reprojected image data may include images corresponding to different times during the time period. The imaging system may identify changes in the state of the object during the time period. The indicator of the state of the object indicates a change in the state of the object. In some aspects, the change in the state of the object includes a change in the lighting characteristics of the light source, wherein the object includes the light source. For example, the display screen of the object 705 (computer) may be an example of a light source, and the warning 745 displayed on the display of the object 705 (computer) may be an example of a change in the lighting characteristics of the light source and / or a change in the state of the object. In some aspects, the imaging system is configured to and can use optical character recognition (OCR) to parse one or more characters on the object in the reprojected image data, wherein the change in the state of the object is associated with the one or more characters. For example, the one or more characters may include one or more characters of the character 545 and / or the warning 745. In some aspects, the change in the state of the object includes a change in the motion of at least a portion of the object. For example, the motion of at least a portion of an object may include movement of object 505 (fire truck) while object 505 is being driven, movement of wheels of object 505 (fire truck), or movement of a visual element (e.g., warning 745) displayed on a display screen of object 705 (computer).

[0137] In some aspects, image data is received from at least one image sensor, such as any of the examples of image sensors listed above. In some aspects, a first subset of image data is received from a first image sensor, and a second subset of image data is received from a second image sensor. The first image sensor and the second image sensor may be image sensors of different cameras of an imaging system, such as in the first camera 330A and the second camera 330B of the HMD 310 or the cameras 430A-430D of the mobile handset 410. In some aspects, detecting the object in the image data includes tracking the object from the first subset of the image data to the second subset of the image data. For example, an object may leave the field of view of the first image sensor and enter the field of view of the second image sensor, or vice versa, based on movement of the object itself (e.g., for a movable object such as the object 505), based on movement of the imaging system that causes the image sensor to move (e.g., movement of the HMD 310 and / or the mobile handset 410), or a combination thereof.

[0138] In some aspects, the imaging system is configured to and may cause a display (e.g., output device 240, output device 1035) to display information based on the reprojected image data. In some aspects, the imaging system includes a display. In some aspects, the imaging system is configured to and may cause a communication interface (e.g., output device 240, output device 1035, communication interface 1040) to send information based on the reprojected image data to a recipient device. In some examples, the imaging system includes a communication interface.

[0139] In some examples, an imaging system may include: a device for receiving image data of an environment, the image data representing the environment from a first perspective; a device for detecting an object in the environment in the image data; and a device for generating reprojected image data of at least a portion of the environment based on the image data, the reprojected image data representing at least the object from a second perspective different from the first perspective.

[0140] In some examples, an apparatus for receiving image data includes image capture and processing system 100, image capture device 105A, image processing device 105B, image processor 150, ISP 154, host processor 152, image sensor 130, sensor 205, first camera 330A, second camera 330B, third camera 330C, fourth camera 330D, first camera 430A, second camera 430B, third camera 430C, fourth camera 430D, an image sensor of camera 540, an image sensor of camera 740, an image sensor for capturing an image used as input data for input layer 810 of NN 800, input device 1045, another image sensor described herein, another sensor described herein, or a combination thereof.

[0141] In some examples, an apparatus for detecting an object includes an image capture and processing system 100, an image processing device 105B, an image processor 150, an ISP 154, a host processor 152, an imaging system 200, an object detector 215, (one or more) trained ML models 250, a NN 800, a computing system 1000, or a combination thereof.

[0142] In some examples, an apparatus for generating a reprojected image includes an image capture and processing system 100, an image processing device 105B, an image processor 150, an ISP 154, a host processor 152, an imaging system 200, a reprojection engine 220, a trained ML model 250, a NN 800, a computing system 1000, or a combination thereof.

[0143] In some examples, the processes described herein (e.g., Figure 1 , 2 , 5, 7, 8 corresponding process, Fig. 9 In some examples, the processes described herein may be performed by the image capture and processing system 100, the image capture device 105A, the image processing device 105B, the image processor 150, the ISP 154, the host processor 152, the imaging system 200, the object detector 215, the reprojection engine 220, the state detection engine 230, the output device 240, the trained ML model 250, the feedback engine 255, the HMD 310, the mobile handset 410, the execution Figure 5 The imaging process 500 includes an imaging system, a user device 530, a camera 540, and an execution Figure 7 Imaging process 700 of the imaging system, user equipment 730, camera 740, neural network 800, execution Fig. 9 The imaging process 900 is performed by the imaging system, the computing system 1000, or a combination thereof.

[0144] The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smart watch or other wearable device), a server computer, an autonomous vehicle or a computing device of an autonomous vehicle, a robotic device, a television, and / or any other computing device having the resource capability to perform the processes described herein. In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to transmit and / or receive data, any combination thereof, and / or other components. The network interface may be configured to transmit and / or receive data based on an Internet Protocol (IP) or other types of data.

[0145] Components of a computing device may be implemented in circuits. For example, a component may include and / or be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0146] The processes described herein are illustrated as logical flow diagrams, block diagrams, or conceptual diagrams, the operations of which represent a series of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, an operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the operation. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform a specific function or implement a specific data type. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0147] In addition, the processes described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, by hardware, or a combination thereof. As described above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0148] Fig.10 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. In particular, Fig.10 An example of a computing system 1000 is shown, which may be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using connection 1005. Connection 1005 may be a physical connection using a bus, or a direct connection into processor 1010, such as in a chipset architecture. Connection 1005 may also be a virtual connection, a networked connection, or a logical connection.

[0149] In some aspects, computing system 1000 is a distributed system, where the functionality described in the present disclosure may be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some aspects, one or more of the described system components represent a number of such components, each component performing some or all of the functionality for which the component is described. In some aspects, a component may be a physical or virtual device.

[0150] The example system 1000 includes at least one processing unit (CPU or processor) 1010 and connections 1005 that couple various system components including system memory 1015, such as read-only memory (ROM) 1020 and random access memory (RAM) 1025, to the processor 1010. The computing system 1000 may include a cache 1012 of high-speed memory directly connected to, in close proximity to, or integrated as a part of the processor 1010.

[0151] Processor 1010 may include any general purpose processor and hardware services or software services, such as services 1032, 1034, and 1036 stored in storage device 1030, configured to control processor 1010 as well as special purpose processors where the software instructions are incorporated into the actual processor design. Processor 1010 may essentially be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0152] To enable user interaction, computing system 1000 includes input device 1045, which can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. Computing system 1000 can also include output device 1035, which can be one or more of a plurality of output mechanisms. In some cases, a multimodal system can enable a user to provide multiple types of input / output to communicate with computing system 1000. Computing system 1000 can include communication interface 1040, which can generally govern and manage user input and system output. The communication interface may use wired and / or wireless transceivers to perform or facilitate receiving and / or transmitting wired or wireless communications, including those utilizing audio jacks / plugs, microphone jacks / plugs, universal serial bus (USB) ports / plugs, ports / plugs, Ethernet ports / plugs, fiber optic ports / plugs, proprietary wired ports / plugs, wireless signal transmission, Bluetooth Low Energy (BLE) wireless signal transmission, wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, Wi-Fi wireless signal transmission, visible light communication (VLC), world interoperability for microwave access (WiMAX), infrared (IR) communication wireless signal transmission, etc. Public Switched Telephone Network (PSTN) Signal Transmission In one embodiment, wireless signal transmission may include wireless signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad-hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 1040 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 1000 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States-based Global Positioning System (GPS), the Russian-based Global Navigation Satellite System (GLONASS), the China-based Beidou Navigation Satellite System (BDS), and the European-based Galileo GNSS. There is no limitation on operating on any particular hardware arrangement, and therefore the basic features here can be easily replaced with improved hardware or firmware arrangements when improved hardware or firmware arrangements are developed.

[0153] The storage device 1030 may be a non-volatile and / or non-transitory and / or computer-readable memory of a device and may be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as a magnetic cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / magnetic stripe, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disk read-only memory (CD-ROM) disc, a rewritable compact disk (CD) disc, a digital video disk (DVD) disc, a Blu-ray disc (BDD) disc, a holographic disc, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a subscriber identity module (SI) M) card, mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM (FLASHEPROM), cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random access memory (RRAM / ReRAM), phase change memory (PCM), spin transfer torque RAM (STT-RAM), another memory chip or box and / or a combination thereof.

[0154] Storage device 1030 may include software services, servers, services, etc., which cause the system to perform functions when the code defining such software is executed by processor 1010. In some aspects, hardware services that perform a particular function may include software components stored in a computer-readable medium in combination with the necessary hardware components (e.g., processor 1010, connection 1005, output device 1035, etc.) to perform that function.

[0155] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data may be stored and which do not include carrier waves and / or transient electronic signals that are propagated wirelessly or via a wired connection. Examples of non-transitory media may include, but are not limited to, disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memories, memories, or memory devices. Computer-readable media may have stored thereon codes and / or machine-executable instructions that may represent programs, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted using any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0156] In some aspects, computer readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0157] Specific details are provided in the above description to provide a thorough understanding of the aspects and examples provided herein. However, it will be understood by those skilled in the art that these aspects can be practiced without these specific details. For clarity, in some cases, the present technology can be presented as including separate functional blocks, which include functional blocks including steps or routines in methods that include devices, device components, software or a combination of hardware and software. In addition to those components shown in the drawings and / or described herein, additional components can be used. For example, circuits, systems, networks, processes and other components can be shown as components in block diagram form, so as not to obscure these aspects with unnecessary details. In other examples, known circuits, processes, algorithms, structures and techniques can be shown without unnecessary details, so as to avoid blurring these aspects.

[0158] Various aspects may be described above as a process or method depicted as a flowchart, flow diagram, data flow diagram, structure diagram, or block diagram. Although a flowchart may describe an operation as a sequential process, many operations may be performed in parallel or simultaneously. In addition, the order of the operations may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the figure. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to a calling function or a main function.

[0159] The processes and methods according to the above examples can be implemented using computer executable instructions stored in or otherwise available from computer readable media. Such instructions may include, for example, instructions and data that enable or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a specific function or group of functions. Parts of the computer resources used may be accessed over a network. Computer executable instructions may be, for example, binary files, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the method according to the described examples include disks or optical disks, flash memory, USB devices provided with non-volatile memory, networked storage devices, etc.

[0160] Devices implementing the processes and methods disclosed herein may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may be implemented in any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. (One or more) processors may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functions described herein may also be embodied in peripheral devices or add-on cards. As another example, such functions may also be implemented between different chips or different processes executed in a single device on a circuit board.

[0161] The instructions, the media for carrying such instructions, the computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functions described in this disclosure.

[0162] In the foregoing description, various aspects of the present application are described with reference to their specific aspects, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the illustrative aspects of the present application have been described in detail herein, it should be understood that the inventive concept can be implemented and adopted differently in other ways, and the appended claims are intended to be interpreted as including such variations, except for being limited by the prior art. The various features and aspects of the above-mentioned application can be used individually or in combination. In addition, without departing from the broader spirit and scope of this specification, various aspects can be utilized in any number of environments and applications outside the environment and application described herein. Therefore, the description and the accompanying drawings are considered to be illustrative rather than restrictive. For illustrative purposes, the method is described in a specific order. It should be understood that, in alternative aspects, the method can be performed in an order different from the described order.

[0163] One of ordinary skill will understand that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by the less than or equal to ("<") and greater than or equal to ("<") symbols, respectively, without departing from the scope of the present specification.

[0164] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform those operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform those operations, or any combination thereof.

[0165] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is in direct or indirect communication with another component (e.g., connected to another component via a wired or wireless connection, and / or other suitable communication interface).

[0166] Claim language or other language reciting "at least one of" a set and / or "one or more of" a set indicates that one member of a set or multiple members of a set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0167] The various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the aspects disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above with respect to their functionality. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. The technician may implement the described functions in different ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0168] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general-purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses (including applications in wireless communication device handsets and other devices). Any features described as modules or components may be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions for performing one or more of the methods described above when executed. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (e.g., synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the techniques may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0169] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. This processor may be configured to perform any of the techniques described in the present invention. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors and a DSP core, or any other such configuration. Therefore, as used herein, the term "processor" may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or implementation of the device suitable for the technology described herein. In addition, in some aspects, the functionality described herein may be provided in a dedicated software module or hardware module configured for encoding and decoding, or incorporated into a combined video encoder-decoder (CODEC).

[0170] Illustrative aspects of the disclosure include:

[0171] Aspect 1: A device for media processing, the device comprising: a memory; and one or more processors coupled to the memory, the one or more processors being configured to: receive image data of an environment, the image data representing the environment from a first perspective; detect an object in the environment in the image data; and generate reprojected image data of at least a portion of the environment based on the image data, the reprojected image data representing at least the object from a second perspective different from the first perspective.

[0172] Aspect 2. The apparatus according to aspect 1, wherein the at least one processor is configured to: receive input using a user interface, wherein the input indicates the second viewing angle.

[0173] Aspect 3. An apparatus according to any one of Aspects 1 to 2, wherein the at least one processor is configured to: generate posture information corresponding to the second perspective based on the position of the feature of the object in the environment.

[0174] Aspect 4. An apparatus according to any one of Aspects 1 to 3, wherein, in order to generate reprojected image data of at least a portion of the environment according to the second perspective, at least one processor is configured to generate the reprojected image data using probabilistic reprojection.

[0175] Aspect 5. An apparatus according to any one of Aspects 1 to 4, wherein, in order to generate reprojected image data of at least a portion of the environment according to a second perspective, at least one processor is configured to use an output of a trained machine learning model in response to an input of the image data into the trained machine learning model.

[0176] Aspect 6. An apparatus according to any one of Aspects 1 to 5, wherein the at least one processor is configured to: receive previous image data of the environment; and determine that the image data is missing visual information of a portion of the object, wherein the portion of the object is discernible from a second perspective, wherein, in order to generate reprojected image data of at least the portion of the environment according to the second perspective, the at least one processor is configured to reconstruct the visual information of the portion of the object based on the previous image data.

[0177] Aspect 7. An apparatus according to any one of Aspects 1 to 6, wherein the at least one processor is configured to: generate a mask associated with an occlusion on the portion of the object, wherein the image data lacks the visual information of the portion of the object because the occlusion obscures the portion of the object in the image data.

[0178] Aspect 8. An apparatus according to any one of Aspects 1 to 7, wherein the at least one processor is configured to: determine that the image data is missing visual information of a portion of the object, wherein the portion of the object is discernible from a second perspective, and wherein, in order to generate reprojected image data of at least the portion of the environment according to the second perspective, the at least one processor is configured to reconstruct the visual information of the portion of the object using an output of the trained machine learning model in response to an input of the image data into the trained machine learning model.

[0179] Aspect 9. An apparatus according to any one of Aspects 1 to 8, wherein the at least one processor is configured to: determine a lighting scheme for illuminating the environment, wherein, in order to generate reprojected image data of at least a portion of the environment according to a second perspective, the at least one processor is configured to reconstruct the lighting scheme according to the second perspective.

[0180] Aspect 10. An apparatus according to any one of Aspects 1 to 9, wherein the at least one processor is configured to: modify the image data to normalize the illumination in the image data before generating reprojected image data of at least a portion of the environment based on the image data.

[0181] Aspect 11. An apparatus according to any one of Aspects 1 to 10, wherein the at least one processor is configured to: output an indicator of the state of the object based on the reprojected image data.

[0182] Aspect 12. An apparatus according to any one of Aspects 1 to 11, wherein the at least one processor is configured to: monitor the state of the object within a time period based on the reprojected image data, wherein both the image data and the reprojected image data include images corresponding to different times during the time period; and identify changes in the state of the object during the time period, wherein an indicator of the state of the object indicates the change in the state of the object.

[0183] Aspect 13. An apparatus according to any one of Aspects 1 to 12, wherein the change in the state of the object comprises a change in an illumination characteristic of a light source, wherein the object comprises the light source.

[0184] Aspect 14. An apparatus according to any one of Aspects 1 to 13, wherein the at least one processor is configured to: use optical character recognition (OCR) to parse one or more characters on the object in the reprojected image data, wherein a change in the state of the object is associated with the one or more characters.

[0185] Aspect 15. An apparatus according to any one of Aspects 1 to 14, wherein the change in the state of the object comprises a change in the movement of at least a part of the object.

[0186] Clause 16. The apparatus according to any one of Clauses 1 to 15, wherein the image data is received from an image sensor.

[0187] Aspect 17. An apparatus according to any one of Aspects 1 to 16, wherein the first subset of the image data is received from a first image sensor, and wherein the second subset of the image data is received from a second image sensor.

[0188] Aspect 18. An apparatus according to any one of Aspects 1 to 17, wherein, in order to detect the object in the image data, the at least one processor is configured to track the object from a first subset of the image data to a second subset of the image data.

[0189] Aspect 19. The apparatus according to any one of Aspects 1 to 18, further comprising: a display configured to display information based on the reprojected image data.

[0190] Aspect 20. The apparatus according to any one of Aspects 1 to 19 further comprises: a communication interface configured to send information based on the reprojected image data to a recipient device.

[0191] Aspect 21. An apparatus according to any one of aspects 1 to 20, wherein the apparatus comprises at least one of a head mounted display (HMD), a mobile handset, or a wireless communication device.

[0192] Aspect 22. A method for imaging, the method comprising: receiving image data of an environment, the image data representing the environment from a first perspective; detecting an object in the environment in the image data; and generating reprojected image data of at least a portion of the environment based on the image data, the reprojected image data representing at least the object from a second perspective different from the first perspective.

[0193] Aspect 23. The method according to aspect 22, further comprising: receiving input using a user interface, wherein the input indicates the second viewing angle.

[0194] Aspect 24. The method according to any one of Aspects 22 to 23 further includes: generating gesture information corresponding to the second perspective based on the position of the feature of the object in the environment.

[0195] Aspect 25. A method according to any one of Aspects 22 to 24, wherein generating the reprojected image data of at least the portion of the environment comprises generating the reprojected image data using probabilistic reprojection.

[0196] Aspect 26. A method according to any one of Aspects 22 to 25, wherein generating reprojected image data of at least a portion of the environment based on a second perspective includes using an output of a trained machine learning model responsive to an input of the image data into the trained machine learning model.

[0197] Aspect 27. The method according to any one of Aspects 22 to 26 further includes: receiving previous image data of the environment; and determining that the image data is missing visual information of a portion of the object, wherein the portion of the object is discernible from a second perspective, and wherein generating reprojected image data of at least the portion of the environment includes reconstructing the visual information of the portion of the object based on the previous image data.

[0198] Aspect 28. The method according to any one of Aspects 22 to 27 further includes: generating a mask associated with an occlusion on the portion of the object, wherein the image data lacks the visual information of the portion of the object because the occlusion obscures the portion of the object in the image data.

[0199] Aspect 29. The method according to any one of Aspects 22 to 28 further includes: determining that the image data lacks visual information of a portion of the object, wherein the portion of the object is discernible from a second perspective, and wherein generating reprojected image data of at least the portion of the environment includes: reconstructing the visual information of the portion of the object using the output of the trained machine learning model in response to the input of the image data into the trained machine learning model.

[0200] Aspect 30. The method according to any one of aspects 22 to 29 further comprises: determining a lighting scheme for the lighting environment, wherein generating reprojected image data of at least a portion of the environment comprises reconstructing the lighting scheme according to the second perspective.

[0201] Aspect 31. The method according to any one of Aspects 22 to 30 further includes: before generating reprojected image data of at least a portion of the environment based on the image data, modifying the image data to normalize the illumination in the image data.

[0202] Aspect 32. The method according to any one of Aspects 22 to 31 further includes: outputting an indicator of the state of the object based on the reprojected image data.

[0203] Aspect 33. The method according to any one of Aspects 22 to 32 further includes: monitoring the state of the object within a time period based on the reprojected image data, wherein both the image data and the reprojected image data include images corresponding to different times during the time period; and identifying changes in the state of the object during the time period, wherein an indicator of the state of the object indicates the change in the state of the object.

[0204] Aspect 34. A method according to any one of Aspects 22 to 33, wherein the change in the state of the object comprises a change in a lighting characteristic of a light source, wherein the object comprises the light source.

[0205] Aspect 35. The method according to any one of Aspects 22 to 34 further includes: using optical character recognition (OCR) to parse one or more characters on the object in the reprojected image data, wherein the change in the state of the object is associated with the one or more characters.

[0206] Aspect 36. A method according to any one of Aspects 22 to 35, wherein the change in the state of the object comprises a change in the movement of at least a part of the object.

[0207] Aspect 37. A method according to any one of aspects 22 to 36, wherein the image data is received from an image sensor.

[0208] Aspect 38. A method according to any one of aspects 22 to 37, wherein a first subset of the image data is received from a first image sensor, and wherein a second subset of the image data is received from a second image sensor.

[0209] Aspect 39. The method of any one of aspects 22 to 38, wherein detecting the object in the image data comprises tracking the object from a first subset of the image data to a second subset of the image data.

[0210] Aspect 40. The method according to any one of aspects 22 to 39, further comprising: causing a display to display information based on the reprojected image data.

[0211] Aspect 41. The method according to any one of aspects 22 to 40 further comprises: causing the communication interface to send information based on the reprojected image data to a recipient device.

[0212] Aspect 42. A method according to any one of aspects 22 to 41, wherein the method is performed using an apparatus comprising at least one of a head mounted display (HMD), a mobile handset, or a wireless communication device.

[0213] Aspect 43: A non-temporary computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to: receive image data of an environment, the image data representing the environment from a first perspective; detect an object in the environment in the image data; and generate reprojected image data of at least a portion of the environment based on the image data, the reprojected image data representing at least the object from a second perspective different from the first perspective.

[0214] Aspect 44: The non-transitory computer-readable medium according to Aspect 43, further comprising operations according to any one of Aspects 2 to 21 and / or any one of Aspects 22 to 42.

[0215] Aspect 45: A device for image processing, the device comprising: a device for receiving image data of an environment, the image data representing the environment from a first perspective; a device for detecting objects in the environment in the image data; and a device for generating reprojected image data of at least a portion of the environment based on the image data, the reprojected image data representing at least the object from a second perspective different from the first perspective.

[0216] Aspect 46: The apparatus according to aspect 45 further comprises means for performing the operations according to any one of aspects 2 to 21 and / or any one of aspects 22 to 42.

Claims

1. A device for imaging, the device comprising: at least one memory; as well as at least one processor coupled to the at least one memory, the at least one processor being configured to: receiving image data of an environment, the image data representing the environment from a first perspective; detecting objects in the environment in the image data; as well as Reprojected image data of at least a portion of the environment is generated based on the image data, the reprojected image data representing at least the object from a second perspective different from the first perspective.

2. The device according to claim 1, wherein: The at least one processor is configured to: An input is received using a user interface, wherein the input is indicative of the second viewing angle.

3. The device according to claim 1, wherein: The at least one processor is configured to: Generating gesture information corresponding to the second perspective based on the position of the feature of the object in the environment.

4. The device according to claim 1, wherein: To generate reprojected image data of at least a portion of the environment according to the second perspective, the at least one processor is configured to generate the reprojected image data using probabilistic reprojection.

5. The device according to claim 1, wherein: In order to generate reprojected image data of at least the portion of the environment according to the second perspective, the at least one processor is configured to use an output of the trained machine learning model in response to an input of the image data into the trained machine learning model.

6. The device according to claim 1, wherein: The at least one processor is configured to: receiving previous image data of the environment; and Determining that the image data is missing visual information of a portion of the object, wherein the portion of the object is discernible from the second perspective, wherein in order to generate reprojected image data of at least the portion of the environment according to the second perspective, the at least one processor is configured to reconstruct the visual information of the portion of the object based on the previous image data.

7. The device according to claim 6, wherein: The at least one processor is configured to: A mask associated with an occlusion on the portion of the object is generated, wherein the image data lacks visual information of the portion of the object due to the occlusion obscuring the portion of the object in the image data.

8. The device according to claim 1, wherein: The at least one processor is configured to: Determining that the image data is missing visual information of a portion of the object, wherein the portion of the object is discernible from the second perspective, wherein, in order to generate reprojected image data of at least the portion of the environment according to the second perspective, the at least one processor is configured to reconstruct the visual information of the portion of the object using an output of the trained machine learning model in response to an input of the image data into the trained machine learning model.

9. The device according to claim 1, wherein: The at least one processor is configured to: A lighting scheme for illuminating the environment is determined, wherein to generate reprojected image data of at least the portion of the environment according to a second perspective, the at least one processor is configured to reconstruct the lighting scheme according to the second perspective.

10. The device according to claim 1, wherein: The at least one processor is configured to: Prior to generating reprojected image data of at least the portion of the environment based on the image data, the image data is modified to normalize illumination in the image data.

11. The device according to claim 1, wherein: The at least one processor is configured to: An indicator of a status of the object is output based on the reprojected image data.

12. The device according to claim 11, wherein The at least one processor is configured to: monitoring a state of the object over a time period based on the reprojected image data, wherein the image data and the reprojected image data both include images corresponding to different times during the time period; and A change in the state of the object during the time period is identified, wherein an indicator of the state of the object indicates the change in the state of the object.

13. The device according to claim 12, wherein: The change in the state of the object includes a change in the lighting characteristics of a light source, wherein the object includes the light source.

14. The device according to claim 12, wherein: The at least one processor is configured to: One or more characters on the object in the reprojected image data are parsed using optical character recognition (OCR), wherein a change in the state of the object is associated with the one or more characters.

15. The device according to claim 12, wherein: The change in state of the object includes a change in motion of at least a portion of the object.

16. The apparatus of claim 1, wherein the image data is received from an image sensor.

17. The apparatus of claim 1, wherein a first subset of the image data is received from a first image sensor, and wherein a second subset of the image data is received from a second image sensor.

18. The apparatus of claim 17, wherein to detect the object in the image data, the at least one processor is configured to track the object from a first subset of the image data to a second subset of the image data.

19. The apparatus according to claim 1, further comprising: A display is configured to display information based on the reprojected image data.

20. The apparatus according to claim 1, further comprising: A communication interface is configured to send information based on the reprojected image data to a recipient device.

21. The device according to claim 1, wherein The apparatus includes at least one of a head mounted display (HMD), a mobile handset, or a wireless communication device.

22. A method for imaging, the method comprising: receiving image data of an environment, the image data representing the environment from a first perspective; detecting objects in the environment in the image data; as well as Reprojected image data of at least a portion of the environment is generated based on the image data, the reprojected image data representing at least the object from a second perspective different from the first perspective.

23. The method according to claim 22, further comprising: An input is received using a user interface, wherein the input is indicative of the second viewing angle.

24. The method of claim 22, further comprising: Gesture information corresponding to the second perspective is generated based on the position of the feature of the object in the environment.

25. The method of claim 22, wherein: Generating reprojected image data of at least the portion of the environment includes generating the reprojected image data using probabilistic reprojection.

26. The method of claim 22, wherein: Generating reprojected image data of at least the portion of the environment according to the second perspective includes: using an output of the trained machine learning model in response to an input of the image data into the trained machine learning model.

27. The method of claim 22, further comprising: Determining that the image data is missing visual information of a portion of the object, wherein the portion of the object is discernible from the second perspective, wherein generating reprojected image data of at least the portion of the environment includes reconstructing the visual information of the portion of the object using an output of the trained machine learning model in response to an input of the image data into the trained machine learning model.

28. The method of claim 22, further comprising: An indicator of a status of the object is output based on the reprojected image data.

29. The method according to claim 28, further comprising: monitoring a state of the object over a time period based on the reprojected image data, wherein both the image data and the reprojected image data include images corresponding to different times during the time period; and A change in the state of the object during the time period is identified, wherein an indicator of the state of the object indicates the change in the state of the object.

30. The method of claim 22, wherein a first subset of the image data is received from a first image sensor, and wherein a second subset of the image data is received from a second image sensor.

Images