Scene depth based exposure control

Abstract

Systems, devices, processes, and computer-readable media for capturing images having subjects at different depths are disclosed. A method of processing image data includes obtaining, at an imaging device, a first image of an environment from an image sensor of the imaging device; determining a region of interest of the first image based on a feature depicted in the first image, wherein the feature is associated with the environment; determining a representative brightness value associated with the first image based on image data in the region of interest of the first image; determining one or more exposure control parameters based on the representative brightness value; and obtaining, at the imaging device, a second image captured based on the one or more exposure control parameters.

Description

Technical Field

[0001] The present application relates to image processing. More specifically, the present application relates to systems and techniques for performing exposure control based on scene depth. Background Art

[0002] Many devices and systems allow for capturing a scene by generating an image (or frame) and / or video data (including multiple frames) of the scene. For example, a camera or a device including a camera (or multiple cameras) may capture a sequence of frames of a scene (e.g., a video of a scene) based on light captured by the camera's image sensor and processed by the camera's processor. In order to improve the quality of the frames captured by the camera, the camera may include a lens to focus the light entering the camera. In some cases, a camera or a device including a camera (or multiple cameras) may include an exposure control mechanism that may control the size of the camera's aperture, the duration that the aperture is open, the duration that the camera's image sensor collects light, the sensitivity of the image sensor, the analog gain applied by the image sensor, or any combination thereof. The sequence of frames captured by the camera may be output for display, may be output for processing and / or consumption by other devices, and for other purposes. Summary of the invention

[0003] In some examples, systems and techniques for performing exposure control based on scene depth are described. According to at least one example, a method for processing one or more images is provided. The method includes: obtaining a first image of an environment from an image sensor of an imaging device at an imaging device; determining a region of interest of the first image based on a feature depicted in the first image, wherein the feature is associated with the environment; determining a representative brightness value associated with the first image based on image data in the region of interest of the first image; determining one or more exposure control parameters based on the representative brightness value; and obtaining a second image captured based on the one or more exposure control parameters at the imaging device.

[0004] In another example, a device for processing one or more images is provided, the device comprising at least one memory and at least one processor, the at least one processor being coupled to the at least one memory. The at least one processor is configured to: obtain a first image of an environment from an image sensor of the imaging device; determine a region of interest of the first image based on a feature depicted in the first image, wherein the feature is associated with the environment; determine a representative brightness value associated with the first image based on image data in the region of interest of the first image; determine one or more exposure control parameters based on the representative brightness value; and obtain a second image captured based on the one or more exposure control parameters.

[0005] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided, which instructions, when executed by one or more processors, cause the one or more processors to: obtain, at an imaging device, a first image of an environment from an image sensor of the imaging device; determine a region of interest of the first image based on a feature depicted in the first image, wherein the feature is associated with the environment; determine a representative brightness value associated with the first image based on image data in the region of interest of the first image; determine one or more exposure control parameters based on the representative brightness value; and obtain, at the imaging device, a second image captured based on the one or more exposure control parameters.

[0006] In another example, an apparatus for processing one or more images is provided. The apparatus includes: a component for obtaining a first image of an environment from an image sensor of an imaging device; a component for determining a region of interest of the first image based on a feature depicted in the first image, wherein the feature is associated with the environment; a component for determining a representative brightness value associated with the first image based on image data in the region of interest of the first image; a component for determining one or more exposure control parameters based on the representative brightness value; and a component for obtaining a second image captured based on the one or more exposure control parameters.

[0007] In some aspects, the apparatus is, is part of, and / or includes a wearable device, an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a head mounted device (HMD) device, a wireless communication device, a mobile device (e.g., a mobile phone and / or a mobile handset and / or a so-called "smartphone" or another mobile device), a camera, a personal computer, a laptop computer, a server computer, a vehicle or a computing device or a component of a vehicle, another device, or a combination thereof. In some aspects, the apparatus includes a camera or cameras for capturing one or more images. In some aspects, the apparatus also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the apparatus may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyrometers, one or more accelerometers, any combination thereof, and / or other sensors).

[0008] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0009] The foregoing and other features and aspects will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Illustrative aspects of the present application are described in detail below with reference to the following drawings:

[0011] Exemplary embodiments of the present application are described in detail below with reference to the following drawings:

[0012] Figure 1 is a block diagram illustrating an example of the architecture of an image capture and processing device according to some examples;

[0013] Figure 2 is a conceptual diagram illustrating an example of a technique for performing visual simultaneous localization and mapping (VSLAM) using a camera of a VSLAM device according to some examples;

[0014] Figure 3A is a block diagram of a device that performs VSLAM or another localization technique according to some examples;

[0015] Figure 3B is a block diagram of another device that performs another positioning technique using a depth sensor according to some examples;

[0016] Figure 4 are example images generated by a device configured to perform positioning techniques in a poor lighting environment according to some aspects of the present disclosure;

[0017] Figure 5 is an example image generated by a device configured to control an image sensor for positioning techniques according to some aspects of the present disclosure;

[0018] Fig. 6A An example of visualization of a three-dimensional (3D) point cloud by a computing device identifying a region of interest according to some aspects of the present disclosure is illustrated;

[0019] Figure 6B illustrates an example of visualization of a 3D point cloud by a computing device identifying a region of interest in accordance with some aspects of the present disclosure;

[0020] Fig. 7A is a perspective view illustrating a ground vehicle performing VSLAM or another localization technique according to some examples;

[0021] Figure 7B is a perspective view illustrating an aerial vehicle performing VSLAM or another localization technique according to some examples;

[0022] Fig. 8A is a perspective view illustrating a head mounted display (HMD) performing VSLAM or another localization technique according to some examples;

[0023] Figure 8B is an example based on some examples Fig. 7A A perspective view of the HMD being worn by a user;

[0024] Figure 8C is a perspective view of a front surface of a mobile phone illustrating use of a front-facing camera to perform VSLAM or another positioning technique according to some examples;

[0025] Fig.8D is a perspective view of a rear surface of a mobile phone illustrating use of a rear camera to perform VSLAM or another positioning technique according to some examples;

[0026] Fig. 9 is a flowchart illustrating an example of an image processing technique according to some examples; and

[0027] Fig.10 is a diagram illustrating an example of a system for implementing certain aspects of the disclosed technology. DETAILED DESCRIPTION

[0028] Certain aspects of the present disclosure are provided below. Some of these aspects may be applied independently, and some of them may be applied in combination, which will be apparent to those skilled in the art. In the following description, specific details are set forth for explanation purposes to provide a thorough understanding of various aspects of the application. However, it will be apparent that various aspects may be practiced without these specific details. Each of the drawings and descriptions is not intended to be restrictive.

[0029] The following description provides only example aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of the example aspects will provide a description that can be used to implement the example aspects to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the essence and scope of the present application as set forth in the appended claims.

[0030] The following description provides only exemplary aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of exemplary aspects will provide an enabling description for implementing aspects of the present disclosure to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of elements without departing from the essence and scope of the present application as set forth in the appended claims.

[0031] The terms "exemplary" and / or "example" are used herein to mean "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" and / or "example" is not necessarily to be construed as preferred or superior to other aspects. Likewise, the term "aspects of the disclosure" does not require that all aspects of the disclosure include the discussed feature, advantage, or mode of operation.

[0032] An image capture device (e.g., a camera) is a device that uses an image sensor to receive light and capture an image frame such as a still image or a video frame. The terms "image," "image frame," and "frame" are used interchangeably herein. An image capture device typically includes at least one lens that receives light from a scene and bends the light toward an image sensor of the image capture device. The light received by the lens passes through an aperture controlled by one or more control mechanisms and is received by the image sensor. The one or more control mechanisms may control exposure, focus, and / or zoom based on information from the image sensor and / or based on information from an image processor (e.g., a host or application process and / or an image signal processor). In some examples, the one or more control mechanisms include a motor or other control mechanism for moving the lens of the image capture device to a target lens position.

[0033] Positioning is a general description of positioning technology used to identify the location of an object in an environment. An example of a positioning technology is the Global Positioning System (GPS) used to identify the location of an outdoor environment. Other types of positioning technology use Angle of Arrival (AoA), Time of Arrival (ToA), Received Signal Strength Indicator (RSSI) to identify the location of an object in an environment.

[0034] Simultaneous Localization and Mapping (SLAM) is a positioning technology used in devices such as robotic systems, autonomous vehicle systems, extended reality (XR) systems, head-mounted displays (HMDs), and the like. As described above, XR systems may include, for example, augmented reality (AR) systems, virtual reality (VR) systems, and mixed reality (MR) systems. An XR system may be an HMD device. Using SLAM, a device may build and update a map of an unknown environment while tracking the device's position in that environment. The device may typically perform these tasks based on sensor data collected by one or more sensors on the device. For example, a device may be activated in a specific room of a building and may be moved throughout the building, mapping the entire interior of the building while tracking its own position within the map as the device draws the map.

[0035] Visual SLAM (VSLAM) is a SLAM technology that performs map building and positioning based on visual data collected by one or more cameras of a device. In some cases, a monocular VSLAM device can perform VSLAM using a single camera. For example, a monocular VSLAM device can capture one or more images of an environment with a camera, and can determine different visual features, such as corner points or other points in one or more images. The device can move in the environment and can capture more images. When the device is in different positions, orientations, and / or postures in the environment, the device can track the movement of those features in the captured continuous images. The device can use these tracked features to generate a three-dimensional (3D) map and determine its own position in the map.

[0036] VSLAM can be performed using a visible light (VL) camera that detects light within the spectrum visible to the human eye. Some VL cameras only detect light within the spectrum visible to the human eye. An example of a VL camera is a camera that captures red (R), green (G), and blue (B) image data (referred to as RGB image data). The RGB image data can then be merged into a full-color image. A VL camera that captures RGB image data may be referred to as an RGB camera. The camera may also capture other types of color images, such as images with luminance (Y) and chrominance (blue chrominance, referred to as U or Cb, red chrominance, referred to as V or Cr) components. Such images may include YUV images, YC b C r Images, etc.

[0037] In some environments (e.g., outdoor environments), image features may be randomly distributed at varying depths (e.g., depths ranging from less than one meter to thousands of meters). Typically, nearby features provide better pose estimates, in which case it is desirable to track as many nearby features as possible. However, due to varying light conditions in certain environments (e.g., outdoor environments), nearby objects may be overexposed or underexposed, which may make feature tracking of nearby features difficult. For example, a VL camera may capture clear images of well-lit and indoor environments. Features such as edges and corners may be easily discerned in clear images of well-lit environments. However, a VL camera may have difficulties in an outdoor environment with a large dynamic range. For example, bright and shadowed areas in an outdoor environment may be very different based on the positioning of the sun, and extremely bright areas may cause the camera to capture the environment at a low exposure, which makes the shadowed area darker. In some cases, it may be difficult to identify objects within the shadowed area based on different amounts of light in the environment. For example, a bright area may be far away, while a shadowed area may be closer. Therefore, a tracking device (e.g., a VSLAM device) using a VL camera may sometimes fail to recognize portions of the environment that the VSLAM device has observed due to lighting conditions in the environment. Failure to recognize portions of the environment that the VSLAM device has observed may result in errors in the localization and / or mapping of the VSLAM device.

[0038] Systems, devices, electronic devices or apparatus, methods (also referred to as processes), and computer-readable media (collectively referred to herein as "systems and techniques") for performing exposure control based on scene depth are described herein. For example, these systems and techniques can capture images in an environment with dynamic lighting conditions or in areas with different lighting conditions, and adjust image exposure settings based on the depth of objects in the environment and the corresponding lighting associated with these objects. For example, these systems and techniques can use image sensors (e.g., visible light (VL) cameras, IR cameras) and / or depth sensors to perform positioning.

[0039] In an illustrative example, a system or device may obtain (e.g., capture) a first image of an environment from an image sensor of an imaging device, and determine a region of interest of the first image based on features depicted in the first image. The region of interest may include features associated with the environment that can be used for tracking and positioning. The device may determine a representative brightness value (e.g., an average brightness value) associated with the first image based on image data in the region of interest of the first image. After determining the representative brightness, the device may determine one or more exposure control parameters based on the representative brightness value. The device may then obtain a second image captured based on the exposure control parameters. In one aspect, if the region of interest is dark and has a low brightness value, the device may increase the exposure time to increase the brightness of the region of interest. The device may also reduce the exposure time, or may perform other changes, such as increasing the gain of the image sensor, which amplifies the brightness of the portion.

[0040] Additional details about the systems and techniques are provided herein with respect to various figures. Although some examples described herein use SLAM as an example of an application that can use the exposure control system and techniques described herein, such techniques can be used by any system that captures images and / or uses images for one or more operations.

[0041] Figure 1 1 is a block diagram illustrating an example of an architecture of an image capture and processing system 100. The image capture and processing system 100 includes various components for capturing and processing an image of a scene (e.g., an image of scene 110). The image capture and processing system 100 may capture individual images (or photographs) and / or may capture a video including multiple images (or video frames) in a particular sequence. A lens 115 of the system 100 faces the scene 110 and receives light from the scene 110. The lens 115 bends the light toward the image sensor 130. The light received by the lens 115 passes through an aperture controlled by one or more control mechanisms 120 and is received by the image sensor 130.

[0042] The one or more control mechanisms 120 may control exposure, focus, and / or zoom based on information from the image sensor 130 and / or based on information from the image processor 150. The one or more control mechanisms 120 may include multiple mechanisms and components; for example, the control mechanism 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. The one or more control mechanisms 120 may also include additional control mechanisms other than those illustrated, such as control mechanisms to control analog gain, flash, HDR, depth of field, and / or other image capture attributes.

[0043] A focus control mechanism 125B of the control mechanism 120 may obtain a focus setting. In some examples, the focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 125B may adjust the positioning of the lens 115 relative to the positioning of the image sensor 130. For example, based on the focus setting, the focus control mechanism 125B may adjust the focus by actuating a motor or servo system (or other lens mechanism) to move the lens 115 closer to the image sensor 130 or further away from the image sensor 130. In some cases, an additional lens may be included in the system 100, such as one or more micro lenses above each photodiode of the image sensor 130, each of which bends the light received from the lens 115 toward the corresponding photodiode before it reaches the photodiode. The focus setting may be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), hybrid autofocus (HAF), or some combination thereof. The focus setting may be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. Focus settings may be referred to as image capture settings and / or image processing settings.

[0044] The exposure control mechanism 125A of the control mechanism 120 may obtain an exposure setting. In some cases, the exposure control mechanism 125A stores the exposure setting in a memory register. Based on the exposure setting, the exposure control mechanism 125A may control the size of the aperture (e.g., aperture size or number of apertures), the duration of the aperture opening (e.g., exposure time or shutter speed), the sensitivity of the image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by the image sensor 130, or any combination thereof. The exposure setting may be referred to as an image capture setting, an image acquisition setting, and / or an image processing setting.

[0045] A zoom control mechanism 125C of the control mechanism 120 may obtain a zoom setting. In some examples, the zoom control mechanism 125C stores the zoom setting in a memory register. Based on the zoom setting, the zoom control mechanism 125C may control the focal length of an assembly of lens elements (lens assembly) including the lens 115 and one or more additional lenses. For example, the zoom control mechanism 125C may control the focal length of the lens assembly by actuating one or more motors or servos (or other lens mechanisms) to move one or more lenses relative to each other. The zoom setting may be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly may include a parfocal zoom lens or a variable focal length zoom lens. In some examples, the lens assembly may include a focusing lens (in some cases, the focusing lens may be the lens 115) that first receives light from the scene 110, wherein the light then passes through an afocal zoom system between the focusing lens (e.g., the lens 115) and the image sensor 130 before the light reaches the image sensor 130. In some cases, the afocal zoom system can include two positive (e.g., converging, convex) lenses with equal or similar focal lengths (e.g., within a threshold difference of each other), with a negative (e.g., diverging, concave) lens between them. In some cases, zoom control mechanism 125C moves one or more of the lenses in the afocal zoom system, such as the negative lens and one or both of the positive lenses.

[0046] Image sensor 130 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a particular pixel in an image generated by image sensor 130. In some cases, different photodiodes may be covered by different color filters and may therefore measure light that matches the color of the color filter covering the photodiode. For example, a Bayer color filter includes a red filter, a blue filter, and a green filter, where each pixel of an image is generated based on red light data from at least one photodiode covered in a red filter, blue light data from at least one photodiode covered in a blue filter, and green light data from at least one photodiode covered in a green filter. Other types of color filters may use yellow, magenta, and / or cyan (also known as "emerald") filters instead of or in addition to red, blue, and / or green filters. Some image sensors (e.g., image sensor 130) may have no color filters at all and may instead use different photodiodes (in some cases stacked vertically) throughout the pixel array. Different photodiodes throughout the pixel array may have different spectral sensitivity curves, thereby responding to different wavelengths of light.Monochrome image sensors may also lack color filters, and therefore lack color depth.

[0047] In some cases, image sensor 130 may alternatively or additionally include an opaque mask and / or a reflective mask that blocks light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles, which may be used for phase detection autofocus (PDAF). Image sensor 130 may also include an analog gain amplifier for amplifying an analog signal output by a photodiode and / or an analog-to-digital converter (ADC) for converting an analog signal output by a photodiode (and / or an analog signal amplified by an analog gain amplifier) ​​into a digital signal. In some cases, certain components or functions discussed with respect to one or more of control mechanisms 120 may alternatively or additionally be included in image sensor 130. Image sensor 130 may be a charge coupled device (CCD) sensor, an electron multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal oxide semiconductor (CMOS), an N-type metal oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0048] The image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), and / or one or more other types of processors 1810 discussed with respect to the computing device 1800. The host processor 152 may be a digital signal processor (DSP) and / or other types of processors. In some implementations, the image processor 150 is a single integrated circuit or chip (e.g., referred to as a system on a chip or SoC) that includes the host processor 152 and the ISP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth ™, global positioning system (GPS), etc.), any combination thereof, and / or other components. The I / O port 156 may include any suitable input / output port or interface according to one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a serial general purpose input / output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface, an Advanced High-Performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In an illustrative example, the host processor 152 may communicate with the image sensor 130 using an I2C port, and the ISP 154 may communicate with the image sensor 130 using a MIPI port.

[0049] The image processor 150 may perform a number of tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. The image processor 150 may store image frames and / or processed images in a random access memory (RAM) 140 / 1020, a read-only memory (ROM) 145 / 1025, a cache, a memory unit, another storage device, or some combination thereof.

[0050] Various input / output (I / O) devices 160 may be connected to the image processor 150. The I / O devices 160 may include a display screen, a keyboard, a keypad, a touch screen, a touchpad, a touch-sensitive surface, a printer, any other output device 1835, any other input device 1845, or some combination thereof. In some cases, subtitles may be input into the image processing device 105B through a physical keyboard or keypad of the I / O device 160, or through a virtual keyboard or keypad of a touch screen of the I / O device 160. The I / O 160 may include one or more ports, jacks, or other connectors that enable a wired connection between the system 100 and one or more peripheral devices, through which the system 100 may receive data from the one or more peripheral devices and / or send data to the one or more peripheral devices. The I / O 160 may include one or more wireless transceivers that enable a wireless connection between the system 100 and one or more peripheral devices, through which the system 100 may receive data from the one or more peripheral devices and / or send data to the one or more peripheral devices. The peripheral devices may include any of the types of I / O devices 160 discussed previously, and may themselves be considered I / O devices 160 once they are coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector.

[0051] In some cases, the image capture and processing system 100 can be a single device. In some cases, the image capture and processing system 100 can be two or more separate devices, including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some implementations, the image capture device 105A and the image processing device 105B can be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly coupled together via one or more wireless transceivers. In some implementations, the image capture device 105A and the image processing device 105B can be disconnected from each other.

[0052] like Figure 1 As shown, the vertical dotted line will Figure 1 The image capture and processing system 100 is divided into two parts, representing an image capture device 105A and an image processing device 105B. The image capture device 105A includes a lens 115, a control mechanism 120, and an image sensor 130. The image processing device 105B includes an image processor 150 (including an ISP 154 and a host processor 152), a RAM 140, a ROM 145, and an I / O 160. In some cases, some components illustrated in the image capture device 105A (such as the ISP 154 and / or the host processor 152) may be included in the image capture device 105A.

[0053] The image capture and processing system 100 may include an electronic device, such as a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the image capture and processing system 100 may include one or more wireless transceivers for wireless communications (such as cellular network communications, 802.11 wi-fi communications, wireless local area network (WLAN) communications, or some combination thereof). In some implementations, the image capture device 105A and the image processing device 105B may be different devices. For example, the image capture device 105A may include a camera device, and the image processing device 105B may include a computing device, such as a mobile phone, a desktop computer, or other computing device.

[0054] Although the image capture and processing system 100 is shown as including certain components, one of ordinary skill will appreciate that the image capture and processing system 100 may include more than Figure 1 . The components of the image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components of the image capture and processing system 100 may include electronic circuits or other electronic hardware, and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include computer software, firmware, or any combination thereof, and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device that implements the image capture and processing system 100.

[0055] In some cases, the image capture and processing system 100 may be part of or implemented by a device (referred to as a VSLAM device) that can perform positioning or a positioning type such as VSLAM. For example, the VSLAM device may include one or more image capture and processing systems 100, image capture systems 105A, image processing systems 105B, computing systems 1000, or any combination thereof. For example, the VSLAM device may include at least one image sensor and a depth sensor. The VL camera and the IR camera may each include at least one of the image capture and processing system 100, the image capture device 105A, the image processing device 105B, the computing system 1800, or some combination thereof.

[0056] Figure 2 2 is a conceptual diagram 200 illustrating an example of a technique for performing visual VSLAM using a camera 210 of a VSLAM device 205. In some examples, the VSLAM device 205 may be a VR device, an AR device, an MR device, an XR device, an HMD, or some combination thereof. In some examples, the VSLAM device 205 may be a wireless communication device, a mobile device (e.g., a mobile phone or so-called "smart phone" or other mobile device), a wearable device, an extended reality (XR) device (e.g., a VR device, an AR device, or an MR device), an HMD, a personal computer, a laptop computer, a server computer, an unmanned ground vehicle, a drone, an unmanned water vehicle, an unmanned underwater vehicle, an unmanned vehicle, an autonomous vehicle, a vehicle, a robot, any combination thereof, and / or other devices.

[0057] The VSLAM device 205 includes a camera 210. The camera 210 may be responsive to light from a particular spectrum. The spectrum may be a subset of the electromagnetic (EM) spectrum. For example, the camera 210 may be a camera responsive to the visible spectrum, an IR camera responsive to the IR spectrum, a UV camera responsive to the ultraviolet (UV) spectrum, a camera responsive to light from another spectrum of another part of the electromagnetic spectrum, or some combination thereof. In some cases, the camera 210 may be a NIR camera responsive to the near infrared (NIR) spectrum. The NIR spectrum may be a subset of the IR spectrum that is close to and / or adjacent to the VL spectrum.

[0058] Camera 210 can be used to capture one or more images, including image 215. VSLAM system 270 can use feature extraction engine 220 to perform feature extraction. Feature extraction engine 220 can use image 215 to perform feature extraction by detecting one or more features in the image. These features can be, for example, edges, corners, areas of color change, areas of brightness change, or a combination thereof. In some cases, when feature extraction engine 220 cannot detect any features in image 215, feature extraction engine 220 may not be able to perform feature extraction on image 215. In some cases, feature extraction engine 220 may fail when it fails to detect at least a predetermined minimum number of features in image 215. If feature extraction engine 220 cannot successfully perform feature extraction on image 215, VSLAM system 270 no longer proceeds, and can wait for the next image frame captured by camera 210.

[0059] When feature extraction engine 220 detects at least a predetermined minimum number of features in image 215, feature extraction engine 220 may successfully perform feature extraction on image 215. In some examples, the predetermined minimum number of features may be one, in which case feature extraction engine 220 successfully performs feature extraction by detecting at least one feature in image 215. In some examples, the predetermined minimum number of features may be greater than one, and may be, for example, 2, 3, 4, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, a number greater than 100, or a number between any two of the previously listed numbers. Images with one or more features clearly delineated may be maintained in a map database as keyframes, and the delineation of their features may be used to track those features in other images.

[0060] Once the feature extraction engine 220 successfully performs feature extraction on one or more images 215, the VSLAM system 270 can use the feature tracking engine 225 to perform feature tracking. The feature tracking engine 225 can perform feature tracking by identifying features in the image 215 that have been previously identified in one or more previous images. The feature tracking engine 225 can also track changes in one or more locations of features between different images. For example, the feature extraction engine 220 can detect the face of a specific person as a feature depicted in a first image. The feature extraction engine 220 can detect the same feature (e.g., the face of the same person) depicted in a second image captured by the camera 210 after the first image and received from it. Feature tracking 225 can identify that these features detected in the first image and the second image are two depictions of the same feature (e.g., the face of the same person). The feature tracking engine 225 can identify that the feature has moved between the first image and the second image. For example, the feature tracking engine 225 can identify that the feature is depicted on the right hand side of the first image and is depicted in the center of the second image.

[0061] The movement of the feature between the first image and the second image may be caused by the movement of the captured object within the captured scene between the capture of the first image by the camera 210 and the capture of the second image. For example, if the feature is a person's face, the person may walk through a portion of the captured scene between the capture of the first image by the camera 210 and the capture of the second image, so that the feature is in a different position in the second image than in the first image. The movement of the feature between the first image and the second image may be caused by the movement of the camera 210 between the capture of the first image by the camera 210 and the capture of the second image. In some examples, the VSLAM device 205 may be a robot or a vehicle, and may move itself and / or its camera 210 between the capture of the first image by the camera 210 and the capture of the second image. In some examples, the VSLAM device 205 may be a head mounted display (HMD) (e.g., an XR headset) worn by a user, and the user may move his or her head and / or body between the capture of the first image by the camera 210 and the capture of the second image.

[0062] For each feature identified by the VSLAM system 270 using the feature extraction engine 220 and / or the feature tracking engine 225, the VSLAM system 270 may identify a coordinate set, which may be referred to as a map point. The coordinate set of each feature may be used to determine the map point 240. The local map engine 250 may use the map point 240 to update the local map. The local map may be a map of a local area of ​​a map of an environment. The local area may be the area where the VSLAM device 205 is currently located. The local area may be, for example, a room or a group of rooms within the environment. The local area may be, for example, a group of one or more rooms visible in the image 215. The coordinate set corresponding to the map point of the feature may be updated by the VSLAM system 270 using the map optimization engine 235 to increase accuracy. For example, by tracking features across multiple images captured at different times, the VSLAM system 270 may generate a coordinate set of the map point of the feature from each image. By triangulating or generating average coordinates based on multiple map points of the feature determined from different images, an accurate coordinate set may be determined for the map point of the feature. The map optimization 235 engine can use the local map construction engine 250 to update the local map to update the coordinate set of the feature, thereby using the accurate coordinate set determined by triangulation and / or averaging. Observing the same feature from different angles can provide additional information about the true location of the feature, which can be used to increase the accuracy of the map point 240.

[0063] Local map 250 can be part of map building system 275 together with global map 255. Global map 255 can map the global area of ​​environment. VSLAM device 205 can be positioned in the global area of ​​environment and / or in the local area of ​​environment. The local area of ​​environment can be smaller than the global area of ​​environment. The local area of ​​environment can be a subset of the global area of ​​environment. The local area of ​​environment can overlap with the global area of ​​environment. In some cases, the local area of ​​environment can include the part of environment that has not been merged into the global map by map merging engine 257 and / or global map building engine 255. In some examples, the local map can include map points in such part of environment that has not been merged into the global map. In some cases, global map 255 can map the whole of environment that VSLAM device 205 has observed. The update of local map by local map building engine 250 can be merged into the global map using map merging engine 257 and / or global map building engine 255, so as to keep the global map up to date. In some cases, after the local map has been optimized using the map optimization engine 235, the local map may be merged with the global map using the map merging engine 257 and / or the global map building engine 255 so that the global map is the optimized map. The map points 240 may be fed into the local map by the local map building engine 250 and / or may be fed into the global map using the global map building engine 255. The map optimization engine 235 may improve the accuracy of the map points 240 and the local and / or global maps. In some cases, the map optimization engine 235 may simplify the local and / or global maps by replacing a bunch of map points with a centroid map point.

[0064] The VSLAM system 270 may also determine a pose 245 of the device 205 based on feature extraction and / or feature tracking performed by the feature extraction engine 220 and / or the feature tracking engine 225. The pose 245 of the device 205 may refer to the position of the device 205, the orientation of the device 205 (e.g., represented as pitch, roll and yaw of the device 205, quaternions, SE3, direction cosine matrix (DCM), or any combination thereof). The pose 245 of the device 205 may refer to the pose of the camera 210, and may therefore include the position of the camera 210 and / or the orientation of the camera 210. The pose 245 of the device 205 may be determined relative to a local map and / or a global map. The pose 245 of the device 205 may be marked on the local map by the local map building engine 250 and / or on the global map by the global map building engine 255. In some cases, the history of poses 245 may be stored in a local map and / or a global map by local mapping engine 250 and / or by global mapping engine 255. Together, the history of poses 245 may indicate a path that VSLAM device 205 has traveled.

[0065] In some cases, feature tracking engine 225 may not successfully perform feature tracking on image 215 when features that were previously identified in a set of earlier captured images are not identified in image 215. In some examples, the set of earlier captured images may include all images captured during a time period that ends before capturing image 215 and begins at a predetermined start time. The predetermined start time may be an absolute time, such as a specific time and date. The predetermined start time may be a relative time, such as a predetermined amount of time (e.g., 30 minutes) before capturing image 215. The predetermined start time may be the time when VSLAM device 205 was last initialized. The predetermined start time may be the time when VSLAM device 205 last received an instruction to start the VSLAM process. The predetermined start time may be the time when VSLAM device 205 last determined that it entered a new room or a new area of ​​the environment.

[0066] If the feature tracking engine 225 cannot successfully perform feature tracking on the image, the VSLAM system 270 can use the relocation engine 230 to perform relocation. The relocation engine 230 attempts to determine where the VSLAM device 205 is located in the environment. For example, the feature tracking engine 225 may not be able to identify any features from one or more previously captured images and / or from the local map 250. The relocation engine 230 may try to see if any features identified by the feature extraction engine 220 match any features in the global map. If one or more features of the VSLAM system 270 identified by the feature extraction engine 220 match one or more features in the global map 255, the relocation engine 230 successfully performs relocation by determining the map points 240 of the one or more features and / or determining the pose 245 of the VSLAM device 205. The relocation engine 230 may also compare any features identified by the feature extraction engine 220 in the image 215 with features stored in the keyframes next to the local map and / or the global map. Each keyframe may be an image that clearly depicts a particular feature, so that the image 230 may be compared to the keyframe to determine whether the image 230 also depicts the particular feature. If none of the features identified by the VSLAM system 270 during feature extraction 220 matches any of the features in the global map and / or any keyframe, the relocalization engine 230 is unable to successfully perform relocalization. If the relocalization engine 230 is unable to successfully perform relocalization, the VSLAM system 270 may exit and reinitialize the VSLAM process. Exiting and reinitializing may include generating the local map 250 and / or the global map 255 from scratch.

[0067] VSLAM equipment 205 may include a transport, and VSLAM equipment 205 may move itself around the environment by the transport. For example, VSLAM equipment 205 may include one or more motors, one or more actuators, one or more wheels, one or more propellers, one or more turbines, one or more rotors, one or more wings, one or more airfoils, one or more gliders, one or more pedals, one or more legs, one or more feet, one or more pistons, one or more nozzles, one or more pushers, one or more sails, the transport of one or more other modes discussed herein or their combination. In some examples, VSLAM equipment 205 may be a vehicle, a robot or any other type of equipment discussed herein. VSLAM equipment 205 including transport may use path planning engine 260 to perform path planning to plan the path for VSLAM equipment 205 to move. Once path planning engine 260 has planned a path for VSLAM equipment 205, VSLAM equipment 205 may use mobile actuator 265 to perform mobile actuation to actuate transport and move VSLAM equipment 205 along the path planned by path planning engine 260. In some examples, the path planning engine 260 may use the Dijkstra algorithm to plan the path. In some examples, the path planning engine 260 may include stationary obstacle avoidance and / or moving obstacle avoidance when planning the path. In some examples, the path planning engine 260 may include a determination of how to best move from a first pose to a second pose when planning the path. In some examples, the path planning engine 260 may plan a path that is optimized to reach and observe each part of each room before moving to other rooms when planning the path. In some examples, the path planning engine 260 may plan a path that is optimized to reach and observe each room in the environment as quickly as possible. In some examples, the path planning engine 260 may plan a path that returns to a previously observed room to observe a specific feature again to improve one or more map points corresponding to the feature in a local map and / or a global map. In some examples, the path planning engine 260 may plan a path that returns to a previously observed room to observe a portion of a previously observed room that lacks a map point in a local map and / or a global map to see if any features can be observed in the portion of the room.

[0068] Although various elements of conceptual diagram 200 are illustrated separately from VSLAM device 205, it should be understood that VSLAM device 205 may include any combination of elements of conceptual diagram 200. For example, at least a subset of VSLAM system 270 may be part of VSLAM device 205. At least a subset of map building system 275 may be part of VSLAM device 205. For example, VSLAM device 205 may include camera 210, feature extraction engine 220, feature tracking engine 225, relocalization engine 230, map optimization engine 235, local map building engine 250, global map building engine 255, map merging engine 257, path planning engine 260, motion actuator 265, or some combination thereof. In some examples, the VSLAM device 205 can capture an image 215, identify features in the image 215 through a feature extraction engine 220, track features through a feature tracking engine 225, optimize a map using a map optimization engine 235, perform relocalization using a relocalization engine 230, determine a map point 240, determine a device pose 245, generate a local map using a local map building engine 250, update a local map using a local map building engine 250, perform map merging using a map merging engine 257, generate a global map using a global map building engine 255, update a global map using a global map building engine 255, plan a path using a path planning engine 260, actuate movement using a motion actuator 265, or some combination thereof. In some examples, the feature extraction engine 220 and / or the feature tracking engine 225 are part of the front end of the VSLAM device 205. In some examples, the relocalization engine 230 and / or the map optimization engine 235 are part of the back end of the VSLAM device 205. Based on the image 215 and / or previous images, the VSLAM device 205 can identify features through feature extraction 220, track features through feature tracking 225, perform map optimization 235, perform relocalization 230, determine map points 240, determine pose 245, generate a local map 250, update the local map 250, perform map merging, generate a global map 255, update the global map 255, perform path planning 260, or some combination thereof.

[0069] In some examples, the map points 240, the device poses 245, the local map, the global map, the path planned by the path planning engine 260, or a combination thereof are stored at the VSLAM device 205. In some examples, the map points 240, the device poses 245, the local map, the global map, the path planned by the path planning engine 260, or a combination thereof are stored away from the VSLAM device 205 (e.g., on a remote server), but can be accessed by the VSLAM device 205 through a network connection. The map building system 275 can be part of the VSLAM device 205 and / or the VSLAM system 270. The map building system 275 can be part of a device (e.g., a remote server) that is away from the VSLAM device 205 but communicates with the VSLAM device 205.

[0070] In some cases, VSLAM device 205 can communicate with a remote server. The remote server may include at least a subset of VSLAM system 270. The remote server may include at least a subset of map construction system 275. For example, VSLAM device 205 may include camera 210, feature extraction engine 220, feature tracking engine 225, relocation engine 230, map optimization engine 235, local map construction engine 250, global map construction engine 255, map merging engine 257, path planning engine 260, mobile actuator 255 or some combination thereof. In some examples, VSLAM device 205 can capture image 215 and transmit image 215 to the remote server. Based on the image 215 and / or the previous image, the remote server may identify features through the feature extraction engine 220, track features through the feature tracking engine 225, optimize the map using the map optimization engine 235, perform relocation using the relocation engine 230, determine map points 240, determine the device pose 245, generate a local map using the local map building engine 250, update the local map using the local map building engine 250, perform map merging using the map merging engine 257, generate a global map using the global map building engine 255, update the global map using the global map building engine 255, plan a path using the path planning engine 260, or some combination thereof. The remote server may transmit the results of these processes back to the VSLAM device 205.

[0071] The accuracy of tracking features used for VSLAM and other localization techniques is based on the identification of features that are closer to the device. For example, a device may be able to identify features of a statue that is 4 meters away from the device, but not be able to identify distinguishing features when the statue is 20 meters away from the device. The lighting of the environment can also affect the tracking accuracy of devices using VSLAM and localization techniques.

[0072] Figure 3Ais a conceptual diagram of a device 300 for capturing images based on positioning and mapping techniques while taking into account lighting conditions of the environment. In some aspects, the device 300 may perform VSLAM techniques to determine the location of the device 300 within the environment. Figure 3A The device 300 may be any type of VSLAM device, including Figure 2 Any type of VSLAM device discussed in the VSLAM device 205. In other aspects, the device 300 can access existing maps locally (e.g., on the device 300) or remotely (e.g., through a service such as a microservice). For example, the device can be configured to perform functions in an existing area based on a known map. An example of a positioning device is an autonomous bus service that uses a known map stored in the bus or available to the bus using wireless communication to serve a park.

[0073] The device 300 includes an image sensor 305 and a motion sensor 310. The image sensor 305 is configured to generate images 320 on a periodic or aperiodic basis and provide the images to a feature extraction engine 330 that is configured to identify various features in the environment. The features may be known to the feature extraction engine 330 (e.g., due to a map) or may be unknown (e.g., a person in the environment). The feature extraction engine 330 extracts information related to the identified features and provides it to the tracking system.

[0074] In some cases, motion sensor 310 may be an inertial measurement unit (IMU), an accelerometer, or other device capable of obtaining motion information 325. Motion information 325 may be a relative position or an absolute position and is provided to motion detection engine 335 to identify motion within the environment. Motion detection engine 335 is configured to receive raw sensor data and process the sensor data to identify movement, rotation, position, orientation, and other relevant information provided.

[0075] In some aspects, tracking system 340 is configured to determine pose information associated with the device using image 320 and motion information provided from motion detection engine 335. For example, pose information may include positioning (e.g., positioning on a map) and the location of device 300 relative to any information. For example, device 300 may be an autonomous bus that uses positioning to identify a fixed route and navigates the fixed route. Device 300 may also be an autonomous ground device, such as an autonomous vacuum cleaner, that uses VSLAM technology to create a map of an environment and navigate within the environment.

[0076] In some aspects, the device 300 may use the image sensor 305 to capture an image and use the map building information to determine the exposure control information for object identification. For example, if the average brightness associated with the area in the image 320 captured by the image sensor 305 is less than a predetermined brightness threshold, the device 300 may determine that the area is poorly illuminated and cannot identify features in the area. In some cases, the poorly illuminated area of ​​the image 320 may be close to the device, while the well-illuminated area may be far from the device. In this example, the device 300 may not be able to identify features in the well-illuminated area based on the distance between the device 300 and the well-illuminated area. For example, if the device 300 is 30 meters away from a fixed statue, the device 300 may not be able to identify the features of the fixed statue based on the distance, and the distance may cause the device 300 to generate errors when performing positioning or SLAM technology. In this case, the positioning indication device 300 does not perform any mapping functions and relies on a known map to navigate the environment (e.g., ground truth), and the SLAM indication device is navigating an area without ground truth based on generating a local map using an existing map or a generated map.

[0077] The device 300 may move throughout the environment, thereby reaching multiple locations along a path through the environment. The tracking system 340 is configured to receive the features extracted from the feature extraction engine 330 and the motion information from the motion detection engine 335 to determine pose information of the device 300. For example, the pose information may determine the location of the device 300, the orientation of the device, and other relevant information that will allow the device 300 to use positioning or SLAM techniques.

[0078] In some aspects, tracking system 340 includes a device pose determination engine 342 that can determine the pose of device 300. Device pose determination engine 370 can be part of device 300 and / or a remote server. The pose of device 300 can be determined based on feature extraction by feature extraction engine 330, determination of map points and updating of the map by map construction system 350, or some combination thereof. In other aspects, device 300 can include other sensors, such as depth sensors, such as Figure 3B , and the features detected by the other sensors may be used to determine the pose information of the device 300. The pose of the device 300 may refer to the position of the device 300 and / or the orientation of the device 300 (e.g., represented as pitch, roll and yaw, quaternions, SE3, DCM, or any combination thereof of the device 300). The pose of the device 300 may refer to the pose of the image sensor 305, and may therefore include the position of the image sensor 305 and / or the orientation of the image sensor 305. The pose of the device 300 may also refer to the orientation of the device, the positioning of the device, or some combination thereof.

[0079] Tracking system 340 may include a device pose determination engine 342 to determine the pose of device 300 relative to a map, in some cases using map building system 350. Device pose determination engine 342 may mark the pose of device 300 on a map, in some cases using map building system 350. In some cases, device pose determination engine 342 may determine and store a pose history within a map or elsewhere. The history of poses may represent the path of device 300. Device pose determination engine 342 may also perform any of the processes discussed with respect to the determination of pose 245 of VSLAM device 205 of conceptual diagram 200. In some cases, device pose determination engine 342 may determine the pose of device 300 by determining the pose of the body of device 300, determining the pose of image sensor 305, determining the pose of another sensor such as a depth sensor, or some combination thereof. One or more of the poses may be a separate output of tracking system 340. Device pose determination engine 342 may, in some cases, merge or combine two or more of the three poses into a single output for tracking system 340 , such as by averaging pose values ​​corresponding to two or more of the three poses.

[0080] Tracking system 340 may include feature tracking engine 344 and identify features within image 320. Feature tracking engine 344 may perform feature tracking by identifying features in image 320 that have been previously identified in one or more previous images or from features identified by map building system 350. For example, based on pose information of device 300, map building system 350 may provide information predicting the location of the feature within image 320. Feature tracking engine 344 may also track changes in one or more locations of features between different images. For example, feature extraction engine 330 may detect lane markings in a first image. Feature extraction engine 330 may detect the same feature (e.g., the same lane marking) depicted in a second image captured by image sensor 305 after the first image and received therefrom. Feature tracking engine 344 may recognize that the features detected in the first image and the second image are two depictions of the same feature (e.g., lane marking). Feature tracking engine 344 may recognize that the feature has moved between the first image and the second image. For example, feature tracking engine 344 may recognize that the feature is depicted on the right hand side of the first image and depicted in the center of the second image.

[0081] In some aspects, the apparatus 300 may include a map building system 350 configured to perform VSLAM or other localization techniques. The map building system 350 may include a stored map, such as a 3D point cloud, which identifies various objects known within the environment. A 3D point cloud is a data structure that includes mathematical relationships and other information such as annotations (e.g., labels describing features), voxels, images, and forms a map that can be used by the map building system 350 to identify locations. In some aspects, although a 3D point cloud is described as an example of a map, a 3D point cloud is a complex data structure that a computing device (e.g., device 300) can use to find out locations and features within the environment of the device 300 using mathematics, logical functions, and machine learning (ML) techniques. In some examples, a 3D point cloud may consist of millions of data points that identify various points in space that the device 300 can use for localization and navigation. In some aspects, the map building system 350 may be configured to predict the location of the device 300 for multiple frames based on the pose and location of the device 300. Device 300 may utilize various information such as velocity, acceleration, etc. to determine positioning over a period of time (e.g., 333 ms or 10 frames if device 300 has an internal processing rate of 30 frames per second (FPS). Mapping system 350 may also identify mapping information including features within a map (e.g., a 3D point cloud) that device 300 is able to perceive based on the pose information. The 3D point cloud is an example of mapping information that device 300 may use, and other types of mapping information may be used.

[0082] In some aspects, the map building system 350 may at least partially use cloud computing capabilities and offload computations to another device. For example, an autonomous vehicle may provide extracted information (e.g., images processed for edge detection, 3D point clouds processed for edge detection, etc.) to another device for map building functions. In this regard, the device 300 may provide pose information to an external system and receive a predicted positioning of the device 300 for multiple frames. In other aspects, the map building system 350 may store 3D point clouds that are mapped by other devices (e.g., other autonomous vehicles) and annotated with features (e.g., road signs, vegetation, etc.) by humans and machine labelers (e.g., AI-based software trained to identify various features in 3D point clouds).

[0083] In some aspects, the map building system 350 may generate a map of the environment based on a set of coordinates determined by the device 300 for all map points for all detected and / or tracked features, including features extracted by the feature extraction engine 330. In some cases, when the map building system 350 first generates a map, the map may begin as a map of a small portion of the environment. As more features are detected from more images, and as more features are converted into map points that the map building system 350 updates the map to include, the map building system 350 may expand the map to map larger and larger portions of the environment. The map may be sparse or semi-dense. In some cases, the selection criteria used by the map building system 340 for map points corresponding to features may be demanding to support robust tracking of features using the feature tracking engine 330.

[0084] In some aspects, the map building system 350 may include a repositioning engine 352 (e.g., repositioning engine 230) to determine the location of the device 300 within the map. For example, if the tracking system 340 cannot identify any features in the image 320 or features identified in previous images, the repositioning engine may reposition the device 300 within the map. The repositioning engine 352 may determine the location of the device 300 within the map by matching the features identified in the image 320 and the feature extraction engine 330 with features corresponding to map points in the map, or some combination thereof. The repositioning engine 352 may be part of the device 300 and / or the remote server. The repositioning engine 352 may also perform any of the processes discussed with respect to the repositioning engine 230 of the conceptual diagram 200.

[0085] The closed loop detection engine 354 may be part of the device 300 and / or the remote server. The closed loop detection engine 354 may identify when the device 300 has completed travel along a path that is shaped like a closed loop or another closed shape without any gaps or openings. For example, the closed loop detection engine 354 may identify that at least some of the features depicted and detected in the image 320 match features identified earlier during travel along the path on which the device 300 is traveling. The closed loop detection engine 354 may detect closed loops based on a map generated and updated by the map building system 350 and based on a pose determined by the device pose determination engine 342. The closed loop detection of the closed loop detection engine 354 prevents the feature tracking engine 344 from mistakenly treating certain features depicted and detected in the image 320 as new features when those features match features previously detected earlier in the same location and / or area during travel along the path that the device 300 has traveled.

[0086] The map construction system 350 may also include a map projection engine 356 configured to identify features within the environment based on the map. For example, based on the map construction information and the pose information, the control system 360 is configured to construct a feature map into a data structure corresponding to the image 320. For example, the control system 360 may mathematically convert 3D data associated with a map (e.g., a 3D point cloud) and project the features into a two-dimensional (2D) coordinate space, such as a 2D map corresponding to the image 320. In one aspect, the map projection engine 356 may create a data structure that identifies features that can be identified in the image 320 based on the pose. This feature allows the features within the 2D image to be associated with the map to perform high-fidelity tracking of the device 300 in the environment. In another example, the control system may generate a 2D array (e.g., a bitmap) having values ​​that identify features associated with the map, and can be used to identify and track features in the image 320.

[0087] The control system 360 is configured to receive a predicted location of the device 300. Although not illustrated, the control system 360 may also receive other information, such as features extracted by the feature extraction engine 330, pose information, and the image 320. In one illustrative aspect, the control system 360 includes an actuator control engine 362 configured to control various actuators (not shown) of the device 300 to move the device 300 within the environment. The control system 360 may also include a path planning engine 364 to plan a path along which the device 300 will travel using a transportation vehicle. The path planning engine 364 may plan a path based on a map, based on the pose of the device 300, based on repositioning performed by the repositioning engine 352, and / or based on closed loop detection performed by the closed loop detection engine 354. The path planning engine 364 may be part of the device 300 and / or a remote server. The path planning engine 364 may also perform any of the processes discussed with respect to the path planning engine 260 of the conceptual diagram 200.

[0088] In some aspects, features within image 320 are not consistently illuminated. For example, a bench along the path of an autonomous bus will have different lighting based on the time of year, time of day, and surrounding environment (e.g., buildings, vegetation, lighting, etc.). In this case, the bench may be in a poorly lit environment due to the positioning of the sun, while other objects in the distance are bright. Device 300 may be positioned in such a dark, poorly lit environment, and the feature to be tracked by device 300 may be positioned at a great distance, which device 300 may not be able to accurately track based on the distance to the feature. Figure 4 An example of a poorly lit environment is illustrated.

[0089] In some aspects, control system 360 may include image acquisition engine 366 configured to control image sensor 305 to optimize image 320. For example, image acquisition engine 366 may determine Figure 4 The foreground area 420 in the image 320 includes multiple features used for SLAM and positioning techniques, but the features in the image 320 are underexposed (e.g., too dark). In this case, based on the underexposure of the features in the image 320 near the device 300, the feature extraction engine 330 may not identify the features in the image 320.

[0090] In some cases, the image acquisition engine 366 may be configured to control exposure based on features that are predicted to be located within the image 320. In one illustrative aspect, the image acquisition engine 366 identifies relevant features in the image 320 based on the pose information and the map construction information. For example, the image acquisition engine 366 may receive a 2D map corresponding to the image 320 from the map projection engine 356. The image acquisition engine 366 may identify features within the image 320 for tracking based on the number of features and the depth of the features (corresponding to the distance of the features within the image from the image sensor 305). The image acquisition engine 366 may determine that the image 320 is underexposed for feature tracking.

[0091] As a result of underexposure of some areas of the image 320, the feature extraction engine 330 may not be able to accurately identify features. In some aspects, the image acquisition engine 366 is configured to analyze the image and control the image sensor 305 to capture the image. For example, the image acquisition engine 366 may control the gain of the image sensor 305 and / or may control the exposure time of the image sensor 305. Controlling the gain and / or exposure time may increase the brightness (e.g., brightness) of the image. After controlling the gain and / or exposure time, the feature extraction engine 330 may be able to identify and extract features and provide the features to the tracking system 340 for tracking.

[0092] Although device 300 is described as using positioning technology for autonomous navigation, device 300 may also be a device capable of moving with 6 degrees of freedom (6DOF). In some aspects, 6DOF refers to the freedom of a rigid body to move in three-dimensional space by translating and rotating in a 3D coordinate system. For example, device 300 may be moved along a path, rotated along a path, or a combination of movement and rotation by a user. For example, if device 300 is a head-mounted display (HMD) (e.g., an XR headset) worn by a user, device 300 may be moved along a path by a user. In some cases, the environment may be a virtual environment or a partially virtual environment presented at least in part by device 300. For example, if device 300 is an AR, VR, or XR headset, at least a portion of the environment may be virtual.

[0093] The device 300 can use the map to perform various functions with respect to the locations depicted or defined in the map. For example, using a robot as an example of a device 300 utilizing the techniques described herein, the robot can actuate a motor to move the robot from a first location to a second location. For example, the second location can be determined using an environmental map to ensure that the robot avoids running into a wall or other obstacle whose location has been identified on the map, or to avoid inadvertently revisiting a location that the robot has already visited. In some cases, the device 300 may plan to revisit a location that the device 300 has already visited. For example, the device 300 may revisit a previous location to verify a previous measurement, to correct drift in the measurement after reaching the end of a long path in a closed-loop path or otherwise, to improve the accuracy of map points that appear to be inaccurate (e.g., outliers) or have low weights or confidence values, to detect more features in an area that includes few and / or sparse map points, or some combination thereof. The device 300 may actuate a motor to move from an initial position to a target position to achieve a goal, such as food delivery, package delivery, package retrieval, capturing image data, mapping an environment, finding and / or reaching a charging station or power outlet, finding and / or reaching a base station, finding and / or reaching an exit from an environment, finding and / or reaching an entrance to an environment or another environment, or some combination thereof.

[0094] In another illustrative aspect, the device 300 may be used to track and plan the movement of the device 300 within an environment. For example, an autonomous vehicle may use features extracted from an image sensor (e.g., image sensor 305) and other sensors to navigate on a road. Features tracked by the autonomous vehicle that may be in a poorly lit environment are lane markings, which the autonomous vehicle may use to prevent collisions, change lanes when appropriate, and the like. When lane markings close to the autonomous vehicle are in a poorly lit environment and other lane markings at a longer distance from the autonomous vehicle are in a higher lit environment, the autonomous vehicle may not be able to accurately identify the lane markings and may incorrectly identify the location and / or pose of the autonomous vehicle. In some aspects, the image acquisition engine 366 may be configured to control the image sensor 305 based on objects closer to the autonomous vehicle to improve tracking and positioning functions.

[0095] In one exemplary aspect, image acquisition engine 366 is configured to identify different regions of image 320 by dividing image 320 into a grid and analyzing features from each bin. In some aspects, features of the bins may be detected from image 320 or may be determined based on a 2D map generated by 356. Figure 5An example illustration of a grid generated based on the division of the image 320 into bins (or cells) of a 4x6 grid is illustrated. The image acquisition engine 366 may be configured to determine an average brightness associated with pixels in each bin. The image acquisition engine 366 may also determine the number of features in each bin based on the image 320. In another illustrative example, the image acquisition engine 366 may also determine the number of features in each bin based on the 2D map generated by the map projection engine 356, as features may not be adequately identified based on underexposure of the image 320. The image acquisition engine 366 may also determine the distance to an object or the depth of a feature of an object within each bin.

[0096] The image acquisition engine 366 is also configured to select a representative bin based on the depth of the features of the object and the number of features in the bin. As mentioned above, tracking of features for positioning may be more accurate for closer features (e.g., features with a lower depth), and the image acquisition engine 366 may identify a candidate bin with the largest number of features (e.g., 2) and features at the smallest depth relative to the image sensor or camera (corresponding to the distance from the image sensor or camera). In other examples, a representative bin may be selected from the candidate bins based on the depth of the features, because tracking accuracy is related to the distance from the image sensor or camera to the object (the depth of the features), where higher tracking accuracy may be achieved based on nearby objects or features.

[0097] Figure 3B 375 is a conceptual diagram of a device 375 for capturing images based on positioning and mapping techniques taking into account lighting conditions of an environment. The device 375 includes an image sensor 305, a motion sensor 310, a feature extraction engine 330, a motion detection engine 335, a tracking system 340, and a local map construction system 350, and a control system 360. The device 375 also includes a depth sensor 380 configured to generate depth information 385 and an object detection engine 390 configured to detect objects within the depth information 385.

[0098] In some aspects, the depth sensor 380 is configured to provide distance information about objects within the environment. Illustrative examples of depth sensors include light detection and ranging (LiDAR) sensors, radar, and time of flight (ToF) sensors. Depth information 385 is provided to the map building system 350 to perform identification of various objects within the environment based on segmentation, edge detection, and other recognition techniques. For example, the map building system 350 can identify edges associated with hard structures, lane markings, and other environmental features based on different reflections from signals associated with the closed-loop detection engine 354.

[0099] Tracking system 340 may use object information detected by map building system 350 to perform the various functions described above in conjunction with or independently of feature extraction engine 330. In one aspect, map building system 350 may be able to identify features that are more distinguishable from image 320, and feature tracking engine 344 may use depth information 385 to track features in image 320 or depth information 385. For example, a pole or sign may be easier to identify in depth information 385 because objects behind road signs are at a greater distance. In some exemplary aspects, map building system 350 may use depth information 385 to create and populate a map based on depth information 385, and continue to update the map based on depth information 385. Map building system 350 may also be configured to combine image 320 with depth information 385 to identify changes within the environment.

[0100] In other aspects, the device 375 may omit the map building system 350 and use an external calibration engine to navigate the environment without map building information, and the external calibration engine and objects detected by the map building system 350 may be used by the tracking system 340 to control image acquisition. The tracking system 340 may include an object projection engine 346 to project objects detected by the map building system 350 into a 2D map corresponding to the image 320. For example, the closed loop detection engine 354 may be located on a different surface of the device 300 and have a different orientation than the image sensor 305, and the features in the image 320 and the external calibration engine are not aligned. In one aspect, the features detected by the map building system 350 are in a 3D coordinate space and may be projected into a 2D map corresponding to the image 320. The feature tracking engine 344 may use the 2D map to track the external calibration engine and the features in the image 320. In one illustrative aspect, the image acquisition engine 366 may compare features identified by the map building system 350 within the 2D map to corresponding areas in the image 320 and control the image sensor 305 based on the brightness of the areas in the image 320. For example, the closed-loop detection engine 354 operates by projecting a signal (e.g., electromagnetic energy) and receiving the signal and may sense features that are independent of lighting, and the brightness of the image 320 is related to lighting. The image acquisition engine 366 may determine that the image 320 is underexposed based on the brightness.

[0101] Figure 4 is an example image 400 that illustrates a poorly lit environment due to the sun-based positioning of the image capture device being blocked by a building, and the background of the background region 410 is fully illuminated. The background region 410 is far away and may not be available to SLAM and localization techniques. The foreground region 420 includes features that can be used for tracking but are underexposed due to shadows provided by the building.

[0102] Figure 55 is an example image 500 generated by a device configured to control an image sensor based on positioning according to some aspects of the present disclosure. For example, the image acquisition engine 366 can control the image sensor 305 to capture the example image 500 based on generating a grid of bins and determining exposure control based on features within a representative bin selected from a plurality of candidate bins 510. The example image 500 corresponds to Figure 4 400 in the image 400 and enhanced to improve object detection within the first bin 512. For example, the image acquisition engine 366 may identify the candidate bins 510 and then select the first bin 512 as representative based on a combination of the depth of the features within the first bin 512 and the number of features within the first bin 512. For example, multiple features from the candidate bins 510 may have different edges that can be tracked by the feature tracking engine 344, and the first bin 512 may be selected based on the depth of the features within the first bin 512 and the number of features. However, as Figure 4 As shown, the corresponding area is underexposed and the feature may not be accurately identified during feature extraction (e.g., by feature extraction engine 330). In one aspect, image acquisition engine 366 is configured to increase the exposure time of image sensor 305 to create example image 500 and increase the brightness associated with first bin 510. For example, image acquisition engine 366 may determine the exposure time based on the average brightness of first bin 510 (e.g., a representative bin) in a previous image.

[0103] By increasing the exposure time, features of example image 500 may be identified by a device (eg, by feature extraction engine 330 of device 300 ) to improve identification and tracking of features that are closer to the device.

[0104] Fig. 6A An example of visualization of a 3D point cloud by a computing device identifying a region of interest according to some aspects of the present disclosure is illustrated. Specifically, Fig. 6A The 3D point cloud illustrated in FIG. 6 is a visualization of a computing device (e.g., computing system 1000) that is required for human understanding. In some aspects, the 3D point cloud is generated by a depth sensor such as a LiDAR sensor, and each of the points corresponds to a numerical distance. Region 602 may be identified as an area of ​​interest because Fig. 6A The edges of objects within can be identified based on comparisons with nearby points in the point cloud. In some aspects, a device using positioning and VSLAM techniques can identify region 602 based on the number of features in region 602 and the depth of the features. Based on region 602, the device can control an image sensor (e.g., image sensor 305) to capture region 602 with an exposure time to ensure that the average brightness of the image (e.g., image 320) has sufficient brightness for object detection (e.g., by map building system 350) and feature extraction (e.g., by feature extraction engine 330).

[0105] Figure 6B An example of visualization of a 3D point cloud by a computing device identifying a region of interest according to some aspects of the present disclosure is illustrated. Specifically, Figure 6B The 3D point cloud illustrated in is a visualization of a computing device (e.g., computing system 1000) required for human understanding. In some aspects, the 3D point cloud is generated by a depth sensor such as a LiDAR sensor, and each point corresponds to a numerical distance. Region 604 can be identified as an area of ​​interest because Figure 6B The edges of the objects within correspond to the particular shapes of interest. For example, based on the density of the points and the pattern corresponding to the lane markings, region 604 may correspond to a lane marking. In some aspects, a device using positioning or VSLAM technology may identify region 604 and control an image sensor (e.g., image sensor 305) to capture region 604 with an exposure time to ensure that the average brightness of the image (e.g., image 320) has sufficient brightness for object detection (e.g., by map building system 350) and feature extraction (e.g., by feature extraction engine 330).

[0106] Fig. 7A is a perspective view 700 illustrating an unmanned ground vehicle 710 performing visual simultaneous localization and mapping (VSLAM). Fig. 7A The ground vehicle 710 illustrated in the perspective view 700 may be a vehicle for performing Figure 2 The VSLAM device 205 of the VSLAM technology illustrated in the conceptual diagram 200, the execution Figure 3A The device 300 and / or implementation of the positioning or other VSLAM technology illustrated in the conceptual diagram 300 Figure 3B . Ground vehicle 710 includes image sensor 305 along the front surface of ground vehicle 710. Ground vehicle 710 may also include depth sensor 380. Ground vehicle 710 includes a plurality of wheels 715 along the bottom surface of ground vehicle 710. Wheels 715 may be used as a means of transportation for ground vehicle 710 and may be motorized using one or more motors. The motors, and therefore wheels 715, may be actuated to move ground vehicle 710 via movement actuator 265.

[0107] Figure 7B is a perspective view 750 illustrating an aerial (or airborne) vehicle 720 implementing VSLAM or other localization techniques. Figure 7B The aerial vehicle 720 illustrated in the perspective view 750 may be a vehicle that performs Figure 2 The VSLAM device 205 of the VSLAM technology illustrated in the conceptual diagram 200, the execution Figure 3AThe VSLAM or positioning technology device 300 illustrated in the conceptual diagram 300, and / or the implementation Figure 3B 70. An example of a device 375 for the positioning technology illustrated in . Aerial vehicle 720 includes image sensor 305 along the front of the body of ground vehicle 710. Aerial vehicle 720 may also include depth sensor 380. Aerial vehicle 720 includes multiple propellers 725 along the top of aerial vehicle 720. Propellers 725 may be spaced apart from the body of aerial vehicle 720 by one or more attachments to prevent propellers 725 from hitting circuits on the body of aerial vehicle 720 and / or to prevent propellers 725 from obstructing the field of view of image sensor 305 and depth sensor 380. Propellers 725 may be used as a means of transportation for aerial vehicle 720 and may be motorized using one or more motors. The motors and therefore propellers 725 may be actuated to move aerial vehicle 720 via motion actuator 265.

[0108] In some cases, the propeller 725 of the aerial vehicle 720 or another part of the VSLAM device 205, device 300, or device 375 (e.g., an antenna) may partially block the field of view of the image sensor 305 and the depth sensor 380. In some examples, the partial occlusion may be edited out from any image and / or IR image in which the partial occlusion occurs before feature extraction is performed. In some examples, the partial occlusion is not edited out from the VL image and / or IR image in which the partial occlusion occurs before feature extraction is performed, but the VSLAM algorithm is configured to ignore the partial occlusion for the purpose of feature extraction, and therefore any portion of the partial occlusion is not considered a feature of the environment.

[0109] Fig. 8A is a perspective diagram 800 illustrating a head mounted display (HMD) 810 performing visual simultaneous localization and mapping (VSLAM). The HMD 810 may be an XR headset. Fig. 8A The HMD 810 illustrated in the perspective view 800 may be an implementation Figure 2 The VSLAM device 205 of the VSLAM technology illustrated in the conceptual diagram 200, the execution Figure 3A The VSLAM technology or other positioning technology device 300 illustrated in the conceptual diagram 300, and / or the implementation Figure 3B . The HMD 810 includes an image sensor 305 and a depth sensor 380 along the front of the HMD 810. The HMD 810 can be, for example, an augmented reality (AR) headset, a virtual reality (VR) headset, a mixed reality (MR) headset, or some combination thereof.

[0110] Figure 8B is an example Fig. 8A 810 is a perspective view 830 of a head mounted display (HMD) being worn by a user 820. The user 820 wears the HMD 810 on the head of the user 820 above the eyes of the user 820. The HMD 810 may capture a VL image with an image sensor 305 and a depth sensor 380. In some examples, the HMD 810 displays one or more images based on the VL image and / or the IR image to the eyes of the user 820. For example, the HMD 810 may provide overlay information on the environment view to the user 820. In some examples, the HMD 810 may generate two images to display to the user 820—one image is displayed to the left eye of the user 820, and one image is displayed to the right eye of the user 820. Although the HMD 810 is illustrated as having only one image sensor 305 and depth sensor 380, in some cases, the HMD 810 (or any other VSLAM device 205 or device 300) may have more than one image sensor 305. For example, in some examples, the HMD 810 may include a pair of cameras located on both sides of the HMD 810. Thus, a stereoscopic view may be captured by the camera and / or displayed to the user. In some cases, the VSLAM device 205, the device 300, or the device 375 may also include more than one image sensor 305 for stereoscopic image capture.

[0111] HMD 810 does not include wheels 715, propellers 725 or other means of transport of its own. Instead, HMD 810 relies on the movement of user 820 to move HMD 810 back and forth in the environment. Therefore, in some cases, when performing VSLAM technology, HMD 810 can skip the path planning using path planning engines 260 and 364 and / or the movement actuation using mobile actuators 265. In some cases, HMD 810 can still perform path planning using path planning engines 260 and 364, and can indicate to user 820 the direction to follow the suggested path to guide the user along the suggested path planned using path planning engines 260 and 364. In some cases, such as when HMD 810 is a VR headset, the environment can be completely or partially virtual. If the environment is at least partially virtual, the movement through the virtual environment can also be virtual. For example, the movement through the virtual environment can be controlled by one or more joysticks, buttons, video game controllers, mice, keyboards, trackpads and / or other input devices. The mobile actuator 265 may include any such input device. The movement by the virtual environment may not require wheels 715, propellers 725, legs or any other form of transport. If the environment is a virtual environment, the HMD 810 may still perform the path planning and / or mobile actuation 265 using the path planning engines 260 and 364. If the environment is a virtual environment, the HMD 810 may perform the mobile actuation using the mobile actuator 265 by performing virtual movement in the virtual environment. Even if the environment is virtual, VSLAM technology may still be valuable because the virtual environment may be de-mapped and / or generated by a device (such as a remote server or a game console associated with a video game or a video game platform) other than the VSLAM device 205, device 300 or device 375. In some cases, VSLAM may even be performed in a virtual environment by a VSLAM device 205, device 300 or device 375 having its own physical transport system that allows it to physically move back and forth in a physical environment. For example, VSLAM may be performed in a virtual environment to test whether the VSLAM device 205, device 300, or device 375 is functioning properly without wasting time or energy on movement and without wearing out the physical transportation system of the VSLAM device 205, device 300, or device 375.

[0112] Figure 8C840 is a perspective view of a front surface 855 of a mobile phone 850 illustrating VSLAM using front cameras 310 and 315 according to some examples. The mobile phone 850 can be, for example, a cellular phone, a satellite phone, a portable game console, a music player, a health tracking device, a wearable device, a wireless communication device, a laptop computer, a mobile device, or a combination thereof. The front surface 855 of the mobile phone 850 includes a display screen 845. The front surface 855 of the mobile phone 850 includes at least one image sensor 305 and may include a depth sensor 380. At least one image sensor 305 and may include a depth sensor 380 are illustrated in a bezel around the display screen 845 on the front surface 855 of the mobile device 850. In some examples, at least one image sensor 305 and the depth sensor 380 may be positioned in a notch or cutout cut out of the display screen 845 on the front surface 855 of the mobile device 850. In some examples, the at least one image sensor 305 and may include a depth sensor 380 may be an under-display camera positioned between the display screen and the rest of the mobile phone 850 such that light passes through a portion of the display screen before reaching the at least one image sensor 305 and may include a depth sensor 380. The at least one image sensor 305 and may include a depth sensor 380 of the perspective view 840 is front-facing. The at least one image sensor 305 and may include a depth sensor 380 faces a direction perpendicular to the planar surface of the front surface 855 of the mobile device 850.

[0113] Fig.8D 860 is a perspective view of a rear surface 865 of a mobile phone 850 illustrating VSLAM using rear cameras 310 and 315 according to some examples. At least one image sensor 305 of the perspective view 860 and may include a depth sensor 380 is rear-mounted. The at least one image sensor 305 and may include a depth sensor 380 faces a direction perpendicular to the flat surface of the rear surface 865 of the mobile phone 850. Although the rear surface 865 of the mobile phone 850 does not have a display screen 845 as illustrated in the perspective view 860, in some examples, the rear surface 865 of the mobile phone 850 may have a display screen 845. If the rear surface 865 of the mobile phone 850 has a display screen 845, any positioning of the at least one image sensor 305 and may include a depth sensor 380 relative to the display screen 845 may be used as discussed with respect to the front surface 855 of the mobile phone 850.

[0114] As with HMD 810, mobile phone 850 does not include wheels 715, propellers 725, or other means of transportation of its own. Instead, mobile phone 850 relies on the movement of the user holding or wearing mobile phone 850 to move mobile phone 850 back and forth in the environment. Therefore, in some cases, when performing VSLAM technology, mobile phone 850 can skip path planning using path planning engines 260 and 364 and / or mobile actuation using mobile actuator 265. In some cases, mobile phone 850 can still perform path planning using path planning engines 260 and 364, and can indicate to the user the direction to follow the suggested path to guide the user along the suggested path planned using path planning engines 260 and 366. In some cases, such as when mobile phone 850 is used for AR, VR, MR, or XR, the environment can be fully or partially virtual. In some cases, the mobile phone 850 can be inserted into the head mounted device so that the mobile phone 850 is used as a display of the HMD 810, wherein the display screen 845 of the mobile phone 850 is used as a display of the HMD 810. If the environment is at least partially virtual, the movement through the virtual environment can also be virtual. For example, the movement through the virtual environment can be controlled by one or more joysticks, buttons, video game controllers, mice, keyboards, trackpads, and / or other input devices coupled to the mobile phone 850 in a wired or wireless manner. The mobile actuator 265 may include any such input device. The movement through the virtual environment may not require wheels 715, propellers 725, legs, or any other form of transportation. If the environment is a virtual environment, the mobile phone 850 can still perform path planning and / or mobile actuation 265 using the path planning engines 260 and 364. If the environment is a virtual environment, the mobile phone 850 can perform mobile actuation using the mobile actuator 265 by performing virtual movement within the virtual environment.

[0115] Fig. 9 is a flowchart illustrating an example of a method 900 for processing audio according to certain aspects of the present disclosure. The method 900 may be performed by a computing device configured to provide an audio stream, such as a mobile wireless communication device, an extended reality (XR) device (e.g., a VR device, an AR device, an MR device, etc.), a network-connected wearable device (e.g., a network-connected watch), a vehicle or a component or system of a vehicle, a laptop computer, a tablet computer, or another computing device. In an illustrative example, the following reference is made to Fig.10 The described computing system 1000 may be configured to perform all or part of the method 900 .

[0116] At block 905, the imaging device may obtain a first image of the environment from an image sensor of the imaging device. In this case, the first image may have brighter areas and darker areas as described above. The brighter areas may be farther away and the darker areas may be closer, which may cause problems with tracking accuracy based on underexposure of objects with smaller feature depths.

[0117] At block 910, the imaging device may determine a region of interest for the first image based on features depicted in the first image. The features may be associated with the environment and may include fixed features such as hardscape, landscape, vegetation, road signs, buildings, etc. The imaging device may determine the region of interest in the first image by predicting the locations of features associated with the environment in a 2D map. The 2D map corresponds to an image obtained by an image sensor, and the imaging device may divide the 2D map into a plurality of bins, classify the bins based on the number of features and the depth of the features, and select one or more candidate bins from the classified bins.

[0118] In one exemplary aspect, the region of interest of the first image may be determined based on depth information obtained using a depth sensor of the imaging device. For example, the depth sensor may include at least one of a LiDAR sensor, a radar sensor, or a ToF sensor.

[0119] In one aspect, in order to predict the location of a feature associated with an environment in a 2D map, the imaging device may determine the position and orientation of the imaging device, obtain three 3D positions of the features associated with the 3D map of the environment based on the position and orientation of the imaging device within the environment, and build the 3D position map of the features associated with the map into the 2D map based on the position and orientation of the imaging device and the position of the image sensor. In some aspects, the 3D position of the features associated with the map may be provided by a map building server. For example, the imaging device may send the position and orientation of the imaging device to a map builder (mapper) server and receive the 3D position of the features associated with the map from the map builder server. On the other hand, the imaging device may store a 3D map, and in order to obtain the 3D position of the features associated with the map, the imaging device may use the position and orientation of the imaging device to determine the 3D position of the features based on the 3D map stored in the imaging device.

[0120] The imaging device may select one or more candidate bins from the classified bins by determining a respective number of features in each bin from the plurality of bins, determining a respective depth of the features within each bin from the plurality of bins, and determining one or more candidate bins from the plurality of bins based on comparing each respective depth of the features in each bin and each respective number of features to a depth threshold and a minimum number of features. For example, a feature may be an edge within an environment, such as hardscape, that corresponds to a feature identified in a map. In one illustrative aspect, the imaging device may select a representative bin from the one or more candidate bins. For example, the imaging device may select a representative bin from the one or more candidate bins based on the number of features in a first bin being greater than a minimum number of features and the first bin having a maximum number of features below a depth threshold compared to the one or more candidate bins.

[0121] At block 915, the imaging device may determine a representative brightness value associated with the first image based on the image data in the region of interest of the first image. In one example, the representative brightness may be determined by the imaging device based on a determination of a representative brightness value associated with the first image based only on the image data in the region of interest. For example, the representative brightness may be determined based on a representative bin. In some aspects, the representative brightness value is an average brightness of the image data in the region of interest. In other aspects, the imaging device may determine the representative brightness value based on the image data in the region of interest by determining a representative brightness value associated with the first image based on an average of the image data in the region of interest that is scaled. For example, an average brightness may be determined for the entire image, but pixels in the representative bin may be weighted to have a greater impact on controlling image capture settings.

[0122] At block 920, the imaging device may determine one or more exposure control parameters based on the representative brightness value. In some aspects, the one or more exposure control parameters include at least one of an exposure duration or a gain setting.

[0123] At block 925, the imaging device may obtain a second image captured based on the one or more exposure control parameters. The imaging device is configured to track the location of the imaging device in the environment based on the location of the feature in the second image.

[0124] Fig.10 is a diagram illustrating an example of a system for implementing certain aspects of the disclosed technology. Specifically, Fig.10An example of a computing system 1000 is illustrated, which may be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any components thereof, wherein the components of the system communicate with each other using a connection 1005. The connection 1005 may be a physical connection using a bus, or a direct connection into the processor 1010, such as in a chipset architecture. The connection 1005 may also be a virtual connection, a networked connection, or a logical connection.

[0125] In some aspects, computing system 1000 is a distributed system in which the functionality described in the present disclosure may be distributed within a data center, multiple data centers, a peer-to-peer network, etc. In some aspects, one or more of the described system components represent a number of such components that each perform some or all of the functionality for which the component is described. In some aspects, each component may be a physical or virtual device.

[0126] The example computing system 1000 includes at least one processing unit (CPU or processor) 1010 and connections 1005 that couple various system components including system memory 1015 such as ROM 1020 and RAM 1025 to the processor 1010. The computing system 1000 may include a cache 1012 of high-speed memory directly connected to the processor 1010, in close proximity to the processor, or integrated as part of the processor.

[0127] Processor 1010 may include any general purpose processor and hardware or software services, such as services 1032, 1034, and 1036 stored in storage device 1030, that are configured to control processor 1010 as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 1010 may essentially be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0128] To enable user interaction, the computing system 1000 includes an input device 1045 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, and the like. The computing system 1000 may also include an output device 1035 that can be one or more of a plurality of output mechanisms. In some cases, a multimodal system may enable a user to provide multiple types of input / output to communicate with the computing system 1000. The computing system 1000 may include a communication interface 1040, which may generally govern and manage user input and system output. The communication interface may perform or facilitate receiving and / or sending wired or wireless communications using a wired and / or wireless transceiver, including utilizing an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, an Apple ® Lightning ® Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Dedicated wired ports / plugs, Bluetooth ® Wireless signal transmission, BLE wireless signal transmission, IBEACON ® Wireless signaling, RFID wireless signaling, near field communication (NFC) wireless signaling, dedicated short range communication (DSRC) wireless signaling, 1002.11 WiFi wireless signaling, WLAN signaling, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), IR communication wireless signaling, public switched telephone network (PSTN) signaling, integrated services digital network (ISDN) signaling, 3G / 4G / 5G / LTE cellular data network wireless signaling, ad hoc network signaling, radio wave signaling, microwave signaling, infrared signaling, visible light signaling, ultraviolet light signaling, wireless signaling along the electromagnetic spectrum, or some combination thereof. The communication interface 1040 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 1000 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' GPS, Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no restriction to operating on any particular hardware arrangement, and thus the base features herein may be easily substituted for improved hardware or firmware arrangements as they are developed.

[0129] The storage device 1030 may be a non-volatile and / or non-transitory and / or computer-readable memory device, and may be a hard disk or other type of computer-readable medium capable of storing data that can be accessed by a computer, such as a magnetic cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette, a floppy disk, a floppy disk, a hard disk, a magnetic tape, a magnetic stripe / magnetic stripe, any other magnetic storage medium, flash memory, a memristor memory, any other solid-state memory, a compact disc-read only memory (CD-ROM) disc, a rewritable compact disc (CD) disc, a digital video disc (DVD) disc, a Blu-ray disc (BDD) disc, a holographic disc, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory stick (Memory Stick) ® card, a smart card chip, a Europay, an EMV chip, a Subscriber Identity Module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a RAM, a static RAM (SRAM), a dynamic RAM (DRAM), a ROM, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin transfer torque RAM (STT-RAM), another memory chip or box, and / or a combination thereof.

[0130] Storage device 1030 may include software services, servers, services, etc., which, when the code defining such software is executed by processor 1010, causes the system to perform functions. In some aspects, hardware services that perform specific functions may include software components for performing functions stored in a computer-readable medium connected to necessary hardware components (such as processor 1010, connection 1005, output device 1035, etc.). The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data may be stored and does not include carrier waves and / or transient electronic signals propagated wirelessly or over a wired connection. Examples of non-transitory media may include, but are not limited to, disks or tapes, optical storage media (such as CDs or DVDs), flash memory, memory, or memory devices. Computer readable media may store thereon code and / or machine executable instructions, which may represent a process, function, subprogram, program, routine, subroutine, module, software package, category, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by transmitting and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be transmitted, forwarded, or sent via any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0131] In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, a computing device may include a display, one or more network interfaces configured to communicate and / or receive data, any combination thereof, and / or other components. The one or more network interfaces may be configured to communicate and / or receive wired and / or wireless data, including data in accordance with 3G, 4G, 5G, and / or other cellular standards, data in accordance with the Wi-Fi (802.11x) standard, data in accordance with the Bluetooth standard, and / or other components. ™ Standard data, data according to IP standards and / or other types of data.

[0132] The components of the computing device may be implemented in circuits. For example, each component may include and / or may be implemented using electronic circuits or other electronic hardware (which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits)), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0133] In some aspects, computer readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0134] Specific details are provided in the above description to provide a detailed understanding of the various aspects and examples provided herein. However, it will be understood by those skilled in the art that these aspects can also be practiced without these specific details. For clarity, in some cases, the present technology may be presented as including separate functional blocks, including functional blocks comprising devices, device components, steps or routines in the method embodied in software or a combination of hardware and software. Additional components other than those components shown in the accompanying drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid confusing these aspects in unnecessary details. In other examples, known circuits, processes, algorithms, structures and techniques may be shown without unnecessary details to avoid confusing various aspects.

[0135] Various aspects may be described above as a process or method, which is depicted as a flow chart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flow chart may describe an operation as a sequential process, many operations in the operation may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operation of the process is completed, but the process may have additional steps not included in the accompanying drawings. A process corresponds to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or the main function.

[0136] The processes and methods according to the above examples can be implemented using stored computer executable instructions or computer executable instructions otherwise obtained from a computer readable medium. Such instructions may include, for example, instructions and data that configure a general-purpose computer, a special-purpose computer, or a processing device to perform a function or a group of functions. Portions of the computer resources used may be accessed via a network. Computer executable instructions may be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include disks or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0137] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. By way of further example, such functionality may also be implemented on circuit boards among different chips or different processes executed on a single device.

[0138] Instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0139] In the above description, various aspects of the present application are described with reference to their specific aspects, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although the exemplary aspects of the present application have been described in detail herein, it is to be understood that each inventive concept can be implemented and adopted in various other ways, and the appended claims are not intended to be interpreted as including such variations, unless limited by the prior art. Various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, various aspects can be utilized in any number of environments and applications beyond those described herein without departing from the broader essence and scope of this specification. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be understood that, in alternative aspects, each method can be performed in a different order than described.

[0140] It should be understood by those skilled in the art that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to (" ”) and greater than or equal to (“ ”) symbol instead.

[0141] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0142] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is directly or indirectly in communication with another component (eg, connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0143] Claim language or other language stating "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language stating "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language stating "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language stating "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0144] The various exemplary logic blocks, modules, circuits, and algorithmic steps described in conjunction with the various aspects disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. The technician may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as departing from the scope of the present application.

[0145] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices, or integrated circuit devices with multiple uses, including applications in wireless communication devices, mobile phones, and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions that, when executed, perform one or more of the above methods. A computer-readable data storage medium may form part of a computer program product, which may include packaging materials. A computer-readable medium may include a memory or data storage medium, such as a RAM (such as a synchronous dynamic random access memory (SDRAM)), a ROM, a non-volatile random access memory (NVRAM), an EEPROM, a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0146] The program code may be executed by a processor, which may include one or more processors, such as one or more DSPs, general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein.

[0147] Illustrative aspects of the present disclosure include:

[0148] Aspect 1. A method for processing one or more images, the method comprising: obtaining, at an imaging device, a first image of an environment from an image sensor of the imaging device; determining a region of interest of the first image based on features depicted in the first image, wherein the features are associated with the environment; determining a representative brightness value associated with the first image based on image data in the region of interest of the first image; determining one or more exposure control parameters based on the representative brightness value; and obtaining, at the imaging device, a second image captured based on the one or more exposure control parameters.

[0149] Aspect 2. The method of aspect 1, wherein the one or more exposure control parameters include at least one of exposure duration or gain setting.

[0150] Aspect 3. A method according to any one of Aspects 1 to 2, wherein determining the one or more exposure control parameters based on the representative brightness value includes: determining at least one of an exposure duration or a gain setting for the second image based on the representative brightness value.

[0151] Aspect 4. A method according to any one of Aspects 1 to 3, wherein determining the representative brightness value based on the image data in the region of interest includes: determining the representative brightness value associated with the first image based only on the image data in the region of interest.

[0152] Aspect 5. The method according to any one of aspects 1 to 4, wherein the representative brightness value is an average brightness of the image data in the region of interest.

[0153] Aspect 6. A method according to any one of Aspects 1 to 5, wherein determining the representative brightness value based on the image data in the region of interest includes: determining the representative brightness value associated with the first image based on scaling the average brightness of the image data in the region of interest.

[0154] Aspect 7. A method according to any one of Aspects 1 to 6, wherein determining the area of ​​interest of the first image comprises: predicting, by the imaging device, the position of the features associated with the environment in a two-dimensional (2D) map, wherein the 2D map corresponds to the image obtained by the image sensor; dividing the 2D map into a plurality of bins; classifying the bins based on the number of the features and the depth of the features; and selecting one or more candidate bins from the classified bins.

[0155] Aspect 8. A method according to any one of Aspects 1 to 7, wherein predicting the position of the feature associated with the environment in the 2D map includes: determining the positioning and orientation of the imaging device; obtaining a three-dimensional (3D) positioning of the feature associated with the 3D map of the environment based on the positioning and orientation of the imaging device within the environment; and constructing, by the imaging device, a 3D positioning map of the feature associated with the map into the 2D map based on the positioning and orientation of the imaging device and the positioning of the image sensor.

[0156] Aspect 9. A method according to any one of Aspects 1 to 8, wherein obtaining the 3D positioning of the feature associated with the map comprises: sending the positioning and the orientation of the imaging device to a map builder server; and receiving the 3D positioning of the feature associated with the map from the map builder server.

[0157] Aspect 10. A method according to any one of Aspects 1 to 9, wherein obtaining the 3D position of the feature associated with the map comprises: using the position and the orientation of the imaging device to determine the 3D position of the feature based on the 3D map stored in the imaging device.

[0158] Aspect 11. A method according to any one of Aspects 1 to 10, wherein selecting the one or more candidate bins from the classified bins includes: determining the corresponding number of features in each bin from the multiple bins; determining the corresponding depth of features in each bin from the multiple bins; and determining the one or more candidate bins from the multiple bins based on comparing each corresponding depth of the features in each bin and each corresponding number of features with a depth threshold and a minimum number of features.

[0159] Aspect 12. According to any one of Aspects 1 to 11, the method further includes: selecting the first bin from the multiple bins based on the number of features in the first bin being greater than the minimum number of features and the first bin having a maximum number of features below the depth threshold compared to the one or more candidate bins.

[0160] Aspect 13. The method according to any one of aspects 1 to 12, wherein the region of interest of the first image is determined based on depth information obtained using a depth sensor of the imaging device.

[0161] Aspect 14. The method according to any one of aspects 1 to 13, wherein the depth sensor comprises at least one of a light detection and ranging (LiDAR) sensor, a radar sensor, or a time of flight (ToF) sensor.

[0162] Aspect 15. The method according to any one of Aspects 1 to 14, further comprising: tracking, at the imaging device, a position of the imaging device in the environment based on the position of the feature in the second image.

[0163] Aspect 16. A device for processing one or more images, the device comprising at least one memory and at least one processor, the at least one processor being coupled to the at least one memory, the at least one processor being configured to: obtain a first image of an environment from an image sensor of an imaging device; determine a region of interest of the first image based on features depicted in the first image, wherein the features are associated with the environment; determine a representative brightness value associated with the first image based on image data in the region of interest of the first image; determine one or more exposure control parameters based on the representative brightness value; and obtain a second image captured based on the one or more exposure control parameters.

[0164] Aspect 17. The system of aspect 16, wherein the one or more exposure control parameters include at least one of exposure duration or gain setting.

[0165] Aspect 18. A system according to any one of Aspects 16 to 17, wherein, in order to determine the one or more exposure control parameters based on the representative brightness value, the at least one processor is configured to: determine at least one of the exposure duration or gain setting for the second image based on the representative brightness value.

[0166] Aspect 19. A system according to any one of Aspects 16 to 18, wherein, in order to determine the representative brightness value based on the image data in the region of interest, the at least one processor is configured to: determine the representative brightness value associated with the first image based only on the image data in the region of interest.

[0167] Clause 20. The system according to any one of clauses 16 to 19, wherein the representative brightness value is an average brightness of the image data in the region of interest.

[0168] Aspect 21. A system according to any one of Aspects 16 to 20, wherein, in order to determine the representative brightness value based on the image data in the region of interest, the at least one processor is configured to: determine the representative brightness value associated with the first image based on scaling the average brightness of the image data in the region of interest.

[0169] Aspect 22. A system according to any one of Aspects 16 to 21, wherein, in order to determine the area of ​​interest of the first image, the at least one processor is configured to: predict the location of the features associated with the environment in a 2D map, wherein the 2D map corresponds to the image obtained by the image sensor; divide the 2D map into a plurality of bins; classify the bins based on the number of the features and the depth of the features; and select one or more candidate bins from the classified bins.

[0170] Aspect 23. A system according to any one of Aspects 16 to 22, wherein, in order to predict the position of the feature associated with the environment in the 2D map, the at least one processor is configured to: determine the position and orientation of the imaging device; obtain a three-dimensional (3D) position of the feature associated with the 3D map of the environment based on the position and orientation of the imaging device within the environment; and construct a 3D position map of the feature associated with the map into the 2D map based on the position and orientation of the imaging device and the position of the image sensor.

[0171] Aspect 24. A system according to any one of Aspects 16 to 23, wherein, in order to obtain the 3D positioning of a feature associated with the map, the at least one processor is configured to: send the positioning and the orientation of the imaging device to a map builder server; and receive the 3D positioning of the feature associated with the map from the map builder server.

[0172] Aspect 25. A system according to any one of Aspects 16 to 24, wherein, in order to obtain the 3D position of the feature associated with the map, the at least one processor is configured to: use the position and the orientation of the imaging device to determine the 3D position of the feature based on the 3D map stored in the imaging device.

[0173] Aspect 26. A system according to any one of Aspects 16 to 25, wherein, in order to select the one or more candidate bins from the classified bins, the at least one processor is configured to: determine the corresponding number of features in each bin from the multiple bins; determine the corresponding depth of features in each bin from the multiple bins; and determine the one or more candidate bins from the multiple bins based on comparing each corresponding depth of the features in each bin and each corresponding number of features with a depth threshold and a minimum number of features.

[0174] Aspect 27. A system according to any one of Aspects 16 to 26, wherein the at least one processor is configured to: select the first bin from the multiple bins based on the number of features in the first bin being greater than the minimum number of features and the first bin having a maximum number of features below the depth threshold compared to the one or more candidate bins.

[0175] Clause 28. A system according to any one of clauses 16 to 27, wherein the region of interest of the first image is determined based on depth information obtained using a depth sensor of the imaging device.

[0176] Aspect 29. The system according to any one of aspects 16 to 28, wherein the depth sensor comprises at least one of a light detection and ranging (LiDAR) sensor, a radar sensor, or a time of flight (ToF) sensor.

[0177] Aspect 30. A system according to any one of Aspects 16 to 29, wherein the at least one processor is configured to: track the location of the imaging device in the environment based on the location of the feature in the second image.

[0178] Aspect 31. A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform operations according to any one of aspects 1 to 15.

[0179] Aspect 32. An apparatus for processing one or more images, the apparatus comprising one or more components for performing the operations according to any one of aspects 1 to 15.

Claims

1. A method for processing one or more images, the method comprising: obtaining, at an imaging device, a first image of an environment from an image sensor of the imaging device; determining a region of interest of the first image based on features depicted in the first image, wherein the features are associated with the environment; determining a representative brightness value associated with the first image based on image data in the region of interest of the first image; determining one or more exposure control parameters based on the representative brightness value; as well as A second image captured based on the one or more exposure control parameters is obtained at the imaging device. 2 . The method of claim 1 , wherein the one or more exposure control parameters include at least one of exposure duration or gain setting.

3. The method of claim 1 , wherein determining the one or more exposure control parameters based on the representative brightness value comprises: At least one of an exposure duration or a gain setting for the second image is determined based on the representative brightness value.

4. The method of claim 1 , wherein determining the representative brightness value based on the image data in the region of interest comprises: The representative brightness value associated with the first image is determined based only on the image data in the region of interest. The method according to claim 1 , wherein the representative brightness value is an average brightness of the image data in the region of interest.

6. The method of claim 1 , wherein determining the representative brightness value based on the image data in the region of interest comprises: The representative brightness value associated with the first image is determined based on scaling an average brightness of the image data in the region of interest.

7. The method of claim 1 , wherein determining the region of interest of the first image comprises: predicting, by the imaging device, a location of the feature associated with the environment in a two-dimensional (2D) map, wherein the 2D map corresponds to an image acquired by the image sensor; Dividing the 2D map into a plurality of bins; classifying the bins based on the number of features and the depth of the features; as well as One or more candidate bins are selected from the classified bins.

8. The method of claim 7, wherein predicting the location of the feature associated with the environment in the 2D map comprises: determining the location and orientation of the imaging device; obtaining a three-dimensional (3D) location of a feature associated with a 3D map of the environment based on the location and the orientation of the imaging device within the environment; as well as A 3D location map of the features associated with the map is constructed into the 2D map by the imaging device based on the location and the orientation of the imaging device and the location of the image sensor.

9. The method of claim 8, wherein obtaining the 3D location of a feature associated with the map comprises: sending the location and the orientation of the imaging device to a map builder server; as well as The 3D location of a feature associated with the map is received from the map builder server.

10. The method of claim 8, wherein obtaining the 3D location of the feature associated with the map comprises: The 3D location of the feature is determined based on the 3D map stored in the imaging device using the location and the orientation of the imaging device.

11. The method of claim 7, wherein selecting the one or more candidate bins from the classified bins comprises: determining a respective quantity of the feature in each bin from the plurality of bins; determining a respective depth of a feature within each bin from the plurality of bins; as well as The one or more candidate bins are determined from the plurality of bins based on comparing each respective depth of the features and each respective number of features in each bin to a depth threshold and a minimum number of features.

12. The method according to claim 11, further comprising: The first bin is selected from the plurality of bins based on the number of features in the first bin being greater than the minimum number of features and the first bin having a maximum number of features below the depth threshold compared to the one or more candidate bins. 13 . The method of claim 1 , wherein the region of interest of the first image is determined based on depth information obtained using a depth sensor of the imaging device.

14. The method of claim 13, wherein the depth sensor comprises at least one of a light detection and ranging (LiDAR) sensor, a radar sensor, or a time of flight (ToF) sensor.

15. The method according to claim 1, further comprising: The location of the imaging device in the environment is tracked at the imaging device based on the location of the feature in the second image.

16. A device, comprising: at least one memory; and at least one processor coupled to at least one memory and configured to: obtaining a first image of the environment from an image sensor of an imaging device; determining a region of interest of the first image based on features depicted in the first image, wherein the features are associated with the environment; determining a representative brightness value associated with the first image based on image data in the region of interest of the first image; determining one or more exposure control parameters based on the representative brightness value; as well as A second image captured based on the one or more exposure control parameters is obtained.

17. The apparatus of claim 16, wherein the one or more exposure control parameters include at least one of exposure duration or gain setting.

18. The device according to claim 16, wherein: To determine the one or more exposure control parameters based on the representative brightness value, the at least one processor is configured to: At least one of an exposure duration or a gain setting for the second image is determined based on the representative brightness value.

19. The device according to claim 16, wherein: To determine the representative brightness value based on the image data in the region of interest, the at least one processor is configured to: The representative brightness value associated with the first image is determined based only on the image data in the region of interest.

20. The apparatus of claim 16, wherein the representative brightness value is an average brightness of the image data in the region of interest.

21. The device according to claim 16, wherein: To determine the representative brightness value based on the image data in the region of interest, the at least one processor is configured to: The representative brightness value associated with the first image is determined based on scaling an average brightness of the image data in the region of interest.

22. The device according to claim 16, wherein: To determine the region of interest of the first image, the at least one processor is configured to: predicting a location of the feature associated with the environment in a two-dimensional (2D) map, wherein the 2D map corresponds to an image acquired by the image sensor; Dividing the 2D map into a plurality of bins; classifying the bins based on the number of features and the depth of the features; as well as One or more candidate bins are selected from the classified bins.

23. The device according to claim 22, wherein: To predict the location of the feature associated with the environment in the 2D map, the at least one processor is configured to: determining the location and orientation of the imaging device; obtaining a three-dimensional (3D) location of a feature associated with a 3D map of the environment based on the location and the orientation of the imaging device within the environment; as well as A 3D positioning map of the features associated with the map is constructed into the 2D map based on the position and the orientation of the imaging device and the position of the image sensor.

24. The device according to claim 23, wherein: To obtain the 3D position of a feature associated with the map, the at least one processor is configured to: sending the location and the orientation of the imaging device to a map builder server; and The 3D location of a feature associated with the map is received from the map builder server.

25. The device according to claim 23, wherein: To obtain the 3D position of a feature associated with the map, the at least one processor is configured to: The 3D location of the feature is determined based on the 3D map stored in the imaging device using the location and the orientation of the imaging device.

26. The device according to claim 22, wherein: To select the one or more candidate bins from the sorted bins, the at least one processor is configured to: determining a respective quantity of the feature in each bin from the plurality of bins; determining a respective depth of a feature within each bin from the plurality of bins; as well as The one or more candidate bins are determined from the plurality of bins based on comparing each respective depth of the features and each respective number of features in each bin to a depth threshold and a minimum number of features.

27. The apparatus of claim 26, wherein the at least one processor is configured to: The first bin is selected from the plurality of bins based on the number of features in the first bin being greater than the minimum number of features and the first bin having a maximum number of features below the depth threshold compared to the one or more candidate bins.

28. The apparatus of claim 16, wherein the region of interest of the first image is determined based on depth information obtained using a depth sensor of the imaging device.

29. The device of claim 28, wherein the depth sensor comprises at least one of a light detection and ranging (LiDAR) sensor, a radar sensor, or a time of flight (ToF) sensor.

30. The apparatus of claim 16, wherein the at least one processor is configured to: The location of the imaging device in the environment is tracked based on the location of the feature in the second image.

Images