Multimodal sensor-independent positioning using adaptive feature maps

Abstract

Techniques and systems for processing image data are provided. For example, a process can include obtaining a first set of image features from one or more images of an environment captured by a camera, transforming the first set of image features to generate a first set of bird's eye view (BEV) image features, obtaining a second set of features obtained using a sensor having a different sensor type than the camera, transforming the second set of features to generate a second set of BEV features, normalizing the first set of BEV image features based on camera configuration information associated with the one or more images, normalizing the second set of BEV features based on sensor configuration information of the sensor, and generating a query graph based on the normalized first set of BEV image features and the normalized second set of BEV features.

Description

Technical Field

[0001] This application relates to processing sensor data. For example, aspects of this application relate to systems and techniques for multimodal sensor-independent localization using one or more adaptive feature maps, such as bird's-eye view (BEV) feature maps. Background Technology

[0002] A growing number of systems and devices (e.g., autonomous vehicles such as autonomous and semi-autonomous cars, drones, mobile robots, mobile devices, extended reality (XR) devices, and other suitable systems or devices) include multiple sensors for collecting information about the environment, and processing systems for processing the collected information, such as for route planning, navigation, collision avoidance, environmental modeling / rendering, etc. An example of such systems is a positioning system for XR devices and / or an advanced driver assistance system (ADAS) for vehicles. In such systems, sensor data (such as images captured from one or more cameras) can be collected, transformed, and analyzed to detect features and / or objects (e.g., targets). The detected features can be compared with features indicated on a map to determine the location of the device and / or vehicle. However, such systems may rely on a perspective view of the environment and are therefore potentially sensitive to sensor placement, occlusion, etc. Summary of the Invention

[0003] The following is a simplified summary of the invention relating to one or more aspects disclosed herein. Therefore, this summary should not be considered an exhaustive overview relating to all conceived aspects, nor should it be considered to identify key or decisive elements relating to all conceived aspects or to depict the scope associated with any particular aspect. Accordingly, the following summary presents certain concepts in a simplified form relating to one or more aspects of the mechanisms disclosed herein, preceding the detailed description that follows.

[0004] Systems and techniques for processing image data are disclosed. In one exemplary example, an apparatus for processing image data is provided. The apparatus includes at least one memory and at least one processor coupled to the at least one memory. The at least one processor is configured to: obtain a first set of image features from one or more images of an environment captured by a camera; transform the first set of image features to generate a first set of bird's-eye view (BEV) image features; obtain a second set of features, the second set of features being generated based on a representation of the environment obtained using a sensor having a different sensor type than the camera; transform the second set of features to generate a second set of BEV features; normalize the first set of BEV image features based on camera configuration information associated with the one or more images; normalize the second set of BEV features based on sensor configuration information of the sensor; and generate a query map based on the normalized first set of BEV image features and the normalized second set of BEV features.

[0005] As another example, a method for processing image data is provided. The method includes: obtaining a first set of image features from one or more images of an environment captured by a camera; transforming the first set of image features to generate a first set of bird's-eye view (BEV) image features; obtaining a second set of features generated based on a representation of the environment obtained using a sensor with a different sensor type than the camera; transforming the second set of features to generate a second set of BEV features; normalizing the first set of BEV image features based on camera configuration information associated with the one or more images; normalizing the second set of BEV features based on sensor configuration information of the sensor; and generating a query graph based on the normalized first set of BEV image features and the normalized second set of BEV features.

[0006] In another example, a non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to: obtain a first set of image features from one or more images of the environment captured by a camera; transform the first set of image features to generate a first set of bird's-eye view (BEV) image features; obtain a second set of features generated based on a representation of the environment obtained using a sensor having a different sensor type than the camera; transform the second set of features to generate a second set of BEV features; normalize the first set of BEV image features based on camera configuration information associated with the one or more images; normalize the second set of BEV features based on sensor configuration information of the sensor; and generate a query graph based on the normalized first set of BEV image features and the normalized second set of BEV features.

[0007] As another example, an apparatus for processing image data. The apparatus includes: components for obtaining a first set of image features from one or more images of an environment captured by a camera; components for transforming the first set of image features to generate a first set of bird's-eye view (BEV) image features; components for obtaining a second set of features, the second set of features being generated based on a representation of the environment obtained using a sensor having a different sensor type than the camera; components for transforming the second set of features to generate a second set of BEV features; components for normalizing the first set of BEV image features based on camera configuration information associated with the one or more images; components for normalizing the second set of BEV features based on sensor configuration information of the sensor; and components for generating a query map based on the normalized first set of BEV image features and the normalized second set of BEV features.

[0008] In some aspects, one or more devices described herein may include, or be part of, extended reality devices (e.g., virtual reality (VR) devices, augmented reality (AR) devices, or mixed reality (MR) devices), mobile devices (e.g., mobile phones or other mobile devices), wearable devices (e.g., network-connected watches or other wearable devices), personal computers, laptop computers, server computers, televisions, video game consoles, or other devices. In some aspects, the device may include at least one camera for capturing one or more images or video frames. For example, the device may include one or more cameras (e.g., an RGB camera) for capturing one or more images and / or one or more videos including video frames. In some aspects, the device may include one or more displays for displaying one or more images, videos, notifications, or other displayable data. In some aspects, the device may include one or more transmitters, one or more receivers, or one or more transceivers configured to transmit and / or receive data or information to / from at least one device via a transmission medium. In some aspects, the device may include one or more processors, which may include a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), or other processing devices or components.

[0009] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to define the scope of the claimed subject matter. This subject matter should be understood with reference to the appropriate portions of the entire specification, any or all drawings, and each claim.

[0010] The foregoing, as well as other features and examples, will become more apparent upon reference to the following description, claims, and drawings. Attached Figure Description

[0011] The following description, with reference to the accompanying drawings, details exemplary examples of this application:

[0012] Figure 1 This is a block diagram illustrating the architecture of an image capture and processing system according to various aspects of this disclosure;

[0013] Figure 2 This is a diagram illustrating the architecture of an example extended reality (XR) system according to some aspects of this disclosure;

[0014] Figure 3A These are illustrations of examples of fully connected neural networks according to some examples of this disclosure;

[0015] Figure 3B This is a diagram illustrating an example of a locally connected neural network according to some examples of this disclosure;

[0016] Figure 3C These are illustrations of examples of convolutional neural networks according to some examples of this disclosure;

[0017] Figure 3D This is an illustration of an example of a deep convolutional network (DCN) for recognizing visual features from an image, according to some examples of this disclosure;

[0018] Figure 4 This is a block diagram illustrating an example deep convolutional network (DCN) according to some examples of this disclosure;

[0019] Figure 5 Techniques for generating multimodal BEV feature maps according to various aspects of this disclosure are illustrated;

[0020] Figure 6 This is a block diagram illustrating techniques for multimodal localization using BEV feature maps according to various aspects of this disclosure;

[0021] Figure 7 This is a flowchart illustrating a process for map construction according to various aspects of this disclosure; and

[0022] Figure 8 This is a diagram illustrating an example of a system used to implement certain aspects of this technology. Detailed Implementation

[0023] Certain aspects and examples of this disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and examples may be applied independently, and some may be applied in combination. In the following description, specific details are set forth for purposes of explanation in order to provide a thorough understanding of the subject matter of this application. However, it will be apparent that the various examples may be practiced without these specific details. The accompanying drawings and descriptions are not intended to be limiting.

[0024] The following description provides only illustrative examples and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the following description will provide those skilled in the art with feasible descriptions for implementing the illustrative examples. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of this application as set forth in the appended claims.

[0025] In some cases, the device may have one or more sensors (e.g., image sensors such as cameras, distance sensors such as radar and / or light detection and ranging (LIDAR) sensors, sonar sensors, etc.). The device may include extended reality (XR) devices (e.g., virtual reality (VR) devices, augmented reality (AR) devices, and / or mixed reality (MR) devices), vehicles (e.g., automated vehicles, semi-automated vehicles, autonomous vehicles, etc.), robotic devices (e.g., automated vacuum cleaners, industrial robots, etc.), or other types of devices. One or more sensors can be used to obtain information about the environment in which the device is located. The device's processing system can be used to process the information to perform one or more operations, such as map building, localization, route planning, navigation, collision avoidance, etc. For example, in some cases, sensor data (e.g., one or more images captured from one or more cameras, depth information captured or determined by one or more radar and / or LIDAR sensors, etc.) may be obtained from one or more sensors, and the sensor data may be transformed and analyzed to determine the device's position in the environment.

[0026] Localization can be used to determine a device's precise location on a map. Localization can be performed based on data input from sensors and a map. For example, a device can use one or more sensors to obtain information about its environment. The device can then compare this information with a map of the environment to locate itself. In some cases, a device can use information about its environment obtained from one or more sensors to build a map of the environment. In some cases, Simultaneous Localization and Mapping (SLAM) techniques can be used for both localization and map building. Existing SLAM techniques often rely on perspectives, which may be sensitive sensor locations, and can use multi-layered maps where different layers correspond to different types of sensors, which can be difficult to generalize. Techniques for augmenting localization may be useful.

[0027] This document describes systems, apparatuses, electronic devices, methods (also referred to as processes), and computer-readable media (collectively referred to herein as "systems and technologies") for multimodal sensor-independent localization using adaptive feature maps. The feature map may be a bird's-eye view (BEV) feature map. For example, SLAM can be enhanced by performing SLAM in a multimodal-normalized BEV feature space using graph representations. In some cases, a set of image features may be obtained based on captured images. Another set of features may be obtained based on a representation of the environment obtained by another sensor of a different sensor type. This other sensor may be a radar sensor, a LiDAR sensor, a sonar sensor, etc. This set of image features and the other set of features can be projected onto the BEV feature space (e.g., a feature map).

[0028] A set of image features in the BEV feature space can be normalized based on camera configuration information. Similarly, another set of features can be normalized based on sensor configuration associated with another sensor. In some cases, camera configuration information may be based on calibration information associated with the camera, including information about the camera's field of view (FOV), principal point, lens distortion, etc. In some cases, sensor configuration information may include information about the sensor's mounting height, tilt angle, FOV, or distance. In some cases, camera configuration information and / or sensor configuration information may be dynamically optimized over time or based on estimates of the environment. For example, the normalization of the camera's FOV may be adapted based on the distance to the object, which may vary over time. In some cases, machine learning models may be used to predict the configuration information used for normalization. In some cases, data obtained from one sensor can be used to determine the configuration used for normalization against another sensor.

[0029] The BEV feature space can then be transformed into a graph by dividing the normalized BEV feature space into a grid of cells. Each cell can be mapped to a node in the graph, and the features in each cell can be aggregated into the corresponding node to generate a query graph. The query graph can be compared with a scene graph to determine the location of the device by identifying the portion of the scene graph that best matches the query graph. In some cases, the comparison can be based on feature descriptors of the query graph and the scene graph. In some cases, the feature descriptors of the query graph can be generated by a graph neural network.

[0030] Various aspects of this application will be described with reference to the accompanying drawings.

[0031] Figure 1This is a block diagram illustrating the architecture of an image capture and processing system 100. The image capture and processing system 100 includes various components for capturing and processing images of a scene (e.g., an image of scene 110). The image capture and processing system 100 can capture individual images (or photographs) and / or capture video comprising multiple images (or video frames) in a specific sequence. In some cases, a lens 115 and an image sensor 130 may be associated with an optical axis. In one exemplary example, both the photosensitive area of ​​the image sensor 130 (e.g., a photodiode) and the lens 115 may be centered on the optical axis. The lens 115 of the image capture and processing system 100 faces scene 110 and receives light from scene 110. The lens 115 bends the incident light from the scene toward the image sensor 130. The light received by the lens 115 passes through an aperture. In some cases, the aperture (e.g., aperture size) is controlled by one or more control mechanisms 120 and received by the image sensor 130. In some cases, the aperture may have a fixed size.

[0032] One or more control mechanisms 120 may control exposure, focus, and / or zoom based on information from image sensor 130 and / or information from image processor 150. One or more control mechanisms 120 may include multiple mechanisms and components; for example, control mechanism 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. One or more control mechanisms 120 may also include additional control mechanisms besides those illustrated, such as controls for analog gain, flash, HDR, depth of field, and / or other image capture properties.

[0033] The focus control mechanism 125B of the control mechanism 120 can obtain the focus setting. In some examples, the focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 125B can adjust the positioning of the lens 115 relative to the positioning of the image sensor 130. For example, based on the focus setting, the focus control mechanism 125B can move the lens 115 closer to or further away from the image sensor 130 by actuating a motor or servo system (or other lens mechanism), thereby adjusting the focus. In some cases, an additional lens may be included in the image capture and processing system 100, such as one or more microlenses on each photodiode of the image sensor 130, each of which bends light received from the lens 115 toward the corresponding photodiode before it reaches the photodiode. The focus setting may be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), hybrid autofocus (HAF), or some combination thereof. The focus setting may be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus settings may be referred to as image capture settings and / or image processing settings. In some cases, the lens 115 may be fixed relative to the image sensor, and the focus control mechanism 125B may be omitted without departing from the scope of this disclosure.

[0034] The exposure control mechanism 125A of the control mechanism 120 can obtain the exposure settings. In some cases, the exposure control mechanism 125A stores the exposure settings in a memory register. Based on the exposure settings, the exposure control mechanism 125A can control the aperture size (e.g., aperture size or aperture value), the duration of aperture opening (e.g., exposure time or shutter speed), the duration of light collection by the sensor (e.g., exposure time or electronic shutter speed), the sensitivity of the image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by the image sensor 130, or any combination thereof. The exposure settings may be referred to as image capture settings and / or image processing settings.

[0035] The zoom control mechanism 125C of the control mechanism 120 can obtain zoom settings. In some examples, the zoom control mechanism 125C stores the zoom settings in a memory register. Based on the zoom settings, the zoom control mechanism 125C can control the focal length of an assembly (lens assembly) of lens elements including lens 115 and one or more additional lenses. For example, the zoom control mechanism 125C can control the focal length of the lens assembly by actuating one or more motors or servo systems (or other lens mechanisms) to move one or more lenses in the lens relative to each other. The zoom settings may be referred to as image capture settings and / or image processing settings. In some examples, the lens assembly may include a parfocal zoom lens or a variable focal length zoom lens. In some examples, the lens assembly may include a focusing lens (in some cases, this focusing lens may be lens 115) that first receives light from scene 110, where the light then passes through a focusless zoom system between the focusing lens (e.g., lens 115) and image sensor 130 before reaching image sensor 130. In some cases, a focusless zoom system may include two positive (e.g., converging, convex) lenses with equal or similar focal lengths (e.g., within a threshold difference between them), with a negative (e.g., diverging, concave) lens between the two positive lenses. In some cases, the zoom control mechanism 125C moves one or more lenses in the focusless zoom system, such as a negative lens and one or both positive lenses. In some cases, the zoom control mechanism 125C can control zoom by capturing images from an image sensor (e.g., including image sensor 130) among a plurality of image sensors at a zoom setting corresponding to the zoom setting. For example, the image processing system 100 may include a wide-angle image sensor with a relatively low zoom and a telephoto image sensor with a greater zoom. In some cases, the zoom control mechanism 125C may capture images from a corresponding sensor based on the selected zoom setting.

[0036] Image sensor 130 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a specific pixel in the image generated by image sensor 130. In some cases, different photodiodes may be covered by different filters. In some cases, different photodiodes may be covered in different color filters, and light that matches the color of the filter covering the photodiode can thus be measured. Various color filter arrays can be used, including Bayer color filter arrays, four-color color filter arrays (also known as four-color Bayer color filter arrays or QCFA), and / or any other color filter array. For example, a Bayer color filter includes a red color filter, a blue color filter, and a green color filter, wherein each pixel of the image is generated based on red light data from at least one photodiode covered in the red color filter, blue light data from at least one photodiode covered in the blue color filter, and green light data from at least one photodiode covered in the green color filter.

[0037] Return to Figure 1 Other types of color filters can be used as alternatives to or supplements to red, blue, and / or green filters, such as yellow, magenta, and / or cyan (also known as "emerald green"). In some cases, photodiodes can be configured to measure infrared (IR) light. In some implementations, the photodiode measuring IR light may not be covered by any filter, thus allowing the IR photodiode to measure both visible light (e.g., color) and IR light. In some examples, the IR photodiode may be covered by an IR filter, thus allowing IR light to pass through and blocking light from other parts of the spectrum (e.g., visible light, color). Some image sensors (e.g., image sensor 130) may lack filters entirely (e.g., color, IR, or any other part of the spectrum) and may alternatively use different photodiodes (in some cases, vertically stacked) throughout the pixel array. Different photodiodes throughout the pixel array can have different spectral sensitivity profiles, thereby responding to light of different wavelengths. Monochrome image sensors may also lack filters and therefore lack color depth.

[0038] In some cases, image sensor 130 may optionally or additionally include opaque and / or reflective masks that block light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles. In some cases, opaque and / or reflective masks may be used for phase detection autofocus (PDAF). In some cases, opaque and / or reflective masks may be used to block portions of the electromagnetic spectrum from reaching the photodiodes of the image sensor (e.g., IR cutoff filters, UV cutoff filters, bandpass filters, low-pass filters, high-pass filters, etc.). Image sensor 130 may also include an analog gain amplifier for amplifying the analog signal output from the photodiodes and / or an analog-to-digital converter (ADC) for converting the analog signal output from the photodiodes (and / or amplified by the analog gain amplifier) ​​into a digital signal. In some cases, certain components or functions discussed relative to one or more control mechanisms in control mechanism 120 may be alternatively or additionally included in image sensor 130. Image sensor 130 may be a charge-coupled device (CCD) sensor, an electron multiplication CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal-oxide semiconductor (CMOS), an N-type metal-oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0039] Image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), and / or relative to... Figure 8The computing system 800 discusses one or more processors of any other type of processor 1010. The host processor 152 may be a digital signal processor (DSP) and / or other types of processor. In some specific implementations, the image processor 150 is a single integrated circuit or chip (e.g., referred to as a system-on-a-chip or SoC) that includes the host processor 152 and the ISP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G, or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth™, Global Positioning System (GPS), etc.), any combination thereof, and / or other components. I / O port 156 may include any suitable input / output port or interface according to one or more protocols or specifications, such as Inter-Integrated Circuit 2 (I2C) interface, Inter-Integrated Circuit 3 (I3C) interface, Serial Peripheral Interface (SPI) interface, Serial General Purpose Input / Output (GPIO) interface, Mobile Industrial Processor Interface (MIPI) (such as MIPI CSI-2 physical (PHY) layer port or interface), Advanced High Performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In an exemplary example, host processor 152 may use the I2C port to communicate with image sensor 130, and ISP 154 may use the MIPI port to communicate with image sensor 130.

[0040] Image processor 150 can perform multiple tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. Image processor 150 can store image frames and / or processed images in random access memory (RAM) 140 / 1025, read-only memory (ROM) 145 / 1020, cache, memory unit, another storage device, or some combination thereof.

[0041] Various input / output (I / O) devices 160 may be connected to the image processor 150. I / O devices 160 may include displays, keyboards, keypads, touchscreens, touchpads, touch-sensitive surfaces, printers, any other output devices, any other input devices, or some combination thereof. In some cases, text may be entered into the image processing device 105B via the physical keyboard or keypad of the I / O device 160, or via a virtual keyboard or keypad on the touchscreen of the I / O device 160. I / O devices 160 may include one or more ports, jacks, or other connectors that enable a wired connection between the image capture and processing system 100 and one or more peripheral devices, through which the image capture and processing system 100 may receive data from and / or send data to one or more peripheral devices. I / O devices 160 may include one or more wireless transceivers that enable a wireless connection between the image capture and processing system 100 and one or more peripheral devices, through which the image capture and processing system 100 may receive data from and / or send data to one or more peripheral devices. Peripheral devices may include any type of I / O device 160 discussed earlier, and they can be considered I / O devices 160 in themselves once they are coupled to ports, jacks, wireless transceivers or other wired and / or wireless connectors.

[0042] In some cases, the image capture and processing system 100 may be a single device. In other cases, the image capture and processing system 100 may consist of two or more independent devices, including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some embodiments, the image capture device 105A and the image processing device 105B may be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly coupled together via one or more wireless transceivers. In some embodiments, the image capture device 105A and the image processing device 105B may be disconnected from each other.

[0043] like Figure 1 As shown, the vertical dashed line will Figure 1 The image capture and processing system 100 is divided into two parts, namely image capture device 105A and image processing device 105B. Image capture device 105A includes a lens 115, a control mechanism 120, and an image sensor 130. Image processing device 105B includes an image processor 150 (including an ISP 154 and a host processor 152), RAM 140, ROM 145, and I / O devices 160. In some cases, certain components illustrated in image capture device 105A (such as ISP 154 and / or host processor 152) may be included in image capture device 105A.

[0044] Image capture and processing system 100 may include electronic devices such as mobile or landline phones (e.g., smartphones, cellular phones, etc.), desktop computers, laptop or notebook computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, Internet Protocol (IP) cameras, or any other suitable electronic devices. In some examples, image capture and processing system 100 may include one or more wireless transceivers for wireless communication (such as cellular network communication, 802.10 Wi-Fi communication, wireless local area network (WLAN) communication, or some combination thereof). In some specific implementations, image capture device 105A and image processing device 105B may be different devices. For example, image capture device 105A may include a camera device, and image processing device 105B may include a computing device, such as a mobile phone, desktop computer, or other computing device.

[0045] Although the image capture and processing system 100 is shown to include certain components, those skilled in the art will understand that the image capture and processing system 100 may include more than [other components]. Figure 1 The components shown herein are additional components. Components of the image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some embodiments, components of the image capture and processing system 100 may include electronic circuitry or other electronic hardware, and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits); and / or may include computer software, firmware, or any combination thereof, and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. Software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing the image capture and processing system 100.

[0046] In some examples, Figure 2 The extended reality (XR) system 200 may include an image capture and processing system 100, an image capture device 105A, an image processing device 105B, or a combination thereof.

[0047] Figure 2This is a diagram illustrating the architecture of an example extended reality (XR) system 200 according to some aspects of this disclosure. The XR system 200 can run (or execute) XR applications and implement XR operations. In some examples, as part of an XR experience, the XR system 200 can perform tracking and localization, mapping of the environment in the physical world (e.g., a scene), and / or localization and rendering of virtual content on a display 209 (e.g., a screen, a visible plane / area, and / or other display). For example, the XR system 200 can generate a map of the environment in the physical world (e.g., a three-dimensional (3D) map), track the pose (e.g., position and localization) of the XR system 200 relative to the environment (e.g., relative to a 3D map of the environment), localize and / or anchor virtual content at a specific location on the map of the environment, and render the virtual content on the display 209 such that the virtual content appears to be at a location in the environment corresponding to a specific location on the map of the scene, where the virtual content is localized and / or anchored. Display 209 may include glass, screen, lens, projector and / or other display mechanisms that allow users to see a real-world environment and also allow XR content to be overlaid, superimposed, blended or otherwise displayed thereon.

[0048] In this exemplary example, the XR system 200 includes one or more image sensors 202, accelerometers 204, gyroscopes 206, storage devices 207, computing components 210, an XR engine 220, an image processing engine 224, a rendering engine 226, and a communication engine 228. It should be noted that... Figure 2 The components 202-228 shown are non-limiting examples provided for illustrative and explanatory purposes, and other examples may include those with... Figure 2 The components shown may be more numerous, fewer, or different than those shown. For example, in some cases, the XR system 200 may include one or more other sensors (e.g., one or more inertial measurement units (IMUs), radio detection and ranging (RADAR), light detection and ranging (LIDAR) sensors, RADAR sensors, sound detection and ranging (SODAR) sensors, sound navigation and ranging (SONAR) sensors, audio sensors, etc.), one or more display devices, one or more other processing engines, one or more other hardware components, and / or Figure 2 One or more other software and / or hardware components not shown herein. While various components of the XR system 200 (such as image sensor 202) may be referred to in the singular herein, it should be understood that the XR system 200 may include multiple components of any component discussed herein (e.g., multiple image sensors 202).

[0049] XR system 200 includes an input device 208 or communicates (wired or wirelessly) with that input device. Input device 208 may include any suitable input device, such as a touchscreen, pen or other pointing device, keyboard, mouse buttons or keys, microphone for receiving voice commands, gesture input device for receiving gesture commands, video game controller, steering wheel, joystick, collection of buttons, trackball, remote control, any other input device 1045 discussed herein, or any combination thereof. In some cases, image sensor 202 may capture images that can be processed to interpret gesture commands.

[0050] The XR system 200 can also communicate with one or more other electronic devices (wired or wireless). For example, the communication engine 228 can be configured to manage connections and communicate with one or more electronic devices. In some cases, the communication engine 228 may correspond to... Figure 8 The communication interface is 840.

[0051] In some embodiments, one or more image sensors 202, accelerometers 204, gyroscopes 206, storage devices 207, computing components 210, XR engines 220, image processing engines 224, and rendering engines 226 may be part of the same computing device. For example, in some cases, one or more image sensors 202, accelerometers 204, gyroscopes 206, storage devices 207, computing components 210, XR engines 220, image processing engines 224, and rendering engines 226 may be integrated into HMDs, extended reality glasses, smartphones, laptops, tablets, gaming systems, and / or any other computing device. However, in some embodiments, one or more image sensors 202, accelerometers 204, gyroscopes 206, storage devices 207, computing components 210, XR engines 220, image processing engines 224, and rendering engines 226 may be part of two or more independent computing devices. For example, in some cases, some of the components in 202-226 may be part of or implemented by a computing device, and the remaining components may be part of or implemented by one or more other computing devices.

[0052] Storage device 207 can be any storage device used for storing data. Furthermore, storage device 207 can store data from any component of the XR system 200. For example, storage device 207 can store data from image sensor 202 (e.g., image or video data), data from accelerometer 204 (e.g., measurement results), data from gyroscope 206 (e.g., measurement results), data from computing component 210 (e.g., processing parameters, preferences, virtual content, rendered content, scene maps, tracking and positioning data, object detection data, privacy data, XR application data, facial recognition data, occlusion data, etc.), data from XR engine 220, data from image processing engine 224, and / or data from rendering engine 226 (e.g., output frames). In some examples, storage device 207 may include a buffer for storing frames processed by computing component 210.

[0053] One or more computing components 210 may include a central processing unit (CPU) 212, a graphics processing unit (GPU) 214, a digital signal processor (DSP) 216, an image signal processor (ISP) 218, and / or other processors (e.g., a neural processing unit (NPU) implementing one or more trained neural networks). Computing component 210 can perform various operations such as image enhancement, computer vision, graphics rendering, extended reality operations (e.g., tracking, localization, pose estimation, map building, content anchoring, content rendering, etc.), image and / or video processing, sensor processing, recognition (e.g., text recognition, face recognition, object recognition, feature recognition, tracking or pattern recognition, scene recognition, occlusion detection, etc.), trained machine learning operations, filtering, and / or any of the various operations described herein. In some examples, computing component 210 may implement (e.g., control, operate, etc.) an XR engine 220, an image processing engine 224, and a rendering engine 226. In other examples, computing component 210 may also implement one or more other processing engines.

[0054] Image sensor 202 may include any image and / or video sensor or capture device. In some examples, image sensor 202 may be part of a multi-camera assembly, such as a dual-camera assembly. Image sensor 202 may capture image and / or video content (e.g., raw image and / or video data), which may then be processed by computing component 210, XR engine 220, image processing engine 224, and / or rendering engine 226, as described herein. In some examples, image sensor 202 may include image capture and processing system 100, image capture device 105A, image processing device 105B, or combinations thereof.

[0055] In some examples, image sensor 202 may capture image data and may generate an image (also referred to as a frame) based on that image data and / or provide the image data or frame to XR engine 220, image processing engine 224, and / or rendering engine 226 for processing. The image or frame may include a video frame in a video sequence or a still image. The image or frame may include an array of pixels representing a scene. For example, the image may be: a red-green-blue (RGB) image with red, green, and blue color components per pixel; a lightness, redness, and blueness (YCbCr) image with a lightness component and two chromaticity (redness and blueness) components per pixel; or any other suitable type of color or monochrome image.

[0056] In some cases, image sensor 202 (and / or other cameras of XR system 200) may also be configured to capture depth information. For example, in some implementations, image sensor 202 (and / or other cameras) may include an RGB depth (RGB-D) camera. In some cases, XR system 200 may include one or more depth sensors (not shown) that are separate from image sensor 202 (and / or other cameras) and capable of capturing depth information. For example, such depth sensors may acquire depth information independently of image sensor 202. In some examples, depth sensors may be physically mounted in the same general location as image sensor 202, but may operate at a different frequency or frame rate than image sensor 202. In some examples, depth sensors may take the form of a light source that can project a structured or textured light pattern (which may include one or more narrowband lights) onto one or more objects in a scene. Depth information can then be obtained by utilizing the geometric deformation of the projected pattern caused by the surface shape of the objects. In one example, depth information may be obtained from a stereo sensor, such as a combination of an infrared structured light projector and an infrared camera registered to a camera (e.g., an RGB camera).

[0057] The XR system 200 may also include other sensors among its one or more sensors. The one or more sensors may include one or more accelerometers (e.g., accelerometer 204), one or more gyroscopes (e.g., gyroscope 206), and / or other sensors. The one or more sensors may provide velocity, orientation, and / or other positioning-related information to the computing component 210. For example, accelerometer 204 may detect the acceleration of the XR system 200 and may generate an acceleration measurement based on the detected acceleration. In some cases, accelerometer 204 may provide one or more translation vectors (e.g., up / down, left / right, forward / backward) that can be used to determine the positioning or pose of the XR system 200. Gyroscope 206 may detect and measure the orientation and angular velocity of the XR system 200. For example, gyroscope 206 may be used to measure the pitch, roll, and yaw of the XR system 200. In some cases, gyroscope 206 may provide one or more rotation vectors (e.g., pitch, yaw, roll). In some examples, the image sensor 202 and / or the XR engine 220 may use measurements obtained by the accelerometer 204 (e.g., one or more translation vectors) and / or measurements obtained by the gyroscope 206 (e.g., one or more rotation vectors) to calculate the pose of the XR system 200. As previously noted, in other examples, the XR system 200 may also include other sensors such as an inertial measurement unit (IMU), a magnetometer, a gaze and / or eye-tracking sensor, a machine vision sensor, a smart scene sensor, a voice recognition sensor, a shock sensor, a vibration sensor, a positioning sensor, a tilt sensor, etc.

[0058] As noted above, in some cases, one or more sensors may include at least one IMU. An IMU is an electronic device that uses a combination of one or more accelerometers, one or more gyroscopes, and / or one or more magnetometers to measure specific forces, angular velocities, and / or orientations of the XR system 200. In some examples, one or more sensors may output measured information associated with the capture of images by the image sensor 202 (and / or other cameras of the XR system 200) and / or depth information obtained using one or more depth sensors of the XR system 200.

[0059] XR engine 220 can use the output of one or more sensors (e.g., accelerometer 204, gyroscope 206, one or more IMUs and / or other sensors) to determine the pose of XR system 200 (also referred to as head pose) and / or the pose of image sensor 202 (or other cameras of XR system 200). In some cases, the pose of XR system 200 and the pose of image sensor 202 (or other cameras) can be the same. The pose of image sensor 202 refers to the positioning and orientation of image sensor 202 relative to a reference frame (e.g., about scene 110). In some specific implementations, camera pose can be determined for 6 degrees of freedom (6DoF), which refers to three translational components (e.g., which can be given by X (horizontal), Y (vertical), and Z (depth) coordinates relative to a reference frame such as the image plane) and three angular components (e.g., roll, pitch, and yaw relative to the same reference frame). In some implementations, camera pose can be determined for 3 degrees of freedom (3DoF), which refers to three angular components (e.g., roll, pitch, and yaw).

[0060] In some cases, a device tracker (not shown) may use measurements from one or more sensors and image data from image sensor 202 to track the pose (e.g., 6DoF pose) of the XR system 200. For example, the device tracker may fuse visual data from the image data (e.g., using a visual tracking solution) with inertial data from the measurements to determine the position and motion of the XR system 200 relative to the physical world (e.g., a scene) and a map of the physical world. As described below, in some examples, when tracking the pose of the XR system 200, the device tracker may generate a three-dimensional (3D) map of the scene (e.g., the real world) and / or generate updates to the 3D map for that scene. 3D map updates may include, for example, but not limited to, new or updated features and / or features or landmarks associated with the scene and / or the 3D map of that scene, positioning updates identifying or updating the position of the XR system 200 within the scene and the 3D map of that scene, etc. The 3D map provides a digital representation of the scene in the real / physical world. In some examples, 3D maps can anchor location-based objects and / or content to real-world coordinates and / or objects. The XR system 200 can use map-constructed scenes (e.g., scenes in the physical world represented by and / or associated with a 3D map) to merge the physical and virtual worlds and / or merge virtual content or objects with the physical environment.

[0061] In some aspects, computing component 210 may use a visual tracking solution to determine and / or track the pose of image sensor 202 and / or the XR system 200 as a whole, based on images captured by image sensor 202 (and / or other cameras of XR system 200). For example, in some examples, computing component 210 may perform tracking using computer vision-based tracking, model-based tracking, and / or simultaneous localization and mapping (SLAM) techniques. For example, computing component 210 may perform SLAM or may communicate (wired or wirelessly) with a SLAM system. SLAM refers to a class of techniques that simultaneously track the pose of a camera (e.g., image sensor 202) and / or XR system 200 relative to an environment while creating a map of the environment (e.g., a map of the environment modeled by XR system 200). This map may be called a SLAM map and may be three-dimensional (3D). SLAM technology can be performed using color or grayscale image data captured by image sensor 202 (and / or other cameras of XR system 200) and can be used to generate an estimate of 6DoF pose measurement of image sensor 202 and / or XR system 200. Such SLAM technology configured to perform 6DoF tracking can be referred to as 6DoF SLAM. In some cases, the output of one or more sensors (e.g., accelerometer 204, gyroscope 206, one or more IMUs and / or other sensors) can be used to estimate, correct, and / or otherwise adjust the estimated pose.

[0062] In some cases, 6DoF SLAM (e.g., 6DoF tracking) can associate features observed from certain input images from image sensor 202 (and / or other cameras) with a SLAM map. For example, 6DoF SLAM can use feature point association from an input image to determine the pose (localization and orientation) of image sensor 202 and / or XR system 200 for that input image. 6DoF map construction can also be performed to update the SLAM map. In some cases, the SLAM map maintained using 6DoF SLAM can contain 3D feature points triangulated from two or more images. For example, keyframes can be selected from an input image or video stream to represent the observed scene. For each keyframe, a corresponding 6DoF camera pose associated with the image can be determined. The pose of image sensor 202 and / or XR system 200 can be determined by projecting features from the 3D SLAM map onto the image or video frame and updating the camera pose according to a verified 2D-3D correspondence.

[0063] In an exemplary example, computational component 210 may extract feature points from some input images (e.g., each input image, a subset of input images, etc.) or from each keyframe. Feature points (also referred to as registration points) as used herein are distinctive or identifiable parts of an image, such as a part of a hand, the edge of a table, etc. Features extracted from a captured image may represent different feature points along three-dimensional space (e.g., coordinates on the X, Y, and Z axes), and each feature point may have an associated feature location. Feature points in a keyframe may match (be identical to or correspond to) feature points in previously captured input images or keyframes, or may not match. Feature detection may be used to detect feature points. Feature detection may include image processing operations that examine one or more pixels of an image to determine if a feature exists at a particular pixel. Feature detection may be used to process the entire captured image or parts of an image. For each image or keyframe, once a feature has been detected, a local image patch around that feature can be extracted. Any suitable technique can be used to extract features, such as Scale Invariant Feature Transform (SIFT) (which localizes features and generates descriptions of them), Learned Invariant Feature Transform (LIFT), Speeded Robust Features (SURF), Gradient Position Orientation Histogram (GLOH), Orientation Fast Rotation Briefing (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retina Keypoints (FREAK), KAZE, Accelerated KAZE (AKAZE), Normalized Cross-Correlation (NCC), Descriptor Matching, another suitable technique, or a combination thereof.

[0064] As an illustrative example, computing component 210 can extract feature points corresponding to a mobile device, etc. In some cases, feature points corresponding to the mobile device can be tracked to determine the pose of the mobile device. As described in more detail below, the pose of the mobile device can be used to determine the position of the projection of AR media content that can enhance the media content displayed on the display of the mobile device.

[0065] In some cases, the XR system 200 may also track the user's hands and / or fingers to allow the user to interact with and / or control virtual content in the virtual environment. For example, the XR system 200 may track the pose and / or movement of the user's hands and / or fingertips to identify or translate the user's interaction with the virtual environment. User interaction may include, but is not limited to, moving virtual content items, resizing virtual content items, selecting input interface elements in the virtual user interface (e.g., a virtual representation of a mobile phone, a virtual keyboard, and / or other virtual interfaces), and providing input through the virtual user interface.

[0066] In some cases, sensor data (such as images captured by an image capture and processing system 100, point clouds captured by a LiDAR / RADAR sensor, etc.) can be processed by neural networks and / or machine learning (ML) systems. Neural networks are examples of ML systems, and a neural network may include an input layer, one or more hidden layers, and an output layer. Data is provided from input nodes in the input layer, processed by hidden nodes in one or more hidden layers, and output is produced by output nodes in the output layer. Deep learning networks typically include multiple hidden layers. Each layer of a neural network may include a feature map or activation map, which may include artificial neurons (or nodes). Feature maps may include filters, kernels, etc. Nodes may include one or more weights used to indicate the importance of nodes in one or more layers. In some cases, a deep learning network may have a series of many hidden layers, where earlier layers are used to determine simple and low-level properties of the input, while later layers build a hierarchy of more complex and abstract properties.

[0067] Deep learning architectures can learn hierarchical structures of features. For example, if presented with visual data, the first layer can learn to recognize relatively simple features in the input stream, such as edges. In another example, if presented with auditory data, the first layer can learn to recognize spectral power at specific frequencies. The second layer, taking the output of the first layer as input, can learn to recognize combinations of features, such as simple shapes in visual data or combinations of sounds in auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Even higher layers can learn to recognize common visual objects or spoken phrases.

[0068] Deep learning architectures perform particularly well when applied to problems with a natural hierarchical structure. For example, the classification of motorized vehicles can benefit from first learning to identify features such as wheels, windshields, and others. These features can then be combined in different ways at higher levels to identify cars, trucks, and airplanes.

[0069] Neural networks can be designed to have multiple connectivity patterns. In feedforward networks, information is passed from lower layers to higher layers, where each neuron in a given layer communicates with neurons in higher layers. As described above, hierarchical representations can be built in successive layers of a feedforward network. Neural networks can also have recurrent or feedback (also known as top-down) connections. In recurrent connections, the output from a neuron in a given layer can be communicated to another neuron in the same layer. Recurrent architectures can help identify patterns across more than one block of input data that is sequentially delivered to the neural network. Connections from neurons in a given layer to neurons in lower layers are called feedback (or top-down) connections. Networks with many feedback connections can be helpful when the recognition of higher-level concepts can aid in discerning specific lower-level features of the input. Connections between layers of a neural network can be fully connected or locally connected. Further details are provided below regarding... Figures 3A to 4 Various examples describing neural network architectures.

[0070] Neural networks can be designed to have multiple connectivity patterns. In feedforward networks, information is passed from lower layers to higher layers, where each neuron in a given layer communicates with neurons in higher layers. As described above, hierarchical representations can be built in successive layers of a feedforward network. Neural networks can also have recurrent or feedback (also known as top-down) connections. In recurrent connections, the output from a neuron in a given layer can be passed to another neuron in the same layer. Recurrent architectures can help identify patterns across more than one block of input data that is sequentially delivered to the neural network. Connections from neurons in a given layer to neurons in lower layers are called feedback (or top-down) connections. Networks with many feedback connections can be helpful when the recognition of higher-level concepts can aid in discerning specific lower-level features of the input.

[0071] The connections between layers in a neural network can be fully connected or locally connected. Figure 3A An example of a fully connected neural network 302 is illustrated. In the fully connected neural network 302, neurons in the first layer can transmit their outputs to each neuron in the second layer, such that each neuron in the second layer will receive inputs from each neuron in the first layer. Figure 3BAn example of a locally connected neural network 304 is illustrated. In the locally connected neural network 304, neurons in a first layer can connect to a limited number of neurons in a second layer. More generally, the locally connected layers of the locally connected neural network 304 can be configured such that each neuron in the layer will have the same or similar connectivity pattern, but the connection strength can have different values ​​(e.g., 310, 312, 314, and 316). The connectivity patterns of locally connected networks may produce spatially different receptive fields in higher layers because higher-layer neurons in a given region can receive inputs that are trained to the properties of a restricted portion of the network's total input.

[0072] An example of a locally connected neural network is a convolutional neural network. Figure 3C An example of a convolutional neural network 306 is illustrated. The convolutional neural network 306 can be configured such that the connection strength associated with the input of each neuron in the second layer is shared (e.g., 308). Convolutional neural networks may be well-suited for problems where the spatial location of the input is meaningful. According to various aspects of this disclosure, the convolutional neural network 306 can be used to perform one or more aspects of video compression and / or decompression.

[0073] One type of convolutional neural network is the deep convolutional network (DCN). Figure 3D A detailed example of the DCN 300 is illustrated. The DCN is designed to receive signals from an image capture device 330 (such as...). Figure 1 The image capture and processing system 100 identifies visual features from the input image 326. The DCN 300 in the current example can be trained to identify traffic signs and the numbers provided on them. Of course, the DCN 300 can be trained for other tasks, such as identifying lane markings or traffic lights.

[0074] Supervised learning can be used to train the DCN 300. During training, an image 326, such as a speed limit sign, can be presented to the DCN 300, and the forward pass can then be computed to produce an output 322. The DCN 300 may include a feature extraction segment and a classification segment. Upon receiving the image 326, a convolutional layer 332 may apply a convolutional kernel (not shown) to the image 326 to generate a first set of feature maps 318. As an example, the convolutional kernel of the convolutional layer 332 may be a 5x5 kernel that generates a 28x28 feature map. In this example, because four different feature maps are generated in the first set of feature maps 318, four different convolutional kernels are applied to the image 326 at the convolutional layer 332. The convolutional kernel may also be referred to as a filter or a convolutional filter.

[0075] The first set of feature maps 318 can be downsampled by a max-pooling layer (not shown) to generate a second set of feature maps 320. The max-pooling layer reduces the size of the first set of feature maps 318. That is, the size of the second set of feature maps 320 (e.g., 14x14) is smaller than the size of the first set of feature maps 318 (e.g., 28x28). The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 320 can be further convolved via one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

[0076] exist Figure 3D In the example, the second set of feature maps 320 is convolved to generate a first feature vector 324. Furthermore, the first feature vector 324 is further convolved to generate a second feature vector 328. Each feature of the second feature vector 328 may include a number corresponding to a possible feature of the image 326, such as "sign", "60", and "100". A softmax function (not shown) converts the numbers in the second feature vector 328 into probabilities. Therefore, the output 322 of the DCN 300 is the probability that the image 326 includes one or more features.

[0077] In this example, the probabilities for "sign" and "60" in output 322 are higher than the probabilities for other numbers in output 322 (such as "30", "40", "50", "70", "80", "90", and "100"). Before training, output 322 generated by DCN 300 may be incorrect. Therefore, the error between output 322 and the target output can be calculated. The target output is the baseline ground truth (e.g., "sign" and "60") of image 326. The weights of DCN 300 can then be adjusted so that output 322 of DCN 300 is more closely aligned with the target output.

[0078] To adjust the weights, the learning algorithm computes the gradient vector of the weights. The gradient indicates how much the error will increase or decrease as the weights are adjusted. At the top layers, the gradient corresponds directly to the values ​​of the weights connecting the activated neurons in the penultimate layer to the neurons in the output layer. In lower layers, the gradient depends on the values ​​of the weights and the error gradient computed in the higher layers. The weights can then be adjusted to reduce the error. This method of adjusting weights is called "backpropagation" because it involves "passing backward" through the neural network.

[0079] In practice, the error gradient of the weights can be calculated using a small number of examples to make the calculated gradient approximate the true error gradient. This approximation method is called stochastic gradient descent. Stochastic gradient descent can be repeated until the achievable error rate of the entire system stops decreasing or until the error rate reaches a target level. After learning, a new image can be presented to the DCN, and the forward pass of the network can produce an output that can be considered an inference or prediction of the DCN.

[0080] Deep convolutional networks (DCNs) are convolutional networks configured with additional pooling and normalization layers. DCNs have achieved state-of-the-art performance on many tasks. DCNs can be trained using supervised learning, where both the input and output targets are known for many paradigms and are used to modify the network's weights using gradient descent.

[0081] DCNs can be feedforward networks. Furthermore, as described above, connections from neurons in the first layer of a DCN to a set of neurons in the next higher layer are shared across neurons in the first layer. The feedforward and shared connections of a DCN can be used for fast processing. For example, the computational cost of a DCN may be much smaller than that of a similarly sized neural network that includes recurrent or feedback connections.

[0082] The processing at each layer of a convolutional network can be considered as a spatially invariant template or base projection. If the input is first decomposed into multiple channels, such as the red, green, and blue channels of a color image, then a convolutional network trained on that input can be considered three-dimensional, with two spatial dimensions along the image's axes and a third dimension capturing color information. The outputs of the convolutional connections can be considered as forming feature maps in subsequent layers, where each element in the feature map (e.g., feature map 320) receives input from a range of neurons in the previous layer (e.g., feature map 318) and input from each of the multiple channels. The values ​​in the feature maps can be further processed non-linearly (e.g., modifiers, max(0,x)). Values ​​from neighboring neurons can be further pooled, which corresponds to downsampling and provides additional local invariance and dimensionality reduction.

[0083] Figure 4 This is a block diagram illustrating an example of a deep convolutional network 450. Based on connectivity and weight sharing, the deep convolutional network 450 can include multiple layers of different types. For example... Figure 4As shown, the deep convolutional network 450 includes convolutional blocks 454A and 454B. Each convolutional block in 454A and 454B can be configured with a convolutional layer (CONV) 456, a normalization layer (LNorm) 458, and a max pooling layer (MAX POOL) 460. It is worth noting that the layers illustrated with respect to convolutional blocks 454A and 454B are examples of layers that can be included in convolutional layers and are not intended to be limiting; other types of layers can be included in any order.

[0084] Convolutional layer 456 may include one or more convolutional filters that can be applied to input data 452 to generate feature maps. Although only two convolutional blocks 454A and 454B are shown, this disclosure is not limited thereto, and any number of convolutional blocks (e.g., convolutional blocks 454A and 454B) may be included in the deep convolutional network 450 according to design preferences. Normalization layer 458 may normalize the output of the convolutional filters. For example, normalization layer 458 may provide whitening or lateral suppression. Max pooling layer 460 may provide spatial downsampling aggregation to achieve local invariance and dimensionality reduction.

[0085] For example, the parallel filter bank of deep convolutional networks can be loaded onto processors such as CPUs or GPUs, or regarding Figure 8 The computing system 800 discusses any other type of processor 810 to achieve high performance and low power consumption. Alternatively, parallel filter banks can be loaded onto... Figure 8 The computing system can be accessed on an 800 DSP or ISP. Furthermore, a deep convolutional network 450 is available and can exist on such systems. Figure 8 Other processing blocks on the computing system 800 include sensor processors and navigation modules, which are dedicated to sensors and navigation, respectively.

[0086] The deep convolutional network 450 may also include one or more fully connected layers, such as layer 462A (labeled "FC1") and layer 462B (labeled "FC2"). The deep convolutional network 450 may also include a logistic regression (LR) layer 464. Weights (not shown) to be updated between each of the layers 456, 458, 460, 462A, 462B, and 464 of the deep convolutional network 450. The output of each of these layers (e.g., 456, 458, 460, 462A, 462B, and 464) can serve as the input to the next layer in the deep convolutional network 450 (e.g., 456, 458, 460, 462A, 462B, and 464) to learn hierarchical feature representations from the input data 452 (e.g., images, audio, video, sensor data, and / or other input data) provided at the initial convolutional block 454A. The output of the deep convolutional network 450 is a classification score 466 of the input data 452. The classification score 466 can be a set of probabilities, where each probability is a probability of the input data (including features from a set of features).

[0087] In some cases, one or more convolutional networks (such as DCNs) can be incorporated into a more complex ML network. As an example, as indicated above, a deep convolutional network 450 can output the probability that input data (such as an image) includes certain features. The deep convolutional network 450 can then be modified to extract (e.g., output) certain features. Additionally, DCNs can be added to extract other features. This set of DCNs can be used as a feature extractor to identify features in an image. In some cases, feature extractors can be used as the backbone of additional ML network components to perform additional operations, such as localization, image segmentation, object detection, etc.

[0088] As indicated above, SLAM can be used to determine the position and / or pose of a device. In some cases, conventional SLAM systems operate on perspective images, potentially making them sensitive to the placement of the imaging sensor and to occlusion, which may degrade performance. Additionally, conventional SLAM systems can be camera-based, and this reliance on a specific sensor can be limiting.

[0089] To enhance SLAM, it can be useful to allow SLAM systems to utilize graphical representations of meta-features from a bird's-eye view (BEV) (e.g., top-down view) multimodal feature map of the environment. Multimodal features can be generated based on data from a variety of different types of sensors, such as image sensors and at least one other type of sensor, such as LiDAR, RADAR, SODAR, SONAR, etc. Using different sensor types helps provide a more comprehensive understanding of the environment, increases robustness to faults and / or noise from individual sensor modalities, and can help overcome occlusion. In some cases, the sensor type can be based on how the sensor senses the environment. For example, two sensors sensing different portions of the electromagnetic spectrum can have different sensor types. Similarly, a sensor sensing the reflection / refraction of projected light can have a different sensor type than another sensor sensing naturally reflected / refracted light. Multimodal features can be transformed into BEV features to help provide view-invariant representations that encode semantic information about the environment. Additionally, BEV features can be normalized based on sensor configuration to help achieve the generalization ability of multimodal BEV features across systems with different sensors. Meta-features can refer to features of features (e.g., features such as those generated from sensor data). For example, as described in detail below, BEV multimodal features can be used to generate graphs, and meta-features can be features of the graphs.

[0090] In some cases, sensor data from multiple sensors can be fused to generate a multimodal BEV feature map. Figure 5 A technique 500 for generating multimodal BEV feature maps according to various aspects of this disclosure is illustrated. In technique 500, input camera data 502 (e.g., images captured by a camera) may be input to a camera data encoder 504. The camera data encoder 504 may include a feature extractor. These feature extractors may be ML-based and used to identify certain features in the camera data. As an example, the feature extractor may include one or more layers or transform blocks, which may include feature maps for identifying certain features. The camera data encoder 504 may output the identified features as intermediate camera features 506. Notably, the input camera data 502 and the camera data encoder 504 may operate in 2D space (e.g., relative to the camera on the height and width axes). A perspective transformation 508 may be applied to the output intermediate camera features, which transforms the intermediate camera features from, for example, a front view of the environment from the vehicle to camera features projected onto the BEV, as if the features were generated based on a camera positioned above the vehicle. In some cases, the perspective transformation 508 may be ML-based.

[0091] In some cases, LIDAR data 510 may be received, for example, as a LIDAR point cloud captured by a LIDAR. The LiDAR can emit a beam of ultraviolet, visible, or near-infrared light into the environment and detect reflections of that beam from objects in the environment. Based on the amount of time required to detect the reflections, the distance to objects in the environment can be determined, and the LIDAR point can be described based on its position relative to the LIDAR on the width, height, and depth axes. Therefore, the LIDAR data is three-dimensional data. LIDAR data 510 may be input to a LIDAR data encoder 512. The LIDAR data encoder 512 may be similar to a camera data encoder 504, but configured (e.g., trained) to operate in 3D space to identify features in the LIDAR data and output the identified features as intermediate LIDAR features 514. Then, for example, by removing height information (e.g., height axis, height channel, height dimension) or averaging that height information, the intermediate LIDAR features 514 may be flattened 516 into BEV-projected LIDAR features.

[0092] In some cases, the input RADAR data 518 can be received, for example, as a RADAR point cloud captured by a RADAR. In other cases, the RADAR operates in a manner similar to LiDAR, but uses radio frequency waves instead of light. The input RADAR data 518 can be fed into a RADAR data encoder 520. The RADAR data encoder 520 can be similar to a LiDAR data encoder 512, and the RADAR data encoder 520 can identify features in the RADAR data and output the identified features as intermediate RADAR features 522. Then, for example, by removing height information (e.g., height axis, height channel, height dimension) or averaging the height information, the intermediate RADAR features 522 can be flattened 524 into RADAR features projected by the BEV.

[0093] The camera features, LiDAR features, and RADAR features projected from the BEV can be combined into a set of multimodal BEV features (Figure 526), ​​for example, by combining the camera features, LiDAR features, and RADAR features projected from the BEV. In some cases, the features projected from the BEV can be combined by cascading the camera features, LiDAR features, and RADAR features projected from the BEV.

[0094] Figure 6 This is a block diagram illustrating a technique 600 for multimodal localization using BEV feature maps according to various aspects of this disclosure. In some cases, technique 600 can operate in a multimodal BEV feature space. Figure 6In this process, the multimodal BEV feature map 602 can be received and input into the feature normalization engine 604. The multimodal BEV feature map 602 can be... Figure 6 Multimodal BEV feature map 526.

[0095] The feature normalization engine 604 can normalize the features of the multimodal BEV feature map 602 based on the sensor configuration of the sensors used to obtain the features of the multimodal BEV feature map 602. For example, the multimodal BEV feature map 602 may include an indication of which sensor (e.g., sensor type) is used to obtain which information, and sensor configuration information (e.g., sensor configuration information 606).

[0096] In some cases, multimodal BEV feature maps can be normalized based on sensor calibration data. As an example, image sensor configuration information may include calibration data such as camera field of view (FOV), principal point, and lens distortion information. In some cases, camera features can be normalized, for example, by dividing pixel intensity by the camera FOV to normalize for the field of view, subtracting the principal point offset to normalize for the camera position, or applying image distortion correction transformation to normalize for lens distortion. For example, when the image sensor configuration includes FOV, mounting height (h), tilt angle α, etc., the normalization function ( () can be defined as combining configuration information and feature maps X As a parameter, making Normalization can be performed by dividing the eigenvalues ​​by the field of view (FOV) to normalize over the field of view, and subtracting from the height values... h To normalize for the installation height, and to apply a rotational transformation based on the tilt angle (e.g., where...). R( ) (representing the rotation transformation matrix), can be expressed as: Where i and j are indexed by spatial dimensions, and c is indexed by feature channels. The resulting normalized image feature map... It can keep sensor parameters constant and can be used to generate feature maps. As another example, the configuration information includes principal point offset (…). pp ), camera intrinsic matrix ( K ) and horizontal field of view ( FOVc In the case of camera position, lens distortion, and camera characteristics, the field of view is normalized. It can be represented as ,in This indicates element-wise operations. Normalized output. It can be aligned to the common BEV coordinate system used for BEV feature fusion.

[0097] As another example, configuration information for a LiDAR sensor may include calibration data such as the sensor's mounting height, tilt angle, and field of view (FOV). In some cases, LiDAR features can be normalized, for example, by subtracting the LiDAR mounting height from the point cloud Z-values, applying a transformation based on the LiDAR tilt angle to normalize for orientation, or dividing the X and Y values ​​by the LiDAR horizontal FOV to normalize for distance. As an example, LiDAR features are normalized for orientation, mounting height, and distance. Xl The normalization performed can be expressed as: ,in Indicates the LIDAR tilt angle. This represents a rotation matrix based on the tilt angle. hl Indicates the LIDAR installation height, and FOVl This represents the horizontal FOV of the LiDAR, where This indicates element-wise operations. Normalized output. It can be aligned to the common BEV coordinate system used for BEV feature fusion.

[0098] In another example, the configuration of the RADAR sensor may include calibration data such as the RADAR sensor's mounting height, tilt angle, distance, etc. In some cases, RADAR features can be normalized, for example, by subtracting the RADAR mounting height from the RADAR return elevation, applying a rotational transformation based on the RADAR tilt angle, or normalizing the distance value by dividing the distance value by the RADAR sensor's maximum distance. As an example, the orientation, mounting location, and distance affect the RADAR features (…). Xr The normalization performed can be expressed as: ,in Indicates the RADAR tilt angle. Indicates the maximum detection distance. hr Indicates the RADAR installation height, and Represents a rotation matrix based on the tilt angle, where This indicates element-wise operations. Normalized output. It can be aligned to the common BEV coordinate system used for BEV feature fusion.

[0099] In some cases, the feature normalization engine 604 can use adaptive normalization. Adaptive (e.g., dynamic) normalization can, for example, dynamically update (e.g., optimize) the normalization parameters based on sensor calibration and / or the environment. In some cases, the normalization parameters... It can be based on dynamic sensor calibration (e.g., where the camera or other sensors are calibrated over time, so that the calibration information is optimized over time) and / or environmental estimation that adapts over time, making Let represent the normalized parameter at time t, and This represents an update to the normalized parameters for each step. For example, environmental estimates of the distance to an object can be used to dynamically adapt the camera's field of view (FOV) for normalization. For adaptive camera FOV normalization, the FOV parameters... αfov Can be updated to In some cases, Based on the estimated distance to the object ( ), making ,in This represents the learning rate (e.g., the frequency at which the normalization parameter is adapted), and This represents the average distance to the object being focused on or detected. As the object gets closer... αfov It can be increased (e.g., by "zooming in"), and as the object moves away, αfov It can be reduced. Calibration loss. Can be used for Provide monitoring signals, so that ,in f Represents the normalization function. This represents the input at time t. This represents the parameter at time t. This indicates the parameter update at time step t, and This represents L² normalization. In some cases, minimizing... Train a model that adapts to normalized parameters to meet calibration constraints. In some cases, online calibration processes, such as target detection and sensor motion analysis, can be optimized. To ensure robust compatibility.

[0100] In some cases, the feature normalization engine 604 can be modularly designed with sub-modules to apply certain normalization transformations based on which sensor is used to obtain specific features, and can perform some operations for multiple types of sensors. In some cases, the modules can be linked together serially or in parallel.

[0101] In some cases, additional normalization techniques can be used. Examples of other normalization techniques include learnable normalization, cross-modal calibration, domain randomization, etc. For learnable normalization, neural networks or other machine learning models can be used to learn optimal normalization parameters from data in a sensor-independent manner (e.g., to determine sensor / camera configuration information) instead of using manually designed transformations. Encoder-decoder networks can be trained to transform raw sensor data into the normalized BEV space. In some cases, more complex sensor-specific distortions beyond simple transformations can be learned. For cross-modal calibration, complementary modalities can be used to co-calibrate with each other. For example, LIDAR points (e.g., data obtained from sensors such as LIDAR) can be used to fine-tune camera normalization parameters. This enables online adaptation and allows for redundancy across sensors. For domain randomization, normalization networks can be trained using, for example, synthetic data augmentation and domain randomization of sensor parameters. In some cases, multiple normalization techniques can be used. For example, calibration-based normalization as described above for image, RADAR, and LIDAR sensors can be followed by cross-modal calibration and domain randomization.

[0102] Return to Figure 6 The normalized BEV feature map can be passed to the graph conversion engine 608. The normalized BEV feature map is invariant to sensor parameters and is sensor-independent. The graph conversion engine 608 can convert the normalized BEV feature map into a graph-based representation, such as BEV feature maps 610A…610T (collectively referred to as BEV feature maps 610). In some cases, the graph conversion engine 608 can receive the normalized BEV feature map at a specific step rate and can generate BEV feature maps 610A, 610B to 610T based on the step rate.

[0103] In graph-based representations, a graph represents the collection of relationships (e.g., edges) between entities (e.g., nodes). For example, feature points in a BEV feature map can be discretized into cells, where each cell becomes a node in the graph, and features can be aggregated from feature points within the nodes. To discretize a normalized BEV feature map into cells, the normalized BEV feature map can be divided into cells using a function. func () is used to aggregate feature points within each cell, and grid cells can be connected to form a graph representation.

[0104] In some cases, the normalized BEV feature map from the feature normalization engine 604 can be divided into a grid, where each grid cell corresponds to a node in the feature map being generated, and the feature corresponding to that node can be an aggregation of all pointwise features (e.g., features within the cell) within the grid cell. For example, for a normalized BEV feature map X of size W x H x C, the BEG feature map X can be discretized into an N x M grid, where N = W / S and M = H / S, with a stride of S. For a total of N... There are M grid cells, each of which can be represented as... ,in i From 1 to N, and j From 1 to M. For each grid cell cause it to fall into that grid cell A set of k feature points within is Then each point p It can have its associated feature vector, such that Feature points can be defined using functions. func () are aggregated into a single feature vector of that unit node within the unit, such that To obtain a feature map of size N x M x C In some cases, the function func () can be a mean / average pooling function, a max pooling function, a learning aggregator such as a transformer, or other aggregation functions.

[0105] Then a graph can be constructed. G .picture G It can be represented as G = ( V , E ), where nodes V = N represents M grid cells. Node features can then be aggregated feature vectors, such that... .side E Adjacent grid cells can be connected based on connectivity. In some cases, graph-based representations of BEV maps can enable more efficient localization by matching subgraphs with the global map graph.

[0106] In some cases, the BEV feature map 610 within a time window can be passed to the query graph engine 612. The query graph engine 612 can generate a query graph 614 by combining the BEV feature maps 610 within the time window. For example, Gq = Vq , EqThis can be represented as a query graph 614 formed by combining BEV feature maps 610{g1, g2, … gt} from a time window t. Each node of the query graph 614... Having feature vectors xi Query graph 614 can be passed to query graph feature engine 616. Query graph feature engine 616 can generate feature descriptors (e.g., meta-features) for the query graph. That is, BEV feature graph 610 can be a representation of graph-based features detected from sensor data, and query graph feature engine 616 can identify and describe features (e.g., meta-features) of the graph-based feature representation in query graph feature descriptor 618. In some cases, the query graph feature engine can be based on a graph neural network (GNN). In some cases, the GNN can perform message passing to update node features (e.g., descriptors), which can be represented as... ,in It is a neural network (e.g., Generative Neural Network) that updates node features based on node features and aggregated neighbor features. In some cases, GNNs may include multi-layer message passing, which can be represented as... , and among them l Index each layer.

[0107] In some cases, each layer further enriches node features by aggregating from a larger neighborhood. Ultimately zi l It can provide a globally context-aware representation for each node. In some cases, this is different from using the original input features. xi In contrast, multi-layer GNNs enable the learning of more discriminative features. In some cases, the query graph feature descriptor 618 can be passed to the query graph comparator engine 620.

[0108] The query graph comparator engine 620 can also receive a scene graph feature descriptor 622. The scene graph feature descriptor 622 can be generated based on the scene graph 624. The scene graph can be represented as... Gs = ( Vs , Es ).

[0109] Scene graph 624 can be a map of features in an environment represented in graph form, and scene graph 624 can be constructed by observing features in the environment multiple times. For example, a query graph can describe features of a portion of the environment described by scene graph 624. In some cases, scene graph 624 can be constructed using observations of features from multiple devices and can be remotely stored and accessed via a network connection (e.g., via a server). Scene graph 624 can be passed to scene graph feature engine 626, which identifies and describes the features of the graph (e.g., meta-features) to generate scene graph feature descriptors 622. Scene graph feature engine 626 can be based on GNN and can operate in a manner similar to query graph feature engine 616. For example, Gs =( Vs , Es Scene graph 624 can be represented by a node. Each node in scene graph 624... Having feature vectors xj Scene graph 624 can be passed to scene graph feature engine 626, which can perform message passing to update node features (e.g., descriptors), which can be represented as... ,in This is a neural network (e.g., a GNN). In some cases, a GNN may include multiple layers of message passing, which can be represented as... , and among them l Index each layer.

[0110] The query graph comparator engine 620 compares the query graph feature descriptor 618 with the scene graph feature descriptor 622 to locate the closest match 628 between a portion of the query graph and the scene graph, thereby localizing features captured by sensors within the environment by finding correspondences between nodes based on learned feature similarity. For example... Gq and Gs The closest matching between them can be represented as ,in It can be a similarity function, such as cosine similarity. In some cases, It can be targeted Gq Each node in In Gs The matched nodes. This allows for the identification of correspondences between nodes based on learned feature similarity. Gq Positioning at Gs Within. GNNs can provide a learnable way to extract feature representations tailored for matching. Based on the location of the features, the location of the device performing technique 600 within the environment can be determined.

[0111] In some cases, information from query graph 614 can be used to update scene graph 624. For example, the environment may have changed since scene graph 624 was generated, and query graph 614 can include observations of such changes. These observations can be used to incrementally aggregate such new information into scene graph 624. As an example, when using a localized query graph... Gq Update the scene images Gs In this case, the matched node (e.g., from localization) can be represented as ,in and These are matched node pairs. For updating... Gs For each matching pair The features of nodes can be aggregated to make ,in These can be aggregation functions such as mean, maximum, and minimum values. Once aggregated, the characteristics of the aggregation can be used to update the nodes. , making Connected to The edges can be updated to make ,in It can refer to the edge Come on Connected to A set of adjacent nodes, among which yes ,in It can be and The connected edges, where It is a node eigenvectors, where yes The updated aggregated features, and where It is to use node features and The updated edge features are used as input and output. The function. Therefore, for Each adjacent New node features can be used ( of and of To update the edge From Aggregated information can be propagated to the local neighborhood of the graph, and this is achieved through feature aggregation and edge updates from the current observation. Gq Incremental aggregation of information Gs In the middle. Over time, this allows for the construction of... Gs The updated environment map.

[0112] Figure 7 This is a flowchart illustrating a process 700 for map construction according to various aspects of this disclosure. Process 700 may be performed by a computing device (or apparatus) or components of a computing device (e.g., chipset, codec, etc.) (such as...). Figure 1 Image capture and processing system 100 Figure 2 XR system 200, Figure 8 The computing device can be a mobile device (e.g., a mobile phone), a networked wearable device (such as a watch), an extended reality (XR) device (such as a virtual reality (VR) device or an augmented reality (AR) device), a vehicle or a component or system of a vehicle, or other types of computing devices. The operation of process 700 can be implemented on one or more processors (e.g., a computing system 800, etc.). Figure 1 Image processor 150 Figure 1 The host processor 152, Figure 2 Computing component 210, Figure 8 Software components that execute and run on the processor 810 and / or other processors. In some cases, the operation of process 700 may be performed by a software component with a processor 810 and / or other processors. Figure 8 The computing system is implemented using the architecture of the 800 system.

[0113] At box 702, the computing device (or a component thereof) can receive information from the environment via a camera (e.g., Figure 1 Image capture device 105A, Figure 2 Image sensor 202, image capture device 330 (Figure 3) Figure 8 One or more images captured by an input device 845, etc. Figure 5 The first set of image features (e.g., input camera data 502) is obtained from the input camera data. Figure 5 (Intermediate camera feature 506). In some cases, computing devices (or components thereof) may include cameras and sensors.

[0114] At box 704, the computing device (or a component thereof) may be transformable (e.g., Figure 5 Perspective transformation 508) The first set of image features is used to generate the first set of bird's-eye view (BEV) image features.

[0115] At box 706, the computing device (or a component thereof) can obtain a second set of features (e.g., Figure 5 Intermediate LiDAR features 514 Figure 5 The first set of features (e.g., intermediate RADAR features 522) is generated based on representations of the environment obtained using sensors with different sensor types than the camera. In some cases, the sensors include one of a LiDAR sensor, a radar sensor, or a sonar sensor.

[0116] At box 708, the computing device (or a component thereof) can be transformed (e.g., Figure 5 516, Figure 5 The second set of features (524, etc.) is used to generate the second set of BEV features.

[0117] At box 710, the computing device (or a component thereof) can, based on camera configuration information associated with one or more images, (e.g., via...) Figure 6 The feature normalization engine 604 normalizes the first set of BEV image features. In some cases, the camera configuration information includes camera calibration information, and the sensor configuration information includes sensor calibration information. In some examples, the camera calibration information includes information associated with at least one of the camera's field of view (FOV), the camera's principal point, and lens distortion information. As an example, camera features may be normalized, for example, by dividing pixel intensity by the camera FOV to normalize for the field of view, subtracting the principal point offset to normalize for the camera position, applying an image distortion correction transform to normalize for lens distortion, etc. In some cases, at least one of the camera calibration information or the sensor calibration information is optimized over time. In some examples, the computing device (or a component thereof) may adapt the camera configuration information or the sensor configuration information based on an estimate of the environment.

[0118] At box 712, the computing device (or a component thereof) may normalize the second set of BEV features based on sensor configuration information of the sensor. In some cases, the sensor calibration information includes information associated with at least one of the sensor's mounting height, tilt angle, FOV, and distance. As an example, the configuration information of a LiDAR sensor may include calibration data (information) such as the LiDAR sensor's mounting height, tilt angle, FOV, etc. In some examples, at least one of the camera configuration information or sensor configuration information is determined by a machine learning model. In some cases, data obtained from the sensor is used by a machine learning model to determine the camera configuration information. In some examples, the normalized first set of BEV image features and the normalized second set of BEV features include a normalized BEV feature map, and at least one processor is configured to: divide the normalized BEV feature map into a cell grid; and generate a BEV feature map based on the divided normalized BEV feature map, wherein features of the normalized BEV feature map in the cells of the cell grid are aggregated into nodes of the BEV feature map.

[0119] At box 714, the computing device (or a component thereof) may generate a query graph based on the normalized first set of BEV image features and the normalized second set of BEV features (e.g., Figure 6(Query graph 614). In some cases, the computing device (or its components) can generate a query graph by aggregating a BEV feature graph with another BEV feature graph within a time window. In some examples, the computing device (or its components) can generate a query graph feature descriptor based on the features of the query graph (e.g., Figure 6 The query graph feature descriptor 618). In some cases, the query graph feature descriptor is generated by a graph neural network. In some examples, the computing device (or a component thereof) can combine the query graph feature descriptor with the scene graph feature descriptor (e.g., Figure 6 The scene graph feature descriptor 622 is compared to identify the scene graph (e.g., Figure 6 The scene graph (624) is part of the query graph that is matched.

[0120] In some examples, the techniques or processes described herein may be performed by a computing device, apparatus, and / or any other computing device. In some cases, the computing device or apparatus may include a processor, microprocessor, microcomputer, or other components of a device configured to perform the steps of the processes described herein. In some examples, the computing device or apparatus may include a camera configured to capture video data (e.g., video sequences) including video frames. For example, the computing device may include a camera device that may or may not include a video codec. As another example, the computing device may include a mobile device with a camera (e.g., camera devices such as digital cameras, IP cameras, etc., mobile phones or tablet computers including cameras, or other types of devices with cameras). In some cases, the computing device may include a display for displaying images. In some examples, the camera or other capturing device that captures video data is separate from the computing device, in which case the computing device receives the captured video data. The computing device may also include a network interface, transceiver, and / or transmitter configured to communicate video data. The network interface, transceiver, and / or transmitter may be configured to communicate Internet Protocol (IP) based data or other network data.

[0121] The processes described herein can be implemented in hardware, computer instructions, or any combination thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a particular function or implement a particular data type. The order in which the operations are described is not intended to be construed as limiting, and any number of the described operations can be combined in any order and / or in parallel to implement the process.

[0122] In some cases, a device or apparatus configured to perform operation of process 700 and / or other processes described herein may include a processor, microprocessor, microcomputer, or other component of the device configured to perform steps of process 700 and / or other processes. In some examples, such a device or apparatus may include one or more sensors configured to capture image data and / or other sensor measurements. In some examples, such a computing device or apparatus may include one or more sensors and / or cameras configured to capture one or more images or videos. In some cases, such a device or apparatus may include a display for displaying images. In some examples, one or more sensors and / or cameras are decoupled from the device or apparatus, in which case the device or apparatus receives the sensed data. Such a device or apparatus may also include a network interface configured to transmit data.

[0123] Components of a device or apparatus configured to perform one or more operations of process 700 and / or other processes described herein may be implemented in a circuit. For example, components may include electronic circuitry or other electronic hardware, and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, graphics processing unit (GPU), digital signal processor (DSP), central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include computer software, firmware, or any combination thereof for performing the various operations described herein, and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein. The computing device may also include a display (as an example of an output device or as a supplement to an output device), a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0124] Process 700 is illustrated as a logic flowchart, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or combinations thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that performs the described operation when executed by one or more processors. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific data type. The order in which the operations are described is not intended to be construed as limiting, and any number of described operations can be combined in any order and / or in parallel to implement the process.

[0125] Additionally, the processes described herein (e.g., process 700 and / or other processes) may be executed under the control of one or more computer systems configured using executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes jointly on one or more processors, by hardware, or a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0126] Additionally, the processes described herein can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes jointly on one or more processors, implemented in hardware, or a combination of the foregoing. As noted above, the code can be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium can be non-transitory.

[0127] Figure 8 This is a diagram illustrating an example of a system used to implement certain aspects of this technology. Specifically, Figure 8 An example of a computing system 800 is illustrated. This computing system can be any computing device, such as constituting an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using a connection 805. The connection 805 can be a physical connection using a bus, or a direct connection to a processor 810, such as in a chipset architecture. The connection 805 can also be a virtual connection, a networking connection, or a logical connection.

[0128] In some examples, the computing system 800 is a distributed system, wherein the functions described in this disclosure may be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some examples, one or more of the described system components represent a plurality of such components, each performing some or all of the functions described for which it is used. In some cases, these components may be physical or virtual devices.

[0129] Example computing system 800 includes at least one processing unit (CPU or processor) 810 and a connection 805 that couples various system components, including system memories 815 such as read-only memory (ROM) 820 and random access memory (RAM) 825, to processor 810. Computing system 800 may include a cache 812 of high-speed memory that is directly connected to, closely proximate to, or integrated into processor 810.

[0130] Processor 810 may include any general-purpose processor and hardware or software services (such as services 832, 834, and 836 stored in storage device 830 and configured to control processor 810), as well as dedicated processors in which software instructions are incorporated into the actual processor design. Processor 810 may be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0131] To enable user interaction, the computing system 800 includes an input device 845 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, a camera, an accelerometer, a gyroscope, etc. The computing system 800 may also include an output device 835 that can be one or more of a plurality of output mechanisms. In some instances, a multi-mode system allows a user to provide multiple types of input / output to communicate with the computing system 800. The computing system 800 may include a communication interface 840 that typically governs and manages user input and system output. The communication interface can perform or facilitate the reception and / or transmission of wired or wireless communications using wired and / or wireless transceivers, including wired and / or wireless transceivers utilizing audio jacks / plugs, microphone jacks / plugs, Universal Serial Bus (USB) ports / plugs, Apple... ® Lightning ® Ports / plugs, Ethernet ports / plugs, fiber optic ports / plugs, dedicated wired ports / plugs, Bluetooth ® Wireless signal transmission, Bluetooth ® Low-power (BLE) wireless signal transmission, IBEACON ®Wireless signal transmission, including radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.10 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), microwave access global interoperability (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 840 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers used to determine the location of the computing system 800 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the U.S. Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There are no limitations on operation on any particular hardware configuration, and therefore the underlying features here can be easily replaced to obtain improved hardware or firmware configurations as they are developed.

[0132] Storage device 830 may be a non-volatile and / or non-transitory and / or computer-readable storage device, and may be a hard disk or other type of computer-readable medium capable of storing data accessible by a computer, such as magnetic tape, flash memory cards, solid-state storage devices, digital multifunction disks, cartridges, floppy disks, hard disks, magnetic tapes, magnetic stripes, any other magnetic storage media, flash memory, memristor memory, any other solid-state storage, CD-ROM discs, rewritable CD discs, DVD discs, Blu-ray discs (BDD discs), holographic discs, another optical medium, secure digital (SD) cards, microSD cards, Memory Sticks. ®Cards, smart card chips, EMV chips, Subscriber Identity Module (SIM) cards, mini / micro / nano / micro SIM cards, another integrated circuit (IC) chip / card, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM, cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random access memory (RRAM / ReRAM), phase change memory (PCM), spin-transfer torque RAM (STT-RAM), another memory chip or card and / or combinations thereof.

[0133] Storage device 830 may include software services, servers, etc., which enable the system to perform functions when the code defining such software is executed by processor 810. In some examples, hardware services that perform specific functions may include software components for performing functions stored in computer-readable media connected to necessary hardware components such as processor 810, connection 805, output device 835, etc.

[0134] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data can be stored and which do not include carrier waves and / or transient electronic signals propagating wirelessly or over a wired connection. Examples of non-transitory media include, but are not limited to, magnetic disks or magnetic tapes, optical storage media (such as compact optical discs (CDs) or digital versatile discs (DVDs)), flash memory, memory, or memory devices. Computer-readable media may store code and / or machine-executable instructions thereon, which may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, forwarded, or transmitted using any suitable means, including memory sharing, messaging, token passing, network transmission, etc.

[0135] In some examples, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly exclude media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0136] Specific details are provided in the foregoing description to provide a thorough understanding of the examples provided herein. However, those skilled in the art will understand that the examples can be implemented without these specific details. For clarity, in some instances, the technology may be presented as comprising individual functional blocks, including functional blocks containing devices, device components, steps or routines in methods embodied in software or a combination of hardware and software. Additional components may be used in addition to those shown in the figures and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the examples with unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the examples.

[0137] The examples above may be described as processes or methods, depicted as flowcharts, diagrams, data flow diagrams, structure diagrams, or block diagrams. While a flowchart may describe operations as a sequential process, many operations within an operation can be executed in parallel or concurrently. Furthermore, the order of operations can be rearranged. A process terminates when its operations are completed, but a process may have additional steps not included in the accompanying diagrams. A process may correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, its termination may correspond to the function returning to the calling function or the main function.

[0138] The processes and methods described in the examples above can be implemented using stored computer-executable instructions or computer-executable instructions otherwise obtainable from a computer-readable medium. Such instructions may include, for example, instructions and data that configure, or otherwise configure, a general-purpose computer, special-purpose computer, or processing device to perform a function or group of functions. The portion may be accessible via a network of the computer resources used. The computer-executable instructions may be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer-readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include disks or optical discs, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0139] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented as software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smartphones, mobile phones, tablet devices, or other small-form-factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, etc. The functionality described herein may also be embodied in peripheral devices or interlocking cards. By further example, such functionality may also be implemented on circuit boards of different chips or different processes executed on a single device.

[0140] Instructions, media for transmitting such instructions, computing resources for executing them, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0141] In the foregoing description, aspects of this application have been described with reference to specific examples of this application, but those skilled in the art will recognize that this application is not limited thereto. Therefore, while illustrative examples of this application have been described in detail herein, it should be understood that the inventive concepts can be implemented and employed in a variety of other ways, and the appended claims are intended to be construed as including such variations unless limited by prior art. Various features and aspects of the applications described above may be used individually or in combination. Furthermore, the examples may be utilized in any number of settings and applications other than those described herein without departing from the broader spirit and scope of this specification. Therefore, the specification and drawings should be considered illustrative rather than restrictive. For illustrative purposes, the methods are described in a particular order. It should be understood that in alternative examples, the methods may be performed in a different order than that described.

[0142] Those skilled in the art will understand that, without departing from the scope of this description, the less than (“<”) and greater than (“>”) symbols or terms used herein may be replaced with less than or equal to (“>”) respectively. ") and greater than or equal to (" The symbol ) is used instead.

[0143] When a component is described as being “configured” to perform certain operations, such configuration can be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., microprocessors or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0144] The phrase “coupled to” means any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0145] The claim language or other language that states "at least one of" and / or "one or more of" in a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, the claim language that states "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, the claim language that states "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, A and B and C, or any repetition is information or data (e.g., A and A, B and B, C and C, A and A and B, etc.), or any other ordering, repetition, or combination of A, B, and C. The language "at least one of" and / or "one or more of" in a set does not limit the set to the items listed in the set. For example, the language of a claim stating "at least one of A and B" or "at least one of A or B" may refer to A, B, or A and B, and may additionally include items not listed in the set of A and B. The phrases "at least one" and "one or more" are used interchangeably herein.

[0146] Claim language or other languages ​​that state "at least one processor, at least one processor is configured to," "at least one processor is configured to," "one or more processors, one or more processors are configured to," "one or more processors are configured to," etc., indicate that one or more processors (in any combination) can perform associated operations. For example, claim language that states "at least one processor, the at least one processor is configured to: X, Y, and Z" means that a single processor can be used to perform operations X, Y, and Z; or that multiple processors are each assigned a specific subset of tasks of operations X, Y, and Z, such that the multiple processors together perform X, Y, and Z; or that a group of multiple processors work together to perform operations X, Y, and Z. In another example, claim language that states "at least one processor, the at least one processor is configured to: X, Y, and Z" could mean that any single processor can perform only a subset of operations X, Y, and Z.

[0147] When referring to one or more elements that perform functions (e.g., steps of a method), one element may perform all functions, or more than one element may jointly perform these functions. When more than one element jointly performs these functions, each function does not need to be performed by every single element (e.g., different functions may be performed by different elements), and / or each function does not need to be performed by only one element as a whole (e.g., different elements may perform different sub-functions of a function). Similarly, when referring to one or more elements configured to cause another element (e.g., a device) to perform functions, one element may be configured to cause another element to perform all functions, or more than one element may be jointly configured to cause another element to perform these functions.

[0148] When referring to an entity that performs or is configured to perform functions (e.g., steps of a method) (e.g., any entity or device described herein), the entity may be configured to cause one or more elements (individually or collectively) to perform those functions. One or more components of the entity may include at least one memory, at least one processor, at least one communication interface, another component configured to perform one or more of those functions, and / or any combination thereof. When referring to an entity that performs functions, the entity may be configured to cause one component to perform all functions, or to cause more than one component to perform those functions collectively. When the entity is configured to cause more than one component to perform those functions collectively, each function does not need to be performed by every single component (e.g., different functions may be performed by different components), and / or each function does not need to be performed by only one component as a whole (e.g., different components may perform different sub-functions of a function).

[0149] The exemplary aspects of this disclosure include:

[0150] Aspect 1. A method for processing image data, the method comprising: obtaining a first set of image features from one or more images of an environment captured by a camera; transforming the first set of image features to generate a first set of bird's-eye view (BEV) image features; obtaining a second set of features, the second set of features being generated based on a representation of the environment obtained using a sensor having a sensor type different from that of the camera; transforming the second set of features to generate a second set of BEV features; normalizing the first set of BEV image features based on camera configuration information associated with the one or more images; normalizing the second set of BEV features based on sensor configuration information of the sensor; and generating a query graph based on the normalized first set of BEV image features and the normalized second set of BEV features.

[0151] Aspect 2. The method according to aspect 1, wherein the sensor comprises one of a light detection and ranging (LIDAR) sensor, a radar sensor, or a sonar sensor.

[0152] Aspect 3. The method according to any one of Aspects 1 to 2, wherein the camera configuration information includes calibration information of the camera, and wherein the sensor configuration information includes calibration information of the sensor.

[0153] Aspect 4. The method according to aspect 3, wherein the calibration information for the camera includes information associated with at least one of the camera's field of view (FOV), the camera's principal point, and lens distortion information.

[0154] Aspect 5. The method according to any one of Aspects 3 to 4, wherein the calibration information of the sensor includes information associated with at least one of the sensor's mounting height, tilt angle, FOV, and distance.

[0155] Aspect 6. The method according to any one of Aspects 3 to 5, wherein at least one of the calibration information of the camera or the calibration information of the sensor is optimized over time.

[0156] Aspect 7. The method according to any one of Aspects 3 to 6, the method further comprising: adapting the camera configuration information or the sensor configuration information based on an estimate of the environment.

[0157] Aspect 8. The method according to any one of Aspects 1 to 7, wherein at least one of the camera configuration information or the sensor configuration information is determined by a machine learning model.

[0158] Aspect 9. The method according to aspect 8, wherein the data obtained by the sensor is used by the machine learning model to determine the camera configuration information.

[0159] Aspect 10. The method according to any one of Aspects 1 to 9, wherein the normalized first set of BEV image features and the normalized second set of BEV features comprise a normalized BEV feature map, and the method further comprises: dividing the normalized BEV feature map into a cell grid; and generating a BEV feature map based on the divided normalized BEV feature map, wherein features of the normalized BEV feature map in the cells of the cell grid are aggregated into nodes of the BEV feature map.

[0160] Aspect 11. The method according to aspect 10, wherein generating the query graph includes aggregating the BEV feature graph with another BEV feature graph within a time window.

[0161] Aspect 12. The method according to aspect 11, the method further comprising: generating a query graph feature descriptor based on the features of the query graph.

[0162] Aspect 13. The method according to aspect 12, wherein the query graph feature descriptor is generated by a graph neural network.

[0163] Aspect 14. The method according to any one of Aspects 12 to 13, the method further comprising: comparing the query graph feature descriptor with the scene graph feature descriptor to identify a portion of the scene graph that matches the query graph.

[0164] Aspect 15. An apparatus for processing image data, the apparatus comprising: a sensor (or a plurality of sensors); at least one memory; and at least one processor coupled to the at least one memory, the at least one processor being configured to: obtain a first set of image features from one or more images of an environment captured by a camera; transform the first set of image features to generate a first set of bird's-eye view (BEV) image features; obtain a second set of features, the second set of features being generated based on a representation of the environment obtained using a sensor having a different sensor type than the camera; transform the second set of features to generate a second set of BEV features; normalize the first set of BEV image features based on camera configuration information associated with the one or more images; normalize the second set of BEV features based on sensor configuration information of the sensor; and generate a query map based on the normalized first set of BEV image features and the normalized second set of BEV features.

[0165] Aspect 16. The apparatus according to aspect 15, wherein the sensor comprises one of a light detection and ranging (LIDAR) sensor, a radar sensor, or a sonar sensor.

[0166] Aspect 17. The apparatus according to any one of Aspects 15 to 16, wherein the camera configuration information includes calibration information of the camera, and wherein the sensor configuration information includes calibration information of the sensor.

[0167] Aspect 18. The apparatus according to aspect 17, wherein the calibration information of the camera includes information associated with at least one of the camera's field of view (FOV), the camera's principal point, and lens distortion information.

[0168] Aspect 19. The apparatus according to any one of Aspects 17 to 18, wherein the calibration information of the sensor includes information associated with at least one of the sensor's mounting height, tilt angle, FOV, and distance.

[0169] Aspect 20. The apparatus according to any one of aspects 17 to 19, wherein at least one of the calibration information of the camera or the calibration information of the sensor is optimized over time.

[0170] Aspect 21. The apparatus according to any one of Aspects 17 to 20, wherein the at least one processor is configured to adapt the camera configuration information or the sensor configuration information based on an estimate of the environment.

[0171] Aspect 22. The apparatus according to any one of aspects 15 to 21, wherein at least one of the camera configuration information or the sensor configuration information is determined by a machine learning model.

[0172] Aspect 23. The apparatus according to aspect 22, wherein data obtained by the sensor is used by the machine learning model to determine the camera configuration information.

[0173] Aspect 24. The apparatus according to any one of Aspects 15 to 23, wherein the normalized first set of BEV image features and the normalized second set of BEV features comprise a normalized BEV feature map, and wherein the at least one processor is configured to: divide the normalized BEV feature map into a cell grid; and generate a BEV feature map based on the divided normalized BEV feature map, wherein features of the normalized BEV feature map in the cells of the cell grid are aggregated into nodes of the BEV feature map.

[0174] Aspect 25. The apparatus according to aspect 24, wherein, in order to generate the query graph, the at least one processor is configured to aggregate the BEV feature graph with another BEV feature graph within a time window.

[0175] Aspect 26. The apparatus according to aspect 25, wherein the at least one processor is configured to generate a query graph feature descriptor based on features of the query graph.

[0176] Aspect 27. The apparatus according to aspect 26, wherein the query graph feature descriptor is generated by a graph neural network.

[0177] Aspect 28. The apparatus according to any one of Aspects 26 to 27, wherein the at least one processor is further configured to compare the query graph feature descriptor with the scene graph feature descriptor to identify a portion of the scene graph that matches the query graph.

[0178] Aspect 29. The apparatus according to any one of aspects 15 to 28, wherein the apparatus further comprises the camera and the sensor.

[0179] Aspect 30. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by at least one processor, causing the at least one processor to: obtain a first set of image features from one or more images of an environment captured by a camera; transform the first set of image features to generate a first set of bird's-eye view (BEV) image features; obtain a second set of features, the second set of features being generated based on a representation of the environment obtained using a sensor having a different sensor type than the camera; transform the second set of features to generate a second set of BEV features; normalize the first set of BEV image features based on camera configuration information associated with the one or more images; normalize the second set of BEV features based on sensor configuration information of the sensor; and generate a query map based on the normalized first set of BEV image features and the normalized second set of BEV features.

[0180] Aspect 31. A non-transitory computer-readable medium having instructions stored thereon, the instructions causing the at least one processor, when executed, to perform one or more operations according to any one of aspects 1 to 14.

[0181] Aspect 32: An apparatus for image generation, the apparatus comprising components for performing one or more operations according to any one of aspects 1 to 14.

Claims

1. An apparatus for processing image data, the apparatus comprising: sensor; At least one memory; and At least one processor, coupled to the at least one memory, is configured to: Obtain a first set of image features from one or more images of the environment captured by the camera; Transform the first set of image features to generate the first set of bird's-eye view (BEV) image features; A second set of features is obtained, which is generated based on representations of the environment obtained using a sensor with a different sensor type than the camera. Transform the second set of features to generate a second set of BEV features; The features of the first group of BEV images are normalized based on camera configuration information associated with the one or more images; The second set of BEV features is normalized based on the sensor configuration information of the sensor. as well as A query graph is generated based on the normalized first set of BEV image features and the normalized second set of BEV features.

2. The apparatus of claim 1, wherein the sensor comprises one of a light detection and ranging (LIDAR) sensor, a radar sensor, or a sonar sensor.

3. The apparatus of claim 1, wherein the camera configuration information includes calibration information of the camera, and wherein the sensor configuration information includes calibration information of the sensor.

4. The apparatus of claim 3, wherein the calibration information of the camera includes information associated with at least one of the camera's field of view (FOV), the camera's principal point, and lens distortion information.

5. The apparatus of claim 3, wherein the calibration information of the sensor includes information associated with at least one of the sensor's mounting height, tilt angle, FOV, and distance.

6. The apparatus of claim 3, wherein at least one of the calibration information of the camera or the calibration information of the sensor is optimized over time.

7. The apparatus of claim 3, wherein the at least one processor is configured to adapt the camera configuration information or the sensor configuration information based on an estimate of the environment.

8. The apparatus of claim 1, wherein at least one of the camera configuration information or the sensor configuration information is determined by a machine learning model.

9. The apparatus of claim 8, wherein the data obtained by the sensor is used by the machine learning model to determine the camera configuration information.

10. The apparatus of claim 1, wherein the normalized first set of BEV image features and the normalized second set of BEV features comprise normalized BEV feature maps, and wherein the at least one processor is configured to: The normalized BEV feature map is divided into a cell grid; and A BEV feature map is generated based on the partitioned normalized BEV feature map, wherein features of the normalized BEV feature map in the cells of the cell grid are aggregated into the nodes of the BEV feature map.

11. The apparatus according to claim 10, wherein, In order to generate the query graph, the at least one processor is configured to aggregate the BEV feature graph with another BEV feature graph within a time window.

12. The apparatus of claim 11, wherein the at least one processor is configured to generate a query graph feature descriptor based on features of the query graph.

13. The apparatus of claim 12, wherein the query graph feature descriptor is generated by a graph neural network.

14. The apparatus of claim 12, wherein the at least one processor is configured to compare the query graph feature descriptor with the scene graph feature descriptor to identify a portion of the scene graph that matches the query graph.

15. The apparatus of claim 1, wherein the apparatus further comprises the camera and the sensor.

16. A method for processing image data, the method comprising: Obtain a first set of image features from one or more images of the environment captured by the camera; Transform the first set of image features to generate the first set of bird's-eye view (BEV) image features; A second set of features is obtained, which is generated based on representations of the environment obtained using a sensor with a different sensor type than the camera. Transform the second set of features to generate a second set of BEV features; The features of the first group of BEV images are normalized based on camera configuration information associated with the one or more images; The second set of BEV features is normalized based on the sensor configuration information of the sensor. as well as A query graph is generated based on the normalized first set of BEV image features and the normalized second set of BEV features.

17. The method of claim 16, wherein the sensor comprises one of a light detection and ranging (LIDAR) sensor, a radar sensor, or a sonar sensor.

18. The method of claim 16, wherein: The camera configuration information includes the camera's calibration information, which includes information associated with at least one of the camera's field of view (FOV), the camera's principal point, and lens distortion information; and The sensor configuration information includes the sensor's calibration information, which includes information associated with at least one of the sensor's installation height, tilt angle, field of view (FOV), and distance.

19. The method of claim 16, wherein the normalized first set of BEV image features and the normalized second set of BEV features comprise normalized BEV feature maps, and the method further comprises: The normalized BEV feature map is divided into a cell grid; as well as A BEV feature map is generated based on the partitioned normalized BEV feature map, wherein features of the normalized BEV feature map in the cells of the cell grid are aggregated into the nodes of the BEV feature map.

20. The method of claim 19, wherein generating the query graph includes aggregating the BEV feature graph with another BEV feature graph within a time window.

Images