Computing offload for distributed processing

Abstract

Systems and techniques for distributed processing are provided. For example, a process may include sending a discovery message; receiving a response to the discovery message, the response including an identifier of the neighboring device; sending an unloading request to an unloading server, wherein the unloading request comprises the identifier of the adjacent equipment; receiving, from the offload server, an indication that the neighboring device is selected to perform an offload process; sending data for processing to the neighboring device; receiving processed data from the neighboring device; and transmitting an indication of the amount of completed offload processing.

Description

Technical Field

[0001] The present application relates to processing data in a distributed processing system (eg, a distributed or split-rendered extended reality system). For example, aspects of the present application relate to systems and techniques for computational offloading of distributed processing in a distributed processing system. Background Art

[0002] Extended reality (XR) systems or devices can provide virtual content to users and / or can combine the real world and physical environment with a virtual environment (composed of virtual content) to provide users with an XR experience. XR systems typically use powerful processors to perform feature analysis (e.g., extraction, tracking, etc.) and other complex functions quickly enough to display output based on those functions to their users. Powerful processors typically draw power at a high rate. Similarly, transmitting large amounts of data to powerful processors typically draws power at a high rate. Headsets and other portable devices typically have small batteries so as not to be uncomfortably heavy for the user. As a result, some XR systems must be plugged into an external power source and are therefore not portable. Portable XR systems often have short battery life and / or are uncomfortably heavy due to the large batteries they include. Summary of the Invention

[0003] The following presents a simplified summary of one or more aspects disclosed herein. Therefore, the following summary should not be considered an exhaustive overview of all contemplated aspects, nor should it be considered to identify key or critical elements related to all contemplated aspects or to delineate the scope associated with any particular aspect. Accordingly, the following summary presents certain concepts related to one or more aspects of the mechanisms disclosed herein in a simplified form prior to the detailed description presented below.

[0004] Systems and techniques for computational offloading of distributed processing are described herein. In one illustrative example, an apparatus for distributed processing is provided, the apparatus comprising at least one memory and at least one processor coupled to the at least one memory. The at least one processor is configured to: send a discovery message; receive a response to the discovery message, the response including an identifier of a neighboring device; send an offload request to an offload server, the offload request including the identifier of the neighboring device; receive an indication from the offload server that the neighboring device has been selected to perform offload processing; send data for processing to the neighboring device; receive the processed data from the neighboring device; and send an indication of the amount of offload processing completed.

[0005] As another example, a first device for distributed processing is provided, the first device including at least one memory including instructions; and at least one processor coupled to the at least one memory. The at least one processor is configured to: send an indication to an offload server that the first device can provide a computing offload service; receive a discovery message from a second device; send a response to the discovery message; receive an indication from the offload server that the first device is selected to provide the computing offload service; receive data for processing from the second device; process the data to obtain processed data; and send the processed data to the second device.

[0006] In another example, an apparatus for distributed processing is provided, the apparatus comprising at least one memory and at least one processor coupled to the at least one memory. The at least one processor is configured to: receive an offload request from a first device, the offload request including an identifier of a second device; receive an indication from the second device that the second device can provide a computational offload service; select the second device to perform the computational offload service; send an indication that the second device is selected to provide the computational offload service; receive an indication from the first device of an amount of completed offload processing; charge a first account associated with the first device based on the indication of the amount of completed offload processing; and credit a second account associated with the second device based on the indication of the amount of completed offload processing.

[0007] As another example, a method for distributed processing is provided. The method includes: sending a discovery message; receiving a response to the discovery message, the response including an identifier of a neighboring device; sending an offload request to an offload server, the offload request including the identifier of the neighboring device; receiving an indication from the offload server that the neighboring device is selected to perform offload processing; sending data for processing to the neighboring device; receiving the processed data from the neighboring device; and sending an indication of the amount of offload processing completed.

[0008] In another example, a method for distributed processing is provided. The method includes: sending an indication to an offload server that a first device can provide a computing offload service; receiving a discovery message from a second device; sending a response to the discovery message; receiving an indication from the offload server that the first device is selected to provide the computing offload service; receiving data for processing from the second device; processing the data to obtain processed data; and sending the processed data to the second device.

[0009] As another example, a method for distributing processing is provided. The method includes: receiving an offload request from a first device, the offload request including an identifier of a second device; receiving an indication from the second device that the second device can provide a computational offload service; selecting the second device to perform the computational offload service; sending an indication that the second device is selected to provide the computational offload service; receiving an indication from the first device of an amount of completed offload processing; charging a first account associated with the first device based on the indication of the amount of completed offload processing; and crediting a second account associated with the second device based on the indication of the amount of completed offload processing.

[0010] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided that, when executed by at least one processor, causes the at least one processor to: send a discovery message; receive a response to the discovery message, the response including an identifier of the neighboring device; send an offload request to an offload server, the offload request including an identifier of the neighboring device; receive an indication from the offload server that the neighboring device is selected to perform offload processing; send data for processing to the neighboring device; receive processed data from the neighboring device; and send an indication of an amount of offload processing completed.

[0011] As another example, a non-transitory computer-readable medium having instructions stored thereon is provided, which, when executed by at least one processor, causes the at least one processor to: send an indication to an offload server that a first device can provide computing offload services; receive a discovery message from a second device; send a response to the discovery message; receive an indication from the offload server that the first device is selected to provide computing offload services; receive data for processing from the second device; process the data to obtain processed data; and send the processed data to the second device.

[0012] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided that, when executed by at least one processor, causes the at least one processor to: receive an offload request from a first device, the offload request including an identifier of a second device; receive an indication from a second device that the second device can provide a computational offload service; select the second device to perform the computational offload service; send an indication that the second device is selected to provide the computational offload service; receive an indication from the first device of an amount of offload processing completed; charge a first account associated with the first device based on the indication of the amount of offload processing completed; and credit a second account associated with the second device based on the indication of the amount of offload processing completed.

[0013] As another example, an apparatus for distributed processing is provided. The apparatus includes: means for sending a discovery message; means for receiving a response to the discovery message, the response including an identifier of a neighboring device; means for sending an offload request to an offload server, the offload request including an identifier of the neighboring device; means for receiving an indication from the offload server that the neighboring device has been selected to perform offload processing; means for sending data for processing to the neighboring device; means for receiving the processed data from the neighboring device; and means for sending an indication of the amount of offload processing completed.

[0014] In another example, an apparatus for distributed processing is provided. The apparatus includes: a component for sending an indication to an offload server that a first device can provide a computing offload service; a component for receiving a discovery message from a second device; a component for sending a response to the discovery message; a component for receiving an indication from the offload server that the first device is selected to provide the computing offload service; a component for receiving data for processing from the second device; a component for processing the data to obtain processed data; and a component for sending the processed data to the second device.

[0015] As another example, an apparatus for distributing processing is provided. The apparatus includes: means for receiving an offload request from a first device, the offload request including an identifier of a second device; means for receiving an indication from the second device that the second device can provide a computational offload service; means for selecting the second device to perform the computational offload service; means for sending an indication that the second device is selected to provide the computational offload service; means for receiving an indication of an amount of completed offload processing from the first device; means for charging a first account associated with the first device based on the indication of the amount of completed offload processing; and means for crediting a second account associated with the second device based on the indication of the amount of completed offload processing.

[0016] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. This subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all of the drawings, and each claim.

[0017] The foregoing and other features and examples will become more apparent after reference to the following description, claims, and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Illustrative examples of the present application are described in detail below with reference to the following drawings:

[0019] Figure 1 is a block diagram illustrating the architecture of an image capture and processing system according to aspects of the present disclosure.

[0020] Figure 2 is a diagram illustrating the architecture of an example extended reality (XR) system according to some aspects of the present disclosure.

[0021] Figure 3 is a block diagram illustrating the architecture of a simultaneous localization and mapping (SLAM) system according to aspects of the present disclosure.

[0022] Figure 4 An example of an augmented reality enhancement application engine according to aspects of the present disclosure is illustrated.

[0023] Figure 5 Examples of a primary application engine and a secondary application engine that can provide augmented reality enhancements to the primary application engine according to aspects of the present disclosure are illustrated.

[0024] Figure 6 is a diagram illustrating techniques for incentivizing computation offloading of a distributed split rendering architecture for devices on the same network, according to aspects of the present disclosure.

[0025] Figure 7 is a diagram illustrating techniques for motivating computation offloading of a distributed split rendering architecture for devices on different networks according to aspects of the present disclosure.

[0026] Figure 8 is a flow diagram illustrating an example of a process for performing distributed processing in accordance with some aspects.

[0027] Figure 9 is a flow diagram illustrating another example of a process for performing distributed processing in accordance with some aspects.

[0028] Figure 10 is a flow diagram illustrating another example of a process for performing distributed processing in accordance with some aspects.

[0029] Figure 11A is a perspective diagram illustrating a head-mounted display (HMD) 1110 performing feature tracking and / or visual simultaneous localization and mapping (VSLAM) according to some examples.

[0030] Figure 11B is an example based on some examples Figure 11A A perspective view of a head-mounted display (HMD) being worn by a user.

[0031] Figure 12A is a perspective view of a front surface of a mobile device illustrating the use of one or more front-facing cameras to perform feature tracking and / or visual simultaneous localization and mapping (VSLAM) according to some examples.

[0032] Figure 12Bis a perspective view illustrating a rear surface of a mobile device 1250 according to aspects of the present disclosure.

[0033] Figure 13 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. DETAILED DESCRIPTION

[0034] Provided below are certain aspects and examples of the present disclosure. As will be apparent to those skilled in the art, some of these aspects and examples can be applied independently, and some of them can be applied in combination. In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of the subject matter of the present application. However, it will be apparent that each example can be implemented without these specific details. Each drawing and description are not intended to be restrictive.

[0035] The following description provides only illustrative examples and is not intended to limit the scope, applicability or configuration of the present disclosure. Specifically, the following description will provide a feasible description for realizing the illustrative examples to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of elements without departing from the essence and scope of the application as set forth in the appended claims.

[0036] Extended reality (XR) systems or devices can provide virtual content to users and / or can combine the real world and physical environment with a virtual environment (composed of virtual content) to provide users with an XR experience. The real-world environment can include real-world objects (also known as physical objects) such as people, vehicles, buildings, tables, chairs, and / or other real-world or physical objects. XR systems or devices can facilitate interaction with different types of XR environments (e.g., users can use XR systems or devices to interact with XR environments). XR systems can include virtual reality (VR) systems that facilitate interaction with VR environments, augmented reality (AR) systems that facilitate interaction with AR environments, mixed reality (MR) systems that facilitate interaction with MR environments, and / or other XR systems. Examples of XR systems or devices include head-mounted displays (HMDs), smart glasses, and the like. In some cases, the XR system can track user parts (e.g., the user's hands and / or fingertips) to allow the user to interact with items of virtual content.

[0037] AR is a technology that provides virtual or computer-generated content (referred to as AR content) over a user's view of a physical, real-world scene or environment. AR content can include virtual content such as video, images, graphical content, location data (e.g., Global Positioning System (GPS) data or other location data), sound, any combination thereof, and / or other augmented content. AR systems or devices are designed to enhance (or augment) rather than replace a person's current perception of reality. For example, a user may see a real, stationary or moving physical object through an AR device display, but the user's visual perception of the physical object may be enhanced or improved by a virtual image of the object (e.g., a real-world car replaced by a virtual image of a DeLorean), by AR content added to the physical object (e.g., virtual wings added to a living animal), by AR content displayed relative to the physical object (e.g., informational virtual content displayed near a sign on a building, a virtual coffee cup virtually anchored to a real-world table in one or more images (e.g., placed on top of the real-world table), etc.), and / or by displaying other types of AR content. Various types of AR systems can be used for gaming, entertainment, and / or other applications.

[0038] In some cases, an XR system may include an optical "see-through" display or "pass-through" display (e.g., a see-through or pass-through AR HMD or AR glasses), allowing the XR system to display XR content (e.g., AR content) directly onto a real-world view without displaying video content. For example, a user may view physical objects through a display (e.g., glasses or lenses), and the AR system may display AR content onto the display to provide the user with an augmented visual perception of one or more real-world objects. In one example, the display of an optical see-through AR system may include a lens or glasses positioned in front of each eye (or a single lens or glasses positioned above both eyes). The see-through display may allow the user to directly see real-world objects or physical objects, and may display (e.g., project or otherwise display) an augmented image of the object or additional AR content to enhance the user's visual perception of the real world.

[0039] Visual Simultaneous Localization and Mapping (VSLAM) is a computational geometry technique used in devices with cameras, such as robots, head-mounted displays (HMDs), mobile phones, and autonomous vehicles. In VSLAM, a device can build and update a map of an unknown environment based on images captured by the device's camera. As the device updates the map, it can keep track of the device's pose (e.g., position and / or orientation) within the environment. For example, a device can be started in a specific room in a building and moved throughout the building's interior, capturing images. The device can build a map of the environment and track its position within the environment by tracking where different objects in the environment appear in different images.

[0040] Degrees of freedom (DoF) refers to the number of fundamental ways a rigid object can move in three-dimensional (3D) space. In some cases, six different DoFs can be tracked. The six DoFs include three translational DoFs corresponding to translational movement along three perpendicular axes. These three axes can be referred to as the x-axis, y-axis, and z-axis. The six DoFs include three rotational DoFs corresponding to rotational movement about three axes, which can be referred to as pitch, yaw, and roll.

[0041] In the context of systems that track movement through an environment (such as XR systems and / or VSLAM systems), degrees of freedom can refer to which of the six degrees of freedom the system is able to track. 3DoF systems typically track three rotational DoFs—pitch, yaw, and roll. For example, a 3DoF headset can track if a user wearing the headset turns their head left or right, tilts their head up or down, and / or tilts their head left or right. A 6DoF system can track three translational DoFs as well as three rotational DoFs. Thus, for example, a 6DoF headset can track a user moving forward, backward, laterally, and / or vertically in addition to tracking three rotational DoFs.

[0042] In some cases, an XR system may include an HMD display, such as an AR HMD or AR glasses, that can be worn by a user of the XR system. Generally, it is desirable to keep the HMD display as light and small as possible. To help reduce the weight and size of the HMD display, the HMD display may be a relatively low-power system (e.g., in terms of battery and computing power) compared to the device to which the HMD display is connected (e.g., wired or wireless connection) (e.g., a companion device such as a mobile phone, server device, or other device).

[0043] In some cases, a distributed processing system (such as a split rendering system) can be implemented to reduce the computational load on an XR system (such as an HMD display). For example, in split rendering, a companion device can perform certain tasks relative to one or more images to be displayed by the HMD display and send the task results to the HMD display. The HMD display can then perform additional image processing tasks and display the one or more images. In some cases, it may be beneficial to offload certain tasks from the companion device (or HMD display) to other devices to assist with processing.

[0044] This document describes systems, apparatuses, methods (also referred to as processes), and computer-readable media (collectively referred to herein as "systems and techniques") for distributed processing. For example, the systems and techniques can offload computational components of image processing tasks to nearby devices for processing. In some cases, the nearby devices can indicate to an offload server that the nearby devices can provide computational offload services. The XR system can use neighbor discovery (also referred to as "discovery") to search for or locate nearby devices. The nearby devices can respond to the neighbor discovery request. The XR system can transmit a request for offload processing to the offload server. The offload server can coordinate the offload processing by pairing the XR system with the nearby devices and setting up the offload. The XR system can then transmit data for the offload processing to the nearby devices. The nearby devices can process the data and transmit the processed data back to the XR system. In some cases, the XR system can communicate directly with the nearby devices via, for example, a sidelink connection or another radio access technology. After at least partially completing the offload processing, the XR system can report the amount of offload processing performed by the nearby devices. The offload server can then credit the nearby devices. In some cases, this credit can be used for further offload services, preferential treatment for future services, or other account credits.

[0045] The systems and techniques described herein can provide various advantages. For example, by providing credits to devices that provide offload services, the techniques can help incentivize the provision of offload services. Additionally, because the nearby device is relatively close to the device seeking offload services (e.g., an XR device), the data to be processed and the results of the processing can be communicated directly between the nearby device and the device seeking offload services, such as by using a sidelink connection or another radio access technology (RAT) (such as Wi-Fi). By allowing the nearby device and the device seeking offload services to communicate directly, transmission bandwidth between the device and a network (such as a cellular network) can be conserved. Additionally, allowing the nearby device and the device seeking offload services to communicate directly helps avoid potential round-trip delays and / or routing delays.

[0046] Various aspects of the application will be described with respect to the accompanying drawings. Figure 1is a block diagram illustrating the architecture of image capture and processing system 100. Image capture and processing system 100 includes various components for capturing and processing images of a scene (e.g., images of scene 110). Image capture and processing system 100 can capture individual images (or photographs) and / or can capture videos comprising multiple images (or video frames) in a particular sequence. In some cases, lens 115 and image sensor 130 may be associated with an optical axis. In one illustrative example, both the photosensitive area of ​​image sensor 130 (e.g., a photodiode) and lens 115 may be centered about the optical axis. Lens 115 of image capture and processing system 100 faces scene 110 and receives light from scene 110. Lens 115 bends incoming light from the scene toward image sensor 130. Light received by lens 115 passes through an aperture. In some cases, the aperture (e.g., aperture size) is controlled by one or more control mechanisms 120 and received by image sensor 130. In some cases, the aperture may have a fixed size.

[0047] The one or more control mechanisms 120 may control exposure, focus, and / or zoom based on information from the image sensor 130 and / or based on information from the image processor 150. The one or more control mechanisms 120 may include a plurality of mechanisms and components; for example, the control mechanisms 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. The one or more control mechanisms 120 may also include additional control mechanisms beyond those illustrated, such as controls for analog gain, flash, HDR, depth of field, and / or other image capture attributes.

[0048] Focus control mechanism 125B of control mechanism 120 may obtain a focus setting. In some examples, focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, focus control mechanism 125B may adjust the position of lens 115 relative to the position of image sensor 130. For example, based on the focus setting, focus control mechanism 125B may actuate a motor or servo (or other lens mechanism) to move lens 115 closer to or further away from image sensor 130, thereby adjusting the focus. In some cases, image capture and processing system 100 may include additional lenses, such as one or more microlenses positioned above each photodiode of image sensor 130, each of which bends light received from lens 115 toward the corresponding photodiode before it reaches the photodiode. The focus setting may be determined via contrast detection autofocus (CDAF), phase detection autofocus (PDAF), hybrid autofocus (HAF), or some combination thereof. The focus setting may be determined using control mechanism 120, image sensor 130, and / or image processor 150. The focus setting may be referred to as an image capture setting and / or an image processing setting.In some cases, the lens 115 may be fixed relative to the image sensor, and the focus control mechanism 125B may be omitted without departing from the scope of the present disclosure.

[0049] Exposure control mechanism 125A of control mechanism 120 may obtain an exposure setting. In some cases, exposure control mechanism 125A stores the exposure setting in a memory register. Based on the exposure setting, exposure control mechanism 125A may control the size of the aperture (e.g., aperture size or f-stop), the duration that the aperture is open (e.g., exposure time or shutter speed), the duration that the sensor collects light (e.g., exposure time or electronic shutter speed), the sensitivity of image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by image sensor 130, or any combination thereof. The exposure setting may be referred to as an image capture setting and / or an image processing setting.

[0050] Zoom control mechanism 125C of control mechanism 120 may obtain a zoom setting. In some examples, zoom control mechanism 125C stores the zoom setting in a memory register. Based on the zoom setting, zoom control mechanism 125C may control the focal length of an assembly of lens elements (lens assembly) including lens 115 and one or more additional lenses. For example, zoom control mechanism 125C may control the focal length of the lens assembly by actuating one or more motors or servos (or other lens mechanisms) to move one or more lenses relative to one another. The zoom setting may be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly may include a parfocal zoom lens or a variable focal length zoom lens. In some examples, the lens assembly may include a focusing lens (in some cases, this focusing lens may be lens 115) that first receives light from scene 110, where the light then passes through an afocal zoom system between the focusing lens (e.g., lens 115) and image sensor 130 before reaching image sensor 130. In some cases, an afocal zoom system may include two positive (e.g., converging, convex) lenses with equal or similar focal lengths (e.g., within a threshold difference of each other), with a negative (e.g., diverging, concave) lens between them. In some cases, zoom control mechanism 125C moves one or more of the lenses in the afocal zoom system, such as the negative lens and one or both of the positive lenses. In some cases, zoom control mechanism 125C may control zoom by capturing images from an image sensor (e.g., including image sensor 130) from a plurality of image sensors at a zoom corresponding to a zoom setting. For example, image processing system 100 may include a wide-angle image sensor with a relatively low zoom and a telephoto image sensor with a greater zoom. In some cases, based on the selected zoom setting, zoom control mechanism 125C may capture images from the corresponding sensor.

[0051] Image sensor 130 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a particular pixel in the image generated by image sensor 130. In some cases, different photodiodes may be covered by different filters. In some cases, different photodiodes may be covered in different color filters and may therefore measure light that matches the color of the filter covering the photodiode. Various color filter arrays may be used, including a Bayer color filter array, a four-color color filter array (also known as a four-color Bayer color filter array or QCFA), and / or any other color filter array. For example, a Bayer color filter includes a red filter, a blue filter, and a green filter, where each pixel of the image is generated based on red light data from at least one photodiode covered in the red filter, blue light data from at least one photodiode covered in the blue filter, and green light data from at least one photodiode covered in the green filter.

[0052] Back to Figure 1 Other types of color filters may use yellow, magenta, and / or cyan (also known as "emerald") filters as an alternative to or in addition to red, blue, and / or green filters. In some cases, some photodiodes may be configured to measure infrared (IR) light. In some implementations, the photodiodes measuring IR light may not be covered by any filters, thereby allowing the IR photodiodes to measure both visible light (e.g., color) and IR light. In some examples, the IR photodiodes may be covered by IR filters, thereby allowing IR light to pass through and blocking light from other parts of the spectrum (e.g., visible light, color). Some image sensors (e.g., image sensor 130) may lack filters entirely (e.g., color, IR, or any other part of the spectrum) and may instead use different photodiodes (in some cases stacked vertically) throughout the pixel array. Different photodiodes throughout the pixel array may have different spectral sensitivity curves, thereby responding to different wavelengths of light. Monochrome image sensors may also lack filters and, therefore, lack color depth.

[0053] In some cases, image sensor 130 may alternatively or additionally include opaque and / or reflective masks that block light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles. In some cases, opaque and / or reflective masks may be used for phase detection autofocus (PDAF). In some cases, opaque and / or reflective masks may be used to block portions of the electromagnetic spectrum from reaching the image sensor's photodiodes (e.g., IR cutoff filters, UV cutoff filters, bandpass filters, low-pass filters, high-pass filters, etc.). Image sensor 130 may also include analog gain amplifiers for amplifying analog signals output by the photodiodes and / or analog-to-digital converters (ADCs) for converting the analog signals output by the photodiodes (and / or the analog signals amplified by the analog gain amplifiers) into digital signals. In some cases, certain components or functionality discussed with respect to one or more of control mechanisms 120 may alternatively or additionally be included in image sensor 130. Image sensor 130 may be a charge coupled device (CCD) sensor, an electron multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal oxide semiconductor (CMOS), an N-type metal oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0054] The image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including the ISP 154), one or more host processors (including the host processor 152), and / or related Figure 101000 . The host processor 152 may be a digital signal processor (DSP) and / or other types of processors. In some implementations, the image processor 150 is a single integrated circuit or chip (e.g., referred to as a system on a chip or SoC) that includes the host processor 152 and the ISP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth™, Global Positioning System (GPS), etc.), any combination thereof, and / or other components. The I / O ports 156 may include any suitable input / output ports or interfaces according to one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a serial General Purpose Input / Output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface), an Advanced High-Performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In one illustrative example, the host processor 152 may communicate with the image sensor 130 using an I2C port, and the ISP 154 may communicate with the image sensor 130 using a MIPI port.

[0055] The image processor 150 may perform a number of tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. The image processor 150 may store image frames and / or processed images in a random access memory (RAM) 140 / 1025, a read-only memory (ROM) 145 / 1020, a cache, a memory unit, another storage device, or some combination thereof.

[0056] Various input / output (I / O) devices 160 may be connected to the image processor 150. The I / O devices 160 may include a display screen, a keyboard, a keypad, a touch screen, a touchpad, a touch-sensitive surface, a printer, any other output device 1035, any other input device 1045, or some combination thereof. In some cases, subtitles may be entered into the image processing device 105B via a physical keyboard or keypad of the I / O device 160, or via a virtual keyboard or keypad of the touch screen of the I / O device 160. The I / O devices 160 may include one or more ports, jacks, or other connectors that enable wired connections between the image capture and processing system 100 and one or more peripheral devices, via which the image capture and processing system 100 can receive data from and / or send data to the one or more peripheral devices. The I / O devices 160 may also include one or more wireless transceivers that enable wireless connections between the image capture and processing system 100 and one or more peripheral devices, via which the image capture and processing system 100 can receive data from and / or send data to the one or more peripheral devices. Peripheral devices may include any of the types of I / O devices 160 discussed previously, and may themselves be considered I / O devices 160 once they are coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector.

[0057] In some cases, the image capture and processing system 100 can be a single device. In some cases, the image capture and processing system 100 can be two or more separate devices, including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some implementations, the image capture device 105A and the image processing device 105B can be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly coupled together via one or more wireless transceivers. In some implementations, the image capture device 105A and the image processing device 105B can be disconnected from each other.

[0058] like Figure 1 As shown, the vertical dotted line will Figure 1 1. The image capture and processing system 100 is divided into two parts, representing image capture device 105A and image processing device 105B. Image capture device 105A includes lens 115, control mechanism 120, and image sensor 130. Image processing device 105B includes image processor 150 (including ISP 154 and host processor 152), RAM 140, ROM 145, and I / O device 160. In some cases, some components illustrated in image capture device 105A (such as ISP 154 and / or host processor 152) may be included in image capture device 105A.

[0059] The image capture and processing system 100 may include an electronic device, such as a mobile or landline telephone handset (e.g., a smartphone, a cell phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the image capture and processing system 100 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 802.11 Wi-Fi communication, wireless local area network (WLAN) communication, or some combination thereof. In some implementations, the image capture device 105A and the image processing device 105B may be different devices. For example, the image capture device 105A may include a camera device, and the image processing device 105B may include a computing device, such as a mobile handset, a desktop computer, or other computing device.

[0060] Although the image capture and processing system 100 is shown as including certain components, one of ordinary skill will appreciate that the image capture and processing system 100 may include more than Figure 1 . The components of the image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components of the image capture and processing system 100 may include and / or be implemented using electronic circuitry or other electronic hardware that may include one or more programmable electronic circuits (e.g., a microprocessor, GPU, DSP, CPU, and / or other suitable electronic circuitry) and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device that implements the image capture and processing system 100.

[0061] In some examples, Figure 2 In some examples, the extended reality (XR) system 200 may include the image capture and processing system 100, the image capture device 105A, the image processing device 105B, or a combination thereof. Figure 3 The simultaneous localization and mapping (SLAM) system 300 may include the image capture and processing system 100 , the image capture device 105A, the image processing device 105B, or a combination thereof.

[0062] Figure 22 is a schematic diagram illustrating the architecture of an example extended reality (XR) system 200 according to some aspects of the present disclosure. The XR system 200 can run (or execute) XR applications and implement XR operations. In some examples, as part of an XR experience, the XR system 200 can perform tracking and positioning, mapping of an environment in the physical world (e.g., a scene), and / or positioning and rendering of virtual content on a display 209 (e.g., a screen, visible plane / area, and / or other display). For example, the XR system 200 can generate a map (e.g., a three-dimensional (3D) map) of the environment in the physical world, track the pose (e.g., position and positioning) of the XR system 200 relative to the environment (e.g., relative to the 3D map of the environment), position and / or anchor virtual content in a specific location on the map of the environment, and render the virtual content on the display 209 so that the virtual content appears to be at a location in the environment corresponding to the specific location on the map of the scene at which the virtual content is positioned and / or anchored. The display 209 may include glass, screens, lenses, projectors, and / or other display mechanisms that allow a user to see the real-world environment and also allow XR content to be overlaid, superimposed, blended, or otherwise displayed thereon.

[0063] In this illustrative example, XR system 200 includes one or more image sensors 202, accelerometer 204, gyroscope 206, storage 207, computing component 210, XR engine 220, image processing engine 224, rendering engine 226, and communication engine 228. It should be noted that Figure 2 The components 202-228 shown are non-limiting examples provided for purposes of illustration and explanation, and other examples may include components related to Figure 2 For example, in some cases, the XR system 200 may include one or more other sensors (e.g., one or more inertial measurement units (IMUs), radars, light detection and ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, sound detection and ranging (SODAR) sensors, sound navigation and ranging (SONAR) sensors, audio sensors, etc.), one or more display devices, one or more other processing engines, one or more other hardware components, and / or Figure 2 One or more other software and / or hardware components not shown. Although various components of XR system 200 (such as image sensor 202) may be referred to herein in the singular, it should be understood that XR system 200 may include multiple components of any component discussed herein (e.g., multiple image sensors 202).

[0064] The XR system 200 includes or communicates (wired or wirelessly) with an input device 208. The input device 208 can include any suitable input device, such as a touch screen, a pen or other pointer device, a keyboard, a mouse, buttons or keys, a microphone for receiving voice commands, a gesture input device for receiving gesture commands, a video game controller, a steering wheel, a joystick, a set of buttons, a trackball, a remote control, any of the other input devices 1045 discussed herein, or any combination thereof. In some cases, the image sensor 202 can capture images that can be processed for interpreting gesture commands.

[0065] The XR system 200 may also communicate with one or more other electronic devices (wired or wirelessly). For example, the communication engine 228 may be configured to manage connections and communicate with one or more electronic devices. In some cases, the communication engine 228 may correspond to Figure 10 Communication interface 1040.

[0066] In some implementations, one or more image sensors 202, accelerometer 204, gyroscope 206, storage 207, computing component 210, XR engine 220, image processing engine 224, and rendering engine 226 can be part of the same computing device. For example, in some cases, one or more image sensors 202, accelerometer 204, gyroscope 206, storage 207, computing component 210, XR engine 220, image processing engine 224, and rendering engine 226 can be integrated into an HMD, extended reality glasses, a smartphone, a laptop, a tablet, a gaming system, and / or any other computing device. However, in some implementations, one or more image sensors 202, accelerometer 204, gyroscope 206, storage 207, computing component 210, XR engine 220, image processing engine 224, and rendering engine 226 can be part of two or more separate computing devices. For example, in some cases, some of the components 202 - 226 may be part of or implemented by one computing device, and the remaining components may be part of or implemented by one or more other computing devices.

[0067] Storage 207 can be any storage device for storing data. Furthermore, storage 207 can store data from any of the components of XR system 200. For example, storage 207 can store data from image sensor 202 (e.g., image or video data), data from accelerometer 204 (e.g., measurements), data from gyroscope 206 (e.g., measurements), data from compute component 210 (e.g., processing parameters, preferences, virtual content, rendered content, scene maps, tracking and positioning data, object detection data, privacy data, XR application data, facial recognition data, occlusion data, etc.), data from XR engine 220, data from image processing engine 224, and / or data from rendering engine 226 (e.g., output frames). In some examples, storage 207 can include a buffer for storing frames for processing by compute component 210.

[0068] One or more computing components 210 may include a central processing unit (CPU) 212, a graphics processing unit (GPU) 214, a digital signal processor (DSP) 216, an image signal processor (ISP) 218, and / or other processors (e.g., a neural processing unit (NPU) implementing one or more trained neural networks). Computing component 210 may perform various operations such as image enhancement, computer vision, graphics rendering, extended reality operations (e.g., tracking, localization, pose estimation, mapping, content anchoring, content rendering, etc.), image and / or video processing, sensor processing, recognition (e.g., text recognition, facial recognition, object recognition, feature recognition, tracking or pattern recognition, scene recognition, occlusion detection, etc.), trained machine learning operations, filtering, and / or any of the various operations described herein. In some examples, computing component 210 may implement (e.g., control, operate, etc.) an XR engine 220, an image processing engine 224, and a rendering engine 226. In other examples, computing component 210 may also implement one or more other processing engines.

[0069] Image sensor 202 may include any image and / or video sensor or capture device. In some examples, image sensor 202 may be part of a multi-camera assembly (such as a dual-camera assembly). Image sensor 202 may capture image and / or video content (e.g., raw image and / or video data), which may then be processed by compute component 210, XR engine 220, image processing engine 224, and / or rendering engine 226, as described herein. In some examples, image sensor 202 may include image capture and processing system 100, image capture device 105A, image processing device 105B, or a combination thereof.

[0070] In some examples, image sensor 202 may capture image data and may generate an image (also referred to as a frame) based on the image data and / or may provide the image data or frame to XR engine 220, image processing engine 224, and / or rendering engine 226 for processing. An image or frame may include a video frame from a video sequence or a still image. An image or frame may include an array of pixels representing a scene. For example, the image may be a red-green-blue (RGB) image with red, green, and blue color components per pixel; a luminance, redness, blueness (YCbCr) image with a luminance component and two chrominance (color) components (redness and blueness) per pixel; or any other suitable type of color or monochrome image.

[0071] In some cases, image sensor 202 (and / or other cameras of XR system 200) may also be configured to capture depth information. For example, in some implementations, image sensor 202 (and / or other cameras) may include an RGB depth (RGB-D) camera. In some cases, XR system 200 may include one or more depth sensors (not shown) that are separate from image sensor 202 (and / or other cameras) and that can capture depth information. For example, such depth sensors may obtain depth information independently of image sensor 202. In some examples, the depth sensor may be physically mounted in the same general location as image sensor 202 but may operate at a different frequency or frame rate than image sensor 202. In some examples, the depth sensor may take the form of a light source that projects a structured or textured light pattern (which may include one or more narrowband lights) onto one or more objects in a scene. Depth information can then be obtained by exploiting the geometric deformation of the projected pattern caused by the surface shape of the objects. In one example, depth information can be obtained from a stereo sensor, such as a combination of an infrared structured light projector and an infrared camera registered to a camera (e.g., an RGB camera).

[0072] XR system 200 may also include other sensors within its sensor(s). The sensor(s) may include one or more accelerometers (e.g., accelerometer 204), one or more gyroscopes (e.g., gyroscope 206), and / or other sensors. The sensor(s) may provide velocity, orientation, and / or other positioning-related information to computing component 210. For example, accelerometer 204 may detect acceleration of XR system 200 and may generate acceleration measurements based on the detected acceleration. In some cases, accelerometer 204 may provide one or more translation vectors (e.g., up / down, left / right, forward / backward) that may be used to determine the position or pose of XR system 200. Gyroscope 206 may detect and measure the orientation and angular velocity of XR system 200. For example, gyroscope 206 may be used to measure the pitch, roll, and yaw of XR system 200. In some cases, gyroscope 206 may provide one or more rotation vectors (e.g., pitch, yaw, roll). In some examples, image sensor 202 and / or XR engine 220 may use measurements obtained by accelerometer 204 (e.g., one or more translation vectors) and / or measurements obtained by gyroscope 206 (e.g., one or more rotation vectors) to calculate the pose of XR system 200. As previously described, in other examples, XR system 200 may also include other sensors, such as an inertial measurement unit (IMU), a magnetometer, gaze and / or eye tracking sensors, machine vision sensors, smart scene sensors, voice recognition sensors, shock sensors, vibration sensors, positioning sensors, tilt sensors, and the like.

[0073] As mentioned above, in some cases, the one or more sensors may include at least one IMU. An IMU is an electronic device that uses a combination of one or more accelerometers, one or more gyroscopes, and / or one or more magnetometers to measure specific forces, angular velocities, and / or orientations of XR system 200. In some examples, the one or more sensors may output measured information associated with the capture of images captured by image sensor 202 (and / or other cameras of XR system 200) and / or depth information obtained using one or more depth sensors of XR system 200.

[0074] The XR engine 220 can use the output of one or more sensors (e.g., accelerometer 204, gyroscope 206, one or more IMUs, and / or other sensors) to determine the pose (also referred to as head pose) of the XR system 200 and / or the pose of the image sensor 202 (or other camera of the XR system 200). In some cases, the pose of the XR system 200 and the pose of the image sensor 202 (or other camera) can be the same. The pose of the image sensor 202 refers to the position and orientation of the image sensor 202 relative to a reference frame (e.g., with respect to the scene 110). In some implementations, the camera pose can be determined for six degrees of freedom (6DoF), which refers to three translational components (e.g., which can be given by X (horizontal), Y (vertical), and Z (depth) coordinates relative to a reference frame (such as the image plane)) and three angular components (e.g., roll, pitch, and yaw relative to the same reference frame). In some implementations, the camera pose can be determined for 3 degrees of freedom (3DoF), which refers to three angular components (eg, roll, pitch, and yaw).

[0075] In some cases, a device tracker (not shown) may use measurements from one or more sensors and image data from image sensor 202 to track the pose (e.g., a 6DoF pose) of XR system 200. For example, the device tracker may fuse visual data from the image data (e.g., using a visual tracking solution) with inertial data from the measurements to determine the position and motion of XR system 200 relative to the physical world (e.g., a scene) and a map of the physical world. As described below, in some examples, while tracking the pose of XR system 200, the device tracker may generate a three-dimensional (3D) map of the scene (e.g., the real world) and / or generate updates to the 3D map of the scene. 3D map updates may include, for example, but not limited to, new or updated features and / or feature or landmark points associated with the scene and / or the 3D map of the scene, position updates that identify or update the position of XR system 200 within the scene and the 3D map of the scene, and the like. The 3D map may provide a digital representation of the scene in the real / physical world. In some examples, the 3D map can anchor location-based objects and / or content to real-world coordinates and / or objects. The XR system 200 can use the mapped scene (e.g., a scene in the physical world represented by a 3D map and / or associated with the map) to merge the physical and virtual worlds and / or merge virtual content or objects with the physical environment.

[0076] In some aspects, the computing component 210 can use a visual tracking solution to determine and / or track the pose of the image sensor 202 and / or the XR system 200 as a whole based on images captured by the image sensor 202 (and / or other cameras of the XR system 200). For example, in some examples, the computing component 210 can perform tracking using computer vision-based tracking, model-based tracking, and / or simultaneous localization and mapping (SLAM) techniques. For example, the computing component 210 can perform SLAM or can communicate with a SLAM system (not shown) (such as Figure 3 SLAM refers to a class of technologies that simultaneously tracks the pose of a camera (e.g., image sensor 202) and / or XR system 200 relative to a map of an environment (e.g., a map of the environment modeled by XR system 200) while creating a map of the environment. This map may be referred to as a SLAM map and may be three-dimensional (3D). SLAM technology can be performed using color or grayscale image data captured by image sensor 202 (and / or other cameras of XR system 200) and can be used to generate an estimate of 6DoF pose measurements of image sensor 202 and / or XR system 200. Such SLAM technology configured to perform 6DoF tracking may be referred to as 6DoF SLAM. In some cases, the output of one or more sensors (e.g., accelerometer 204, gyroscope 206, one or more IMUs, and / or other sensors) may be used to estimate, correct, and / or otherwise adjust the estimated pose.

[0077] In some cases, 6DoF SLAM (e.g., 6DoF tracking) can associate features observed from certain input images from image sensor 202 (and / or other cameras) to a SLAM map. For example, 6DoF SLAM can use feature point associations from an input image to determine the pose (position and orientation) of image sensor 202 and / or XR system 200 for that input image. 6DoF map building can also be performed to update the SLAM map. In some cases, a SLAM map maintained using 6DoF SLAM can include 3D feature points triangulated from two or more images. For example, keyframes can be selected from an input image or video stream to represent the observed scene. For each keyframe, a corresponding 6DoF camera pose associated with the image can be determined. The pose of image sensor 202 and / or XR system 200 can be determined by projecting features from the 3D SLAM map into the image or video frame and updating the camera pose based on verified 2D-3D correspondences.

[0078] In one illustrative example, the computing component 210 may extract feature points from certain input images (e.g., each input image, a subset of the input images, etc.) or from each keyframe. As used herein, a feature point (also referred to as a registration point) is a unique or identifiable portion of an image, such as a portion of a hand, the edge of a table, and other examples. Features extracted from a captured image may represent different feature points along three-dimensional space (e.g., coordinates on the X, Y, and Z axes), and each feature point may have an associated feature location. Feature points in a keyframe may match (be identical to or correspond to) or fail to match feature points from a previously captured input image or keyframe. Feature detection may be used to detect feature points. Feature detection may include image processing operations for examining one or more pixels of an image to determine whether a feature exists at a particular pixel. Feature detection may be used to process the entire captured image or portions of an image. For each image or keyframe, once a feature has been detected, a local image patch surrounding the feature may be extracted. Features may be extracted using any suitable technique, such as Scale Invariant Feature Transform (SIFT) (which localizes features and generates their descriptions), Learned Invariant Feature Transform (LIFT), Speeded Up Robust Features (SURF), Gradient Location Orientation Histogram (GLOH), Orientation Rapid Rotation Brief (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retinal Keypoints (FREAK), KAZE, Accelerated KAZE (AKAZE), Normalized Cross Correlation (NCC), descriptor matching, another suitable technique, or a combination thereof.

[0079] As an illustrative example, the computing component 210 may extract a message corresponding to a mobile device (e.g., Figure 4 Mobile devices 440, Figure 5 In some cases, feature points corresponding to the mobile device may be tracked to determine the pose of the mobile device. As described in more detail below, the pose of the mobile device may be used to determine the location of a projection of AR media content that may augment media content displayed on the display of the mobile device.

[0080] In some cases, the XR system 200 may also track the user's hands and / or fingers to allow the user to interact with and / or control virtual content in the virtual environment. For example, the XR system 200 may track the pose and / or movement of the user's hands and / or fingertips to identify or translate user interactions with the virtual environment. User interactions may include, for example, but are not limited to, moving virtual content items, resizing virtual content items, selecting input interface elements in a virtual user interface (e.g., a virtual representation of a mobile phone, a virtual keyboard, and / or other virtual interfaces), providing input through the virtual user interface, and the like.

[0081] Figure 3 is a block diagram illustrating the architecture of a simultaneous localization and mapping (SLAM) system 300. In some examples, the SLAM system 300 may be or may include an extended reality (XR) system such as Figure 2 In some examples, the SLAM system 300 can be a wireless communication device, a mobile device or cell phone (e.g., a mobile phone or so-called "smartphone" or other mobile device), a wearable device, a personal computer, a laptop computer, a server computer, a portable video game console, a portable media player, a camera device, a manned or unmanned ground vehicle, a manned or unmanned aerial vehicle, a manned or unmanned water vehicle, a manned or unmanned underwater vehicle, a manned or unmanned vehicle, an autonomous vehicle, a vehicle, a computing system of a vehicle, a robot, another device, or any combination thereof.

[0082] Figure 3 The SLAM system 300 includes or is coupled to each of one or more sensors 305. The one or more sensors 305 may include one or more cameras 310. Each of the one or more cameras 310 may include an image capture device 105A, an image processing device 105B, an image capture and processing system 100, another type of camera, or a combination thereof. Each of the one or more cameras 310 may respond to light from a particular spectrum. The spectrum may be a subset of the electromagnetic (EM) spectrum. For example, each of the one or more cameras 310 may be a visible light (VL) camera that responds to the VL spectrum, an infrared (IR) camera that responds to the IR spectrum, an ultraviolet (UV) camera that responds to light from another spectrum in another portion of the EM spectrum, or some combination thereof.

[0083] The one or more sensors 305 may include one or more other types of sensors in addition to the camera 310, such as one or more of each of the following: an accelerometer, a gyroscope, a magnetometer, an inertial measurement unit (IMU), an altimeter, a barometer, a thermometer, a radio detection and ranging (RADAR) sensor, a light detection and ranging (LIDAR) sensor, a sound navigation and ranging (SONAR) sensor, a sound detection and ranging (SODAR) sensor, a global navigation satellite system (GNSS) receiver, a global positioning system (GPS) receiver, a BeiDou navigation satellite system (BDS) receiver, a Galileo receiver, a GLONASS satellite navigation system (GLONASS) receiver, an Indian constellation navigation (NavIC) receiver, a Quasi-Zenith Satellite System (QZSS) receiver, a Wi-Fi positioning system (WPS) receiver, a cellular network positioning system receiver, a Bluetooth® Beacon positioning receivers, short-range wireless beacon positioning receivers, personal area network (PAN) positioning receivers, wide area network (WAN) positioning receivers, wireless local area network (WLAN) positioning receivers, other types of positioning receivers, other types of sensors discussed herein, or combinations thereof. In some examples, the one or more sensors 305 may include Figure 2 Any combination of sensors of the XR System 200.

[0084] Figure 3 The SLAM system 300 includes a visual-inertial odometry (VIO) tracker 315. The term "visual-inertial odometry" may also be referred to herein as visual odometry. The VIO tracker 315 receives sensor data 365 from the one or more sensors 305. For example, the sensor data 365 may include one or more images captured by the one or more cameras 310. The sensor data 365 may include other types of sensor data from the one or more sensors 305, such as data from any of the types of sensors 305 listed herein. For example, the sensor data 365 may include inertial measurement unit (IMU) data from one or more of the one or more sensors 305.

[0085] After receiving sensor data 365 from one or more sensors 305, the VIO tracker 315 performs feature detection, extraction, and / or tracking using the feature tracking engine 320 of the VIO tracker 315. For example, where the sensor data 365 includes one or more images captured by one or more cameras 310 of the SLAM system 300, the VIO tracker 315 may identify, detect, and / or extract features in each image. Features may include visually distinct points in an image, such as portions of an image that depict edges and / or corners. The VIO tracker 315 may periodically and / or continuously receive sensor data 365 from the one or more sensors 305, for example, by continuously receiving more images from the one or more cameras 310 as the one or more cameras 310 capture a video, where the images are video frames of the video. The VIO tracker 315 may generate descriptors for the features. The feature descriptors may be generated, at least in part, by generating a description of the feature, as depicted in a local image patch extracted around the feature. In some examples, the feature descriptors may describe the feature as a collection of one or more feature vectors. The VIO tracker 315, in some cases in conjunction with the mapping engine 330 and / or the relocalization engine 355, may associate multiple features with a map of the environment based on such feature descriptors. The feature tracking engine 320 of the VIO tracker 315 may perform feature tracking by identifying features in each image that the VIO tracker 315 has previously identified in one or more previous images (in some cases, based on identifying features with matching feature descriptors in different images). The feature tracking engine 320 may track changes in one or more locations at which a feature is depicted in each of the different images. For example, the feature extraction engine may detect a particular corner of a room depicted in a first image captured by a first camera in the cameras 310 on the left side. The feature extraction engine may detect the same feature depicted (e.g., the same particular corner of the same room) on the right side of a second image captured by the first camera. The feature tracking engine 320 may recognize that the feature detected in the first image and the second image are two depictions of the same feature (e.g., the same particular corner of the same room), and that the feature appears in two different locations in the two images. The VIO tracker 315 may determine that the first camera has moved based on the appearance of the same feature on the left side of the first image and on the right side of the second image, for example, where the feature depicts a static portion of the environment (eg, a particular corner of a room).

[0086] The VIO tracker 315 may include a sensor integration engine 325. The sensor integration engine 325 may use sensor data from other types of sensors 305 (besides the cameras 310) to determine information that can be used by the feature tracking engine 320 when performing feature tracking. For example, the sensor integration engine 325 may receive IMU data from an IMU in one or more of the sensors 305 (e.g., which may be included as part of the sensor data 365). Based on the IMU data in the sensor data 365, the sensor integration engine 325 may determine that the SLAM system 300 has rotated 15 degrees clockwise from the time a first camera in the cameras 310 acquired or captured a first image to the time a second image was acquired or captured. Based on this determination, the sensor integration engine 325 may identify that a feature depicted at a first location in the first image is expected to appear at a second location in the second image, and that the second location is expected to be a predetermined distance (e.g., a predetermined number of pixels, inches, centimeters, millimeters, or another distance metric) to the left of the first location. The feature tracking engine 320 may consider this expectation when tracking the feature between the first and second images.

[0087] Based on feature tracking performed by feature tracking engine 320 and / or sensor integration performed by sensor integration engine 325, VIO tracker 315 may determine a 3D feature location 373 for a particular feature. 3D feature location 373 may include one or more 3D feature locations and may also be referred to as a 3D feature point. 3D feature location 373 may be a set of coordinates along three different axes perpendicular to each other, such as an X-axis (e.g., horizontally), a Y-axis (e.g., vertically) perpendicular to the X-axis, and a Z-axis (e.g., depthwise) perpendicular to both the X-axis and the Y-axis. VIO tracker 315 may also determine one or more keyframes 370 (hereinafter referred to as keyframes 370) corresponding to the particular feature. A keyframe (from one or more keyframes 370) corresponding to a particular feature may be an image in which the particular feature is clearly depicted. In some examples, a keyframe (from one or more keyframes 370) corresponding to a particular feature may be an image in which the particular feature is clearly depicted. In some examples, a keyframe corresponding to a particular feature can be an image that, when considered by the feature tracking engine 320 and / or the sensor integration engine 325 for determining the 3D feature location 373, reduces the uncertainty of the 3D feature location 373 of the particular feature. In some examples, a keyframe corresponding to a particular feature also includes data associated with the pose 385 of the SLAM system 300 and / or the camera 310 during the capture of the keyframe. In some examples, the VIO tracker 315 can transmit the 3D feature location 373 and / or keyframes 370 corresponding to one or more features to the mapping engine 330. In some examples, the VIO tracker 315 can receive a map slice 375 from the mapping engine 330. The VIO tracker 315 can perform feature extraction on the information within the map slice 375 for feature tracking using the feature tracking engine 320.

[0088] Based on the feature tracking performed by the feature tracking engine 320 and / or the sensor integration performed by the sensor integration engine 325, the VIO tracker 315 can determine a pose 385 of the SLAM system 300 and / or the camera 310 during the capture of each of the images in the sensor data 365. The pose 385 can include the position of the SLAM system 300 and / or the camera 310 in 3D space, such as a set of coordinates along three different axes that are perpendicular to each other (e.g., an X coordinate, a Y coordinate, and a Z coordinate). The pose 385 can include the orientation of the SLAM system 300 and / or the camera 310 in 3D space, such as pitch, roll, yaw, or some combination thereof. In some examples, the VIO tracker 315 can transmit the pose 385 to the relocalization engine 355. In some examples, the VIO tracker 315 can receive the pose 385 from the relocalization engine 355.

[0089] The SLAM system 300 also includes a map building engine 330. The map building engine 330 generates a 3D map of the environment based on the 3D feature locations 373 and / or keyframes 370 received from the VIO tracker 315. The map building engine 330 may include a map densification engine 335, a keyframe remover 340, a bundle adjuster 345, and / or a loop closure detector 350. The map densification engine 335 may perform map densification, in some examples, increasing the number and / or density of 3D coordinates describing the map geometry. The keyframe remover 340 may remove keyframes and / or, in some cases, add keyframes. In some examples, the keyframe remover 340 may remove keyframes 370 corresponding to areas of the map to be updated and / or areas with low corresponding confidence values. In some examples, the bundle adjuster 345 may refine the 3D coordinates describing the scene geometry, parameters of relative motion, and / or optical properties of the image sensor used to generate the frames according to an optimality criterion involving corresponding image projections of all points. The loop closure detector 350 can identify when the SLAM system 300 has returned to a previously mapped area and can use this information to update the map slice and / or reduce uncertainty in certain 3D feature points or other points in the map geometry. The map engine 330 can output a map slice 375 to the VIO tracker 315. The map slice 375 can represent a 3D portion or subset of the map. The map slice 375 can include a map slice 375 representing a new, previously unmapped area of ​​the map. The map slice 375 can also include a map slice 375 representing an update (or modification or revision) to a previously mapped area of ​​the map. The map engine 330 can output map information 380 to the relocalization engine 355. The map information 380 can include at least a portion of the map generated by the map engine 330. The map information 380 can include one or more 3D points that make up the map geometry, such as one or more 3D feature locations 373. The map information 380 can include one or more keyframes 370 corresponding to certain features and certain 3D feature locations 373.

[0090] The SLAM system 300 also includes a relocalization engine 355. The relocalization engine 355 can perform relocalization, for example, when the VIO tracker 315 fails to identify more than a threshold number of features in an image and / or when the VIO tracker 315 loses track of the pose 385 of the SLAM system 300 within the map generated by the mapping engine 330. The relocalization engine 355 can perform relocalization by performing extraction and matching using the extraction and matching engine 360. For example, the extraction and matching engine 360 ​​can extract features from an image captured by the camera 310 of the SLAM system 300 when the SLAM system 300 is in the current pose 385, and can match the extracted features with features depicted in different keyframes 370, identified by the 3D feature localization 373, and / or identified in the map information 380. By matching these extracted features with previously identified features, the relocalization engine 355 can identify that the pose 385 of the SLAM system 300 is the pose 385 at which the previously identified feature was visible to the camera 310 of the SLAM system 300, and is therefore similar to one or more previous poses 385 at which the previously identified feature was visible to the camera 310. In some cases, the relocalization engine 355 can perform relocalization based on a wide baseline map build or the distance between the current camera position and the camera position at which the feature was initially captured. The relocalization engine 355 can receive information about the pose 385 from the VIO tracker 315 (e.g., information about one or more recent poses of the SLAM system 300 and / or camera 310), and the relocalization engine 355 can make its relocalization determination based on this information. Once the relocalization engine 355 relocalizes the SLAM system 300 and / or camera 310 and determines the pose 385 accordingly, the relocalization engine 355 can output the pose 385 to the VIO tracker 315.

[0091] In some examples, the VIO tracker 315 may modify the image in the sensor data 365 before performing feature detection, extraction, and / or tracking on the modified image. For example, the VIO tracker 315 may rescale and / or resample the image. In some examples, rescaling and / or resampling the image may include downscaling, downsampling, subscaling, and / or subsampling the image one or more times. In some examples, the VIO tracker 315 modifying the image may include converting the image from color to grayscale, or from color to black and white, for example, by desaturating the colors in the image, removing certain color channels, reducing the color depth in the image, replacing the colors in the image, or a combination thereof. In some examples, the VIO tracker 315 modifying the image may include the VIO tracker 315 masking certain areas of the image. A dynamic object may include an object whose appearance changes from one image to another. For example, a dynamic object may be an object that moves within an environment, such as a person, a vehicle, or an animal. A dynamic object may be an object whose appearance changes at different times, such as a display screen that may display different things at different times. Dynamic objects can be objects that have a changing appearance based on the pose of the camera 310, such as reflective surfaces, prisms, or mirrored surfaces that reflect, refract, and / or scatter light differently depending on the position of the camera 310 relative to the dynamic object. The VIO tracker 315 can detect dynamic objects using facial detection, facial recognition, facial tracking, object detection, object recognition, object tracking, or a combination thereof. The VIO tracker 315 can detect dynamic objects using one or more artificial intelligence algorithms, one or more trained machine learning models, one or more trained neural networks, or a combination thereof. The VIO tracker 315 can mask one or more dynamic objects in an image by overlaying a mask over an area of ​​the image that includes a depiction of the one or more dynamic objects. The mask can be an opaque color (such as black). The region can be a bounding box having a rectangular or other polygonal shape. The region can be determined on a pixel-by-pixel basis.

[0092] Figure 4 An example of an augmented reality application engine 400 is illustrated. In the illustrated example, the augmented reality application engine 400 includes a simulation engine 405, a rendering engine 410, a main rendering module 415, and an AR rendering module 460. As illustrated, the main rendering module 415 may include an effect rendering engine 420, a post-processing engine 425, and a user interface (UI) rendering engine 430. The AR rendering module 460 may include an AR effect rendering engine 465 and an AR UI rendering engine 470. It should be noted that Figure 4 The components 405 to 470 shown are non-limiting examples provided for purposes of illustration and explanation, and other examples may include those related to Figure 4 More, fewer or different components than those shown.

[0093] In some cases, the augmented reality augmented application engine 400 is included in and / or communicates (wired or wirelessly) with the mobile device 440. In some examples, the augmented reality augmented application engine 400 is included in and / or communicates (wired or wirelessly) with the XR system 450.

[0094] exist Figure 4 In the illustrated example, simulation engine 405 can generate simulations for augmented reality augmented application engine 400. In some cases, the simulation can include, for example, one or more images, one or more videos, one or more character strings (e.g., alphanumeric characters, numbers, text, Unicode characters, symbols, and / or icons), one or more two-dimensional (2D) shapes (e.g., circles, ellipses, squares, rectangles, triangles, other polygons, rounded polygons with one or more rounded corners, portions thereof, or combinations thereof), one or more three-dimensional (3D) shapes (e.g., spheres, cylinders, cubes, pyramids, triangular prisms, rectangular prisms, tetrahedrons, other polyhedrons, rounded polyhedrons with one or more rounded edges and / or rounded corners, portions thereof, or combinations thereof), shape textures, bump mapping of the shapes, lighting effects, or combinations thereof. In some examples, the simulation can include at least a portion of an environment. The environment can be a real-world environment, a virtual environment, and / or a hybrid environment including elements of a real-world environment and a virtual environment.

[0095] In some cases, the simulation generated by the simulation engine 405 can be dynamic. For example, the simulation engine 405 can update the simulation based on different triggers, including but not limited to physical contact, sound, gestures, input signals, the passage of time, and / or any combination thereof. As used herein, at a particular moment, the application state of the augmented reality enhancement application engine 400 may include any information associated with the following: the simulation engine 405, the rendering engine 410, the main rendering module 415, the effect rendering engine 420, the post-processing engine 425, the UI rendering engine 430, the AR rendering module 460, the AR effect rendering engine 465, the AR UI rendering engine 470, input to the augmented reality enhancement application engine 400, output from the augmented reality enhancement application engine 400, and / or any combination thereof.

[0096] As illustrated, simulation engine 405 can obtain mobile device input 441 from mobile device 440. In some cases, simulation engine 405 can obtain XR system input 451 from XR system 450. Mobile device input 441 and / or XR system input 451 can include, for example, user input through a user interface of an application displayed on a display of mobile device 440, input from an input device (e.g., Figure 2input device 208), one or more sensors (e.g., Figure 2 In some cases, the simulation engine 405 can update the application state of the augmented reality enhancement application engine 400 based on the mobile device input 441, the XR system input 451, and / or any combination thereof.

[0097] exist Figure 4 In the illustrative example of , the rendering engine 410 may obtain application state information from the simulation engine 405. In some cases, the rendering engine 410 may determine the portion of the application state information that will be rendered by a display that can be used by the augmented reality enhancement application engine 400. For example, the rendering engine 410 may determine whether a connection (wired or wireless) has been established between the XR system 450 and the mobile device 440. In some cases, the rendering engine 410 may determine the application state information that will be rendered by the main rendering module 415 and the AR rendering module 460. In some cases, the rendering engine 410 may determine that the XR system 450 is not connected (wired or wirelessly) to the mobile device 440. In some cases, the rendering engine 410 may determine the application state information of the main rendering module 415 and forgo determining the application state information that will be rendered by the AR rendering module 460 that will not be displayed. Thus, the rendering engine 410 may facilitate an adaptive rendering configuration for the augmented reality enhancement application engine 400 based on the availability and / or type of available displays. In some implementations, the following may be excluded: Figure 4 In one illustrative example, the main rendering module 415 and / or the AR rendering module 460 may include at least a portion of the functionality of the rendering engine 410 described above.

[0098] The main rendering module 415 may include an effects rendering engine 420, a post-processing engine 425, and a UI rendering engine 430. In some cases, the main rendering module 415 may render image frames configured for display on a display of a mobile device 440. As illustrated, the main rendering module 415 may output the generated image frames (e.g., media content) for display on the display of the mobile device 440. In some cases, the effects rendering information may render application state information generated by the simulation engine 405. For example, the effects rendering engine may generate a 2D projection of a portion of a 3D environment included in the application state information. For example, the rendering engine 420 may generate a perspective projection of the 3D environment using a virtual camera. In some cases, the application state information may include the pose of the virtual camera within the environment. In some cases, the effects rendering engine 420 may generate additional visual effects not included in the 3D environment. For example, the rendering engine 420 may apply texture mapping to enhance the visual appearance of the effects generated by 420. In some cases, the rendering engine 420 may exclude portions of the application state information specified by the rendering engine 410 for the AR rendering module 460. For example, the main rendering module 415 may exclude effects present in the simulated environment.

[0099] In some cases, post-processing engine 425 may provide additional processing to the rendered effects generated by effects rendering engine 420. For example, post-processing engine 425 may perform scaling, image smoothing, z-buffering, contrast enhancement, gamma, color mapping, any other image processing, and / or any combination thereof.

[0100] In some implementations, the UI rendering engine 430 may render a UI. In some cases, the user interface may provide application state information in addition to effects rendered based on the application environment (e.g., a 3D environment). In some cases, the UI may be generated as an overlay on a portion of the image frame output by the post-processing engine 425.

[0101] The AR rendering module 460 may include an AR effect rendering engine 465 and an AR UI rendering engine 470. In some cases, the AR effect rendering engine 465 may render the application state information generated by the simulation engine 405. For example, the AR effect rendering engine 465 may generate a 2D projection of a 3D environment included in the application state information. In some cases, the AR effect rendering engine 465 may generate an effect that appears to protrude from the display surface of the display of the mobile device 440.

[0102] In some cases, the display of XR system 450 may have different display parameters than the display of mobile device 440 (e.g., different resolution, frame rate, aspect ratio, and / or any other display parameters). In some cases, display parameters may also vary between different types of output devices (e.g., different HMD models, other XR systems, etc.). Therefore, rendering display data for 450 using AR rendering module 460 may affect the performance of main rendering module 415 (e.g., by consuming computing resources such as GPU, CPU, memory, etc.). In some cases, including AR rendering module 460 within augmented reality augmented application engine 400 may require periodic updates to provide compatibility with different devices.

[0103] Figure 5 An example of a primary application engine 500 and a secondary application engine 560 that can provide augmented reality enhancements to the primary application engine 500 is illustrated. Figure 5 In the illustrated example, the primary application engine 500 includes a simulation engine 505, a rendering engine 515, an encoding engine 520, and a communication engine 525. In the illustrated example, the secondary application engine 560 includes a tracking engine 565 (e.g., Figure 2 XR Engine 220, Figure 3 VIO tracker 315), AR rendering engine 575, decoding engine 580 and communication engine 585. As illustrated, the main application engine 500 and the auxiliary application engine 560 can communicate via a (wired or wireless) communication link 530. It should be noted that Figure 5 The components 505-525 shown in the main application engine 500 are non-limiting examples provided for purposes of illustration and explanation, and other examples may include Figure 5 More, fewer, or different components than those shown. Similarly, it should be noted that Figure 5 The components 565-585 shown in the auxiliary application engine 560 are non-limiting examples provided for illustrative and explanatory purposes, and other examples may include the following: Figure 5 More, fewer or different components than those shown.

[0104] exist Figure 5In the illustrated example, the simulation engine 505 of the main application engine 500 can generate a simulation for an application on the mobile device 540. In some cases, the simulation can include, for example, one or more images, one or more videos, one or more character strings (e.g., alphanumeric characters, numbers, text, Unicode characters, symbols, and / or icons), one or more two-dimensional (2D) shapes (e.g., circles, ellipses, squares, rectangles, triangles, other polygons, rounded polygons with one or more rounded corners, portions thereof, or combinations thereof), one or more three-dimensional (3D) shapes (e.g., spheres, cylinders, cubes, pyramids, triangular prisms, rectangular prisms, tetrahedrons, other polyhedrons, rounded polyhedrons with one or more rounded edges and / or rounded corners, portions thereof, or combinations thereof), shape textures, bump mapping of the shapes, lighting effects, or combinations thereof. In some examples, the simulation can include at least a portion of an environment. The environment can be a real-world environment, a virtual environment, and / or a hybrid environment that includes elements of a real-world environment and a virtual environment.

[0105] In some cases, the simulation generated by the simulation engine 505 can be dynamic. For example, the simulation engine 505 can update the simulation based on different triggers, including but not limited to physical contact, sound, gestures, input signals, the passage of time, and / or any combination thereof. As used herein, at a particular moment in time, the application state of the main application engine 500 may include any information associated with the simulation engine 505, the effects rendering engine 515, the communication engine 525, and / or any combination thereof.

[0106] The rendering engine 515 may correspond to Figure 4 The main rendering module 415 of FIG. 5 and performs similar functions. For example, the rendering engine 515 may include a module for effect rendering (e.g., Figure 4 Rendering engine 420), modules for post-processing (e.g., Figure 4 post-processing engine 425) and / or a module for UI rendering (e.g., Figure 4 UI rendering engine 430).

[0107] The communication engine 525 of the primary application engine 500 and the communication engine 585 of the secondary application engine 560 can communicate via a communication link 530. In some cases, the communication link 530 can be bidirectional. In some examples, the communication engine 525 can send application state information (e.g., from the simulation engine 505) to the communication engine 585 of 560. In some cases, the application state information can include information that can be used to generate AR effects. In some examples, the application state information can include data that can be used by the secondary application engine 560 to generate an AR UI. In some cases, the communication engine 525 can also send input received from the mobile device 540 to the communication engine 585 via the communication link 530. In some cases, the communication engine 585 of the secondary application engine 560 can send pose information, connectivity status, user input, etc. to the communication engine 525 of the primary application engine 500. The communication engines 525 and 585 can also send and / or receive synchronization signals for synchronizing the display between the display of the mobile device 540 and the display of the HMD 550. The examples of communication between the communication engine 525 and the communication engine 585 provided herein are non-limiting and are provided as examples. In some cases, more, less, and / or different information may be communicated over the communication link 530 without departing from the scope of this disclosure. While the HMD 550 is used herein as an illustrative example of an XR device, the systems and techniques may be used for any type of XR device, such as AR, VR, or MR glasses.

[0108] With reference to the auxiliary application engine 560, the tracking engine 565 may use the information generated by the sensor (e.g., Figure 2 Image sensor 202, accelerometer 204, gyroscope 206, Figure 3 Tracking (e.g., SLAM, VIO, etc.) is performed using information captured by one or more sensors 305, cameras 310, etc. of the mobile device 540. In some cases, the tracking engine 565 may determine the pose of the mobile device 540, the pose of the HMD 550, a map of the environment, etc. In some aspects, the tracking engine 565 may determine the outline of the display of the mobile device 540. In some cases, the outline of the display of the mobile device 540 may include a boundary. In some cases, the pose of the mobile device 540 and / or the outline and / or boundary of the display of the mobile device 540 may be output to the AR rendering engine 575 to provide a target for displaying AR information (e.g., AR effects, AR UI) on the display of the HMD 550.

[0109] The AR rendering module 460 may be similar to Figure 4 For example, in some implementations, the HMD 550 may include an AR effect rendering engine (e.g., Figure 4AR effect rendering engine 465) and / or AR UI rendering engine (e.g., Figure 4 540 ). In some cases, the AR rendering engine 575 may output AR media content to the HMD 550 having different display parameters (e.g., a different resolution, frame rate, aspect ratio, and / or any other display parameters) than the media content output from the rendering engine 515 to the mobile device 540. In some cases, by dividing the rendering functionality between the primary application engine 500 and the secondary application engine 560, computing resources for providing an AR-enhanced application experience may be shared among computing resources of multiple devices, such as the mobile device 540 and the HMD 550. Furthermore, providing a standalone AR rendering engine 575 in the secondary application engine 560 may simplify the development of the primary application engine 500. For example, the rendering engine 515 of the primary application engine 500 may not need to maintain compatibility with a variety of different mobile devices having different display configurations.

[0110] In some cases, to allow HMD 550 to be wearable, HMD 550 may be relatively battery- and processing-constrained compared to mobile device 540. To reduce processing requirements on HMD 550, images for display by HMD 550 may be rendered by mobile device 540 and transmitted to HMD 550 via communication link 530. In some cases, the HMD may concurrently receive multiple images for display to the user. For example, rendering engine 515 of mobile device 540 may render a left-eye image, a right-eye image, and, in some cases, provide depth information. Depth information may include information indicating the distance of a point in a scene (e.g., a point corresponding to the surface of an object) from a viewpoint (such as a camera viewpoint). In some cases, depth information may be inferred based on, for example, the difference between the left-eye and right-eye images received by HMD 550. In some cases, the depth information can be used to warp (e.g., apply a displacement vector to) portions of an image to help adjust for the movement of objects that may have moved independently of a camera (such as a camera on the HMD 550) between the time the image is rendered by the rendering engine 515 and the time the image is received by the HMD 550. The rendered image can be in any known image or video format. In some cases, the image can include objects that will be overlaid on the environment visible through the HMD 550. The encoding engine 520 can encode the rendered image to reduce the size of the image for transmission. The encoded image can be transmitted to the HMD 550 via the communication engine 525 and the communication link 530.

[0111] The HMD may receive encoded images via the communication engine 585. These received images may then be decoded, for example, by the decoding engine 580. In some cases, there may be delays (e.g., display latency) introduced by the rendering, encoding, transmission, reception, and decoding processes, and during this display latency, the user may, for example, move the HMD 550. This movement may not be accounted for by the images rendered by the rendering engine 515, and any objects in the rendered images may appear in different locations than intended due to this movement. To account for potential movement of the HMD 550, the AR rendering engine 575 may warp the received images based on the pose and / or tracking information from the tracking engine 565 describing the movement of the HMD 550. Consequently, the rendering of the final image displayed to the user may be split across multiple devices (e.g., the mobile device 540 and the HMD 550). In some cases, the techniques discussed herein may be applicable to other scenarios where there is sufficient lag between when images are rendered and when those images are displayed for noticeable movement to occur.

[0112] In some cases, there may be an opportunity to convert images into images because the images may be encoded on a device separate from the HMD 550 (such as the mobile device 540) and then transferred to the HMD 550 for additional processing. Figure 5 The XR system shown is extended to include other devices. For example, XR systems may often be located near other devices whose computing resources are underutilized. Being able to use some of the underutilized computing resources for rendering may be beneficial. However, neighboring devices with underutilized computing resources are not obligated to help. To encourage neighboring devices to provide processing resources, incentives for computation offloading in a distributed split rendering architecture may be useful.

[0113] Figure 6 is a diagram 600 illustrating a technique for stimulating computation offloading of a distributed split rendering architecture for devices on the same network according to aspects of the present disclosure. For example, an XR device 602 may be an XR system such as Figure 5 The XR device may be part of a mobile device 540 and may be communicatively coupled to a network, such as a cellular communication network. Although discussed in the context of a cellular communication network, it will be understood that the techniques discussed herein are applicable to any communication network.

[0114] Neighboring device 604 may be a device such as a mobile device or handset (e.g., a mobile phone or so-called "smartphone" or other mobile device), a wearable device, a personal computer, a laptop computer, a server computer, or other device with available computing resources (e.g., a processor and / or memory capacity that is not in use). Neighboring device 604 is also communicatively coupled to a network. In some cases, neighboring device 604 is not participating in an XR session, and neighboring device 604 may indicate to network 606 that neighboring device 604 is offering to provide a computing offload session with XR device 602. Service 608. In some cases, network 606 may be a wireless carrier's network that serves XR device 602 and neighboring device 604. In some cases, one or more offload servers may be hosted on network 606 (e.g., hosted on the core network of a cellular network), and the offload servers may coordinate and perform the functions discussed in the context of network 606, as discussed below. In some cases, neighboring device 604 may report information regarding computing resources available to network 606. This computing resource information may indicate the number of CPUs, GPUs, NPUs, and / or the amount of memory available on neighboring device 604. In some cases, the information about computing resources may also indicate the availability of computing resources. For example, neighboring device 604 may indicate that a certain percentage of processor (e.g., CPU, GPU, NPU) cycles are available for offloading. In some cases, neighboring device 604 may update network 606 regarding available computing resources. For example, when the amount of available computing resources changes significantly, neighboring device 604 may update network 606 based on a schedule, periodically, or the like.

[0115] In some cases, the XR device 602 may participate in or begin an XR session or perform another computing task that can be easily divided into discrete parts. In this example, the XR device 602 may use split rendering for the XR session. The XR device 602 may determine that it can offload a computing task, such as a portion of the rendering. The XR device 602 may then perform neighbor discovery 610 to locate nearby computing devices, such as neighboring device 604, that are providing computing offload services. In some cases, the XR device 602 may send (e.g., broadcast) a neighbor discovery message (also referred to herein as a "discovery message") to discover nearby devices. In some cases, the neighbor discovery message may be sent using the same or a different radio access technology (RAT). As an example, the XR device 602 may attempt to establish a sidelink connection using a sidelink neighbor discovery mechanism on a physical sidelink discovery channel (PSDCH). In some cases, the XR device 602 may send a sidelink discovery message, such as a sidelink synchronization block (S-SSB). Another device, such as neighboring device 604, can detect a sidelink discovery message (e.g., an S-SSB) and use it to establish a direct connection with XR device 602. In examples where the sidelink discovery message includes an S-SSB, the S-SSB can include a sidelink primary synchronization signal (SPSS) and a sidelink secondary synchronization signal (SSSS), and in some cases, other signals. In some cases, XR device 602 can indicate that it is requesting a compute offload service. For example, a request for a compute offload service can be indicated in one or more fields of the S-SSB.

[0116] In some cases, if other devices are providing computational offload services, other devices (such as neighboring device 604) may respond to the neighbor discovery message. For example, by causing other devices providing computational offload services to reply, the processing load on the XR device 602 can be reduced. In some cases, the response to the neighbor discovery message includes an identifier that can identify the neighboring device 604 to the network 606. Based on the response to the neighbor discovery message, the XR device 602 may send an offload request 612 to the network 606. In some cases, the offload request 612 may indicate which computational modules can be offloaded to another device, such as the neighboring device 604. The indication of which computational modules can be transmitted along with an estimate of the amount of computational offload services required to process the computational modules. In some cases, the computational modules may be discrete processing tasks that can be performed on a dataset. As an example, the XR device 602 may include a subset of computational modules in a motion-to-photon loop among the offloaded computational modules. The motion-to-photon loop involves capturing the user's motion and, based on the captured motion and possibly the user's surrounding environment, updating a 3D model of a virtual object that reacts to the captured motion and / or environment. The updated 3D model can then be rendered (e.g., rasterized) as an image of the virtual object for display, as if the virtual object were in the environment surrounding the user. Portions of this motion-to-photon loop can be offloaded to neighboring devices 604 for processing. Examples of computational modules from the motion-to-photon loop can include 3D to 2D mapping, pose correction, synthesis, object recognition, semantic segmentation, etc. In some cases, the offload request 612 can also include identifiers received from other devices in response to neighbor discovery messages.

[0117] Based on the offload request 612 received from the XR device 602, the network 606 may select one or more other devices (such as the neighboring device 604) to perform the compute offload service. In some cases, when selecting the one or more other devices, the network 606 may consider reported information regarding the compute resources available to the other devices (such as the neighboring device 604). For example, the network may select a device with more available compute resources. When selecting the one or more other devices, the network 606 may also consider the compute modules indicated by the XR device 602 in the offload request 612. For example, the compute modules that can be offloaded may include an indication of compute resources that may be more suitable for processing the compute modules. As a more specific example, the compute module may indicate that the compute module is better offloaded to a device with available GPU cycles rather than another device with available CPU cycles. In some cases, the neighboring device 604 may also indicate, for example in its offer to provide the compute offload service 608, the types of compute modules that the neighboring device 604 may be capable of processing. The network 606 can compare the type of computing module indicated by the XR device 602 in the offload request 612 with the computing resources available on the neighboring device 604 (e.g., the indicated type of computing module that the neighboring device 604 may be able to handle). In some cases, the network 606 can allocate the computing module indicated by the XR device 602 in the offload request 612 to the selected one or more other devices, such as the neighboring device.

[0118] The network 606 may, for example, set up offloading 614 by transmitting to the neighboring device 604 an indication that the neighboring device 604 has been (or is) selected to perform computation offloading. In some cases, the network 606 may also transmit an indication of the computation module selected for execution by the neighboring device 604. In some cases, the indication of the computation module may provide, for example, code to be executed to perform the offloaded processing. In other cases, the code to be executed to perform the offloaded processing may be provided by the XR device 602 or available on the neighboring device 604, for example, as part of a defined standard or procedure for offloading processing. As part of setting up offloading 614, the network 606 may also transmit to the XR device 602 an indication of the selected other devices, such as the neighboring device 604. In some cases, the network 606 may also indicate to the XR device 602 which computation modules may be offloaded to the selected other devices, such as the neighboring device 604.

[0119] After the offload is set up 614, the XR device 602 may transmit input data to the neighboring device 604 to be processed 616 by the offload service. In some cases, the XR device 602 may transmit the input data directly to the neighboring device 604 via a sidelink connection. In some cases, such as when the offload service is being used for an XR application, the input data may include images, video, audio, haptics, pose information, etc. In some cases, the input data for the offload processing and the output from the neighboring device may be controlled by the network 606. For security and privacy, the input data and output may be encrypted, and the processing on the neighboring device may be performed in a container, a trusted execution environment, a secure enclave, a virtualized environment, or other similar environment that is protected from access by local processes that may be executing on the neighboring device 604. In some cases, the XR device 602 may transmit the input data for processing 616 to the network 606 via the offload service, which may then forward the input data to the neighboring device 604.

[0120] After processing at least a portion of the input data, the neighboring device 604 can send processed data results 618 back to the XR device 602. For example, the neighboring device 604 can send a rendered image, pose information, semantic segmentation information of the image, etc. to the XR device 602. In some cases, the neighboring device 604 can begin sending the processed data results 618 to the XR device before completing processing of all the input data. In some cases, the neighboring device 604 can send the processed data results 618 to the network 606, and the network 606 can then forward the processed data results to the XR device 602.

[0121] In some cases, after receiving at least some of the processed data results 618, the XR device 602 may report to the network 606 the amount 620 of completed offload processing provided by the neighboring device 604. In some cases, to help incentivize offload processing, the network 606 may apply a credit to the neighboring device 604 for performing the offload service. The network 606 may perform charging, account crediting, and debiting 622. For example, the network 606 may charge the XR device 602 for the offload service received. In some cases, the debit charged to the device for the offload service received and the credit applied to the device providing the offload service may be used as a criterion for qualifying the device for premium service (e.g., offload service, preferential treatment for some future service, etc.) in the future.

[0122] Figure 7700 is a diagram illustrating techniques for stimulating computational offload for a distributed split rendering architecture for devices on different networks, in accordance with aspects of the present disclosure. In some cases, the techniques for stimulating computational offload for a distributed split rendering architecture for devices on different networks are similar to techniques for stimulating computational offload for a distributed split rendering architecture for devices on the same network. In diagram 700, XR device 702 may be on a different communication network than neighboring device 704. For example, XR device 702 may be served by a different wireless carrier than the neighboring device.

[0123] The neighboring device 704 may indicate to one or more offload servers 706 that the neighboring device 704 proposes to provide a computation offload service 708. In some cases, the one or more offload servers 706 may be hosted on one or more operator networks or a public network (such as the Internet). The offload servers 706 may be configured to operate across different communication networks (e.g., network independent). In some cases, the proposal to provide a computation offload service 708 may be similar to the proposal to provide a computation offload service 708. Figure 6 6. In some cases, the identifiers used for the neighboring device 704 and the XR device 702 can be separate identifiers from the identifiers used to identify the neighboring device 704 and the XR device 702 on their respective networks. In some cases, the neighboring device 704 and the XR device 702 can use different RATs. For example, the XR device 702 can communicate with the offload server 706 using a cellular connection, possibly cascaded by (or routed through) the internet, while the neighboring device 704 can communicate with the offload server 706 using Wi-Fi, possibly cascaded by (or routed through) the internet.

[0124] XR device 702 may perform neighbor discovery 710 to locate nearby computing devices that are providing computation offload services, such as neighboring device 704. In some cases, neighbor discovery 710 may be performed in a manner similar to Figure 6 In some cases, neighbor discovery 710 may be performed across more frequencies and / or frequency bands, and to account for different operating frequencies / frequency bands between different wireless carriers. In other cases, neighbor discovery 710 may be performed using a different RAT than the RAT used to communicate with the offload server 706. For example, the XR device 702 may communicate with the offload server using cellular communication, and the XR device 702 may use Wi-Fi for neighbor discovery. In the case where cellular communication (e.g., sidelink) is used for neighbor discovery 710, the cellular core networks for the different cellular networks may be configured to allow and coordinate sidelink neighbor discovery.

[0125] In some cases, the XR device 702 may be configured as described above with respect to Figure 6 The uninstall request 712 is sent to the uninstall server 706 in a substantially similar manner as described above with respect to the uninstall request 612. The uninstall server 706 may send an uninstall request 712 to the uninstall server 706 in a substantially similar manner as described above with respect to the uninstall request 612. Figure 6 Offload 714 is set up in substantially the same manner as described for offload 614 in network 606 .

[0126] After setting up the offload 714, the XR device 702 may transmit input data 716 to be processed by the offload service to the neighboring device 704, and the neighboring device 704 may send processed data results 718 back to the XR device 702. In some cases, the input data and the processing of the input data may be similar to those described above with respect to Figure 6 7. In some cases, the XR device 702 may connect to the neighboring device 704 and send input data to the neighboring device via a different RAT than the RAT used to communicate with the offload server 706. For example, the XR device 702 and the neighboring device 704 may communicate with the offload server 706 using cellular communication. However, the XR device 702 and the neighboring device 704 may use another RAT (such as Wi-Fi, ultra-wideband (UWB), etc.) to send data 716 for offload processing. Similarly, the neighboring device may use the other RAT (e.g., Wi-Fi, UWB, etc.) to send processed data results 718 to the XR device 702. In some cases, the XR device 702 may transmit the input data 716 to be processed by the offload service to the offload server 706. The offload server 706 may then forward the input data to the neighboring device 704. In some cases, the neighboring device 704 may send the processed data results 718 to the offload server 706 , and the offload server 706 may then forward the processed data results 718 to the XR device 702 .

[0127] The XR device 702 can be used in the same manner as above with respect to Figure 6 The amount of offload processing completed 720 is reported to the offload server 706 in substantially the same manner as described above for the XR device 602 reporting the amount of offload processing completed 620 in FIG. The offload server 706 can perform charging, account crediting, and debiting 722. For example, the offload server 706 can charge the XR device 702 for the offload services received, and the offload server 706 can credit the neighboring device 704 for performing the offload services. In some cases, the offload server 706 can apply a credit to the neighboring device for providing the offload service, which can be used as a criterion for the device to qualify for premium services in the future (e.g., offload services, preferential treatment for future services, etc.).

[0128] Figure 8is a flow chart illustrating a process 800 for distributed processing according to aspects of the present disclosure. Process 800 may be performed by a computing device (or apparatus or system) or a component of a computing device (e.g., a chipset, a codec, etc.), such as Figure 2 XR System 200, Figure 3 SLAM system, Figure 4 Augmented reality enhanced application engine 400, Figure 5 The main application engine 500, Figure 5 Auxiliary application engine 560, Figure 13 The computing system 1300 of the present invention, any combination thereof, and / or other computing devices or systems. The computing device can be a mobile device (e.g., a mobile phone), a network-connected wearable device such as a watch, an extended reality (XR) device such as a virtual reality (VR) device or an augmented reality (AR) device, a vehicle or a component or system of a vehicle, or other type of computing device. The operations of process 800 can be implemented as a processor (e.g., Figure 2 One or more computing components in computing components 210, Figures 2 to 5 One or more of the engines, Figure 13 Software components executed and run on the processor 1310 and / or other processors).

[0129] At block 802, a computing device (or a component thereof) may send a discovery message. In some cases, the computing device may be an XR device, such as Figure 6 XR device 602 or Figure 7 XR device 702. At block 804, the computing device (or a component thereof) may receive a response to the discovery message. In some cases, the response includes neighboring devices (e.g., Figure 6 Neighboring device 604, Figure 7 In some cases, the discovery message includes a sidelink discovery message.

[0130] At block 806, the computing device (or a component thereof) may send a request to an offload server (e.g., Figure 6 Network 606 or Figure 7 The offload server 706 sends an offload request. In some cases, the offload request includes an identifier of the neighboring device. In some cases, the offload request includes an indication of the computing module to be offloaded. In some cases, the offload request includes an estimate of the amount of computing offload service required to process the computing module. In some cases, the offload server is hosted on a core network. In some cases, the device and the neighboring device are connected to the core network. In some cases, the offload server is hosted on the Internet. In some cases, the offload server may be hosted on another network accessible via the Internet.

[0131] At block 808, the computing device (or a component thereof) may receive an indication from the offload server that a neighboring device has been selected to perform offload processing. The computing device (or a component thereof) may establish a sidelink connection with the neighboring device. The computing device (or a component thereof) may send data for processing via the sidelink connection. The computing device (or a component thereof) may receive the processed data via the sidelink connection.

[0132] At block 810, the computing device (or its components) may send data for processing to a neighboring device. The computing device (or its components) may communicate with an offload server using a first radio access technology. The computing device (or its components) may communicate with the neighboring device using a second radio access technology. At block 812, the computing device (or its components) may receive the processed data from the neighboring device. At block 814, the computing device (or its components) may send an indication of the amount of offload processing completed.

[0133] Figure 9 900 is a flow chart illustrating a process 900 for distributed processing according to aspects of the present disclosure. Process 900 may be performed by a computing device (or apparatus) or a component of a computing device (e.g., a chipset, codec, etc.). The computing device may be a mobile device (e.g., a mobile phone), a network-connected wearable device such as a watch, an extended reality (XR) device such as a virtual reality (VR) device or an augmented reality (AR) device, a vehicle or a component or system of a vehicle, or other type of computing device. The operations of process 900 may be implemented as software components that execute and run on one or more processors.

[0134] At block 902, a computing device (or a component thereof) may send a request to an offload server (e.g., Figure 6 Network 606 or Figure 7 The offloading server 706) sends an indication that the first device can provide computing offloading services. In some cases, the computing device can be a neighboring device, such as Figure 6 Neighboring device 604 or Figure 6 The neighboring device 704. In some cases, the indication that the first device can provide a computation offload service includes an indication of a computational resource of the first device that is available for use.

[0135] At block 904, the computing device (or a component thereof) may receive a request from a second device (e.g., Figure 6 XR device 602 or Figure 7The computing device (or its component) may establish a sidelink connection with the second device. The computing device (or its component) may receive data via the sidelink connection. The computing device (or its component) may transmit processed data via the sidelink connection. At block 906, the computing device (or its component) may transmit a response to the discovery message.

[0136] At block 908, the computing device (or a component thereof) may receive an indication from the offload server that the first device is selected to provide computing offload services. In some cases, the indication that the first device is selected to provide computing offload services includes an indication of a computing module for processing.

[0137] At block 910, a computing device (or a component thereof) may receive data for processing from a second device. The computing device (or a component thereof) may communicate with an offload server using a first radio access technology. The computing device (or a component thereof) may communicate with the second device using a second radio access technology.

[0138] At block 912, the computing device (or a component thereof) may process the data to obtain processed data.The computing device (or a component thereof) may process the data by processing the data based on instructions to the computing module for processing.

[0139] At block 914, the computing device (or a component thereof) may send the processed data to the second device.The computing device (or a component thereof) may receive a credit for processing the data.

[0140] Figure 10 1000 is a flow chart illustrating a process 1000 for distributed processing according to aspects of the present disclosure. Process 1000 may be performed by a computing device (or apparatus) or a component of a computing device (e.g., a chipset, codec, etc.). The computing device may be a mobile device (e.g., a mobile phone), a network-connected wearable device such as a watch, an extended reality (XR) device such as a virtual reality (VR) device or an augmented reality (AR) device, a vehicle or a component or system of a vehicle, or other type of computing device. The operations of process 1000 may be implemented as software components that execute and run on one or more processors.

[0141] At block 1002, a computing device (or a component thereof) may receive a request from a first device (e.g., Figure 6 XR device 602 or Figure 7 The XR device 702 receives an uninstall request, which includes the second device (eg, Figure 6 Neighboring device 604 or Figure 7The offload request includes an identifier of a neighboring device 704. In some cases, the offload request includes an indication of a computing module to be used for offloading. In some cases, the computing device is hosted on a core network. In some cases, the first device and the second device are connected to the core network. In some cases, the computing device is hosted on the Internet. In some cases, the computing device may be hosted on another network accessible via the Internet.

[0142] At block 1004, the computing device (or a component thereof) may receive an indication from a second device that the second device can provide a computational offload service. In some cases, the indication that the second device can provide a computational offload service includes an indication of computational resources available for use on the second device. The computing device (or a component thereof) may communicate with the first device using a first radio access technology. The computing device (or a component thereof) may communicate with the second device using a second radio access technology.

[0143] At block 1006, the computing device (or a component thereof) may select a second device to perform the computation offload service. The computing device (or a component thereof) may select the second device by identifying the second device based on an identifier of the second device. The computing device (or a component thereof) may select the second device based on a comparison between the indicated computation modules for offloading and the indicated available computation resources.

[0144] At block 1008, the computing device (or a component thereof) may send an indication that the second device has been selected to provide computing offload services. The computing device (or a component thereof) may send the indication that the second device has been selected by sending an indication of a computing module for offloading to the second device. At block 1010, the computing device (or a component thereof) may receive an indication of the amount of offload processing completed from the first device.

[0145] At block 1012, the computing device (or a component thereof) may charge a first account associated with the first device based on the indication of the amount of offload processing completed. At block 1014, the computing device (or a component thereof) may credit a second account associated with the second device based on the indication of the amount of offload processing completed.

[0146] Figure 11A1100 is a perspective view illustrating a head-mounted display (HMD) 1110 according to some examples. HMD 1110 may be, for example, an augmented reality (AR) headset, a virtual reality (VR) headset, a mixed reality (MR) headset, an extended reality (XR) headset, or some combination thereof. HMD 1110 may be an example of XR system 200, SLAM system 300, or a combination thereof. HMD 1110 includes a first camera 1130A and a second camera 1130B along the front of HMD 1110. First camera 1130A and second camera 1130B may be both of one or more cameras 310. In some examples, HMD 1110 may have only a single camera. In some examples, HMD 1110 may also include one or more additional cameras in addition to first camera 1130A and second camera 1130B. In some examples, HMD 1110 may include one or more additional sensors in addition to first camera 1130A and second camera 1130B.

[0147] Figure 11B is an example based on some examples Figure 11A A perspective view 1130 of a head-mounted display (HMD) 1110 being worn by a user 1120. User 1120 wears HMD 1110 on their head, above their eyes. HMD 1110 can capture images using a first camera 1130A and a second camera 1130B. In some examples, HMD 1110 displays one or more display images based on the images captured by first camera 1130A and second camera 1130B, toward the eyes of user 1120. The display images can provide a stereoscopic view of the environment, in some cases with superimposed information and / or other modifications. For example, HMD 1110 can display a first display image based on the image captured by first camera 1130A to the right eye of user 1120. HMD 1110 can also display a second display image based on the image captured by second camera 1130B to the left eye of user 1120. For example, the HMD 1110 may provide overlay information in the displayed image that is overlaid on the images captured by the first camera 1130A and the second camera 1130B.

[0148] HMD 1110 may not include wheels, propellers, or other forms of transport of its own. Instead, HMD 1110 relies on the movement of user 1120 to move HMD 1110 around the environment. In some cases, such as when HMD 1110 is a VR headset, the environment may be fully or partially virtual. If the environment is at least partially virtual, movement through the virtual environment may also be virtual. For example, movement through the virtual environment may be controlled by input devices 208. Movement actuators may include any such input devices 208. Movement through the virtual environment may not require wheels, propellers, legs, or any other form of transport. Even if the environment is virtual, SLAM technology may still be valuable because the virtual environment can be mapped and / or generated by devices other than HMD 1110, such as a remote server or console associated with a video game or video game platform. In some cases, feature tracking and / or SLAM can even be performed within the virtual environment by a vehicle or other device that has its own physical transport system that allows it to physically move around the physical environment. For example, SLAM can be performed in a virtual environment to test whether the SLAM system 300 is working properly without wasting time or energy on movement and without wearing out the physical conveyor system.

[0149] Figure 12A1200 is a perspective view of a front surface 1255 of a mobile device 1250 illustrating the use of one or more front-facing cameras 1230A-B to perform features described herein, including, for example, feature tracking and / or visual simultaneous localization and mapping (VSLAM), according to some examples. Mobile device 1250 can be, for example, a cellular phone, a satellite phone, a portable game console, a music player, a fitness tracking device, a wearable device, a wireless communication device, a laptop, a mobile device, any other type of computing device or computing system discussed herein, or a combination thereof. Front surface 1255 of mobile device 1250 includes display 1245. Front surface 1255 of mobile device 1250 includes first camera 1230A and second camera 1230B. First camera 1230A and second camera 1230B are illustrated in a bezel surrounding display 1245 on front surface 1255 of mobile device 1250. In some examples, first camera 1230A and second camera 1230B can be positioned in a notch or cutout cut out of display screen 1245 on front surface 1255 of mobile device 1250. In some examples, first camera 1230A and second camera 1230B can be under-display cameras positioned between display screen 1245 and the rest of mobile device 1250, such that light passes through a portion of display screen 1245 before reaching first camera 1230A and second camera 1230B. First camera 1230A and second camera 1230B in perspective view 1200 are front-facing cameras. First camera 1230A and second camera 1230B face in a direction perpendicular to the planar surface of front surface 1255 of mobile device 1250. First camera 1230A and second camera 1230B can be both of one or more cameras 310. In some examples, front surface 1255 of mobile device 1250 can have only a single camera. In some examples, the mobile device 1250 can include one or more additional cameras in addition to the first camera 1230A and the second camera 1230B. In some examples, the mobile device 1250 can include one or more additional sensors in addition to the first camera 1230A and the second camera 1230B.

[0150] Figure 12B1290 is a perspective view illustrating a rear surface 1265 of a mobile device 1250. Mobile device 1250 includes a third camera 1230C and a fourth camera 1230D on the rear surface 1265 of mobile device 1250. Third camera 1230C and fourth camera 1230D of perspective view 1290 are rear-facing. Third camera 1230C and fourth camera 1230D face a direction perpendicular to the planar surface of rear surface 1265 of mobile device 1250. Although rear surface 1265 of mobile device 1250 does not have a display screen 1245 as illustrated in perspective view 1290, in some examples, rear surface 1265 of mobile device 1250 may have a second display screen. If the back surface 1265 of the mobile device 1250 has a display screen 1245, any positioning of the third camera 1230C and the fourth camera 1230D relative to the display screen 1245 can be used, as discussed with respect to the first camera 1230A and the second camera 1230B at the front surface 1255 of the mobile device 1250. The third camera 1230C and the fourth camera 1230D can be both of the one or more cameras 310. In some examples, the back surface 1265 of the mobile device 1250 may have only a single camera. In some examples, the mobile device 1250 may include one or more additional cameras in addition to the first camera 1230A, the second camera 1230B, the third camera 1230C, and the fourth camera 1230D. In some examples, the mobile device 1250 may include one or more additional sensors in addition to the first camera 1230A, the second camera 1230B, the third camera 1230C, and the fourth camera 1230D.

[0151] Like HMD 1110, mobile device 1250 does not include wheels, propellers, or other means of transport of its own. Instead, mobile device 1250 relies on the movements of a user holding or wearing mobile device 1250 to move it around in the environment. In some cases, such as when mobile device 1250 is used for AR, VR, MR, or XR, the environment may be fully or partially virtual. In some cases, mobile device 1250 may be inserted into a head-mounted device (HMD) (e.g., into a cradle of the HMD) such that mobile device 1250 functions as a display for the HMD, with display screen 1245 of mobile device 1250 serving as the HMD's display. If the environment is at least partially virtual, movement through the virtual environment may also be virtual. For example, movement through the virtual environment may be controlled by one or more joysticks, buttons, video game controllers, mice, keyboards, trackpads, and / or other input devices coupled to mobile device 1250, either wired or wirelessly.

[0152] Figure 13is a diagram illustrating an example of a system for implementing certain aspects of the present technology. Specifically, Figure 13 An example of a computing system 1300 is illustrated, which can be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using a connection 1305. The connection 1305 can be a physical connection using a bus, or a direct connection to the processor 1310, such as in a chipset architecture. The connection 1305 can also be a virtual connection, a networked connection, or a logical connection.

[0153] In some examples, computing system 1300 is a distributed system, in which the functionality described in this disclosure can be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some examples, one or more of the described system components represent a plurality of such components, each of which performs some or all of the functionality for which the component is described. In some cases, these components can be physical or virtual devices.

[0154] Example system 1300 includes at least one processing unit (CPU or processor) 1310 and connections 1305 that couple various system components including system memory 1315, such as read-only memory (ROM) 1320 and random access memory (RAM) 1325, to processor 1310. Computing system 1300 may include a cache 1312 of high-speed memory directly connected to, in close proximity to, or integrated as part of processor 1310.

[0155] Processor 1310 may include any general-purpose processor and hardware or software services, such as services 1332, 1334, and 1336 stored in storage device 1330, configured to control processor 1310 as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 1310 may be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0156] To enable user interaction, the computing system 1300 includes an input device 1345 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, speech, a camera, an accelerometer, a gyroscope, and the like. The computing system 1300 may also include an output device 1335 that can be one or more of a plurality of output mechanisms. In some instances, a multimodal system may enable a user to provide multiple types of input / output to communicate with the computing system 1300. The computing system 1300 may include a communication interface 1340, which generally governs and manages user input and system output. The communication interface may perform or facilitate the use of wired and / or wireless transceivers to receive and / or send wired or wireless communications, including utilizing an audio jack / plug, a microphone jack / plug, a Universal Serial Bus (USB) port / plug, an Apple ® Lightning ® Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Dedicated wired ports / plugs, Bluetooth ® Wireless signal transmission, Bluetooth ® Low energy (BLE) wireless signal transmission, IBEACON ® The communication interface 1340 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 1300 based on one or more signals received from one or more satellites associated with the one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no restriction on operating on any particular hardware arrangement, and thus the base features herein may be readily substituted for improved hardware or firmware arrangements as they are developed.

[0157] The storage device 1330 may be a non-volatile and / or non-transitory and / or computer-readable memory device and may be a hard disk or other type of computer-readable medium that can store data that can be accessed by a computer, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a magnetic cassette, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / strip, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disc read-only memory (CD-ROM) disc, a rewritable compact disc (CD) disc, a digital video disc (DVD) disc, a Blu-ray disc (BDD) disc, a holographic disc, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a Memory Stick ® card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin-transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.

[0158] Storage devices 1330 may include software services, servers, services, etc., which, when the code defining such software is executed by processor 1310, causes the system to perform a function. In some examples, hardware services that perform a particular function may include software components stored in a computer-readable medium connected to the necessary hardware components (such as processor 1310, connections 1305, output devices 1335, etc.) to perform the function.

[0159] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media that can store data and does not include carrier waves and / or transient electronic signals propagating wirelessly or over a wired connection. Examples of non-transitory media include, but are not limited to, magnetic disks or tapes, optical storage media (such as compact discs (CDs) or digital versatile discs (DVDs)), flash memory, memory, or storage devices. A computer-readable medium may have stored thereon code and / or machine-executable instructions, which may represent a procedure, function, subroutine, program, routine, subroutine, module, software package, class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted using any suitable means, including memory sharing, message passing, token passing, network transmission, and the like.

[0160] In some examples, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media specifically excludes media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0161] Specific details are provided in the above description to provide a thorough understanding of the examples provided herein. However, it will be understood by those skilled in the art that examples can be implemented without these specific details. For clarity of explanation, in some cases, the present technology may be presented as including separate functional blocks, including functional blocks that include devices, device components, steps in the method embodied in software or a combination of hardware and software or routines. Additional components other than those shown in the accompanying drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form so as not to obscure the examples in unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures and techniques may be shown without unnecessary details in order to avoid obscuring the examples.

[0162] The various examples above may be described as processes or methods, which may be depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flow chart may describe operations as a sequential process, many of the operations may be performed in parallel or concurrently. Furthermore, the order of the operations may be rearranged. A process is terminated when its operations are completed, but a process may have additional steps not included in the accompanying figures. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or the main function.

[0163] The processes and methods according to the examples described above can be implemented using stored computer-executable instructions or computer-executable instructions otherwise obtained from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binary, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0164] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., a computer program product) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. By way of further example, such functionality may also be implemented on circuit boards in different chips or different processes executed on a single device.

[0165] Instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0166] In the foregoing description, various aspects of the present application have been described with reference to the specific examples of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the illustrative examples of the present application have been described in detail herein, it should be understood that the inventive concept can be implemented and adopted in various other ways, and the appended claims are intended to be interpreted as including these variations, unless limited by the prior art. The various features and aspects of the above-mentioned applications can be used individually or in combination. Further, without departing from the broader essence and scope of this specification, each example can be utilized in any number of environments and applications other than the environment and application described herein. Therefore, the description and the accompanying drawings should be considered to be illustrative and not restrictive. For illustrative purposes, each method is described in a specific order. It should be understood that in an alternative example, the method can be performed in an order different from the described order.

[0167] It should be understood by those skilled in the art that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to (" ") and greater than or equal to (" ) symbol instead.

[0168] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0169] The phrase “coupled to” refers to any component being directly or indirectly physically connected to another component, and / or any component being in direct or indirect communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0170] Claim language or other language that recites "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A, B, and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0171] The various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the examples disclosed herein can be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. Technicians may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as departing from the scope of this application.

[0172] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices, or integrated circuit devices with multiple uses, including applications in wireless communication devices and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium containing program code, including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, and the like. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0173] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Therefore, the term "processor," as used herein, may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein may be provided within dedicated software or hardware modules configured for encoding and decoding, or incorporated into a combined video encoder-decoder (CODEC).

[0174] Illustrative aspects of the present disclosure include:

[0175] Aspect 1. An apparatus for distributed processing, the apparatus comprising: a memory comprising instructions; and a processor coupled to the memory and configured to: send a discovery message; receive a response to the discovery message, the response comprising an identifier of the neighboring device; send an offload request to an offload server, the offload request comprising the identifier of the neighboring device; receive an indication from the offload server that the neighboring device is selected to perform offload processing; send data for processing to the neighboring device; receive processed data from the neighboring device; and send an indication of the amount of offload processing completed.

[0176] Aspect 2. The apparatus according to aspect 1, wherein the offload request includes an indication of a computing module for offloading.

[0177] Aspect 3. The apparatus of aspect 2, wherein the offload request includes an estimate of an amount of computation offload service to process the computation module.

[0178] Aspect 4. An apparatus according to any one of aspects 1 to 3, wherein the discovery message comprises a sidelink discovery message.

[0179] Aspect 5. An apparatus according to Aspect 4, wherein the processor is further configured to: establish a side link connection with the neighboring device; send the data for processing via the side link connection; and receive the processed data via the side link connection.

[0180] Aspect 6. The apparatus according to any one of aspects 1 to 5, wherein the offload server is hosted on a core network, and wherein the apparatus and the neighboring device are connected to the core network.

[0181] Aspect 7. The apparatus according to any one of aspects 1 to 6, wherein the processor is further configured to: communicate with the offload server using a first radio access technology; and communicate with the neighboring device using a second radio access technology.

[0182] Aspect 8. A first device for distributed processing, the first device comprising: a memory comprising instructions; and a processor coupled to the memory and configured to: send an indication to an offloading server that the first device is capable of providing computing offloading services; receive a discovery message from a second device; send a response to the discovery message; receive an indication from the offloading server that the first device is selected to provide computing offloading services; receive data for processing from the second device; process the data to obtain processed data; and send the processed data to the second device.

[0183] Aspect 9. The first device according to aspect 8, wherein the processor is further configured to receive a credit for processing the data.

[0184] Aspect 10. The first device according to any one of aspects 8 to 9, wherein the indication that the first device is capable of providing a computation offloading service comprises an indication of computational resources of the first device available for use.

[0185] Aspect 11. The first device according to any one of aspects 8 to 10, wherein the discovery message comprises a sidelink discovery message.

[0186] Aspect 12. The first device according to aspect 11, wherein the processor is further configured to: establish a side link connection with the second device; receive the data via the side link connection; and send the processed data via the side link connection.

[0187] Aspect 13. The first device according to any one of aspects 8 to 12, wherein the indication that the first device is selected to provide a computation offloading service comprises an indication of a computation module for processing.

[0188] Aspect 14. The first device according to aspect 13, wherein, in order to process the data, the processor is configured to process the data based on the instruction to the computing module for processing.

[0189] Aspect 15. The first device according to any one of aspects 8 to 14, wherein the processor is further configured to: communicate with the offload server using a first radio access technology; and communicate with the second device using a second radio access technology.

[0190] Aspect 16. An apparatus for distributed processing, the apparatus comprising: a memory comprising instructions; and a processor coupled to the memory and configured to: receive an offload request from a first device, the offload request comprising an identifier of a second device; receive an indication from the second device that the second device is capable of providing a computational offload service; select the second device to perform the computational offload service; send an indication that the second device is selected to provide the computational offload service; receive an indication from the first device of an amount of completed offload processing; charge a first account associated with the first device based on the indication of the amount of completed offload processing; and credit a second account associated with the second device based on the indication of the amount of completed offload processing.

[0191] Aspect 17. The apparatus according to aspect 16, wherein the offload request includes an indication of a computing module for offloading.

[0192] Aspect 18. The apparatus according to aspect 17, wherein the indication that the second device is capable of providing computation offload services comprises an indication of computational resources available for use on the second device.

[0193] Aspect 19. An apparatus according to Aspect 18, wherein, in order to select the second device, the processor is configured to: identify the second device based on the identifier of the second device; and select the second device based on a comparison between the indicated computing module for offloading and the indicated available computing resources.

[0194] Aspect 20. The apparatus according to aspect 17, wherein, in order to send the indication that the second device is selected, the processor is configured to send the indication of the computing module for offloading to the second device.

[0195] Aspect 21. An apparatus according to any one of aspects 16 to 20, wherein the apparatus is hosted on a core network, and wherein the first device and the second device are connected to the core network.

[0196] Aspect 22. An apparatus according to any one of aspects 16 to 21, wherein the processor is further configured to: communicate with the first device using a first radio access technology; and communicate with the second device using a second radio access technology.

[0197] Aspect 23. A method for distributed processing, the method comprising: sending a discovery message; receiving a response to the discovery message, the response including an identifier of the neighboring device; sending an offload request to an offload server, the offload request including the identifier of the neighboring device; receiving an indication from the offload server that the neighboring device is selected to perform offload processing; sending data for processing to the neighboring device; receiving processed data from the neighboring device; and sending an indication of the amount of offload processing completed.

[0198] Aspect 24. The method according to aspect 23, wherein the offload request includes an indication of a computing module for offloading.

[0199] Aspect 25. The method of aspect 24, wherein the offload request includes an estimate of an amount of computation offload service to process the computation module.

[0200] Aspect 26. A method according to any one of Aspects 23 to 25, wherein the discovery message comprises a sidelink discovery message.

[0201] Aspect 27. The method according to aspect 26 further includes: establishing a side link connection with the neighboring device; sending the data for processing via the side link connection; and receiving the processed data via the side link connection.

[0202] Aspect 28. The method of any one of aspects 23 to 27, wherein the offload server is hosted on a core network, and wherein the neighboring device is connected to the core network.

[0203] Aspect 29. The method according to any one of aspects 23 to 28, further comprising: communicating with the offload server using a first radio access technology; and communicating with the neighboring device using a second radio access technology.

[0204] Aspect 30. A method for distributed processing, the method comprising: sending an indication to an offload server that a first device is capable of providing computing offload services; receiving a discovery message from a second device; sending a response to the discovery message; receiving an indication from the offload server that the first device is selected to provide computing offload services; receiving data for processing from the second device; processing the data to obtain processed data; and sending the processed data to the second device.

[0205] Aspect 31. The method of aspect 30, further comprising receiving a credit for processing the data.

[0206] Aspect 32. The method according to any one of aspects 30 to 31, wherein the indication that the first device is capable of providing a computation offload service comprises an indication of computational resources of the first device available for use.

[0207] Aspect 33. A method according to any one of Aspects 30 to 32, wherein the discovery message comprises a sidelink discovery message.

[0208] Aspect 34. The method according to aspect 33 further includes: establishing a side link connection with the second device; receiving the data via the side link connection; and sending the processed data via the side link connection.

[0209] Aspect 35. The method according to any one of aspects 30 to 34, wherein the indication that the first device is selected to provide a computation offload service comprises an indication of a computation module for processing.

[0210] Aspect 36. The method according to aspect 35, further comprising processing the data based on the instruction to the computing module for processing.

[0211] Aspect 37. The method according to any one of aspects 30 to 36, further comprising: communicating with the offload server using a first radio access technology; and communicating with the second device using a second radio access technology.

[0212] Aspect 38. A method for distributed processing, the method comprising: receiving an offload request from a first device, the offload request including an identifier of a second device; receiving an indication from the second device that the second device is capable of providing computing offload services; selecting the second device to perform the computing offload service; sending an indication that the second device is selected to provide computing offload services; receiving an indication of the amount of completed offload processing from the first device; charging a first account associated with the first device based on the indication of the amount of completed offload processing; and crediting a second account associated with the second device based on the indication of the amount of completed offload processing.

[0213] Aspect 39. The method according to aspect 38, wherein the offload request includes an indication of a computing module for offloading.

[0214] Clause 40. The method of clause 39, wherein the indication that the second device is capable of providing computation offload services comprises an indication of computational resources available for use on the second device.

[0215] Aspect 41. A method according to Aspect 40, wherein selecting the second device includes: identifying the second device based on the identifier of the second device; and selecting the second device based on a comparison between the indicated computing module for offloading and the indicated available computing resources.

[0216] Clause 42. The method of clause 39, wherein sending the indication that the second device is selected comprises sending the indication of a computing module for offloading to the second device.

[0217] Aspect 43. The method according to any one of aspects 38 to 42, wherein the apparatus is hosted on a core network, and wherein the first device and the second device are connected to the core network.

[0218] Aspect 44. The method according to any one of aspects 3 to 438, further comprising: communicating with the first device using a first radio access technology; and communicating with the second device using a second radio access technology.

[0219] Aspect 45: The apparatus of any one of aspects 1 to 15, wherein the apparatus is a mobile device.

[0220] Aspect 46. A non-transitory computer-readable medium having instructions stored thereon that, when executed by at least one processor, cause the at least one processor to perform one or more of the operations of any one of Aspects 23 to 44.

[0221] Aspect 47: An apparatus for image generation, the apparatus comprising means for performing one or more of the operations according to any one of aspects 23 to 44.

Claims

1. A device for distributed processing, comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: Send discovery message; receiving a response to the discovery message, the response including an identifier of a neighboring device; Sending an uninstall request to an uninstall server, where the uninstall request includes the identifier of the neighboring device; receiving, from the offloading server, an indication that the neighboring device is selected to perform an offloading process; sending data for processing to the neighboring device; receiving processed data from the neighboring device; as well as Sends an indication of the amount of offload processing completed. 2 . The apparatus of claim 1 , wherein the offload request comprises an indication of a computing module for offloading.

3. The apparatus of claim 2, wherein the offload request includes an estimate of an amount of computation offload service to process the computation module. The apparatus of claim 1 , wherein the discovery message comprises a sidelink discovery message.

5. The apparatus of claim 4 , wherein the at least one processor is further configured to: Establishing a side link connection with the adjacent device; sending the data for processing via the sidelink connection; and The processed data is received via the side link connection.

6. The apparatus of claim 1, wherein the offload server is hosted on a core network, and wherein the apparatus and the neighboring device are connected to the core network.

7. The apparatus of claim 1 , wherein the at least one processor is further configured to: communicating with the offload server using a first radio access technology; and Communicating with the neighboring device using a second radio access technology.

8. A first device for distributed processing, the first device comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: Sending an indication to the offloading server that the first device is capable of providing computing offloading services; receiving a discovery message from a second device; sending a response to the discovery message; receiving, from the offload server, an indication that the first device is selected to provide a computation offload service; receiving data for processing from the second device; processing the data to obtain processed data; as well as The processed data is sent to the second device.

9. The first device of claim 8, wherein the at least one processor is further configured to receive a credit for processing the data.

10. The first device of claim 8, wherein the indication that the first device is capable of providing a computation offload service comprises an indication of computational resources of the first device available for use. The first device of claim 8 , wherein the discovery message comprises a sidelink discovery message.

12. The first device of claim 11, wherein the at least one processor is further configured to: establishing a side link connection with the second device; receiving the data via the sidelink connection; and The processed data is sent via the sidelink connection.

13. The first device of claim 8, wherein the indication that the first device is selected to provide a computation offload service comprises an indication of a computation module for processing.

14. The first device according to claim 13, wherein: To process the data, the at least one processor is configured to process the data based on the instruction to the computing module for processing.

15. The first device of claim 8, wherein the at least one processor is further configured to: communicating with the offload server using a first radio access technology; and Communicating with the second device using a second radio access technology.

16. An apparatus for distributed processing, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: receiving an uninstall request from the first device, the uninstall request including an identifier of the second device; receiving, from the second device, an indication that the second device is capable of providing a computation offloading service; selecting the second device to perform the computation offloading service; sending an indication that the second device is selected to provide a computation offloading service; receiving, from the first device, an indication of an amount of offload processing completed; charging a first account associated with the first device based on the indication of the amount of completed offload processing; as well as A second account associated with the second device is credited based on the indication of the amount of offload processing completed. The apparatus of claim 16 , wherein the offload request includes an indication of a computing module for offloading.

18. The apparatus of claim 17, wherein the indication that the second device is capable of providing a computation offload service comprises an indication of computational resources available for use on the second device.

19. The device according to claim 18, wherein To select the second device, the at least one processor is configured to: identifying the second device based on the identifier of the second device; as well as The second device is selected based on a comparison between the indicated computing modules for offloading and the indicated available computing resources.

20. The apparatus according to claim 17, wherein To send the indication that the second device is selected, the at least one processor is configured to send the indication of the computing module for offloading to the second device.

21. The apparatus of claim 16, wherein the apparatus is hosted on a core network, and wherein the first device and the second device are connected to the core network.

22. The apparatus of claim 16, wherein the at least one processor is further configured to: communicating with the first device using a first radio access technology; and Communicating with the second device using a second radio access technology.

23. A method for distributed processing, the method comprising: Send discovery message; receiving a response to the discovery message, the response including an identifier of a neighboring device; Sending an uninstall request to an uninstall server, where the uninstall request includes the identifier of the neighboring device; receiving, from the offloading server, an indication that the neighboring device is selected to perform an offloading process; sending data for processing to the neighboring device; receiving processed data from the neighboring device; as well as Sends an indication of the amount of offload processing completed.

24. The method of claim 23, wherein the offload request includes an indication of a computing module for offloading.

25. The method of claim 24, wherein the offload request includes an estimate of an amount of compute offload service to process the compute module.

26. The method of claim 23, wherein the discovery message comprises a sidelink discovery message.

27. The method according to claim 26, further comprising: Establishing a side link connection with the adjacent device; sending the data for processing via the sidelink connection; as well as The processed data is received via the side link connection.

28. The method of claim 23, wherein the offload server is hosted on a core network, and wherein the neighboring device is connected to the core network.

29. The method of claim 23, further comprising: communicating with the offload server using a first radio access technology; as well as Communicating with the neighboring device using a second radio access technology.

Images