Dynamic capture parameter processing for low power consumption

Abstract

In one general aspect, a method can include capturing a first raw image at a first resolution using an image sensor, converting the first raw image to a digitally processed image using an image signal processor, and analyzing at least a portion of the digitally processed image based on a processing condition. The method can include determining that the first resolution does not satisfy the processing condition, and triggering capture of a second raw image at the image sensor at a second resolution greater than the first resolution.

Description

[Technical Field]

[0001] Related Applications This application is a continuation of and claims priority to U.S. Application No. 17 / 819,154 and U.S. Application No. 17 / 819,170, filed August 11, 2022, which claim the benefit of U.S. Provisional Application No. 63 / 260,206 and U.S. Provisional Application No. 63 / 260,207, filed August 12, 2021, the disclosures of which are incorporated herein by reference in their entireties.

[0002] This application also claims priority to U.S. Provisional Application No. 63 / 260,206, filed August 12, 2021, and U.S. Provisional Application No. 63 / 260,207, filed August 12, 2021, the disclosures of which are incorporated herein by reference in their entireties.

[0003] Technical Field This specification relates generally to methods, devices, and algorithms used to process image content. [Background technology]

[0004] background Computer vision techniques enable computers to analyze and extract information from images. Such computer vision techniques can be costly in terms of processing and power consumption. As the demand for balancing performance and power consumption in mobile computing devices continues to increase, device manufacturers are tasked with configuring devices to balance image degradation and device performance so as not to overtax the limitations of the mobile computing device. Summary of the Invention

[0005] overview One or more computer systems can be configured to perform particular operations or actions by installing software, firmware, hardware, or a combination thereof on the system that causes the system to perform the actions during operation. One or more computer programs can be configured to perform particular operations or actions by containing instructions that, when executed by a data processing device, cause the device to perform the actions.

[0006] This disclosure describes systems and methods for performing image processing that may enable a computing device (e.g., a wearable computing device) to efficiently perform (e.g., implement) computer vision tasks. The systems and methods described herein may reduce (e.g., minimize) the amount of image data used in performing computer vision and image analysis tasks, reducing the use of certain hardware and / or device resources (e.g., memory, processor, network bandwidth, etc.). Configuring a computing device (e.g., a head-mounted device (HMD), augmented reality (AR) smart glasses) to reduce resource use may provide the advantage of allowing such devices to perform relatively low-power computational tasks on-board the device and offload (e.g., push) complex and / or high-power processing to a companion device (e.g., a server, a mobile device connected to the computing device, another computer, etc.). For example, an initial assessment of the feasibility of optical character recognition (OCR) may be performed on the computing device (e.g., HMD), and when it is determined that images can be captured with desirable capture parameters (e.g., sufficient resolution) to perform OCR, the images may be sent (e.g., wirelessly sent) to a companion device for additional (e.g., more detailed, more intensive) OCR processing.

[0007] In at least one general aspect, an image processing method can include capturing a first raw image at a first resolution using an image sensor, converting the first raw image to a digitally processed image using an image signal processor, and analyzing at least a portion of the digitally processed image based on a processing condition. The method can include determining that the first resolution does not satisfy the processing condition, and triggering capture of a second raw image at the image sensor at a second resolution greater than the first resolution. The method can include any combination of the following:

[0008] In another general aspect, a non-transitory, machine-readable, tangible medium has stored thereon instructions that can be read and executed by one or more processors to perform an image processing method, the method including capturing a first raw image at a first resolution using an image sensor, converting the first raw image into a digitally processed image using an image signal processor, and analyzing at least a portion of the digitally processed image based on a processing condition. The method can include determining that the first resolution does not satisfy the processing condition, and triggering capture of a second raw image at the image sensor at a second resolution greater than the first resolution. The method can include any combination of the following:

[0009] In yet another general aspect, a system can include a processor and a memory device communicatively coupled to the processor and storing a plurality of instructions that, when executed by the processor, cause the processor to capture a first raw image at a first resolution using an image sensor, convert the first raw image into a digitally processed image using an image signal processor, and analyze at least a portion of the digitally processed image based on a processing condition. The processor can be configured to determine that the first resolution does not satisfy the processing condition and to trigger capture of a second raw image at the image sensor at a second resolution greater than the first resolution.

[0010] Each of the above aspects may include any combination of the following. In some embodiments, at least a portion of the digitally processed image includes text characters. In some embodiments, at least a portion of the digitally processed image includes real-world objects. In some embodiments, the processing condition is at least one of a pixel-per-character condition, a height condition, and an area condition. In some embodiments, a first resolution for capturing the first raw image corresponds to bins defined by the image sensor. In some embodiments, the first resolution is a default resolution, and the second resolution is defined based on entries in a lookup table. In some embodiments, the digitally processed image is a first digitally processed image captured by a computing device. The method may also include converting the second raw image to a second digitally processed image, determining that the second resolution meets the processing condition, and, in response to determining that the second resolution meets the processing condition, transmitting the second digitally processed image to a companion device of the computing device for further processing. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 illustrates an example of a computing device for generating and processing image content using limited computational and / or power resources, according to implementations described throughout this disclosure. [Figure 2] FIG. 1 illustrates a system for performing image processing on a computing device, according to implementations described throughout this disclosure. [Figure 3A] 1A-1C show signs with printed words that can be seen in the HMD display in various scenarios. [Figure 3B] 1A-1C show signs with printed words that can be seen in the HMD display in various scenarios. [Figure 3C]1A-1C show signs with printed words that can be seen in the HMD display in various scenarios. [Figure 4A] FIG. 1 illustrates an example of a wearable computing device, according to implementations described throughout this disclosure. [Figure 4B] FIG. 1 illustrates an example of a wearable computing device, according to implementations described throughout this disclosure. [Figure 5] 1 is a flowchart illustrating a method associated with embodiments described herein. [Figure 6] 1 illustrates examples of computing devices and mobile computing devices that can be used with the techniques described herein. DETAILED DESCRIPTION OF THE INVENTION

[0012] Like reference symbols in the various drawings indicate like elements. Detailed Description This disclosure describes systems and methods for performing image processing that may enable a computing device (e.g., a wearable computing device) to efficiently perform (e.g., implement) computer vision tasks. For example, the systems and methods described herein may utilize processors, sensors, neural networks, and / or image analysis algorithms to recognize text, symbols, and / or objects in images and / or extract information from images. In some implementations, the systems and methods described herein may perform such tasks while operating in reduced computational and / or reduced power modes. For example, the systems and methods described herein may reduce (e.g., minimize) the amount of image data used when performing computer vision and image analysis tasks, thereby reducing the use of certain hardware and / or device resources (e.g., memory, processor, network bandwidth, etc.).

[0013] Configuring computing devices (e.g., head-mounted devices (HMDs), augmented reality (AR) smart glasses) to reduce resource usage can provide the advantage of allowing such devices to perform relatively low-power computational tasks on-board the device and offload (e.g., push) complex and / or high-power processing to companion devices (e.g., servers, mobile devices connected to the computing device, other computers, etc.).

[0014] For example, an initial assessment of the feasibility of optical character recognition (OCR) may be performed on the computing device (e.g., HMD), and when it is determined that images can be captured with desirable capture parameters (e.g., sufficient resolution) to perform OCR, the images may be sent (e.g., wirelessly sent) to a companion device for additional (e.g., more detailed, more intensive) OCR processing.

[0015] As another example, one or more capture parameters (e.g., resolution, number of pixels per character) for performing an early stage of OCR (e.g., an early stage of an OCR pipeline such as text recognition) may be performed on a computing device. When it is determined that an image can be captured with the desired capture parameters for performing OCR, the image may be sent (e.g., wirelessly sent) to a companion device for detailed OCR processing. In some embodiments, the detailed OCR processing (or other processing) may be performed on the computing device, and the techniques described herein may help conserve energy during at least some (e.g., any) downstream processing.

[0016] In some implementations, initial low-power processing of an image can be used to determine target capture parameters (e.g., minimum resolution, cropping to a region of interest (ROI)) for capturing an image that can be sent to another device for higher power and / or more complex processing. The specific capture parameters can be determined based on one or more processing conditions. The initial processing to determine the target capture parameters can be referred to as the capture parameter selection processing stage, and subsequent processing can be referred to as the downstream image processing stage.

[0017] As a specific example, at least a portion of an image may be captured with first capture parameters, and a capture parameter selection processing stage may determine, based on processing conditions, that the first capture parameters do not satisfy (e.g., are not high enough to) be processed in a subsequent processing stage. The processing conditions may be conditions used to determine whether processing in a subsequent processing stage (e.g., downstream processing) can be performed in a desirable manner. In response to determining the insufficiency of the first solution, a portion of the image may be captured with second capture parameters, and a determination may be made, based on the processing conditions, that the second capture parameters are sufficient to be processed in a subsequent processing stage. The downstream image processing stage may be, for example, processing related to optical character recognition.

[0018] As another example, at least a portion of an image may be captured with first capture parameters, and a capture parameter selection processing stage may determine, based on processing conditions, that the first capture parameters meet (e.g., are sufficiently high) the conditions for processing in a subsequent processing stage. In response to the determination of sufficiency, the portion of the image may be processed in a downstream image processing stage (e.g., processing related to optical character recognition).

[0019] In some implementations, the capture parameter selection process can be based on specific regions of interest determined using a separate region-of-interest selection technique. For example, instead of analyzing the entire image, regions of interest (ROIs) can be identified in the image data, and one or more of those regions can be loaded onto a computing device for analysis. Reducing the amount of information to be analyzed while maintaining accurate results can allow complex image processing tasks to be performed on a computing device without the resource burden of analyzing the entire image. In some implementations, the systems and methods described herein can be configured to run on a device such that such a device can perform efficient image processing tasks while reducing computational load and / or power. For example, the systems and methods described herein can enable a device to perform object detection tasks, optical character recognition (OCR) tasks, and / or other image processing tasks while utilizing certain techniques to reduce power, memory, and / or processing consumption.

[0020] In some implementations, the systems and methods described herein may ensure that complex image processing calculations can be performed on the device without assistance from other resources and / or devices. For example, conventional systems may require assistance from other communicatively coupled mobile devices, servers, and / or off-board systems to perform computationally intensive image processing tasks. The systems and methods described herein offer the advantage of providing complete and accurate image processing capabilities while generating portions of an image that can be manipulated by devices with less computational power than, for example, a server.

[0021] The systems and methods described herein may enable computing devices to perform computer vision tasks such as object detection, motion tracking, face recognition, OCR tasks, etc. using machine learning intelligence (e.g., neural networks, algorithms, etc.) that utilize low power and / or processing consumption.

[0022] 1 illustrates an example of a computing device 100 (e.g., an HMD) for generating and processing image content using limited computational and / or power resources, according to implementations described throughout this disclosure. In this example, computing device 100 is depicted in the form of AR smart glasses. However, any form factor may be substituted and combined with the systems and methods described herein. In some implementations, computing device 100 includes a system-on-chip (SOC) architecture (not shown) combined with one or more sensors, a low-power island processor, a high-power island processor, a core processor, an encoder, or the like.

[0023] In operation, computing device 100 (which may also be referred to as a device) can capture an image of a scene 102 using, for example, a camera, an image sensor, etc. Device 100 may be worn and operated by a user 104. Scene 102 may include physical content as well as augmented reality (AR) content. Scene 102 is shown from the perspective of a view through device 100. In some implementations, computing device 100 may be communicatively coupled to a companion device 106 (e.g., a mobile computing device).

[0024] In some implementations, computing device 100 may be triggered to begin real-time image processing (e.g., real-time image and / or character recognition) in response to a request to identify image content from scene 102. In a non-limiting example, computing device 100 may be triggered to begin real-time image processing in response to a request to identify image content associated with optical data captured by a sensor included in device 100. The request may come from a user wearing device 100 (e.g., user 104).

[0025] In operation, computing device 100 may include a capture parameter selection processing stage 100-1. Initial low-power processing of an image in capture parameter selection processing stage 100-1 may be used to determine target capture parameters (e.g., minimum resolution, cropping to a region of interest (ROI)) for capturing an image of at least a portion of scene 102 that is desirable for (or to trigger) further processing. The target capture parameters determined in capture parameter selection processing stage 100-1 may be used to capture one or more additional images that may be sent to companion device 106 for higher-power and / or more complex processing in downstream image processing stage 100-2. Although downstream image processing stage 100-2 is illustrated as being within companion device 106, any of the processing in downstream image processing stage 100-2 may be performed in computing device 100.

[0026] Target capture parameters for capturing an image of at least a portion of a scene 102 may be determined based on one or more processing conditions 101 in a capture parameter selection processing stage 100-1. The target capture parameters may be capture parameters that are optimized (based on the processing conditions) for processing the image in a downstream image processing stage 100-2. Specifically, the target capture parameters may be selected (e.g., converged) based on one or more processing conditions 101 being satisfied.

[0027] In some implementations, the target capture parameter may be selected based on the initial capture parameter for capturing the content. In some implementations, the target capture parameter may be the same as the initial capture parameter or may be identified to be the same as the initial capture parameter. In some implementations, in addition to the initial capture parameter, multiple intermediate capture parameters may be tested (e.g., considered against one or more processing conditions) until the target capture parameter is identified (e.g., until convergence occurs). At least one of the intermediate capture parameters may be identified as the target capture parameter if the intermediate capture parameter satisfies one or more processing conditions 101.

[0028] In some implementations, content for processing by capture parameter selection processing stage 100-1 may be targeted to a region of interest 130. In some implementations, a region of interest (e.g., a region of interest narrowed by cropping) may be selected in conjunction with or after target capture parameter processing. In some implementations, a region of interest may be selected or processed by any element of capture parameter selection processing stage 100-1 (e.g., an image sensor, an image signal processor, on-device processing, etc.).

[0029] As a specific example, text in a scene 102 may be captured in a first image using initial capture parameters representing a first resolution (low resolution, low resolution level). The capture parameter selection processing stage 100-1 may determine that the text in the first image does not satisfy processing condition 101 (e.g., characters in the text are below a pixel-per-character threshold (e.g., pixel height per character)). Because processing condition 101 is not satisfied based on analysis of the first image (e.g., text) by the capture parameter selection processing stage 100-1, a second image of the text in the scene may be captured using intermediate capture parameters representing a second resolution (higher resolution, high resolution level). If the capture parameter selection processing stage 100-1 determines that the text in the second image satisfies processing condition 101 (e.g., characters in the text meet or exceed the pixel-per-character threshold), the captured second image of the text may be sent to a companion device for processing (e.g., OCR processing) in the downstream image processing stage 100-2.

[0030] In some implementations, multiple images can be captured simultaneously or nearly simultaneously with different capture parameters. An evaluation may be performed as to whether a first image of the multiple images satisfies one or more processing conditions 101. The first image of the multiple images may be the image that uses fewer processing resources (e.g., the image that uses the fewest processing resources of the multiple images). If the first image of the multiple images satisfies one or more processing conditions 101, further evaluation of other images of the multiple images may be unnecessary. If the first image of the multiple images does not satisfy one or more processing conditions 101, an evaluation may be performed as to whether a second image of the multiple images satisfies one or more processing conditions 101. The second image of the multiple images may be the image that uses more (e.g., incrementally more) processing resources than the first image of the multiple images. This processing of the multiple images may be performed until one of the multiple images is identified as satisfying one or more processing conditions 101 while using the least amount of processing resources (from the multiple images).

[0031] 2 illustrates a system 200 for performing image processing on a computing device 100, according to implementations described throughout this disclosure. In some implementations, image processing is performed on the computing device 100. In some implementations, image processing is shared among one or more devices. For example, image processing may be completed partially on the computing device 100 and partially on a companion device 202 (e.g., a mobile computing device 106) and / or a server computing device 204. In some implementations, image processing is performed on the computing device 100, and output from such processing is provided to the companion device 202 and / or the server computing device 204.

[0032] 2 illustrates an implementation of capture parameter selection processing stage 100-1 included in computing device 100. Although not shown, downstream image processing stage 100-2 may be included in companion device 202. In this implementation, capture parameter selection processing stage 100-1 includes image sensor 216, image signal processor 224, and parameter selection processor 227.

[0033] The image signal processor 224 can be configured to convert raw images (e.g., raw images, analog images) captured by the image sensor 216 into a digitally processed image format that can be processed by the parameter selection processor 227. For example, the raw images can be uncompressed images that do not contain information for further image processing. The image signal processor 224 can process the raw images into a digital format that can be further processed by downstream processing, such as the parameter selection processor 227. The image signal processor 224 can perform, for example, noise reduction, automatic white balance and color correction, color interpolation, gamma correction, etc., to generate a digitally processed image. In some implementations, the image signal processor 224 can be part of the sensor system 214. For simplicity of discussion, the conversion of the images generated by the image sensor 216 into a digital representation by the image signal processor 224 will generally be omitted from the description, even if such conversion occurs before downstream processing (e.g., processing by the parameter selection processor 227). Thus, processing by downstream components generally occurs after the digital processing by the image signal processor 224, even if not mentioned within the description herein.

[0034] The embodiment shown in Figure 2 is discussed in the context of resolution (e.g., resolution capture parameters 221, resolution conditions 222, resolution selection algorithm 223) associated with the OCR process to focus the discussion and explain the principles in this description. Different types of capture parameters, conditions, and / or processes other than or in addition to resolution may be used in conjunction with at least the embodiment of Figure 2. For example, the techniques described herein may be applied to any type of processing (e.g., computer vision, machine learning) of real-world objects, such as, but not limited to, barcodes, products, signs, storefronts, landmarks, plants, food, faces, hands (e.g., pointing or holding), objects (e.g., cars, buildings, bicycles, books, people), feature point tracking, and / or other real-world objects.

[0035] The image sensor 216, image signal processor 224, and parameter selection processor 227 can function as a pipeline within the capture parameter selection processing stage 100-1. Image resolution is reduced or minimized as early in the pipeline as possible to reduce or minimize energy consumption. For example, a minimum resolution image can be captured by the image sensor 216 so that processing in any downstream processing (e.g., image signal processor 224, parameter selection processor 227, communications module 252, companion device 202) can be reduced or minimized. Capturing images at a higher resolution in the image sensor 216 can result in increased power usage, increased latency (e.g., transmission latency), etc. in downstream processing.

[0036] As described above, capture parameter selection process stage 100-1 may be used to determine target resolution capture parameters for capturing an image of at least a portion of scene 102 that is desirable for further processing (and for power conservation). The target resolution capture parameters determined in capture parameter selection process stage 100-1 may be used to capture one or more additional images that may be sent to companion device 202 for higher power and / or more complex OCR processing within downstream image processing stages.

[0037] The resolution capture parameters 221 may represent, for example, parameters that may be used by the image sensor 216 to capture an image. Although not shown, each of the different types of resolution capture parameters (e.g., an initial resolution capture parameter, an intermediate resolution capture parameter, and a target resolution capture parameter) may be represented by, or generally referred to as, a resolution capture parameter 221. For example, an initial or default resolution capture parameter may be referred to as a resolution capture parameter 221. The resolution capture parameters 221 may represent, for example, different values ​​or levels of resolution.

[0038] The capture parameter selection process stage 100-1 may be configured to begin capturing images using one or more initial capture parameters using the image sensor 216, and adjust and / or define additional resolution capture parameters (e.g., one or more intermediate capture parameters) until one or more target resolution capture parameters are determined for further processing. In some implementations, the initial capture parameters may be identified as the target resolution capture parameters.

[0039] For example, image sensor 216 may be configured to capture an image using initial resolution capture parameters (e.g., default resolution capture parameters, first resolution). Parameter selection processor 227 may be configured to determine whether the content captured by parameter selection processor 227 has sufficient resolution for further processing by analyzing at least a portion of the content within the image. The sufficiency of the resolution may be determined based on one or more of resolution processing conditions 222. Specifically, if the content within the image satisfies one or more of resolution processing conditions 222 (e.g., if the resolution is sufficient for further processing), parameter selection processor 227 identifies the initial resolution capture parameters as target resolution capture parameters for capturing image content using image sensor 216.

[0040] As another example, the image sensor 216 may be configured to capture an image using initial resolution capture parameters (e.g., a first resolution). The parameter selection processor 227 may be configured to determine whether the content captured by the parameter selection processor 227 has sufficient resolution for further processing by analyzing at least a portion of the content in the image. If the resolution is not sufficient for further processing, the parameter selection processor 227 identifies intermediate resolution capture parameters.

[0041] After the initial resolution capture parameters are determined to be inappropriate for capturing images by the image sensor 216 for downstream processing based on one or more resolution criteria 222, a resolution selection algorithm 223 can be used to determine one or more intermediate resolution capture parameters until a target resolution capture parameter is identified. The resolution algorithm 223 can be based on a predetermined set of resolution capture parameters. The resolution algorithm 223 can be based on a predetermined order of the resolution capture parameters. The resolution algorithm 223 can be based on identifying a resolution capture parameter that results in minimal processing.

[0042] The resolution selection algorithm 223 can be, for example, or can be based on, a look-up table (LUT) that specifies the resolution capture parameter order after the initial resolution capture parameter. In other words, different capture parameters (e.g., capture parameter levels) can be determined using the look-up table. The look-up table can be used to quickly determine a desired resolution capture parameter (or other type of capture parameter).

[0043] For example, the lookup table may include initial resolution capture parameters (e.g., default resolution capture parameters). If a first image captured based on the initial resolution capture parameters is not acceptable for further processing (e.g., downstream processing), a second image may be captured based on intermediate resolution capture parameters identified in the lookup table. If the second image captured based on the intermediate resolution capture parameters is acceptable for further processing, further processing may be performed (the intermediate resolution capture parameters may be final resolution capture parameters). If the second image captured based on the intermediate resolution capture parameters is not acceptable for further processing (e.g., downstream processing), a third image may be captured based on yet another intermediate resolution capture parameter identified in the lookup table. The processing may proceed as described above in an iterative manner until a final resolution capture parameter is identified. Other parameters other than the resolution capture parameters may also be identified.

[0044] In some implementations, the resolution selection algorithm 223 can be based on various algorithms and / or equations.

[0045] Computing device 100 includes a sensor system 214. Sensor system 214 includes an image sensor 216 configured to detect and / or acquire image data. In some implementations, sensor system 214 includes multiple image sensors 216. As shown, sensor system 214 includes one or more image sensors 216. Image sensor 216 can capture and record images (e.g., pixels, frames, and / or portions of images) and video.

[0046] In some implementations, the resolution (e.g., level of resolution) of the image sensor 216 may be defined by a predefined (e.g., preprogrammed) binning (e.g., binning of pixels) of the image sensor 216. For example, binning an increased number of pixels of the image sensor 216 can result in a reduced resolution or resolution level. Binning an increased number of pixels can result in an increased resolution or resolution level.

[0047] Satisfaction of a pixel-by-pixel threshold, which is one type of processing condition (e.g., processing condition 101), may depend on many factors, including distance and real-world image element size (e.g., actual font size). While the techniques described herein can be applied to any type of image or image element, for convenience, the techniques described herein are described with respect to fonts.

[0048] For example, an image (e.g., a raw image) that captures characters with a relatively large font, but far away from the image sensor 216, may still meet the pixel-by-pixel threshold because the relatively large font size can be resolved from a relatively large distance away from the image sensor 216. Such an example is shown in FIG.

[0049] As shown in FIG. 3A , a sign 350-1 (e.g., a real-world sign) with the word “the” printed on it is visible within the HMD display 310 (e.g., a display window having pixel size P1×P2) at a distance D1. A field of view 320 associated with an outward-facing image sensor (e.g., image sensor 216 shown in FIG. 2 ) of the HMD (not shown) may generate an image (e.g., a raw image) within the HMD display 310. In this example, the size of the word “the” on the sign is relatively large at distance D1, and therefore the size of the word within the HMD display 310 is also relatively large. The size (e.g., height in this example) in pixel number Q1 of the letter “t” from the word “the” may be greater than a threshold value (e.g., sufficiently large) so that further processing in a downstream process (e.g., downstream image processing stage 100-2 shown in FIG. 1 ) can be performed in a desired manner.

[0050] Returning to Figure 2, as another example, an image (e.g., a raw image) capturing characters in a relatively small font far away from image sensor 216 may not meet the pixel-by-pixel threshold because the relatively small size may not be resolved from a relatively large distance from image sensor 216. Such an example, which is a variation of Figure 3A, is shown in Figure 3B.

[0051] As shown in FIG. 3B , a sign 350-2 (e.g., a real-world sign) with the word “the” printed on it can be viewed within the HMD display 310 (e.g., a display window having pixel size P1×P2) at a distance D1 (the same distance as in FIG. 3A ). A field of view 320 associated with an outward-facing image sensor in the HMD (not shown) can generate an image (e.g., a raw image) within the HMD display 310. In this example, the size of the word “the” on the sign is relatively small at distance D1, and therefore the size of the word within the HMD display 310 is also relatively small. The size, in pixel number Q2, of the letter “t” from the word “the” may be smaller than a threshold value such that further processing in a downstream process (e.g., downstream image processing stage 100-2 shown in FIG. 1 ) cannot be performed in a desirable manner. In such a situation, the capture resolution may be increased so that the pixel number of the word “the” can be greater than Q2 so that further processing in a downstream process (e.g., downstream image processing stage 100-2 shown in FIG. 1 ) can be performed in a desirable manner.

[0052] In some implementations, a low-resolution image (e.g., a raw image) and a high-resolution image can be captured simultaneously or nearly simultaneously. An evaluation can be performed to determine whether the low-resolution image satisfies a processing condition (e.g., a pixel-by-pixel threshold). If the low-resolution image satisfies the processing condition, high-resolution processing may not be performed or may be omitted. If the low-resolution image does not satisfy the processing condition, an evaluation can be performed to determine whether the high-resolution image satisfies the processing condition (e.g., a pixel-by-pixel threshold). If the high-resolution image satisfies the processing condition, the high-resolution image may be used (or triggered) for further processing. If the high-resolution image does not satisfy the processing condition, a higher-resolution image may be captured, and an evaluation can be performed to determine whether the higher-resolution image satisfies the processing condition (e.g., a pixel-by-pixel threshold).

[0053] In some embodiments, processing can begin at a high processing level. For example, in some embodiments, a low-resolution image (e.g., a raw image) and a high-resolution image (e.g., a raw image) can be captured simultaneously or nearly simultaneously. An evaluation of whether the high-resolution image satisfies processing conditions (e.g., a pixel-by-pixel threshold) can be performed. If the high-resolution image satisfies the processing conditions, low-resolution processing may not be performed or may be omitted.

[0054] In some embodiments, processing can begin at a high processing level and be reduced until a desired condition is met. For example, in some embodiments, low-resolution and high-resolution images can be captured simultaneously or nearly simultaneously. An evaluation can be performed of whether the high-resolution image satisfies the processing condition (e.g., a per-pixel threshold). If the high-resolution image satisfies the processing condition, an evaluation can also be performed of whether the low-resolution image satisfies the processing condition (e.g., a per-pixel threshold). If the low-resolution image satisfies the processing condition, the low-resolution image may be used (or triggered) for further processing (rather than the high-resolution image). Images may be captured and evaluated until a resolution is captured that is likely to result in minimal further processing (determined based on the processing condition).

[0055] As another example, an image (e.g., a raw image) capturing characters with a relatively small font but close to the image sensor 216 may meet the per-pixel threshold even though the font is small because the characters can be resolved from a relatively short distance. Such an example, which is a variation of Figure 3B, is shown in Figure 3C.

[0056] As shown in FIG. 3C , a sign 350-3 (e.g., a real-world sign) with the word “the” printed on it is viewable within the HMD display 310 (e.g., a display window having pixel size P1×P2) at a distance D2 (less than distance D1 in FIGS. 3A and 3B ). A field of view 320 associated with an outward-facing image sensor in the HMD (not shown) may generate an image (e.g., a raw image) within the HMD display 310. In this example, the size of the word “the” on the sign is relatively large at distance D2, and therefore the size of the word within the HMD display 310 is also relatively large. The size (e.g., height in this example) in pixel number Q3 of the letter “t” from the word “the” may be greater than a threshold value (e.g., sufficiently large) so that further processing in a downstream process (e.g., downstream image processing stage 100-2 shown in FIG. 1 ) can be performed in a desired manner.

[0057] In accordance with the above-described embodiments, the per-pixel threshold can be a powerful tool for determining an appropriate capture resolution in many situations. The per-pixel threshold can be, for example, a single metric that can be determined without information about font size (e.g., real-world font size) and / or distance.

[0058] In some implementations, the pixel-by-pixel threshold may be determined based on the script or language of the text information. In some implementations, the one or more processing conditions 101 may be based on the angular resolution or visual acuity of the system. The angular resolution may be based on the field of view of an image sensor (e.g., a camera) that captures the image (e.g., a raw image). In some implementations, the one or more processing conditions 101 may be based on sharpness, contrast, accuracy, confidence in character identification, and / or any of the foregoing.

[0059] In some implementations, image sensor 216 is a red-green-blue (RGB) camera. In some examples, image sensor 216 includes a pulsed laser sensor (e.g., a LiDAR sensor) and / or a depth camera. For example, image sensor 216 may be a camera configured to detect and communicate information used to create the images represented by image frames 226. Image sensor 216 can capture and record both images and video.

[0060] In operation, image sensor 216 is configured to acquire (e.g., capture) image data (e.g., optical sensor data) continuously or periodically while device 100 is powered on. In some implementations, image sensor 216 is configured to operate as an always-on sensor. In some implementations, image sensor 216 may be powered on in response to detecting an object or region of interest.

[0061] In some implementations, computing device 100 includes one or more computing devices, at least one of which is a display device wearable on or near a person's skin. In some examples, computing device 100 is or includes one or more computing device components. In some implementations, computing device 100 may include a head-mounted display (HMD) device, such as an optical head-mounted display (OHMD) device, a transparent head-up display (HUD) device, a virtual reality (VR) device, an AR device, or other devices such as goggles or a headset having sensors, a display, and computing capabilities. In some implementations, computing device 100 includes AR glasses (e.g., smart glasses). AR glasses refer to an optical head-mounted display device designed in the shape of a pair of glasses. In some implementations, computing device 100 is or includes a smartwatch. In some implementations, computing device 100 is or includes jewelry. In some implementations, computing device 100 is or includes a ring controller device or other wearable controller. In some implementations, computing device 100 is or includes earphones / headphones or smart earphones / headphones.

[0062] 2, system 200 includes computing device 100 communicatively coupled to a companion device 202 and optionally a server computing device 204. In some implementations, the communicative coupling may occur over a network 206. In some implementations, the communicative coupling may occur directly between computing device 100, companion device 202, and / or server computing device 204.

[0063] Computing device 100 includes one or more processors 208, which may be formed on a substrate configured to execute one or more machine-executable instructions or software, firmware, or a combination thereof. Processor 208 may be semiconductor-based and may include semiconductor material capable of executing digital logic. Processor 208 may include a CPU, GPU, and / or DSP, to name just a few examples.

[0064] Computing device 100 may also include one or more memory devices 210. Memory device 210 may include any type of storage device that stores information in a format that can be read and / or executed by processor 208. Memory device 210 may store applications and modules that, when executed by processor 208, perform certain operations. In some examples, applications and modules may be stored in an external storage device and loaded into memory device 210. Memory 210 may include or have access to a buffer 212, for example, to store and retrieve image and / or audio content for device 100.

[0065] The sensor system 214 may also include an inertial motion unit (IMU) sensor 218. The IMU sensor 218 may detect movement, movement, and / or acceleration of the computing device 100. The IMU sensor 218 may include a variety of different types of sensors, such as, for example, an accelerometer, a gyroscope, a magnetometer, and other such sensors.

[0066] In some implementations, sensor system 214 may also include an audio sensor 220 configured to detect audio received by computing device 100. Sensor system 214 may include other types of sensors, such as light sensors, distance and / or proximity sensors, contact sensors such as capacitance sensors, timers, and / or other sensors and / or different combinations of sensors. Sensor system 214 may be used to obtain information related to the position and / or orientation of computing device 100.

[0067] Computing device 100 may also include one or more antennas 242 configured to communicate with other computing devices via wireless signals. For example, computing device 100 may receive one or more wireless signals and use the wireless signals to communicate with other devices, such as companion device 202 and / or server computing device 204, or other devices that are within range of antenna 242. The wireless signals may be triggered via a wireless connection, such as a short-range connection (e.g., a Bluetooth connection or a near-field communication (NFC) connection) or an Internet connection (e.g., Wi-Fi or a mobile network).

[0068] Computing device 100 includes a display 244. Display 244 may include a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting display (OLED), an electrophoretic display (EPD), or a microprojection display employing an LED light source. In some examples, display 244 is projected onto a user's field of view. In some examples, in the case of AR glasses, display 244 may provide a transparent or translucent display so that a user wearing the AR glasses can see not only the image provided by display 244 but also information located within the field of view of the AR glasses behind the projected image.

[0069] Computing device 100 also includes a control system 246 that includes various control system devices for facilitating operation of computing device 100. Control system 246 may utilize a processor 208 and / or a sensor system 214 operatively coupled to components of computing device 100.

[0070] Computing device 100 also includes a UI renderer 250. UI renderer 250 may work in conjunction with display 244 to depict user interface objects or other content to a user of computing device 100. For example, UI renderer 250 may receive images captured by device 100 and generate and render additional user interface content on display 244.

[0071] Computing device 100 also includes a communications module 252. Communications module 252 may enable computing device 100 to communicate to exchange information with another computing device that is within range of device 100. For example, computing device 100 may be operatively coupled to the other computing device to facilitate communication, e.g., via a wired connection, a wireless connection, e.g., via Wi-Fi or Bluetooth, or other type of connection.

[0072] In some implementations, computing device 100 is configured to communicate with server computing device 204 and / or companion device 202 over network 206. Server computing device 204 may represent one or more computing devices in the form of a number of different devices, such as, for example, a standard server, a group of such servers, or a rack server system. In some implementations, server computing device 204 is a single system that shares components such as a processor and memory. Network 206 may include the Internet and / or other types of data networks, such as a local area network (LAN), a wide area network (WAN), a cellular network, a satellite network, or other types of data networks. Network 206 may also include any number of computing devices (e.g., computers, servers, routers, network switches, etc.) configured to receive and / or transmit data within network 206.

[0073] 4A and 4B show various views of an example of an AR wearable computing device according to implementations described throughout this disclosure. FIG. 4A is a front view of an example of a wearable computing device according to implementations described throughout this disclosure. In this example, the wearable computing device is a pair of AR glasses 400A (e.g., wearable computing device 100 of FIG. 1). In general, AR glasses 400A may include any or all of the components of system 200. AR glasses 400A may also be referred to as smart glasses, which represent an optical head-mounted display device designed in the shape of a pair of glasses. For example, smart glasses are glasses that add information (e.g., project a display) in parallel with what the wearer sees through the glasses.

[0074] Although AR glasses 400A are shown as the wearable computing device described herein, other types of wearable computing devices are possible. For example, the wearable computing device may include a head-mounted display (HMD) device, such as an optical head-mounted display (OHMD) device, a transparent head-up display (HUD) device, an augmented reality (AR) device, or other devices such as goggles or headsets having sensors, displays, and computing capabilities. In some examples, the wearable computing device may be a watch, a mobile device, jewelry, a ring controller, or other wearable controller.

[0075] 4A , the AR glasses 400A include a frame 402 having a display device 404 coupled within the frame 402 (or within a glass portion of the frame 402). The AR glasses 400A also include an audio output device 406, an illumination device 408, a sensing system 410, a control system 412, at least one processor 414, and a camera 416.

[0076] The display device 404 can include a see-through near-eye display, such as one that uses birdbath or waveguide optics. For example, such an optical design can project light from a display light source onto a portion of teleprompter glass that acts as a beam splitter positioned at a 45-degree angle. The beam splitter can enable reflection and transmission values ​​that partially reflect the light from the display light source and transmit the remaining light. Such an optical design allows the user to see both physical items in the world next to the digital images (e.g., UI elements, virtual content, etc.) generated by the display. In some implementations, waveguide optics can be used to render content on the display device 404 of the AR glasses 400A.

[0077] An audio output device 406 (e.g., one or more speakers) may be coupled to the frame 402. The sensing system 410 may include a control system 412 that includes various sensing devices and various control system devices to facilitate operation of the AR glasses 400A. The control system 412 may include a processor 414 operably coupled to the components of the control system 412.

[0078] The camera 416 can capture still and / or moving images. In some implementations, the camera 416 can be a depth camera that can collect data related to the distance of external objects from the camera 416. In some implementations, the camera 416 can be a point-tracking camera that can detect and track one or more optical markers on an external device, such as, for example, an optical marker on an input device or a finger on a screen. In some implementations, the AR glasses 400A can include an illumination device 408 that can, for example, selectively operate with the camera 416 to detect objects (e.g., virtual and physical) within the field of view of the camera 416. The illumination device 408 can, for example, selectively operate with the camera 416 to detect objects within the field of view of the camera 416.

[0079] The AR glasses 400A may include a communications module (e.g., communications module 252) that communicates with the processor 414 and the control system 412. The communications module can provide for communications between devices housed within the AR glasses 400A as well as communications with external devices, such as a controller, a mobile device, a server, and / or other computing devices. The communications module can enable the AR glasses 400A to communicate with another computing device to exchange information and to authenticate other devices within range of the AR glasses 400A or other identifiable elements in the environment. For example, the AR glasses 400A may be operably coupled to other computing devices to facilitate communications, such as via a wired connection, a wireless connection, such as via Wi-Fi or Bluetooth, or other types of connections.

[0080] FIG. 4B is a back view 400B of AR glasses 400A according to embodiments described throughout this disclosure. The AR glasses 400B may be an example of the wearable computing device 100 of FIG. 1. The AR glasses 400B are glasses that add information (e.g., project onto a display 420) in parallel with what the wearer sees through the glasses. In some embodiments, instead of projecting information, the display 420 is an in-lens microdisplay. In some embodiments, the AR glasses 400B (e.g., eyeglasses or spectacles) are visual aids that include lenses 422 (e.g., glass or hard plastic lenses) attached to a frame 402 that holds the lenses in front of a person's eyes, typically utilizing a bridge 424 above the nose and temples 426 (e.g., temples or temple pieces) that rest on the ears.

[0081] 5 is a flow chart illustrating a method associated with embodiments described herein, which may be implemented at least within the capture parameter selection process stage 100-1 shown in FIGS.

[0082] The method may include capturing a first raw image at a first resolution using an image sensor (block 510). The first image may be captured using, for example, image sensor 216 shown in Figure 2. The first resolution may be included in resolution capture parameters 221 shown in Figure 2.

[0083] The method may include converting the first raw image into a digitally processed image using an image signal processor (block 520). The image signal processor may be, for example, image signal processor 224 shown in FIG. 2.

[0084] The method may include analyzing at least a portion of the digitally processed image based on processing conditions (block 530). The analysis may be performed, for example, by the parameter selection processor 227 shown in Figure 2. The processing conditions may include, for example, the processing conditions 101 shown in Figure 1. An example of a processing condition is shown as the resolution processing condition 222 shown in Figure 1.

[0085] The method may include determining that the first resolution does not meet the processing conditions (block 540). The determination may be performed, for example, by the parameter selection processor 227 shown in Figure 2. The determination may be based on the resolution selection algorithm 223 shown in Figure 2.

[0086] The method may include triggering capture of a second raw image at an image sensor at a second resolution greater than the first resolution (block 550). The second image may be captured using, for example, image sensor 216 shown in Figure 2. The second resolution may be included in resolution capture parameters 221 shown in Figure 2.

[0087] 6 illustrates examples of a computing device 1000 and a mobile computing device 1050 (e.g., implementing client computing device 100, server computing device 204, and / or mobile device 202) that may be used with the techniques described herein. Computing device 1000 includes a processor 1002, a memory 1004, a storage device 1006, a high-speed interface 1008 that connects to memory 1004 and a high-speed expansion port 1010, and a low-speed interface 1012 that connects to a low-speed bus 1014 and storage device 1006. Each of the components 1002, 1004, 1006, 1008, 1010, and 1012 are interconnected using various buses and may be mounted on a common motherboard or in any other suitable manner. The processor 1002 can process instructions for execution within the computing device 1000, including instructions stored in the memory 1004 or storage device 1006, to display graphical information for a GUI on an external input / output device, such as a display 1016 coupled to a high-speed interface 1008. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and memory types, as appropriate. Also, multiple computing devices 1000 may be connected, with each device providing a portion of the required operations (e.g., as a bank of servers, a group of blade servers, or a multiprocessor system).

[0088] The memory 1004 stores information within the computing device 1000. In one implementation, the memory 1004 is a volatile memory unit or units. In another implementation, the memory 1004 is a non-volatile memory unit or units. The memory 1004 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0089] The storage device 1006 can provide mass storage for the computing device 1000. In one embodiment, the storage device 1006 can be or include a computer-readable medium such as a floppy disk device, a hard disk device, an optical disk device, or an array of devices including a tape device, flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. A computer program product can be embodied on an information carrier. The computer program product can include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as memory 1004, the storage device 1006, or memory on the processor 1002.

[0090] The high-speed controller 1008 manages bandwidth-intensive operations of the computing device 1000, while the low-speed controller 1012 manages low-bandwidth-intensive operations. This allocation of functionality is an example. In one implementation, the high-speed controller 1008 is coupled to the memory 1004, the display 1016 (e.g., via a graphics processor or accelerator), and the high-speed expansion port 1010, which can accept various expansion cards (not shown). In this implementation, the low-speed controller 1012 is coupled to the storage device 1006 and the low-speed expansion port 1014. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input / output devices, such as a keyboard, pointing device, scanner, etc., or to a network device, such as a switch or router, for example, via a network adapter.

[0091] Computing device 1000 may be implemented in many different forms, as shown. For example, computing device 1000 may be implemented as a standard server 1020 or multiple times in a cluster of such servers. Computing device 1000 may also be implemented as part of a rack server system 1024. Furthermore, computing device 1000 may be implemented in a personal computer, such as a laptop computer 1022. Alternatively, components from computing device 1000 may be combined with other components in a mobile device (not shown), such as device 1050. Each such device may include one or more of computing devices 1000, 1050, and an entire system may be made up of multiple computing devices 1000, 1050 communicating with each other.

[0092] Computing device 1050 includes, among other components, a processor 1052, memory 1064, input / output devices such as a display 1054, a communication interface 1066, and a transceiver 1068. Device 1050 may also include a storage device, such as a microdrive or other device, to provide additional storage. Each of the components 1050, 1052, 1064, 1054, 1066, and 1068 are interconnected using various buses, and some of the components may be mounted on a common motherboard or in any other suitable manner.

[0093] The processor 1052 can execute instructions within the computing device 1050, including instructions stored in the memory 1064. The processor may be implemented as a chipset of chips including separate analog and digital processors. The processor can provide, for example, control of a user interface, applications executed by the device 1050, and coordination of other components of the device 1050, such as wireless communication by the device 1050.

[0094] The processor 1052 can communicate with a user via a control interface 1058 and a display interface 1056 coupled to a display 1054. The display 1054 can be, for example, a TFT LCD (thin film transistor liquid crystal display), an LED (light emitting diode) or an OLED (organic light emitting diode) display, or other suitable display technology. The display interface 1056 can include appropriate circuitry for driving the display 1054 to present graphical and other information to the user. The control interface 1058 can receive and convert commands from a user for transmission to the processor 1052. Additionally, an external interface 1062 can be provided in communication with the processor 1052 to enable short-range communication between the device 1050 and other devices. The external interface 1062 can, for example, provide for wired communication in some implementations and wireless communication in other implementations; multiple interfaces can also be used.

[0095] The memory 1064 stores information within the computing device 1050. The memory 1064 may be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a nonvolatile memory unit or units. Expansion memory 1074 may also be provided and connected to the device 1050 via an expansion interface 1072, which may include, for example, a SIMM (single in-line memory module) card interface. Such expansion memory 1074 may provide extra storage space for the device 1050 or may store applications or other information for the device 1050. Specifically, the expansion memory 1074 may include instructions that perform or supplement the processes described above and may also include secure information. Thus, for example, the expansion memory 1074 may be provided as a security module for the device 1050 and programmed with instructions that enable secure use of the device 1050. Furthermore, secure applications may be provided via a SIMM card, along with additional information, such as placing identifying information on the SIMM card in an unhackable manner.

[0096] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one embodiment, a computer program product may be embodied on an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is, for example, a computer- or machine-readable medium, such as memory 1064, expansion memory 1074, or memory on processor 1052, which may be received via transceiver 1068 or external interface 1062.

[0097] Device 1050 can communicate wirelessly via communication interface 1066, which may include digital signal processing circuitry as needed. Communication interface 1066 can provide communication in various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS. Such communication occurs, for example, via radio frequency transceiver 1068. Additionally, short-range communication may occur, such as using Bluetooth, Wi-Fi, or other such transceivers (not shown). Additionally, GPS (Global Positioning System) receiver module 1070 may provide additional navigation- and location-related wireless data to device 1050, which may be used as appropriate by applications executing on device 1050.

[0098] Device 1050 may also communicate voice using voice codec 1060, which can receive voice information from a user and convert it into usable digital information. Voice codec 1060 may also generate audible sounds for the user, such as through a speaker in a handset of device 1050. Such sounds may include sounds from voice calls, recorded sounds (e.g., voice messages, music files, etc.), and sounds generated by applications running on device 1050.

[0099] The computing device 1050, as shown, may be implemented in many different forms. For example, the computing device 1050 may be implemented as a mobile phone 1080. The computing device 1050 may also be implemented as part of a smartphone 1082, a personal digital assistant, or other similar mobile device.

[0100] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may be special purpose or general purpose, and may include implementation in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0101] These computer programs (also known as programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0102] To provide for user interaction, the systems and techniques described herein can be implemented on a computer that has a display device (such as an LED (light-emitting diode), or OLED (organic LED), or LCD (liquid crystal display) monitor / screen) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction. For example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user can be received in any form, including acoustic input, voice input, or tactile input.

[0103] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0104] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0105] In some implementations, the illustrated computing device may include sensors that interface with the AR headset / HMD device 1090 to generate an augmented environment for viewing inserted content within a physical space. For example, one or more sensors included in the computing device 1050 or other illustrated computing devices may provide input to the AR headset 1090 or generally to the AR space. The sensors may include, but are not limited to, a touchscreen, an accelerometer, a gyroscope, a pressure sensor, a biometric sensor, a temperature sensor, a humidity sensor, and an ambient light sensor. The computing device 1050 may use the sensors to determine the absolute position and / or detected rotation of the computing device in the AR space, which may then be used as input to the AR space. For example, the computing device 1050 may be incorporated into the AR space as a virtual object such as a controller, a laser pointer, a keyboard, a weapon, or the like. When incorporated into the AR space, a user's positioning of the computing device / virtual object may allow the user to position the computing device to view the virtual object in a particular manner in the AR space.

[0106] In some implementations, one or more input devices included in or connected to the computing device 1050 can be used as input to the AR space. The input devices can include, but are not limited to, a touchscreen, a keyboard, one or more buttons, a trackpad, a touchpad, a pointing device, a mouse, a trackball, a joystick, a camera, a microphone, earphones or buds with input capabilities, a gaming controller, or other connectable input devices. When the computing device is integrated into the AR space, a user interacting with an input device included in the computing device 1050 can cause certain actions to occur in the AR space.

[0107] In some implementations, the touchscreen of the computing device 1050 may be rendered as a touchpad in the AR space. A user can interact with the touchscreen of the computing device 1050. The interaction is rendered, for example, in the AR headset 1090, as movements on the rendered touchpad in the AR space. The rendered movements can control virtual objects in the AR space.

[0108] In some implementations, one or more output devices included in the computing device 1050 can provide output and / or feedback to a user of the AR headset 1090 in the AR space. The output and feedback can be visual, tactile, or audio. The output and / or feedback can include, but is not limited to, vibration, turning one or more lights or strobes on / off or blinking and / or flashing, sounding an alarm, playing a chime, playing a song, and playing an audio file. Output devices include, but are not limited to, vibration motors, vibration coils, piezoelectric devices, electrostatic devices, light-emitting diodes (LEDs), strobes, and speakers.

[0109] In some implementations, the computing device 1050 may appear as another object in the computer-generated 3D environment. A user's interaction with the computing device 1050 (e.g., rotating, shaking, touching the touchscreen, swiping a finger across the touchscreen) may be interpreted as an interaction with an object in the AR space. In the example of a laser pointer in the AR space, the computing device 1050 appears as a virtual laser pointer in the computer-generated 3D environment. As the user manipulates the computing device 1050, the user in the AR space sees the movement of the laser pointer. The user receives feedback from their interaction with the computing device 1050 in the AR environment on the computing device 1050 or on the AR headset 1090. The user's interaction with the computing device may be translated into an interaction with a user interface generated in the AR environment of the controllable device.

[0110] In some implementations, the computing device 1050 may include a touchscreen. For example, a user can interact with the touchscreen to interact with a user interface for the controllable device. For example, the touchscreen may include user interface elements, such as sliders, that can control properties of the controllable device.

[0111] Computing device 1000 is intended to represent various forms of digital computers and devices, including, but not limited to, laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 1050 is intended to represent various forms of mobile devices, such as personal digital assistants, mobile phones, smartphones, and other similar computing devices. The components, their connections and relationships, and their functions depicted herein are intended to be exemplary only and are not intended to limit the practice of the subject matter described and / or claimed herein.

[0112] A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the present invention.

[0113] Additionally, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. Furthermore, other steps may be provided or removed from the described flows, and other components may be added to or removed from the described systems. Accordingly, other embodiments are within the scope of the following claims.

[0114] In addition to the above, users may be provided with controls that allow them to make choices both when the systems, programs, or functionality described herein enable the collection of user information (e.g., information about the user's social networks, social actions or activities, occupation, user preferences, or the user's current location) and when the user is sent content or communications from the server. Furthermore, certain data may be processed in one or more ways so that personally identifiable information is removed before it is stored or used. For example, the user's identity may be processed so that personally identifiable information is not determined about the user, and the user's geographic location may be generalized where the location information is obtained (e.g., to the city, zip code, state level) so that the user's specific location is not determined. Thus, users can control what information is collected about them, how that information is used, and what information is provided to them.

[0115] While certain features of the described embodiments have been described herein, many modifications, substitutions, changes, and equivalents will occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and variations as fall within the scope of the embodiments. These have been presented by way of example only, and not limitation, and it should be understood that various changes in form and detail may be made. Any portion of the apparatus and / or methods described herein may be combined in any combination, except in mutually exclusive combinations. The embodiments described herein may include various combinations and / or subcombinations of the functions, components, and / or features of the different embodiments described.

Claims

1. Analyzing the size of text characters in a first image having a first resolution; triggering capture of a second image at a second resolution greater than the first resolution based on the analysis of the size of the text characters in the first image; analyzing the size of text characters in the second image; Based on the analysis of the size of the text characters in the second image and in response to the second resolution satisfying a given condition, transmitting the second image from the computing device to the companion device for optical character recognition (OCR) processing by the companion device; An image processing method comprising:

2. The image processing method of claim 1 , wherein the first image further includes a real-world object.

3. An image processing method as described in claim 1, wherein the size of the text characters is analyzed to determine at least one of the following conditions: number of pixels per character, height, and area of ​​the text characters.

4. An image processing method described in any one of claims 1 to 3, further comprising capturing the first image at the first resolution using an image sensor, the first resolution corresponding to a bin defined by the image sensor.

5. 4. The image processing method according to claim 1, wherein the first resolution is a default resolution and the second resolution is defined based on entries in a look-up table.

6. comprising instructions that, when read and executed by one or more processors, cause the one or more processors to perform an image processing method; The image processing method includes: Analyzing the size of text characters in a first image having a first resolution; triggering capture of a second image at a second resolution greater than the first resolution based on the analysis of the size of the text characters in the first image; analyzing the size of text characters in the second image; Based on the analysis of the size of the text characters in the second image and in response to the second resolution satisfying a given condition, transmitting the second image from the computing device to the companion device for optical character recognition (OCR) processing by the companion device; a computer program comprising:

7. The computer program product of claim 6 , wherein the first image further includes a real-world object.

8. A computer program as described in claim 6, wherein the size of the text characters is analyzed to determine at least one of the following conditions: number of pixels per character, height, and area of ​​the text characters.

9. A computer program described in any one of claims 6 to 8, wherein the image processing method further includes capturing the first image at the first resolution using an image sensor, the first resolution corresponding to a bin defined by the image sensor.

10. The computer program product of any one of claims 6 to 8, wherein the first resolution is a default resolution and the second resolution is defined based on entries in a look-up table.

11. one or more processors; communicatively coupled to the one or more processors, and when executed by the one or more processors, causing the one or more processors to: Analyzing the size of text characters in a first image having a first resolution; triggering capture of a second image at a second resolution greater than the first resolution based on the analysis of the size of the text characters in the first image; analyzing the size of text characters in the second image; Based on the analysis of the size of the text characters in the second image and in response to the second resolution satisfying a given condition, transmitting the second image from the computing device to the companion device for optical character recognition (OCR) processing by the companion device; and a memory device storing a plurality of instructions for causing the execution of the

12. The system of claim 11 , wherein the first image further includes a real-world object.

13. The system described in claim 11, wherein the size of the text characters is analyzed to determine at least one of a pixel number condition per character, a height condition, and an area condition for the text characters.

14. A system as described in any one of claims 11 to 13, wherein the plurality of instructions further cause the one or more processors to capture the first image at the first resolution using an image sensor, the first resolution corresponding to a bin defined by the image sensor.

15. The system of any one of claims 11 to 13, wherein the first resolution is a default resolution and the second resolution is defined based on entries in a look-up table.

Images