Dynamic capture parameter processing for low power
The system optimizes image processing in mobile devices by capturing at reduced resolution and offloading complex tasks, addressing power and resource inefficiencies in computer vision tasks.
Patent Information
- Application Number
- KR1020247007934
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2022-08-11
- Filing Date
- 2022-08-12
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2042-08-12
AI Technical Summary
Computer vision techniques in mobile computing devices face challenges in balancing performance and power consumption, leading to resource overextension and inefficiencies in image processing tasks.
A system and method for image processing that reduces resource usage by capturing images at initial low resolution, determining target capture parameters, and offloading complex processing to companion devices, utilizing neural networks and algorithms to perform tasks like OCR efficiently.
Enables low-power computing devices to perform computer vision tasks effectively by minimizing resource consumption and offloading complex processing, ensuring accurate image processing without overwhelming device resources.
Smart Images

Figure 112024026473620-PCT00001_ABST
Abstract
Description
Technology Field
[0001] Related applications
[0002] This application is a continuation of U.S. applications No. 17 / 819,170 and 17 / 819,154 filed on August 11, 2022, and claims priority thereto, and both applications claim priority to U.S. provisional applications No. 63 / 260,206 and 63 / 260,207 filed on August 12, 2021. The contents of all of these applications are incorporated herein by reference in their entirety.
[0003] This application claims priority to U.S. Provisional Application No. 63 / 260,206 filed August 12, 2021 and U.S. Patent Application No. 63 / 260,207 filed August 12, 2021, the contents of which are incorporated herein by reference in their entirety.
[0004] This specification relates to methods, devices, and algorithms generally used for processing image content. Background Technology
[0005] Computer vision techniques enable computers to analyze and extract information from images. These techniques can be costly in terms of processing and power consumption. As the demand to balance the performance and power consumption of mobile computing devices continues to grow, device manufacturers must configure devices to strike a balance between image degradation and device performance to avoid overextending the limits of mobile computing devices.
[0006] One or more computer systems may be configured to perform a specific operation or action by software, firmware, hardware, or a combination thereof installed in the system so as to cause the system to perform an action while in operation. One or more computer programs may be configured to perform a specific operation or action by including instructions that cause said device to perform an operation when executed by said device.
[0007] The present invention describes a system and method for performing image processing that enables a computing device (e.g., a wearable computing device) to efficiently perform computer vision tasks. The system and method described herein can reduce the use of specific hardware and / or device resources (e.g., memory, processor, network bandwidth, etc.) by reducing (e.g., minimizing) the amount of image data used when performing computer vision and image analysis tasks. Configuring a computing device (e.g., a head-mounted device (HMD), augmented reality (AR) smart glasses) to reduce resource usage can provide the advantage of allowing such a device to perform relatively low-power computing tasks onboard the device and offload (e.g., push) complex and / or high-power processing to a companion device (e.g., a server, a mobile device connected to the computing device, another computer, etc.). For example, an initial evaluation of the feasibility of OCR (Optical Character Recognition) can be performed on a computing device (e.g., HMD), and if it is determined that an image can be captured with desired capture parameters (e.g., sufficient resolution) to perform OCR, the image can be transmitted to a companion device (e.g., wireless transmission) for additional (e.g., more detailed and intensive) OCR processing.
[0008] In at least one general embodiment, an image processing method may include the step of capturing a first raw image at a first resolution using an image sensor; the step of converting the first raw image into a digitally processed image using an image signal processor; and the step of analyzing at least a portion of the digitally processed image based on processing conditions. The method may include the step of determining that the first resolution does not satisfy the processing conditions; and the step of triggering the capture of a second raw image at a second resolution higher than the first resolution from the image sensor. The method may include any combination of the following.
[0009] In another general embodiment, a non-transient machine-readable type medium storing instructions read and executed by one or more processors to perform an image processing method, wherein the method may include the steps of: capturing a first raw image at a first resolution using an image sensor; converting the first raw image into a digitally processed image using an image signal processor; and analyzing at least a portion of the digitally processed image based on processing conditions. The method may include the steps of: determining that the first resolution does not satisfy the processing conditions; and triggering the capture of a second raw image at a second resolution higher than the first resolution from the image sensor. The method may include any combination of the following.
[0010] In another general embodiment, the system may include a processor and a memory device communicationly connected to the processor and storing a plurality of instructions, wherein when the instructions are executed by the processor, the processor causes: to capture a first raw image at a first resolution using an image sensor; to convert the first raw image into a digitally processed image using an image signal processor; and to analyze at least a portion of the digitally processed image based on processing conditions. The processor may determine that the first resolution does not satisfy the processing conditions and trigger the capture of a second raw image from the image sensor at a second resolution higher than the first resolution.
[0011] Each embodiment described above may include any combination of the following. In some embodiments, at least a portion of the digitally processed image includes text characters. In some embodiments, at least a portion of the digitally processed image includes real-world objects. In some embodiments, the processing condition is at least one of pixels per character, height, or area conditions. In some embodiments, the first resolution for capturing the first raw image corresponds to a bin defined in the image sensor. In some embodiments, the first resolution is a base resolution, and the second resolution is defined based on an entry in a lookup table. In some embodiments, the digitally processed image is the first digitally processed image captured by the computing device. Additionally, the method may further include the steps of converting the second raw image into a second digitally processed image, determining whether the second resolution satisfies the processing condition, and transmitting the second digitally processed image to a companion device for the computing device for further processing in response to the determination that the second resolution satisfies the processing condition. Brief explanation of the drawing
[0012] FIG. 1 is an example of a computing device for generating and processing image content using limited computation and / or power resources, according to an embodiment described throughout the present disclosure. FIG. 2 illustrates a system for performing image processing on a computing device according to an embodiment described throughout the present disclosure. FIGS. 3a to 3c illustrate signs with printed words that can be seen within an HMD display in various scenarios. FIGS. 4a and 4b illustrate examples of wearable computing devices according to embodiments described throughout this specification. FIG. 5 is a flowchart illustrating a method related to an embodiment described in this specification. FIG. 6 illustrates examples of computer devices and mobile computer devices that can be used with the techniques described in this specification. In various drawings, symbols represent the same element. Specific details for implementing the invention
[0013] The present invention describes a system and method for performing image processing that enables a computing device (e.g., a wearable computing device) to efficiently perform computer vision tasks. For example, the system and method described herein may utilize a processor, a sensor, a neural network, and / or an image analysis algorithm to recognize text, symbols, and / or objects in an image and / or extract information from the image. In some embodiments, the system and method described herein may perform these tasks while operating in a reduced computation mode and / or a reduced power mode. For example, the system and method described herein may reduce the use of specific hardware and / or device resources (e.g., memory, processor, network bandwidth, etc.) by reducing (e.g., minimizing) the amount of image data used when performing computer vision and image analysis tasks.
[0014] Configuring computing devices (e.g., head-mounted devices (HMDs), augmented reality (AR) smart glasses) to reduce resource usage can provide the advantage of allowing these devices to perform relatively low-power computing tasks onboard and offload (e.g., push) complex and / or high-power processing to companion devices (e.g., servers, mobile devices connected to the computing device, other computers, etc.).
[0015] For example, an initial evaluation of the feasibility of OCR (Optical Character Recognition) can be performed on a computing device (e.g., HMD), and if it is determined that an image can be captured with desired capture parameters (e.g., sufficient resolution) to perform OCR, the image can be transmitted to a companion device (e.g., wireless transmission) for additional (e.g., more detailed and intensive) OCR processing.
[0016] As another example, one or more capture parameters (e.g., resolution, pixels per character) for performing an initial stage of OCR (e.g., an initial stage of an OCR pipeline such as text recognition) may be performed on a computing device. If it is determined that an image can be captured with desirable capture parameters for performing OCR, the image may be transmitted (e.g., wirelessly) to a companion device for detailed OCR processing. In some embodiments, detailed OCR processing (or other processing) may be performed on a computing device, and the technique described herein may help conserve energy during at least some (e.g., any) of downstream processing.
[0017] In some embodiments, initial low-power processing of the image may be used to determine target capture parameters (e.g., minimum resolution, region of interest (ROI) cropping) for capturing an image that can be transmitted to another device for higher-power and / or more complex processing. Specific capture parameters may be determined based on one or more processing conditions. The initial processing for determining the target capture parameters may be referred to as the capture parameter selection processing step, and subsequent processing may be referred to as the downstream image processing step.
[0018] As a specific example, at least a portion of the image may be captured by a first capture parameter, and in the capture parameter selection processing step, a decision may be made that the first capture parameter does not satisfy a condition (e.g., is not sufficiently high) based on a processing condition to be processed in a subsequent processing step. The processing condition may be a condition used to determine whether processing in a later processing step (e.g., downstream processing) can be performed in a desirable manner. In response to the decision that the first solution is insufficient, a portion of the image may be captured by a second capture parameter, and a decision may be made that the second capture parameter is sufficient to be processed in a later processing step based on the processing condition. The downstream image processing step may be processing related to, for example, optical character recognition.
[0019] As another specific example, at least a portion of the image may be captured in a first capture parameter, and in the capture parameter selection processing step, a decision may be made that the first capture parameter satisfies a condition (e.g., sufficiently high) based on a processing condition to be processed in a subsequent processing step. In response to the determination of sufficiency, a portion of the image may be processed in a downstream image processing step (e.g., processing related to optical character recognition).
[0020] In some embodiments, the capture parameter selection processing may be based on a specific region of interest determined using a separate region of interest selection technique. For example, instead of analyzing the entire image, regions of interest (ROIs) may be identified in the image data, and one or more of these regions may be analyzed on the computing device. By reducing the amount of information to be analyzed while maintaining accurate results, complex image processing tasks can be performed on the computing device without the resource burden of analyzing the entire image. In some embodiments, the system and method described herein may be configured to run on the device so that the device can perform efficient image processing tasks while reducing computational load and / or power. For example, the system and method described herein may enable the device to perform object detection tasks, Optical Character Recognition (OCR) tasks, and / or other image processing tasks while utilizing specific techniques to reduce power, memory, and / or processing consumption.
[0021] In some embodiments, the system and method described herein can ensure that complex image processing calculations can be executed on a device without the assistance of other resources and / or devices. For example, an existing system may request support from other mobile devices, servers, and / or off-board systems that are communicably connected to perform computationally intensive image processing tasks. The system and method described herein provide the advantage of generating portions of an image that can be operated by a device with less computational power than a server, for example, while providing complete and accurate image processing capabilities.
[0022] The systems and methods described herein may enable a computing device to use machine learning intelligence (e.g., neural networks, algorithms, etc.) that utilizes low power consumption and / or low processing consumption to perform computer vision tasks such as object detection, motion tracking, face recognition, OCR tasks, etc.
[0023] FIG. 1 is an example of a computing device (100) (e.g., HMD) for generating and processing image content using limited computation and / or power resources, according to an embodiment described throughout the present disclosure. In this example, the computing device (100) is illustrated in the form of AR smart glasses. However, any form factor may be substituted and combined with the systems and methods described herein. In some embodiments, the computing device (100) includes a system-on-chip (SOC) architecture (not illustrated) combined with any or all of one or more sensors, a low-power island processor, a high-power island processor, a core processor, an encoder, etc.
[0024] When in operation, the computing device (100) (also referred to as the device) may capture an image of the scene (102) using, for example, a camera, an image sensor, etc. The device (100) may be worn and operated by a user (104). The scene (102) may include not only physical content but also augmented reality (AR) content. The scene (102) is illustrated in terms of a view through the device (100). In some embodiments, the computing device (100) may be coupled to communicate with a companion device (106) (e.g., a mobile computing device).
[0025] In some embodiments, the computing device (100) may be triggered to start real-time image processing (e.g., real-time image and / or character recognition) in response to a request to identify image content from a scene (102). In non-limiting examples, the computing device (100) may be triggered to start real-time image processing in response to a request to identify image content associated with optical data captured by a sensor included in the device (100). The request may be from a user wearing the device (100) (e.g., user (104)).
[0026] When in operation, the computing device (100) may include a capture parameter selection processing step (100-1). In the capture parameter selection processing step (100-1), initial low-power processing of the image may be used to determine target capture parameters (e.g., minimum resolution, cropping the region of interest (ROI)) for capturing at least some image of the desired scene (102) for additional processing (or triggering additional processing). The target capture parameters determined in the capture parameter selection processing step (100-1) may be used to capture one or more additional images that can be transmitted to the companion device (106) for higher-power and / or more complex processing within the downstream image processing step (100-2). Although the downstream image processing step (100-2) is depicted as being in the companion device (106), any of the processing of the downstream image processing step (100-2) may be performed in the computing device (100).
[0027] A target capture parameter for capturing an image of at least a portion of a scene (102) may be determined based on one or more processing conditions (101) in a capture parameter selection processing step (100-1). The target capture parameter may be a capture parameter optimized (based on processing conditions) for processing an image in a downstream image processing step (100-2). Specifically, the target capture parameter may be selected (e.g., converged) based on one or more of the processing conditions (101) being satisfied.
[0028] In some embodiments, the target capture parameter may be selected based on the initial capture parameter for capturing content. In some embodiments, the target capture parameter may be the same as the initial capture parameter or may be identified as the same. In some embodiments, in addition to the initial capture parameter, a number of intermediate capture parameters may be tested until the target capture parameter is identified (e.g., until convergence). (e.g., considered for one or more processing conditions). At least one of the intermediate capture parameters may be identified as the target capture parameter if the intermediate capture parameter satisfies one or more of the processing conditions (101).
[0029] In some embodiments, the content processed by the capture parameter selection processing step (100-1) may be targeted to a region of interest (130). In some embodiments, the region of interest (e.g., a region of interest refined by cropping) may be selected together with or after the target capture parameter processing. In some embodiments, the region of interest may be selected or processed at any element of the capture parameter selection processing step (100-1) (e.g., an image sensor, an image signal processor, on-device processing, etc.).
[0030] As a specific example, text within a scene (102) can be captured in a first image using an initial capture parameter representing a first resolution (low resolution, low resolution level). The capture parameter selection processing step (100-1) may determine that the text within the first image does not satisfy the processing condition (101) (e.g., the characters of the text are below a pixel threshold per character (e.g., pixel height per character). Based on the analysis of the first image (e.g., text) by the capture parameter selection processing step (100-1), since the processing condition (101) is not satisfied, an intermediate capture parameter representing a second resolution (high resolution, high resolution level) is used to capture a second image of the text within the scene. If the capture parameter selection processing step (100-1) determines that the text in the second image satisfies the processing condition (101) (e.g., the characters of the text satisfy or exceed a pixel threshold per character), the second image of the captured text is transmitted to a companion device for processing (e.g., OCR processing) within the downstream image processing step (100-2).
[0031] In some embodiments, multiple images may be captured simultaneously or nearly simultaneously with different capture parameters. An evaluation may be performed to determine whether a first image among the multiple images satisfies one or more processing conditions (101). The first image among the multiple images may be an image that uses less processing resources (e.g., uses the least processing resources among the multiple images). If the first image among the multiple images satisfies one or more processing conditions (101), further evaluation of other images among the multiple images may not be necessary. If the first image among the multiple images does not satisfy one or more processing conditions (101), an evaluation may be performed to determine whether a second image among the multiple images satisfies one or more processing conditions (101). The second image among the multiple images may be an image that uses more processing resources (e.g., progressively more) than the first image among the multiple images. This processing of the multiple images may be performed until one of the multiple images is identified as satisfying one or more processing conditions (101) while using the minimum processing resources (from the multiple images).
[0032] FIG. 2 illustrates a system (200) for performing image processing in a computing device (100) according to an embodiment described throughout the present disclosure. In some embodiments, image processing may be performed in the computing device (100). In some embodiments, image processing is shared among one or more devices. For example, image processing may be partially completed in the computing device (100) and partially completed in a companion device (202) (e.g., a mobile computing device (106)) and / or a server computing device (204). In some embodiments, image processing is performed in the computing device (100), and the output from such processing is provided to the companion device (202) and / or the server computing device (204).
[0033] FIG. 2 illustrates an example of an implementation of a capture parameter selection processing step (100-1) included in a computing device (100). Although not illustrated, a downstream image processing step (100-2) may be included in a companion device (202). In this embodiment, the capture parameter selection processing step (100-1) includes an image sensor (216), an image signal processor (224), and a parameter selection processor (227).
[0034] The image signal processor (224) may be configured to convert a raw image (e.g., raw image, analog image) captured by the image sensor (216) into a digitally processed image format that can be processed by a parameter selection processor (227). For example, the raw image may be an uncompressed image that lacks some information for further image processing. The image signal processor (224) may process the raw image into a digital format that can be further processed by downstream processing, such as by a parameter selection processor (227). The image signal processor (224) may generate a digitally processed image by performing, for example, noise reduction, automatic white balance and color correction, color interpolation, gamma correction, etc. In some embodiments, the image signal processor (224) may be part of the sensor system (214). For the sake of simplicity, the conversion of an image generated by an image sensor (216) into a digital representation by an image signal processor (224) is generally omitted from the description, even though such conversion occurs before downstream processing (e.g., processing by a parameter selection processor (227)). Accordingly, processing by downstream components is generally performed after digital processing by the image signal processor (224), even if not mentioned in the description of this specification.
[0035] The embodiment illustrated in FIG. 2 is discussed in the context of the resolution associated with the OCR process (e.g., resolution capture parameter (221), resolution condition (222), resolution selection algorithm (223)), and the principles of this description will be explained with a focus on the discussion. Various types of capture parameters, conditions, and / or processes other than or in addition to resolution may be used in connection with at least the embodiment of FIG. 2. For example, the techniques described herein may be applied to the processing of any type of real object (e.g., computer vision, machine learning) such as (but not limited to) barcodes, products, signs, storefronts, landmarks, plants, food, faces, hands (e.g., pointing or holding), objects (e.g., cars, buildings, bicycles, books, people, etc.), feature point tracking, and / or others.
[0036] The image sensor (216), image signal processor (224), and parameter selection processor (227) can function as a pipeline within the capture parameter selection processing step (100-1). To reduce or minimize energy consumption, the image resolution is reduced or minimized as much as possible at the beginning of the pipeline. For example, a minimum resolution image can be captured from the image sensor (216) so that processing in any downstream processing (e.g., image signal processor (224), parameter selection processor (227), communication module (252), companion device (202)) can be reduced or minimized. Capturing an image from the image sensor (216) at a higher resolution can result in increased power consumption in downstream processing and increased latency (e.g., transmission latency).
[0037] As described above, the capture parameter selection processing step (100-1) may be used to determine target resolution capture parameters for capturing images of at least a portion of the scene (102) that are desirable for additional processing (and power saving). The target resolution capture parameters determined in the capture parameter selection processing step (100-1) may be used to capture one or more additional images that can be transmitted to a companion device (202) for higher power and / or more complex OCR processing within a downstream image processing step.
[0038] The resolution capture parameter (221) may represent a parameter that can be used to capture an image, for example, by an image sensor (216). Although not illustrated, each different type of resolution capture parameter (e.g., initial, intermediate, and target resolution capture parameters) may be represented or generally referred to as the resolution capture parameter (221). For example, the initial or basic resolution capture parameter may be referred to as the resolution capture parameter (221). The resolution capture parameter (221) may represent various values or levels of resolution, for example.
[0039] The capture parameter selection processing step (100-1) may be configured to start image capture using one or more initial capture parameters using an image sensor (216), and may adjust and / or define additional resolution capture parameter(s) (e.g., one or more intermediate capture parameters) until one or more target resolution capture parameters for additional processing are determined. In some embodiments, the initial capture parameters may be identified as target resolution capture parameters.
[0040] For example, the image sensor (216) may be configured to capture an image using an initial resolution capture parameter (e.g., a primary resolution capture parameter at a first resolution). The parameter selection processor (227) may be configured to determine whether the content captured by the parameter selection processor (227) has sufficient resolution for further processing by analyzing at least a portion of the content within the image. Sufficiency of resolution may be determined based on one or more of the resolution processing conditions (222). Specifically, if the content of the image satisfies one or more of the resolution processing conditions (222) (e.g., if the resolution is sufficient for further processing), the parameter selection processor (227) identifies the initial resolution capture parameter as a target resolution capture parameter for capturing image content using the image sensor (216).
[0041] As another example, the image sensor (216) may be configured to capture an image using an initial resolution capture parameter (e.g., at a first resolution). The parameter selection processor (227) may be configured to determine whether the content captured by the parameter selection processor (227) has sufficient resolution for further processing by analyzing at least a portion of the content within the image. If the resolution is not sufficient for further processing, the parameter selection processor (227) identifies an intermediate resolution capture parameter.
[0042] After an initial resolution capture parameter is determined not to be suitable for capturing an image by an image sensor (216) for downstream processing based on one or more resolution processing conditions (222), a resolution selection algorithm (223) may be used to determine one or more intermediate resolution capture parameters until a target resolution capture parameter is identified. The resolution algorithm (223) may be based on a predetermined set of resolution capture parameters. The resolution algorithm (223) may be based on a predetermined order of resolution capture parameters. The resolution algorithm (223) may be based on identifying a resolution capture parameter that results in minimal processing.
[0043] The resolution selection algorithm (223) may be, for example, a lookup table (LUT) that identifies the order of resolution capture parameters after the initial resolution capture parameter, or may be based thereon. That is, different capture parameters (e.g., capture parameter levels) may be determined using the lookup table. The desired resolution capture parameter (or other types of capture parameters) can be quickly determined using the lookup table.
[0044] For example, the lookup table may include initial resolution capture parameters (e.g., base resolution capture parameters). If the first image captured based on the initial resolution capture parameters is not acceptable for further processing (e.g., downstream processing), the second image may be captured based on intermediate resolution capture parameters identified within the lookup table. If the second image captured based on the intermediate resolution capture parameters is acceptable for further processing, further processing may be performed (and the intermediate resolution capture parameters may be final resolution capture parameters). If the second image captured based on the intermediate resolution capture parameters is not acceptable for further processing (e.g., downstream processing), the third image may be captured based on another intermediate resolution capture parameter identified within the lookup table. As described above, processing may proceed in an iterative manner until the final resolution capture parameters are identified. Parameters other than resolution capture parameters may be identified.
[0045] In some embodiments, the resolution selection algorithm (223) may be based on various algorithms and / or formulas.
[0046] A computing device (100) includes a sensor system (214). The sensor system (214) includes an image sensor (216) configured to detect and / or acquire image data. In some embodiments, the sensor system (214) includes a plurality of image sensors (216). As illustrated, the sensor system (214) includes one or more sensors (216). The image sensor (216) can capture and record images (e.g., pixels, frames, and / or parts of an image) and video.
[0047] In some embodiments, the resolution (e.g., resolution level) of the image sensor (216) may be defined through predefined (e.g., pre-programmed) binning (e.g., pixel binning) of the image sensor (216). For example, binning an increased number of pixels of the image sensor (216) may decrease the resolution or resolution level. Binning an increased number of pixels may increase the resolution or resolution level.
[0048] The satisfaction of a per-pixel threshold, which is a type of processing condition (e.g., processing condition (101)), can depend on many factors, including distance and the actual image element size (e.g., actual font size). Although the technique described herein can be applied to all types of images or image elements, for convenience, the technique described herein is described in relation to fonts.
[0049] For example, an image (e.g., raw image) that captures characters using a relatively large font but is far from the image sensor (216) can still satisfy the threshold per pixel because the relatively large font size can be interpreted from a relatively far distance from the image sensor (216). Such an example is illustrated in FIG. 3a.
[0050] As illustrated in FIG. 3a, a sign (350-1) (e.g., a real sign) with the word "the" printed on it can be seen within an HMD display (310) (e.g., a display window of pixel size P1 x P2) at a distance D1. The field of view (320) associated with the outward-facing image sensor of the HMD (e.g., the image sensor (216) shown in FIG. 2, not shown) can generate an image (e.g., a raw image) within the HMD display (310). In this example, because the size of the word "the" on the sign is relatively large at a distance D1, the size of the word within the HMD display (310) is also relatively large. The size of the pixel count Q1 of the letter "t" in the word "the" (e.g., height in this example) can be larger than a threshold (e.g., sufficiently large), allowing additional processing to be performed in a desirable manner in a downstream process (e.g., the downstream image processing step (100-2) shown in FIG. 1).
[0051] Referring again to FIG. 2, as another example, an image (e.g., a raw image) that captures characters with a relatively small font and is located far from the image sensor (216) may not satisfy the threshold per pixel because the relatively small size may not be interpreted from the image sensor (216) from a relatively long distance. Such an example, which is a variation of FIG. 3a, is illustrated in FIG. 3b.
[0052] As illustrated in FIG. 3b, a sign (350-2) (e.g., a real sign) with the word "the" printed on it can be seen within an HMD display (310) (e.g., a display window of pixel size P1 x P2) at a distance D1 (the same distance as in FIG. 3a). A field of view (320) associated with an outward-facing image sensor of an HMD (not shown) can generate an image (e.g., a raw image) within the HMD display (310). In this example, because the size of the word "the" on the sign is relatively small at a distance D1, the size of the word within the HMD display (310) is also relatively small. The size of the pixel count Q2 of the letter "t" in the word "the" can be smaller than a threshold, preventing additional processing from being performed in a desirable manner in a downstream process (e.g., a downstream image processing step (100-2) illustrated in FIG. 1). In this situation, the number of pixels of the word "the" can be increased significantly compared to Q2, so that the capture resolution can be increased to enable additional processing to be performed in a desirable manner in the downstream process (e.g., the downstream image processing step (100-2) shown in FIG. 1).
[0053] In some embodiments, a low-resolution image (e.g., raw image) and a high-resolution image may be captured simultaneously or nearly simultaneously. An evaluation may be performed to determine whether the low-resolution image satisfies processing conditions (e.g., threshold per pixel). If the low-resolution image satisfies processing conditions, processing of the high-resolution image may not be performed or may be excluded. If the low-resolution image does not satisfies processing conditions, an evaluation may be performed to determine whether the high-resolution image satisfies processing conditions (e.g., threshold per pixel). If the high-resolution image satisfies processing conditions, the high-resolution image may be used (or triggered) for additional processing. If the high-resolution image does not satisfies processing conditions, a higher-resolution image may be captured, and an evaluation may be performed to determine whether the higher-resolution image satisfies processing conditions (e.g., threshold per pixel).
[0054] In some embodiments, processing may be initiated at a high processing level. For example, in some embodiments, a low-resolution image (e.g., raw image) and a high-resolution image (e.g., raw image) may be captured simultaneously or nearly simultaneously. An evaluation may be performed to determine whether the high-resolution image satisfies processing conditions (e.g., threshold per pixel). If the high-resolution image satisfies processing conditions, processing of the low-resolution image may not be performed or may be excluded.
[0055] In some embodiments, processing may start at a high processing level and then decrease until a desired condition is met. For example, in some embodiments, a low-resolution image and a high-resolution image may be captured simultaneously or nearly simultaneously. An evaluation may be performed to determine whether the high-resolution image satisfies a processing condition (e.g., a threshold per pixel). If the high-resolution image satisfies the processing condition, an evaluation may also be performed to determine whether the low-resolution image satisfies a processing condition (e.g., a threshold per pixel). If the low-resolution image satisfies the processing condition, the low-resolution image may be used (or triggered) for additional processing (instead of the high-resolution image). Images may be captured and evaluated until a resolution is captured that minimizes additional processing (determined by the processing condition).
[0056] As another example, an image (e.g., raw image) that captures characters close to the image sensor (216) using a relatively small font can satisfy the per-pixel threshold because it can be interpreted at a relatively short distance even though the font is small. Such an example, a variation of FIG. 3b, is illustrated in FIG. 3c.
[0057] As illustrated in FIG. 3c, a sign (350-3) (e.g., a real sign) with the word "the" printed on it can be seen within an HMD display (310) (e.g., a display window of pixel size P1 x P2) at a distance D2 (closer distance in FIG. 3a and 3b). A field of view (320) associated with an outward-facing image sensor of an HMD (not shown) can generate an image (e.g., a raw image) within the HMD display (310). In this example, because the size of the word "the" on the sign is relatively large at distance D2, the size of the word within the HMD display (310) is also relatively large. The size of the pixel count Q3 of the letter "t" in the word "the" (e.g., height in this example) can be larger than a threshold (e.g., sufficiently large), allowing additional processing to be performed in a desirable manner in a downstream process (e.g., a downstream image processing step (100-2) illustrated in FIG. 1).
[0058] As illustrated in the example above, the per-pixel threshold can be a powerful tool for determining the appropriate capture resolution in many situations. The per-pixel threshold can be a single metric that can be determined without information regarding, for example, font size (e.g., actual font size) and / or distance.
[0059] In some embodiments, the per-pixel threshold may be determined based on the script or language of the text information. In some embodiments, one or more processing conditions (101) may be based on the angular resolution or visual acuity of the system. The angular resolution may be based on the field of view of an image sensor (e.g., camera) that captures an image (e.g., raw image). In some embodiments, one or more processing conditions (101) may be based on sharpness, contrast, accuracy, reliability of character identification, etc.
[0060] In some embodiments, the image sensor (216) is an RGB (Red Green Blue) camera. In some examples, the image sensor (216) includes a pulse laser sensor (e.g., a LiDAR sensor) and / or a depth camera. For example, the image sensor (216) may be a camera configured to detect and transmit information used to create an image displayed as an image frame (226). The image sensor (216) may capture and record both images and videos.
[0061] When in operation, the image sensor (216) is configured to acquire (e.g., capture) image data (e.g., light sensor data) continuously or periodically while the device (100) is active. In some embodiments, the image sensor (216) is configured to operate as a sensor that is always on. In some embodiments, the imaging sensor (216) may be activated in response to the detection of an object of interest or a region of interest.
[0062] In some embodiments, the computing device (100) comprises one or more computing devices, wherein at least one of the devices is a display device that can be worn on or near the skin of a person. In some embodiments, the computing device (100) is or comprises one or more computing device components. In some embodiments, the computing device (100) may include a head-mounted display (HMD) device such as an optical head-mounted display (OHMD) device, a transparent head-up display (HUD) device, a virtual reality (VR) device, or an AR device, or other devices such as goggles or a headset having sensors, displays, and computing functions. In some embodiments, the computing device (100) comprises AR glasses (e.g., smart glasses). AR glasses represent an optical head-mounted display device designed in the shape of glasses. In some embodiments, the computing device (100) is or comprises a smartwatch. In some embodiments, the computing device (100) is or comprises a piece of jewelry. In some embodiments, the computing device (100) is or comprises a ring controller device or other wearable controller. In some embodiments, the computing device (100) is an earbud / headphone or a smart earbud / headphone or includes the same.
[0063] As illustrated in FIG. 2, the system (200) includes a computing device (100) that is communicably connected to a companion device (202) and optionally a server computing device (204). In some embodiments, the communicable connection may occur via a network (206). In some embodiments, the communicable connection may occur directly between the computing device (100), the companion device (202), and / or the server computing device (204).
[0064] A computing device (100) includes one or more processors (208) that may be formed on a substrate configured to execute one or more machine-executable instructions or pieces of software, firmware, or a combination thereof. The processors (208) may be semiconductor-based and may include semiconductor materials capable of performing digital logic. The processors (208) may include, for example, a CPU, a GPU, and / or a DSP.
[0065] The computing device (100) may also include one or more memory devices (210). The memory device (210) may include any type of storage device that stores information in a format that can be read and / or executed by the processor(s) (208). The memory device (210) may store applications and modules that cause specific actions to be performed when executed by the processor(s) (208). In some examples, applications and modules may be stored in an external storage device and loaded into the memory device (210). The memory (210) may include a buffer (212) or access the buffer (212) to store and retrieve image content and / or audio content for the device (100), for example.
[0066] The sensor system (214) may include an inertial motion unit (IMU) sensor (218). The IMU sensor (218) may detect motion, movement, and / or acceleration of the computing device (100). The IMU sensor (218) may include various different types of sensors, such as, for example, an accelerometer, a gyroscope, a magnetometer, and other such sensors.
[0067] In some embodiments, the sensor system (214) may include an audio sensor (220) configured to detect audio received by the computing device (100). The sensor system (214) may include other types of sensors, such as a light sensor, a distance and / or proximity sensor, a contact sensor such as a capacitive sensor, a timer and / or other sensors, and / or different combinations of sensors. The sensor system (214) may be used to obtain information related to the position and / or orientation of the computing device (100).
[0068] A computing device (100) may include one or more antennas (242) configured to communicate with other computing devices via wireless signals. For example, the computing device (100) may receive one or more wireless signals and use the wireless signals to communicate with other devices, such as a companion device (202) and / or a server computing device (204), or other devices within the range of the antenna (242). The wireless signals may be triggered via a wireless connection, such as a short-range connection (e.g., a Bluetooth connection or a near-field communication (NFC) connection) or an internet connection (e.g., Wi-Fi or a mobile network).
[0069] A computing device (100) includes a display (244). The display (244) may include a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED), an electrophoretic display (EPD), or a micro-projection display employing an LED light source. In some examples, the display (244) is projected into the user's field of vision. In some examples, for AR glasses, the display (244) may be a transparent or translucent display so that a user wearing AR glasses can see not only the image provided by the display (244) but also information located in the field of vision of the AR glasses behind the projected image.
[0070] The computing device (100) also includes a control system (246) comprising various control system devices that facilitate the operation of the computing device (100). The control system (246) may include a processor (208) and / or a sensor system (214) operably connected to a component of the computing device (100).
[0071] The computing device (100) includes a UI renderer (250). The UI renderer (250) functions in conjunction with a display (244) to display user interface objects or other content to a user of the computing device (100). For example, the UI renderer (250) may receive an image captured by the device (100) to create and render additional user interface content on the display (244).
[0072] The computing device (100) may include a communication module (252). The communication module (252) may enable the computing device (100) to communicate and exchange information with another computing device within the range of the device (100). For example, the computing device (100) may be operably connected to another computing device to facilitate communication, for example, through a wired connection, for example, a wireless connection via Wi-Fi or Bluetooth, or other types of connections.
[0073] In some embodiments, the computing device (100) may communicate with a server computing device (204) and / or a companion device (202) via a network (206). The server computing device (204) may represent one or more computing devices taking the form of a number of different devices, such as a standard server, a group of such servers, or a rack server system. In some embodiments, the server computing device (204) is a single system that shares components such as a processor and memory. The network (206) may include the Internet and / or other types of data networks, such as a local area network (LAN), a wide area network (WAN), a cellular network, a satellite network, or other types of data networks. The network (206) may also include any number of computing devices (e.g., computers, servers, routers, network switches, etc.) configured to receive and / or transmit data within the network (206).
[0074] FIGS. 4a and 4b illustrate various views of an example of an AR wearable computing device according to an embodiment described throughout this specification. FIG. 4a is a front view of an example of a wearable device according to an embodiment described throughout this disclosure. In this example, the wearable device is AR glasses (400A) (e.g., the wearable computing device (100) of FIG. 1). Generally, the AR glasses (400A) may include some or all components of the system (200). The AR glasses (400A) may be represented as smart glasses, which represent an optical head-mounted display device designed in the shape of glasses. For example, smart glasses are glasses that add information (e.g., display projection) to what the wearer sees through the glasses.
[0075] Although the wearable computing device described herein exemplifies AR glasses (400A), other forms of wearable computing devices are also possible. For example, a wearable computing device may include a head-mounted display (HMD) device such as an optical head-mounted display (OHMD) device, a transparent head-up display (HUD) device, or an augmented reality (AR) device, or other devices such as goggles or headsets having sensors, displays, and computing functions. In some examples, the wearable computing device may be a watch, a mobile device, jewelry, a ring controller, or other wearable controller.
[0076] As illustrated in FIG. 4a, AR glasses (400A) include a frame (402) having a display device (404) coupled to the frame (402) (or to the glass portion of the frame (402)). AR glasses (400A) also include an audio output device (406), a lighting device (408), a sensing system (410), a control system (412), at least one processor (414) and a camera (416).
[0077] The display device (404) may include a see-through near-eye display, such as one using a birdbath or a waveguide optical device. For example, such an optical design may project light from a display source onto a portion of teleprompter glass that functions as a beam splitter positioned at a 45-degree angle. The beam splitter may allow for reflection and transmission values such that the remaining light is transmitted while the light from the display source is partially reflected. Such an optical design may allow the user to see both physical items of the world next to a digital image (e.g., UI elements, virtual content, etc.) generated by the display. In some embodiments, waveguide optics may be used to depict content on the display device (404) of the AR glasses (400A).
[0078] An audio output device (406) (e.g., one or more speakers) may be coupled to the frame (402). The sensing system (410) may include a control system (412) comprising various sensing devices and various control system devices to facilitate the operation of the AR glasses (400A). The control system (412) may include a processor (414) operably connected to a component of the control system (412).
[0079] The camera (416) can capture still images and / or videos. In some embodiments, the camera (416) may be a depth camera capable of collecting data related to the distance of an external object from the camera (416). In some embodiments, the camera (416) may be a point tracking camera capable of detecting and following one or more optical markers on an external device, such as an optical marker on an input device or a finger on a screen. In some embodiments, the AR glasses (400A) may include a lighting device (408) that can optionally operate with the camera (416), for example, to detect objects (e.g., virtual and physical) in the field of view of the camera (416). The lighting device (408) may optionally operate with the camera (416), for example, to detect objects in the field of view of the camera (416).
[0080] AR glasses (400A) may include a communication module (e.g., communication module (252)) that communicates with a processor (414) and a control system (412). The communication module may provide communication between devices housed within the AR glasses (400A), as well as communication with external devices such as, for example, a controller, a mobile device, a server, and / or other computing devices. The communication module may enable the AR glasses (400A) to communicate to exchange information with other computing devices and to authenticate other devices or other identifiable elements of the environment within the range of the AR glasses (400A). For example, the AR glasses (400A) may be operably connected to another computing device to facilitate communication, for example, via a wired connection, for example, a wireless connection via Wi-Fi or Bluetooth, or other types of connections.
[0081] FIG. 4b is a rear view (400B) of AR glasses (400A) according to an implementation described throughout this specification. AR glasses (400B) may be an example of the wearable device (100) of FIG. 1. AR glasses (400B) are glasses that add information (e.g., project a display (420)) along with what the wearer sees through the glasses. In some embodiments, instead of projecting information, the display (420) is an in-lens micro-display. In some embodiments, AR glasses (400B) (e.g., glasses or a vanity) is a visual aid device comprising a lens (422) (e.g., glass or hard plastic lens) mounted in a frame (402) that secures the lens in front of the person's eyes by utilizing a bridge (424) over the nose and a stand (426) placed over the ears (e.g., temples or temple portions).
[0082] FIG. 5 is a flowchart illustrating a method related to an embodiment described herein. The method may be performed within at least the capture parameter selection processing step (100-1) illustrated in FIG. 1 and FIG. 2.
[0083] The method may include the step (block 510) of capturing a first raw image at a first resolution using an image sensor. The first image may be captured using, for example, the image sensor (216) shown in FIG. 2. The first resolution may be included within the resolution capture parameter(s) (221) shown in FIG. 2.
[0084] The above method may include the step (block 520) of converting a first raw image into a digitally processed image using an image signal processor. The image signal processor may be, for example, the image signal processor (224) shown in FIG. 2.
[0085] The above method may include a step (block 530) of analyzing at least a portion of a digitally processed image based on processing conditions. The analysis may be performed, for example, in a parameter selection processor (227) illustrated in FIG. 2. Processing conditions may include, for example, processing condition(s) (101) illustrated in FIG. 1. A specific example of a processing condition is illustrated as the resolution processing condition (222) illustrated in FIG. 1.
[0086] The above method may include a step (block 540) of determining that the first resolution does not satisfy the processing conditions. The determination may be performed, for example, in a parameter selection processor (227) illustrated in FIG. 2. The determination may be based on a resolution selection algorithm (223) illustrated in FIG. 2.
[0087] The above method may include the step (block 550) of triggering the capture of a second raw image from an image sensor at a second resolution higher than the first resolution. The second image may be captured using, for example, the image sensor (216) shown in FIG. 2. The second resolution may be included within the resolution capture parameter(s) (221) shown in FIG. 2.
[0088] FIG. 6 illustrates examples of a computer device (1000) and a mobile computer device (1050) that may be used with the techniques described herein (e.g., to implement a client computing device (100), a server computing device (204) and / or a mobile computer device (202)). The computing device (1000) includes a processor (1002), memory (1004), a storage device (1006), a high-speed interface (1008) connected to the memory (1004) and a high-speed expansion port (1010), and a low-speed bus (1014) and a low-speed interface (1012) connected to the storage device (1006). Each of the components (1002, 1004, 1006, 1008, 1010 and 1012) may be interconnected using various buses and mounted on a common motherboard or in other suitable ways. A processor (1002) can process instructions for execution within a computing device (1000), including instructions stored in memory (1004) or a storage device (1006), and display graphic information for a GUI on an external input / output device, such as a display (1016) connected to a high-speed interface (1008). In other embodiments, multiple processors and / or multiple buses may be appropriately used together with multiple memories and multiple types of memory. Additionally, multiple computing devices (1000) may be connected to each device that provides parts of the necessary operation (e.g., a server bank, a blade server group, or a multiprocessor system).
[0089] Memory (1004) stores information within a computing device (1000). In one embodiment, memory (1004) is a volatile memory unit or units. In another embodiment, memory (1004) is a non-volatile memory unit or units. Additionally, memory (1004) may be another form of computer-readable medium, such as a magnetic or optical disk.
[0090] A storage device (1006) may provide large storage for a computing device (1000). In one embodiment, the storage device (1006) may be a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device or a tape device, a flash memory or other similar solid-state memory device, or an array of devices including a storage area network or other configurations, or may include a computer-readable medium. A computer program product may be tangibly stored in an information carrier. A computer program product may also include instructions that perform one or more methods as described above when executed. The information carrier is a computer or machine-readable medium, such as memory (1004), a storage device (1006), or memory on a processor (1002).
[0091] The high-speed controller (1008) manages bandwidth-intensive operations for the computing device (1000), whereas the low-speed controller (1012) manages low-bandwidth-intensive operations. The assignment of these functions is merely exemplary. In one embodiment, the high-speed controller (1008) is connected to a high-speed expansion port (1010) capable of accommodating memory (1004), a display (1016) (e.g., via a graphics processor or accelerator), and various expansion cards (not shown). In an embodiment, the low-speed controller (1012) is coupled to a storage device (1006) and a low-speed expansion port (1014). The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router via a network adapter, for example.
[0092] The computing device (1000) may be implemented in a number of different forms as illustrated in the drawing. For example, it may be implemented as a standard server (1020) or as a group of such servers. It may also be implemented as part of a rack server system (1024). It may also be implemented in a personal computer such as a laptop computer (1022). Alternatively, components from the computing device (1000) may be combined with other components within a mobile device (not shown), such as a device (1050). Each of the devices may include one or more of the computing devices (1000, 1050), and the entire system may be composed of a number of computing devices (1000, 1050) communicating with each other.
[0093] The computing device (1050) includes, in particular among several other components, an input / output device such as a processor (1052), memory (1064), and a display (1054), a communication interface (1066), and a transceiver (1068). The device (1050) may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components (1050, 1052, 1064, 1054, 1066, and 1068) is interconnected using various buses, and some components may be mounted on a common motherboard or in other suitable ways.
[0094] The processor (1052) can execute instructions within the computing device (1050) including instructions stored in memory (1064). The processor may be implemented as a chipset of chips including separate and multiple analog and digital processors. The processor may provide coordination of other components of the device (1050), such as user interfaces, applications executed by the device (1050), and wireless communication by the device (1050).
[0095] The processor (1052) can communicate with the user through a control interface (1058) and a display interface (1056) connected to the display (1054). The display (1054) may include, for example, a TFT LCD (Thin-Film Transistor Liquid Crystal Display), an LCD (Light Emitting Diode) or OLED (Organic Light Emitting Diode) display, or other suitable display technology. The display interface (1056) may include suitable circuitry for driving the display (1054) to provide graphics and other information to the user. The control interface (1058) may receive commands from the user and convert them to submit to the processor (1052). Additionally, an external interface (1062) may be provided for communication with the processor (1052) to enable short-range communication between the device (1050) and other devices. The external interface (1062) may be provided for wired communication in some embodiments or for wireless communication in other embodiments, for example, and multiple interfaces may be used.
[0096] Memory (1064) stores information within the computing device (1050). Memory (1064) may be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. Additionally, an extended memory (1074) may be provided to and connected to the device (1050) through an extended interface (1072) which may include, for example, a Single In Line Memory Module (SIMM) card interface. The extended memory (1074) may provide additional storage space for the device (1050) or may store applications or other information about the device (1050). In particular, the extended memory (1074) may include instructions that perform or supplement the aforementioned processes and may also include security information. Thus, for example, the extended memory (1074) may be provided as a security module for the device (1050) and may be programmed with instructions that allow the secure use of the device (1050). In addition, security applications may be provided with additional information through SIMM cards, such as by deploying identification information on the SIMM card in a way that makes it impossible to hack.
[0097] The memory may include, for example, flash memory and / or NVRAM memory as described below. In one embodiment, a computer program product is tangibly stored in an information carrier. The computer program product also includes instructions that perform one or more methods as described above when executed. The information carrier is a computer or machine-readable medium, such as memory (1064), extended memory (1074), or memory on a processor (1052), which can be received, for example, through a transceiver (1068) or an external interface (1062).
[0098] The device (1050) may communicate wirelessly through a communication interface (1066) that may include a digital signal processing circuit as needed. The communication interface (1066) may be provided for communication under various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS, among others. Such communication may occur, for example, through a radio frequency transceiver (1068). Additionally, short-range communication may occur, such as using Bluetooth, Wi-Fi, or other transceivers (not shown). Additionally, a Global Positioning System (GPS) receiver module (1070) may provide additional navigation and location-related wireless data to the device (1050) that can be appropriately used by applications running on the device (1050).
[0099] The device (1050) may also communicate audibly using an audio codec (1060) capable of receiving information spoken by a user and converting it into available digital information. The audio codec (1060) may likewise generate audible sound for the user, such as through a speaker within the handset of the device (1050), for example. Such sound may include sounds from voice phone calls, recorded sounds (e.g., voice messages, music files, etc.), and sounds generated by applications running on the device (1050).
[0100] The computing device (1050) may be implemented in a number of different forms as illustrated in the drawing. For example, it may be implemented as a cellular phone (1080). It may also be implemented as part of a smartphone (1082), a personal digital assistant (PDA), or other similar mobile device.
[0101] Various embodiments of the systems and techniques described herein may be implemented in digital electronic circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include embodiments in one or more computer programs executable and / or interpretable on a programmable system comprising at least one programmable processor, which may be dedicated or general-purpose, and may be connected to receive data and instructions from a storage system, at least one input device, and at least one output device and to transmit data and instructions thereto.
[0102] These computer programs (also known as programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in high-level procedural languages and / or object-oriented programming languages and / or assembly / machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program, device, and / or device, e.g., magnetic disk, optical disk, memory, programmable logic device (PLD), used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.
[0103] To provide interaction with a user, the systems and techniques described herein may be implemented in a computer having a display device (a light-emitting diode, or OLED, or liquid crystal display monitor / screen) to display information to a user, and a keyboard and pointing device, such as a mouse or trackball, to which a user can provide input to the computer. Other types of devices may also be used to provide interaction with a user. For example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form including sound, voice, or tactile input.
[0104] The systems and techniques described herein may be implemented in a computing system comprising, for example, a backend component such as a data server, a middleware component such as an application server, a user computer having a graphical user interface, a frontend component such as a web browser through which a user can interact with an embodiment of the systems and techniques described herein, or any combination of one or more of said backend, middleware, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication, for example, by a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.
[0105] A computing system may include users and servers. Users and servers are typically located far apart from each other and generally interact through a communication network. The relationship between users and servers is generated by computer programs running on their respective computers that have a user-server relationship with each other.
[0106] In some embodiments, the computing device illustrated in the drawing may include sensors that interface with an AR headset / HMD device (1090) to create an augmented environment for viewing content inserted within a physical space. For example, one or more sensors included in the computing device (1050) or other computing devices illustrated in the drawing may provide input to the AR headset (1090) or generally to the AR space. The sensors may include, but are not limited to, touch screens, accelerometers, gyroscopes, pressure sensors, biosensors, temperature sensors, humidity sensors, and ambient light sensors. The computing device (1050) may use the sensors to determine the absolute position and / or detected rotation of the computing device within the AR space, which can be used as input to the AR space. For example, the computing device (1050) may be integrated into the AR space as a virtual object, such as a controller, laser pointer, keyboard, weapon, etc. When integrated into the AR space, the position of the computing device / virtual object by the user causes the user to position the computing device and view the virtual object in a specific way within the AR space.
[0107] In some embodiments, one or more input devices included in or connected to the computing device (1050) may be used as inputs to the AR space. The input devices may include, but are not limited to, a touch screen, a keyboard, one or more buttons, a trackpad, a touchpad, a pointing device, a mouse, a trackball, a joystick, a camera, a microphone, earphones or earbuds with input functions, a game controller, or other connectable input devices. When the computing device is integrated into the AR space, a user interacting with the input device included in the computing device (1050) may cause a specific action to occur in the AR space.
[0108] In some embodiments, the touchscreen of the computing device (1050) may be rendered as a touchpad in AR space. A user may interact with the touchscreen of the computing device (1050). Interactions are rendered, for example, according to movements on the touchpad rendered in AR space on an AR headset (1090). The rendered movements may control virtual objects in AR space.
[0109] In some embodiments, one or more output devices included in the computing device (1050) may provide output and / or feedback to a user of the AR headset (1090) in the AR space. The output and feedback may be visual, tactile, or auditory. The output and / or feedback may include, but are not limited to, vibration, turning on and off or blinking and / or flashing of one or more lights or strobes, sounding an alarm, chiming, playing a song, and playing an audio file. The output devices may include, but are not limited to, a vibration motor, a vibration coil, a piezoelectric device, an electrostatic device, a light-emitting diode (LED), a strobe, and a speaker.
[0110] In some embodiments, the computing device (1050) may be displayed as another object in a computer-generated 3D environment. Interactions with the computing device (1050) by a user (e.g., rotation, shaking, touchscreen touch, finger swiping across the touchscreen) may be interpreted as interactions with the object in AR space. In the example of a laser pointer in AR space, the computing device (1050) is displayed as a virtual laser pointer in a computer-generated 3D environment. As the user manipulates the computing device (1050), the user in AR space sees the movement of the laser pointer. The user receives feedback from interactions with the computing device (1050) in the AR environment on the computing device (1050) or with the computing device (1050) on the AR headset (1090). Interactions between the computing device and the user may be converted into interactions with a user interface generated in the AR environment for the controllable device.
[0111] In some embodiments, the computing device (1050) may include a touchscreen. For example, a user may interact with the touchscreen to interact with a user interface for a controllable device. For example, the touchscreen may include user interface elements, such as a slider, that can control attributes of the controllable device.
[0112] The computing device (1000) is intended to represent various forms of digital computers, including but not limited to laptops, desktops, workstations, personal digital terminals, servers, blade servers, mainframes, and other suitable computers. The computing device (1050) is intended to represent various forms of mobile devices, such as personal digital terminals, cellular phones, smartphones, and other similar computing devices. The components illustrated herein, their connections and relationships, and their functions are merely illustrative and do not imply limiting the implementation of the invention described and / or claimed herein.
[0113] Many embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the invention.
[0114] Additionally, the logical flows depicted in the drawings do not necessarily require a specific depicted order or chronological order to achieve the desired results. Furthermore, other steps may be provided, steps may be omitted from the described flow, and other components may be added to or removed from the described system. Accordingly, other embodiments are also within the scope of the following claims.
[0115] In addition to the above description, controls may be provided to the user to enable choices regarding when the systems, programs, or configurations described herein collect user information (e.g., information about the user's social networks, social actions or activities, occupation, user preferences, or user's current location) and when the user is transmitted to a content or server for communication. Additionally, certain data is treated in one or more different ways before being stored or used, so that personally identifiable information is removed. For example, the user's identity is treated so that personally identifiable information about the user cannot be determined, or the user's geographic location is generalized from where the location information was obtained (to the city, zip code, or state level) so that the user's specific location cannot be determined. Thus, the user may have control over what information about the user is collected, how the information is used, and what information is provided to the user.
[0116] Although specific configurations of the described embodiments have been described as set forth in this specification, many modifications, substitutions, changes, and equivalents will arise for those skilled in the art. Therefore, it should be understood that the appended claims are intended to include all such modifications and changes within the scope of the embodiment. These embodiments are merely illustrative and not limiting, and it should be understood that various changes in form and detail may be made. Any part of the apparatus and / or method described in this specification may be combined in any combination except mutually exclusive combinations. The embodiments described in this specification may include various combinations and / or sub-combinations of functions, components, and / or configurations of the other embodiments described.
Claims
Claim 1 An image processing method comprising: using an image sensor to capture a first image at a first resolution; using an image signal processor to convert the first image into a processed image; analyzing at least a portion of the processed image based on processing conditions; determining that the first resolution does not satisfy the processing conditions; and in response to the processing conditions not being satisfied, triggering the capture of a second image at a second resolution higher than the first resolution from the image sensor. Claim 2 An image processing method according to claim 1, wherein at least a portion of the processed image includes text characters. Claim 3 An image processing method according to claim 1, wherein at least a portion of the processed image comprises a real-world object. Claim 4 An image processing method according to claim 1, wherein the processing condition is at least one of character-specific pixel, height, and area conditions. Claim 5 An image processing method according to claim 1, wherein the first resolution for capturing the first image corresponds to a bin defined in the image sensor. Claim 6 An image processing method according to claim 1, wherein the first resolution is a basic resolution and the second resolution is defined based on an item of a lookup table. Claim 7 An image processing method according to any one of claims 1 to 6, wherein the processed image is a first processed image captured by a computing device, and the method further comprises: a step of converting a second image into a second processed image; a step of determining whether a second resolution satisfies processing conditions; and a step of transmitting the second processed image to a companion device for the computing device for further processing in response to the determination that the second resolution satisfies processing conditions. Claim 8 A non-transient machine-readable type medium storing instructions read and executed by one or more processors to perform an image processing method, wherein the method comprises: capturing a first image at a first resolution using an image sensor; converting the first image into a processed image using an image signal processor; analyzing at least a portion of the processed image based on processing conditions; determining that the first resolution does not satisfy the processing conditions; and triggering the capture of a second image at a second resolution higher than the first resolution from the image sensor in response to the processing conditions not being satisfied. Claim 9 A non-transient machine-readable type medium according to claim 8, wherein at least a portion of the processed image comprises text characters. Claim 10 A non-transient machine-readable type medium according to claim 8, wherein at least a portion of the processed image comprises a real-world object. Claim 11 A non-transient machine-readable type medium according to claim 8, wherein the processing condition is at least one of character-specific pixel, height, and area conditions. Claim 12 In claim 8, the first resolution for capturing the first image corresponds to a bin defined in the image sensor, a non-transient machine-readable type medium. Claim 13 A non-transient machine-readable type medium according to claim 8, wherein the first resolution is a basic resolution and the second resolution is defined based on an entry in a lookup table. Claim 14 A non-transient machine-readable type medium according to any one of claims 8 to 13, wherein the processed image is a first processed image captured by a computing device, and the method further comprises: converting a second image into a second processed image; determining whether a second resolution satisfies processing conditions; and transmitting the second processed image to a companion device for the computing device for further processing in response to the determination that the second resolution satisfies processing conditions. Claim 15 A system comprising: a processor; and a memory device communicationly connected to the processor and storing a plurality of instructions, wherein when the instructions are executed by the processor, the processor causes: to capture a first image at a first resolution using an image sensor; to convert the first image into a processed image using an image signal processor; to analyze at least a portion of the processed image based on a processing condition; to determine that the first resolution does not satisfy the processing condition; and, in response to the processing condition not being satisfied, to trigger the capture of a second image at a second resolution higher than the first resolution from the image sensor. Claim 16 A system according to claim 15, wherein at least a portion of the processed image comprises text characters. Claim 17 A system according to claim 15, wherein at least a portion of the processed image comprises a real-world object. Claim 18 A system according to claim 15, wherein the processing condition is at least one of a character-specific pixel, height, and area condition. Claim 19 A system according to claim 15, wherein the first resolution for capturing the first image corresponds to a bin defined in the image sensor. Claim 20 A system according to claim 15, wherein the first resolution is a basic resolution and the second resolution is defined based on an item of a lookup table. Claim 21 A system according to any one of claims 15 to 20, wherein the processed image is a first processed image captured by a computing device, and the instructions, when executed by the processor, further cause the processor to: convert a second image into a second processed image; determine whether a second resolution satisfies processing conditions; and, in response to the determination that the second resolution satisfies processing conditions, transmit the second processed image to a companion device for the computing device for further processing.
Citation Information
Patent Citations
Composing a focused document image from multiple image captures or portions of multiple image captures
US10402790B1
Apparatus and method for analyzing images
US20140253701A1
Image Resizing For Optical Character Recognition In Portable Reading Machine
US20150042562A1
Vehicle promotion aggregator systems
US20200027111A1
Dynamically optimizing photo capture for multiple subjects
US20200125882A1
