Cross-platform camera-based distance detection

Abstract

Described and claimed is a computer-implemented method for resource-constrained and / or cross-platform camera-based distance detection without knowledge of camera specifications. The method comprises the steps of instructing a camera to capture calibration camera data of an object located at a calibration distance, detecting the object, determining a calibration width of the object, instructing the camera to capture additional camera data for distance detection, detecting the object on the additional camera data, determining an object width based on the additional camera data, and estimating the object distance based on the object width, the calibration width, and the calibration distance. Further, a computer-implemented method, an execution environment, and a computer-readable storage media for placing a user at a specific distance from a visual target for performing a vision test using a resource-constrained and / or cross-platform camera-based distance detection are disclosed and claimed.

Description

CROSS-PLATFORM CAMERA-BASED DISTANCE DETECTIONBACKGROUND

[0001] This application claim priority to U.S. Provisional Patent Application No. 63 / 594,234, which was filed on October 30, 2023 and entitled “Cross-platform camera-based distance detection, the content of which is incorporated herein by reference in its entirety.BACKGROUND

[0002] Modem computing devices often include cameras for capturing video or images used to implement image-based functions (e.g., object detection, facial recognition, video conferencing, image-based search). Camera technology, and execution environments for applications that implement these image-based functions, can vary across different devices and platforms (e.g., operating systems). An application may need greater camera capabilities and / or additional computing resources than its execution environment allows. Similarly, the available camera capabilities and computing resources will vary across devices and an application may need to perform its image-based functions regardless of the device being used. Further, determining the specifications of the camera might not always be possible, especially in a cross-platform execution environment.SUMMARY OF THE INVENTION

[0003] This document is directed to techniques for enabling cross-platform camera-based distance detection, including in some instances, even if the camera specifications are unknown. In some examples, computer-implemented methods are described for determining distance to an object, in near real-time, using an application running natively on a resource-constrained device or non- natively (i.e.. cross-platform) in an application container (e.g., a web browser). In furtherance of these examples, a system or apparatus is described including a processor configured to perform steps ofthese methods. For example, computer-readable storage media stores or includes instructions that, when executed, cause at least one processor to perform these computer-implemented method steps.

[0004] This Summary introduces simplified concepts for efficient camera-based distance detection that are further described in the Detailed Description and Drawings. This Summary' is not intended to identify essential features of the claimed subject matter, nor is it intended for use in determining the scope of the claimed subject matter.

[0005] In a first aspect, an example of a computer implemented method for resource- constrained and / or cross-platform camera-based distance detection without knowledge of camera specifications is described. The method comprising the steps of instructing a camera to capture calibration camera data of an object, wherein the object is located at a calibration distance from the camera and wherein the calibration camera data is an image captured using the camera (108), or a frame of a video captured using the camera (108), detecting the object on the calibration camera data, determining a calibration width of the object detected on the calibration camera data, instructing the camera to capture additional camera data of the object for distance detection, detecting the object on the additional camera data captured for distance detection, determining an object width of the object detected on the camera data captured for distance detection, and estimating the object distance based on the object width, the calibration width, and the calibration distance.

[0006] Preferably, after determining a calibration width of the object, the camera is instructed to continuously capture camera data in the form of video frames or images. The method further comprises the steps of selecting a plurality of the video frames or images captured as additional camera data for distance detection, detecting the object on each video frame or image selected for distance detection, determining an object width of the object detected on each video frame or image selected for distance detection, and estimating the distance of the object on each video frame or image selected for distance detection based on the object width of the object on the selected video frame or image, the calibration width, and the calibration distance.

[0007] In a second aspect, an example of a computer-implemented method for placing a user at a specific distance from a visual target for performing a vision test using a resource-constrained and / or cross-platform camera-based distance detection without knowledge of camera specifications is described. The method comprises the steps of instructing a camera to capture calibration camera data of an object formed by the user using a camera attached to the visual target, wherein the object is located at a calibration distance from the camera (108) and wherein the camera data is an image or a video frame captured using the camera (108), detecting the object on the calibration camera data, determining a calibration width of the object detected on the calibration camera data, instructing the user to increase or decrease the distance between the user and the camera (108) until the user is at the specific distance based on a detected object distance of the user from the camera (108), wherein for estimating the estimated distance of the user from the camera (108), the method further comprises the steps of instructing the camera (108) to continuously capture additional camera data in the form of video frames or images, selecting a plurality of the video frames or images captured as additional camera data for distance detection, detecting the object formed by the user on each video frame or image selected for distance detection, determining an object width of the object detected on each video frame or image selected for distance detection, and estimating the distance of the user on each video frame or image selected for distance detection based on the object width of the user on the selected video frame or image, the calibration width, and the calibration distance.

[0008] Preferably, the object distance is estimated based on an estimated focal length of the camera, wherein the estimated focal length of the camera is determined from an estimated field of view of the camera and an image size of the camera or wherein the object distance is estimated without knowing or estimating the focal length of the camera.

[0009] Preferably, of the continuously captured additional camera data, between two and five images or video frames captured per second are selected for distance detection.

[0010] Preferably, the detected object distance is determined as a median of the distances estimated of the object on a plurality of video frames or images selected over a specific time frame.

[0011] Preferably, the calibration width and the object width are determined as a number of pixels the object spans across the image or the camera frame, respectively.

[0012] Preferably, the calibration distance corresponds approximately to an arm's length, wherein an arm’s length is one of 30.48 cm (one foot), 45.72 cm (18 inches), or 60.96 cm (two feet).

[0013] Preferably, the calibration distance is a distance at which the object covers a predetermined section of a field of view of the camera. Further preferably, the method is configured to instruct a user to change a distance between the object and the camera for capturing the calibration camera data in case the object covers a larger section of the field of view of the camera or a smaller section of the field of view of the camera than the predetermined section.

[0014] Preferably, a first object recognition model is used for detecting the object on the calibration camera data and wherein a different second object recognition model is used for detecting the object on the additional camera data for distance detection. Further preferably, the object is a user, the first object recognition model is a facial recognition model, and the second object recognition model is one of an upper-body recognition model and a full-body recognition model.

[0015] Preferably, the camera is part of a computing device and wherein the calibration camera data and the additional camera data captured with the camera are transmitted by the computing device to a remote computing system for detecting the object on the camera data, determining the calibration width and object width, and estimating the distance of the object.

[0016] In a third aspect, an example of an execution environment comprising a computing device with a camera is described. The execution environment is configured for implementing a method according to any of the preceding exemplary embodiments.

[0017] Preferably, the execution environment further comprises a remote computing system.

[0018] Finally, in a fourth aspect, an example of a computer-readable storage media storing or including instructions that, when executed, cause at least one processor to perform a computer- implemented method according of any of the preceding examples.BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Techniques for efficient camera-based distance detection are described with reference to the following drawings, which use some of the same numbers throughout to reference like features and components, or examples thereof.Fig. 1 illustrates an execution environment for enabling cross-platform camera-based distance detection, in accordance with techniques of this disclosure.Fig. 2 illustrates a computing device for enabling cross-platform camera-based distance detection, in accordance with techniques of this disclosure.Fig. 3 illustrates a data flow for enabling cross-platform camera-based distance detection, in accordance with techniques of this disclosure.Figs. 4-1 and 4-2 illustrate an example use case for enabling cross-platform camera-based distance detection, in accordance with techniques of this disclosure.Fig. 5 illustrates a computer-implemented process for enabling cross-platform camera-based distance detection, in accordance with techniques of this disclosure.DETAILED DESCRIPTION

[0020] The described techniques relate to implementing efficient camera-based distance detection, including image-based or video-based distance detection within resource-constrained and / or cross-platform execution environments. Whether running natively or non-natively. computer applications are allocated finite computing resources (e.g., bandwidth, processor time, storage, memory allocation, process / thread resource availability). While existing camera applications mayperform automatic distance determinations, these traditional approaches are unfeasible in certain execution environments where computing resources are limited and / or the specifications of the camera are unknown. Existing camera applications may not be adaptable for execution in a resource- constrained native environment, such as an loT (Intemet-of-Things) device or an embedded system. Likewise, non-native environments, such as web browsers or other application containers, can be particularly restrictive and may limit a traditional application’s ability to handle unknown and inconsistent camera specifications and unknown size / dimensions of the target object. Furthermore, existing camera-based distance detection techniques tend to be resource-intensive in terms of processor and memory needs to rely on parallel execution of numerous native and / or non-native processes and threads, which can overburden devices.

[0021] In contrast, the described techniques enable near real-time determinations about distances between objects and device cameras to be made efficiently, even when camera specifications are unknown, and without wasting computing resources. For example, efficiencies may be obtained without having to perform complex calibrations. In other words, rather than computing various camera properties that are normally needed to make accurate distance estimations, such as focal length and field of view (FOV), the described techniques apply various assumptions or rules about device cameras and object reference sizes, which enable accurate distance estimations to be made quickly.

[0022] In addition, other efficiencies are gained through careful processing of a camera feed. Select portions of the video or image data in the camera feed may be analyzed instead of the entire data package. Offloading processing to a support application executing on a server or to a background process or thread may also be utilized to reduce the latency of the distance estimates and improve efficiency. Local processing techniques, as well as remote processing techniques, may further benefit from smoothing techniques that rely on averages, means, and median measurements to improve accuracy and reduce lag in updates to the distance estimates. The techniques are described in relationto improving efficiency for native or non-native environments, especially where resources are especially limited.

[0023] The efficiency gained from the described techniques significantly extends browser, device, and operating system capabilities across many different types of mobile devices, desktop computers, and other camera-equipped devices. Through these techniques, user accessibility7to camera functions is significantly expanded, even when device and / or operating system computing resources are constrained. Numerous use cases for the described near real-time camera-based distance detection processes exist. As some examples, a web browser can implement the described techniques to perform distance verification for a vision test (e.g., a visual acuity test, visual refraction test, or other vision test), a proximity sensor alert system, a user interface (e.g., a distance-based graphical element size adjustment, an audible element sound volume adjustment), and an augmented reality application (e.g., a camera-based distance measurement tool).

[0024] Fig. 1 illustrates an execution environment 100 for enabling efficient and / or crossplatform camera-based distance detection, in accordance with techniques of this disclosure. The execution environment 100 includes a computing device 102 (e g., a mobile device), shown communicatively and operatively connected via an optional network 104 (e.g., a connection, a wired network, a wireless network, a cloud) to an optional computing system 106 (e.g., a remote server, another processor in the same device). A more detailed example of the computing device 102 is illustrated in Fig. 2 as computing device 200; likewise, a data flow 300 is illustrated in Fig. 3 to provide additional details of the execution environment 100, including illustrating examples where the techniques are optionally implemented in conjunction with the network 104 and the computing system 106.

[0025] The computing device 102 is a computing platform having one or more central processing units (CPUs) and / or graphical processing units (GPUs) operable to execute instructions to perform computing functions. The computing device 102 includes memory and storage units forretaining data and storing the instructions executed by the CPUs or GPUs. The computing device 102 includes various input / output interfaces to enable user experiences and provide machine interactions with external devices and networks. Additionally, the computing device 102 may include specialized hardware and / or software components for executing specific applications, improving efficiency, and enhancing performance. The computing device 102 may be a mobile device, desktop computer, laptop, tablet, smartphone, loT device, embedded system, or other device (e.g., baby monitor, surveillance camera unit, vehicle system).

[0026] The network 104 interconnects a group of systems and devices that communicate and / or share resources through established rules or protocols. The computing device 102 and the computing system 106 are among the computers, servers, routers, switches, and other devices capable of transmitting and receiving data as part of the network 104. The network 104 may provide wired (e.g., Ethernet) or wireless (e.g., Wi-Fi) communication channels that are organized into local area networks (LANs) and / or wide area networks (WANs) (e.g., the Internet). A cellular network or mobile network (e g., LTE, 3G, 4G, 5G) are further examples of the network 104. The network 104 enables data to be transferred between the computing device 102 and the computing system 106. This data communication may be facilitated using one or more network communication protocols. As some examples, the protocols that coordinate the information exchange over the network 104 can include User Datagram Protocol (UDP), Transmission Control Protocol (TCP), Internet Protocol (IP), Hypertext Transfer Protocol (HTTP), WebSocket, Short Message Service (SMS), Multimedia Messaging Service (MMS), MQ Telemetry Transport (MQTT), Constrained Application Protocol (CoAP), Extensible Messaging and Presence Protocol (XMPP). Advanced Message Queuing Protocol (AMQP), and Data Distribution Service (DDS).

[0027] The computing system 106 may be a remote computer, such as a server, which is operable to exchange data with the computing device 102 using a shared connection or communication channel within the network 104. Based on information exchanged through the network 104, thecomputing device 102 may offload, to one or more processing units of the computing system 106, certain complex computer functions, which might otherwise be too resource-intensive to be performed locally on the computing device 102.

[0028] The computing device 102 includes a camera 108, which uses one or more lenses and imagers to capture camera data (e.g., images, video). The camera 108 may be integrated into the computing device 102 or communicatively coupled to it via a wired or wireless connection, such as Bluetooth or Wi-Fi. The camera 108 outputs the camera data to a camera feed 110, which is typically accessed as a video stream. Application processes or threads that execute on the computing device 102 may access the camera feed 110 to read the camera data coincident with or nearly coincident with when the camera 108 generates the camera data. For example, each frame or portion of video output to the camera feed 110 is accessible from an application program interface (API), without having to wait for a subsequent portion of video data to be captured or stored as a recorded video fde. To minimize latency and interruptions in the stream, the camera feed 110 may be buffered in memory of the computing device 102; the memory allocated to the camera feed 110 may be of sufficient size to include multiple milliseconds, seconds, minutes, or hours of video data captured by the camera 108. For example, the camera feed 110 may store several individual images or frames of previously captured video, before overwriting these older video portions with new camera data captured more recently.

[0029] The computing device 102 executes a device application 112. The device application 1 12 may run as one or more native processes or threads, one or more non-native (i.e., cross-platform) processes or threads, or a combination of these native and non-native programs or applications. In general, native applications are optimized to execute on a specific platform, including for operability with a specific camera and / or certain device. Device- or platform-specific source code (e.g., C++, Python) may be necessary for implementing a native application that benefits from low-level access to hardware capabilities of aspecific device (e.g., camera 108). Examples of processes include console applications, native applications, background processes, web applications. Processes can run on avariety of hosts: on the host machine, in a Docker container, in a web server, in a Function-as-a-Service cloud function (e.g., AWS Lambda, Google Cloud Functions, Azure Functions). Each of these application ty pes has a primary ‘‘application process’" or “application thread” that either runs in the foreground or (e.g., UI thread) or in the background. Each application process or thread can additionally spaw n one or more processes (multi-processing) or one or more threads (multi-threading) itself. While other numerous native-implementations of camera-based distance detections may exist, these approaches are camera-specific and platform-constrained; their functions depend on knowing the camera details and are available only to specific devices and operating systems, and therefore, not applicable to cross-platform environments, such as executing from within a web browser on a variety of devices. Unlike native applications targeting specific cameras and platforms, web browsers and other types of application containers provide execution environments for cross-platform or non-native applications intend to support a variety' of cameras, devices, browsers, operating systems, and platforms written in a cross-platform source code (e.g., Java, JavaScript); their functions are operable regardless of the specific camera, device, browser, operating system, or platform used to support the container. As such, these cross-platform non-native applications expand user reach to their functions to more devices and computing architectures than their counterparts that target specific platforms native application counterparts. However, non-native applications may have restricted ability’ (relative to their native application variants targeting specific platforms) to access and interface with certain hardware capabilities of a specific device and camera. Further drawbacks to cross-platform or non- native applications are that their execution typically happens under more constraints than in a nativeexecution environment and without a priori knowledge of or guarantees of camera specifications or resource availability. These additional constraints may otherwise hinder the performance of camerabased distance detection functions implemented therein.

[0030] Regardless of whether implemented in native, non-native, or hybrid (i.e., non-native and native) execution environments, the device application 112 may cause the computing device 102to output (e.g.. for display) a camera interface 116 (e.g., a graphical user interface). There may be instances where the camera interface 116 is not necessary7. For instance, the computing device 102 may determine the distance an object is away from the camera 108 without verifying the object is necessarily at a particular distance. The camera feed 110 is accessible to processes or threads of the device application 112. Based on the camera data obtained from the camera feed 110, the device application 112 may cause the camera interface 116 to include a preview 118 of the images or video frames (or a subset thereol), which are displayed by the computing device 102 in near real-time (e.g., coincident with capturing by the camera 108).

[0031] In some examples, the computing system 106 executes a server application 114 that communicates over the network 104 to enhance or support operations of the device application 112. The server application 114 may perform one or more functions that enable the device application 112 to generate the camera interface 116 more efficiently. For example, the server application 114 may process camera data on behalf of the device application 112, by accessing the video stream provided in the camera feed 110, which may improve performance (e.g., reduce latency) in the camera interface 1 16. The server application 114 may access the camera feed 110; the device application 112 selects a version or subset of the camera data to share over the network 104 with the server application 114 rather than providing the server application 114 with access to the entire video stream. For example, each time camera data is sent to the computing system 106, the device application 112 batches one or more images or one or more video frames into a file that is transmitted via the network 104 to the server application 1 14. The server application 114 may perform computationally complex vision analysis of the camera data, such as. to perform object recognition and to determine the distance to the object in question, which may be the object presented in the camera interface 116. As explained throughout, utilizing the computing system 106 is optional; the device application 112 may analyze or process the camera feed 110 independently, without relying on the computing system 106. For example, the device application 112 may locally (e.g., without sending camera data to the serverapplication 114) batch or select portions of the camera data generated in the camera feed 110 to perform object recognition and distance estimations for updating the camera interface 116.

[0032] The device application 112 enables camera-based distance determinations from a resource-constrained and / or cross-platform computing environment. That is, examples of the device application 112 include one or more processes or threads executing on the computing device 102 with only limited available storage, minimal available memory', or with restricted processor compute-time. In other examples, the device application 112 includes one or more threads or processes that execute in a web browser or other application container, which similarly limits the amount of computing resources available to complete its computing functions. Still, in other versions, a combination of foreground and background threads or processes may execute on the computing device 102 to implement the functions of the device application 112. For instance, the application process or thread that controls the camera 108 may be invoked to access the camera feed, then delegate task operations to performance by one or more Web Workers that perform object detection and distance estimation from a web browser. The background process(es) or thread(s) have access to the camera feed 110 data passed by the application process or thread. The device application 112 may be a function of a user interface process or thread that executes as part of the web browser; the device application 1 12 may execute as a separate JavaScript or other cross-platform run-time compilation source code from within the user interface process or thread, which utilizes the camera 108 and accesses the camera feed 110. Regardless of its native or non-native environment, execution of the device application 112 enables efficient camera-based distance determinations to occur without overloading processing resources of the computing device 102 or with constrained processing resources allocated to the device application 112. To understand how the device application 112 performs efficient camera-based distance detection, it is important to discuss traditional ways that other camera applications measure a distance between an object and a camera based on a video feed.

[0033] In a traditional distance detection function, the application determines the camera’s focal length f and the object’s size R. For example, by processing each frame of a video captured by the camera (e.g., one frame at a time), the distance between the object and the camera can be determined for any given time. That is, if the focal length of the camera is known, and the size of the object that is known to the camera process or thread is R, then the distance d between the object and the camera 108 can be calculated from Equation 1 : f / d = 1 / REquation 1. Unfortunately, the variables of Equation 1 may be difficult to derive. Even if their values are derivable, their solutions may take too long to support real-time distance estimations in conjunction with a realtime video feed.

[0034] Determining the focal length / requires knowing about the physical camera hardware itself as it varies based on the camera lens. Further, in a multi-lens camera the focal length / changes based on the crop factor. Ensuring the ability to determine the focal length / is why distance detection using a device camera is typically limited to execution environments (e.g., native applications) where these camera specifications are easily obtained or at least consistent on a particular device model or platform. Determining the focal length / of a camera given unknown camera specifications can be impractical to implement for supporting near real-time distance detection. Calculating the focal length f may require manually measuring how the camera performs in the physical world. This calculation can be error prone; the most accurate way to determine the focal length / when it is unknown is to perform camera calibration, essentially taking several pictures of a checkerboard or rings of circles from different angles, or manually measuring the focal length / with a tape measure. This step makes sense if calibrating a camera that will be used to take pictures for model training, for example, but may be too complicated for most other use cases.

[0035] Determining the size R or dimensions of the object can also be too complex and impractical to implement within a near real-time distance detection application. For example, to calculate the distance of an object from the camera, the width of the object must be known. If the width of the object is not known, a calibration step using an object of known size (e.g., a frame of reference) is required. For example, a standard-size piece of paper or a standard-width credit card may be included in a camera FOV and used as a reference to determine the relative size of the target object in a frame. In other solutions, estimated object sizes can be used based on the precision requirements of the solution. For example, when determining a distance from a camera to a person’s face, an average pupil separation distance of sixty -three millimeters can be used as an approximate size R. w hich can then be extrapolated based on pixel distances for determining an overall width of that person’s face.

[0036] Further challenges to implementations of camera-based distance detection include not overloading computing resources on the device (or those made available to the application) due to the complexity' of processing a video stream frame-by-frame. For example, detecting the distance of an object from a camera in near real-time traditionally entails processing a video stream frame-by-frame using computer vision techniques to detect the object in each frame of the video. Even though nonnative application libraries, such as OpenCV.js, offer computer vision functions that enable web applications to do this, processing each frame of a video is resource-intensive and may result in poor performance and user dissatisfaction. Often video is captured at a high frame rate, such as twenty-four or thirty frames per second (FPS). Processing many frames of a video feed from a non-native execution environment (e.g., in a web browser) is not practical, especially on already resource-constrained mobile devices with limited compute time, reduced memory, and minimal process or thread resource availability.

[0037] Despite the challenges of resource-constrained and / or cross-platform execution environments, the device application 112 implements camera-based functions, including video-based distance detection, without consuming excessive processor time, memory, or storage. In general, thedevice application 112 determines object distance estimates by: processing the camera feed 110, including extracting one or more video frames (or snippets), followed by performing object recognition on the extracted video frames or images, and lastly by calculating the relative distance of the objects recognized. Because the details of the camera specifications may generally be unknown, and a manual calibration step to determine a camera focal length f can be too complex, time-consuming, or inconvenient to support real-time decisions, the device application 112 avoids calculating the focal length of the camera 108. Instead, the device applications 112 executes using various assumptions about the camera 108 and the camera data in the camera feed 110. These assumptions enable the device application 112 to determine object distances without needing to know certain technical specifications for the camera 108, improving the efficiency of the end-user experience associated with the camera interface 116.

[0038] In addition, knowing the object size R is often a limiting factor when attempting to detect the distance of an obj ect from a camera. The device application 112 avoids calculating the obj ect size R. by instead inferring the object pixel size for a well-known distance. In the camera interface 1 16, a reference width 120 between pupil centers of a user’s face or across an entire width of the user’s face, is annotated within the preview- 1 1 of FIG. 1 . The reference width 120 is not typically annotated or displayed in the preview 118. By extrapolating the reference width 120 for different predetermined distances from the well-known distance, the device application 112 can determine an accurate face width to be used in efficiently estimating the object distance.

[0039] Fig. 2 illustrates the computing device 200 for enabling efficient camera-based distance detection, in accordance with techniques of this disclosure. For example, the computing device 200 performs near real-time distance detection using the camera 108 and executing the device application 112.

[0040] The computing device 200 is an example of the computing device 102 and includes the camera 108, the camera feed 110. and the device application 112. The computing device 200 furtherincludes one or more processors 202, a computer-readable storage media 204 (referred to simply as CRM 204). one or more communication units 206, one or more input / output units 208, and an optional, application container 210. As one example, the application container 210 is a web browser that executes the device application 112 as part of a user interface process or thread, and optionally executes one or more background processes or threads (e.g., Web Workers).

[0041] The CRM 204 maintains instructions and data accessed by the processors 202 to perform computing functions. For example, the CRM 204 stores the camera feed 110 (or a portion thereof) and executable code associated with the device application 112. The CRM 204 may include a memory' portion, which may be volatile and / or non-volatile memory'. Examples of memory' include Random Access Memory (RAM) for temporary7storage of data and instructions (e.g., for storing runtime instructions), and Read-Only Memory (ROM) for persistent storage of data and instructions (e.g., for storing critical system instructions). The CRM 204 may include storage apart from memory portions; examples of storage include hard drives, solid-state drives, or other storage media. Data and instructions maintained in the storage of the CRM 204 may be for persistent, long-term data storage and retrieval.

[0042] The processors 202 may be CPUs and / or GPUs. The processors 202 retrieve instructions and data from the CRM 204 to execute applications, processes or threads, and other modules implemented in software or firmware. For example, the processors 202 execute instructions associated with the application container 210, yvhich within one or more process or thread execution environments can execute the device application 112. The processors 202 may facilitate the camera 108 to write information to the camera feed 110, which may also be maintained in the CRM 204. The processors 202 include hardyvare processors configured to execute instructions maintained in the CRM 204 to execute the device application 112 as one or more processes or threads. The processors 202 may execute the application container 210 based on instructions stored in the CRM 204. In cases yvhere the application container 210 is executed by the processors 202, the application container 210 mayfunction as a virtual machine configured to execute the device application 112 within an execution environment provided by the application container 210. In other words, the processors 202 may be hardware processors configured to execute instructions of the application container 210. The application container 210 may function as a software processor or virtual machine that processes an instruction set associated with the device application 112 to perform camera-based distance prediction functions within the cross-platform environment of the application container 210.

[0043] The communication unit 206 enables communication between the computing device 200 and a remote computing device, such as the computing system 106. The communication unit 206 may be a telecommunications radio configured to communicate over wired or wireless (e.g., cellular, Wi-Fi, Ethernet) connections (e.g., made on the network 104). The communication unit 206 may package and output information exchanged over the network 104 and unpack and receive information exchanged therein. The server application 114 executing on the computing system 106 may access the camera feed 110 by communicating with the communication unit 206. The server application 1 14 may receive subsets of the camera feed 110 batched and output from the device application 112 using the communication unit 206.

[0044] The input / output units 208 can include various input components (e.g., microphones, keyboards, touchscreens, buttons, switches) and various output components (e.g., displays, speakers, haptic devices). The input / output units 208 may facilitate a user interface on the computing device 200. For example, a display of the input / output units 208 may generate graphics for the camera interface 116 and activate certain pixels of the display to convey the preview 1 18.

[0045] The processors 202 can retrieve the instructions and data associated with the application container 210 to provide an executable environment for invoking and executing the device application 112. In some examples, the application container 210 is omitted and the device application 112 executes outside the application container 210 in a native, resource-constrained execution environment.

[0046] The device application 112 is an executable set of instructions that, when executed on or by the processors 202, performs near real-time distance detection from within the execution environment of the application container 210 (e.g., a web browser) or other (i.e., native) resource- constrained execution environment. Within the device application 112 are one or more submodules, functions, and / or libraries. A distance detection instruction set within the device application 112 may be executed for determining image-based distances of objects captured by the videos obtained through the camera feed 110. The camera 108 can write video or image data to the camera feed 110, which is maintained by the CRM 204 and read by the device application 112. The distance detection function of the device application 112 may rely on the server application 114 to support its execution. The device application 112 processes the camera feed 110 (e.g., with or without utilizing support from the server application 114) as a live video feed, including extraction of video frames, performing object recognition on video frames, and calculating the distance of the objects.

[0047] The device application 112 is operable to infer camera properties needed to perform camera-based distance detection, without having to determine the focal length of the camera 108 or the precise object size R. The manual calibration steps normally required to determine the camera 108 focal length / are painstaking and not practical as part of an efficient, user experience. The device application 1 12 operates under various rules or assumptions to avoid needing to know about the camera 108 hardware itself. Eliminating the need-to-know hardware specifications of the camera 108 represents a major simplification when performing real-time distance detection, especially from execution environments like the application container 210.

[0048] Instead of calculating the variables of Equation 1, the device application 112 uses an assumed FOV size to quickly arrive at the focal length / Specifically, if the FOV of the camera 108 is known, the focal length / of the camera 108 can be quickly derived from the following formula: f=A / tan(a)Equation 2.In Equation 2, A represents half of the image size, and a represents one half of the size of the FOV. If the camera 108 is a ty pical webcam, it may have an average FOV of between sixty and ninety' degrees horizontally, vertically, and diagonally . In fact, in many real-world front-facing camera use cases, after being cropped, most every device camera has a consistent effective FOV size.

[0049] In other cases, with a known range of FOV as a guideline, it is possible for the device application 112 to estimate the FOV with some accuracy. The exact approach differs per use case. For example, if the use case uses facial recognition, the ratio of the face width measurement in pixels to the frame width in pixels (which is fairly consistent) can be used to estimate the FOV. This assumes that the facial recognition will consistently occur when users are at a similar distance from the camera 108. This approach takes the crop factor into account as well. Further, as it turns out, most web applications, whether running on a desktop or mobile device, seem to apply a crop factor such that the effective FOV regardless of the particular camera is sixty' -five degrees. Regardless of the type of device (e.g., mobile phone, desktop, laptop, loT device), and regardless of the camera, most Internet- connected devices, crop the FOV of the camera 108 to be about sixty -five degrees for front-facing camera use cases. This means, regardless of variability in the FOV size across different cameras, the effective FOV used for the camera 108, wtien adjusting for crop factor, has a small range. This makes estimating the FOV based on Equation 2 an accurate approach to calculating the focal length f As such, the device application may use a fixed constant for FOV and, likewise, for the focal length f.

[0050] Alternatively, instead of estimating the FOV to calculate the focal length f the device application 112 may obtain device and camera characteristics from the application container 210. For example, in the case of a web browser, the application container 210 maintains a “User Agent” string. From querying the User Agent string from the application container 210, the device application 112 can determine information about the computing device 200 and infer information about the camera 108. This approach to determining the focal length / requires the User Agent string maintained by theapplication container 210 to contain sufficient information to distinguish between different devices and different cameras.

[0051] Knowing the object size R is another limiting factor when attempting to detect the distance of an object from the camera 108, regardless of whether the device application 112 is running in the application container 210, or in another example, with it running natively. In lieu of needing to know the object size R. the device application 112 instead has different ways to use an object’s pixel size when taken from a well-known or assumed distance to estimate this parameter.

[0052] The need to know the object size R is unnecessary if its width in pixels (e.g., original pixel width) is known at a certain distance from the camera 108. The original pixel width can be determined by the device application 112 by determining an initial pixel measurement at an initial, known distance. For example, an initial pixel measurement can betaken at approximately arm’s length away (e.g., one or two feet) from the camera 108, which is a typical distance from a webcam or from a mobile device when a user is holding it in front of their face to take a “selfie”. Other instructions can be provided to a user, such as “come closer” or “back up,” to get the user into the desired target range for calibration. The camera interface 116 shown in Fig. 1 includes a text prompt indicating the device application 1 12 is calibrating or initializing while the user of the computing device 102 holds the camera 108 at approximately 1.2 feet from her face (e.g., approximate arm’s length).

[0053] In this scenario, the calculation of the distance of an object from a camera distills down to a simple ratio: calibration width in pixels object width in pixels calibration distance object distanceEquation 3. Equation 3 may be re ritten as: object width in pixels X calibration distance object distance = - — - — - ; - calibration width in pixelsEquation 4.

[0054] Fig. 3 illustrates a data flow 300 for enabling efficient and / or cross platform camerabased distance detection, in accordance with techniques of this disclosure. The data flow 300 occurs in modules or instruction sets associated with the camera 108, the device application 112, and the server application 114. As shown, the server application 114 is optional and the data flow 300 may execute in its entirety on local processors (e.g., the processor 202) of the computing device 102. In addition, the device application 112 may execute in a native environment or within the optional application container 210.

[0055] The camera 108 is shown providing a camera API 316 from which the camera feed 110 is accessed. The device application 112 is shown executing a device thread 302 and one or more background threads 304, which may optionally be referred to as device and background processes. For instance, the device thread 302 may be a function executed with a user interface thread of a web brow ser and the background threads 304 may be Web Worker threads of that same brow ser. The server application 114 executes a server thread 306 or server process. A camera link 308 is established between the camera API 316 and the device thread 302. An application link 310 is made to enable communication between the device thread 302 and the background threads 304. A server link 312 and a server link 314 are made via the network 104 between the device application 112 and the server application 1 14 to allow communication to occur between, respectively, the server thread 306 and the device thread 302 or the server thread 306 and the background threads 304.

[0056] The background threads 304 and the server thread 306 are support threads to the device application 1 12. As discussed already, the device application 112 may be running in the application container 210 (e.g., web browser, other cross-platform environment) or in other applications, in a native, resource-constrained environment. In either case, with limited bandwidth, compute-time, memory, and thread resources, the device application 112 cannot easily process every video frame in the camera feed 110. Retying on the device application 112 to process every video frame may be impractical, if not impossible. The device application 112 may rely on the background threads 304and / or the server thread 306 to handle some of the distance detection processing. By utilizing the background threads 304 and / or the server thread 306, resource consumption by the device thread 302 may be significantly reduced. The background threads 304 and / or the server thread 306 may assist the device thread 302 to provide various advantages, including more efficient near real-time video processing.

[0057] The device thread 302 invokes the camera API 316 to have access to the camera feed 110. For example, the camera link 308 may be accessible via the camera API 316. The device thread 302 may issue a tum-on command over the camera link 308 to cause the camera 108 to power on and gain access to the camera feed 110. In some cases, permissions to utilize the camera 108 may be validated by the camera API 316 based on information the device thread 302 sends to the camera 108 via the camera link 308.

[0058] Once permission has been validated and the camera API 316 is invoked by the device thread 302, the device application 1 12 may give the background threads 304 and / or the server thread 306 access to the camera feed 110 through the camera API 316, as well. For example, the application link 310 is established to share information between the device thread 302 and the background thread 304. The background threads 304 may access the camera feed 1 10 via information relayed by the device thread 302 over the application link 310. In other examples, rather than rely on the device thread 302 to package or select data from the camera feed 110 for processing by the background threads 304, the background threads 304 establish their own communication link with the camera 108 (e.g., by negotiating permission with the camera API 316 to access the camera feed 1 10 like is done by the device thread 302). Similar to the background threads 304, the server thread 306 can access the camera feed 110 via information relayed by the device thread 302 over the server link 312. The server thread 306 may access the camera feed 110 via information relayed by the background threads over the server link 314. In other examples, rather than rely on the device thread 302 or the background threads 304, the server thread 306 establishes a communication link with the camera 108.

[0059] Causing more efficient real-time video processing using the background threads 304 and / or the server thread 306 can be difficult. Offloading process intensive work from the device thread 302 to the server thread 306, with voluminous data (e.g., the camera feed 110), can be difficult because limitations in speed and bandwidth associated with data transfers over the network 104 can eliminate efficiency gains otherwise gained from processing off-device. In other words, to improve efficiency, the offloading needs to occur quickly and with high bandwidth. Offloading from the device thread 302 to the background threads 304 may be less of a problem than offloading to the server thread 306 because often the background threads 304 share memory space with the offloading, device thread 320. However, this is not always the case (e.g., when executing in the application container 210, such as a web browser). For example, offloading to Web Worker threads can be challenging just like offloading to a server thread. To accomplish low latency, offloading to the server thread 306 or to the background threads 304 using HTTP, for example, is too slow to support near real-time functions. However, using WebSockets can improve performance.

[0060] In addition, reducing the amount of data offloaded can improve performance. Many devices, especially mobile devices, do not have enough local processing capability to encode and upload an entire video stream to a server or share information between processes or threads on the local device. Further, due to the nature of how video files are packaged and transmitted, the server thread 306 and / or the background threads 304 may require an entire video file to be loaded prior to extracting individual frames. One solution is to limit the amount of video processed from the camera feed 110, limit the amount of video data uploaded from the device thread 302 to the background threads 304 and / or to the server thread 306. and minimize the size of the video data (e.g., the video size) to make it easier for the background threads 304 and / or the server thread 306 to process. Sending smaller batches of video data (e.g., quarter second of video per second instead of the full second of video per second) may improve performance.

[0061] For example, the device thread 302 may create a new MediaRecorder instance on some interval of the video from the camera feed 110. The device thread 302 may create a new MediaRecorder every' one second, which uses a quarter second of video data (e.g., six to eight frames as opposed to twenty -four or thirty' frames). When each recording is complete, the device thread 302 uploads or shares the recording with the server thread 306 and / or the background threads 304 for further processing. In this scenario, the amount of video data uploaded to the server thread 306 and / or the background threads 304 is reduced by seventy-five percent; the server thread 306 and / or the background threads 304 can load and individually process each of these smaller video files more quickly, thereby eliminating the need to stream the video file between threads, or for the server thread 306 and / or the background threads 304 to load the entire camera feed 110 for processing. The server thread 306 can use robust Python or C++ based OpenCV (computer vision) libraries to perform object recognition and the distance calculation.

[0062] In other examples, the device thread 302 may take a screenshot of the preview 118 displayed in the camera interface 1 16. For example, this can be performed using an HTML canvas element every N seconds. However, the WebM MediaRecorder file type may be proportionally much smaller than these types of image files and may have more frames / images contained within. That is, using a canvas to create a base64 PNG data URL every second may be of similar size to a quartersecond WebM or MediaRecorder file, but the video file will contain many frames / images (typically around 6 per quarter second) versus the single PNG image.

[0063] Using traditional HTTP transport may be too slow for a near real-time experience when fractions of a second matter. To reduce latency to the extent possible, the device thread 302 may upload video data to the server thread 306 and / or the background threads 304 using a Web Socket API (Application Program Interface). Similarly, using an in-memory database, like Redis, can be useful if there is a need to distribute the video processing between nodes of the computing system 106 (e.g., different processors executing the sen' er application 114).

[0064] Regardless, even when implemented per the above, bandwidth and connectivity constraints can lead to poor performance and moderately powered devices may still struggle to process and upload video data quickly enough. Pursuing a browser-only implementation (i.e., without relying on the server thread 306) may be a more reliable way to accomplish near real-time distance detection, unless bandwidth and device performance over the network 104 is known.

[0065] Figs. 4-1 and 4-2 illustrate an example use case for enabling efficient and / or crossplatform, camera-based distance detection, in accordance with techniques of this disclosure. There are many possible use cases for adopting the described techniques to determine how far an object is from a camera. A visual acuity test or other vision tests are examples for camera-based distance verification. Vision tests require users to be at a specific distance (or a small range thereof) before recognizing visual targets to check for visual health. When applied to this scenario, the described techniques enable vision test applications to execute in resource- constrained and / or cross-platform environments, which then help users get to a correct distance from a visual target (e.g., an eye chart displayed on a device screen) without needing to use a tape measure or counting steps.

[0066] Turning first to Fig. 4-1, after presenting the camera interface 116, the computing device 102 presents a camera interface 400, which includes a video preview 402 and a status indicator 404 that is overlaid on the video preview 402. The estimated distance inferred by the device application 1 12 is displayed along with instructions for the user of the computing device 102 to create more distance between her and the camera 108. In some examples, the device application 112 determines that too much latency or lag in determining the distance is caused by the user moving too quickly. The camera interface 400 may convey to the user to slow dow n if they are moving away too fast and their face cannot be recognized with sufficient accuracy. In other words, the object recognition function may report an error or low confidence associated with the pixel width determined and refrain from updating the distance estimate accordingly.

[0067] Turning next to Fig. 4-2, after presenting the camera interface 400, the computing device 102 presents a camera interface 406 as the user steps back. A video preview 408 and a status indicator 410 are overlaid on the video preview 408. The estimated distance inferred by the device application 112 is displayed along with instructions for the user of the computing device 102 to keep looking at the camera 108. When the status indicator 410 completely encircles the video preview 408, the user can see that the acuity test is complete.

[0068] Another reason to adopt the described techniques is to implement a resource constrained and / or cross platform, camera-based proximity' sensor alert system. For instance, the described ways to perform camera-based distance detection may enable a system to generate alarms when people, animals, or machines (e.g., vehicles) encroach upon protected areas. The described techniques are universally adoptable across different devices and platforms, in addition to being computationally efficient and lightweight for use in loT devices. This technology may be incorporated into traditional cameras, baby monitors, and other loT or embedded systems to make them smart without having to integrate more expensive processors and memory devices.

[0069] Other use cases for the described techniques include distance-based device controls. Audio volume adjustment and / or graphical element (e.g., icon, text) size within a graphical user interface (GUI) can be automatically adjusted based on camera-based distance detection. For instance, the disclosed techniques enable the creation of web applications that can automatically change attributes based on how far away a user is from a device screen. This may be useful for video streaming applications, video conferencing, gaming, or any web application consumed from some distance away from the device.

[0070] Another use case for this distance detection approach is to support augmented reality. For example, the techniques can be used to implement a cross-platform, cross-browser compatible distance measurement tool. This may be used to determine relative distance and / or size attributes ofpeople and objects captured in a camera FOV, which may replace the need for a physical tape measure or laser measurement tool.

[0071] Fig. 5 illustrates a computer-implemented process 500 for enabling efficient and / or cross platform camera-based distance detection, in accordance with techniques of this disclosure. The process 500 includes steps 502 through 510, which may be performed in a different order than that shown in the drawing. In addition, some of the steps 502 through 510 may be omitted or repeated or additional steps may be included. For example, the steps 506 and 508 may be repeated multiple times before continuing to the step 510. The process 500 is described in the context of Fig. 1, including operations performed by the device application 112. It should also be understood that the device application 112 may rely on assistance from the server application 114 to execute one or more of the steps of the process 500.

[0072] Imagine a user of the computing device 102 wishes to perform an online vision acuity test to check the health of their eyes and / or whether a prescription for eyeglasses is warranted. From a web browser executing on the computing device 102, the user navigates to a website that hosts the vision test. The user can log in to the website and click on a web element to initiate the vision acuity test. In response to the user input for initiating the test, the device application 112 begins executing within the web browser execution environment (e.g., as provided by the application container 210).

[0073] At step 502, a camera is activated to capture video of an object at a predetermined distance. For example, upon clicking on an element displayed in the web browser to activate a vision test, the device application 1 12 causes the camera 108 to turn on. This causes the camera 108 to begin outputting camera data through the camera feed 110. The camera feed 110 streams video captured of the user as the user looks into a lens of the camera 108. The device application 112 causes the camera interface 116 to be output for display on a screen of the computing device 102. The camera interface 116 may include a prompt (e.g., text, audio) for instructing the user to hold the computing device 102such that the camera 108 is at an original or predetermined distance (e.g., one or two feet, arm’s length) away to cause their face to fill up the preview 118.

[0074] At step 504, an object pixel width at the predetermined distance is determined. For example, when the user is at an original or predetermined distance of approximately an arm’s length from the camera 108, the device application 112 performs object recognition techniques to determine the reference width 120 between the pupils of the user’s face, or across the entire width of the user’s face. If the user’s face is too close and not far enough away from the camera 108, then the entire width of the user’s face cannot be measured. Likewise, if too much margin exists in the video around the person’s face, then the user is too far from the camera 108 to begin the test. The device application 112 can output instructions in the user interface 116 to indicate to the user to move closer or further away from the camera 108 until their face fills the FOV and minimizes the amount of margin on either side. The device application 112 determines the number of pixels that span across the reference width 120 and saves that parameter as an original width in pixels. The original width in pixels is stored with the predetermined distance, which corresponds to the arm’s length (e.g., one foot, eighteen inches, two feet). Applying the ratio of Equations 3 and 4, the device application 112 can determine subsequent object distances from subsequent video frames in seemingly real-time by updating the object width measured in pixels.

[0075] At step 506, an increased distance between the camera and the object (e.g., the user) is estimated based on video of the object. For example, with the object pixel width at the predetermined distance known, the device application 112 can update the camera interface 116 to indicate that the user is to move away from the camera 108. The user may prop the computing device 102 on a table or tripod while keeping their face in the FOV of the camera 108. The device application 112 can execute an object recognition routine to identify the person’s face within the video and count the new quantity or number of pixels that span the reference width 120. As the quantity of pixels along the reference width 120 changes, the device application 112 determines the distance between the user and the camera108 is increased or decreased. The device application 112 determines the object distance by multiplying the original distance (e.g., one foot) with a ratio between the new quantity' of pixels and the original width in pixels. The reference width 120 may be some other measurement on the person’s face; for example, rather than ty pical pupil distance, there may be other features of people that are common / consistent (e.g., human body part ratios) that can be extrapolated to be an overall face width.

[0076] At step 508, whether the increased distance satisfies a desired distance threshold is determined. For example, the camera interface 116 may prompt the user to back up from the camera 108 to ten feet. As the user backs up, the camera interface 116 can indicate the estimated distance computed at step 506. If the useris not far enough away from the camera 108 (i.e., a NO outcome from step 508), then step 506 is repeated with further emphasis from the camera interface 116 that the user keeps backing up. If the user gets to a position that is far enough away from the camera 108 (i.e., a YES outcome from step 508), then the process continues to step 510, and information displayed in the camera interface 1 16 communicates to the user to stop and remain at their current position. As shown in Fig. 4, a progress indicator may be used that encircles the preview 118 completely when the YES outcome from step 508 is achieved.

[0077] At step 510, a computing function is executed based on the video of the object at the increased distance. For example, the device application 112 determines that the user is located at the proper position for performing the vision acuity test. The camera interface 116 may convey this by outputting a large, green check mark to indicate success reaching ten feet away. The device application 1 12 can then output a graphical user interface associated w ith the vision test. The information collected from further audio and video captured of the user at the distance from the camera 108 may be used to diagnose their vision.

[0078] During steps 504 and 506, the device application 112 can rely on the server application 114, as discussed above with reference to Fig. 3. to cause more efficient and / or cross-platform realtime video processing by the device application 112. There are other ways that more efficient and / orcross platform video processing can be performed without relying on the computing system 106 and / or the server application 114. For example, several ways that performance can improve on the computing device 102, and within the device application 112 itself, include: reducing the number of processed video frames (e.g., to significantly reduce web browser resource utilization), maximizing object recognition effectiveness at low FPS using multiple models and / or using distance-optimized models, and reducing measurement inconsistencies at low FPS using median or modified median techniques.

[0079] The device application 112 may analyze the video stored in the camera feed 110 in part by reducing the number of processed video frames, which can significantly reduce the web browser (or application container) resource utilization. The device application 112 may rely on the OpenCV.js library available from the web browser (i.e., the application container 210) as a most efficient way to perform object detection in support of real-time distance detection using the video captured by the camera 108. However, using this library as recommended to process a typical video frame rate of thirty FPS is not practical on typical mobile devices. Instead, reducing the number of video frames processed, as much as possible, is desired. Too much frame reduction may negatively impact accuracy of the realtime distance detection; care should be taken to not impact accuracy while still reducing the computetime, memory, and process or thread resources required by the device application 1 12. For example, typically, anything lower than two FPS reduces the ability of the device application 112 to accurately account for movement. However, even at five FPS, some mobile devices lack available resources to keep up with the distance estimations, and struggle in performance. Therefore, a balance needs to be struck between object detection accuracy and resource availability in executing the device application 1 12 to use just enough video frames, but not too few or too many.

[0080] In some examples, when executing in a web browser or the application container 210, to reduce the number of frames, the device application 112 may use a recursive setTimeout function that only processes a predetermined number of frames per second of video captured in the camera feed 110. This implementation may work well as it is time-based and, therefore, does not attempt to processany frames of the video while waiting, which frees up computing resources to enable the computing device 102 to perform other tasks. This approach may require the use of an HTML Canvas API or an HTML Offscreen Canvas API, which have high cross-browser compatibility, but can otherwise be resource intensive. There is a potentially more resource-optimized way to capture an image frame-by- frame using MediaStream Image Capture API, which can be used to process every N frames (where N is any integer). However, this approach may have limited browser support and not preferred in more advanced application containers or browser environments.

[0081] Reducing the number of video frames processed can significantly conserve resources. However, depending on the robustness of the object recognition model, lowering the FPS could significantly impact the effectiveness of the real-time distance detection itself.

[0082] The device application 112 may analyze the video stored in the camera feed 110 in part by maximizing the effectiveness of its object recognition routines at low FPS in part by using distance- optimized obj ect recognition models. In other words, in addition to the device application 112 reducing the number of video frames processed to reduce the compute, memory, and process or thread resources required to perform the steps 504 and 506, the device application 112 may also utilize efficient and effective object recognition models when performing real-time distance detection using a camera in a web browser.

[0083] In some examples, the device application 1 12 executes an object recognition model that is a single, efficient, and effective model, which works well and is robust at all of the distances being measured. For example, the model used may be suited for near distances from the camera 108 (e.g., up to twelve feet) as opposed to other models that are suitable for long range (e.g.. greater than ten feet). In some cases, multiple models may be used; the device application 112 may use one model for the calibration or initialization step, which is optimized for two feet or less. The device application 112 may use a second model that has improved performance over the first model for distances greater than two feet (e.g., up to fifteen feet).

[0084] For example, with the device application 112 being a browser application intended to detect how far a person is from the camera 108, using a facial recognition model in the first zero to four feet might be highly reliable, but that same model’s performance might start to degrade beyond four feet. Therefore, instead of using the facial recognition model beyond four feet, the device application 112 may use an upper-body recognition model (instead of the facial recognition model). The device application 112 may use a full-body recognition model beyond 10 feet. The device application 112 may execute the frontal face recognition model in conjunction with the upper-body recognition model for estimating distances from the camera 108 of between three feet and five feet; the upper-body recognition model and full-body recognition model might execute by the device application for estimating distances between nine feet and eleven feet.

[0085] When the device application 112 is executing the steps 504 and 506 to perform near real-time distance detection using the camera 108 (e.g., while executing in a web browser process or thread at a low frame rate), any anomalies in the object recognition models can become pronounced. For example, sometimes artifacts appear in video frames, which are incorrectly recognized as target objects. When performing distance detection using the camera 108 in a native application, these inconsistencies are often masked or omitted because the native application can process a large volume of frames (e.g., thirty frames per second). However, the device application 112 performs the analysis at a low frame rate from the non-native execution environment (e.g., within a web browser), and as a result, these inconsistencies negatively impact the accuracy of object recognition models. The inconsistencies diminish the effectiveness of any real-time distance measurements performed by the device application 112, which may ultimately lead to a poor user experience. The device application 112 may reduce measurement inconsistencies that may otherwise occur at these low frame rates by using mean, median, or modified median model measurements.

[0086] For example, to minimize the impact of anomalous measurements, the device application 112 may determine at the steps 504 and 506 a mean value, which is taken as an averageover a specific time frame. Although less complex to implement, taking the average or mean value may not sufficiently reduce the inconsistencies enough to provide a noticeable improvement in accuracy. Therefore, in other examples, the device application 112 determines a median measurement value over a specific time frame. For example, the median measurement value can be computed from the following:Equation 5. Using a median over a mean average may be useful for enabling the device application 112 to prevent any anomalous high and low outlier values from impacting a final distance measurement.

[0087] Using median measurements, however, causes other challenges. Due to the reduced frame rate, capturing enough measurements to be used as inputs to a median function takes time, which limits the ability of the device application 112 to perform distance detections in near real-time. Therefore, to ensure the real-time nature of the device application 112 it is important to minimize the time for capturing measurements. For example, at one FPS, using a rolling median of the last three measurement values takes three seconds, which results in a three-second delay for the end user. A solution may be to find a balance between FPS and processing delay, which is sufficient for the pertinent use case.

[0088] Further, if it is known that anomalous measurements from the model(s) being used by the device application 1 12 are typically smaller (or larger) than the object itself, a modified median value can be used to reduce the number of frames needed for its output to be effective within one frame per second. The maximum of either the (^)thterm or the + l)t / lterm from Equation 5 can beused as the modified median when the measurements tend to be too small. When measurements tend to be too large, the minimum of either the (^)thterm or the+ l)t / lterm from the Equation 5 can be used. For example, at four frames per second, if it is known that anomalous outputs of the model (s)of the device application 112 tend to be smaller than the object itself, a modified median, which is the maximum of the middle two values is used, instead of the mean of the middle two values. Otherwise, five frames per second would be required to throw out up to two low outliers. In this example, given a set of four samples (e.g., 10, 67, 100, and 101) where the expected value is around 100, the median is 83.5, which is 16.5 different from the expected value of 100. Whereas the modified mean that is equal to the maximum of the two middle terms is exactly 100, which is the expected value.

[0089] While various embodiments of the disclosure are described in the foregoing description and show n in the drawings, it is to be understood that this disclosure is not limited thereto but may be variously embodied to practice within the scope of the following claims. From the foregoing description, it will be apparent that various changes may be made without departing from the spirit and scope of the disclosure as defined by the following claims.

[0090] The use of “or” and grammatically related terms indicates non-exclusive alternatives without limitation unless the context clearly dictates otherwise. As used herein, a phrase referring to “at least one of’ a list of items refers to any combination of those items, including single members. As an example, “at least one of: a, b, or c” is intended to cover a, b, c, a-b, a-c, b-c, and a-b-c, as well as any combination with multiples of the same element (e.g., a-a, a-a-a, a-a-b, a-a-c, a-b-b, a-c-c, b-b, b- b-b, b-b-c, c-c, and c-c-c or any other ordering of a, b, and c).

Claims

CL IMS1. A computer implemented method for resource-constrained and / or cross-platform camera-based distance detection without knowledge of camera specifications, the method comprising the steps of instructing a camera (108) to capture calibration camera data of an object, wherein the object is located at a calibration distance from the camera (108) and wherein the calibration camera data is an image captured using the camera (108), or a frame of a video captured using the camera (108), detecting the object on the calibration camera data, determining a calibration width of the object detected on the calibration camera data, instructing the camera to capture additional camera data of the object for distance detection, detecting the obj ect on the additional camera data captured for distance detection, determining an object width of the object detected on the additional camera data captured for distance detection, and estimating the object distance based on the object width, the calibration width, and the calibration distance.

2. Method according to claim 1, wherein after determining a calibration width of the object, the camera (108) is instructed to continuously capture camera data in the form of video frames or images, and wherein the method further comprises the steps of selecting a plurality of the video frames or images captured as additional camera data for distance detection, detecting the object on each video frame or image selected for distance detection, determining an object w idth of the object detected on each video frame or image selected for distance detection, and estimating the distance of the object on each video frame or image selected for distance detection based on the object width of the object on the selected video frame or image, the calibration width, and the calibration distance.

3. A computer-implemented method for placing a user at a specific distance from a visual target for performing a vision test using a resource-constrained and / or cross-platform camera-based distance detection without knowledge of camera specifications, the method comprising the steps of instructing a camera (108) to capture calibration camera data of an object formed by the user using a camera (108) attached to the visual target, wherein the obj ect is located at a calibration distance from the camera (108) and wherein the camera data is an image or a video frame captured using the camera (108), detecting the object on the calibration camera data, determining a calibration width of the object detected on the calibration camera data, instructing the user to increase or decrease the distance between the user and the camera (108) until the user is at the specific distance based on a detected object distance of the user from the camera (108), wherein for estimating the estimated distance of the user from the camera (108), the method further comprises the steps of instructing the camera (108) to continuously capture additional camera data in the form of video frames or images, selecting a plurality of the video frames or images captured as additional camera data for distance detection, detecting the object formed by the user on each video frame or image selected for distance detection, determining an object width of the object detected on each video frame or image selected for distance detection, and estimating the distance of the user on each video frame or image selected for distance detection based on the object width of the user on the selected video frame or image, the calibration width, and the calibration distance.

4. Method according to claim 2 or 3, wherein the object distance is estimated based on an estimated focal length of the camera (108), wherein the estimated focal length of the camera (108) is determined from an estimated field of view of the camera (108) and an image size of the camera (108) or wherein the obj ect distance is estimated without knowing or estimating the focal length of the camera (108).

5. Method according to claim 2, 3 or 4, wherein of the continuously captured additional camera data, between two and five images or video frames captured per second are selected for distance detection.

6. Method according to any of the preceding claims, wherein the detected object distance is determined as a median of the distances estimated of the object on a plurality of video frames or images selected over a specific time frame.

7. Method according to any of the preceding claims, wherein the calibration width and the object width are determined as a number of pixels the object spans across the image or the camera frame, respectively.

8. Method according to any of the preceding claims, wherein the calibration distance corresponds approximately to an arm’s length, wherein an arm’s length is one of 30.48 cm (one foot), 45.72 cm (18 inches), or 60.96 cm (two feet).

9. Method according to any of the claims 1 to 7. wherein the calibration distance is a distance at which the object covers a predetermined section of a field of view of the camera (108).

10. Method according to claim 9, wherein the method is configured to instruct a user to change a distance between the object and the camera (108) for capturing the calibration camera data in case the object covers a larger section of the field of view of the camera (108) or a smaller section of the field of view of the camera (108) than the predetermined section.

11. Method according to any of the preceding claims, wherein a first object recognition model is used for detecting the object on the calibration camera data and wherein a different second object recognition model is used for detecting the object on the additional camera data for distance detection.

12. Method according to claim 11, wherein the object is a user, and wherein the first object recognition model is a facial recognition model, and wherein the second object recognition model is one of an upper-body recognition model and a full-body recognition model.

13. Method according to any of the preceding claims, wherein the camera (108) is part of a computing device (102) and wherein the calibration camera data and the additional camera data captured with the camera (108) are transmitted by the computing device (102) to a remote computing system (106) for detecting the object on the camera data, determining the calibration width and object width, and estimating the distance of the object.

14. An execution environment (100) comprising a computing device (102) with a camera (108), wherein the execution environment (100) is configured for implementing a method according to any of the preceding claims.

15. Execution environment (100) according to claim 14, wherein the execution environment (100) further comprises a remote computing system (106) and wherein the execution environment (100) is configured for implementing the method according to claim 12.

16. A computer-readable storage media storing or including instructions that, when executed, cause at least one processor to perform a computer-implemented method according of any of claim 1 to 13.

Images