Portable data processing device and target detection method using the same

Abstract

A portable or wearable data processing device and a method for detecting targets using the device are disclosed. The device, e.g., a head-mounted display device, includes a data processing unit, at least one high-resolution imaging sensor operable to capture an environment around the device within a field of view as image data, a low-resolution ranging sensor configured to detect one or more targets within the field of view within a range interval of the device, and, optionally, a display. Upon detecting the target or targets, the ranging sensor outputs target data encoding detected target characteristics. The data processing unit receives at least the target data and filters targets therein by comparing the detected target characteristics with a target detection threshold. The filtered target data is mapped to a corresponding portion of the image data as the imaging sensor captures the environment.

Description

[Technical Field]

[0001] The present invention resides in the field of portable or wearable data processing devices with an imaging sensor and the detection of targets in image data of the environment surrounding the device captured by the sensor. [Background technology]

[0002] Target tracking and identification as visual cues are increasingly being used to interact with and control portable and wearable electronic devices, such as smartphones and head-mounted display ("HMD") devices, as an alternative to, or in addition to, auditory cues and physical tactile input. HMD devices are digital display devices with data processing capabilities, either built-in or housed in a tethered companion module, that project digital video content into the user's direct field of view, either superimposed on the user's actual physical environment in the case of see-through augmented reality ("AR") glasses, or alternatively as computer-generated imagery in the case of virtual reality ("VR") headsets, or more recently as a combination of a captured physical environment and computer-generated imagery in the case of mixed reality ("MR") headsets.

[0003] Whether in an HMD, smartphone, or the like, visual cues are detected, optically tracked, and then analyzed by the device to recognize structures, such as limbs or hands in the context of a gesture-based interface, and translate them into predefined device commands, such as "pinch" or "close," to command device and / or program functionality. This form of interaction is particularly relevant in the medical field, where sterility requirements and glove wearing often prevent users from using tactile input interfaces such as touchscreens.

[0004] To recognize such visual cues during use, high-resolution image frames of the actual physical environment captured by the camera(s) of the HMD or similar device are input into an image analysis algorithm, which processes them to recognize one or more targets, such as one or both of the cameras or the user's hands present within the field of view of each camera. The recognition is typically based on principles of computer vision or artificial intelligence, e.g., a machine learning algorithm trained on a corpus of images depicting targets, e.g., various hands forming various command gestures, taking into account the size, shape, and color of the targets, differences in environmental brightness and lighting, and other factors, such as the reflectivity and color of the particular material of gloves if the intended context of use is medical.

[0005] A recent example of this technique is the open-source MediaPipe® framework, which first detects the general region of each hand present in an image frame and then detects each actual hand within each such general region. While this approach improves hand and articulated finger detection, it processes the entire image frame, which is not important in the specific context of a gesture user interface, resulting in wasteful data processing. A further drawback of this approach is that it is subject to the optical resolution of the image sensor, and within that range, initial processing of the entire image frame may detect all hands at any distance from the image sensor, thus including the hands of any person in the image frame, even though that person is not the device's user, which the framework may treat as, and ultimately interpret as, an unintended command, i.e., a "false positive" detection that interferes with the intended use of the device.

[0006] Thus, both computer vision and artificial intelligence techniques for target detection require significant computing resources, with correspondingly large power consumption and processing times, which scale with the resolution of the image frames being analyzed. While these techniques continuously process image data to detect targets in virtually real time, their application to and use by portable or wearable electronic devices remains problematic due to limited on-board data processing capabilities and battery power that limits autonomy by design. Summary of the Invention

[0007] Aspects of the present invention are set out in the accompanying claims and are directed to various embodiments of a portable or wearable data processing device and various embodiments of a method of detecting a target using the device, respectively.

[0008] In a first aspect, the present invention provides a portable data processing device comprising: data processing means; at least one high resolution imaging sensor operable to capture an environment around the device within a field of view as image data; a low resolution ranging sensor configured to detect one or more targets within the field of view within a distance interval of the device and to output target data encoding detected target characteristics; and power storage means connected to supply power to the imaging sensor, the ranging sensor and the data processing means, wherein the data processing means is configured to receive at least the target data from the ranging sensor, filter the received target data by comparing the detected target characteristics with a target detection threshold, and map the filtered target data to a corresponding portion of the image data when the or each high resolution imaging sensor is used to capture the environment.

[0009] The low resolution ranging sensor in the device of the present invention advantageously provides the device with a target pre-detection capability, thereby removing the computational overhead associated with performing target recognition from computationally expensive image processing, particularly in situations where recognizing close targets needs to be a permanent technical feature of the device when in use, such as when detecting hands in a gesture user interface. Targets present in the field of view beyond the range interval that would be detected and treated as false positives by prior art techniques are not detected, and targets present in the field of view within the range interval are detected but filtered out because the detected target characteristics exceed the target detection threshold, whereby the data processing means does not process the corresponding image data, thus saving computational power.

[0010] In certain embodiments of the device in which detection is range-based, the detected target characteristics include a distance between the detected target and the low-resolution ranging sensor, and the target detection threshold includes a proximity threshold. In variations that further improve detection accuracy, the proximity threshold can be configurable to be less than the range interval, such that the data processing means filters received target data only if at least a first target reaches the proximity threshold within the range interval. These configurations can be particularly useful when the device, or an application program processed by the device, is intended for use in an environment rich in potential targets within a relatively short range of the device.

[0011] As used herein, a high-resolution imaging sensor should be understood as any camera or other light-based sensor, whether of, for example, RGB, three-dimensional time-of-flight ("3D ToF"), thermal, near-infrared ("NIR"), or event-based type, with an imaging resolution substantially higher, i.e., at least ten times higher, than that of the ranging sensor, with correspondingly substantially higher data output and power consumption. In an embodiment of the apparatus, the data processing means may be further configured to receive image data from the high-resolution imaging sensor and crop a corresponding portion from the received image data. Such an embodiment usefully maintains the full resolution of the image data generated by the imaging sensor(s), particularly with regard to pixel density, but removes redundant image data from the image frame as passed to prior art gesture recognition algorithms, e.g., examples from the MediaPipe® framework, and performs initial hand region recognition based on the cropped partial image data according to the target data provided by the ranging sensor, rather than a pixel-by-pixel or other analysis of the entire image frame using the data processing means.

[0012] In particularly power-efficient embodiments of the device, the or each high-resolution imaging sensor may be switched and triggered to capture image data only when a target is detected. Thus, either the ranging sensor is preferably further configured to switch imaging sensors when it outputs target data, or the data processing means is preferably further configured to switch imaging sensors when it receives target data. In such a configuration, power consumption associated with operation of one or more full-resolution imaging sensors can be conserved until the target is finally verified. Such a configuration is particularly suitable for augmented reality HMDs, which allow the wearer to observe their surroundings through see-through lenses, but imaging of that environment is often redundant.

[0013] In an embodiment of the device, the ranging sensor may be configured to capture the field of view into multiple distinct zones with low-resolution features. Inspired by single-point detection sensors, very low-resolution, low-power sensors of this configuration are known that implement an orthogonal array of distinct cells, such as an 8x8 or 16x16 cell matrix. In such an embodiment, where each cell is assigned a respective identifier, the target data may advantageously be as simple as the respective identifier of the or each zone or cell corresponding to a target in the field of view, and the respective detected target characteristics of such zone or each zone or cell.

[0014] Low-resolution ranging sensors for use in embodiments of the device include a wide variety of sensor types, each preferably implementing time-of-flight (ToF) technology, e.g., integrated circuit (IC) 2 C) operatively connected to data processing means via a protocol-compliant low-bandwidth data connection. According to the principles explained above, a low-resolution ranging sensor is to be understood as any optical, acoustic or other wavelength-based sensor suitable for measuring distance to a target, the resolution of which is substantially lower than that of the imaging sensor, i.e. at least 10 times lower, and the data output and power consumption are correspondingly substantially lower.

[0015] An area in which the present invention is expected to be particularly useful is in gesture user interfaces, where the configuration and / or movement of the device wearer's hands and / or fingers are optically recognized and translated into data processing commands. Accordingly, device embodiments may be particularly developed for use cases in which the or each detectable target is a human hand, and corresponding portions include image data representing the or each human hand. In such embodiments, the data processing means is preferably further configured to process matched portions of the image data into user commands.

[0016] In a variation of such an embodiment, when the low-resolution data from the ranging sensor encodes gesture information sufficient to be translated into a user command, e.g., when the device wearer moves their hand according to a particular direction corresponding to a particular user command, the data processing means may also, or instead, be further configured to process the target data into a user command, this further configuration usefully saving data processing overhead associated with processing portions of high-resolution image data into the same command.

[0017] In another aspect, the present invention provides a method of detecting targets with a portable data processing device, the device comprising data processing means, at least one high resolution imaging sensor operable to capture an environment around the device within a field of view as image data, a low resolution ranging sensor configured to detect one or more targets within the field of view within a distance interval of the device, and power means operatively connected to the imaging sensor, the ranging sensor and the data processing means, the method comprising the steps of: upon detecting the or each target, outputting target data using the ranging sensor encoding detected target characteristics; receiving at least the target data in the data processing means; filtering the received target data by comparing the detected target characteristics with a target detection threshold; and mapping the filtered target data to a corresponding portion of the image data when capturing the environment using the or each high resolution imaging sensor.

[0018] In an embodiment of the method, the detected target characteristics may include a distance between the detected target and the low-resolution ranging sensor, the target detection threshold may include a proximity threshold, and the filtering step may further include comparing the distance between the detected target and the low-resolution ranging sensor to the proximity threshold to exclude target data distal to the proximity threshold.

[0019] Embodiments of the method may include the further steps of capturing the environment with a high resolution imaging sensor, outputting the captured image data to a data processing means, and using the data processing means to crop the image data to the mapped portion.

[0020] In an embodiment of the method, the or each imaging sensor of the device is switchable, and the method preferably includes the further step of switching the or each imaging sensor to capture target data either as it is output by the low-resolution ranging sensor or as it is received by the data processing means.

[0021] In an embodiment of the method, the step of outputting the target data may further include dividing the field of view in the ranging sensor into separate zones, whereby the target data may be a zone or a respective identifier for each zone.

[0022] In an embodiment of the method particularly aimed at gestural user interfaces, the or each target is a human hand and the corresponding portion comprises image data representing the or each human hand, and the method preferably comprises the further step of processing the corresponding portion of the image data, or the target data itself if it encodes sufficient context information, into user commands using data processing means.

[0023] The present invention is particularly intended to detect targets in close proximity to the device, and therefore the distance interval may be 10 to 400 centimeters, or even less, for example 30 to 70 centimeters, to help mitigate false positive detections.

[0024] Other aspects of the invention are set out in the accompanying claims. [Brief explanation of the drawings]

[0025] The invention will be more clearly understood from the following description of embodiments thereof, given by way of example only, with reference to the accompanying drawings, in which: [Figure 1] A prior art handheld device, in this example a head mounted display (HMD) device with a full resolution imaging sensor, and a prior art method for detecting targets are shown. [Figure 2A] 1 illustrates respective embodiments of a wearable apparatus according to the present invention, in this example an HMD device having a low-power ranging sensor and one or more full-resolution imaging sensors. [Figure 2B] 1 illustrates respective embodiments of a wearable apparatus according to the present invention, in this example an HMD device having a low-power ranging sensor and one or more full-resolution imaging sensors. [Figure 2C] 1 illustrates respective embodiments of a wearable apparatus according to the present invention, in this example an HMD device having a low-power ranging sensor and one or more full-resolution imaging sensors. [Figure 3A] FIG. 2C is a functional diagram of an example hardware architecture for the HMD devices shown in FIGS. 2A-2C, each including a memory. [Figure 3B] FIG. 2C is a functional diagram of an example hardware architecture for the HMD devices shown in FIGS. 2A-2C, each including a memory. [Figure 4] 2A to 3B, and shows the fields of view of the image sensor and the distance measurement sensor shown in FIG. 2A to 3B. [Figure 5] The full-resolution image frame and low-resolution detector array corresponding to each field of view shown in FIG. 4 are shown. [Figure 6A] The data processing steps performed by the low-power ranging sensors of FIGS. 2A-4 to populate the array of FIG. 5 are detailed below. [Figure 6B] In addition to the data processing steps of FIG. 6A, optional data processing steps performed by the low-power ranging sensor are detailed. [Figure 7A]2A-6A or 6B show in detail the data processing steps performed by the apparatus of FIG. [Figure 7B] In addition to the steps of FIG. 7A, optional data processing steps performed by the device are detailed. [Figure 8] 3A or 3B at run time when performing the steps of FIG. 7A or 7B. [Figure 9] 5 shows the full resolution image frame of FIG. 4 and the low resolution detector array with data processing steps performed by the apparatus of FIGS. 2A-6A or 6B in an alternative embodiment. [Figure 10] 2A-6A or 6B according to an alternative embodiment, where target data from a ranging sensor is processed into movement and / or orientation data. Detailed Description of the Drawings

[0026] Examples of specific embodiments contemplated by the inventors will now be described. In the following description and the accompanying drawings, numerous specific details are set forth to provide a thorough understanding, and like reference numerals refer to like features. It will be readily apparent to those skilled in the art that the present invention can be practiced without being limited to these specific details. In other instances, well-known methods and structures have not been described in detail to avoid unnecessarily obscuring the description.

[0027] 1, a prior art portable data processing device, in this example an augmented reality ("AR") head-mounted display ("HMD") device, is shown along with a prior art method of detecting targets with the device. The AR HMD comprises a wearer visor 20 including a main see-through portion 22 and video display portions 24A, 24B for each eye positioned equidistantly in a central bridge portion that covers the wearer's nose in use.

[0028] Perceptually, display portions 24A, 24B implement a single video display that occupies a subset of the front of the HMD, allowing the wearer to observe both the surrounding physical environment and the display. Each video display portion 24A, 24B consists of a respective video display unit 26A, 26B, in this example a micro OLED panel having a minimum 60 Hz frame refresh rate and a resolution of 1920 x 1080 pixels, and the video display units 26A, 26B are positioned adjacent to the lower edge of the visor such that the see-through portion 22 extends upward to the upper edge of the visor to ensure no visual obstruction when the VDU is displayed.

[0029] The HMD further comprises a high-resolution optical sensor 30 that captures visible light, typically in the wavelength range of 400 nm to 700 nm, within a field of view of typically 70 to 90 degrees, and outputs image data as a sequence of RGB image frames at a rate of 60 frames per second or greater, with a resolution of at least 1920 x 1080 pixels.

[0030] Upon power-up, the HMD first loads firmware and an operating system ("OS") in step 1, then initializes the imaging sensor 30 in step 2, and then optionally loads additional data processing functions in step 3, such as application programs for processing and rendering information into a user interface that is initialized and output to VDUs 26A, 26B in step 4. If the optional application programs are not present in step 3, the HMD also initializes the OS's user interface in step 4.

[0031] The image data generated by the imaging sensor 30 in step 3 is continuously input to a target detection algorithm, which may be a subroutine of the OS and / or optional application program. Each high-resolution image frame is fully processed, i.e., scanned, by the algorithm in step 5 (e.g., the first stage of the MediaPipe® prior art) to identify one or more targets therein, such as the HMD wearer's hands. If at least one target is successfully detected, the HMD proceeds to recognize whether the identified target encodes a command in step 6 (e.g., the second stage of the MediaPipe® prior art). Thus, in step 7, a question is asked as to whether a command was recognized. If so, the OS or optional application program executes the corresponding data processing command in step 8. Immediately thereafter, the HMD updates the user interface in step 4, as if the question in step 7 had been answered in the negative. A next question is then asked as to whether the HMD should be powered down or whether optional application programs should be terminated and unloaded from memory, which is answered in the negative as long as the HMD remains in use, thereby returning control to step 5 identification of the next image frame from the imaging sensor 30, and so on.

[0032] Considering the context of the prior art described with reference to FIG. 1 , the inventors believed that image data to be processed for target recognition and conversion using augmented reality (AR) HMDs and similar low-power electronic devices should be as small as possible. The inventors also observed that reducing the resolution of the image data is not a desirable solution because it interferes with the recognition and conversion stage. The inventors then determined that a solution should preferably be to crop a full-resolution image frame or each full-resolution image frame to a minimum region of interest (ROI) that includes the entire target, without processing the balance of the image data within the frame or each frame. The inventors then recognized that a low-power, low-resolution ranging sensor can usefully detect targets in distinct portions of a field of view substantially similar to that of the device's camera(s), thereby identifying corresponding portions of interest within the device's camera's full-resolution image frame with minimal computational and power requirements.

[0033] The inventive concepts herein may be implemented in a wide variety of data processing devices, and are expected to be particularly relevant to portable or wearable devices that are powered by an on-board power source and operate separate from substantial computing resources, such as a desktop computer. Accordingly, several exemplary embodiments of portable head-mounted display ("HMD") devices are shown in Figures 2A-3B, where like numerals refer to like features, by way of non-limiting example.

[0034] The first embodiment shown in FIG. 2A is an augmented reality ("AR") type HMD 10A similar to that shown in FIG. 1. The HMD 10A also includes a wearer visor 20 having a main see-through portion 22 and respective eye video display portions 24A, 24B consisting of video display units 26A, 26B, each having a resolution of 1920 x 1080 pixels, positioned adjacent the lower edge of the visor such that the see-through portion 22 extends upward to the upper edge of the visor to avoid visual obstruction when a VDU is displayed. The technical principles disclosed herein may also be implemented in other HMD types, such as a virtual reality ("VR") or mixed reality ("MR") enclosed display device 10B shown in FIG. 2B, in which the respective eye video display portions 24A, 24B perceptually implement a single video display portion 24 occupying substantially the entire inner front surface of the HMD. In such an HMD, each video display portion 24A, 24B may consist of an RGB low-persistence panel 26A, 26B with a minimum 60 Hz frame refresh rate and an individual resolution of 2048 x 1080 pixels per eye, perceived as a single video display with a resolution of 4096 x 2160 pixels.

[0035] Each HMD embodiment 10A, 10B further includes at least one high- or full-resolution optical sensor 30, typically having a respective field of view (FoV) 40 of 70 to 90 degrees, that captures the environment around the HMD as seen within the FoV as visible light, typically in the wavelength range of 400 to 700 nm. In an exemplary operating room, an HMD wearer 36 points a finger 38 at a patient lying on an operating table. The imaging sensor 30, according to this embodiment, permanently or selectively outputs image data 400 as a stream of RGB image frames 410 at a rate of 60 frames per second or greater. Each image frame 400 has a resolution of at least 1920 x 1080 pixels for the AR HMD 10A and at least 2048 x 1080 pixels for the VR / MR HMD 10B.

[0036] Each HMD according to the present invention further comprises at least one low-resolution ranging sensor 32, which includes a respective field of view (FoV) 42 that is similar to the FoV 40 of the imaging sensor 30, i.e., substantially coincident between 70 and 90 degrees, as shown in Figure 4. The low-resolution ranging sensor continuously or periodically polls the same environment around the HMD as seen by the imaging sensor FoV 40 for targets, but within a relatively short distance interval d, ranging from 10 to 400 centimeters from the HMD, that are expected to appear, in this example, the HMD wearer's hand 38.

[0037] The low-resolution ranging sensor 32 may be, by way of non-limiting example, a low-power multi-zone time-of-flight (ToF) sensor that does not require a specific computing unit, as even a microcontroller can process its output data even at relatively high acquisition or polling rates, such as 60 Hz, such as the sensor model VL53L7CX manufactured by STMicroelectronics, Geneva, Switzerland. Those skilled in the art will understand that the present technology may be implemented with other types of low-resolution ranging sensors that, depending on their characteristics and capabilities, may enable selective detection of targets according to type and distance, and / or color-based target detection, such that the dominant color of the FoV 40 can be dynamically detected.

[0038] Ranging sensor 32 is configured to divide observed field of view 42 into a number of distinct zones, in this example 64 zones arranged as a matrix 420 of 8x8 cells 421, the matrix representing the low resolution of the ranging sensor. The aspect ratio of matrix 420 is preferably the same as the aspect ratio of image frame 410, with each cell 421 corresponding to a respective portion of image data 400 within image frame 410, measuring, for example, 240x135 pixels for an image frame size of 1920x1080 pixels, or 256x135 pixels for an image frame size of 2048x1080 pixels.

[0039] Each time the ranging sensor 32 detects one or more targets 38 within each FoV 42 within the distance interval d, it generates a value for each cell 421 representing a characteristic of the target detected within the cell—in this example, because the ranging sensor 32 is a ToF sensor, the distance between the ranging sensor 32 and the target within the FoV 42. Referring to the exemplary scene shown in FIG. 5 , a hand with fingers 38 is located approximately 50 centimeters from the sensor 32, and its corresponding cell within the FoV 42 is assigned a distance value of “50.” A patient, for example, lying on an operating table, is located approximately 100 centimeters from the sensor 32, and its corresponding cell within the FoV 42 is assigned a distance value of “100.”

[0040] The ranging sensor 32 then outputs target data 422 consisting of characteristics of each target detected across the matrix 420. In this example, the output target data 422 accordingly includes each cell 421's respective cell identifier and the respective distance to the detected target, encoding both the distance and matrix location of each target detected within the FoV 42. Thus, depending on the target's size and its proximity to the ranging sensor, a target may be defined by multiple adjacent cells 421 forming a cluster 423. By referencing the same aspect ratio between the image frame 410 and the ranging sensor detection matrix 420, the perimeter of each cell 421, or cluster 423 of multiple cells, defines and bounds a corresponding portion 424 of the image data 400 within the image frame 410, within which the detected or each detected target 38 is or can be captured at full resolution by the imaging sensor 30, representing a valid detection of the target according to principles described below.

[0041] Embodiments of HMDs according to the present invention can include additional sensors, such as an additional high-resolution or full-resolution optical sensor 35 identical to first sensor 30 to provide stereoscopic capture of the surrounding physical environment with visual depth information, as shown in VR / MR HMD embodiment 10C shown in Figure 2C. In further embodiments believed to be advantageous for surgical use, the additional optical sensor 35 can instead capture light in a different spectrum than first sensor 30, for example, in the wavelength range of 800 nm to 2,500 nm corresponding to near-infrared ("NIR") light, thereby allowing the wearer to observe aspects of the object made fluorescent by an NIR contrast agent.

[0042] All embodiments of HMDs according to the present invention further include data processing capabilities, and optionally data connectivity capabilities. Exemplary hardware architectures of HMDs 10A and 10C will now be described in further detail with specific reference to Figures 3A and 3B, respectively, where like numerals again refer to like features by way of non-limiting example.

[0043] In addition to sensors 30, 32, and optionally 35, each HMD 10A, 10B, 10C includes a data processing unit 301, e.g., a general-purpose microprocessor according to the Cortex™ architecture manufactured by ARM™, that functions as the HMD's main controller. CPU 301 may further include a dedicated image signal processing (“ISP”) unit or module for receiving and preprocessing image data generated by optical sensor 30 before outputting corresponding image data to CPU 301. If present, this ISP unit is integral with or coexists with CPU 301, which is programmed to perform other data processing tasks, as described below. CPU 301 is coupled to memory means 302, which may include volatile random access memory (RAM), nonvolatile random access memory (NVRAM), or a combination thereof, by a data input / output bus 303, through which they communicate and through which other components of HMD 10 are similarly connected to provide headset functionality and receive user commands.

[0044] The data connection between the full resolution imaging sensor(s) 30, 35 and the CPU 301, via bus 303 or otherwise, is a high frequency data communication interface that is sensitive to external electromagnetic interference (EMI) and therefore must be shielded. The data connection between the ranging sensor 32 and the CPU 301, via bus 303 or otherwise, is a high frequency data communication interface that is sensitive to external electromagnetic interference (EMI) and therefore must be shielded. 2 C protocol, and any EMI on the interface is negligible considering the type and amount of data output by the ranging sensor.

[0045] User input data may be received directly from one or more buttons, including at least an on / off switch, and / or a physical input interface 304, which may be part of the HMD casing configured for tactile interaction with the wearer's touch. User input data may also be received indirectly, such as gestures optically captured by optical sensor(s) 30 and / or spoken words captured as analog sound wave data by microphone 305, for which CPU 301 (or a DSP unit or module, not shown) implements analog-to-digital conversion functions, both of which CPU 301 then interprets according to principles already introduced herein, but which are beyond the scope of this disclosure. The processed audio data is output to a speaker unit 306, and all components are powered by an electrical circuit 307 interfaced with an internal battery module 308, which is periodically recharged by an electrical converter 309.

[0046] At any particular point in time during runtime, data circulating within the exemplary architecture includes one or more of target data 422 whenever output by ranging sensor 32 in accordance with the principles described herein, image data 400 whenever generated by imaging sensor(s) 30, display data output by CPU 301 to display units 26A, 26B, and processed audio data output to speaker unit 306. Power is provided to the above components by electrical circuitry 307 interfaced with an internal battery module 308, the battery being periodically recharged by electrical converter 309.

[0047] An embodiment of an HMD according to the present invention may further include networking means 310, shown in dotted lines in the figure as a wireless network interface card or module (WNIC) also connected to data input / output bus 303 and electrical circuitry 307, suitable for interfacing the HMD with a wireless local area network ("WLAN") created by a local wireless router. Alternative or additional wireless data communication functionality may be provided by the same or a different module, for example implementing short-range data communication via Bluetooth™ and / or Near Field Communication (NFC) interoperability and data communication protocols.

[0048] In a computing context, the processing required for conventional target detection, as described with reference to FIG. 1, typically resides at a higher computing layer, e.g., the application level, which favors processing operations involving complex models but requires significant computational resources and power consumption. The present invention improves on this technique by significantly reducing computing and power resources by moving the operational requirement of detecting targets and cropping full-resolution image frame 400 to portion(s) 424 having target(s) of interest to a lower computing layer at the OS kernel level. This is made possible by the low-frequency interface between low-power depth sensor 32 and CPU 301, but it is recalled that even a simple data processing unit such as a microcontroller can process sensed data and output the sensed data to its interface for transmission to the CPU.

[0049] Accordingly, the basic and improved data processing configurations and functionality of HMDs 10A, 10B, 10C of Figures 2A-5 will now be described with reference to Figures 6A-7B, and the data structures stored in memory 302 and processed by CPU 301 are shown in Figure 8, with like numerals referring to like features.

[0050] In a first embodiment of the operating mode of the ranging sensor 32 shown in FIG. 6A , upon power-on of the HMD, the sensor loads a separate set of operating instructions, i.e., sensor firmware, in step 601, thereby initializing the detection matrix 420 in step 602. Depending on the type of ranging sensor used, the sensor can optionally begin emitting a signal, e.g., a light wave, to illuminate a target(s) within its FoV 42 to trigger detection in step 603. Then, in step 604, a query is asked as to whether one or more of its zones 421, representing a detection within the FoV 42, have been triggered. If not, control returns to the optional emission of step 603, or, if there is no emission, the sensor 32 executes a wait command and then resumes polling the query 604. The query of step 604 is finally answered in the affirmative, and in the next step 605 the sensor processes the detection event by determining a property value for each cell 421 involved in the detection event and mapping the determined property to the cell 421 in matrix 420, which in the example of Figure 5 is the distance from the sensor 32 to the HMD wearer's hand 38, or the patient lying on the table, or the operating room lamp (all within FoV 42). The sensor 32 then outputs target data 422 corresponding to matrix 420 to the CPU 301, or to the ISP if present, in step 606, and control returns again to the optional emission of step 603 or the query of step 604.

[0051] In a second embodiment of the operating mode of the ranging sensor 32 shown in FIG. 6B, where like reference numbers refer to like data processing steps, the sensor firmware implements user adjustments to the characteristics of the detectable target, such as a proximity threshold representing a shorter distance to the sensor than the distance interval d, consisting of a minimum or maximum distance value that can be assigned by the sensor to cell 421, or a configurable signal strength value in the case of a ranging sensor 32 with signal emission capability.

[0052] Thus, upon power-up of the HMD, the sensor reloads its firmware in step 601, thereby initializing the detection matrix 420 and, in this embodiment, the target characteristic threshold in step 612, which the user may have entered into a startup configuration file at startup or during a previous runtime instance. Control proceeds to the optional emission in step 603 or to the detection query in step 604. If answered negatively, control returns to the optional emission in step 603, or the sensor 32 waits and then resumes polling query 604. If the query in step 604 is finally answered affirmatively, the sensor processes the detection event in the next step 605, and in this embodiment, the mapping of the determined characteristic to cells 421 in the matrix 420 is filtered in sub-step 615 by comparing the determined characteristic to the target characteristic threshold in step 612 and setting the characteristic value of the cell to the maximum allowed value whenever the detected value exceeds the target characteristic threshold.

[0053] In this example, the sensor 32 filters the detection event accordingly by generating a distance value for the or each cell included in the detection event, comparing each cell's distance value with a proximity threshold, mapping distance values ​​below the proximity threshold to the respective cell, and mapping the maximum distance value to other cells whose respective distance values ​​exceed the proximity threshold. The sensor 32 then outputs the target data 422 to the CPU 301 in step 606, and control again returns to the optional emission in step 603 or the query in step 604.

[0054] The ranging sensor 32 remains operational independent of the activities and tasks of the CPU 301 at all times while the HMD is in use, and its firmware may implement additional functionality, in particular switching between active and idle states according to preset non-detection periods, to enhance power savings regardless of the embodiment of the operating mode. A first embodiment of an operating mode of an HMD, based on the detection and target data output of the ranging sensor 32 regardless of its operating mode, is shown in Figure 7A with reference to the prior art of Figure 1, where like numerals refer to like data processing steps.

[0055] Upon power-on of the HMD 10A, an operating system ("OS") 801 is again first loaded in step 1 to manage the basic data handling, interdependencies, and interoperability of the HMD components 301-309, including the WNIC 310, if present. The HMD OS may be based on Android™, distributed by Google™, Mountain View, California, USA. The OS 801 includes subroutines for reading and processing input and output data, and optionally includes a subroutine 802 for configuring the HMD 10A for bilateral network communication with remote terminals via the WNIC 310, which interfaces with a network router device.

[0056] Further in step 1, an instruction set 803 embodying a target recognition-driven human-machine user interface 803, in this example a gesture user interface application, is loaded either as a subroutine of the OS 801 or as a separate application at a higher computational layer, this distinction being indicated by the dotted line in Figure 8. The application 803 comprises a target recognition engine 804, e.g., a trained model as described previously herein, and is interfaced with the optical sensor(s) 30, 35 and low-resolution ranging sensor 32 via one or more application programmer interfaces (APIs) 805 through the OS 601.

[0057] In addition to initializing the imaging sensor 30 in step 2, the optional loading of the application program 806 in step 3, and the initialization of the OS 801 user interface 807 or its optional application 806 program in step 4, when the ranging sensor is initialized in accordance with steps 601, 602 in parallel with or as part of step 1 and ultimately begins outputting target data 422 in step 606, a question is first asked in step 701 as to whether such target data 422 is being received by the CPU 301, or alternatively the ISP, over the low frequency data connection.

[0058] If so, the CPU 301 or ISP, in step 702, verifies the presence of detected targets 38 of interest in the received target data by filtering out redundant target data based on a comparison of characteristics encoded in the target data 422, such as the respective distance values ​​of each cell 421, with a target detection threshold, e.g., a maximum distance of the HMD wearer's hand from the ranging sensor 32, set at, e.g., 50 centimeters. Cells whose distance values ​​are found to exceed the target detection threshold are excluded from further analysis. For cells remaining after the initial filtering, their identifiers 421 encoded in the target data 422 are also input into a clustering analysis, implemented, for example, using region growing or K-means techniques, which outputs one or more clusters 423 of cells 421, each deemed to contain a detected target 38 of interest.

[0059] Alternatively, filtering may be performed using a point-based interpolation technique as shown in FIG. 9, where like numbers are assigned to opposite corners, e.g., the bottom right 922, of a cluster 923 defined by a set of cell identifiers 421 with matching characteristics. A and top left 922 B, which still defines and bounds the corresponding portion 924 of the image data 400 in the image frame 410, and the or each detected target 38 is or can be imaged at full resolution by the imaging sensor 30. This step advantageously mitigates false positive detections at or near the edges of the distance interval d, for example, in cases where the target data should only encode pairs of adjacent cells 421, to prevent redundant processing of the image data by later steps in the logic.

[0060] Accordingly, in step 703, a question is asked as to whether the comparison calculated in step 702 indicates a detected target, such as the user's hand 38, within distance interval d. If so, the CPU 301 or its ISP maps each cell 421 contained in the or each cluster 423 of known equivalent dimensions in the image frame 410 to the image frame 410, and thus, in step 704, determines the or each corresponding portion 424 of the image data 400 as a respective region of interest in the full resolution image frame 410. The CPU 301 or its IPS then, in step 705, crops the image frame 410 to the calculated region of interest or each calculated region of interest 424.

[0061] The cropped image data 424 generated in step 705 is then input into a conventional target detection algorithm, and the or each portion 424 of the high resolution image frame 410 is scanned by the algorithm to identify the target therein corresponding to the portion, i.e., the hand 38 with the index finger of the HMD wearer 36 in step 5. The conventional recognition data processing continues as above until the HMD updates its user interface in step 9, at which point control proceeds directly to step 9 whenever the question in either step 701 or step 703 is answered negatively, control eventually returning to the target data polling question of step 701, and so on, until the HMD needs to be powered off.

[0062] In the first embodiment described above, the imaging sensor 30 continuously captures the environment around the HMD after initialization in step 3, whereby the data processing steps 701-705 based on target data from the ranging sensor usefully save the computational overhead associated with performing optical target recognition from and within the full resolution image frames 410. A second embodiment of the HMD's operating mode is shown in Figure 7B, where like numerals refer to like data processing steps in Figures 7A and 1, and which is more power efficient.

[0063] This second embodiment implements selective switching between active and idle states of the imaging sensor(s) 30, 35 to enhance power conservation and regardless of the operating mode of the ranging sensor 32, with the imaging sensor(s) being initialized as before in step 2 but remaining in an idle state by default and not generating image frames 400 until and unless instructed to do so by the CPU 301 or the ISP. Thus, in this embodiment, once the query in step 703 is answered in the affirmative, the CPU 301 or the ISP first switches the imaging sensor(s) 30, 35 to an active state in step 711 to generate image frames 400 before proceeding to map the cluster(s) to the first generated image frame in step 704.

[0064] The imaging sensor or sensors 30 remain active and continue capturing image frames 400 as long as the conventional target detection algorithm continues to receive and process each image frame portion 424 generated in each iteration of step 705 to interpret the captured gesture into a command. The query in step 7 is eventually answered either affirmatively when a command is recognized or negatively, for example, when a preset number of interpretation attempts from consecutive frame portions is reached. In this embodiment, if the query in step 7 is answered affirmatively, the CPU 301 executes the interpreted command in step 8 and then returns the imaging sensor or sensors 30, 35 to an idle state in step 721, ceasing image frame generation. If the query in step 7 is answered negatively, the CPU 301 proceeds directly to the switch in step 721. Thus, the image data processing and corresponding power consumption associated with generating image data 400 for conventional target recognition, transmitting it over bus 303, and processing part(s) thereof is conditional on preliminary detection of at least one target 38 by low-resolution ranging sensor 32, and is therefore further reduced compared to the first embodiment of FIG. 7A.

[0065] 10, an alternative embodiment is proposed that leverages the target pre-detection capabilities of the ranging sensor 32 in the specific context of a gesture user interface by identifying application commands from directional movements determined using the target data 422. That is, in a gesture user interface where one or more directional movements of a user's hand(s), for example, are known to be associated with respective specific data processing tasks or commands, the filtered target data is analyzed to determine whether targets 38 therein exhibit relevant directional movements before mapping the clusters 423, 923 to corresponding portions 424, 924 of the image data, thus avoiding the redundant computational cost of the mapping and cropping steps 704, 705 and processing of the cropped high-resolution data in steps 5-7.

[0066] In such an embodiment, target data 422 is generated by ranging sensor 32 per steps 601-606 as described above with reference to FIG. 6A or 6B, and output to CPU 301 per steps 701-703 as described above with reference to FIG. 7A or 7B, where it is filtered by CPU 301. However, in addition to an affirmative answer to the query in step 703, in step 1001, the current filtered target data for the data processing cycle, or the initial filtered target data, is processed into first motion data, i.e., a motion start data point, in accordance with the motion detection algorithm implemented for the embodiment, and temporarily stored, e.g., buffered, until filtered target data for the next data processing cycle, or second filtered target data, is received in a subsequent iteration of query 703 in which the query is answered affirmatively. The second filtered target data is similarly processed into second motion data, i.e., a motion end data point, in step 1002, where such motion start and motion end data points may be, for example, the centroids of the respective clusters 423 in successive captures by ranging sensor 32.

[0067] In step 1003, a difference between the first and second motion data is calculated, representing the directional movement of the or each target 38. For example, a motion vector is calculated using the initial motion start and motion end data points, and its direction is determined by referencing the Cartesian plane of matrix 420. In step 1004, the directional vector is compared to a library of application tasks and commands defined by directional user input, such as "left to view the next record" or "right to view the previous record," similar to turning the pages of a book. In response, in step 1005, a query is asked as to whether the comparison identified a matching application task or command. If the query is answered in the affirmative, CPU 301 proceeds directly to executing the task or command according to step 8, advantageously without performing mapping or cropping and recognition processing of the high-resolution image data, as described above. Alternatively, the logic loops back to step 1001 to wait for and process the next instance of filtered target data.

[0068] Depending on the operational requirements of the application(s) to be controlled, one skilled in the art can devise an inexpensive wearable device implementing the embodiment described with reference to Figure 10 with a simplified architecture relative to that described with reference to Figures 3A and 3B, for example, comprising a CPU 301, memory 302, ranging sensor 32, associated data (303) and power (307, 308, 309) circuitry, and a wired or preferably wireless data interface (e.g., 310) for relaying determined directional movement data to a remote computer or the like that performs corresponding data processing tasks or commands.

[0069] While exemplary embodiments have been described using the HMD wearer's hand(s) as target 38, the principles disclosed herein are readily adaptable to and usable with alternative targets, such as medical markers or tags increasingly used as visual references for determining and tracking the position and / or orientation of a patient, their limb, or treatment site relative to HMDs 10A, 10B, 10C in medical procedures, particularly surgery. Because several markers may be placed at different respective locations on the patient, such that the markers fall within or outside of FoV 42 and distance interval d depending on how the HMD wearer positions and / or orients relative to the patient, the HMD wearer is expected to benefit from isolating the marker(s) of interest according to their distance relative to the HMD in accordance with the principles disclosed herein.

[0070] 2A-9, the configuration and logic of each of the ranging sensor 32 and HMDs 10A, 10B, 10C described herein are substantially identical. Optional application 806 processes the cropped image data portion output in step 705 to identify medical markers by targets 38 as before, and performs further data processing tasks based on this recognition to at least determine and maintain alignment of the HMD coordinate system of the patient, limb, or surgical site, but also to calculate and adjust, for example, the position and / or orientation of a computer-generated image composited onto an image of the patient (or, in the case of HMD 10A, the patient that can be directly observed) within the HMD wearer's field of view 40 within user interface 807.

[0071] Thus, the combination of a ranging sensor and the methods described herein optimizes user-machine interactions that involve machine commands based on the detection of a nearby target, typically a user's hand, but alternatively, markers, tags, and other visual cues as simple as a particular color.

[0072] In this specification, the terms "comprise, comprises, comprised, and comprising" or any variation thereof and the terms "include, includes, included, and including" or any variation thereof are considered to be fully interchangeable and should all be given the broadest possible interpretation, and vice versa. The present invention is not limited to the embodiments described above, which may be modified both in arrangement and detail.

Claims

1. A portable data processing device, the portable data processing device comprising: data processing means; at least one high resolution imaging sensor operable to capture an environment around the device within a field of view as image data; a low-resolution ranging sensor configured to detect one or more targets in the field of view within a range interval of the device and to output target data encoding detected target characteristics; a portable data processing device comprising: power means connected to supply power to the imaging sensor, the ranging sensor, and the data processing means, the data processing means being configured to receive at least the target data from the low-resolution ranging sensor, filter the received target data by comparing the detected target characteristics with a target detection threshold, and map the filtered target data to a corresponding portion of the image data when the or each high-resolution imaging sensor is used to capture the environment.

2. The handheld device of claim 1 , wherein the detected target characteristics include a distance between the detected target and the low-resolution ranging sensor, and the target detection threshold includes a proximity threshold.

3. 3. The portable device of claim 2, wherein the proximity threshold is configurable to be less than the distance interval, and the data processing means filters received target data only if at least a first target reaches the proximity threshold within the distance interval.

4. 4. The portable device of claim 1, wherein the data processing means is further configured to receive the image data from the high-resolution imaging sensor and to crop the corresponding portion from the received image data.

5. the or each high resolution imaging sensor is switchable; the ranging sensor is further configured to switch the or each imaging sensor for acquisition upon outputting the target data; or 5. A handheld device according to any preceding claim, wherein the data processing means is further configured to switch the or each imaging sensor for acquisition upon receiving the target data.

6. 6. The handheld device of claim 1, wherein the ranging sensor is configured to detect a plurality of distinct zones within the field of view, and the target data further includes a respective identifier for the or each zone.

7. 7. The portable device of claim 6, wherein the ranging sensor is a low power time-of-flight (ToF) sensor operatively connected to the data processing means via a low bandwidth data connection.

8. 8. A portable device as described in any preceding claim, wherein the or each target is a human hand, and wherein the portion of the image data includes image data representing the or each human hand, and wherein the data processing means is further configured to process either the portion of image data or the target data into user commands.

9. A portable device according to any preceding claim, wherein the or each target is a medical marker or tag, and wherein the portion of the image data comprises image data representative of the or each medical marker or tag.

10. 1. A method for detecting a target with a portable data processing device, comprising: the handheld device comprises data processing means, at least one high resolution imaging sensor operable to capture an environment around the device within a field of view as image data, a low resolution ranging sensor configured to detect one or more targets within the field of view within a range interval of the device, and power means operably connected to the imaging sensor, the ranging sensor, and the data processing means; The method comprises: upon detecting the or each target, outputting target data using the ranging sensor encoding detected target characteristics; receiving at least the target data in the data processing means; filtering the received target data by comparing the detected target characteristics to a target detection threshold; and mapping the filtered target data to a corresponding portion of the image data when the or each high resolution imaging sensor is used to capture the environment.

11. The method of claim 10 , wherein the detected target characteristics include a distance between the detected target and the low-resolution ranging sensor, and the target detection threshold includes a proximity threshold.

12. 12. The method of claim 11, including the further step of configuring the proximity threshold to be less than the distance interval, and wherein the filtering step further includes comparing the distance between the detected target and the low-resolution ranging sensor to the proximity threshold to filter out target data distal to the proximity threshold.

13. capturing the environment with the high-resolution imaging sensor; outputting the captured image data to the data processing means; A method according to any of claims 10 to 12, comprising the further step of using said data processing means to crop said image data to said mapped portion.

14. A method according to any one of claims 10 to 13, wherein the high resolution imaging sensor is switchable, and the method includes the further step of switching the high resolution imaging sensor to capture image data either when target data is output by the low resolution ranging sensor or when target data is received by the data processing means.

15. 15. The method of claim 10, wherein the step of outputting target data using the ranging sensor further comprises dividing the field of view at the ranging sensor into separate zones, and the target data further comprises a respective identifier for the or each zone.

16. 16. A method according to any of claims 11 to 15, wherein the or each target is a human hand and the corresponding portion comprises image data representing the or each human hand, the method comprising the further step of processing the corresponding portion of image data, or the filtered target data, into user commands using the data processing means.

17. A method according to any of claims 11 to 16, wherein the or each target is a medical marker or tag, and wherein the portion of the image data comprises image data representative of the or each medical marker or tag.

18. A wearable user interface device, comprising: a data processing means having at least one output; a low resolution ranging sensor operable to poll an environment around the device within a field of view using waveform signals, detect one or more targets within the field of view within a range interval of the device in accordance with said polling, and output target data encoding detected target characteristics; and power storage means connected to supply power to the distance measurement sensor and the data processing means, wherein the data processing means receiving at least the target data from the low resolution ranging sensor; filtering the received target data by comparing the detected target characteristics to a target detection threshold; processing the first filtered target data into first motion data and processing the second filtered target data into second motion data; calculating a difference between the first movement data and the second movement data representing directional movement of the or each target; A wearable user interface device configured to compare the calculated difference with a library of data processing commands associated with each directional movement to identify a matching data processing command.

19. 20. The wearable user interface device of claim 18, wherein the separation distance is in the range of 30 to 70 centimeters and the or each target is a human hand.

Images