Supervisory setup of control devices with imagers

Abstract

The present invention discloses a system and method for determining a target at which a pointing device is pointed. One embodiment of the present invention is a method performed using a control device having a pointing direction and an onboard imager. The onboard imager has a field of view that includes the pointing direction. The method includes the following steps: capturing a reference image, determining a feature value based on the reference image, generating a feedback message based on the feature value, and predicting the termination of a setup phase of the control device based on the feature value.

Description

[0001] Cross-references to Related Patent Applications

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 884,278, filed on August 8, 2019. Background Art

[0003] Consumers intuitively align their remotes with the devices they want to interact with by pointing them at them. This behavior stems from the fact that traditional remote controls send radio signals directly in the direction they're pointed. Therefore, aligning the remote with the device provides the best chance for the radio signal to reach its intended target.

[0004] More recently, remote controls using the technology described in U.S. Patent No. 10,068,463 identify the device to which commands are sent through one system, while the commands themselves are sent through other means. For example, a set of beacons can determine the direction the remote is pointing, while a separate radio system indirectly transmits commands from the remote to the controllable device. These approaches preserve the traditional user experience of pointing at the controllable device while expanding the range of devices that can receive commands from the remote. However, these devices now require the system to identify where the pointing device is pointing so that the commands can be sent to the appropriate device.

[0005] There are various methods for identifying an object's position and orientation. For example, geolocation, such as GPS, can be used to determine an object's position and orientation. However, GPS is not precise enough to accurately determine pointing direction, especially in indoor environments, and cannot distinguish between pointing directions corresponding to one of two adjacent devices. Indoor positioning systems, such as the beacons mentioned above, are an alternative to GPS. These beacons use locally generated wireless signals to estimate an object's position. However, the placement restrictions of these beacons can make installation expensive and complex. Summary of the Invention

[0006] The present invention discloses a system and method for determining the direction a pointing device is pointing at a target using an onboard imager located on an object. The shape of the pointing device can define the natural pointing direction of the pointing device. For example, the device can be a rectangle with a distinct short side, where the distinguishing features of the long and short sides of the rectangle naturally indicate the direction in which the object is pointing. Generally speaking, a pointing device can have a pointing direction. For example, the device can be a disk whose pointing direction is indicated by an arrow icon on the surface of the device. In other examples, any indicator that allows a person to determine the direction in which the device is pointing can be considered to provide a pointing direction to the device (as used in this invention). The onboard imager can be aligned with the pointing direction of the object. The onboard imager can be connected to the exterior of the pointing device or be an integral component of the pointing device. The onboard imager can have a field of view. The onboard imager can be oriented so that the field of view includes the pointing direction.

[0007] The system described herein can be the pointing device itself, but may also include a support device (e.g., a dock or charger for the pointing device) and a remote control device, such as a server or cloud architecture in operational communication with the support device or the pointing device. Reference will be made to non-transitory computer-readable media storing instructions for performing certain operations of the disclosed system. In these embodiments, the computer-readable media may be located entirely within the pointing device, disposed on the support device, the remote control device, and the pointing device, or located entirely on the support device and / or the remote control device.

[0008] In a specific embodiment of the present invention, the pointing device can be a control device. The pointing device can be a remote control that selects a pointing target in the form of a controllable object or, typically, a communication object. By pointing the pointing device at a specific pointing target, an association can be formed between the controllable object or communication object and the sending system. This association can then be used to send commands to the currently associated controllable object or to send communication information from the currently associated communication object. For example, if the object is a controllable object such as a television, commands obtained from the user on the pointing device can be sent to the controllable object while maintaining the association. As another example, if the object is a communication object such as a weather service on a remote server, communication information obtained from the remote server can be sent to the pointing device while maintaining the association. In this way, the user can receive communication information from and send commands to various objects based on where the pointing device is pointing at any given time.

[0009] Object associations formed by pointing a pointing device at a given target can also modify the user interface by displaying controls for the currently associated object on the user interface. The user interface can be provided on the pointing device. For example, the pointing device can include a touch display, and when an association is formed, controls for the currently associated controllable object can be displayed on the touch display. When a user points the pointing device at a television, the touch display can display the television's channel and volume control interface. When a user points the device at a light fixture, the touch display can display the light fixture's brightness control interface.

[0010] The above associations can be predefined by a setup program. The system can be described as operating in a setup phase during the execution of the setup program. The setup program may involve associating a target area with a pointing target and defining a signature for the target area. The physical area can be a specific spatial volume within a physical location, such as a portion of a room near a television, or a specific surface such as a wall, ceiling, floor, or their intersections. The pointing target can be the center of the target area. For example, the pointing target can be a small device (such as a DVR or a compact streaming box), and the target area can be a cabinet and surrounding area where the small device is placed. The setup program can associate the target area with the pointing target and further associate the pointing target with a controllable object or communication object. The pointing device is then used to associate the signature involved in identifying the target area, calling the pointing target associated with the target area, calling the controllable object or communication object associated with the pointing target, and forming an association with the object of the sending system. This operating phase of the system can be referred to as the operational phase. The various nodes of the system can be said to be in a deployed state during the operational phase.

[0011] In certain embodiments of the present invention, the system can use images captured by an imager to determine a target for a pointing device. The images can be applied to a system that uses these images to identify a signature of a target area. The pointing target can be the center of the target area. Figure 1 An example of a user 100 pointing a pointing device 101 at an audio / video (A / V) device 102 is shown as an example of a user attempting to form an association between a controllable object and the pointing device. In this example, an imager is aligned with the pointing device 101 to capture an image of the A / V device 102. Figure 1An example of image 103 is shown, where receiver 105 is located within cabinet 104, serving as A / V equipment 102. The system can then operate on image 103, identifying the signature of target area 106. In this case, the target area could include the entire corner of the room occupied by cabinet 104. Assuming that target area 106 has been previously associated with cabinet 104 (the pointing target), and cabinet 104 has been previously associated with receiver 105 (the controllable object), the entire control system can be associated with receiver 105 and, based on this determination, send commands to receiver 105. As mentioned above, the pointing target and the controllable device need not be the same. Instead, a user can associate specific areas within any given area with specific controllable devices (e.g., a window in a room could be associated with a weather service on a remote server). In the illustrated example, the imager can obtain image 107, in which only cabinet 104 is visible, but the system still understands that when pointing at area 106, a command should be sent to receiver 105.

[0012] Detecting a pointed target at any given time for association with a pointing device can be difficult for a number of reasons. For example, small pointed targets, such as a compact streaming media box, can be difficult to detect in images captured of a general area. Furthermore, a venue may include many identical copies of the same controllable device, such as multiple versions of a home automation assistant or a home stereo system. Distinguishing between the two devices based solely on the device being identified by an image recognition system would be nearly impossible, as the two devices would be identical copies of the same design. Furthermore, venues such as homes often include multiple pieces of matching furniture and other repetitive design elements on the walls, floors, and ceilings that may appear identical to an image recognition system. Furthermore, areas of a given venue may appear very different at different times due to changes in lighting, changes in the viewing angle of a pointing device, changes in the spatial configuration of objects within the area, and the introduction or removal of objects within the area.

[0013] In certain embodiments of the present invention, focusing on the target area rather than the pointing target helps to reduce many of the problems pointed out in the previous paragraph. The target area will include additional information and signals from which the signature is derived. For example, this additional stronger signal can prevent the system from confusing cabinet 108 in image 109 with the identical cabinet 104. This is because the target area may include additional information (in the form of which corner of the room the cabinet is located) that would be hidden if only the characteristics of the cabinet were used to generate the signature. Certain aspects of these embodiments are described in detail below that enhance the system's ability to collect information about the target area rather than the specific pointing target.

[0014] In certain embodiments of the present invention, the problems identified in the previous paragraph are mitigated by constructing a robust signature creation and detection system for identifying the target of a pointing device. In these embodiments, the signature creation process involves a collection of different images of the same target area. The different images captured during the setup phase are referred to herein as reference images because they provide a reference for later identifying the signature of the target area. The signature creation process can be performed during the system's setup phase and is designed to ensure that the different images are highly distinct, thereby increasing the strength of the signature. The images can be selected to vary based on the viewpoint from which the image is captured, the lighting conditions under which the image is captured, the configuration of the target area itself during image capture, and other variations. The variability in the images used to create the signature during the setup phase results in a recognizable signature despite the aforementioned variability in the images when the system is deployed. For example, a system with a robust signature, when provided with image 110, can still identify target area 106 even though cabinet 104 is partially covered by items 111, 112, and 113 that modify the area's appearance. The robust signature has a sufficiently high signal-to-noise ratio that the noise represented by these items does not cause the signature detection system to fail.

[0015] In certain embodiments of the present invention, a user may be relied upon to capture reference images that are used to generate signatures for target areas identified during operation of the pointing device. However, in practice, there is no guarantee that the reference images captured by the user will be sufficient for the system to work properly when deployed. For example, the user may not capture enough images, may capture multiple identical images, thereby not adding any useful information, may capture disturbed images (e.g., occluded images where an object briefly appears in the field of view), may capture blurred images (e.g., the imager moves too quickly when capturing the reference images), or may capture images under extreme lighting conditions that are not suitable for the sensor characteristics (e.g., at night). Therefore, in certain embodiments of the present invention, supervision is provided to ensure that the set of reference images acquired during setup is sufficient to generate a strong, robust signature for the target area. Supervision can take the form of feedback to the user on the adequacy of the reference images that have been acquired so far, as well as potential encouragement for additional images to be acquired.

[0016] In certain embodiments of the present invention, the system may calculate a feature quantity for a set of reference images and predict the termination of the setup phase based on the feature quantity exceeding a sufficiency threshold. The feature quantity may be a quantity representing the amount of signal contained in the set of reference images, which can be manipulated to generate a target region signature. The feature quantity may be a quantity representing the signal-to-noise ratio contained in the set of reference images, where the signal represents the uniqueness of the target region signature generated from the set of reference images. For example, the feature quantity may be a number calculated based on the statistical degree of variation between images in the set of reference images. Many other examples are described in detail below. Alternatively, or in combination, feedback may be provided to the user during the setup phase to ensure the sufficiency of the set of reference images. For example, the feedback may provide encouragement for taking additional steps to complete the setup procedure, such as capturing additional images. In certain embodiments of the present invention, the feedback may be generated based on the feature quantity. For example, instructions may be provided on a display of a pointing device indicating where the user should move the imager to capture additional images with a higher degree of variation, thereby further improving the feature quantity. Many other examples of such feedback are described in detail below.

[0017] In a specific embodiment of the present invention, a system is provided. The system includes a control device. The shape of the control device defines a pointing direction of the control device. The system also includes an onboard imager located on the control device. The field of view of the onboard imager includes the pointing direction. The system also includes one or more computer-readable media storing instructions for performing the following operations: receiving a reference image; determining a feature value based on the reference image; generating a feedback message based on the feature value; and predicting the termination of a setup phase of the control device based on the feature value.

[0018] In a specific embodiment of the present invention, a computer-implemented method is provided. The method is performed by a control device having the following characteristics: (i) a shape defining a pointing direction of the control device; and (ii) a field of view of an onboard imager including the pointing direction. The method includes capturing a reference image, determining a feature quantity based on the reference image, generating a feedback message based on the feature quantity, and predicting the termination of a setup phase of the control device based on the feature quantity.

[0019] In a specific embodiment of the present invention, a system is provided. The system includes a control device. The system has a pointing direction. The system also includes an onboard imager located on the control device. The field of view of the onboard imager includes the pointing direction. The system also includes one or more computer-readable media storing instructions for performing the following operations: receiving a set of reference images; determining a feature quantity based on the set of reference images; and based on the feature quantity: (i) generating a feedback message; or (ii) terminating a setup phase of the control device. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A set of images captured by a user using a pointing device and an imager onboard the pointing device according to certain embodiments of the present disclosure is shown.

[0021] Figure 2 A system block diagram and accompanying flow chart are shown illustrating a computer-implemented method for performing this system setup phase according to certain embodiments of the present disclosure.

[0022] Figure 3 The present invention discloses a specific embodiment of the use of reference Figure 2 The process is a method for performing the operational phase of system setup.

[0023] Figure 4 A user interface displayed on a pointing device according to a specific embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0024] The present invention discloses a system and method for determining a target at which a pointing device is pointed using an imager on a pointing device. In a specific embodiment of the present invention, the imager is aligned with the pointing direction of the pointing device. The image of the imager is then used to identify the target area at which the pointing device is currently pointed. Images, referred to as sample images, can be acquired during the operation phase of the system. Subsequently, the system can determine the pointed target within the target area and then create an association with the pointed target according to the methods in the above invention. The following disclosure includes specific methods in the setup phase of the system for identifying the target area according to the above invention. A set of reference images acquired during the system setup phase can facilitate the identification of the target area. The examples provided in this section are non-limiting embodiments of the present invention and should not be construed as limiting the scope of the invention.

[0025] The present invention discloses a pointing device in the form of a remote control that integrates a touch display, a radio frequency transmitter, and a visible light airborne imager. However, the pointing device may take various alternative forms. The pointing device may include various components, including various user interface elements, additional sensors, and specialized hardware for processing reference images and identifying target areas. As described in the Summary of the Invention, the pointing device may also be used in conjunction with a support device and a remote control device to perform the method described herein.

[0026] In a specific embodiment of the present invention, the system that is responsible for the setup phase of the operation and determines that the pointing device is pointing to the target during the operation phase may be an independently operated pointing device. However, the system responsible for these operations may also include a support device (such as a charger for the pointing device) and may also include a remote control device (such as a server or cloud architecture that communicates with the pointing device for operation). In a specific embodiment of the present invention, the system responsible for receiving user commands, presenting a user interface to the user and / or providing information to the user from the pointing target may be an independent pointing device. However, these tasks may also be performed by independent support devices and remote control devices. For example, the support device may directly receive user commands (for example, through a built-in microphone) or relay commands input through the pointing device to the currently associated controllable object. These detachable components (pointing device, support device and remote control device) may be referred to as nodes of the system.

[0027] Depending on the hardware and other design constraints of the pointing device, the operation of the system can be divided among the nodes in a variety of ways. For example, if the pointing device has limited battery, the memory and logic used to perform resource-intensive operations (such as storing images or identifying signatures of target areas) may be located on a support device (such as a charging base for the pointing device). As another example, more resource-intensive operations, such as initially generating a signature for the target area, calculating feature quantities of a set of reference images used to generate the signature, and generating feedback for the user based on the feature quantities, may be performed on a server or cloud architecture compared to the pointing device or any computing device in the same physical environment as the pointing device. In a specific embodiment of the present invention, the system can be a multi-processor system in which a main processor collaborates with an AI accelerator. Both the main processor and the AI ​​accelerator can be located on the pointing device. However, the multi-processor system can also be a distributed system in which a host processor collaborates with a remote coprocessor (e.g., an AI accelerator). The remote coprocessor can be located on a remote server.

[0028] In certain embodiments of the present invention, one or both of determining the signature or determining whether the image is a target area based on the signature may be performed on the support device. The support device may use its internal processor and memory to perform these operations. In certain embodiments of the present invention, the support device will have a power connection to a wall outlet and will not be as sensitive to power as the pointing device when performing heavier load calculations. In certain embodiments of the present invention, the support device will include an AI accelerator that communicates with the support device or a main processor on the pointing device. Alternatively, the support device may receive a reference image and store it for later transmission to a cloud architecture to analyze the reference image and / or generate a signature for the target area. The support device may also send commands to the appropriate device when the pointing device is set up and deployed.

[0029] In certain embodiments of the present invention, a cloud architecture in communication with a pointing device can perform various operations. For example, the cloud architecture can train an initial state for a trainable directed graph, which is deployed with the device. The cloud architecture can provide the initial state to the pointing device or other supporting device as a download when the device is first initialized. The cloud architecture can also provide instructions for capturing reference images necessary for accurate setup. The cloud architecture can also review the image and the progress of determining reference image features, and generate feedback to the user as needed. Images are captured by an imager on the pointing device or an imager on an alternative device.

[0030] The pointing device and any associated support devices can be enhanced by dedicated hardware associated with identifying the target area signature. The dedicated hardware can execute image processing and recognition algorithms more efficiently than a general-purpose processor. The dedicated hardware can be an AI accelerator as described in the present invention. The dedicated hardware can more efficiently train and utilize a trainable machine intelligence system. The trainable machine intelligence system can be trained using a set of reference images acquired during the system setup phase. The signature of the specific target area is then reflected in the weights of the trained system. During deployment, the input image from the imager on the pointing device is fed to the machine intelligence system to determine that the pointing device is pointing at the target area.

[0031] The dedicated hardware can take a variety of forms. For example, a pointing device or associated support device may include a dedicated digital signal processor for instantiating, training, or drawing inferences from a trainable image recognition system. For example, the digital signal processor may be embodied as a GPU, FPGA, chipset, or ASIC specifically designed for such calculations. The ASIC may be designed to generate inferences about signatures associated with a sample image and consume power on the order of microjoules for each inference. The GPU or ASIC may be mounted on the pointing device. The digital signal processor may be optimized to accelerate calculations, such as linear matrix operations performed by an artificial neural network (ANN) performing image classification inferences. The digital signal processor may perform multiple computational operations in parallel. As another example, the pointing device or support device, such as a charger, may be enhanced with a dedicated digital signal processor to perform training or inference operations for a trainable machine intelligence system, which operations can be easily performed on the dedicated digital signal processor without excessive battery consumption or excessive processing time.

[0032] The pointing device and any associated support device may include one or more interfaces for receiving user commands. The pointing device may include a display for displaying information to the user. The pointing device may also receive user commands through an interface (e.g., a keyboard, touch display, or microphone, or any other known user interface technology). Alternatively, the pointing device may not have any interface for receiving commands, while the support device includes a touch display, microphone, or gesture recognition interface for receiving user commands. For example, a smartphone, tablet, or hub device may receive user commands, and the pointing device may be used to identify the controllable device. The pointing device may also include a speaker or tactile feedback system that provides information to the user. When the system selects or locks onto a pointing target, the display, speaker, or tactile feedback system may provide a prompt to the user, thereby alerting the user that the system has formed an association based on this pointing target.

[0033] In addition to the imager, the pointing device can include additional sensors, such as a motion tracker. The motion tracker can be an inertial motion unit (IMU). The pointing device can include an integrated motion tracker (e.g., a magnetometer, accelerometer, gyroscope, or any 9-axis sensor). The motion tracker can be used to activate the imager. For example, the motion tracker can determine when the pointing device moves and then remains stationary, indicating that the user is pointing at a pointing target. Upon detecting this motion pattern, the imager can be triggered to acquire a reference image or a sample image. Additional sensors, such as motion trackers, can also be used in data fusion applications, as described below.

[0034] As described above, a reference image of the target area can be acquired during the setup phase to generate a signature for the target area, and this signature can be used to analyze sample images later when the device is deployed. In certain embodiments of the present invention, both steps can be performed by an onboard imager. The reference image can be captured by the same onboard imager used to identify the target area and determine the pointing direction of the pointing device when it is deployed. However, a separate imager can also be used to capture the reference image, while the onboard imager captures the image analyzed during device deployment. The separate imager can be mounted on a companion device. The companion device can be one of the support devices described above. The companion device can be a smartphone or a dedicated device for the setup phase. The companion device can include features not available on the pointing device that facilitate the setup phase. For example, the companion device can include a higher resolution display than the pointing device, making it easier for the user to check the quality of the reference image or receive feedback from the system regarding a set of reference images. The companion device can also include a device for providing feedback to the user regarding the reference image. This feedback can be provided via the companion device's display, the companion device's speakers, or other user interface of the companion device. In a specific embodiment of the present invention, the accessory device will be paired with the pointing device via a wireless connection (such as a Bluetooth connection), and the reference image will be captured by the onboard imager of the pointing device. At the same time, the reference image for inspection and any additional feedback for the user, such as instructions for additional images, will be displayed on the display of the accessory device.

[0035] The onboard imager or the imager on the accompanying device can take various forms and can be enhanced to capture additional information from the physical environment in which it operates. The imager can capture one-, two-, or three-dimensional images. Images can be generated using any form of electromagnetic energy (e.g., visible light, infrared, or ultraviolet) or any combination of multiple frequency bands. To assist the imager, the imager can operate in conjunction with a projector that generates and projects visible electromagnetic energy to the imager. The projector can be used to illuminate the environment or generate a structured light pattern within the environment, allowing the imager to capture more information from the environment. For example, the imager can include a night vision infrared camera with infrared LEDs or an ultraviolet camera with an ultraviolet structured light projector. In certain embodiments, the infrared LEDs or other projectors can be mounted on a support device around the physical environment so that they do not drain excessively from the pointing device's battery. The support device can also be a charger for the pointing device. The imager can include multiple sub-imagers, such as a visible light camera for detecting visible light in the environment and an ultraviolet camera for detecting the pattern projected by the ultraviolet projector.

[0036] In certain embodiments of the present invention, the imager has a large capture area to maximize the information available to the system, enabling robust signature generation during setup and signature recognition during operation. For example, the imager can be a wide-angle visible light camera, capturing a larger area per image to obtain more information about the target area. Additional information can include the relative position of adjacent corners of the room and / or the distance from the pointed target to the ceiling and floor. The imager can be a fisheye imager. In other embodiments, the imager can capture panoramic images, such as full spherical panoramas.

[0037] In certain embodiments of the present invention, the imager's field of view will include the pointing direction but will not be centered on it. For example, the imager may be tilted relative to the pointing direction. The degree of tilt may be selected for a particular application. For example, for a pointing device with a user interface, which a user typically reads while operating the device, the imager may be tilted perpendicularly relative to the pointing direction to offset common biases associated with this usage pattern. As another example, the corners of a room are often more visible from the ceiling-wall junction than the ceiling-floor junction due to furniture and other items on the floor. Therefore, the imager may be tilted slightly perpendicularly toward the ceiling to capture information about the room's layout.

[0038] The imager may include a combination of sensors that project a pattern in the direction the device is pointing, such as a visible light camera and a UV LED or laser-enhanced UV camera. In addition to the visible light emitted by the camera, the combined image may include depth information captured by a depth sensor, such as a UV or infrared camera that detects the projected pattern. Any form of projected structured light may be projected to capture depth information that includes visible, UV, or infrared light. The depth sensor may be any form of depth sensor, including LIDAR, stereo imaging devices, or any other form of sensor that can capture depth information or derive depth information from it. In these embodiments, the image may be an RGB-D matrix or a depth cloud. In other approaches, the imager and image may be based solely on depth information and completely ignore texture data. For example, the image may capture the locations of infrared dots in a two-dimensional image illuminated by the projection of structured light. The specific sensor used to acquire the image may vary depending on the environmental conditions in which the imager operates. For example, a pointing device or a supporting device (e.g., a charging station) may include an ambient light sensor (ALS) and may disable the visible light sensor if it is determined that there is insufficient ambient light in the environment for the visible light sensor to provide actionable information.

[0039] Figure 2A system block diagram 200 and accompanying flowchart 210 are shown for a computer-implemented method performed using the system, as described in certain embodiments of the present disclosure. Block diagram 200 includes a control device 201. The shape of the control device 201 defines a pointing direction 202 for the control device. Block diagram 200 also includes an onboard imager 203 located on the control device 201. As shown, a field of view 204 of the onboard imager 203 includes the pointing direction 202. Block diagram 200 also includes a computer-readable medium 205 storing instructions for performing each step of flowchart 210. As described in the Summary of the Invention, computer-readable medium 205 may be located entirely within the control device 201, for example, in a memory internal to the control device 201, or may be distributed throughout a system including the control device 201.

[0040] Flowchart 210 begins at step 211 (Capture Reference Image). The reference image can be captured by the onboard imager 203 or an imager on a companion device. Flowchart 210 continues with step 212 (Receive Reference Image). This step involves receiving an image from the onboard imager of a processing device and storing it in the memory of the pointing device, or receiving an image from another node in the system that captures the image (e.g., capturing the image on a companion device in the form of a smartphone and then transmitting it to a pointing device charger or remote server that receives the image).

[0041] Flowchart 210 continues with step 213 (determining a feature quantity based on the reference images). Step 213 may also be performed based on a set of reference images if the system has previously captured additional reference images at this point. The feature quantity may be a value calculated for the most recent reference image or for the entire set of reference images that has been acquired. The feature quantity may be a quality measure of the image quality of the most recent reference image, the number of reference images in a set of reference images, or a more complex value that reflects a set of reference images calculated by graphical and geometric analysis (evaluating image variations in terms of image viewpoint, image brightness, etc.). The feature quantity may be a measure of the completeness of a 3D reconstruction of at least a portion of a target region or pointing to a target in a reference image (e.g., the ratio of discovered surfaces to the total surface to be discovered). The feature may be a diversity measure calculated based on the spectral content of different reference images. The feature quantity may be a diversity measure of the estimated viewpoints of the reference images.

[0042] Flowchart 210 continues with step 214 (Predicting the Termination of the Setup Phase of the Pointing Device Based on a Feature Quantity). In this step, the feature quantity generated in step 213 can be evaluated and used to determine whether the setup phase is complete. Thus, the termination of the setup phase can be predicted based on the feature quantity. Evaluation of the feature quantity may involve comparing the feature quantity to a threshold (e.g., whether the feature quantity is greater than, equal to, or less than a given threshold). This comparison will depend on the type of feature quantity calculated in step 214. In a particular embodiment, the feature quantity is the number of reference images in a set of reference images. The threshold is a predetermined required number of reference images that the user needs to capture. In another particular embodiment, the feature quantity is a measure of disparity or correlation between different images in the set of reference images. The disparity or correlation can be measured based on the overall texture map, imager pose, brightness, or other aspects of the image. However, regardless of the specific quantity calculated, step 214 is based on a feature quantity that indicates that the reference image or set of reference images evaluated in step 213 contains sufficient information to generate a robust, unique signature for the target region of the reference image. For example, the feature quantity can be the degree of correlation between the reference images in the set of reference images. In this case, the evaluation would require that the feature quantity be less than a threshold, indicating that the image is sufficiently sharp. As another example, the feature quantity could be the degree of difference between reference images in a set of reference images. In this case, the evaluation would require that the feature quantity be greater than a threshold, indicating that the image is sufficiently sharp.

[0043] Flowchart 210 continues with step 215 (Generating Feedback Message Based on Feature Quantities). In this branch of the flowchart, the setup phase is not terminated because the set of reference images was found to be incomplete or insufficient in step 214. The feedback message can provide guidance or encouragement for the user to continue the setup phase and improve the set of reference images. The feedback message can be a textual message or a symbolic message. For example, the feedback message can be a textual instruction displayed on a pointing device display to capture additional images (e.g., move left, move right, move back, move forward, capture again while holding the imager steady, turn on more lights, turn off lights). The textual message can also be provided audibly using a speaker. As another example, the feedback message can be a score, progress bar, or other symbolic representation indicating the ratio of the number of reference images captured to the required number of reference images. The feedback message can be provided visually via the display or audibly via the speaker. The feedback message can be generated based on the feature quantities or directly based on the reference images. The feedback message can include the reference images themselves so that the user can examine the reference images and diagnose why the system found the reference image or set of reference images to be insufficient. Various combinations of the types of feedback messages described herein may also be provided to the user. For example, the feedback message may include a reference image requesting the user to confirm the quality of the reference image, and the feedback message may also include a progress indicator showing the number of reference images received by the user compared to a target threshold.

[0044] In certain embodiments of the present invention, the feedback message is displayed on a companion device (e.g., a smartphone). These embodiments offer certain advantages when the companion device includes a display that is superior to the pointing device, and when visual inspection of the feedback message is important (e.g., the feedback message includes the reference image itself). In these embodiments, the pointing device can capture the image and wirelessly transmit it to the companion device for display. Alternatively, the companion device can be used to capture and display the reference image.

[0045] In certain embodiments of the present invention, the feedback message may include instructions on how to capture at least one additional reference image. The provided instructions may ensure that the next reference image obtained has the greatest value to the system in terms of its impact on the feature quantity. The instructions may specify at least one position that the pointing device or the companion device (whichever device is acquiring the reference image) should be in when capturing the at least one additional reference image. For example, the user may see an arrow on the display indicating the direction they need to move to capture the next reference image. As another example, the user may receive text instructions via a visual display or audible instructions such as, "Take an additional reference image for this area from the left of the image just acquired." Feedback may be provided in real time to guide the user in maintaining the correct imager posture. For example, the feedback may be an augmented reality (AR) signal. For example, the feedback may highlight which parts of the area have not been fully captured (e.g., a visible highlight is displayed in real time on the image of the area). As another example, the feedback may display an arrow that continuously updates how the device should move until it is in the optimal position to create the maximum value for the set of reference images.

[0046] In certain embodiments of the present invention, the feedback message may be directed to a recently captured reference image (e.g., determining the characteristic quantity may include determining that the recently acquired reference image is of insufficient quality due to poor lighting, blur, or insufficient variation from a previous reference image). The feedback message may simply include instructions to recapture the same reference image, or instructions to modify the reference image before capturing it from the same vantage point (e.g., "steady the imager" or "turn on more lights").

[0047] In certain embodiments of the present invention, the feedback message may be considered an improvement to the entire set of reference images (e.g., determining the feature quantity may determine that the set of reference images lacks a certain form of variance). The feedback message indicates that at least one additional reference image needs to be captured to improve the feature quantity of the set of reference images. For example, the evaluation of the feature quantity may indicate that the user needs to capture additional images from different angles, at different brightness levels, remove objects from the target area, etc. The feedback message may then provide this information to the user (e.g., instructions to move to a specific angle to complete the desired set of unique viewpoints, or instructions to turn on additional lights).

[0048] In certain embodiments, step 211 is performed while simultaneously changing the environment to increase the diversity of the reference images. The differentiation of the environment can be performed according to a set program or a program that varies according to a characteristic quantity during the acquisition of a set of reference images. The differentiation of the environment can be created by an environment differentiation system. Because the pointing device can be a control device that acts as a controller for a large number of devices, and the environment differentiation system can be a component of an entire system including the pointing device, the environment differentiation system can access a large number of devices that can be used to modify the environment. In fact, in certain embodiments, the pointing device can be part of the environment differentiation system and serve as the initiator of commands for modifying the environment.

[0049] Depending on the device controlled by the environmental differentiation system, various environmental differences can be introduced using various methods. For example, the brightness of a room can be changed by turning a light on or off, changing the color of any color-changing lamp, opening or closing curtains, or turning a television or other display device on or off. As another example, the visual appearance of a room can be altered by displaying different color patterns or images on a television or other display device. These environmental changes can be created based on characteristic quantities as part of the aforementioned process. In an embodiment, where the characteristic quantity is a measure of the difference or correlation between the brightness or overall texture of different images, a feedback message can be generated using computer-readable instructions for a desired difference from the next reference image. The environmental differentiation system can create the desired difference and set the environment to the conditions required to respond to the feedback message. For example, the characteristic quantity may indicate that the reference image lacks sufficient brightness difference. The feedback message may be an instruction to increase the brightness. The environmental differentiation system can then select a potential method for increasing the brightness in response. If the environmental differentiation system only has access to a single light, the light's brightness can be reduced based on the feedback message. If the ambient difference system has access to a lamp and a curtain, it can choose different approaches and study their effects on the diversity of the reference image (e.g., try the curtain first, and if subsequent feedback still indicates that a greater brightness difference is needed, then dim the lamp).

[0050] As shown, if the setup phase has not terminated, the flowchart returns to steps 212 and 213, where additional reference images are captured, received, and added to the set of reference images; and the feature quantities are re-determined using the additional images. This process includes receiving at least one additional reference image and re-determining the feature quantities based on the reference image captured in the first iteration and the at least one additional reference image. After re-determining the feature quantities, the flowchart returns to a point where the system can trigger the end of the setup phase based on the feature quantities. This step may involve further iterations of step 214, which will not loop through another iteration to trigger the end of the setup phase. Thus far, the steps described in flowchart 200 can be performed by a companion device (such as a smartphone) rather than the control device 201. However, in certain specific embodiments, the control device, and particularly an imager on the control device, can be used to capture the reference image and additional reference images in step 211, while displaying the feedback in step 215 on a display of the companion device (such as a smartphone).

[0051] As shown, the flowchart 210 may continue from step 214 to step 216, which triggers the termination of the setup phase of the pointing device. Step 216 may be performed at any time during the setup phase when the system determines that a set of reference images is sufficient. As shown, the termination of the setup phase is triggered based on the behavior of predicting the termination of the setup phase based on the feature quantity. Figure 2 The setup phase is shown. However, the setup phase may include acquiring reference images for multiple target regions at once. Therefore, the steps of flowchart 200 discussed so far may be repeated multiple times (including multiple iterations of step 215) to acquire reference images for these additional target regions. The setup phase may also be revisited when a user wishes to add an additional target region to an existing system. A user command to add an additional region may trigger re-entry into the setup phase.

[0052] The termination of the setup phase may trigger additional steps required to place the system in an operational phase condition. These steps include generating a signature for the target area and associating a pointing target or object with the target area. Flowchart 210 includes step 217 (generating a signature for the target area using the reference image captured in step 212). The signature may be generated by a set of reference images captured over multiple iterations in step 212. Flowchart 210 also includes step 218 of associating an object with the target area. The object may be a communication object or a control object. In the operational phase, the association between the object and the target area may be used to create an appropriate association so that communications or commands are sent through the system when the pointing device is pointed at the target area, and the target area is identified using the generated signature.

[0053] The signature generated in step 217 may represent the target region and may be derived from two-dimensional or three-dimensional data of the target region. The signature may be embodied as weights used by a trained ANN to process an image to determine whether it is an image of the target region. The signature may be a trained directed graph, a series of points / coordinates, or other forms of compressed information that can be used to identify a region. The signature may be a feature vector that is used to match the output of a classification system (provided with an input image of the target region). In certain embodiments of the present invention, a region may have multiple signatures (e.g., multiple two-dimensional or three-dimensional features captured from different perspectives) or a common signature (e.g., a three-dimensional model of the region created by reconstruction based on multiple two-dimensional reference images or features of these reference images). Because lighting conditions affect reference images, a signature may be a combination of daytime and nighttime signatures. In certain embodiments of the present invention, different signature libraries are accessed based on the features of sample images acquired while the pointing device is in an operational phase. For example, the system may maintain a library of low-light signatures for target regions and a library of high-light signatures for the same target regions. Using an ALS on the pointing device, determining a target region during the operational phase may include accessing only the low-light signature if the ALS detects low-light conditions, and vice versa if it detects high-light conditions.

[0054] In certain embodiments of the present invention, the feature quantity is a measure of the diversity of the estimated viewpoints of the reference images (e.g., measuring or calculating an estimate of the viewpoint of each reference image, which is then analyzed to determine whether the variation between the viewpoints is sufficient). The system can determine and analyze the viewpoint differences between the reference images. This step can be performed by calculating the points of interest (e.g., ORB descriptors) in the two reference images, correlating these points across the entire image, and calculating an essential matrix based on the correlation, which then derives the direction of the change in orientation and the change in position between the viewpoints. In these embodiments, the threshold value of the feature quantity used to evaluate the termination of the predicted setup phase can be the absolute angular distance between the orientations of the two images to determine whether they are sufficiently different. The number of sufficiently different orientations using this measure can also be calculated between the reference images and displayed to the user as a feedback message. The threshold used to predict the termination of the setup phase can also be a large number of reference images with sufficiently different orientations measured using the above measure.

[0055] In certain embodiments of the present invention, the feature quantity can be a complete metric of a three-dimensional reconstruction of at least a portion of the target region or the target being pointed at. The reference image can be used to reconstruct a three-dimensional representation of the target region, and if the three-dimensional reconstruction is sufficiently complete (i.e., there are no dead zones in the reconstruction), the setup phase is terminated (or the user is advised to terminate it). This three-dimensional reconstruction can employ a procedure similar to that considered in the previous paragraph, wherein the depth of the point of interest is retrieved from the two images in which the point of interest appears, and the relative orientation and translation direction of the two images are calculated based on the intrinsic matrix.

[0056] In certain embodiments of the present invention, the process of acquiring images from the user during the setup phase is performed using a real-time image stream captured by an onboard imager or an imager of an associated device. The pointing device may also analyze reference images captured in this real-time stream, so that the feature quantities of a set of reference images are continuously updated as the imager captures images. Similarly, feedback may be calculated and provided to the user in real-time. The feedback in these embodiments may be the AR signals described above. For example, an arrow may be placed on the display of the real-time image stream to indicate the direction the imager should move, or different surfaces in the real-time image stream may be highlighted to indicate that the surface was fully captured and added to the 3D reconstruction of the physical space.

[0057] In certain embodiments of the invention, data fusion is employed to improve the setup phase and / or the operation phase. Data fusion may be used to provide more accurate feedback or better estimates of feature quantities. Data fusion may involve adding additional data to the imager data, such as motion tracking and / or position data. For example, an estimate of the pose of the imager is added to an image taken from that pose. In certain embodiments of the invention, the system will include a motion tracker and a computer readable medium located on a control device, the computer readable medium storing instructions for determining feature quantities based on a reference image and data from the motion tracker, generating feedback messages based on the feature quantities and data from the motion tracker, or identifying a signature of a target area using the motion tracker data during the operation phase.

[0058] Using a visible light camera as the only sensing element makes the system susceptible to blur, visual occlusion, and visual blur between multiple places (such as two blank walls). To address these issues, one possibility is to supplement the visible light imager with another information source (such as an IMU) to perceive the pose of the imager (i.e., its joint position and orientation). The pose estimate can then serve as a backup when the imager's sensor is obstructed or the ambient brightness is too low for the sensor to operate properly. The pose estimate can then be used as additional information to distinguish between two areas with the same visual aspects during the operation phase. In addition, the pose information can be used to assist in the calculation of feature quantities for a set of reference images, especially those based on capturing the desired degree of difference between the viewpoints of each reference image.

[0059] In certain embodiments of the present invention, the pose estimation of the imager can be aided by having prior knowledge of the positions of the various target regions. This can be accomplished by: (1) capturing the pose of the pointing device each time a reference image is taken; and (2) triangulating the position of the reference region from at least two pose estimates captured in (1). A similar approach can be used to correct for drift in the sensor used to determine the position of the pointing device. The pose estimate of the IMU is subject to drift, meaning that the estimate slowly deviates from the true pose. To address this, the output of the imager can be used to correct for drift each time a known target region is identified.

[0060] Figure 3 A flow chart illustrating the operational stages of a system according to a particular embodiment of the invention is shown. Flowchart 300 begins at step 301 (receive a sample image). The sample image may be an image of a pointing target from an onboard imager on a pointing device. As shown, sample image 310 includes a visible light encoding or TV 311. The TV represents a target area. Flowchart 300 continues with step 302 (identify a signature using the sample image). The signature will be a signature of the target area represented in image 310. This step involves providing information from image 310 to a classifier, a trained machine learning system, a trained support vector machine, or any other system capable of identifying a target area signature from an image.

[0061] In certain embodiments of the present invention, once a signature is recognized, the system may associate it with a controllable device or other target with which the system is communicating. Therefore, flowchart 300 includes step 304 (Associating with an object by the system). In certain embodiments of the present invention, the system may also provide a user interface to the user upon signature recognition. As shown, flowchart 300 continues with step 303 (Displaying a controllable object interface on a display for the recognized signature). Figure 3 The remote control is shown in a first state 312 with a display 313. Therefore, step 303 is executed to cause the same remote control to transition to state 314, where a controllable object interface 315 of a television 311 is displayed on the display 313. This transition can be performed after the target area signature is detected if the target area was previously associated with a controllable object, for example, according to the setup phase procedure of the present invention.

[0062] Once a pointing device's pointing target is selected, this information can be used in various ways. The pointing target can remain selected until cleared by the user, or it can change in sync with the pointing device's instantaneous pointing direction. While a specific pointing target remains selected, the pointing device itself or the system to which it belongs can interact with the specific device or system associated with that specific pointing target. In certain embodiments of the present invention, the pointing device can be a remote control, and the pointing target can be selected to identify the controllable device that the remote control should interact with at any given time. In certain embodiments of the present invention, the pointing device can have a display, and the pointing target can be selected to identify the control interface that the display should display at any given time. The display can be a touchscreen display, or some other combination of display and input interface, for displaying information to the user and receiving control input from the user. As an example combining the two aforementioned sets of embodiments, when a user points a pointing device at a television, the display may display a channel selector and a volume selector, as well as the word "TV" identifying the currently selected device. Subsequently, when the user points the pointing device at a light bulb, the display may display a switch, dimmer, or color selector, depending on the characteristics associated with the light bulb.

[0063] When a specific pointing target is selected, the relationship between that pointing target and the association generated by the system can take various forms. For example, the pointing target itself may be a controllable object, which should be the subject of the control system association. However, the pointing target may be a physical area defined by certain features detected by the imager but not physically associated with the controllable object. Conversely, the system may have previously associated the pointing target with a specific controllable object. Thus, the controllable object can be real or virtual. For example, a controllable object can be a physical automated building component or smart home device that receives commands or provides information, such as a lighting fixture, television, electronic window shades, thermostat, actuators for commercial HVAC equipment, smoke alarms, chemical sensors, security devices, etc. However, a controllable object can also be a virtual object that receives commands or provides information, such as a web-accessible API or other virtual object. Furthermore, the flexible association between pointing targets and controllable objects ensures that the user does not need to physically see the location of the controllable object in order to select it.

[0064] The chain of operations from acquiring an image, selecting a pointing target, associating the pointing target or controllable object, and utilizing this association according to the examples above can be implemented in various ways. In particular, commands or information can be sent by various nodes in the system based on the currently selected pointing target. For example, commands can be received through an interface within the pointing device itself (e.g., a touchscreen display or microphone) or through a completely separate system (e.g., a microphone located in the same room as the pointing device). Furthermore, the pointing device can simply transmit images from the imager to a separate node in the system, or it can perform all necessary steps to generate a command specifically addressed to a given controllable device. The pointing device itself can capture the images captured on it and store and process them locally, or it can transfer them to an alternative device for storage and processing. For example, the pointing device can send images to a charger, which can then determine the object to which the control system should be associated and send commands to that object. Similarly, the pointing device can be associated with the controllable device alone through a command, or only with the pointing target through a command, and the support system proceeds to the next step, that is, to associate the command with the controlled device through the stored association between the pointing target and the controlled device stored and maintained by this support system.

[0065] In certain embodiments of the present invention, the manner in which commands are actually sent through the system may vary depending on the specific implementation. Figure 3 317 is an example of a pointing device in the form of a remote control 316 that has been associated with a control device in the form of a television 317, such that a transmission system 318 can transmit commands from the remote control 316 to the television 317. Transmission system 318 can be fully integrated into the remote control 316 and include an IR transmitter that can be tuned to transmit signals to various devices, including the IR receiver of the television 317. However, transmission system 318 can also include additional devices, such as a charging station for the remote control 316. Transmission system 318 can also include any number of local or wide area networks, the internet, and remote devices, such as servers and supporting cloud infrastructure. The charging station can include an IR transmitter that communicates with the television 317 and a different wireless communication system that communicates with the remote control 316. Furthermore, as described elsewhere herein, a remote control or other pointing device can be used solely to form association 302, while commands transmitted by transmission system 318 originate from a separate device. For example, these commands can be received via a microphone on the charger for the remote control 316 or any other microphone that has a communication interface with the transmission system 318.

[0066] Figure 4A potential user interface 400, mounted on a display of a remote control 410, is shown for guiding a user in acquiring a set of reference images, according to a particular embodiment of the present invention. The user interface is displayed on the touchscreen display of the remote control 410. At this point in the setup process, the system has identified a multi-colored light on a potential pointing target. The pointing target is referred to herein as a potential pointing target because it has not yet been added to the set of pointing targets known to the system. An area 401 of the display identifies the pointing device for the current portion of the setup process. The user interface 400 also displays a counter 402 indicating the number of pointing target images captured by the user. Additional instructions can be provided to help guide the user in acquiring appropriate images. A control 404 is also provided for the user to instruct the imager to acquire images. The counter and area for additional instructions can be used to provide the user with other types of information regarding perspective changes, lighting changes, and other system feedback to guide the user in successfully executing the image acquisition process. Figure 4 The user interface can provide a similar user experience flow for acquiring images of a target area. In fact, in certain embodiments, the user may not know that the system is capturing information of the target area, but rather provides instructions for pointing to reference images related to the target in this area.

[0067] Before the image received in step 212 is applied to determine the feature quantity or identification signature, it may undergo various forms of pre-processing. For example, if the image is a visible light image, the image may be processed through automatic white balance, autofocus and image stabilization, as well as any other image normalization procedures, so that the image can be used as a reference for how the pointing target is later displayed. The image can be corrected using a rolling shutter camera model. In a related method, the image can be acquired using a global shutter camera to help standardize the image when the image is acquired. The pre-processing may further include image processing to compensate for brightness changes during the setup phase. This type of processing may also be applied to images collected when the pointing device is operating in a deployed state. In embodiments where the pointing device acquires various data, the pre-processing may also include fusing data from various sensors together, for example, by combining visible light and depth data to form an RGB-D point cloud.

[0068] The same type of pre-processing described above can be applied to images acquired when the pointing device is operating in a deployed state, such as the images received in step 301. In fact, various pre-processing methods for normalizing images help the system match images of the same pointing type because spurious differences are eliminated.

[0069] In certain embodiments of the present invention, methods (e.g., the method exemplified by flowchart 300) can adapt to system-detected fault conditions or significant changes in data availability. For example, in the event of a fault condition or missing data, the pointing device can default to manual mode, in which the user identifies the device to be controlled through other means, such as voice commands or selecting the device from a menu on the pointing device's touchscreen display. Missing data or fault conditions can be detected in a variety of ways. For example, an ambient light sensor, or a control system that identifies the lighting conditions in a room and the time of day, can control which type of imager is used to capture images and what modifications may be required to the inference-generating process. Following this example, a pointing device with both a visible light sensor and an IR sensor can disable the visible light sensor if the device is operated in darkness. As another example, the system can determine whether a significant modification has been made to the physical location or any given area, rendering the system no longer able to reliably identify that area. More specifically, the system can determine whether the user rearranged the physical space during the setup process, thereby returning the process to the setup phase to recalibrate and refine the signature associated with that area.

[0070] In any of the above examples, rather than automatically exiting the imager mode or automatically changing the performance of the imager mode, the user can be provided with the option to switch to the default mode. The option to switch to the default mode can also be continuously presented to the user via a configurable setting, regardless of whether a fault condition or missing data condition exists.

[0071] In certain embodiments of the present invention, the components of the system can be connected to a mobile robot so that the physical position of the components can be automatically adjusted during the use setup or operation phase. For example, the pointing device, support device, control device and supporting device described above can be mounted on a mobile robot in whole or in isolation. The mobile robot can be a pedal-type, wheeled or legged robot. The mobile robot can be an aerial robot, for example, a micro quadcopter. The mobile robot can be a fixed tripod or other support connected to a rotating imager base, where the robot can change and set the position of the connected imager. Any of the above-mentioned mobile robots can also be enhanced by such a rotating imager support base. The mobile robot can provide the device with six degrees of freedom (DOF) mobility, allowing the device to have different x, y and z positions in the environment, as well as different pitch, yaw and roll imager positions. The robot can also automatically change the area of ​​the imager.

[0072] In certain embodiments of the present invention, the mobile robot can change the position of the device during the setup phase to increase the diversity of the reference images. The position change can be based on a set program or a program that changes based on a characteristic quantity during the acquisition of a set of reference images. The position change is created based on the characteristic quantity as part of the program. In an embodiment, the characteristic quantity is a measure of the difference or correlation between the camera poses or physical appearance of the environment in different images, and a feedback message is generated by computer-readable instructions regarding the required difference for the next reference image. The mobile robot can create the required difference and set the environment to the conditions required to respond to the feedback message. For example, the characteristic quantity may indicate that the reference images lack sufficient difference in camera pose, and the mobile robot may move the imager to a different position to obtain another reference image. As another example, the characteristic quantity may indicate that an area lacks sufficient physical difference, and the mobile robot may move itself or other objects within the area to increase such diversity.

[0073] While this specification has been described in detail with respect to specific embodiments of the present invention, it will be appreciated that those skilled in the art, upon understanding the foregoing disclosure, will readily envision variations, modifications, and equivalents of these embodiments. For example, the present invention uses a remote control as an example of a pointing device, where the pointing direction is determined to transmit commands to a controllable device. However, this method is more broadly applicable to any pointing device, such as a presentation pointer, an inventory management device, or a wirelessly tagged toy. Furthermore, the same method can be used to determine the heading of any device in a specific physical environment, such as a robot or drone used for autonomous navigation within a given space. Furthermore, many of the methods described herein can be applied to devices with built-in imagers, even if these devices were not originally intended for use as pointing devices. For example, a camera mounted on the back of a smartphone camera can be used as an imager aligned with the smartphone's pointing direction, while the display can serve as a control interface. These and other modifications and variations may be made by those skilled in the art without departing from the scope of the present invention, which is more fully described in the appended claims.

Claims

1. An identification system comprising: a control device, wherein a shape of the control device defines a pointing direction of the control device; an onboard imager located on the control device, wherein the field of view of the onboard imager includes the pointing direction; and One or more computer-readable media storing instructions to: receiving a reference image indicating a target area in a physical location; determining a feature quantity based on a reference image, wherein the feature quantity represents the uniqueness of the signature of the target area; generating a feedback message according to the feature amount; and The setting phase of the control device is terminated according to the prediction of the characteristic quantity; The system further comprises one of the following features (i) to (iii): (i) the feedback message indicates that it is necessary to capture at least one additional reference image in order to improve the feature quantity, and the feedback message includes instructions on how to capture the at least one additional reference image; (ii) determining the feature amount includes determining that a reference image has insufficient quality, and the feedback message includes an instruction to recapture the reference image; or (iii) The one or more computer-readable media further store instructions to disable a capture interface on the control device based on the characteristic quantity, wherein determining the characteristic quantity based on the reference image is performed in real time.

2. The system according to claim 1, wherein: The one or more computer-readable media further store instructions to: receiving at least one additional reference image; re-determining the feature amount based on the reference image and the at least one additional reference image; and After the characteristic quantity has been re-determined, termination of the system setting phase is triggered according to the characteristic quantity.

3. The system according to claim 1, wherein: The one or more computer-readable media further store instructions to: changing the environment of the control device according to the feedback message; Wherein, changing the environment includes changing one of: the brightness of the environment, the position of the mobile robot in the environment, and the image displayed on the screen in the environment.

4. The system of claim 1 , wherein the one or more computer-readable media further store instructions to: generating a signature for a target region using the reference image; Associating a communication object with the target area; receiving a pointing target image from the airborne imager; recognizing the signature using the pointing target image; and A communication interface of the communication partner is activated in response to identifying the signature.

5. The system according to claim 1, further comprising: a display located on the control device; as well as The one or more computer-readable media further store instructions to: generating a signature for a target region using the reference image; associating a controllable object with the target area; receiving a pointing target image from the airborne imager; recognizing the signature using the pointing target image; and A controllable object user interface is displayed on the display in response to recognizing the signature.

6. The system according to claim 1, further comprising: a motion tracker located on the control device; as well as The one or more computer-readable media further store instructions to: Determining the feature quantity based on the reference image and data from a motion tracker; or The feedback message is generated based on the feature quantity and data from a motion tracker.

7. A recognition method, said method being performed using a control device having the following features: (i) a shape defining a pointing direction of the control device; and (ii) an onboard imager having a field of view including the pointing direction, the method comprising the steps of: capturing a reference image indicating a target area in a physical location; determining a feature quantity based on the reference image, wherein the feature quantity represents the uniqueness of the target area signature; generating a feedback message according to the feature amount; and The setting phase of the control device is terminated according to the prediction of the characteristic quantity; The method further comprises one of the following features (i) or (ii): (i) the step of determining the feature quantity includes determining that a reference image is of insufficient quality, and the feedback message includes an instruction to recapture the reference image; or (ii) The method further includes disabling a capture interface on the control device according to the feature quantity, wherein the step of determining the feature quantity according to the reference image is performed in real time.

8. The method according to claim 7, further comprising the steps of: receiving at least one additional reference image; re-determining the feature quantity based on the reference image and the at least one additional reference image; as well as After the characteristic quantity has been re-determined, termination of the setup phase is triggered according to the characteristic quantity.

9. The method according to claim 7, further comprising the steps of: generating a signature for a target region using the reference image; Associating the communication object with the target area; receiving a pointing target image from the airborne imager; recognizing the signature using the pointing target image; and A communication interface of the communication partner is activated in response to identifying the signature.

10. The method according to claim 7, further comprising the steps of: generating a signature for a target region using the reference image; associating a controllable object with the target area; receiving a pointing target image from the airborne imager; recognizing the signature using the pointing target image; as well as A controllable object user interface is displayed on a display on the control device in response to identifying the signature.

11. The method according to claim 7, further comprising the steps of: determining the feature quantity based on data from a motion tracker on the control device and the reference image; or The feedback message is generated based on the feature quantity and data from a motion tracker.

12. An identification system comprising: a control device, wherein the control device has a pointing direction; an onboard imager located on the control device, wherein the field of view of the onboard imager includes the pointing direction; and One or more computer-readable media storing instructions to: receiving a set of reference images indicating a target area in a physical location; determining a feature quantity based on a set of reference images, wherein the feature quantity represents the uniqueness of the target region signature; generating a feedback message according to the feature amount, and continuously capturing a real-time image stream using the onboard imager, wherein the feedback message includes an augmented reality image stream generated using the real-time image stream; and Based on the characteristic quantity, a setting phase of the control device is terminated.

13. The system of claim 12, wherein the one or more computer-readable media further store instructions to: generating a signature for the target region using the set of reference images; Associating a communication object with the target area; receiving a pointing target image from the airborne imager; recognizing the signature using the pointing target image; and A communication interface of the communication partner is activated in response to identifying the signature.

14. The system of claim 12, further comprising: a motion tracker located on the control device; as well as The one or more computer-readable media further store instructions to: determining the feature quantity based on the set of reference images and data from a motion tracker; or A feedback message is generated based on the feature quantity and data from the motion tracker.

Images