User interface for selecting field of view of camera in smart glasses

Abstract

A wearable device for immersive reality applications is provided. The wearable device includes an eyepiece to provide a forward image to a user, a first forward-looking camera mounted on a frame and having a field of view, a processor configured to identify a region of interest within the forward image, and an interface device to indicate to the user that the field of view of the first forward-looking camera is misaligned with the region of interest. A method of using the device, a memory storing instructions, and a processor executing the instructions to cause the device to perform the method of using are also provided.

Description

Technical Field

[0001] This disclosure relates to a user interface in a smart glasses device, the smart glasses device including one or more cameras for recording images and videos. More specifically, this disclosure relates to a method for using an eye-tracking tool to identify a region of interest in a user's field of vision and provide non-invasive feedback to the user so that the user's region of interest aligns with the field of view of one or more cameras in the smart glasses. Background Technology

[0002] Many modern electronic devices include one or more cameras embedded within a wearable frame, which users can activate to capture photos or videos. However, in many cases, the captured images and videos do not match the region of interest (ROI) in the user's field of view at the time of capture. This occurs because the angle and field of view of the one or more cameras often differ from the user's angle and field of view. In some types of electronic devices, this inconsistency can be addressed by providing the user with direct feedback on the ROI of one or more camera attachments on a display. The user can then manually adjust the camera position and optical configuration (magnification and aperture, etc.) to align the ROI of the one or more camera attachments with the ROI. However, this method requires a display, which may not be available in some devices, and requires the user's focused attention to correct the inconsistency. In some devices with displays, it may still be desirable to use the immovable elements within the device for maximizing user enjoyment rather than for hardware adjustments. Summary of the Invention

[0003] In a first embodiment, an apparatus includes: a frame including an eyepiece for providing a forward-facing image to a user; a first forward-facing camera mounted on the frame; a processor configured to identify a region of interest within the user's field of view; and an interface device for instructing the user that the field of view of the first forward-facing camera is misaligned with the region of interest.

[0004] In another embodiment, an apparatus is provided comprising: a frame including an eyepiece for providing a forward-facing image to a user; a first forward-facing camera mounted on the frame; a processor configured to identify a region of interest within the forward-facing image; and an interface device for instructing the user that the field of view of the first forward-facing camera is misaligned with the region of interest.

[0005] In some embodiments, the device may further include a sensing device facing the user for tracking the user's pupil position, wherein the processor identifies a region of interest based on the user's pupil position.

[0006] In some embodiments, the processor may identify the region of interest based on the user's gaze direction within the field of vision.

[0007] In some embodiments, the device may further include a memory that stores instructions that, when executed by a processor, cause the device to locate a gaze vector indicating the center of a region of interest within the user's field of vision.

[0008] In some embodiments, the interface device may include an audio device for providing signals to the user to adjust head posture and align the field of view of the first forward-facing camera with the region of interest.

[0009] In some embodiments, the interface device may include a vibration component configured to instruct a user to move their head position in a desired direction so that the field of view of the first forward-facing camera is aligned with the region of interest.

[0010] In some embodiments, the eyepiece may include a liquid crystal configured to block a vertical or horizontal portion of the eyepiece when activated, to guide the user to move their head horizontally or vertically, respectively, so that the center of the field of view of the first forward-facing camera is aligned with the region of interest.

[0011] In some embodiments, the device may further include an actuator configured to modify the color of a portion of the field of view of the eyepiece corresponding to the region of interest.

[0012] In some embodiments, the device may further include a microphone for receiving voice commands from a user to identify objects of interest within the field of view.

[0013] In some embodiments, the device may further include a second forward-viewing camera, wherein the interface device is configured to provide a user with an option to activate the second forward-viewing camera when the field of view of the first forward-viewing camera is not aligned with the region of interest.

[0014] In another embodiment, a computer-implemented method includes: receiving an indication from a user of smart glasses of an object of interest within a forward-facing image viewed by the user; comparing a region of interest centered on the object of interest with the field of view of a first forward-facing camera mounted on the frame of the smart glasses; and providing feedback to the user via an interface actuator mounted on the smart glasses based on the relative position between the region of interest and the field of view of the first forward-facing camera, optionally wherein the feedback is used to enable the user to adjust their head position relative to the object of interest.

[0015] In some embodiments, receiving an indication of an object of interest within the user's field of vision may include: tracking the position of the user's pupils and identifying the object of interest based on the position of the user's pupils.

[0016] In some embodiments, receiving an indication of an object of interest within a user's field of vision may include the eye position of the user receiving the smart glasses, which indicates the location of the object of interest within the user's field of vision.

[0017] In some embodiments, receiving a designation for an object of interest within the user's field of vision may include determining the user's gaze direction based on the positions of the user's two pupils.

[0018] In some embodiments, receiving an indication of an object of interest within the user's field of vision may include receiving the positions of the user's two pupils, and determining the gaze direction may include determining the convergence or divergence of the positions of the user's two pupils.

[0019] In some embodiments, the computer-implemented method may further include determining a region of interest, which has a field of view of a first forward-facing camera and is centered on the object of interest.

[0020] In some embodiments, the smart glasses may include a second front-view camera, and may also include receiving a command from a user to turn on the second front-view camera when the region of interest is not aligned with the field of view of the first front-view camera.

[0021] In some embodiments, the smart glasses may include a second forward-facing camera, and wherein providing feedback to the user includes determining the relative position of the region of interest with respect to the field of view of the second forward-facing camera.

[0022] In some embodiments, the smart glasses may include a speaker, and providing feedback to the user may include giving the user instructions to move the smart glasses in a given direction.

[0023] In some embodiments, smart glasses may include a light emitter, and providing feedback to the user may include using the light emitter to indicate the direction in which the user's gaze moves.

[0024] In another embodiment, a computer-implemented method for a camera interface in smart glasses includes identifying a mismatch between a region of interest centered on a user's object of interest and the field of view of a first camera mounted on the smart glasses. The computer-implemented method further includes automatically updating the position, orientation, or other settings of the first camera without notifying the user, or selecting a second camera on the smart glasses that is better positioned relative to the region of interest.

[0025] In another embodiment, a system includes one or more processors and a memory storing instructions that, when executed by the one or more processors, cause the system to perform a method. The method includes: receiving an indication from a user of smart glasses of an object of interest within the user's field of vision; comparing a region of interest centered on the object of interest with the field of view of a first forward-facing camera mounted on the frame of the smart glasses; and providing feedback to the user via an interface actuator mounted on the smart glasses, based on the relative position between the region of interest and the field of view of the first forward-facing camera, the feedback being used to enable the user to adjust their head position relative to the object of interest.

[0026] In another embodiment, a system includes: a first means for storing instructions, and a second means for executing the instructions and causing the system to perform a method. The method includes: receiving an indication from a user of smart glasses of an object of interest within the user's field of vision; comparing a region of interest centered on the object of interest with the field of view of a first forward-facing camera mounted on the frame of the smart glasses; and providing feedback to the user via an interface actuator mounted on the smart glasses based on the relative position between the region of interest and the field of view of the first forward-facing camera.

[0027] These and other embodiments will be clear from the following disclosure. It will be understood that any feature described herein that is suitable for incorporation into one or more aspects or embodiments of this disclosure is intended to be generalizable in any and all aspects and embodiments of this disclosure. Other aspects of this disclosure will be understood by those skilled in the art based on the specification, claims, and drawings of this disclosure. The foregoing general description and the following detailed description are exemplary and illustrative only and do not limit the scope of the claims. Attached Figure Description

[0028] Figure 1 An architecture comprising one or more wearable devices according to some embodiments is shown, the one or more wearable devices being coupled to each other, coupled to a mobile device, a remote server, and coupled to a database.

[0029] Figures 2A to 2C The illustration shows a scene viewed by a user wearing smart glasses, according to some embodiments.

[0030] Figure 3 A visual indicator in the eyepiece of smart glasses according to some embodiments is shown, which is used to help the user adjust the camera field of view based on the region of interest for image capture.

[0031] Figure 4Multiple front-view cameras in smart glasses according to some embodiments are shown, as well as interference with the field of view of one of the multiple front-view cameras.

[0032] Figure 5 Smart glasses with glass-like light-blocking capabilities are shown according to some embodiments.

[0033] Figures 6A to 6B An example of spatial modulation of the transparent and shading states of a cross-eyepiece is shown according to some embodiments.

[0034] Figures 7A to 7B A smart glasses with two eyepieces in a single transmission state, oriented towards a scene, is shown according to some embodiments.

[0035] Figure 8 This is a flowchart illustrating steps in a method for matching a region of interest with the field of view of a camera in smart glasses, according to some embodiments.

[0036] Figure 9 This is a block diagram illustrating an exemplary computer system according to some embodiments, which can be used to implement a headset and its usage.

[0037] In the accompanying drawings, unless otherwise expressly stated, elements with the same or similar numerals share the same or similar features. Detailed Implementation

[0038] In the following detailed description, numerous specific details are set forth to provide a full understanding of this disclosure. However, it will be apparent to those skilled in the art that embodiments of this disclosure can be practiced without some of these specific details. In other instances, well-known structures and techniques have not been shown in detail so as not to obscure this disclosure. The embodiments disclosed herein should be considered within the scope of the features shown in the accompanying drawings and other embodiments, as follows.

[0039] Traditional cameras or imaging systems allow users to view the object or scene being captured from the camera's perspective. Some methods include viewfinders, eyepieces, displays with previews, and live streaming. However, systems where the camera lacks a viewfinder, display, or other such means of viewing the captured object or scene from the camera's perspective present the user with unwanted uncertainty about the camera's field of view. While capturing an image through the camera can be triggered, without feedback to correct the camera's field of view before triggering capture, the user may not know the exact scene or perspective being captured at the moment of capture.

[0040] To address the aforementioned problems in the field of wearable devices with camera accessories, the embodiments disclosed herein allow users to understand the field of view to be covered by the camera. In some embodiments, the system includes smart glasses with a built-in or attached camera. The system may or may not have a display for the user. If a display is present, it may not be available for the camera's field of view, or it may preferably be dedicated to other utilities.

[0041] Furthermore, to address the aforementioned problems in the field of wearable devices with video camera accessories (e.g., "smart glasses"), the embodiments disclosed herein allow users to understand the field of view to be covered by the camera. In some embodiments, the system includes smart glasses having a built-in or attached camera.

[0042] The smart glasses device disclosed herein includes: a frame including eyepieces for providing a field of view to a user; a forward-facing camera mounted on the frame; a sensing device facing the user for tracking the user's pupil position; processing circuitry configured to identify a region of interest (ROI) within the user's field of view; and an interface device for instructing the user that the field of view of the forward-facing camera is misaligned with the ROI. In some embodiments, a memory may store instructions that, when executed by a processor, cause the smart glasses to perform one or more steps of a method consistent with this disclosure. The memory may also store data, such as calibration data of the position and orientation of the forward-facing camera relative to the user's field of view.

[0043] The embodiments disclosed herein use eye-tracking tools to identify regions of interest (ROIs) in a scene viewed through smart glasses as described above. Furthermore, the embodiments disclosed herein read user gestures to identify ROIs in a scene viewed through smart glasses as described above. In some embodiments, the system is designed such that a camera captures a scene, including several example gestures from the user. A gesture recognition system is used to process frames including gestures to identify moments when the user performs appropriate gestures, and ROIs are identified based on reading the gestures from the user.

[0044] Figure 1An architecture 10 is illustrated, comprising one or more wearable devices 100-1 (e.g., smart glasses) and 100-2 (e.g., smartwatches) (hereinafter collectively referred to as "wearable device 100") according to some embodiments, which are coupled to each other, to a mobile device 110, a remote server 130, and to a database 152. The smart glasses 100-1 may be configured for augmented reality (AR) / virtual reality (VR) applications, and the mobile device 110 may be a smartphone. All these devices may communicate with each other wirelessly and exchange datasets 103-1. Datasets 103-1 may include recorded video, audio, or other files or streaming media. The user 101 of the wearable device 100 is also the owner of or associated with the mobile device 110. In some embodiments, the smart glasses may communicate directly with a remote server, database, or any other client device (e.g., smartphones of different users, etc.) via a network. Mobile devices can be communicatively coupled to remote servers and databases via network 150, and can transfer / share information, files, etc., such as datasets 103-2 and 103-3 (hereinafter collectively referred to as "dataset 103"). For example, network 150 may include any one or more of a local area network (LAN), a wide area network (WAN), and the Internet. Furthermore, the network may include, but is not limited to, any one or more of the following network topologies: bus networks, star networks, ring networks, mesh networks, star-bus networks, and tree or hierarchical networks.

[0045] Smart glasses 100-1 may include a frame 105 including eyepieces 107 for providing a forward-facing image to a user 101. A (e.g., forward-looking) camera 115 is mounted on the frame 105. A user-facing sensing device 128 is configured to track the user's pupil position. A processor 112 is configured to identify the user's region of interest (ROI). An interface device 129 indicates to the user 101 that the field of view (FOV) of the camera 115 is misaligned with the ROI. In some embodiments, smart glasses 100-1 may automatically identify misalignment between the ROI and the FOV of the camera 115 by applying algorithms such as object recognition or scene understanding. In the event of a static offset between the FOV and the ROI of the camera 115, user calibration may be implemented to apply a strain gauge to the user on the display. In some embodiments, the smart glasses 100-1 may further include a haptic actuator 125 and a speaker 127. The haptic actuator is used to reconstruct tactile feedback to the user for use in VR / AR applications, and the speaker is used to transmit voice or sound signals to the user 101 instructing them to adjust their gaze direction to improve the field of view (FOV) of the camera 115. For example, in some embodiments, the haptic actuator 125 may include a vibration component that instructs the user to move their head position in a desired direction (e.g., applying vibration to the user's left temple to move the user's gaze slightly to the left, etc.) to align the FOV of the forward-looking camera 115 with the area of ​​interest (ROI).

[0046] In some embodiments, the smart glasses 100-1 may include a plurality of sensors 121, such as an inertial measurement unit (IMU), a gyroscope, a microphone, and a capacitive sensor configured as a touch interface for a user. Other touch sensors may include pressure sensors, thermometers, etc.

[0047] Furthermore, the wearable device 100 or mobile device 110 may include memory circuitry 120 and processor circuitry 112. The memory circuitry stores instructions, and the processor circuitry is configured to execute these instructions to cause the smart glasses 100-1 to at least partially perform some steps of the methods consistent with this disclosure. The memory circuitry 120 may also store data, such as calibration data of the position and orientation of the camera 115 relative to the user's field of view (FOV). In some embodiments, the smart glasses 100-1, mobile device 110, remote server 130, and / or database 152 may also include a communication module 118 that enables the device to wirelessly communicate with the remote server 130 via network 150. Thus, the smart glasses 100-1 can download multimedia online content (e.g., dataset 103-1) from the remote server 130 to at least partially perform some operations of the methods disclosed herein. In some embodiments, the memory 120 may include instructions that cause the processor 112 to receive and combine signals from the sensor 121, avoid false affirmations, and better evaluate user intentions and commands when receiving input signals from the user interface.

[0048] In some embodiments, the smartwatch or wristband device 100-2 may include contact and motion sensors that capture gestures of the user 101. Gestures can be used to increase, expand, pan, or move the field of view (FOV) of one or more cameras 115 based on the user 101's region of interest (ROI). In some embodiments, the processor 112 may be configured to recognize user gestures provided by any of the sensors in the plurality of wearable devices 100 and adjust the FOV of the camera 115 accordingly.

[0049] Figures 2A to 2C A scene 210 viewed by a user wearing smart glasses 200, according to some embodiments, is illustrated. The user's gaze can be identified using an eye-tracking mechanism 228, such as gaze vector 229A. For example, the user may scan towards a frame (e.g., ROI 238). In some embodiments, the center of ROI 238 is selected as the convergence / divergence point of gaze vector 229A. As shown, according to some embodiments, gaze vector 229A is aligned with the FOV of a first forward-facing camera 215-1 or a second forward-facing camera 215-2 (hereinafter collectively referred to as "forward-facing camera 215"), which is shown here in a dashed box. In some embodiments, previously calibrated settings stored in memory may allow the system to estimate the desired framing to capture the object of interest within ROI 238.

[0050] When gaze vector 229B is outside the field of view (FOV) of the forward-facing camera 215-1, the system can provide feedback to the user indicating that the ROI 238 is outside the FOV of one of the forward-facing cameras 215. This indication can be provided, for example, by an audio cue from the microphone 227 audible to the user of the smart glasses 200; or by tactile feedback on the right side of the glasses; or by some other means. Upon receiving this feedback, the user can then choose to move their head, along with the eyepiece 207 and the forward-facing camera 215, to bring the ROI 238 into the field of view of the forward-facing camera 215. When the gaze overlaps with or is concentric with the FOV of at least one of the forward-facing cameras 215, the system can issue a "green light" notification to indicate that the ROI 238 has been transmitted to the capture system.

[0051] The ROI 238 for smart glasses users is selected according to some embodiments. In some embodiments, the system may capture / trigger still image capture or video capture. In some embodiments, the system may apply optimization techniques to the region of interest (e.g., exposure optimization, camera focus optimization, etc.) or to other modules / events that benefit from the region of interest information.

[0052] Figure 3 Visual indicators 350 in the eyepiece 307 of smart glasses 300 according to some embodiments are shown to help the user adjust the FOV 338 of the first front-view camera 315-1 for image capture based on the region of interest (ROI). Scene 310 can also be captured by a second front-view camera 315-2 (hereinafter, the first and second front-view cameras 315-1 and 315-2 will be referred to as "camera 315"). In fact, in some embodiments, the system can identify objects outside the FOV 338 indicated by the gaze vector 329. For this purpose, the system may include an eye-tracking sensor 328. Thus, the system can instruct the user to move their head slightly in a given direction (e.g., to the right and slightly upward), or it can determine that the FOV of the second front-view camera 315-2 is better aligned with the user's ROI. Therefore, the system can turn on the second front-view camera 315-2 and turn off the first front-view camera 315-1.

[0053] Figure 4The illustration shows front-view cameras 415-1 and 415-2 (hereinafter collectively referred to as "front-view cameras 415") in a smart glasses 400 according to some embodiments, and interference 410 on the field of view (FOV) 438 of at least one of the plurality of front-view cameras 415. The smart glasses 400 includes an eyepiece 407. For example, interference 410 could be the brim of a hat worn by a user that unintentionally obstructs the FOV 438. In some embodiments, the system can automatically crop the interference 410 from the image and notify the user that automatic cropping has been used. Thus, in such a configuration, the processor 412 can identify the interference and alert the user via a speaker 427 that a hat or "foreign object" is obstructing the FOV 438.

[0054] In some embodiments, the user may have actually captured multiple images, including interference 410, with camera 415-1. These images can then be uploaded to a mobile device (see mobile device 110 and dataset 103-1) via wireless data transmission. An algorithm running on the processor in mobile device 110 can identify interference 410 and provide voice suggestions to smart glasses 400 via microphone 427.

[0055] In some embodiments, the gaze vector 429 determined using the eye-tracking sensor 428 may select the FOV 438 from the first forward-looking camera 415-1. Clearly, the eye-tracking sensor 428 may not recognize or even perceive interference 410, which may also obstruct a portion of the FOV of the second forward-looking camera 415-2. In response to information from the speaker 427, the user can adjust the hat or remove it completely. Once the FOV 438 is free from interference 410, the system can issue a clearance notification instructing the user that it is now possible to take a picture using any of the forward-looking cameras 415.

[0056] Figure 5 A smart glasses 500 with glass-like light-blocking capabilities is illustrated according to some embodiments. In some embodiments, an active liquid crystal with polarizing components (e.g., a linear polarizer, a half-wave plate, and / or a quarter-wave plate, etc.) can be switched to increase or decrease the transmittance of light passing through the lenses or eyepieces of the glasses. Thus, a transparent state 501A (high transparency) and a light-blocking state 501B (low transparency, "sunglasses" state) can be provided to the eyepiece 507. Spatially modulating this characteristic across the eyepiece 507 can be used to provide the user with feedback on the relative position of the camera's field of view and the forward-facing image viewed by the user.

[0057] In some embodiments, each eyepiece 507 includes an actuator configured to modify the optical transparency of at least a portion of the eyepiece. The smart glasses also include a camera 515 mounted on a frame 505. The camera 515 faces forward toward the user's field of view (FOV). In some embodiments, the smart glasses 500 also includes a processor 512 configured to compare the FOV of the camera 515 with a forward-facing image viewed by the user. The processor 512 may be configured to indicate to the user that the camera's FOV lacks a portion of the forward-facing image viewed by the user by adjusting the optical transparency of at least a portion of the eyepiece 507. A memory 520 stores instructions that, when executed by the processor 512, cause the smart glasses 500 to perform the steps of the methods disclosed herein.

[0058] Figures 6A to 6B Examples of spatial modulation of the following across eyepiece 607 according to some embodiments are shown: a partially vertically transparent state 601A and a partially horizontally transparent state 601B (hereinafter collectively referred to as "transparent state 601") and a partially vertically light-blocking state 602A and a partially horizontally light-blocking state 602B (hereinafter collectively referred to as "light-blocking state 602"). The smart glasses 600 may include one or two forward-facing cameras 615-1 and 615-2 (hereinafter collectively referred to as "cameras 615"). Either or both of the cameras 615 can be active at any given time to ensure better capture of the forward-facing image according to the user's ROI. To fully utilize the power used by the smart glasses 600, the system can be configured to keep only one of the multiple cameras 615 'on' at any given time. Therefore, a suitably designed liquid crystal can be configured to switch locally within each eyepiece 607. In the example shown, the vertical portion of eyepiece 607 can be blocked, and the vertical portion can be made more or less transparent to provide feedback to the user. This feedback can indicate that the field of view (FOV) of one of the multiple cameras 615 is active. For example, when the partially vertically transparent state 601A of eyepiece 607 provides a clear vertical stripe in the middle of the forward-facing image viewed by the user, this can indicate that the camera's FOV is correctly centered in the horizontal direction. Similarly, moving the partially vertically transparent state 601A to the left or right along eyepiece 607 can provide the user with a clear indication that the frustum of one of the active multiple cameras 615 is not aligned in the horizontal direction and that head posture may need to be adjusted before acquiring an image or video by moving left or right, respectively, or that the user should activate another of the multiple cameras 615.

[0059] Similarly, a clear horizontal stripe 601B in the center of the user's field of view can indicate that the field of view of one of the multiple cameras 615 is correctly centered in the vertical direction. The horizontal stripe 601B is defined upper and lower by a semi-transparent or "light-blocking" stripe 602B. Likewise, by moving the transparent stripe 601B up or down, the eyepiece can provide the user with a clear indication that the field of view of one of the multiple cameras 615 is not aligned in the vertical direction and that head posture adjustments may be necessary before acquiring images or video.

[0060] Therefore, the eyepiece 607 with this partial transmission switching state can be used to provide the user with some broad level of feedback regarding what is seen through the camera 615 mounted on the smart glasses 600. In some embodiments, the eye-tracking sensor 628 can be used to identify the user's ROI, in which case the transparency state 601 is adjusted accordingly.

[0061] Alternatively, some embodiments may include side bars and top / bottom bars in photo / video capture mode, so that the user will automatically understand how the camera is framing the shot, even if the display in the eyepiece is not partially obscured.

[0062] Figures 7A to 7B A smart glasses 700 with two eyepieces 707 in a single transmission state, facing a scene, is shown according to some embodiments. A camera 715, mounted on the frame 705 of the smart glasses 700, captures the field of view 738 of the entire scene 710 as viewed by the user of the smart glasses 700. Therefore, the embodiments disclosed herein help the user understand the camera coverage 748 of the scene at the time of capture without requiring a viewfinder / display (which displays the camera's field of view), as shown below.

[0063] In some embodiments, the eyepiece 707 has locally switchable transmission characteristics, so, according to some embodiments (see smart glasses 700), the (vertical) portion 701B at the outer edge of the eyepiece is in a darker transmission state. Furthermore, the clearer portion 701A overlaps with the camera's FOV 738 (e.g., the field of view of camera 715). Therefore, the clearer portion 701A will appear in the photograph, while portion 701B will be cropped from the camera capture. In some embodiments, based on the position and orientation of the camera relative to the eyepiece, the position and size of the clearer portion 701A can be calibrated to the camera's field of view. Therefore, the user can be informed that the clearly visible portion of the view will be captured by the camera.

[0064] When the camera's field of view (FOV) 738 covers the user's ROI, the user can activate image acquisition. When the camera's FOV 738 does not cover the user's ROI, the user can move their head to adjust the camera's FOV 738. In cases where the smart glasses 700 includes more than one camera 715 and the camera's FOV 738 fails to cover the user's ROI, the user can select another camera for image capture, and the system can indicate the size and position of the selected camera's field of view to the user before image acquisition.

[0065] Figure 8 This is a flowchart illustrating steps in method 800, according to some embodiments, for matching a region of interest with the field of view of cameras in smart glasses (e.g., smart glasses 100-1, 200, 300, 400, 500, 600, and 700, and cameras 115, 215, 315, 415, 515, 615, and 715). In addition to cameras, smart glasses may also include eye-tracking sensors, a memory storing instructions, and a processor (e.g., sensing devices 128, 228, 328, and 428, memories 120 and 520, and processors 112 and 512) configured to execute these instructions to at least partially perform one or more steps in method 800. Methods consistent with this disclosure may include at least one step of method 800, or two or more steps of method 800 executed in different orders, simultaneously, quasi-simultaneously, or overlapping in time.

[0066] Step 802 includes receiving an instruction from a user of the smart glasses indicating an object of interest within the user's field of vision. In some embodiments, step 802 includes tracking the position of the user's pupils and identifying the object of interest based on the position of the user's pupils. In some embodiments, step 802 includes receiving the position of the user's eyes, which indicates the position of the object of interest within the user's field of vision. In some embodiments, step 802 includes determining the user's gaze direction based on the positions of the user's two pupils. In some embodiments, step 802 includes receiving the positions of the user's two pupils, and determining the gaze direction includes determining the convergence and divergence of the positions of the user's two pupils. In some embodiments, step 802 may include displaying two converging lines in the eyepiece of the smart glasses indicating the gaze direction, the two converging lines converging on an object of obvious interest to the user, showing the user what the new crop in the camera image is. In some embodiments, step 802 includes determining a region of interest centered on the object of interest, having the field of view of a first forward-facing camera. In some embodiments, the smart glasses include a second front-view camera, and step 802 further includes receiving an instruction from the user to activate the second front-view camera when the region of interest is not aligned with the field of view of the first front-view camera.

[0067] Step 804 includes comparing the region of interest centered on the object of interest with the field of view of a first forward-facing camera mounted on the frame of the smart glasses.

[0068] Step 806 includes providing feedback to the user via an interface actuator mounted on the smart glasses device, based on the relative position between the region of interest and the field of view of a first forward-facing camera. In some embodiments, the smart glasses include a second forward-facing camera, and step 806 includes determining the relative position between the region of interest and the field of view of the second forward-facing camera. In some embodiments, the smart glasses include a speaker, and step 806 includes providing the user with an instruction to move the smart glasses in a given direction. In some embodiments, the smart glasses include a light emitter, and step 806 includes using the light emitter to indicate the direction of the user's gaze movement.

[0069] Hardware Overview

[0070] Figure 9 This is a block diagram illustrating an exemplary computer system 900 according to some embodiments, which can be used to implement... Figure 1The head-mounted device 100 and method 800. In some aspects, the computer system 900 may be implemented using hardware or a combination of software and hardware, either in a dedicated server, integrated into another entity, or distributed across multiple entities. The computer system 900 may include a desktop computer, laptop computer, tablet computer, phablet, smartphone, feature phone, server computer, or others. The server computer may be located remotely in a data center or stored locally.

[0071] Computer system 900 includes a bus 908 or other communication mechanism for transmitting information, and a processor 902 (e.g., processor 112) coupled to the bus 908 for processing information. As an example, computer system 900 may be implemented using one or more processors 902. Processor 902 may be a general-purpose microprocessor, microcontroller, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), programmable logic device (PLD), controller, state machine, gated logic, discrete hardware components, or any other suitable entity capable of performing computations or other information operations on information.

[0072] In addition to hardware, computer system 900 may also include code that creates the execution environment for the computer program in question, such as code constituting the following stored in included memory 904 (e.g., memory 120): processor firmware, protocol stack, database management system, operating system, or a combination thereof, such as random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable PROM (EPROM), registers, hard disk, removable disk, compact optical disc read-only memory (CD-ROM), digital versatile optical disc (DVD), or any other suitable storage device, coupled to bus 908 for storing information and instructions to be executed by processor 902. Processor 902 and memory 904 may be supplemented by or incorporated into dedicated logic circuitry.

[0073] Instructions may be stored in memory 904 and may be implemented in one or more computer program products, such as one or more modules of computer program instructions encoded on a computer-readable medium for execution by or control of the operation of computer system 900, and such computer program instructions include, but are not limited to, computer languages, such as data-oriented languages ​​(e.g., SQL, dBase), system languages ​​(e.g., C, Objective-C, C++, assembly), structured languages ​​(e.g., Java, .NET), and application languages ​​(e.g., PHP, Ruby, Perl, Python). Instructions can also be implemented using computer languages, such as array languages, aspect-oriented languages, assembly languages, written languages, command-line interface languages, compiled languages, concurrent languages, brace languages, dataflow languages, data structure languages, declarative languages, esoteric languages, extended languages, fourth-generation languages, functional languages, interactive mode languages, interpreted languages, iterative languages, list-based languages, small languages, logic-based languages, machine languages, macro languages, metaprogramming languages, multi-paradigm languages, numerical analysis, non-English-based languages, class-based object-oriented languages, prototype-based object-oriented languages, offside rule languages, procedural languages, reflection languages, rule-based languages, scripting languages, stack-based languages, synchronous languages, syntax processing languages, visual languages, Word languages, and XML-based languages. Memory 904 can also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 902.

[0074] The computer program discussed herein does not necessarily correspond to a file in a file system. A program may be stored as a portion of a file containing other programs or data (e.g., one or more scripts stored in a markup language document), a single file dedicated to the program discussed, or multiple coordinating files (e.g., a file storing one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one or more computers located at a single site or distributed across multiple sites and interconnected via a communication network. The processes and logical flows described in this specification may be executed by one or more programmable processors that execute one or more computer programs to perform functions by manipulating input data and generating output.

[0075] Computer system 900 also includes a data storage device 906, such as a disk or optical disk, coupled to bus 908 for storing information and instructions. Computer system 900 can be coupled to various devices via input / output module 910. Input / output module 910 can be any input / output module. Exemplary input / output module 910 includes a data port such as a Universal Serial Bus (USB) port. Input / output module 910 is configured to connect to communication module 912. Exemplary communication module 912 includes a network interface card, such as an Ethernet card and a modem. In some aspects, input / output module 910 is configured to connect to multiple devices, such as input device 914 and / or output device 916. Exemplary input device 914 includes a keyboard and pointing device, such as a mouse or trackball, through which a user can provide input to computer system 900. Other types of input devices 914 can also be used to provide interaction with the user; these input devices are, for example, haptic input devices, visual input devices, audio input devices, or brain-computer interface devices. For example, the feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound input, voice input, tactile input, or brainwave input. Exemplary output device 916 includes a display device for displaying information to the user, such as a liquid crystal display (LCD) monitor.

[0076] According to one aspect of this disclosure, a wearable device 100 can be at least partially implemented using a computer system 900 in response to processor 902 executing one or more sequences of one or more instructions contained in memory 904. These instructions may be read into memory 904 from another machine-readable medium (e.g., data storage device 906). Execution of the instruction sequence contained in memory 904 causes processor 902 to perform the process steps described herein. One or more processors in a multiprocessor configuration may also be employed to execute the instruction sequence contained in memory 904. Alternatively, hardwired circuitry may be used instead of software instructions, or hardwired circuitry may be combined with software instructions to implement various aspects of this disclosure. Therefore, aspects of this disclosure are not limited to any particular combination of hardware circuitry and software.

[0077] The aspects of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer with a graphical user interface or web browser through which a user can interact with embodiments of the subject matter described in this specification); or the aspects of the subject matter described in this specification can be implemented in any combination of one or more such back-end components, one or more such middleware components, or one or more such front-end components. The components of the system can be interconnected via any form or medium of digital data communication (e.g., a communication network). A communication network (e.g., network 150) may include, for example, any one or more of the following: LAN, WAN, and the Internet. Furthermore, for example, the communication network may include, but is not limited to, any one or more of the following network topologies: bus network, star network, ring network, mesh network, star-bus network, and tree or hierarchical network. For example, a communication module may be a modem or an Ethernet card.

[0078] Computer system 900 may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is generated by computer programs running on their respective computers and having a client-server relationship with each other. For example, computer system 900 may be, but is not limited to, a desktop computer, laptop computer, or tablet computer. Computer system 900 may also be embedded in another device, such as, but not limited to, a mobile phone, a personal digital assistant (PDA), a mobile audio player, a Global Positioning System (GPS) receiver, a video game console, and / or a set-top box.

[0079] As used herein, the terms "machine-readable storage medium" or "computer-readable medium" refer to any medium or medium that participates in providing instructions to processor 902 for execution. Such media can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical discs or magnetic disks, such as data storage device 906. Volatile media include dynamic memory, such as memory 904. Transmission media include coaxial cables, copper wires, and optical fibers, which include conductors forming bus 908. Common forms of machine-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs, any other optical media, punched cards, paper tapes, any other physical media with a perforated pattern, RAM, PROMs, EPROMs, flash EPROMs, any other memory chips or cartridges, or any other media that a computer can read. A machine-readable storage medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a combination of substances that affect machine-readable propagation signals, or a combination of one or more of these.

[0080] To illustrate the interchangeability of hardware and software, various descriptive blocks, modules, components, methods, operations, instructions, and algorithms have been described in general terms of their functionality. Whether this functionality is implemented as hardware, software, or a combination of both depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the described functionality in different ways for each specific application.

[0081] As used herein, the phrase “at least one of” precedes a list of items, separated by the terms “and” or “or”, and modifies the entire list, not each element of the list (e.g., each item). The phrase “at least one of” does not require the selection of at least one item; rather, the phrase means to include at least one of any of these items, and / or at least one of any combination of these items, and / or at least one of each of these items. As an example, each of the phrases “at least one of A, B, and C” or “at least one of A, B, or C” refers to: only A, only B, or only C; any combination of A, B, and C; and / or, at least one of each of A, B, and C.

[0082] The term “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments. The following phrases are for convenience and do not imply that the disclosure associated with one or more such phrases is essential to the subject matter or that the disclosure applies to all configurations of the subject matter: these phrases are, for example, one aspect, that aspect, another aspect, some aspects, one or more aspects, one implementation, that implementation, another implementation, some implementations, one or more implementations, an embodiment, that embodiment, another embodiment, some embodiments, one or more embodiments, a configuration, that configuration, another configuration, some configurations, one or more configurations, the subject matter, the disclosure, this disclosure, and other variations thereof. The disclosure associated with one or more such phrases may apply to all configurations, or one or more configurations. The disclosure associated with one or more such phrases may provide one or more examples. Phrases such as one aspect or some aspects may refer to one or more aspects, and vice versa, and this also applies to other foregoing phrases.

[0083] Unless otherwise stated, references to elements in the singular form are not intended to mean "one and only one," but rather "one or more." The term "some" refers to one or more. Underlined and / or italicized headings and subheadings are used for convenience only and do not limit the subject matter, nor do they imply any interpretation of the description of the subject matter. Related terms such as "first" and "second" are used to distinguish one entity or action from another without necessarily requiring or implying any actual such relationship or order between these entities or actions. All structural and functional equivalents of elements in the various configurations described herein that are known or will be known hereafter by one of ordinary skill in the art are intended to be expressly incorporated herein by reference and are intended to be included in the subject matter. Furthermore, nothing disclosed herein is intended to be offered to the public, whether or not such disclosure is expressly stated in the foregoing description.

[0084] While this specification contains numerous details, these details should not be construed as limiting the scope of what may be described, but rather as descriptions of particular embodiments of the subject matter. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations, and even initially described as such, in some cases, one or more features from the described combination may be removed from that combination, and the described combination may involve sub-combinations or variations thereof.

[0085] The subject matter of this specification has been described in certain aspects, but other aspects may be implemented and fall within the scope of the appended claims. For example, although operations are depicted in a specific order in the drawings, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order, or requiring all shown operations to achieve the desired result. The actions described in the claims may be performed in different orders and still achieve the desired result. As an example, the process depicted in the drawings does not necessarily require the specific order or sequential order shown to achieve the desired result. In some cases, multitasking in parallel may be advantageous. Furthermore, the separation of the various system components in the foregoing aspects should not be construed as requiring such separation in all aspects, but rather it should be understood that the described program components and systems can generally be integrated together in a single software product or encapsulated in multiple software products.

[0086] The title, background art, description of the drawings, abstract, and drawings are incorporated herein by reference and are provided as illustrative examples rather than limiting descriptions. It should be understood that this abstract is not intended to interpret or limit the scope or meaning of the claims. Furthermore, as will be seen in the detailed description, the description provides illustrative examples, and different features are combined in different embodiments to simplify the disclosure. The approach of this disclosure should not be construed as reflecting an intention that the described subject matter requires more features than are expressly recited in each claim. Rather, as reflected in the claims, the inventive subject matter lies in fewer features than all the features of a single disclosed configuration or operation. The claims are incorporated herein by reference, each claim being an independent subject matter described separately.

[0087] The claims are not intended to be limited to the aspects described herein, but should be given the full scope consistent with the language claims and cover all legal equivalents. Nevertheless, none of the claims are intended to include subject matter that does not meet the requirements of applicable patent law, nor should they be interpreted in this way.

Claims

1. A smart glasses device, comprising: A frame, the frame including an eyepiece for providing a forward-facing image to a user; A first forward-facing camera is mounted on the frame. A processor configured to identify regions of interest within the forward-facing image; An interface device, the interface device being used to indicate to the user that the first field of view of the first forward-looking camera is not aligned with the region of interest; as well as A second forward-viewing camera is mounted on the frame, wherein the second forward-viewing camera is configured to be turned on when the region of interest is better aligned with the second field of view of the second forward-viewing camera, and the first forward-viewing camera is configured to be turned off.

2. The smart glasses device according to claim 1 further includes a sensing device, the sensing device facing the user for tracking the position of the user's pupils, wherein, The processor identifies the region of interest based on the user's pupil position.

3. The smart glasses device according to claim 1 or 2, wherein, The processor identifies the region of interest based on the user's gaze direction within the field of view.

4. The smart glasses device according to claim 1 further includes a memory storing instructions that, when executed by the processor, cause the smart glasses device to find a gaze vector indicating the center of the region of interest within the user's field of vision.

5. The smart glasses device according to claim 1, wherein, The interface device includes an audio device for providing signals to the user to adjust head posture and align the first field of view of the first forward-facing camera with the region of interest.

6. The smart glasses device according to claim 1, wherein, The interface device includes a vibration component configured to instruct the user to move their head in a desired direction so that the first field of view of the first forward-facing camera is aligned with the region of interest.

7. The smart glasses device according to claim 1, wherein, The eyepiece includes a liquid crystal configured to, when activated, block a vertical portion or a horizontal portion of the eyepiece to guide the user to move their head horizontally or vertically, respectively, so that the center of the field of view of the first forward-facing camera is aligned with the region of interest.

8. The smart glasses device of claim 1, further comprising an actuator configured to modify the color of a portion of the field of view of the eyepiece corresponding to the region of interest.

9. The smart glasses device according to claim 1 further includes a microphone for receiving voice commands from the user regarding objects of interest within the user's field of vision.

10. The smart glasses device according to claim 1, wherein, The interface device is configured to provide the user with an option to activate the second front-view camera when the first field of view of the first front-view camera is not aligned with the region of interest.

11. A computer-implemented method, comprising: Receive indications from the user of the smart glasses regarding objects of interest within the forward-facing image being viewed by the user; The region of interest centered on the object of interest is compared with the first field of view of the first forward-facing camera, which is mounted on the frame of the smart glasses. When the region of interest is better aligned with the second field of view of the second front-view camera mounted on the frame of the smart glasses, the second front-view camera is activated, wherein the first front-view camera is deactivated when the second front-view camera is activated; as well as Feedback is provided to the user via an interface actuator mounted on the smart glasses, based on the relative position between the region of interest and the first field of view of the first forward-facing camera.

12. The computer-implemented method according to claim 11, wherein, Receiving an indication of an object of interest within the user's field of vision includes one or more of the following: i. Track the position of the user's pupils and identify the object of interest based on the position of the user's pupils; ii. Receive the eye position of the user of the smart glasses, the eye position indicating the location of the object of interest within the user's field of vision; iii. Determine the user's gaze direction based on the positions of the user's two pupils; iv. Receive the positions of the user's two pupils, and determine the gaze direction by determining the convergence or divergence of the positions of the user's two pupils.

13. The computer-implemented method of claim 11 or 12, further comprising determining the region of interest, the region of interest having a region centered on the object of interest within the first field of view of the first forward-looking camera.

14. The computer-implemented method according to claim 11, 12, or 13, further comprising one or more of the following: i. When the region of interest is not aligned with the first field of view of the first forward-facing camera, receive an instruction from the user to open the second forward-facing camera; and / or ii. Providing feedback to the user includes determining the relative position of the region of interest with the second field of view of the second forward-facing camera.

15. The computer-implemented method according to any one of claims 11 to 14, wherein, The smart glasses include: i. A speaker, and providing feedback to the user including giving the user instructions to move the smart glasses in a given direction; and / or ii. A light emitter, and providing feedback to the user includes using the light emitter to indicate the direction in which the user's gaze moves.

Images