Intelligent glasses control method, intelligent glasses, storage medium and program product

Abstract

The invention discloses an intelligent glasses control method, intelligent glasses, a storage medium and a program product, and relates to the technical field of intelligent glasses, and the intelligent glasses control method comprises the steps: obtaining a global depth-of-field image of an environment where the intelligent glasses are located, and recognizing an attention direction pointed by an interaction operation of a wearer of the intelligent glasses; based on the attention direction, identifying an attention target of the wearer for the global depth-of-field image; and focusing and shooting the concerned target. The technical problem that the quality of photos shot by existing intelligent glasses is low is solved.

Description

Technical Field

[0001] This application relates to the field of smart glasses technology, and in particular to a smart glasses control method, smart glasses, storage medium, and program product. Background Technology

[0002] In recent years, with the rapid development of smart wearable devices, smart glasses, as an emerging product that combines fashion and technology, have quickly attracted the attention of a wide range of consumers.

[0003] However, when smart glasses cannot use the display function (such as smart camera glasses without a display module), or when it is temporarily inconvenient to use the display function or display the shooting scene, users cannot enjoy the convenience of real-time preview of the shooting scene when taking pictures, nor can they accurately focus. Currently, most smart camera glasses on the market adopt a camera module design with a wide field of view, fixed focus, and large depth of field. While this design can expand the shooting range to a certain extent, it also brings significant problems such as obvious wide-angle distortion, insufficient overall image sharpness, and difficulty in focusing to highlight the subject, seriously affecting the quality of the photos taken.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide a smart glasses control method, smart glasses, storage medium and program product, which aims to solve the technical problem of low image quality obtained by existing smart glasses.

[0006] To achieve the above objectives, this application proposes a smart glasses control method, which includes: Acquire a global depth image of the environment in which the smart glasses are located, and identify the direction of attention of the smart glasses wearer's interactive operations; Based on the direction of attention, the wearer's focus on the global depth image is identified; Focus on the target of interest and take a picture.

[0007] In one embodiment, the step of acquiring a global depth image of the environment in which the smart glasses are located includes: A first depth-of-field image is obtained by focusing on a distant view of the environment in which the smart glasses are located. A close-up focusing shot is taken of the environment where the smart glasses are located to obtain a second depth-of-field image; The first depth image and the second depth image are fused together to obtain a global depth image.

[0008] In one embodiment, the step of fusing the first depth image and the second depth image to obtain a global depth image includes: Generate a corresponding mask image based on the first depth-of-field image and the second depth-of-field image; Evaluate the sharpness of each pixel region in the mask image for the first depth-of-field image and the second depth-of-field image; If the clarity of the first depth-of-field image in the pixel region is higher than the clarity of the second depth-of-field image in the same pixel region, then the region image of the first depth-of-field image in the pixel region is selected as the selected image. If the clarity of the second depth-of-field image in the pixel region is higher than the clarity of the first depth-of-field image in the same pixel region, then the region image of the second depth-of-field image in the pixel region is selected as the selected image; Selected images of each pixel region are fused to obtain a global depth image.

[0009] In one embodiment, the interaction operation is an eye-tracking interaction operation, and the step of identifying the direction of attention of the smart glasses wearer's interaction operation includes: The wearer's eye-tracking interaction operations are captured to obtain the wearer's eye-tracking image data; The eye-tracking image data is identified to obtain the wearer's gaze direction, which is then used as the focus of the interactive operation.

[0010] In one embodiment, the step of identifying the wearer's focus target in the global depth image based on the focus direction includes: Target detection is performed on the global depth image to obtain each environmental target in the global depth image; The focus direction is mapped onto the global depth image to obtain the focus point corresponding to the focus direction; Environmental targets that match the stated environmental objectives with the stated points of interest are designated as the focus targets.

[0011] In one embodiment, the step of selecting environmental targets that match the points of interest as targets of interest includes: Obtain the location coordinates of the points of interest and the location regions of each environmental target; If the location coordinates of the point of interest are within the location area, then the environmental target corresponding to the location area where the location coordinates are located will be taken as the target of interest. If the location coordinates of the point of interest are not within the location area, then the environmental target corresponding to the location area closest to the location coordinates will be taken as the target of interest.

[0012] In one embodiment, the step of focusing on and capturing the target of interest includes: The shape of the adjustable lens on the camera of the smart glasses is adjusted, and the shooting preview of the camera during the adjustment process is obtained; Once the target's sharpness is maximized in the shooting preview screen, take the picture.

[0013] In addition, to achieve the above objectives, this application also proposes a smart glasses, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the smart glasses control method described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the smart glasses control method described above.

[0015] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the smart glasses control method described above.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: This application acquires a global depth-of-field image of the environment in which the smart glasses are located and identifies the direction of attention of the wearer's interactive operations. This provides a relatively clear global image of the environment and determines the direction the wearer intends to take a picture. Based on this direction of attention, the application identifies the wearer's focus target within the global depth-of-field image, allowing for focus adjustment to capture the target. Because this application can accurately locate the wearer's intended focus target, it eliminates the need for a large-field-of-view camera module, effectively avoiding the wide-angle distortion problems associated with such modules. Furthermore, by focusing on and capturing the focus target, this application effectively ensures image clarity and highlights the subject, thus significantly improving the quality of photos taken by the smart glasses. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating an embodiment of the smart glasses control method of this application. Figure 2 This is a structural example diagram of the smart glasses involved in the embodiments of this application; Figure 3 This is a flowchart illustrating Embodiment 2 of the smart glasses control method of this application; Figure 4 This is a flowchart illustrating Embodiment 3 of the smart glasses control method of this application; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the smart glasses control method in this application embodiment.

[0020] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] The main solution of this application embodiment is: to acquire a global depth image of the environment in which the smart glasses are located, and to identify the direction of attention of the smart glasses wearer's interactive operation; based on the direction of attention, to identify the target of attention of the wearer in the global depth image; and to focus on and take a picture of the target of attention.

[0024] In this embodiment, for ease of description, the following description uses smart glasses as the execution subject.

[0025] When smart glasses cannot use the display function (e.g., smart camera glasses lack a display module), or when it is temporarily inconvenient to use the display function or display the shooting scene, users cannot enjoy the convenience of real-time preview of the shooting scene when taking photos, nor can they accurately focus. Currently, most smart camera glasses on the market adopt a camera module design with a wide field of view, fixed focal length, and large depth of field. While this design can expand the shooting range to a certain extent, it also brings significant problems such as obvious wide-angle distortion, insufficient overall image sharpness, and difficulty in focusing to highlight the subject, seriously affecting the quality of the photos taken.

[0026] This application provides a solution that acquires a global depth-of-field image of the environment in which the smart glasses are located, identifies the direction of attention of the wearer's interactive operations, obtains a relatively clear global image of the environment, and determines the direction in which the wearer intends to take a picture. Based on this direction of attention, the application can identify the wearer's focus target in the global depth-of-field image, and then adjust the focus to capture the target. Because this application can accurately locate the wearer's intended focus target, it eliminates the need for a large-field-of-view camera module, effectively avoiding the wide-angle distortion problems associated with such modules. Furthermore, by focusing on and capturing the focus target, this application effectively ensures the clarity of the captured image and highlights the subject, thereby significantly improving the quality of photos taken by the smart glasses.

[0027] Based on this, embodiments of this application provide a smart glasses control method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the smart glasses control method of this application.

[0028] In this embodiment, the smart glasses control method includes steps S10 to S30: Step S10: Obtain a global depth image of the environment where the smart glasses are located, and identify the direction of attention of the smart glasses wearer's interactive operation. It should be noted that the smart glasses can be smart wearable devices in the form of smart camera glasses, AR (Augmented Reality) glasses, MR (Mixed Reality) glasses, etc.

[0029] Furthermore, it should be noted that the global depth image is an image in which the entire image area (or the largest possible image area) of the environment captured by the smart glasses presents a relatively clear depth effect. The interactive operations can be gesture-based, eye-tracking-based, voice-based, or other interactive methods.

[0030] To obtain a clear global image of the environment surrounding the smart glasses, facilitating subsequent identification of the wearer's focus, this embodiment processes the environmental images captured by the smart glasses using methods such as focus stacking, depth map synthesis, and image fusion to obtain a global depth-of-field image of the environment. Focus stacking is a method of merging multiple photos of the same environment taken at different focus points into a single, fully clear global depth-of-field image. Depth map synthesis generates a depth map of the current scene and then blurs or sharpens the captured environmental images based on the depth map to simulate a global depth-of-field effect, thus obtaining a global depth-of-field image. Image fusion combines images with different focus points, exposures, or viewpoints using image fusion algorithms (such as multi-band fusion, Poisson fusion, etc.) into a single global depth-of-field image.

[0031] Therefore, this embodiment can collect the interaction operations of the wearer of the smart glasses, and then identify the direction of attention of the wearer's interaction operations. It can be understood that the direction of the interaction operation can be a real pointing behavior, such as pointing with a finger or pointing with the eyes, or it can be a virtual pointing behavior, such as pointing to the content being described in voice interaction.

[0032] In one feasible implementation, the interaction operation in step S10 is an eye-tracking interaction operation, and the step of identifying the direction of attention of the smart glasses wearer's interaction operation may include steps S11-S12: Step S11: Image acquisition is performed on the wearer's eye-tracking interaction to obtain the wearer's eye-tracking image data; Step S12: The eye-tracking image data is identified to obtain the wearer's gaze direction as the focus direction for interactive operations.

[0033] It should be noted that the interaction is an eye-tracking interaction, meaning that the wearer interacts with the smart glasses through eye movements.

[0034] In this embodiment, image acquisition of the wearer's eye-tracking interaction operations is performed to obtain eye-tracking image data. This data is then used for recognition to track the wearer's eye movements and determine the direction of their gaze as the focus of the interaction. For example, this embodiment can activate a camera to acquire images of the wearer's eye-tracking interaction operations, capturing eye-tracking image data. This data is then recognized to determine the positions of the wearer's pupil center and corneal reflex center. The direction of the wearer's gaze is then calculated by calculating the vector relationship between the pupil center and the corneal reflex center. To further improve the accuracy of the gaze direction, the wearer's head position information can also be incorporated. Finally, the wearer's gaze direction is used as the focus of the interaction.

[0035] Step S20: Based on the direction of attention, identify the wearer's focus target in the global depth image; This embodiment maps the direction of interest onto the global depth image to obtain the intersection point of the direction of interest and the image plane of the global depth image. Then, the environmental targets at the intersection point in the global depth image are taken as targets of interest. For example, this embodiment can perform target detection on the global depth image to obtain each environmental target in the global depth image; map the direction of interest onto the global depth image to obtain the attention point corresponding to the direction of interest; and take the environmental targets that match the attention points as targets of interest. Further, this embodiment can also map the direction of interest onto the global depth image to obtain the attention point corresponding to the direction of interest, and then segment an image of a preset size region from the global depth image with the attention point as the center point as the matching image. Then, target detection is performed on the matching image to obtain each environmental target in the matching image, and the environmental targets that match the attention points are taken as targets of interest. This reduces the image area requiring target detection, decreases the time required for target detection, and improves the recognition efficiency of the targets of interest.

[0036] In one feasible implementation, step S20 may include steps S21 to S23: Step S21: Target detection is performed on the global depth image to obtain each environmental target in the global depth image; Step S22: Map the direction of interest onto the global depth image to obtain the point of interest corresponding to the direction of interest; Step S23: Select the environmental targets that match the points of interest as the targets of interest.

[0037] This embodiment employs an object detection algorithm to detect objects in the global depth image, obtaining various environmental objects within the global depth image. The object detection algorithm can be R-CNN (Region-CNN), Fast R-CNN, YOLO (You Only Look Once), or other algorithms used for object detection. Furthermore, this embodiment maps the focus direction onto the global depth image, and then uses the intersection points of each focus direction with the image plane of the global depth image as the focus points corresponding to the focus directions. Finally, environmental objects that match the focus points are designated as focus targets, i.e., the environmental targets to which the focus points fall are considered focus targets.

[0038] In one feasible implementation, step S23 may include steps A10 to A30: Step A10: Obtain the location coordinates of the point of interest and the location regions of each environmental target; Step A20: If the location coordinates of the point of interest are within the location area, then the environmental target corresponding to the location area where the location coordinates are located is taken as the target of interest. Step A30: If the location coordinates of the point of interest are not within the location area, then the environmental target corresponding to the location area closest to the location coordinates is taken as the target of interest.

[0039] This embodiment obtains the location coordinates of the point of interest and the location regions of each environmental target. The location coordinates are the coordinates of the point of interest on the global depth image, and the location regions of the environmental targets are the areas occupied by the environmental targets on the global depth image, such as the target detection bounding box of the environmental target or the internal region of the edge contour of the environmental target. It then determines whether the location coordinates are within the location regions of the environmental targets. If the location coordinates of the point of interest are within the location regions, the environmental target corresponding to the location region where the location coordinates are located is taken as the target of interest. If the location coordinates of the point of interest are not within the location regions, the environmental target corresponding to the location region closest to the location coordinates is taken as the target of interest. Thus, this embodiment identifies environmental targets matching the point of interest as targets of interest by using the location of the point of interest and the regions occupied by each environmental target.

[0040] Step S30: Focus on the target of interest and take a picture.

[0041] It should be noted that the camera of the smart glasses can be configured with an adjustable lens to adjust the focal length. Of course, it can also use optical zoom module, digital zoom module, hybrid zoom module or other modules to adjust the focal length of the camera.

[0042] This embodiment allows for focus adjustment via the camera of the smart glasses to focus on the target of interest, and then takes a picture after receiving a shooting command. Furthermore, this embodiment can also execute steps S10 to S30 only after the smart glasses receive the shooting command, thereby reducing the power consumption of the smart glasses.

[0043] See Figure 2 , Figure 2 This is a structural example diagram of the smart glasses involved in an embodiment of this application. Figure 2 As shown, smart glasses may include a frame and lenses. The frame includes a lens frame, a left temple, and a right temple. An eye-tracking camera (i.e., a camera for tracking eye movements) is located on the inner left and right sides of the lens frame, and an environmental camera (a camera for capturing images of the environment in which the smart glasses are located) is located on the outer side of the lens frame. It is understood that... Figure 2 The structures of smart glasses presented here are merely examples; in reality, smart glasses can include more or fewer components. Understandably, the environmental camera can use adjustable lenses (such as Tuneable Lenses, which can change the lens shape and thus the optical path and focal length by adjusting the input voltage) to achieve focusing. Compared to traditional optical zoom modules, adjustable lenses offer advantages such as smaller size, faster focusing speed, lower power consumption, and no magnetic interference or pose error. Compared to digital zoom modules, images captured with adjustable lenses have higher resolution and clarity.

[0044] The first embodiment of this application provides a smart glasses control method. By acquiring a global depth-of-field image of the environment in which the smart glasses are located, and identifying the direction of attention of the wearer's interactive operations, a relatively clear global image of the environment in which the smart glasses are located can be obtained, as well as the direction in which the wearer intends to take a picture can be determined. Furthermore, based on the direction of attention, the wearer's focus on the global depth-of-field image can be identified, and the focus can be adjusted to focus on and capture the target of attention. Because this embodiment can accurately locate the target of attention that the wearer intends to capture, it eliminates the need for a large-field-of-view camera module, effectively avoiding the wide-angle distortion problem caused by such camera modules. Moreover, by focusing on and capturing the target of attention, this embodiment effectively ensures the clarity of the captured image and highlights the target of attention in the captured image, thereby effectively improving the quality of the photos taken by the smart glasses.

[0045] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 The step of obtaining the global depth image of the environment where the smart glasses are located in step S10 includes steps B10 to B30: Step B10: Take a distant focus shot of the environment where the smart glasses are located to obtain a first depth-of-field image; Step B20: Take a close-up focusing shot of the environment where the smart glasses are located to obtain a second depth-of-field image; Step B30: Perform depth fusion on the first depth image and the second depth image to obtain a global depth image.

[0046] This embodiment obtains a first depth-of-field image by focusing on and capturing a distant view of the environment where the smart glasses are located, according to a preset distant focal length. The preset distant focal length can include at least one distant focal length, which is a pre-set focal length for capturing distant views, thereby obtaining at least one distant image as the first depth-of-field image. Then, a second depth-of-field image is obtained by focusing on and capturing a close-up view of the environment where the smart glasses are located, according to a preset close-up focal length. The preset close-up focal length can include at least one close-up focal length, which is a pre-set focal length for capturing close-up views, thereby obtaining at least one close-up image as the second depth-of-field image. It is understood that the preset close-up focal length is shorter than the preset distant focal length. The first depth-of-field image and the second depth-of-field image can then be fused to obtain a global depth-of-field image. This embodiment can evaluate the sharpness of the first depth-of-field image and the second depth-of-field image; for example, gradient, Laplacian transform, or other edge detection methods can be used to evaluate the image sharpness. Then, based on the sharpness, the weight of each pixel region in the first depth image and the second depth image is determined, with pixel regions having higher sharpness being assigned higher weights. Then, based on the weights, the first depth image and the second depth image are fused to obtain a global depth image. For example, the image fusion method can be a spatial domain-based fusion method (such as selection based on sharpness measurement and weight calculation based on Laplacian transform), a transform domain-based fusion method (such as Laplacian pyramid transform, contrast pyramid transform, complex wavelet transform, dual-tree complex wavelet transform, etc.), or an image fusion method based on neural networks, fuzzy theory, etc.

[0047] In one feasible embodiment, step B30 includes steps C10 to C50: Step C10: Generate a corresponding mask image based on the first depth image and the second depth image; Step C20: Evaluate the sharpness of each pixel region of the first depth-of-field image and the second depth-of-field image in the mask image; Step C30: If the clarity of the first depth-of-field image in the pixel region is higher than the clarity of the second depth-of-field image in the same pixel region, then the region image of the first depth-of-field image in the pixel region is selected as the selected image. Step C40: If the clarity of the second depth image in the pixel region is higher than the clarity of the first depth image in the same pixel region, then the region image of the second depth image in the pixel region is selected as the selected image. Step C50: Perform image fusion on the selected images of each pixel region to obtain a global depth image.

[0048] Since the first depth-of-field image and the second depth-of-field image are essentially images captured from the same environment at different focal lengths, this embodiment can align the first depth-of-field image and the second depth-of-field image and then perform region segmentation to generate a corresponding mask image. The region segmentation method can be mesh segmentation, edge contour segmentation, etc., and this embodiment is not limited to this. This embodiment can evaluate the sharpness of each pixel region of the first depth-of-field image and the second depth-of-field image in the mask image using gradient, Laplacian transform, or other edge detection methods; that is, the higher the edge contour sharpness and the higher the contrast, the higher the sharpness. Then, based on the sharpness, a comparison is made between the first depth-of-field image and the second depth-of-field image in each pixel region to determine whether the first depth-of-field image or the second depth-of-field image is sharper, and the sharpest region image of the first depth-of-field image in each pixel region is selected as the image. Specifically, if the sharpness of the first depth-of-field image in a pixel region is higher than the sharpness of the second depth-of-field image in the same pixel region, then the region image of the first depth-of-field image in that pixel region is selected as the image; if the sharpness of the second depth-of-field image in a pixel region is higher than the sharpness of the first depth-of-field image in the same pixel region, then the region image of the second depth-of-field image in that pixel region is selected as the image. Then, the selected images of each pixel region are fused to obtain a global depth image. The image fusion method for each selected image can be an image fusion algorithm such as weighted averaging or multi-resolution fusion to ensure a smooth transition between the selected images and effectively improve the clarity and naturalness of the global depth image.

[0049] In the second embodiment of this application, a first depth-of-field image is obtained by focusing on a distant view of the environment where the smart glasses are located; a second depth-of-field image is obtained by focusing on a close-up view of the environment where the smart glasses are located; and the first depth-of-field image and the second depth-of-field image are fused to obtain a global depth-of-field image. Therefore, this embodiment achieves global depth-of-field image acquisition without the need for a special camera by capturing a first depth-of-field image at a distant focal length and a second depth-of-field image at a close-up focal length, and then fusing the first and second depth-of-field images. This facilitates the subsequent identification of various environmental targets in the environment where the smart glasses are located.

[0050] Based on the first embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 Step S30 includes steps S31 to S32: Step S31: Adjust the shape of the adjustable lens on the camera of the smart glasses and obtain the shooting preview screen of the camera during the adjustment process; Step S32: After the sharpness of the target in the shooting preview screen reaches its maximum, take a picture.

[0051] It should be noted that the adjustable lens is an optical lens whose shape can be changed to alter the focal length, such as a TuneableLens lens. The lens shape can be changed by adjusting the input voltage, thereby changing the optical path and focal length.

[0052] Taking a TuneableLens lens as an example, this embodiment can adjust the input voltage of the adjustable lens on the camera of the smart glasses according to a predetermined voltage change curve to adjust the lens shape, thereby changing the focal length of the adjustable lens and obtaining a shooting preview image of the camera during the adjustment process. Then, the sharpness of the target in the shooting preview image is detected. For example, the sharpness can be evaluated using the contrast, gradient, and sharpness of the outline edge of the target. When the sharpness of the target in the shooting preview image reaches its maximum, it indicates that the target has been focused, and a shot can be taken. The resulting image, having achieved focus on the target, effectively ensures the sharpness of the captured image and highlights the target as the main subject, thus effectively improving the quality of the photos taken by the smart glasses.

[0053] In the third embodiment of this application, the shape of the adjustable lens on the camera of the smart glasses is adjusted, and a shooting preview image of the camera is acquired during the adjustment process. Once the sharpness of the target in the shooting preview image reaches its maximum, a shot is taken. This embodiment uses an adjustable lens for focal length adjustment. Compared to traditional optical zoom modules, adjustable lenses offer advantages such as smaller size, faster focusing speed, lower power consumption, and no magnetic interference or pose error. Compared to digital zoom modules, images captured with an adjustable lens have higher resolution and sharpness. Therefore, this embodiment further ensures the sharpness of the captured image and the shooting speed.

[0054] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the smart glasses control method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0055] This application provides a smart glasses, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the smart glasses control method in Embodiment 1 above.

[0056] The following is for reference. Figure 5 The diagram illustrates a structure suitable for implementing smart glasses in the embodiments of this application. The smart glasses in the embodiments of this application may include smart camera glasses, AR (Augmented Reality) glasses, MR (Mixed Reality) glasses, and other forms of smart wearable devices. Figure 5 The smart glasses shown are merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0057] like Figure 5As shown, the smart glasses may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the smart glasses. The processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, display modules, speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tape, hard disks, etc.; and communication devices 1009. The communication device 1009 allows the smart glasses to communicate wirelessly or wiredly with other devices to exchange data. While the diagram shows smart glasses with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0058] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0059] The smart glasses provided in this application, employing the smart glasses control method described in the above embodiments, can solve the technical problem of low image quality obtained by existing smart glasses. Compared with the prior art, the beneficial effects of the smart glasses provided in this application are the same as those of the smart glasses control method provided in the above embodiments, and other technical features of the smart glasses are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0060] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0061] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0062] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the smart glasses control method in the above embodiments.

[0063] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0064] The aforementioned computer-readable storage medium may be included in the smart glasses; or it may exist independently and not assembled into the smart glasses.

[0065] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the smart glasses, cause the smart glasses to: acquire a global depth image of the environment in which the smart glasses are located, and identify the direction of attention of the wearer's interactive operation; based on the direction of attention, identify the target of the wearer's attention in the global depth image; and focus on and capture the target of attention.

[0066] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0067] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0068] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0069] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described smart glasses control method, which can solve the technical problem of low image quality obtained by existing smart glasses. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the smart glasses control method provided in the above embodiments, and will not be repeated here.

[0070] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the smart glasses control method described above.

[0071] The computer program product provided in this application can solve the technical problem of low image quality obtained by existing smart glasses. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the smart glasses control method provided in the above embodiments, and will not be repeated here.

[0072] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for controlling smart glasses, characterized in that, The smart glasses control method includes: Acquire a global depth image of the environment in which the smart glasses are located, and identify the direction of attention of the smart glasses wearer's interactive operations; Based on the direction of attention, the wearer's focus on the global depth image is identified; Focus on the target of interest and take a picture.

2. The method as described in claim 1, characterized in that, The step of acquiring a global depth image of the environment in which the smart glasses are located includes: A first depth-of-field image is obtained by focusing on a distant view of the environment in which the smart glasses are located. A close-up focusing shot is taken of the environment where the smart glasses are located to obtain a second depth-of-field image; The first depth image and the second depth image are fused together to obtain a global depth image.

3. The method as described in claim 2, characterized in that, The step of fusing the first depth image and the second depth image to obtain a global depth image includes: Generate a corresponding mask image based on the first depth-of-field image and the second depth-of-field image; Evaluate the sharpness of each pixel region in the mask image for the first depth-of-field image and the second depth-of-field image; If the clarity of the first depth-of-field image in the pixel region is higher than the clarity of the second depth-of-field image in the same pixel region, then the region image of the first depth-of-field image in the pixel region is selected as the selected image. If the clarity of the second depth-of-field image in the pixel region is higher than the clarity of the first depth-of-field image in the same pixel region, then the region image of the second depth-of-field image in the pixel region is selected as the selected image; Selected images of each pixel region are fused to obtain a global depth image.

4. The method as described in claim 1, characterized in that, The interaction is an eye-tracking interaction, and the step of identifying the direction of attention of the smart glasses wearer's interaction includes: The wearer's eye-tracking interaction operations are captured to obtain the wearer's eye-tracking image data; The eye-tracking image data is identified to obtain the wearer's gaze direction, which is then used as the focus of the interactive operation.

5. The method as described in claim 1, characterized in that, The step of identifying the wearer's focus target in the global depth image based on the focus direction includes: Target detection is performed on the global depth image to obtain each environmental target in the global depth image; The focus direction is mapped onto the global depth image to obtain the focus point corresponding to the focus direction; Environmental targets that match the stated environmental objectives with the stated points of interest are designated as the focus targets.

6. The method as described in claim 5, characterized in that, The step of selecting environmental targets that match the points of interest as targets of interest includes: Obtain the location coordinates of the points of interest and the location regions of each environmental target; If the location coordinates of the point of interest are within the location area, then the environmental target corresponding to the location area where the location coordinates are located will be taken as the target of interest. If the location coordinates of the point of interest are not within the location area, then the environmental target corresponding to the location area closest to the location coordinates will be taken as the target of interest.

7. The method according to any one of claims 1 to 6, characterized in that, The step of focusing on and taking a picture of the target of interest includes: The shape of the adjustable lens on the camera of the smart glasses is adjusted, and the shooting preview of the camera during the adjustment process is obtained; Once the target's sharpness is maximized in the shooting preview screen, take the picture.

8. A type of smart glasses, characterized in that, The smart glasses include: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the smart glasses control method as described in any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the smart glasses control method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the smart glasses control method as described in any one of claims 1 to 7.

Images