Intelligent glasses operation method

Abstract

The invention discloses a smart glasses operation method and a computing device. The operation method comprises the following steps: acquiring a plurality of motions on the intelligent glasses and a posture and a pressure change value of the intelligent glasses detected by a pressure sensor; judging that the posture and / or pressure change value reaches a preset threshold range; sending the posture and the pressure change value to the electronic equipment; receiving the posture and the pressure change value on the electronic equipment; a corresponding control instruction is obtained based on the posture and the pressure change value; executing the control instruction; and sending a result of executing the control instruction back to the intelligent glasses. Therefore, by integrating the motion sensor and the pressure sensor in the intelligent glasses and cooperating with a flexible control action recognition algorithm, changes generated due to the fact that a user operates the glasses with hands (for example, the user pushes the glasses with the hands) can be quickly captured and accurately recognized, and quick adjustment of relevant control functions of the intelligent glasses is achieved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of smart glasses, and in particular to a method for operating smart glasses. Background Art

[0002] Smart glasses are a head-mounted device integrating augmented reality (AR) technology. Through a micro display, a camera, sensors, and an audio system, they provide functions such as information display, environmental perception, and remote interaction for users. Users can interact with the glasses through touch, voice, and gesture recognition to achieve operations such as navigation, real-time translation, and shooting records. Smart glasses can achieve seamless information transmission and efficient interaction without disturbing the user's natural line of sight, and can be widely used in various scenarios in daily life.

[0003] However, there are still some problems with existing smart glasses interaction methods. The interaction area of touch interaction is usually very small, so the operation is inconvenient, or additional touch accessories (such as touch rings) need to be worn. Voice interaction has a low recognition rate in noisy environments and there is also a risk of privacy leakage; gesture recognition interaction based on camera shooting has poor stability and high overhead in complex environments. In addition, these existing interaction methods are quite power-consuming and not intuitive and convenient to operate. Summary of the Invention

[0004] One technical problem to be solved by the present disclosure is to provide a method for operating smart glasses. The smart glasses enable users to control the smart glasses through natural manipulation actions by means of multiple built-in motion and pressure sensors, thereby enhancing the interaction experience. Since the manipulation instruction is triggered only by the collaborative determination of the readings of multiple motion and pressure sensors, misjudgment can be avoided to a great extent.

[0005] According to a first aspect of the present disclosure, there is provided a method for operating smart glasses, including: on the smart glasses: obtaining the attitude change values of the smart glasses detected by multiple motion sensors; obtaining the pressure change values of the smart glasses at the positions of the multiple pressure sensors detected by the multiple pressure sensors; determining that the attitude change values reach an attitude change threshold range and / or determining that the pressure change values reach a pressure change threshold range; sending the attitude change values and the pressure change values to an electronic device; on the electronic device: receiving the attitude change values and the pressure change values; obtaining a corresponding manipulation instruction based on the attitude change values and the pressure change values; executing the manipulation instruction; and sending the result of executing the manipulation instruction back to the smart glasses.

[0006] Optionally, obtaining a corresponding control instruction based on the posture change value and the pressure change value includes: splicing the posture change value and the pressure change value to obtain a multi-dimensional control vector; inputting the multi-dimensional control vector into a pre-trained control action recognition model; determining a control action based on the output of the control action recognition model; and determining the control instruction according to the control action.

[0007] Optionally, obtaining a corresponding control instruction based on the posture change value and the pressure change value includes: determining a presumed motion mode of the smart glasses based on the posture change value; determining a presumed contact mode of the smart glasses based on the pressure change value; determining a control action according to a matching result of the presumed motion mode and the presumed contact mode; and determining the control instruction according to the control action.

[0008] Optionally, the plurality of motion sensors and the plurality of pressure sensors include motion sensors and pressure sensors that are cooperatively arranged within the smart glasses, and the plurality of motion sensors and the plurality of pressure sensors are respectively arranged at at least two of the following positions: at the left and right temple arms; at the left and right frames; and at the nose pad.

[0009] Optionally, the method further includes: on the smart glasses: obtaining deformation change values of the smart glasses detected by a plurality of strain sensors at positions of the plurality of strain sensors; sending the deformation change values to the electronic device; on the electronic device: obtaining the corresponding control instruction based on the posture change value, the pressure change value, and the deformation change value.

[0010] Optionally, the deformation change value is used to verify a control action corresponding to the control instruction, and the plurality of strain sensors are respectively arranged at at least two of the following positions: at the left and right temple arms; at the left and right hinges; and at the nose pad.

[0011] Optionally, the posture change threshold range and the pressure change threshold range are determined according to empirical values and / or user-defined values.

[0012] Optionally, the method further includes: on the electronic device: in response to a user selecting to enter a control action recognition adjustment page, the adjustment page including a prompt for performing a specific control action on the smart glasses; obtaining a custom posture change value for the motion sensors and a custom pressure change value for the pressure sensors returned by the smart glasses under the prompt; and determining the posture change threshold range and the pressure change threshold range for the user according to the custom posture change value and the custom pressure change value.

[0013] Optionally, determining the posture change threshold range and the pressure change threshold range according to the custom posture change value and the custom pressure change value includes: using the custom posture change value and the custom pressure change value as the input of the fine-tuning samples of a pre-trained manipulation action recognition model to generate a custom threshold range for this user; and obtaining the posture change threshold range and the pressure change threshold range for this user output by the manipulation action recognition model.

[0014] Thus, by integrating motion and pressure sensors in the smart glasses and cooperating with a flexible manipulation action recognition algorithm, the present disclosure can quickly capture and accurately identify the changes caused by the user's hand manipulation of the glasses (for example, pushing with the hand), and realize the quick activation of the relevant control functions of the smart glasses. The user can set the mapping relationship between the manipulation action and the manipulation instruction in the smart glasses APP of the electronic device according to personal habits and preferences, making the interaction more personalized and adapting to the operation habits of different users. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] By describing the exemplary embodiments of the present disclosure in more detail in conjunction with the accompanying drawings, the above and other objects, features, and advantages of the present disclosure will become more apparent. Among them, in the exemplary embodiments of the present disclosure, the same reference numerals generally represent the same components.

[0016] Figure 1 An example of the structure of the smart glasses is shown.

[0017] Figure 2 A schematic flowchart of a smart glasses operation method according to an embodiment of the present disclosure is shown.

[0018] Figure 3 An example of setting multiple IMUs in the smart glasses is shown.

[0019] Figure 4 A schematic flowchart of obtaining a manipulation instruction with the participation of the cloud is shown.

[0020] Figure 5 A schematic structural diagram of a computing device that can be used to participate in implementing the above smart glasses operation method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] Preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art. In addition, it should be understood that the "first", "second", and "third" or similar expressions in the present disclosure are only for descriptive and distinguishing purposes, and should not be construed as indicating or implying the order of appearance or importance of the corresponding objects.

[0022] Smart glasses are a type of transmissive head-mounted device that can be used not only to display images but also to allow users to see the images behind the display screen. This feature is often used in the development of AR applications. Figure 1 An example of the structure of smart glasses is shown. As shown in the figure, smart glasses 100 are first of all a pair of glasses, so they can have a structure similar to that of conventional glasses, including lenses 110, frames 120, and temple arms 130. Further, smart glasses 100 can provide various "intelligent" functions that conventional glasses cannot provide, such as voice interaction, real-time translation, navigation, and audio-visual playback. For this purpose, smart glasses 100 also need to integrate relevant components required for intelligent computing devices. Specifically, the processing unit (for example, the main control chip) of smart glasses 100 can be arranged inside the temple arm 130, for example, at the position shown as 131 in the figure. The processing unit can perform various operations according to the obtained instructions, such as using left and right display units (not shown in the figure) to project left and right images onto the left and right lenses 110 for display respectively. In order to receive voice commands from the wearer, smart glasses 100 can also be equipped with microphones, such as the microphones 122 and 132 shown in the figure. Microphone 122 is set on the frame 120, and microphone 132 can be set on the temple arm 130 to pick up sounds from different directions. For example, microphone 132 is in a position where it is easy to obtain clear speech from the wearer to facilitate receiving oral instructions from the wearer, and microphone 122 can be used to pick up the voices of others talking to the wearer. Correspondingly, smart glasses 100 can also include a speaker 133 for playing voice feedback to the wearer or the specified sound content, such as playing music. In order to provide a better sound effect to the wearer, speakers 133 can be provided at the positions of the left and right temple arms close to the ears as shown in the figure.

[0023] One or more cameras can also be provided on smart glasses 100, such as cameras integrated in the center and on both sides of the frame (not shown in the figure). The camera in the center of the frame is mainly used to capture the images from the user's perspective and is suitable for daily photography and video recording; the cameras on both sides of the frame are helpful for functions such as environmental perception, object recognition, and AR interaction.

[0024] As a computing device, in addition to including the processing unit 131 shown in the figure, the smart glasses 100 should also include a storage unit and a communication unit (not shown in the figure).

[0025] The processing unit 131 can be a processor or can include multiple processors. In some embodiments, the processing unit 131 can include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), and so on. In some embodiments, at least part of the processing unit 131 can be implemented using custom circuits, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0026] The storage unit can include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, the ROM can store static data or instructions required by the processing unit or other modules of the computing device. The permanent storage device can be a readable and writable storage device. The permanent storage device can be a non-volatile storage device that does not lose the stored instructions and data even when the computing device is powered off. The system memory can be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. The system memory can store some or all of the instructions and data required by the processing unit during operation. Executable code is stored on the storage unit, and when the executable code is processed by the processing unit, it can cause the processing unit to execute various methods required to implement the functions related to the smart glasses.

[0027] The communication unit can be used for wireless communication with the outside world. For example, the smart glasses can interact with an electronic device (such as a smartphone) installed with a smart glasses APP through the communication unit to implement all its functions. By using the communication function of the electronic device, support from the cloud can also be obtained to implement the functions related to the smart glasses.

[0028] It should be understood that although Figure 1 a specific frame glasses shape is shown, in other embodiments, the smart glasses can also have a frameless design. At this time, the components originally arranged in the frame can be arranged at other positions of the glasses. In some other embodiments, the smart glasses can also include other components outside the glasses main body structure, such as an image projection device attached to the outside of the lens, a detachable additional lens with myopia / hyperopia degrees located inside the lens, or a detachable in-ear earplug, etc. The present disclosure does not limit the specific implementation form of the smart glasses here.

[0029] In the prior art, users can usually interact with devices through touch, voice, and gesture recognition to achieve navigation, real-time translation, shooting records, and auxiliary operations in other professional fields. However, there are still some problems with these existing interaction methods. For example, the interaction area of touch interaction is usually very limited, which causes inconvenience in operation. One existing solution is to wear additional touch accessories (such as touch rings), but this increases the cost and makes the use of smart glasses troublesome. The recognition rate of voice interaction is limited in noisy environments, and there is also a risk of privacy leakage. Gesture recognition interaction based on visual recognition (such as camera shooting) has poor stability and high overhead in complex environments. In addition, these existing interaction methods are quite power-consuming and not intuitive and convenient to operate.

[0030] For this reason, the present disclosure provides a smart glasses operation method and an electronic device. By integrating multiple motion and pressure sensors, the smart glasses enable users to control the smart glasses through natural manipulation actions, thereby enhancing the interaction experience. Since the collaborative determination and matching of these sensor readings are required to trigger a manipulation instruction, misjudgment can be largely avoided. Further, since modern sensors, such as IMUs (Inertial Measurement Units) and certain high-precision pressure sensors, adopt a low-power mode, and the sensors trigger the interaction with the electronic device and the generation of manipulation instructions only when detecting a change value within a threshold range, the power consumption of this interaction method is very low.

[0031] Figure 2 FIG. shows a schematic flowchart of a smart glasses operation method according to an embodiment of the present disclosure. Figure 2 The shown operation method can be jointly executed by a smart glasses internally provided with multiple motion and pressure sensors and an electronic device paired with the smart glasses. By jointly judging these readings with the participation of the electronic device, a manipulation gesture is determined, and then the execution of a manipulation instruction and the corresponding operation of the smart glasses are triggered.

[0032] First, on the smart glasses side, a preliminary determination of the manipulation action can be made according to the threshold range. In step S210, the attitude change values of the smart glasses detected by multiple motion sensors are obtained, and in step S220, the pressure change values of the smart glasses detected by multiple pressure sensors at the positions of the multiple pressure sensors are obtained. The multiple motion sensors and the multiple pressure sensors are arranged inside the smart glasses. Among them, the motion sensors are usually arranged at positions where the movement of the glasses can be well detected, such as the nose pad, the frame, and the temple, while the pressure sensors are usually arranged at parts where the pressure needs to be directly measured, such as the bridge of the nose, the contact part between the temple and the ear, etc. Since the manipulation action on the glasses usually causes changes in both the attitude of the glasses and the pressure at the contact position at the same time, steps S210 and S220 are usually executed simultaneously or alternately.

[0033] The motion sensors are used to detect the motion state of the smart glasses. In the present disclosure, the motion sensors arranged in the smart glasses can be inertial motion sensors, which are used to detect the attitude changes of the smart glasses, such as changes in acceleration, angular velocity, and / or direction. In one embodiment, the motion sensor can especially be an IMU. An IMU is an integrated sensor device used to detect and measure the motion state of an object. That is, an IMU can be regarded as a motion sensor, but it is usually a measurement unit integrating multiple sensors. Specifically, an IMU usually consists of an accelerometer, a gyroscope, and an optional magnetometer (the accelerometer, the gyroscope, and the magnetometer respectively correspond to a kind of motion sensor). The accelerometer provides linear motion data, such as linear acceleration, for sensing tilt and gravity effects. The gyroscope provides rotational motion data, such as angular velocity, to make up for the insufficient dynamic performance of the accelerometer. The magnetometer is used to calibrate the direction deviation and determine the direction of the device relative to the earth's magnetic field, so as to provide accurate direction information. After being processed in the present disclosure (for example, through comprehensive calculation), these data can determine the three-dimensional motion trajectory and spatial attitude of the device, and further infer that the user has performed a specific manipulation action on the smart glasses.

[0034] A pressure sensor is a sensor that operates based on physical principles such as the piezoresistive effect and the piezoelectric effect. For example, in a piezoresistive pressure sensor, the internal resistance changes with the applied pressure. When pressure acts on the sensitive element of the sensor, the geometric shape and internal structure of the sensitive element change, resulting in a change in its resistance value. By measuring the change in resistance, the magnitude of the pressure can be obtained. The pressure sensor directly measures the magnitude of the pressure, and the unit is usually Pascal (Pa), kilopascal (kPa), etc. It focuses on the vertical force applied to the surface of the sensor. In smart glasses, when a user's finger presses on parts such as the nose bridge, for example, a pressure sensor located in the nose bridge or the nose pad can sense this pressure change (e.g., the pressure change caused by the user's finger pressing on the nose bridge or the pressure change caused by the nose pad making deeper contact with the user's nose bridge) and convert it into an electrical signal for output.

[0035] To achieve accurate judgment of multiple manipulation actions, multiple motion sensors can be set inside the smart glasses, such as multiple IMUs, and these IMUs can be arranged at different parts of the smart glasses. The pressure sensor can usually be set in cooperation with the motion sensor at this part (i.e., the pressure sensor is usually set near the IMU), and the readings measured by these sensors are used to jointly determine the occurrence of various manipulation actions.

[0036] Figure 3 An example of setting multiple IMUs inside the smart glasses is shown. For the convenience of explanation, in the Figure 3 example, other components arranged in the smart glasses frame are hidden, and only the IMUs set inside the smart glasses are shown (shown by thick long dashed boxes in the figure). As shown in the figure, an IMU 124 can be set at the right nose pad (optionally, another IMU 124 can be set at the left nose pad, not shown in the figure due to the viewing angle; or the IMU 124 can be set at the nose bridge), and it is mainly used to detect the user's manipulation actions near the nose bridge, for example, pushing up the nose bridge. The IMU 125 can also be set at the edge of the frame (for example, one is set at the left frame edge and one is set at the right frame edge), and it is mainly used to detect the user's manipulation actions near the frame edge, for example, pushing up or pushing the frame sideways. Two IMUs 135 can also be set in the two temple arms respectively (one is set in the left temple arm and one is set in the right temple arm), and it is mainly used to detect the user's manipulation actions on the temple arms, for example, pushing up or pushing the temple arms sideways. Correspondingly, multiple pressure sensors can also be set inside or on the surface of the smart glasses ( Figure 3(not shown). Multiple pressure sensors can be arranged in cooperation with the IMU, for example, arranged at the bridge of the glasses (e.g., in front of the bridge of the glasses) to detect the contact generated when the user pushes up the bridge of the glasses with a finger; two pressure sensors can also be respectively arranged at the left and right temple arms. In some embodiments, two more pressure sensors can also be arranged on the left and right frames. It should be understood that in different implementations, the motion sensors and the corresponding pressure sensors can also be at more, fewer, or different positions than Figure 3 shown to detect different manipulation actions.

[0037] The "posture change value" detected by the motion sensor can refer to the linear acceleration value measured by the accelerometer, the angular velocity value measured by the gyroscope, and optionally the direction value measured by the magnetometer; it can also refer to the change value of these values over time, for example, the change value compared to the previous measurement value, the change value compared to the measurement values within a previous period of time (e.g., 100 ms), etc. Similarly, the "pressure change value" detected by the pressure sensor can also refer to the pressure value or the change value of these state values over time.

[0038] The motion sensor and the pressure sensor can detect the posture change value and the pressure change value of the smart glasses in real time when the smart glasses are in the working state (e.g., the smart glasses are being worn by the user) and / or when the corresponding functions based on the measurements of the motion sensor and the pressure sensor are enabled (e.g., the smart glasses are being worn by the user and the function of controlling the smart glasses based on the manipulation action is enabled).

[0039] In the present disclosure, the determination of the manipulation action is made by the electronic device side based on the sensor readings. For this purpose, the smart glasses need to send the posture change value and the pressure change value to the electronic device. To avoid unnecessary transmission and power consumption, data transmission can be performed only when the sensor readings reach a certain intensity (i.e., reach a predetermined threshold range). Therefore, in step S230, it is determined that the posture change value reaches a posture change threshold range and / or it is determined that the pressure change value reaches a pressure change threshold range, and in step S240, the posture change value and the pressure change value are sent to the electronic device.

[0040] Determining that the posture change value reaches a posture change threshold range can mean that the posture change value is within the threshold range set for the posture change value, and determining that the pressure change value reaches a pressure change threshold range can mean that the pressure change value is within the threshold range set for the pressure change value. In some cases, transmission can be performed only when one of the posture or pressure change values is within the preset threshold range, while in other embodiments, only when both the posture change value and the pressure change value are within their respective preset threshold ranges, will the smart glasses trigger the transmission of the above change values.

[0041] In one embodiment, a pressure sensor arranged in cooperation with a motion sensor may be disposed at a position where contact is not or is not likely to be caused when the user normally wears the smart glasses. In this case, the reading of the pressure sensor exceeding a specific threshold (i.e., corresponding to the pressure change value being within a certain threshold range for the pressure change value) can clearly reflect that the smart glasses have made contact other than normal wearing at the part where the pressure sensor is installed (i.e., have made contact with an object other than the user's face, for example, have made contact with the user's finger).

[0042] For example, for the IMU 124 arranged in the nose pad, the corresponding pressure sensor may be arranged on the front or lower surface of the bridge of the nose. When the smart glasses are worn, the front and lower surfaces of the bridge of the nose usually do not contact the user's nose and are in a suspended state. Similarly, for the IMU 135 arranged at the temple, the corresponding strain gauge or pressure sensor may be arranged on the outer side of the temple. When the smart glasses are worn, the outer surface of the temple does not contact the user's head skin (although it may contact the user's hair, but the pressure generated by this contact is usually very small). Thus, when the pressure sensor detects a large enough reading, it can usually indicate that the corresponding position of the smart glasses has experienced contact from the outside (i.e., outside the head). At this time, the attitude change value and the pressure change value being within the threshold range may only mean that the pressure change value is within the threshold range for the pressure change value, that is, as long as the reading of the pressure sensor reaches a certain intensity, the smart glasses send the pressure change value and the attitude change value of the motion sensor at the corresponding position within the corresponding time period to the electronic device, and the electronic device makes a judgment on the control action for the pressure change value and the attitude change value.

[0043] Since the user's head position is usually changing during the wearing of the smart glasses, the attitude values detected by the IMU in real time are also constantly changing. That is, in the normal use state of the smart glasses, the IMU will always detect attitude changes (i.e., the collected linear acceleration values, angular velocity values, and optionally direction values are always changing). However, these normal changes in attitude values (such as those caused by the user walking) should not cause additional operations of the smart glasses, and only when these change values can indicate (or indicate with a relatively high probability) that the user has performed a certain control action, will the smart glasses perform the corresponding transmission operation. In the present disclosure, the occurrence of a control action determined on the smart glasses side is characterized by the attitude change value and the pressure change value falling within the threshold range (although, as will be described in detail below, the final determination of the control action and the generation of the corresponding control instruction are performed on the electronic device side, and this final determination also requires model reasoning in some embodiments).

[0044] Herein, a "manipulation action" refers to an action performed by a user on the smart glasses that can trigger a manipulation instruction. Which actions are supported for recognition is usually determined by the manufacturer of the smart glasses (and also requires the cooperation of motion and pressure sensors set at specific positions on the smart glasses), while which operation instruction each action corresponds to is usually selected by the user or determined by the user based on the system default settings. For example, the user can make settings on the manipulation action recognition page of the smart glasses APP installed on the electronic device, which will be described in detail below. In some embodiments, these manipulation actions are actions performed by the user's hand touching the smart glasses (usually causing the smart glasses to move). In other embodiments, these manipulation actions can also be coordinated actions of head actions and hand actions (for example, tilting the head in cooperation with pushing the frame of the glasses, etc.). In the present disclosure, in order to avoid misrecognition, it can be specified that the manipulation action requires the participation of the hand, and can be a touch action of the hand on the smart glasses, or a coordinated action of a head action and a hand touch action.

[0045] In one example, the smart glasses support the recognition of five actions, such as pushing up the nose bridge, pushing the left / right frame of the glasses, and pushing up the left / right leg of the glasses (herein, pushing the left frame of the glasses and pushing the right frame of the glasses can be recognized as the same or different actions, and the same applies to pushing up the left and right legs of the glasses). The user can select from these five actions supported by the system. For example, a certain user is used to pushing the glasses with the left hand, so the user selects to enable pushing up the nose bridge, pushing the left frame of the glasses, and pushing up the left leg of the glasses as manipulation actions, and associates these three actions with, for example, turning on the navigation, switching the display mode (for example, switching from AR to VR (virtual reality) display, or from VR to AR display), and triggering the music playing function, respectively.

[0046] After determining the manipulation actions, a threshold range of the attitude change value detected by the motion sensor corresponding to each manipulation action and a threshold range of the pressure change value detected by the pressure sensor at the corresponding pressure sensor can be determined. The significance of the threshold range is that when the state change value detected by the sensor falls within this threshold range, it can be determined on the smart glasses side that the user has a certain probability of performing the corresponding manipulation action.

[0047] In different embodiments, the threshold range of the attitude change value for a specific action can be a specific range of one or more attitude values of a certain IMU (or multiple IMUs), or a change range of one or more attitude values of a certain IMU (or multiple IMUs) within a predetermined time period, or a combination of both. Similarly, the threshold range of the pressure change value for the specific action can be a specific range of one or more pressure values of a certain pressure sensor (or multiple pressure sensors), or a change range within a predetermined time period, or a combination of both.

[0048] For example, if it is determined that a manipulation action is to push up the nose bridge, the threshold range of the corresponding attitude change value can be determined as follows: the linear acceleration value of the IMU 124 set inside the nose bridge (or inside the nose pad) has a change not less than a0 within a predetermined time range t0 to t1, and the measured pressure change value of the corresponding pressure sensor has a change not less than b0 within t0 to t1. If the IMU 124 detects a change in the linear acceleration value within the time of t i during which a i changes, where t i falls within the predetermined time range t0 to t1, and a i is not less than a0, and at the same time, the pressure sensor has a change in b i during the time of t i and b i is not less than b0, then the smart glasses can preliminarily determine that they have received the manipulation action corresponding to this threshold range (in this example, pushing up the nose bridge).

[0049] Corresponding threshold ranges can be set for different manipulation actions respectively. Each threshold range can correspond to a reasonable sensor reading range when the smart glasses receive this manipulation action. For example, the threshold range of the attitude change value corresponding to pushing up the nose bridge can be that the change amplitude of the linear acceleration of the IMU 124 in the vertical direction is greater than 5 m / s 2 , and the duration is between 0.2 seconds and 0.5 seconds. Another example is that the threshold range for pushing the frame sideways is that the change amplitude of the angular velocity of the IMU 125 in the horizontal direction is greater than 100 degrees per second, and the duration is between 0.1 seconds and 0.3 seconds. These threshold ranges can be set based on different principles in different embodiments. In some embodiments, these threshold ranges can be determined based on empirical values. Through a large number of experiments and data analysis, the sensor data ranges corresponding to different manipulation actions can be determined. For example, the action of pushing up the nose bridge usually causes a change in the linear acceleration of the IMU in a certain direction exceeding a certain value. In some other embodiments, these threshold ranges can be custom threshold ranges obtained by combining user input. The user can be allowed to perform a series of manipulation actions on the manipulation action recognition adjustment page, and the system automatically adjusts the recognition parameters to adapt to the individual operation habits. In some other embodiments, the determination of the threshold range can be based on model participation. Specifically, the manipulation action data executed by the user can be uploaded to a pre-trained manipulation action recognition model (usually arranged on the server) for fine-tuning to generate a custom threshold range for this user. In still some other embodiments, these threshold ranges can be obtained by comprehensively combining the model output value and the user input value. The generation and distribution of the threshold range will be described in detail below.

[0050] Regardless of how these threshold ranges are obtained, after determining the manipulation action, these threshold ranges can be stored locally in the smart glasses, and when the change values detected by the sensors fall within the threshold ranges, these posture change values and pressure change values are sent to the electronic device.

[0051] The electronic device is a device that is used in conjunction with the smart glasses and has the smart glasses APP installed, and is usually a smartphone. The realization of the functions of the smart glasses requires the cooperation of the electronic device. For example, to realize the navigation function of the smart glasses, it is necessary to start a navigation thread on the electronic device, query the route, and transmit the navigation information to the smart glasses to realize the real-time display of the navigation information on the smart glasses. Another example is that to realize the music playback function of the smart glasses, it is necessary to start a music playback thread on the electronic device and then transmit the audio to the speaker on the smart glasses for playback, and so on. The smart glasses operation method of the present disclosure also requires the participation of the electronic device.

[0052] Therefore, as Figure 2 shown, the smart glasses operation method of the present disclosure further includes steps S250 to S280 executed on the electronic device.

[0053] In step S250, the posture change value and the pressure change value are received on the electronic device. Subsequently, in step S260, a manipulation instruction is obtained based on the posture change value and the pressure change value from the smart glasses. Here, the "manipulation instruction" is an instruction executed by the electronic device to realize the function corresponding to the manipulation action. Thus, in step S270, the manipulation instruction is executed on the electronic device, and in step S280, the result of executing the manipulation instruction is sent back to the smart glasses. Thus, the smart glasses can receive the execution result returned by the electronic device and perform corresponding operations. For example, when the change values from the motion and pressure sensors at the nose bridge received on the electronic device are within the threshold range, it is necessary to perform a collaborative analysis on the posture change value and the pressure change value to determine the manipulation action of pushing up the nose bridge (that is, analyze the change values from these two types of sensors at the same time, and these two types of changes need to conform to a predetermined pattern or match in the occurrence time to determine the manipulation action of pushing up the nose bridge), and determine the manipulation instruction for displaying navigation content according to the previous settings of the user (that is, the user has previously corresponded the action of pushing up the nose bridge to the manipulation instruction for displaying navigation content). Specifically, the electronic device can execute the manipulation instruction, that is, start the navigation thread to obtain the navigation content to be displayed on the smart glasses, and send the navigation content as the execution result to the smart glasses, so that the navigation content can be displayed at a predetermined position on the lens of the smart glasses.

[0054] In one embodiment, the manipulation instruction can be determined in a relatively simple manner. At this time, obtaining a corresponding manipulation instruction based on the attitude change value and the pressure change value includes: determining a presumed motion mode of the smart glasses based on the attitude change value, determining a presumed contact mode of the smart glasses based on the pressure change value, determining a manipulation action according to the matching result of the presumed motion mode and the presumed contact mode, and determining a corresponding manipulation instruction according to the manipulation action. Further, determining the manipulation action according to the matching result of the presumed motion mode and the presumed contact mode may include: determining a reference motion mode corresponding to a specific manipulation action of the presumed motion mode; determining a reference contact position corresponding to the specific manipulation action of the presumed contact mode; determining that the presumed motion mode and the presumed contact mode match in time; and determining the specific manipulation action as the manipulation action.

[0055] The attitude change values sent to the electronic device may include a sequence of attitude change values each containing a timestamp, and the pressure change values sent to the electronic device include a sequence of pressure change values each containing a timestamp. The electronic device can determine a time-related presumed motion mode based on the sequence of attitude change values, determine a time-related presumed contact mode based on the sequence of pressure change values, and determine a manipulation action when the presumed motion mode conforms to the reference motion mode of a specific manipulation action, the presumed contact mode conforms to the reference contact mode of the specific manipulation action, and the presumed motion mode and the presumed contact mode match in time. For example, for the action of pushing up the nose bridge, the motion direction and amplitude detected by the IMU sensor should match the motion mode of the glasses caused by an upward thrust on the nose bridge, while the pressure change is mainly concentrated in the nose bridge area. For example, the pressure peak appears at the position where the nose bridge contacts the finger skin (consistent with the reference contact mode of the manipulation action of pushing up the nose bridge), and the overall motion of the glasses shows an upward rotation trend centered on the nose bridge (consistent with the reference motion mode of the manipulation action of pushing up the nose bridge), and the time point when the pressure peak appears matches the time point of the change in the motion state of the glasses, then it can be determined that the smart glasses receive the manipulation action of pushing up the nose bridge, determine the corresponding manipulation instruction and execute it.

[0056] Thus, by integrating a motion sensor in the smart glasses to detect the displacement and acceleration changes of the smart glasses in physical space and integrating a pressure sensor to detect physical contacts at corresponding positions, the user can trigger preset functions through specific manipulation actions (such as gestures like pushing or tilting the glasses). In a specific implementation, sensor data is sent to the electronic device when the change values detected by the sensors fall within a predetermined threshold, thereby realizing the recognition of the user's manipulation action and the execution of the corresponding manipulation instruction under the final judgment of the electronic device, making the user's manipulation of the smart glasses more natural and convenient.

[0057] The present disclosure realizes the detection of user manipulation actions by cleverly and coordinately arranging motion and pressure sensors in the frame structure of the smart glasses. The specific arrangement can be as described above Figure 3 shown, at key parts such as the temple, the frame, and the nose bridge (or nose pad). The IMU at the temple can better detect the rotation and tilt of the head, the IMU on the frame can capture the translation and rotation actions of the glasses, and the IMU at the nose bridge (or nose pad) can sense the upward rotation trend centered on the nose bridge. The pressure sensors or strain gauges at the corresponding positions can sense the pressure changes caused by the contact of the user's hand at the corresponding positions.

[0058] High-precision and low-power MEMS (Micro-Electro-Mechanical System) type IMUs can be selected, such as multi-axis MEMS IMUs. These IMUs can include, for example, precise three-axis accelerometers, three-axis gyroscopes, and three-axis magnetometers. These IMUs can be cleverly embedded in the frame structure of the smart glasses to ensure that, without affecting the appearance design and wearing comfort of the glasses, the motion state of the glasses in three-dimensional space can be monitored comprehensively, and it is ensured that the position, attitude, and motion state of the glasses can be accurately monitored in real time. Similarly, MEMS pressure sensors can be selected to achieve high-precision and low-power pressure sensors.

[0059] On the smart glasses side, the raw data collected by the motion sensors (such as IMUs) can go through certain preprocessing steps, such as filtering and noise reduction and feature extraction steps, to obtain gesture feature information. Specifically, algorithms such as Kalman filtering and complementary filtering can be used to remove noise and interference in the sensor data and improve the accuracy and stability of the data. Subsequently, gesture feature information can be extracted from the filtered data. Gesture feature information refers to the quantitative features that can characterize specific gestures extracted from the IMU sensor data. The features can include the amplitude of the acceleration, the rate of change of the angular velocity, the change in the direction of the acceleration, etc. The acceleration features can include, for example, the amplitude of the acceleration, the change in the direction of the acceleration, the integral of the acceleration (displacement), etc. For example, the action of pushing up the nose bridge may cause a peak in the acceleration of the IMU in the vertical direction. The angular velocity features can include the amplitude of the angular velocity, the change in the direction of the angular velocity, etc. For example, the action of rotating the glasses will cause an obvious change in the angular velocity of the IMU on a certain axis. The comprehensive features can combine the information of the acceleration and the angular velocity, such as the correlation between the acceleration and the angular velocity, the synchronous change of the acceleration and the angular velocity, etc. These features can more accurately describe the dynamic characteristics of the gesture. For example, for the action of pushing up the nose bridge, the features that may be extracted include the change amplitude of the linear acceleration value of the IMU at the nose bridge within a specific time, and the direction of the acceleration (such as upward). In some embodiments, the extracted gesture feature information (for example, the following ax i ,…,mz i)It can be spliced with pressure characteristic information (e.g., p1, p2, p3 below) for subsequent determination of manipulation actions based on the manipulation action recognition model.

[0060] When the smart glasses initially determine that a preset manipulation action has been received based on the readings falling within the threshold range, the sensor readings can be sent to the electronic device via wireless communication means such as Bluetooth or Wi-Fi. For example, it can be sent to the smart glasses APP installed on the smartphone. To ensure stable signal transmission and low latency, efficient data compression and encryption algorithms can be adopted to improve communication efficiency.

[0061] After receiving the sensor readings, the smart glasses APP can make a final judgment based on these sensor readings. When it determines that the sensor readings correspond to a preset gesture, it determines the manipulation execution according to the preset gesture, executes the manipulation instruction and returns the corresponding execution result to the smart glasses. For example, if the user pushes the glasses upward, after the smart glasses APP receives the sensor readings and determines the corresponding manipulation action, it will automatically launch the navigation application on the mobile phone and send the corresponding execution result to the smart glasses to display the navigation information on the display screen of the smart glasses. If the user pushes the glasses sideways, the APP will switch the display mode, such as switching from the augmented reality mode to the virtual reality mode. These displays can adopt smooth transitions and animation effects to enhance the smoothness and naturalness of the user experience. To prevent mis-triggering, a certain gesture confirmation mechanism can also be set, such as requiring the user to repeat the same gesture action within a short time to trigger the function or making a determination via a deep learning model, etc.

[0062] For the final determination of whether the user has performed a certain manipulation action, in some embodiments, the participation of a pre-trained model can also be included. For the convenience of explanation, Figure 4 The schematic flowchart of obtaining the manipulation instruction with the participation of the model is shown.

[0063] When the attitude change value and the pressure change value of the smart glasses fall within the threshold range, the smart glasses send these sensor data (i.e., the attitude and pressure change values of the smart glasses) to the electronic device. After receiving these sensor data, the electronic device can have the model participate in the judgment of the manipulation action. For example, it is performed by a pre-trained manipulation action recognition model arranged on the manipulation action recognition server. At this time, steps S4601 to S4604 can be regarded as Figure 2 A specific implementation of the shown step S260.

[0064] After the electronic device receives the attitude change value and the pressure change value, the electronic device can obtain the multi-dimensional vector as the input of the cloud model based on these change values in step S4601. In Figure 4 In the shown example, the attitude change value and the pressure change value can be spliced to obtain the multi-dimensional manipulation vector.

[0065] For example, it can be assumed that there are 5 IMUs (left and right temple arms, left and right frames, and nose pads) and 3 pressure sensors (left and right temple arms, nose pads) in the smart glasses, and the data of each sensor can form a vector. For example, each IMU can output acceleration in three directions (i.e., x, y, and z directions), angular velocity in three directions, and magnetic field data in three directions, totaling 9-dimensional data; while each pressure sensor outputs a pressure value, totaling 1-dimensional data.

[0066] Therefore, the data of all sensors can form a 5×9 + 3 = 48-dimensional vector. This vector can be expressed as:

[0067] v = [ax1, ay1, az1, ωx1, ωy1, ωz1, mx1, my1, mz1, …, p1, p2, p3]

[0068] where ax i , ωx i , mx i respectively represent the acceleration, angular velocity, and magnetic field data of the i-th IMU in the x direction; ay i , ωy i , my i respectively represent the acceleration, angular velocity, and magnetic field data of the i-th IMU in the y direction; az i , ωz i , mz i respectively represent the acceleration, angular velocity, and magnetic field data of the i-th IMU in the z direction; p i represents the pressure value of the i-th pressure sensor.

[0069] After obtaining the multi-dimensional manipulation vector, in step S4602, the electronic device may input the multi-dimensional manipulation vector into a pre-trained manipulation action recognition model. Since the recognition model has been pre-trained, it has the ability to reason based on the input multi-dimensional manipulation vector to identify the corresponding manipulation action. Since the recognition model is usually arranged on the server side, in specific operations, the electronic device may upload the multi-dimensional manipulation vector to the manipulation action recognition server. Correspondingly, on the server side, the multi-dimensional manipulation vector can be received, and the pre-trained manipulation action recognition model on it can perform model reasoning and generate a model reasoning output. Then, in step S4603, the electronic device may determine the manipulation action based on the output of the manipulation action recognition model. For example, the manipulation action recognition model may be a classification model for outputting recognition results corresponding to different manipulation actions based on the input multi-dimensional manipulation vector, and the recognition results can be sent to the electronic device, so that the electronic device can determine the manipulation action corresponding to the multi-dimensional manipulation vector. Subsequently, in step S4604, the electronic device can determine the manipulation instruction according to the manipulation action. For example, according to the mapping relationship set by the user or the system default, it can be determined which manipulation instruction the manipulation action specifically corresponds to.

[0070] The manipulation action recognition model can be obtained by training a large number of manipulation action sample data based on machine learning and pattern recognition techniques. The training of the manipulation action recognition model is usually carried out on the server side. In some implementations, it may specifically include the following steps:

[0071] · Data collection: First, a large-scale gesture sample database can be constructed, and a large amount of sensor data when users perform different manipulation actions can be collected, covering various common manipulation actions and professional manipulation actions in specific application scenarios. Data augmentation techniques, such as rotation, scaling, translation, etc., can also be used to expand the sample data to improve the generalization ability of the model.

[0072] · Data preprocessing: Perform preprocessing operations such as filtering and normalization on the collected data.

[0073] · Feature engineering: Extract feature vectors from the preprocessed data, such as the above-mentioned 48-dimensional vector.

[0074] · Model Training: Use machine learning algorithms to train the feature vectors to identify different manipulation actions. In some embodiments, algorithms such as Support Vector Machine (SVM), Random Forest, etc. can be used to classify and identify different gestures. In other embodiments, more complex models can be adopted. For example, deep learning algorithms such as Deep Convolutional Neural Network (CNN) or Long Short-Term Memory Network (LSTM) can be used to train the gesture sample data. By continuously adjusting the network structure and optimizing the hyperparameters, the model can learn the unique feature patterns of different gestures. For example, CNN can effectively extract the spatial features of gestures, while LSTM is suitable for processing time series data, such as the dynamic change process of gestures.

[0075] · Model Optimization: Optimize the model performance by adjusting the model parameters, increasing the training data, etc.

[0076] Furthermore, the attention mechanism and transfer learning technology can be introduced to improve the accuracy and speed of gesture recognition. The attention mechanism can make the model pay more attention to the key parts of the gesture, while transfer learning can utilize the model pre-trained on other related tasks to accelerate the training process of the gesture recognition model. Fuzzy logic and probability statistics methods can also be used to perform uncertainty analysis and decision fusion on the recognition results. When there is a certain degree of uncertainty in the sensor data, multiple factors are comprehensively considered, such as the likelihood of the gesture, historical operation habits, environmental factors, etc., and the final gesture recognition result is determined through methods such as Bayesian inference to improve the reliability of the system. The online learning algorithm can also be updated in real time to continuously optimize the gesture recognition model to adapt to the usage habits of different users and environmental changes.

[0077] In some implementations, a series of standard gesture actions can be defined, such as pushing up glasses, pushing glasses to the side, and gently shaking glasses, etc. In some embodiments, by analyzing and processing the acceleration, angular velocity, and magnetic field data collected by the IMU sensor, and simultaneously co-processing the pressure data collected by the pressure sensor, the feature vectors of the gesture are extracted and matched with the preset standard gesture features, thereby identifying the user's gesture action. The above data processing, gesture feature vector extraction, and matching with the standard gesture features can also be implemented on the server. In addition, the Dynamic Time Warping (DTW) algorithm can be introduced to solve the problems of differences in gesture speed and amplitude among different users and improve the robustness of gesture recognition. Specifically, the acceleration, angular velocity, and / or magnetic field data included in the continuously received attitude change values can be processed to obtain a manipulation action sequence; the manipulation action sequence is time-aligned with a reference manipulation action sequence; and the aligned manipulation action sequence is compared with the action module to identify the specific manipulation action made by the user.

[0078] In some embodiments, in addition to the pressure sensor, a plurality of strain gauges can also be arranged on the smart glasses. The working principle of the strain gauge is mainly based on the strain effect of metals. When a metal wire undergoes mechanical deformation under the action of an external force, its resistance value will change accordingly. A strain gauge is usually made of a very thin metal wire or semiconductor material and is pasted on the surface of the object to be measured. When the object to be measured is stressed and generates strain, the strain gauge will also deform accordingly, thereby causing a change in resistance. Different from the pressure sensor that directly measures the magnitude of pressure, the strain gauge measures the strain of the object, that is, the relative deformation amount of the object when it is stressed, which is dimensionless but is usually expressed in microstrain (με). The strain gauge indirectly reflects the force and stress conditions of the object by measuring the strain.

[0079] The pressure sensor usually needs to be set at the position where the pressure is directly measured. For example, in smart glasses, the pressure sensor is often installed at the nose bridge, the contact part between the temple and the ear, etc. Its function is to sense the pressure interaction between the user and the glasses, such as detecting whether the user presses the nose bridge and the magnitude of the pressing force, and providing pressure data for gesture recognition and operation judgment. The strain gauge is more suitable for installation at the positions where small deformations need to be detected, such as the key stress-bearing structures of the frame (for example, at the left and right temples, the left and right hinges, and / or the nose pads), and can detect the degree of small deformation of the frame caused by various user operations (such as wearing, adjusting, etc.). If the frame undergoes a small deformation due to stress, the strain gauge pasted on the frame can detect this deformation and convert it into a resistance change signal. By analyzing the strain data, the stress distribution and structural state of the frame can be understood, which helps to judge the user's operation actions, such as judging whether it is a push-up, a side push, or other complex operations that cause the frame to deform.

[0080] In smart glasses, the pressure sensor (especially some high-precision pressure sensors) can meet the application scenarios with strict requirements for pressure measurement and can accurately sense the pressure changes caused by a slight finger press. The sensitivity of the strain gauge usually depends on its material and manufacturing process, and generally can also detect very small strain changes. However, its accuracy is affected by various factors, such as the pasting process, environmental temperature, etc. However, in detecting small deformations of objects, the strain gauge has high resolution and sensitivity and can capture subtle deformation information of the frame. Therefore, in some embodiments, by additionally arranging the strain gauges, the detection of manipulation actions can be more accurately achieved.

[0081] At this time, the smart glasses operation method of the present disclosure may further include: obtaining, on the smart glasses, the deformation change values of the smart glasses at the positions of the plurality of strain sensors detected by the plurality of strain sensors; and sending the deformation change values to the electronic device; and on the electronic device: obtaining the corresponding manipulation instruction based on the attitude change value, the pressure change value, and the deformation change value.

[0082] In different embodiments, the deformation change values collected by the strain gauges can be utilized in different ways. In one embodiment, the deformation change values directly participate in the determination of manipulation actions. Specifically, it can be determined whether the deformation change values are within the range of the deformation change threshold, and the deformation change values, together with the attitude change values and the pressure change values, are sent to the electronic device. The electronic device can splice the deformation change values when constructing the multi-dimensional manipulation vector, so that the model inference also covers the deformation change values. In another embodiment, the deformation change values can be used as additional verification signals for further verification after a specific manipulation action is determined. At this time, the smart glasses side can directly send the deformation change values to the electronic device without performing threshold determination on the deformation change values. After the electronic device determines the manipulation action, it further determines whether the deformation change values fall within the range of the deformation change value threshold corresponding to the manipulation action, so as to conduct verification.

[0083] As described above, after determining the manipulation action, the corresponding function can be invoked at the electronic device (i.e., the execution of the manipulation instruction) and the generation of the execution result for sending to the smart glasses can be completed. Which function the manipulation action specifically corresponds to can be set by the user in the smart glasses APP installed on the electronic device. For this purpose, the method for operating the smart glasses executed by the electronic device according to the present disclosure may further include: responding to the user's selection to enter the manipulation action recognition setting page; in the manipulation action recognition setting page, in response to the manipulation function assigned by the user to the manipulation action, generating a mapping from the manipulation action to the manipulation instruction.

[0084] An intuitive and easy-to-use manipulation action recognition setting page can be designed and developed for the smart glasses APP. In this recognition setting page, the user can be allowed to perform personalized mapping of different actions to various functions of the smart glasses according to their own needs and preferences, so as to achieve the custom setting of the manipulation actions that can invoke the corresponding functions. For example, the user can set the action of pushing the glasses to open a specific application program, and set the action of rotating the glasses to adjust the display parameters, etc. in a simple and clear graphical configuration interface.

[0085] Furthermore, rich function customization options can also be provided in the manipulation action recognition setting page, so that the user can perform detailed parameter settings for each function. For example, the user can adjust the sensitivity, trigger threshold, response time, and animation effects of the manipulation action, etc., to adapt to different usage scenarios and personal habits. It can support the user to create and save multiple different function mapping configuration files for quick switching in different usage scenarios. It can also support the import and export of function mapping configuration files, so that the user can share their manipulation action settings with other users, or import the existing settings from other users, improving the usage convenience and personalization level.

[0086] Considering the differences in the operating habits and physical characteristics of different users, a personalized control action adjustment function is also developed. When first using or needing to adjust, users can perform a series of control actions on the control action recognition and adjustment page to allow the system to automatically adjust the recognition parameters to adapt to their individual operation methods. For example, according to the user's hand strength and movement amplitude, the trigger threshold of the control action is adjusted. To this end, the intelligent glasses operation method executed by the electronic device in the present disclosure may include: in response to the user selecting to enter the control action recognition and adjustment page, the adjustment page including prompts for performing specific control actions on the intelligent glasses; obtaining the custom attitude change value for the motion sensor and the custom pressure change value for the pressure sensor returned by the intelligent glasses under the prompts; and determining the attitude change threshold range and the pressure change threshold range for the user according to the custom attitude change value and the custom pressure change value.

[0087] In some embodiments, these custom attitude change values returned by the user under the prompts can be directly used to adjust the threshold range. In other embodiments, these custom attitude change values and custom pressure change values returned by the user under the prompts can be input into the control action recognition model, and the control action recognition model generates a custom threshold range for the user. At this time, determining the attitude change threshold range and the pressure change threshold range for the user according to the custom attitude change value and the custom pressure change value may include: using the custom attitude change value and the custom pressure change value as the input of the fine-tuning samples of the pre-trained control action recognition model (at this time, the label of the sample may be the corresponding specific control action) to generate a custom threshold range for the user; and obtaining the custom threshold range output by the control action recognition model. Specifically, the custom attitude change value and the custom pressure change value can be uploaded to the control action recognition server, and the recognition model arranged thereon processes them and the server issues the processing result. Thus, through the participation of the recognition model, a custom threshold range that better meets the user's needs can be obtained.

[0088] In actual operation, in order to improve the convenience of control action interaction and avoid mis-triggering, a real-time feedback mechanism can also be introduced. For example, after detecting the user's control action and successful recognition, real-time feedback can be provided to the user in various ways. For example, corresponding icons or prompt messages are displayed on the display screen of the glasses, or feedback is provided through a vibration motor, audio feedback, etc. In addition, in order to help the user correctly complete the control action, the user can also be allowed to obtain the system status information and operation prompts at any time during the execution of the control gesture. For example, AR technology can be used to superimpose virtual operation guides and feedback information on the display screen of the glasses.

[0089] In addition, the intelligent glasses operation solution based on manipulation actions in the present disclosure can also be deeply integrated with other interaction methods of the intelligent glasses, such as voice control, touch operation, etc., to achieve a richer and more diverse interaction experience. For example, while using manipulation actions for control, the user can perform further operations and adjustments through voice commands, or perform auxiliary operations through a touchpad, etc. For example, when the user raises the navigation function on the mobile phone by pushing up the nose bridge of the intelligent glasses and displays the navigation content on the lens of the intelligent glasses, the user can use a voice command to further clarify the navigation destination, thereby further improving the interaction convenience and operation friendliness of the intelligent glasses.

[0090] Furthermore, comprehensive performance optimization of the operation method in the present disclosure can also be carried out at the system level. For example, the power consumption of the sensor can be reduced, the data processing speed can be increased, and the communication delay can be reduced, etc. Specifically, low-power design techniques, such as dynamic voltage and frequency adjustment, sensor sleep mechanisms, etc., can be adopted to reduce the energy consumption of the system, especially on the intelligent glasses side. The algorithm and data structure can also be optimized to improve the efficiency of data processing and reduce the response time of the system.

[0091] In addition, in order to ensure privacy, strict security measures can be taken to ensure the confidentiality and integrity of the user's manipulation action data. For example, encrypt the transmission of sensor data, use secure communication protocols and encryption algorithms to prevent data from being stolen or tampered with. At the same time, set up user authentication and permission management mechanisms to restrict unauthorized access and operations.

[0092] Comprehensive system testing and verification of the implementation of the operation method in the present disclosure can also be carried out, including functional testing, performance testing, compatibility testing, security testing, etc. Use automated testing tools and test frameworks to strictly test each functional module of the system to ensure the quality and reliability of the system. User feedback and opinions can also be collected to continuously improve and optimize the performance and functions of the system.

[0093] Thus, the present disclosure provides a more natural and intuitive interaction method for the operation of intelligent glasses. By integrating motion and pressure sensors in the intelligent glasses and the cooperation of electronic devices, the user can quickly activate specific functions of the intelligent glasses through manipulation actions (for example, physical gestures for the intelligent glasses), thereby enhancing the user experience. This method can reduce the dependence on traditional interaction methods (such as the rotary wheel on the intelligent glasses or a separate touch ring), and provide a faster and more intuitive operation method. The manipulation actions can also be extended to specific movements or angles of the head to provide convenient interaction options for the user to adapt to specific application scenarios.

[0094] Figure 5FIG. shows a schematic structural diagram of a computing device that can be used to participate in implementing the above-mentioned smart glasses operation method according to an embodiment of the present invention. Refer to Figure 5 , the computing device 500 includes a memory 510 and a processor 520.

[0095] The processor 520 can be a multi-core processor or can include multiple processors. In some embodiments, the processor 520 can include a general main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), and so on. In some embodiments, the processor 520 can be implemented using custom circuits, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0096] The memory 510 can include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, the ROM can store static data or instructions required by the processor 520 or other modules of the computer. The permanent storage device can be a readable and writable storage device. The permanent storage device can be a non-volatile storage device that does not lose the stored instructions and data even when the computer is powered off. In some embodiments, the permanent storage device uses a mass storage device (such as a magnetic or optical disk, flash memory) as the permanent storage device. In some other embodiments, the permanent storage device can be a removable storage device (such as a floppy disk, optical drive). The system memory can be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. The system memory can store some or all of the instructions and data required by the processor during operation. In addition, the memory 510 can include any combination of computer-readable storage media, including various types of semiconductor storage chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and magnetic disks and / or optical disks can also be used. In some embodiments, the memory 510 can include a removable storage device that is readable and / or writable, such as a compact disc (CD), a read-only digital versatile disc (such as a DVD-ROM, a dual-layer DVD-ROM), a read-only Blu-ray disc, a high-density optical disc, a flash memory card (such as an SD card, a mini SD card, a Micro-SD card, etc.), a magnetic floppy disk, and so on. The computer-readable storage medium does not include carrier waves and instantaneous electronic signals transmitted wirelessly or wired.

[0097] An executable code is stored on the memory 510. When the executable code is processed by the processor 520, it can cause the processor 520 to execute the above-mentioned smart glasses operation method. When executing Figure 2 the method shown, the computing device 500 can correspond to the smart glasses or an electronic device, especially a smart phone installed with a smart glasses APP.

[0098] The method for operating smart glasses according to the present disclosure and the computing device using the method have been described in detail above with reference to the accompanying drawings. By integrating motion and pressure sensors in the smart glasses and cooperating with a flexible manipulation action recognition algorithm, the present disclosure can accurately capture and identify the displacement, angle, and acceleration changes generated by the user through manipulating the glasses (e.g., pushing with hands), thereby quickly invoking the control function of the glasses. The user can set the mapping relationship between each manipulation action and a specific function in the smart glasses APP of the electronic device according to personal habits and preferences, making the interaction more personalized and adapting to the operation habits of different users.

[0099] In addition, the method according to the present disclosure can also be implemented as a computer program or a computer program product, which includes computer program code instructions for performing the above-described steps defined in the above method of the present disclosure.

[0100] Alternatively, the present disclosure can also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium), on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor is caused to execute each step of the above method according to the present disclosure.

[0101] Those skilled in the art will also understand that the various exemplary logical blocks, modules, circuits, and algorithm steps described in connection with the present disclosure can be implemented as electronic hardware, computer software, or a combination of both.

[0102] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the systems and methods according to multiple embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0103] The embodiments of the present disclosure have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, the practical application, or the improvement of the technology in the market, or to enable other ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for operating smart glasses, comprising: On said smart glasses: Acquire posture change values ​​of the smart glasses detected by multiple motion sensors; Acquire pressure change values ​​of the smart glasses detected by multiple pressure sensors at the positions of the multiple pressure sensors; Determining that the posture change value reaches a posture change threshold range and / or determining that the pressure change value reaches a pressure change threshold range; Sending the posture change value and the pressure change value to an electronic device; On the electronic device: receiving the posture change value and the pressure change value; Obtain a corresponding control instruction based on the posture change value and the pressure change value; executing the control instruction; as well as The result of executing the control instruction is sent back to the smart glasses.

2. The method of claim 1, wherein: A corresponding control instruction based on the posture change value and the pressure change value includes: splicing the posture change value and the pressure change value to obtain a multi-dimensional control vector; Inputting the multi-dimensional manipulation vector into a pre-trained manipulation action recognition model; Determining a manipulation action based on an output of the manipulation action recognition model; and The corresponding control instruction is determined according to the control action.

3. The method of claim 1, wherein: A corresponding control instruction based on the posture change value and the pressure change value includes: Determining an estimated motion mode of the smart glasses based on the posture change value; determining an estimated contact pattern of the smart glasses based on the pressure change value; determining a manipulation action according to a matching result between the estimated motion pattern and the estimated contact pattern; and The manipulation instruction is determined according to the manipulation action.

4. The method of claim 1, wherein: The plurality of motion sensors and the plurality of pressure sensors include motion sensors and pressure sensors that are cooperatively arranged in the smart glasses, and the plurality of motion sensors and the plurality of pressure sensors are respectively arranged at at least two of the following positions: At the left and right temples; On the left and right mirror frames; and Nose pads.

5. The method of claim 1, further comprising: On said smart glasses: Acquire deformation change values ​​of the smart glasses at the positions of the multiple strain sensors detected by the multiple strain sensors; sending the deformation change value to the electronic device; On the electronic device: The corresponding control instruction is obtained based on the posture change value, the pressure change value and the deformation change value.

6. The method of claim 5, wherein: The deformation change value is used to verify the manipulation action corresponding to the manipulation instruction, and the plurality of strain sensors are respectively arranged at at least two of the following positions: At the left and right temples; At the left and right hinges; and Nose pads.

7. The method of claim 1, wherein: The posture change threshold range and the pressure change threshold range are determined according to empirical values ​​and / or user-defined values.

8. The method of claim 7, comprising: On the electronic device: In response to a user selecting to enter a manipulation action recognition adjustment page, wherein the adjustment page includes a prompt for performing a specific manipulation action for the smart glasses; Acquire the custom posture change value for the motion sensor and the custom pressure change value for the pressure sensor returned by the smart glasses under the prompt; and The posture change threshold range and the pressure change threshold range for the user are determined according to the custom posture change value and the custom pressure change value.

9. The method of claim 8, wherein: Determining the posture change threshold range and the pressure change threshold range according to the custom posture change value and the custom pressure change value includes: Using the custom posture change value and the custom pressure change value as inputs of a fine-tuning sample of a pre-trained manipulation action recognition model to generate a custom threshold range for the user; and The posture change threshold range and the pressure change threshold range for the user output by the manipulation action recognition model are obtained.

Images