Control method based on ar device, and device and storage medium

Abstract

A control method based on an AR device, and a device and a storage medium. The control method based on an AR device comprises: displaying at least one electronic device within the field of vision of an AR device (S301), wherein the electronic device establishes a communication connection with the AR device; and acquiring a hand action of a user when the user is intended to operate the electronic device, and controlling the electronic device on the basis of the hand action of the user (S302).

Description

Control method, device and storage medium based on AR device

[0001] This disclosure claims priority to Chinese patent application No. 202311259637.8, filed on September 26, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present disclosure relates to the field of augmented reality technology, and in particular to a control method, device, and storage medium based on an AR device. Background Art

[0003] With the development of the Internet of Things (IoT), smart homes are becoming increasingly integrated into our daily lives. Smart homes integrate electronic devices related to daily life into a centralized, intelligently controlled management system, providing convenience for users' daily use.

[0004] Currently, the control of electronic devices in smart homes is generally based on proprietary applications (APPs) provided by electronic device manufacturers. Users control electronic devices by operating on the proprietary user interface (UI) of the proprietary APP.

[0005] Summary of the Invention

[0006] In one aspect, a control method based on an AR device is provided. The method includes:

[0007] Display at least one electronic device within the field of view of the AR device; the electronic device establishes a communication connection with the AR device;

[0008] Acquire the user's hand movements when they intend to operate the electronic device, and control the electronic device according to the user's hand movements.

[0009] In another aspect, an electronic device is provided, which includes a display module and an acquisition module.

[0010] The display module is used to display at least one electronic device within the field of view of the AR device; the electronic device establishes a communication connection with the AR device;

[0011] The acquisition module is used to acquire the hand movements of the user when he / she intends to operate the electronic device, and control the electronic device according to the hand movements of the user.

[0012] In some embodiments, having an operational intention includes at least one of the following: the position of the user's hand coincides with the position of the electronic device within the field of view; the user's line of sight is directed towards the electronic device; the user's hand movement is consistent with the preset movement of the electronic device; the user's brain waves match the corresponding brain waves preset by the electronic device.

[0013] In some embodiments, the control module is used to control the electronic device to turn on or off when it is detected that the hand movement is the first type of operation.

[0014] In some embodiments, the first type of operation is a click operation or a rotation operation.

[0015] In some embodiments, when the electronic device is in the on state, a click operation or a rotation operation in a first preset direction is used to control the electronic device to turn off; when the electronic device is in the off state, a click operation or a rotation operation in a second preset direction is used to control the electronic device to turn on; the first preset direction is opposite to the second preset direction.

[0016] In some embodiments, the control module is used to adjust the operating parameters of the electronic device when the electronic device is turned on and the hand motion detected is the second type of operation.

[0017] In some embodiments, the second type of operation is a sliding operation, and the sliding mode corresponding to the sliding operation is horizontal sliding, vertical sliding or clockwise sliding.

[0018] In some embodiments, sliding operations of different sliding modes correspond to adjusting different operating parameters of the electronic device.

[0019] In some embodiments, the display module is also used to display an operation interface in the surrounding area of ​​the electronic device within the field of view; the operation interface includes multiple controls; and the control module is used to control the electronic device based on the control command corresponding to the target control when a hand movement is detected to operate a target control among the multiple controls.

[0020] In some embodiments, the hand motion includes multiple gestures; the control module is used to control the electronic device according to the user's hand motion when it is detected that the holding time of the last gesture among the multiple gestures reaches a preset time.

[0021] In some embodiments, the display module is further used to display the current switch state of the electronic device and / or the current operating parameters of the electronic device.

[0022] In some embodiments, at least one electronic device is highlighted within the field of view.

[0023] In some embodiments, the highlighting method is any one of the following: a stroke prompt method, a semi-transparent cover layer prompt method, and a highlight prompt method.

[0024] In another aspect, an electronic device is provided. The electronic device includes: a processor and a memory for storing instructions executable by the processor; the processor is configured to execute the instructions so that the electronic device performs the control method based on the AR device described in any of the above embodiments.

[0025] On the other hand, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the control method based on the AR device described in any of the above embodiments is implemented.

[0026] On the other hand, a computer program product is provided, which includes computer program instructions, and when the computer program instructions are executed by a processor, the control method based on the AR device described in any of the above embodiments is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] To more clearly illustrate the technical solutions of the present disclosure, the following briefly introduces the drawings required for use in some embodiments of the present disclosure. Obviously, the drawings described below are only drawings of some embodiments of the present disclosure, and those skilled in the art can also derive other drawings based on these drawings.

[0028] FIG1 is a schematic diagram of an application scenario according to some embodiments of the present disclosure.

[0029] FIG2 is a schematic diagram of the composition of an AR device according to some embodiments of the present disclosure.

[0030] FIG3 is a flow chart of a control method based on an AR device according to some embodiments of the present disclosure.

[0031] FIG4 is a schematic diagram of a highlight display according to some embodiments of the present disclosure.

[0032] FIG5 is a schematic diagram of a control process of an electric lamp according to some embodiments of the present disclosure.

[0033] FIG6 is a schematic diagram of a control process of an oven according to some embodiments of the present disclosure.

[0034] FIG7 is a schematic diagram of adjusting operating parameters of a lamp according to some embodiments of the present disclosure.

[0035] FIG8 is a schematic diagram of an operation interface according to some embodiments of the present disclosure.

[0036] FIG9 is a schematic diagram of a temperature adjustment process according to some embodiments of the present disclosure.

[0037] FIG10 is a schematic diagram of a form of misidentification according to some embodiments of the present disclosure.

[0038] FIG11 is a schematic diagram showing the composition of an electronic device according to some embodiments of the present disclosure.

[0039] FIG12 is a schematic structural diagram of an electronic device according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0040] To help those skilled in the art better understand the technical solutions of the embodiments of the present disclosure, the technical solutions of the present disclosure will be clearly and completely described below in conjunction with the drawings in the present disclosure. Obviously, the embodiments described are only some of the embodiments of the present disclosure, not all of them. Based on the embodiments of the present disclosure, all other embodiments obtained by those of ordinary skill in the art without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0041] It should be noted that in this disclosure, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this disclosure as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner.

[0042] In the following, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the technical features being referred to. Therefore, a feature defined with the terms "first," "second," etc., may explicitly or implicitly include one or more of such features.

[0043] In the description of this disclosure, unless otherwise specified, " / " means "or." For example, A / B can mean A or B. "And / or" herein is merely a description of an association relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: only A, only B, and both A and B. Furthermore, "at least one" means one or more, and "a plurality" means two or more.

[0044] Currently, the control of electronic devices in smart homes is generally based on proprietary applications (APPs) provided by electronic device manufacturers. Users control electronic devices through the proprietary user interface (UI) of the proprietary APP. Current mobile phone terminals themselves have communication functions based on infrared remote control, wireless fidelity (WIFI) protocol, Bluetooth protocol, etc. Taking infrared remote control as an example, a proprietary APP can be used to build in infrared remote control codes for as many electronic devices as possible, so that users can control different electronic devices through their mobile phone terminals. Alternatively, through alliances jointly established by various smart home gateways or different manufacturers, universal protocols can be used to a certain extent to achieve the control of different smart home electronic devices using a single remote control or proprietary APP.

[0045] However, this control method is not convenient and requires users to be familiar with the proprietary app's user interface, which places a high barrier to entry. Furthermore, with the increasing variety and number of electronic devices in smart homes, which may come from different manufacturers, users may have to familiarize themselves with the proprietary user interfaces provided by multiple manufacturers, further raising the barrier to entry.

[0046] With the rapid development of AR technology, more and more users will come into contact with and use AR devices in their daily lives. Furthermore, as AR devices become more widespread and lightweight, they will become the next evolutionary trend and technological replacement for mobile phones in the near future. This trend will provide users with a wider display space, a more real-time online experience for receiving and sending information and commands, and will make them ever-connected users of the future. In the AR era, user interaction methods will undergo new changes, resulting in more convenient and diverse operation methods.

[0047] Based on this, the embodiments of the present disclosure provide a control method based on AR devices, which utilizes the AR technology's ability to recognize user hand movements and detect the relative positions of different objects in space, and uses computer vision and other detection methods to detect and judge in real time the electronic devices and target movements that the user wants to control, allowing the user to directly turn on and off electronic devices, and / or adjust the operating parameters of electronic devices without opening more complex APPs and more dedicated user interfaces.

[0048] Referring to Figure 1 , which is a schematic diagram of an application scenario involved in an embodiment of the present disclosure, the application scenario includes an AR device 101 and at least one electronic device 102 (three electronic devices are shown as an example, but this is not intended to be limiting). The AR device can be connected to at least one electronic device for communication.

[0049] The AR device may be AR glasses, for example, monocular glasses or binocular glasses.

[0050] The electronic device may be a smart device with communication connection functions, including but not limited to infrared remote control, Wi-Fi, Bluetooth, Zigbee, etc. For example, the electronic device may be a smart light, a smart air conditioner, a smart oven, a television, etc.

[0051] In some embodiments, as shown in FIG1 , when a user wears an AR device and looks at an electronic device, the AR device can determine that the user wants to operate the electronic device, recognize the user's hand movements, determine the corresponding operation instructions, and send the operation instructions to the electronic device to achieve contactless control.

[0052] It should be noted that the application scenarios described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure, and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Ordinary technicians in this field can know that with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.

[0053] Figure 2 is a schematic diagram of the components of an AR device provided by an embodiment of the present disclosure. As shown in Figure 2, the AR device includes an IoT device management module, a user gesture operation detection module, an AR module, and a central control module.

[0054] The IoT device management module is used to detect electronic devices in the surrounding environment and establish communication connections with the electronic devices to send instructions to the electronic devices for control. The user gesture operation detection module can perform image recognition processing based on the pictures taken by the external field of view camera of the AR device to determine the user's operation intention. The AR module is used to provide related functions of AR display, such as displaying the current operating parameters of the electronic device, displaying the operation interface of the electronic device, etc. The central control module is used to coordinate the direct interaction between the above three modules.

[0055] Figure 3 is a flow chart of a control method based on an AR device provided by an embodiment of the present disclosure. For example, the control method based on an AR device provided by the present disclosure can be applied to the network architecture shown in Figure 1, and specifically to the AR device shown in Figure 2.

[0056] As shown in FIG3 , the control method based on the AR device provided in the present disclosure may include the following S301 - S302 .

[0057] S301: Display at least one electronic device within the field of view of the AR device.

[0058] The electronic device establishes a communication connection with the AR device.

[0059] In some embodiments, when a user wears an AR device and turns it on, the AR device can display at least one electronic device within the user's field of view and with which they can establish a communication connection. Display here primarily refers to the AR device presenting the electronic device within the user's field of view to the user, either in real-time preview or after processing.

[0060] For example, when the AR device is powered on, the IoT device management module in the AR device can search for connectable electronic devices in the surrounding environment and automatically establish a communication connection. Communication connection methods include but are not limited to infrared remote control, Bluetooth, WiFi, ZigBee, and ultra wideband (UWB) communication. Furthermore, at least one electronic device is displayed within the field of view.

[0061] In some embodiments, at least one electronic device is highlighted within the field of view.

[0062] In some embodiments, the above-mentioned highlighting method is any one of the following: by stroking prompts, by semi-transparent covering layer prompts, and by highlighting prompts. For example, Figure 4 is a schematic diagram of a highlighting provided by an embodiment of the present disclosure. Figure 4 takes a lamp and a table as an example, where the lamp is an electronic device that can be connected to communicate with an AR device, and provides the user with the ability to control the device in a highlighted manner. As shown in Figure 4, the lamp in (a) is highlighted in a stroking prompt, the lamp in (b) is highlighted in a semi-transparent covering layer prompt, and the lamp in (c) is highlighted in a highlighting prompt.

[0063] S302: Acquire hand movements of a user when the user intends to operate the electronic device, and control the electronic device according to the hand movements of the user.

[0064] In some embodiments, when a user uses an AR device, the external field of view camera on the AR device can capture the user's hand movements during use. After determining that the user has an intention to operate the electronic device, the electronic device can be controlled according to the user's hand movements. The AR device can use image recognition technology and a pre-trained gesture recognition model to identify the user's hand movements. In the process of controlling the electronic device, the AR device can convert the control instructions corresponding to the hand movements into the same protocol as the electronic device through the IoT device management module, and send the converted control instructions to achieve control of the electronic device.

[0065] In some embodiments, having an operational intention includes at least one of the following: the position of the user's hand coincides with the position of the electronic device within the field of view; the user's line of sight is directed towards the electronic device; the user's hand movement is consistent with the preset movement of the electronic device; the user's brain waves match the corresponding brain waves preset by the electronic device.

[0066] It is understandable that if the user needs to operate an electronic device, he will look at the electronic device. Since the external field of view camera on the AR device is close to the position of the human eye, the AR device can obtain the field of view content consistent with what the human eye sees. If the user needs to operate an electronic device, he can also move his hand so that the position of his hand coincides with the electronic device within the field of view. For example, the user can point to the electronic device with one finger, or use gestures such as two-finger selection to circle the electronic device to indicate the intention to operate the electronic device. Moreover, the user's hand movements need to comply with the preset actions of the electronic device. For example, the preset actions of an electronic device only support click operations, and the user performs a sliding operation on the electronic device. This may be an unconscious misoperation by the user, and the operation can be considered an invalid operation. The AR device can then determine that this situation is a situation where there is no intention to operate.

[0067] The above description is based on the example of an AR device identifying a user's more obvious operational intentions (e.g., user's line of sight, user's hand movements, etc.). It should be understood that with the development of artificial intelligence technology in the future, AR devices can also recognize users' "potential" or "implicit" operational intentions, thereby achieving control over electronic devices. For example, an AR device can recognize a user's brain waves. When the user's brain waves match the brain waves preset by the electronic device, it can be considered that the user currently has an operational intention for the electronic device.

[0068] Several implementations of the above S302 are described below in conjunction with specific embodiments and the accompanying drawings.

[0069] In some embodiments, controlling the electronic device according to the user's hand motion in S302 may be implemented as: controlling the electronic device to turn on or off when the hand motion is detected to be a first type of operation. That is, the first type of operation is an operation to control the electronic device to turn on or off.

[0070] For example, the first type of operation is a click / press operation or a rotation operation. When the electronic device is turned on, the click operation or the rotation operation in a first preset direction is used to control the electronic device to be turned off; when the electronic device is turned off, the click operation or the rotation operation in a second preset direction is used to control the electronic device to be turned on; the first preset direction is opposite to the second preset direction.

[0071] For example, Figure 5 is a schematic diagram of a light control process provided by an embodiment of the present disclosure. As shown in Figure 5(a), the light is off and the user's hand gesture when intending to operate the light is a click. The AR device then sends a control instruction to the light to turn it on. As shown in Figure 5(b), the light is on and the user's hand gesture when intending to operate the light is a click. The AR device then sends a control instruction to the light to turn it off.

[0072] For another example, Figure 6 is a schematic diagram of the control process of an oven provided by an embodiment of the present disclosure. In the figure, the oven is controlled by a knob. As shown in Figure 6 (a), the oven is in the off state. If the user rotates the knob clockwise (a first preset direction), the AR device sends a control instruction to the oven to turn it on. As shown in Figure 6 (b), the oven is in the on state. If the user rotates the knob counterclockwise (a second preset direction), the AR device sends a control instruction to the oven to turn it off.

[0073] In some embodiments, controlling the electronic device based on the user's hand motion in S302 may be implemented as: when the electronic device is turned on and the hand motion detected is a second type of operation, adjusting the operating parameters of the electronic device. That is, the second type of operation is an operation to adjust the operating parameters of the electronic device.

[0074] Exemplarily, the second type of operation is a sliding operation, and the sliding mode corresponding to the sliding operation is horizontal sliding, vertical sliding, or clockwise sliding (wherein the clockwise direction is clockwise or counterclockwise).

[0075] In addition, sliding operations in different sliding modes correspond to adjusting different operating parameters of the electronic device.

[0076] For example, Figure 7 is a schematic diagram of adjusting the operating parameters of a light provided by an embodiment of the present disclosure. When the light is on, it supports two types of operating parameter adjustments, one for adjusting the color of the light and the other for adjusting the brightness of the light. As shown in Figure 7 (a), the AR device detects that the user's finger makes a horizontal sliding motion, and can virtually display a color palette below the light, as well as the currently selected color adjustment position. The user can adjust the color of the light by sliding his finger horizontally on the virtually displayed color palette. As shown in Figure 7 (b), the AR device detects that the user's finger makes a vertical sliding motion, and can virtually display a brightness adjustment dial on the left side of the light, as well as the currently selected brightness position. The user can adjust the brightness of the light by vertically sliding his finger on the virtually displayed brightness adjustment section.

[0077] In some scenarios, if the operation functions of an electronic device are relatively complex, simple click and slide operations alone cannot realize all the functions of the electronic device. In this case, the AR device can assist the user in operation by displaying the operation interface.

[0078] Therefore, in some embodiments, after obtaining the user's hand motion when intending to operate the electronic device in S302 above, the control method provided by the embodiment of the present disclosure further includes: displaying an operation interface in the surrounding area of ​​the electronic device within the field of view, where the operation interface includes multiple controls. In the case of displaying the operation interface, controlling the electronic device based on the user's hand motion in S302 above can be implemented as follows: when detecting that the hand motion operates a target control among the multiple controls, controlling the electronic device based on the control command corresponding to the target control.

[0079] In summary, the above S302 can be described as: a. After obtaining the hand movement of the user when he has the intention to operate the electronic device, the operation interface is displayed in the surrounding area of ​​the electronic device within the field of view; b. When it is detected that the hand movement operates the target control among multiple controls, the electronic device is controlled based on the control command corresponding to the target control.

[0080] For example, Figure 8 is a schematic diagram of an operation interface provided by an embodiment of the present disclosure. Figure 8 takes the controlled electronic device as an example of a television, which has many operation functions, such as a power switch function, a channel selection function, a volume adjustment function, and a screen brightness adjustment function. Therefore, after detecting that there is an intention to operate the television, the AR device can virtually display an operation interface (in the form of an overlay) in the area around the television. The operation interface includes a power switch control, a channel selection control (including controls 1-9), a volume adjustment control, and a screen brightness adjustment control. For the channel selection function, if the AR device detects that the user clicks on the target control (for example, control 2), the AR device sends a control instruction to the television to control the television to switch to the channel corresponding to control 2. Similarly, if a user sliding operation on the volume adjustment control or the screen brightness adjustment control is detected, a corresponding instruction can be sent to the television to control the television to adjust the brightness or adjust the volume.

[0081] In some scenarios, the results of AR device control of electronic devices can be displayed within the field of view or not. For example, when adjusting the color or brightness of a light, the adjustment results are immediately fed back, so the AR device does not need to display the control results. For example, when adjusting the temperature of an air conditioner, due to the lag in ambient temperature changes, the AR device can immediately display the current adjustment results to inform the user of the current temperature setting, ensuring the user's perceptual experience.

[0082] Therefore, the control method based on the AR device provided in the embodiment of the present disclosure further includes the following operation: displaying the current switch device of the electronic device and / or the current operating parameters of the electronic device in the surrounding area of ​​the electronic device within the field of view. In this way, the current adjustment results can be fed back in a timely manner, improving the user's sensory experience.

[0083] For example, FIG9 is a schematic diagram of a temperature adjustment process provided by an embodiment of the present disclosure. The figure shows a temperature adjustment panel installed on the wall. When the user intends to operate the temperature adjustment panel, the user slides up and down with his fingers to adjust the indoor temperature. At the same time, an overlay is displayed around the temperature adjustment panel, which can display the specific value of the currently set temperature and the temperature range within a certain range around the currently set temperature. The specific data of the current device temperature (26°C in the figure) can be displayed in an enlarged form to serve as a prompt to the user.

[0084] It should be noted that, in the control method based on the AR device provided in the embodiment of the present disclosure, the user's hand movement is an air operation, and there is no specific contact object. Therefore, compared with the actual touch control of the electronic device, the AR device has a lower recognition accuracy for the end process of the user's hand movement. That is to say, after the user completes the parameter adjustment, the action of the finger leaving is easily misrecognized by the AR device. For example, Figure 10 is a schematic diagram of a form of misrecognition provided by the embodiment of the present disclosure. As shown in Figure 10, the user has completed the temperature adjustment, and the current adjusted target temperature is 26°C. Then, the user's finger leaves the temperature adjustment position, and during the process of leaving, the AR device mistakenly recognizes the hand movement when leaving as an action to continue adjustment, and further adjusts the current temperature to 25°C. This adjustment is obviously not in line with the user's wishes and will greatly affect the user's experience.

[0085] Based on the above situations that may cause erroneous operations, in the control method based on AR devices provided by the embodiments of the present disclosure, the user's hand movements in the above S302 include multiple gestures; the operation of the electronic device is controlled according to the user's hand movements in the above S302, which can be implemented as follows: when it is detected that the holding time of the last gesture among the multiple gestures reaches a preset time length, the electronic device is controlled according to the user's hand movements.

[0086] That is to say, when the AR device recognizes the user's hand movements, it can continuously capture pictures of multiple gestures. When the user completes the hand movement to control the electronic device and maintains a pause for a period of time (conscious or unconscious), it can be regarded as a confirmation of the control operation. That is, when the AR device recognizes that the duration of the last gesture reaches the preset duration, it means that the user has confirmed the current operation, and the AR device can send instructions to control the electronic device according to the current operation. In this way, precise control can be achieved to avoid misidentification and a bad user experience.

[0087] The disclosed embodiments provide a control method based on an AR device, wherein the method displays at least one electronic device within the field of view of the AR device, then obtains the user's hand movements indicating specific operational intentions for the electronic device, and then controls the operation of the electronic device based on the hand movements. Compared to some technologies that use proprietary apps and relatively proprietary user interfaces to control electronic devices, the present method uses the AR device to recognize the user's hand movements, enabling direct control of the electronic device without contact or a proprietary user interface, thereby lowering the operational threshold requirements for the user and providing the user with a smarter and more intuitive user interaction experience.

[0088] Furthermore, this solution determines whether the user intends to operate the electronic device by factors such as the user looking at the electronic device and the overlap of the hand position with the electronic device. After determining that the user intends to operate the electronic device, the electronic device is controlled according to the user's hand movements, which can effectively avoid the user's unconscious misoperation and ensure the accuracy of the control of the electronic device. In addition, considering the possible misidentification of contactless control, this method further improves the accuracy of controlling the electronic device based on the AR device by identifying the pause of the user's hand movement and treating the pause as a confirmation of the user's operation, thereby ensuring the user's interactive experience.

[0089] It is understandable that, in order to realize the above functions, the electronic device (the above-mentioned AR device) includes hardware structures and / or software modules corresponding to the execution of each function. It should be easy for those skilled in the art to realize that, in combination with the algorithm steps of each example described in the embodiments of the present disclosure, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0090] The embodiment of the present disclosure can divide the electronic device into functional modules according to the above-mentioned method embodiment. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one functional module. The above-mentioned integrated module can be implemented in the form of hardware or software. It should be noted that the division of modules in the embodiment of the present disclosure is schematic and is only a logical function division. There may be other division methods in actual implementation. The following is an example of dividing each functional module corresponding to each function.

[0091] FIG11 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure, which can execute the control method based on an AR device provided by the above method embodiment. As shown in FIG11 , the electronic device includes: a display module 1101 and a control module 1102.

[0092] The display module 1101 is used to display at least one electronic device within the field of view of the AR device; the electronic device establishes a communication connection with the AR device;

[0093] The control module 1102 is used to obtain the hand movements of the user when the user intends to operate the electronic device, and control the electronic device according to the hand movements of the user.

[0094] In some embodiments, having an operational intention includes at least one of the following: the position of the user's hand coincides with the position of the electronic device within the field of view; the user's line of sight is directed towards the electronic device; the user's hand movement is consistent with the preset movement of the electronic device; the user's brain waves match the corresponding brain waves preset by the electronic device.

[0095] In some embodiments, the control module 1102 is used to control the electronic device to turn on or off when the hand motion is detected to be the first type of operation.

[0096] In some embodiments, the first type of operation is a click operation or a rotation operation.

[0097] In some embodiments, when the electronic device is in the on state, a click operation or a rotation operation in a first preset direction is used to control the electronic device to turn off; when the electronic device is in the off state, a click operation or a rotation operation in a second preset direction is used to control the electronic device to turn on; the first preset direction is opposite to the second preset direction.

[0098] In some embodiments, the control module 1102 is configured to adjust operating parameters of the electronic device when the electronic device is turned on and the hand motion detected is a second type of operation.

[0099] In some embodiments, the second type of operation is a sliding operation, and the sliding mode corresponding to the sliding operation is horizontal sliding, vertical sliding or clockwise sliding.

[0100] In some embodiments, sliding operations of different sliding modes correspond to adjusting different operating parameters of the electronic device.

[0101] In some embodiments, the display module 1101 is also used to display an operation interface in the surrounding area of ​​the electronic device within the field of view; the operation interface includes multiple controls; the control module 1102 is used to control the electronic device based on the control command corresponding to the target control when a hand movement is detected to operate a target control among multiple controls.

[0102] In some embodiments, the hand movement includes multiple gestures; the control module 1102 is used to control the electronic device according to the user's hand movement when it is detected that the holding time of the last gesture among the multiple gestures reaches a preset time.

[0103] In some embodiments, the display module 1101 is further used to display the current switch state of the electronic device and / or the current operating parameters of the electronic device.

[0104] In some embodiments, at least one electronic device is highlighted within the field of view.

[0105] In some embodiments, the highlighting method is any one of the following: a stroke prompt method, a semi-transparent cover layer prompt method, and a highlight prompt method.

[0106] In the case of implementing the functions of the above-mentioned integrated modules in the form of hardware, the embodiment of the present disclosure provides another structure of the electronic device involved in the above-mentioned embodiment. As shown in Figure 12, the electronic device 120 includes: a memory 1201, a processor 1202, a communication interface 1203, and a bus 1204.

[0107] The memory 1201 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store dynamic information and instructions, an electrically erasable programmable read-only memory (EEPROM), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0108] The processor 1202 may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the embodiments of the present disclosure. The processor 1202 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array, or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. The processor 1202 may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the embodiments of the present disclosure. The processor 1202 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0109] The communication interface 1203 is used to connect to other devices via a communication network, such as Ethernet, wireless access network, or wireless local area network (WLAN).

[0110] In some embodiments, the memory 1201 may exist independently of the processor 1202 and may be connected to the processor 1202 via a bus 1204 for storing instructions or program codes. When the processor 1202 calls and executes the instructions or program codes stored in the memory 1201, the control method for the AR device provided in the embodiments of the present disclosure can be implemented.

[0111] In some embodiments, the memory 1201 may also be integrated with the processor 1202 .

[0112] Bus 1204 can be an Extended Industry Standard Architecture (EISA) bus, for example. Bus 1204 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, FIG12 shows only one thick solid line, but this does not mean that there is only one bus or only one type of bus.

[0113] In some embodiments, the memory 1201 stores executable instructions. When the processor 1202 executes the executable instructions, the electronic device executes the control method based on the AR device as described in any of the above embodiments.

[0114] Some embodiments of the present disclosure provide a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium), which stores computer program instructions. When the computer program instructions are executed on a computer, the computer executes the control method based on the AR device as described in any of the above embodiments.

[0115] Exemplarily, the above-mentioned computer-readable storage media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes, etc.), optical disks (e.g., compact disks (CDs), digital versatile disks (DVDs), etc.), smart cards, and flash memory devices (e.g., erasable programmable read-only memories (EPROMs), cards, sticks, or key drives, etc.). The various computer-readable storage media described in the present disclosure may represent one or more devices and / or other machine-readable storage media for storing information. The term "machine-readable storage medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0116] An embodiment of the present disclosure provides a computer program product including instructions. When the computer program product is run on a computer, the computer is enabled to execute the control method based on an AR device as described in any of the above embodiments.

[0117] The above is only a specific embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or replacements within the technical scope disclosed in the present disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A control method based on augmented reality (AR) equipment, comprising: Display at least one electronic device within the field of view of the AR device; The electronic device establishes a communication connection with the AR device; Acquire the hand movements of the user when he / she intends to operate the electronic device, and control the electronic device according to the hand movements of the user.

2. The method according to claim 1, wherein: The operation intention includes at least one of the following: The position of the user's hand coincides with the position of the electronic device within the field of view; The user's line of sight is directed toward the electronic device; The hand movement of the user conforms to the preset movement of the electronic device; or The user's brain waves match corresponding brain waves preset in the electronic device.

3. The method according to claim 1, wherein: Controlling the electronic device according to the hand movement of the user includes: When it is detected that the hand movement is a first type of operation, the electronic device is controlled to be turned on or off.

4. The method according to claim 3, wherein: The first type of operation is a click operation or a rotation operation.

5. The method according to claim 4, wherein: When the electronic device is in an on state, the clicking operation or the rotating operation in the first preset direction is used to control the electronic device to be turned off; When the electronic device is in a turned-off state, the clicking operation or the rotating operation in the second preset direction is used to control the electronic device to turn on; The first preset direction is opposite to the second preset direction.

6. The method according to claim 1, wherein: Controlling the electronic device according to the hand movement of the user includes: When the electronic device is in the turned-on state and when it is detected that the hand motion is the second type of operation, the operating parameters of the electronic device are adjusted.

7. The method according to claim 6, wherein: The second type of operation is a sliding operation, and the sliding mode corresponding to the sliding operation is horizontal sliding, vertical sliding or clockwise sliding.

8. The method according to claim 7, wherein: The sliding operations in different sliding modes correspond to adjusting different operating parameters of the electronic device.

9. The method according to claim 1, wherein: After obtaining the hand motion of the user when the user intends to operate the electronic device, the method further includes: Within the visual field, displaying an operation interface in a surrounding area of ​​the electronic device; the operation interface includes a plurality of controls; Controlling the electronic device according to the hand movement of the user includes: When it is detected that the hand action operates a target control among the multiple controls, the electronic device is controlled based on a control command corresponding to the target control.

10. The method according to claim 1, wherein: The hand motion includes a plurality of gestures; and controlling the electronic device according to the hand motion of the user includes: When it is detected that the holding time of the last gesture among the multiple gestures reaches a preset time, the electronic device is controlled according to the hand movement of the user.

11. The method according to claim 1, further comprising: In the surrounding area of ​​the electronic device within the field of view, the current switch state of the electronic device and / or the current operating parameters of the electronic device are displayed.

12. The method according to claim 1, wherein: Within the field of view, the at least one electronic device is highlighted.

13. The method according to claim 12, wherein: The highlighting method is any one of the following: a stroke prompt method, a semi-transparent cover layer prompt method, and a highlight prompt method.

14. An electronic device comprising: a processor and a memory for storing instructions executable by the processor; The processor is configured to execute the instructions so that the electronic device executes the control method based on the augmented reality (AR) device according to any one of claims 1 to 13.

15. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on the communication device, the communication device executes the control method based on the augmented reality (AR) device according to any one of claims 1 to 13.

Images